Speaking of social, I am sure that if you have not been hiding under a rock you have heard of Chatroulette (no link; if you're at work, you can thank me later). The 17-year-old creator had this to say about the phenomenal growth of his site:

from Brad Stone's New York Times weblog, by way of Andy Baio

Each time the user count grew, I had to rewrite my code completely, because my software and hardware couldn’t handle it all. I never thought that handling the heavy user load would be the most difficult part of my project.

Imagine having to re-write your whole architecture from scratch each time you want to scale to the next level. That may be fine for a one- or two-page site, but think about your own infrastructure, and how much an entire re-write would cost you. Andrey Ternofskiy is 17 years old. What's your excuse?

Jon Harris' most recent blog posting reminded me of many conversations that I have had with colleagues regarding software quality assurance as applied to automated performance testing. Jon’s blog posting questions whether programming skills are required for a successful automated performance testing initiative.  It reminded me of how I always get weary when I encounter marketing campaigns or sales pitches for automated performance testing “solutions” that claim "No technical skills are necessary!", "Start testing your application within 30 minutes!", "Click, Record, Test, Done!", etc. These claims are very misleading and quite frankly undermine the perceived level of effort and qualifications required for value-added performance testing. How comfortable would you be if someone offered to do your complicated federal income taxes, if they simply had an electronic calendar in their hand? Would you feel comfortable with the claim that this can be done in 30 minutes? Heck, it takes a lot more knowledge than knowing how to use a calculator to submit that federal tax return and it certainly takes more than 30 minutes! So, simply having a tool accessible is not going to provide added value; it actually can cause more trouble (i.e. think tax audit). Education, training, mentoring, and experience will provide the foundation of managing a wide cross-section of varying income portfolios; it's not a one-size-fits-all solution. Even when software packages, such as TurboTax, are provided to streamline the process, there is almost always a unique situation that requires the tax preparer to adapt. So, what does this have to do with using tools, such as HP LoadRunner, IBM Rational Performance Tester, Microsoft VSTS, Apache JMeter, Micro Focus QALoad, etc.? It's just that; these products are all just tools. They become a solution when used in conjunction with an adaptive methodology that meets both the business goals of the target application and its technical requirements. And the combinations of business requirements and technical requirements are limitless and can’t be grouped into a single turn-key solution. Technical skills are necessary to manage constantly evolving technology and non-technical skills are needed to address the business requirements of the application. These are skills that don't come in shrink-wrapped boxes or an electronic download. And they definitely don't evolve in the time it takes for a Domino's Pizza Delivery. How can we start to dissuade this negative perception towards automated performance testing and set better expectations?

Does the sun rise in the morning?  Ok, that's not fair unless I am more specific. I've been reading articles/blogs/opinions/comments, etc on whether testers need programming skills. All of these articles make sound arguments and all are logical and on and on. I agree that testing skills and programming skills are two totally different things and that a good tester may not necessarily be a good programmer (or vice versa). The context of all of these is in regards to functional testing.  When it comes to performance testing, the answer must be that the tester has to have programming skills.  They do not need to be a rocket scientist in the language, and I tell everyone who asks and who I teach, that they need to be proficient in string manipulation coding.  Every tool I know and use has a programming language behind it (C, C++, C#, Java, JSP, VB, VB.NET, and a few proprietary) to extend the tool, which at some point in every engagement comes into play.  The only way around programming would be if someone  came up with a performance testing tool that used artificial intelligence so that you did not need custom coding to get an automated transaction to work - and they  would corner the market.!

Just one additional skill that is needed in conjunction with programming expertise is the ability to analyze and find patterns. Without these you would not be able to use your programming skills to manipulate a server response to get what you need.

As usual, if you have any comments or anything to add, please feel free to do so.

When Microsoft released Visual Studio Team System (VSTS) 2005, I was excited to take a look at the new web testing component. Over the past 20+ years, I have used most of the performance testing tools on the market, so typically I first look at the basic functionality; how to record, how to playback and what numbers it gives me. I was pleasantly surprised at the ease of use and depth of functionality I found. The only awkward feature I mucked around with was pulling data from external files (datapools). The setup of this was not straightforward. The newest release, VSTS 2008, makes this task simple and very straightforward, so kudos to that enhancement. RTTS has performed numerous highly successful engagements utilizing VSTS because of its rich feature set and the price is attractive for clients. I am looking forward to the next release in 2010.

For anyone out there using VSTS, how do you like the tool?

I've been using IBM Rational Performance Tester (RPT) since it was the Prevue tool long ago. I've used it in many ways, outside of the box, so to speak, for things other than performance testing. One of those things is for monitoring, specifically for production monitoring to verify the target system(s) are up and running correctly. Currently I am involved in an engagement utilizing IBM's ITCAM (IBM Tivoli Composite Application Management) product. As part of its use, we are creating RPT tests (tests is the term used in RPT for automated scripts). For those who may use ITCAM in the future, here are a few up-front working tips. 

(1)  After creating tests in RPT, you upload them to ITCAM. ITCAM will only see tests located in the root directory of the project, so you must place the final working versions there.

(2) ITCAM will only see custom code located in the "src\test" directory, so make sure you keep them in this default location.

(3) ITCAM does not currently see additional Java classes you create and import into your custom code (i.e. utility/reusable functions). You can get around this by creating a new custom code file and adding it to the beginning of the test. Don't add any executable code to the "exec" function. Place all of your utility functions after the "exec" function. ITCAM will now upload the custom code and since it is located in the default "src\test" location, you do not need to "import" it into the other custom code you create.

I hope this helps some. If anyone has similar tips regarding RPT use in ITCAM, please feel free to share in the comments.

I recently read a blog entry by Jason Gorman on "Architecture vs. Design - An Artificial Distinction?" where he talked about the scope of architecture and design being arbitrary and the meaning ultimately resides on your point of view. I agree 100% that almost anything is arbitrary based on your point of view, but being in the software testing arena, architecture and design are two totally different things. The distinction comes into play based on a timeline. In the beginning, software and systems are designed (conceptualized) as to what they are intended to do and what they are supposed to support. The architecture of these applications and systems denote their implementation. Many definitions of design talk about planning. Definitions of architecture talk about structure. It makes me think about the comic strip that came out many years ago where each cell in the strip shows the interpretation of a tree and swing by each of the producers.

As proposed by the project sponsor

As specified in the project request

As designed by the senior analyst

As produced by the programmers

As installed at the user's site

What the user wanted

From conception to delivery you never really know what you'll get. In the end in my world, the proof is in the pudding. Design is one thing, but what you end up with can be something entirely different. When I teach methodology, my definition of architecture is that it defines how components work, how they interact with other components and how we interact with them. Ultimately, architecture determines how well or poor a complete solution will work. There are so many ways to put today's technologies together and depending upon the circumstances, needs, resources, requirements, etc., a system with an elegant design may or may not be put into architectural practice to yield the best possible results. Again, my position on architecture and design is that there is a definite distinction.

Now you know my position on architecture and design, what's yours?

I’m sure I am not the only one that brings my career experience into my daily non-work life. So, when I attempted to send several text messages to friends and family members shortly after midnight this past New Year’s Eve (12/31/2007), I was not surprised to encounter messages on my cell phone stating System Busy or Failed to Send Message. It seems that many of us in the larger metropolitan areas had the same idea at the same time and there was just not enough capacity to handle the volume of calls and text messages. So, it was at this point in the evening, over a glass of tasty sparkling white wine, and after wishing good tidings with my newlywed wife, that I had some thoughts pertaining to the technologies and challenges that I will probably encounter within the upcoming year, similar to a press release  that RTTS distributed discussing similar concepts.

Here were some of my thoughts before resuming my social engagements for the evening:

• RIA Rises - Rich Internet Applications using Web 2.0 technologies will probably make performance and scalability testing more challenging due to the dispersion of physical and logical data sources, the complexities that are now being pushed onto the web clients and the quick paced evolution of communication protocols, including SOAP, REST, JSON, and XML-RPC, amongst others. I don’t think that the Click-N-Test Methodology is going to be very effective in this arena.
• Enter the Enterprise - Lately I’ve been seeing corporations creating Intranets that go way beyond the role of providing static documents. Intranets are starting to be integrated into the role of the business, providing process workflow, enabling access to corporate applications, establishing social networks, and facilitating collaboration via Wikis. If providing this functionality to a business with hundreds or thousands of employees, performance becomes a critical factor in determining the success of the application, the productivity of employees, and in some cases corporate morale. Again, applying the principles of Performance-Testing-In-30-Minutes-Or-Less will not work to one's advantage.
• Meet the Mainframe - No, I don’t think that Mainframe systems are on the rise again.  However, with the momentum of server virtualization picking up, the Mainframe paradigm is taking hold again. Server consolidation will mean that great care needs to be taken when establishing application capacity and assigning system resources to components that are going to share a physical platform.

So, if I had to sum up my thoughts, I would say that performance testing will become even more critical as business and technology continue to merge, and automated performance testing will become even more challenging as result of the increasing complexity of this generation of applications.

What are your thoughts pertaining to these technologies and challenges?  Which other ones do you expect to encounter within the upcoming year?

Black Friday. Cyber Monday. Green Monday. What's next, Plaid Friday?  I don't think I could ever slip in a reference to Spaceballs, but hey I'll try anything for a laugh, or two.

Anyway, it's been quite some time since my last entry, but it's not due to lack of eagerness. I have simply been engrossed in a couple of time-sensitive customer engagements, during which we assisted with holiday e-commerce preparations and an enterprise rollout of a Microsoft SharePoint solution. So, my free time has been costly and as a result this holiday season is rushing past me at ludicrous speed. And I have for the first time purchased all of my holiday gifts through online vendors. There are lots of reports out there publishing positive news for this 2007 Holiday Shopping Season and sales seem to be projected to grow by 20% when compared to last year's online purchases. I have heard of several e-commerce vendors that have experienced significant customer dissatisfaction due to sluggish responses and/or the inability to complete the checkout process due to some form of application error. However, it seems that the vendors that had issues during the 2006 Holiday Shopping Season have prepared themselves very well. One of my personal favorites, Overstock.com, was featured in several articles discussing its investment of almost ten times the amount of resources in previous years towards the goal of better application performance management. And yes, I did shop there this season. For the most part I have had few issues with the large online vendors, most of which occurred on Black Friday. From a personal experience, what I do see is that some of the brick 'n mortar vendors may be experiencing an unanticipated deluge in sales. Many of these smaller web sites have been slow, but even more significant, the order fulfillment process has been painful; I am still waiting for several gifts from these smaller vendors.

So, why did I make the Back-To-School reference? Well, there was a published report this past September 2007 that Online Back-To-School Sales were ranked second in volume when compared to the holiday shopping season. If there were any vendors that experienced issues this past Back-To-School shopping season, I urge them to see what Overstock.com has accomplished. Their plans took some time to implement, but I feel it was well worth it, since I continue to be a return customer.

Happy Holidays!

How far from the truth is that? When it comes to sophisticated performance testing tools, and even the not so sophisticated ones, how many times has anyone been able to just use the out-of-the-box functionality of a testing tool to accurately and thoroughly test an application? Let me guess at this... (thinking) (thinking) (thinking)... None! Ok, in all fairness, there are rare instances when someone tests a simple application, really simple (like a static web site). Anyway, here is my thought as I was flying home from a recent mentoring engagement.

Actually, this applies to all of the tools I use or have used. They all pretty much have the same base functionality. They record, they play back, and they have some form of programming language (compiled or interpreted). The implementation of these functionalities differs, of course. Regarding programming related functionality, in most cases if a specific functionality is not there, you can usually program around it (yeah, right!). For the newbie’s out there, programming around it only applies if you know it is something you need. This is all well and good, but what I was thinking about was the fact that I need to keep programming (or using) the same custom functionality.

Specifically, since the mentoring I was just doing was against a web application, I needed to use custom code functions that are typical and customary to doing web testing. For instance, pulling a list of values from a listbox, randomly picking one and using it in the next request. What about a function to randomly pick a link on a web page? Wouldn't it be great if there was a function that performed this act that comes with the test tool? Of course there are always unique situations for each application that will require custom coding, but the typical things like I just mentioned would serve 2 purposes. First it is code you do not need to maintain or carry with you (importing) and second if it is coded into the test tool itself, it would probably be just as efficient or maybe even more efficient than your custom code (assuming the tool vendors have the right programmers and/or you’re not dealing with interpreted script code).

This blog entry was geared towards newbies since I was recently training people new to these tools. They were all programmers in some form or other (very technical, which is always a pleasure when teaching a technical subject). So for typical code neccessities, let’s have the tool functionality built in. But for custom code – hey, all die-hard programmers out there, we all love our own code creations, right? (wink, wink)

Well these days it seems that you are even lucky to get that apathetic acknowledgement of what is happening. It’s very frustrating to know that “something” is happening, but not to get any feedback or information pertaining to your delay. If someone, please someone just stand up and explain to us what is happening. Inform us. Set expectations. Manage my satisfaction. Keep me as a customer. That’s fairly simple, uh?
 
Segue to application performance and customer satisfaction…

So, this morning when I boot up my computer and try to load any program, such as Firefox, nothing happens. I try again. I still see nothing. Gees, I just want to see if that flat screen TV that I ordered had been shipped. Nada. Niente. Nothing. I see nothing except my Desktop and its lineup of icons, but I can hear the hard disk churning away. What is happening? Cue Microsoft Task Manager. Exit stage left LIRR (for now). Microsoft Task Manager is a nifty program that allows me to quickly glance at how my computer’s resources, such as the memory utilization and CPU utilization (amongst other things), are being used. When Task Manager finally loads, I can start to poke around and see what is causing my hard disk to spin like the dickens. I look at the process list. I sort the list by CPU to see what process is currently requiring the most horsepower on my computer. Low and behold, it’s one of the well known antivirus programs. As it turns out, it was performing a system wide scan on my computer. OK, that’s why I have this software; to look for virus and spyware that might be lurking in the depths of my computer. But again, just like the gate agent on my most recent flight who couldn’t seem to manage an answer why we were still waiting to board the plane after 2 hours, why couldn’t this program actually inform me what it was doing? Why couldn’t a simple notification that it was scanning my computer be produced? Maybe a simple message box could have sufficed? Let me know what’s happening.  Managing customer expectations goes well beyond running load and volume tests in order to determine application bottlenecks. In the more holistic process of software performance engineering, providing feedback is a key attribute in keeping end-users satiated.

So, I’m not here to pick on this antivirus program, but the lack of feedback from software is more of a norm, rather than the exception.  When I submitted my credit card for that flat screen TV the e-commerce web site that I was using gave me no indication of what it was doing while I waited for 20 to 30 seconds to get a confirmation web page back. I think that flag or globe in the Internet Explorer toolbar was waving or doing something to indicate that I actually did submit my payment information. Or did it? I tell you it was not comforting knowing that I just made a large financial investment over the internet as I waited for some clue of what was going on. All I did know is that I just clicked a button to send my credit card information to some remote location and I all had to show for it so far was a blank screen in my web browser. I was thinking, “should I hit the back button and tried it again?”.  I’m sure I’m not alone in thinking this way. And I’m sure that non-technical people are even more perplexed on what to do. What could have been done? Perhaps the web application that I was using could have launched a little progress window stating that it was doing something, rather than relying on the web browser’s flashy toolbar as a progress indicator. Perhaps rather than relying on the mouse cursor to change to an hourglass, the application could have provided feedback that was pertinent to the operation it was currently processing. Perhaps the next time I won’t have to rely on Microsoft’s Task Manager to tell me that my antivirus program has taken over my computer to run a system scan.  I am betting that this not-so-groundbreaking realization of software performance engineering is much easier to implement when compared to getting some clue why my train is not moving. And I’m sure that the customer perceived performance of your application will be well revered, unlike applications that require people like me to use Task Manager to figure out what is going on.

"Houston, we've had a problem." This is a well known quote from Tom Hanks (??); no - it was really James Lovell during the Apollo 13 Mission in 1970, which was supposed to be the 3rd manned lunar-landing mission under NASA's Project Apollo. Although the mission was catastrophic at the time and almost fatal, this particular quote has lived on in infamy to be used in humorous contexts or in my case when I am commercially using or testing an application that starts behaving badly, prompting me with cryptic error messages, taking what seems like eons to respond to a request, or even just refusing to respond to me at all whatsoever. And for some reason this has occurred to me quite often within this past year. This past Black Friday 2006, both Wal-Mart’s and Amazon's e-commerce web applications experienced significant outages and performance-related issues. Well, I didn't want that Microsoft Xbox anyway! On Christmas Day 2006 I could not even create a new account for my nephew on Apple iTunes, so that he could download songs for his brand new Apple iPod. Thankfully we were able to download songs from Green Day the following weekend. Even more recently, using Turbo Tax, 2006 tax returns could not be electronically submitted to the IRS starting several hours prior to the tax deadline. Well, I guess I deserved it for waiting until the last minute. That flat panel TV that I was going to buy using my tax refund would just have to wait a little longer. What's going on here?! Well, as it turns out, these companies just did not anticipate the large demand for their services and merchandise. Whether it was due to stellar marketing, an outstanding sales force, or drastic price slashing for 20-minute sales, their computing infrastructure was just not sized and/or configured properly to sustain the subsequent onslaught of their customer’s demands. So, why am I here? Well, hopefully moving forward we can start to use this Blog to start discussing issues related to the ones I've highlighted here and conversing about better ways to manage the life cycle of application performance. I'm looking forward to see where this is going to lead us.