Jon Harris' most recent blog posting reminded me of many conversations that I have had with colleagues regarding software quality assurance as applied to automated performance testing. Jon’s blog posting questions whether programming skills are required for a successful automated performance testing initiative.  It reminded me of how I always get weary when I encounter marketing campaigns or sales pitches for automated performance testing “solutions” that claim "No technical skills are necessary!", "Start testing your application within 30 minutes!", "Click, Record, Test, Done!", etc. These claims are very misleading and quite frankly undermine the perceived level of effort and qualifications required for value-added performance testing. How comfortable would you be if someone offered to do your complicated federal income taxes, if they simply had an electronic calendar in their hand? Would you feel comfortable with the claim that this can be done in 30 minutes? Heck, it takes a lot more knowledge than knowing how to use a calculator to submit that federal tax return and it certainly takes more than 30 minutes! So, simply having a tool accessible is not going to provide added value; it actually can cause more trouble (i.e. think tax audit). Education, training, mentoring, and experience will provide the foundation of managing a wide cross-section of varying income portfolios; it's not a one-size-fits-all solution. Even when software packages, such as TurboTax, are provided to streamline the process, there is almost always a unique situation that requires the tax preparer to adapt. So, what does this have to do with using tools, such as HP LoadRunner, IBM Rational Performance Tester, Microsoft VSTS, Apache JMeter, Micro Focus QALoad, etc.? It's just that; these products are all just tools. They become a solution when used in conjunction with an adaptive methodology that meets both the business goals of the target application and its technical requirements. And the combinations of business requirements and technical requirements are limitless and can’t be grouped into a single turn-key solution. Technical skills are necessary to manage constantly evolving technology and non-technical skills are needed to address the business requirements of the application. These are skills that don't come in shrink-wrapped boxes or an electronic download. And they definitely don't evolve in the time it takes for a Domino's Pizza Delivery. How can we start to dissuade this negative perception towards automated performance testing and set better expectations?

I’m sure I am not the only one that brings my career experience into my daily non-work life. So, when I attempted to send several text messages to friends and family members shortly after midnight this past New Year’s Eve (12/31/2007), I was not surprised to encounter messages on my cell phone stating System Busy or Failed to Send Message. It seems that many of us in the larger metropolitan areas had the same idea at the same time and there was just not enough capacity to handle the volume of calls and text messages. So, it was at this point in the evening, over a glass of tasty sparkling white wine, and after wishing good tidings with my newlywed wife, that I had some thoughts pertaining to the technologies and challenges that I will probably encounter within the upcoming year, similar to a press release  that RTTS distributed discussing similar concepts.

Here were some of my thoughts before resuming my social engagements for the evening:

• RIA Rises - Rich Internet Applications using Web 2.0 technologies will probably make performance and scalability testing more challenging due to the dispersion of physical and logical data sources, the complexities that are now being pushed onto the web clients and the quick paced evolution of communication protocols, including SOAP, REST, JSON, and XML-RPC, amongst others. I don’t think that the Click-N-Test Methodology is going to be very effective in this arena.
• Enter the Enterprise - Lately I’ve been seeing corporations creating Intranets that go way beyond the role of providing static documents. Intranets are starting to be integrated into the role of the business, providing process workflow, enabling access to corporate applications, establishing social networks, and facilitating collaboration via Wikis. If providing this functionality to a business with hundreds or thousands of employees, performance becomes a critical factor in determining the success of the application, the productivity of employees, and in some cases corporate morale. Again, applying the principles of Performance-Testing-In-30-Minutes-Or-Less will not work to one's advantage.
• Meet the Mainframe - No, I don’t think that Mainframe systems are on the rise again.  However, with the momentum of server virtualization picking up, the Mainframe paradigm is taking hold again. Server consolidation will mean that great care needs to be taken when establishing application capacity and assigning system resources to components that are going to share a physical platform.

So, if I had to sum up my thoughts, I would say that performance testing will become even more critical as business and technology continue to merge, and automated performance testing will become even more challenging as result of the increasing complexity of this generation of applications.

What are your thoughts pertaining to these technologies and challenges?  Which other ones do you expect to encounter within the upcoming year?

Black Friday. Cyber Monday. Green Monday. What's next, Plaid Friday?  I don't think I could ever slip in a reference to Spaceballs, but hey I'll try anything for a laugh, or two.

Anyway, it's been quite some time since my last entry, but it's not due to lack of eagerness. I have simply been engrossed in a couple of time-sensitive customer engagements, during which we assisted with holiday e-commerce preparations and an enterprise rollout of a Microsoft SharePoint solution. So, my free time has been costly and as a result this holiday season is rushing past me at ludicrous speed. And I have for the first time purchased all of my holiday gifts through online vendors. There are lots of reports out there publishing positive news for this 2007 Holiday Shopping Season and sales seem to be projected to grow by 20% when compared to last year's online purchases. I have heard of several e-commerce vendors that have experienced significant customer dissatisfaction due to sluggish responses and/or the inability to complete the checkout process due to some form of application error. However, it seems that the vendors that had issues during the 2006 Holiday Shopping Season have prepared themselves very well. One of my personal favorites, Overstock.com, was featured in several articles discussing its investment of almost ten times the amount of resources in previous years towards the goal of better application performance management. And yes, I did shop there this season. For the most part I have had few issues with the large online vendors, most of which occurred on Black Friday. From a personal experience, what I do see is that some of the brick 'n mortar vendors may be experiencing an unanticipated deluge in sales. Many of these smaller web sites have been slow, but even more significant, the order fulfillment process has been painful; I am still waiting for several gifts from these smaller vendors.

So, why did I make the Back-To-School reference? Well, there was a published report this past September 2007 that Online Back-To-School Sales were ranked second in volume when compared to the holiday shopping season. If there were any vendors that experienced issues this past Back-To-School shopping season, I urge them to see what Overstock.com has accomplished. Their plans took some time to implement, but I feel it was well worth it, since I continue to be a return customer.

Happy Holidays!

Well these days it seems that you are even lucky to get that apathetic acknowledgement of what is happening. It’s very frustrating to know that “something” is happening, but not to get any feedback or information pertaining to your delay. If someone, please someone just stand up and explain to us what is happening. Inform us. Set expectations. Manage my satisfaction. Keep me as a customer. That’s fairly simple, uh?
 
Segue to application performance and customer satisfaction…

So, this morning when I boot up my computer and try to load any program, such as Firefox, nothing happens. I try again. I still see nothing. Gees, I just want to see if that flat screen TV that I ordered had been shipped. Nada. Niente. Nothing. I see nothing except my Desktop and its lineup of icons, but I can hear the hard disk churning away. What is happening? Cue Microsoft Task Manager. Exit stage left LIRR (for now). Microsoft Task Manager is a nifty program that allows me to quickly glance at how my computer’s resources, such as the memory utilization and CPU utilization (amongst other things), are being used. When Task Manager finally loads, I can start to poke around and see what is causing my hard disk to spin like the dickens. I look at the process list. I sort the list by CPU to see what process is currently requiring the most horsepower on my computer. Low and behold, it’s one of the well known antivirus programs. As it turns out, it was performing a system wide scan on my computer. OK, that’s why I have this software; to look for virus and spyware that might be lurking in the depths of my computer. But again, just like the gate agent on my most recent flight who couldn’t seem to manage an answer why we were still waiting to board the plane after 2 hours, why couldn’t this program actually inform me what it was doing? Why couldn’t a simple notification that it was scanning my computer be produced? Maybe a simple message box could have sufficed? Let me know what’s happening.  Managing customer expectations goes well beyond running load and volume tests in order to determine application bottlenecks. In the more holistic process of software performance engineering, providing feedback is a key attribute in keeping end-users satiated.

So, I’m not here to pick on this antivirus program, but the lack of feedback from software is more of a norm, rather than the exception.  When I submitted my credit card for that flat screen TV the e-commerce web site that I was using gave me no indication of what it was doing while I waited for 20 to 30 seconds to get a confirmation web page back. I think that flag or globe in the Internet Explorer toolbar was waving or doing something to indicate that I actually did submit my payment information. Or did it? I tell you it was not comforting knowing that I just made a large financial investment over the internet as I waited for some clue of what was going on. All I did know is that I just clicked a button to send my credit card information to some remote location and I all had to show for it so far was a blank screen in my web browser. I was thinking, “should I hit the back button and tried it again?”.  I’m sure I’m not alone in thinking this way. And I’m sure that non-technical people are even more perplexed on what to do. What could have been done? Perhaps the web application that I was using could have launched a little progress window stating that it was doing something, rather than relying on the web browser’s flashy toolbar as a progress indicator. Perhaps rather than relying on the mouse cursor to change to an hourglass, the application could have provided feedback that was pertinent to the operation it was currently processing. Perhaps the next time I won’t have to rely on Microsoft’s Task Manager to tell me that my antivirus program has taken over my computer to run a system scan.  I am betting that this not-so-groundbreaking realization of software performance engineering is much easier to implement when compared to getting some clue why my train is not moving. And I’m sure that the customer perceived performance of your application will be well revered, unlike applications that require people like me to use Task Manager to figure out what is going on.

"Houston, we've had a problem." This is a well known quote from Tom Hanks (??); no - it was really James Lovell during the Apollo 13 Mission in 1970, which was supposed to be the 3rd manned lunar-landing mission under NASA's Project Apollo. Although the mission was catastrophic at the time and almost fatal, this particular quote has lived on in infamy to be used in humorous contexts or in my case when I am commercially using or testing an application that starts behaving badly, prompting me with cryptic error messages, taking what seems like eons to respond to a request, or even just refusing to respond to me at all whatsoever. And for some reason this has occurred to me quite often within this past year. This past Black Friday 2006, both Wal-Mart’s and Amazon's e-commerce web applications experienced significant outages and performance-related issues. Well, I didn't want that Microsoft Xbox anyway! On Christmas Day 2006 I could not even create a new account for my nephew on Apple iTunes, so that he could download songs for his brand new Apple iPod. Thankfully we were able to download songs from Green Day the following weekend. Even more recently, using Turbo Tax, 2006 tax returns could not be electronically submitted to the IRS starting several hours prior to the tax deadline. Well, I guess I deserved it for waiting until the last minute. That flat panel TV that I was going to buy using my tax refund would just have to wait a little longer. What's going on here?! Well, as it turns out, these companies just did not anticipate the large demand for their services and merchandise. Whether it was due to stellar marketing, an outstanding sales force, or drastic price slashing for 20-minute sales, their computing infrastructure was just not sized and/or configured properly to sustain the subsequent onslaught of their customer’s demands. So, why am I here? Well, hopefully moving forward we can start to use this Blog to start discussing issues related to the ones I've highlighted here and conversing about better ways to manage the life cycle of application performance. I'm looking forward to see where this is going to lead us.