Jon has made performance and scalability testing the focus of his career since 1988. He has completed hundreds of engagements encompassing thousands of tests against applications from every major industry and then some. To him, technology is a playground to which the rules of the game are constantly changing... Jon works out of the Arizona office (360+ sunny days a year; sorry east coasters) and lives in Goodyear with his wife and college son (Go ASU).
Do testers need Programming Skills?
Does the sun rise in the morning? Ok, that's not fair unless I am more specific. I've been reading articles/blogs/opinions/comments, etc on whether testers need programming skills. All of these articles make sound arguments and all are logical and on and on. I agree that testing skills and programming skills are two totally different things and that a good tester may not necessarily be a good programmer (or vice versa). The context of all of these is in regards to functional testing. When it comes to performance testing, the answer must be that the tester has to have programming skills. They do not need to be a rocket scientist in the language, and I tell everyone who asks and who I teach, that they need to be proficient in string manipulation coding. Every tool I know and use has a programming language behind it (C, C++, C#, Java, JSP, VB, VB.NET, and a few proprietary) to extend the tool, which at some point in every engagement comes into play. The only way around programming would be if someone came up with a performance testing tool that used artificial intelligence so that you did not need custom coding to get an automated transaction to work - and they would corner the market.!
Just one additional skill that is needed in conjunction with programming expertise is the ability to analyze and find patterns. Without these you would not be able to use your programming skills to manipulate a server response to get what you need.
As usual, if you have any comments or anything to add, please feel free to do so.
Posted by Jonathan Harris on Thursday, June 25, 2009 2:27 PM EDT
Leave a comment | View Comments (0)
Getting It Right The First Time
When Microsoft released Visual Studio Team System (VSTS) 2005, I was excited to take a look at the new web testing component. Over the past 20+ years, I have used most of the performance testing tools on the market, so typically I first look at the basic functionality; how to record, how to playback and what numbers it gives me. I was pleasantly surprised at the ease of use and depth of functionality I found. The only awkward feature I mucked around with was pulling data from external files (datapools). The setup of this was not straightforward. The newest release, VSTS 2008, makes this task simple and very straightforward, so kudos to that enhancement. RTTS has performed numerous highly successful engagements utilizing VSTS because of its rich feature set and the price is attractive for clients. I am looking forward to the next release in 2010.
For anyone out there using VSTS, how do you like the tool?
Posted by Jonathan Harris on Wednesday, March 11, 2009 11:52 AM EDT
Leave a comment | View Comments (0)
Building a Reliable Scripting Framework
Every performance engagement I do is both unique and repetitive. Unique in regards to a new application in a new test environment. By new I mean that it's different from any other test I've done before, even if some or many of the components are similar. They are brought together and used differently from site to site. By repetitive I am referring to the methodology I use to get the job done. Even though the process is the same, the details of getting the tasks completed change.
In every engagement I do, during the scripting phase when I create the automated test scripts, I automatically think of ways to make the scripts robust and reliable. Robust in the way it is scripted so that if changes are needed, it would have the least amount of impact on time and complexity to make the changes. Making the script reliable means handling known and possibly unknown conditions that would otherwise hang or stop the user. There are other factors in making a script reliable as well, such as after detecting issues having the ability to report on them. This way you can roll it back into the reliability of the script for next time it occurs.
This is something I have been doing for a long time, but recently put a name to it; a "Reliable Scripting Framework." Here is an example. Let's say you are testing a Citrix application and there are issues when running large numbers of virtual-users from a single test driver. This has something to do with the ICA middleware not being written to support many connections from a single machine. Sometimes typed characters get lost. This can cause the script to get an unexpected response. You have two options: (1) add code to detect that something went wrong and attempt to correct it, or (2) add code to test that what you typed was correct and avoid the error in the first place. Both approaches are acceptable, but it would take less thought and code to avoid the problem in the first place.
Here is a second example. In web testing there are times when a request fails intermittently due to a network or application glitch. If you try the same call again, it works perfectly. Though these are serious issues to attend to, when you performance test, having the ability to report on the error and retry would be nice. In most test tools you need to handle this yourself in code. If you do not explicitly handle it you may end up getting cascading errors which muddy up the waters when you try to analyze what happened. So putting a framework in place to handle something like this would make the script robust and more reliable.
If you have any form of scripting framework you use, I'd like to hear about it in the comments. Take a look in the near future for a whitepaper I'm writing about creating a Reliable Scripting Framework using these two examples.
Posted by Jonathan Harris on Wednesday, February 11, 2009 2:21 PM EST
Leave a comment | View Comments (0)
Rational Performance Tester Tips for use with ITCAM
I've been using IBM Rational Performance Tester (RPT) since it was the Prevue tool long ago. I've used it in many ways, outside of the box, so to speak, for things other than performance testing. One of those things is for monitoring, specifically for production monitoring to verify the target system(s) are up and running correctly. Currently I am involved in an engagement utilizing IBM's ITCAM (IBM Tivoli Composite Application Management) product. As part of its use, we are creating RPT tests (tests is the term used in RPT for automated scripts). For those who may use ITCAM in the future, here are a few up-front working tips.
(1) After creating tests in RPT, you upload them to ITCAM. ITCAM will only see tests located in the root directory of the project, so you must place the final working versions there.
(2) ITCAM will only see custom code located in the "src\test" directory, so make sure you keep them in this default location.
(3) ITCAM does not currently see additional Java classes you create and import into your custom code (i.e. utility/reusable functions). You can get around this by creating a new custom code file and adding it to the beginning of the test. Don't add any executable code to the "exec" function. Place all of your utility functions after the "exec" function. ITCAM will now upload the custom code and since it is located in the default "src\test" location, you do not need to "import" it into the other custom code you create.
I hope this helps some. If anyone has similar tips regarding RPT use in ITCAM, please feel free to share in the comments.
Posted by Jonathan Harris on Monday, January 26, 2009 12:02 PM EST
Leave a comment | View Comments (0)
Tools that make consulting a pleasure
Over the years I have had many different laptop computers that I used on client sites. One of the nicer ones was a Dell Inspiron with a 15-inch screen and a 1600x1200 screen resolution. The drawback was it weighed over 10 pounds. In recent years I downsized my machines to a 12-inch screen with a 1280x800 resolution. I did this for a few reasons. First was that they only weighed about five pounds and second is that on my numerous plane trips I could actually open it up without having it pressed against my stomach; especially when the person in front of me reclined their seat.
Well, over the past year and a half I flew enough to attain platinum level status in the frequent flyer program for US Airways. I am getting upgraded to first class ahead of the hordes of gold level people. So I upgraded my laptop from a Dell XPS1210 to an Inspiron 1720. This baby rocks. It has a 17-inch screen with a 1900x1200 resolution. Here is the kicker; it has a full keyboard with a numeric keypad. It is a pleasure to use on client sites. I outfitted it with VMware so that I can create a virtual machine and load whatever tools I need in a matter of minutes, and when I'm done I just blow it away.
For added toys of convenience I found a USB powered network hub with four ports. I can't tell you how many times I was in a room with other consultants and there was only a single network cable. I also purchased a USB mobile battery to power USB devices (and my cell phone) and a USB fan and light. To round it all off I got a 320 GB mini drive. With all of this cool stuff I bought a rolling laptop case that also fits nicely on top of my suitcase for the airport (until I check my luggage). Now I have my sights on a mobile printer. I'm looking at a Cannon BJC85.
Does anyone have a BJC85 that can share their opinions of it? What about any cool gadgets or software?
Posted by Jonathan Harris on Thursday, August 28, 2008 11:46 PM EDT
Leave a comment | View Comments (0)
Not seeing the forest for the trees
Seeing the bigger picture is an important skill in all aspects of business (and life). Many times I see someone performing a task a certain way over and over, while I’m thinking “Why not do it this way?” or, “If you do this part of the task this way once, the rest would be easier and repeating it would be more efficient.” What makes me smack my head and proclaim "Doh!" is when I'm the recipient of such advice. Sometimes when you are down in the thick of detail, you don't see the forest for the trees. In my last blog entry I talked about statements-of-work (SOWs). Because creating them is a significant component of my work, here is one head-smacker to add; and by the way it also falls under the heading of Evaluation and Improvement, both part of the RTTS methodology (Doh! times two).
Usually, after creating and submitting the SOW we have a follow-up call with the prospective client to discuss the document. We seem to always get into a conversation on process, needs, requirements and tools. Even though I enjoy talking about performance testing and tools, it would probably be a good idea to include an explanation of the previously mentioned topics within the SOW itself. For example, regarding process, describe the steps and artifacts; what is involved, setup and requirements, etc. Since there are a number of performance and scalability test tools available, the SOW could include a generalization of what the available tools are; how they work; and what they can and cannot tell you.
Fitting title for the subject. Have you had any head smacking Doh's lately?
Posted by Jonathan Harris on Monday, April 28, 2008 5:19 PM EDT
Leave a comment | View Comments (0)
Following Directions and Asking the Right Questions
I write quite a few Statements of Work (SOW); some call them proposals and I'm sure there are a number of other terms as well. The RTTS format I follow fits perfectly with its intended response; acceptance by the client or winning the project in a competitive situation. I take the client’s information and requirements, sift through them for relevance to the project at hand, and match the details against what can and cannot be done with automated performance and scalability testing. When communicating with potential clients and putting together the SOW, there are two aspects that must be covered even before getting to the “how long” and “how much” part. The first is that we understand their business motivations and have the methodologies in place to address them successfully. The second is that we understand their technical requirements and have the tools, skill sets, methodologies and a plan to successfully address them. However, I have found that not all companies maintain relevancy to the project at hand.
Recently, a prospective client issued a Request For Proposal (RFP) and supplied quite a bit of information regarding the testing environment, the application, business transaction volume requirements, expectations and instructions on what to include and exclude from the proposal and strategy presentation. Following the initial RFP was the opportunity for all responding vendors to ask questions. The unusual part, or at least out of the ordinary as I have not had this occur before, is that all of the questions and their answers were combined into a single response document and returned to all of the vendors. I believe this was in response to one of the questions asked by another vendor asking for access to all of the questions and answers (maybe to size up the competition, huh...).
After going through all of the questions and answers, I can surmise that there were maybe four or possibly five vendors involved. Some of the questions were duplicates that I had asked as well. Some of the questions, though valid, were coming out of left field from my perspective. One of the questions referred to additional services and tools when the RFP explicitly stated to include only tools and services needed to test the application. Another question asked for the procedure for reporting issues. As I can see it is an important point once the project starts, I cannot see its relevance in forming a testing strategy. One of the questions asked if the demonstration was against the client’s application. One of the requirements of the presentation was to do a demonstration of the tools selected for the testing with an overall length of the strategy presentation and demo not to exceed two hours. If you've ever performed a demo against a live application you have never seen before, it can take hours if not longer just set it up, not to mention the possibility of discovering the nuances of the script modifications needed to get it to play back correctly. There were a number of questions asked where the answers referred vendors back to information contained in the RFP. I can only guess whether the people asking the questions were testers or salesmen.
Anyway, from my perspective, it's all about decoding information, inferring, making educated hypotheses and putting together the best SOW possible. And, there are always good questions to ask, just make sure they are the right questions.
Posted by Jonathan Harris on Sunday, March 23, 2008 12:04 PM EDT
Leave a comment | View Comments (0)
Architecture vs. Design - An Artificial Distinction?
I recently read a blog entry by Jason Gorman on "Architecture vs. Design - An Artificial Distinction?" where he talked about the scope of architecture and design being arbitrary and the meaning ultimately resides on your point of view. I agree 100% that almost anything is arbitrary based on your point of view, but being in the software testing arena, architecture and design are two totally different things. The distinction comes into play based on a timeline. In the beginning, software and systems are designed (conceptualized) as to what they are intended to do and what they are supposed to support. The architecture of these applications and systems denote their implementation. Many definitions of design talk about planning. Definitions of architecture talk about structure. It makes me think about the comic strip that came out many years ago where each cell in the strip shows the interpretation of a tree and swing by each of the producers.
As proposed by the project sponsor
As specified in the project request
As designed by the senior analyst
As produced by the programmers
As installed at the user's site
What the user wanted
From conception to delivery you never really know what you'll get. In the end in my world, the proof is in the pudding. Design is one thing, but what you end up with can be something entirely different. When I teach methodology, my definition of architecture is that it defines how components work, how they interact with other components and how we interact with them. Ultimately, architecture determines how well or poor a complete solution will work. There are so many ways to put today's technologies together and depending upon the circumstances, needs, resources, requirements, etc., a system with an elegant design may or may not be put into architectural practice to yield the best possible results. Again, my position on architecture and design is that there is a definite distinction.
Now you know my position on architecture and design, what's yours?
Posted by Jonathan Harris on Wednesday, February 06, 2008 1:53 PM EST
Leave a comment | View Comments (0)
Road Warrior-ism
This may be a little off the blog topic as it does not refer to technology, but since I've have been away from home a lot lately on assignment, thought I’d say something about it. All hail the Road Warrior; those that get to travel all over the world to new and exciting places, and do all sorts of fantastic things. I've been a Road Warrior since I started consulting in the field of testing back in 1988 and whenever I talk with someone about where I've been and for who and what I did there, I get questions like "did you go see this?" or "did you go do that?" Of course they are referring the attractions somewhere within a hundred miles of where I was. In most cases the answer is, "no, I didn't really have time to, but I'll make time for it the next time I go." Now don't get me wrong, I do get to do some fun things. I've visited the airplane graveyard and the Biosphere in Arizona, taken the underground city tour in Seattle, dug for gold and gems in the north-east, gone to countless amusement parks, museums, visited famous beaches and have eaten at many high class restaurants; all of which have been made possible in conjunction with business trips. First and foremost I service the client who has contracted to have me there and in many cases I continue to work back at the hotel. For me this is great since my personal view of my career is that I get paid to play; the work is very challenging and I'm always getting into new situations and devising new solutions.
One of the greatest things about traveling is the frequent traveler programs. When I lived in New Jersey I flew out of Newark and used Continental Airlines since Newark was their hub. I made Gold level status year after year (almost), a couple of them I almost made platinum. Continental had a promotion that if are Gold for 5 years in a row, you stay gold for life. Wouldn't you know it, the fifth year I was on a long term assignment locally and didn't make it. Now I live new Phoenix, so I use US Air (formerly flew America West but bought out by US Air). I've made gold all five years since I've been here and this year will make platinum (RTTS has been busy). One of the benefits of the frequent flyer program is that you get to board the plane before non-frequent flyers, thus ensuring overhead space for your roll-aboard luggage. Others include, depending upon the status level you are, space available for free upgrades to first class and a tag added to any checked luggage marking it as priority so that it comes up on the baggage carrousel first. Also, depending upon your status level you accumulate a percentage of flight miles above and beyond the actual miles flown, which are good towards free flights.
Then there is the hotel frequent stayer program. I like staying at Marriott brands. There are status levels depending upon the number of nights you stay. I've been Gold on the Marriott Rewards program forever. You get points depending upon the amount of money you spend at the hotel that can be used for free nights and they even have a catalog of merchandise you can use your points on. Two of the perks of having elite status at Marriot are the free upgrades to bigger and better rooms and at the full service hotels, admittance to the concierge level and free food (continental breakfast and cocktail hour at night).
Then there are the frequent renter programs for vehicles. I typically use Avis. There aren't any formal status levels, but you get additional flight frequent flyer miles when you rent from them and free upgrades to better class cars. The biggest perk is that you don't have to wait on line. When you arrive, your car is ready with your contract inside.
The points I accumulated over the first 3 years as a road warrior, I was able to take my immediate family (my wife, 5 kids and me) to Cancun, Mexico. All of the flights and the 2 hotel rooms were free. Now that the family is spread out between Arizona, Florida and New Jersey, we use the points to visit the kids and for the kids to visit us. Priorities change over time.
Enter the demons. I was waiting at the gate for a flight recently, watching all of the people and started thinking about traveling. What I find is that when its getting close to boarding time I automatically migrate as close to the jet way door as I can get without crowding anybody, knowing I am elite status and can get on the plane sooner than most. This particular time I was upgraded to first class. The normal process is that those needing assistance are called to board, followed by first class customers, then it’s by boarding zone number. I start to get irritated by the zone 4 person standing between me and the jet way door. Why are they in the way of me and the others who can board before them? Don't they know the boarding sequence? I plan my route on how to get around them when they call first class to board. I'll say "excuse me" and proceed around to their right. Then a woman behind me asks if I am in first class. I say "yes" thinking to myself "first class and elite, thank you very much!" She was in first class as well, but it made me think that you really cannot judge by looks since I see all kinds of people in first class dressed in the full range of attire.
When arriving at my destination airport, my checked bag was one of the last to come up onto the carrousel; so much for tagging it priority. Irritated, but outwardly showing it, I headed to the rental car shuttle. What I've learned is that some airports appear to honor the orange "Priority" tags and some don't. Case in point, US Air at Newark airport never does.
OK! I have an "Avis First" priority reservation. I've had it for a week now. I was here last week and the week before without an issue. I'm dropped off at the "Preferred" counter and have to show my driver’s license and credit card, etc. Hello! Where's the consistency?
I am assuming for the most part that you probably don't know me personally and have not met me personally. I am not one that tends to get stressed or irritated as I mention here. I believe things happen and we just have to navigate our way around them and through them. I have always enjoyed traveling as part of my job and have no plans to stop. The benefits, both for career and personal endeavors, make those momentary demons seem insignificant.
Posted by Jonathan Harris on Sunday, December 16, 2007 9:56 PM EST
Leave a comment | View Comments (0)
Point, Click, Done!
How far from the truth is that? When it comes to sophisticated performance testing tools, and even the not so sophisticated ones, how many times has anyone been able to just use the out-of-the-box functionality of a testing tool to accurately and thoroughly test an application? Let me guess at this... (thinking) (thinking) (thinking)... None! Ok, in all fairness, there are rare instances when someone tests a simple application, really simple (like a static web site). Anyway, here is my thought as I was flying home from a recent mentoring engagement.
Actually, this applies to all of the tools I use or have used. They all pretty much have the same base functionality. They record, they play back, and they have some form of programming language (compiled or interpreted). The implementation of these functionalities differs, of course. Regarding programming related functionality, in most cases if a specific functionality is not there, you can usually program around it (yeah, right!). For the newbie’s out there, programming around it only applies if you know it is something you need. This is all well and good, but what I was thinking about was the fact that I need to keep programming (or using) the same custom functionality.
Specifically, since the mentoring I was just doing was against a web application, I needed to use custom code functions that are typical and customary to doing web testing. For instance, pulling a list of values from a listbox, randomly picking one and using it in the next request. What about a function to randomly pick a link on a web page? Wouldn't it be great if there was a function that performed this act that comes with the test tool? Of course there are always unique situations for each application that will require custom coding, but the typical things like I just mentioned would serve 2 purposes. First it is code you do not need to maintain or carry with you (importing) and second if it is coded into the test tool itself, it would probably be just as efficient or maybe even more efficient than your custom code (assuming the tool vendors have the right programmers and/or you’re not dealing with interpreted script code).
This blog entry was geared towards newbies since I was recently training people new to these tools. They were all programmers in some form or other (very technical, which is always a pleasure when teaching a technical subject). So for typical code neccessities, let’s have the tool functionality built in. But for custom code – hey, all die-hard programmers out there, we all love our own code creations, right? (wink, wink)
Posted by Jonathan Harris on Monday, July 16, 2007 5:58 PM EDT
Leave a comment | View Comments (1)
Thank God You're Here
Thought I'd start off with a little thought dump. I just watched a few episodes of "Thank God You're Here", a comedy show where actors are dressed up and thrown into a scene with absolutely no knowledge of what they are getting into and have to improvise in front of an audience and home viewers until a judge has had enough and stops the scene. Some of the actors do quite well, stealing the scene and making it work (mostly through comedy). Others get thrown for a loop and struggle through it (gets boring). This is not too far from my world. There are largely two situations I find myself in. First is when a company has performance testing as part of their pre-deployment in which case my job is to help them “make it work.” Second is when a company encounters a performance problem and calls upon RTTS to help them resolve the problem as quickly and efficiently as possible. That’s the “Thank God You’re Here!” This is the time when we (RTTS) are thrown into the scene, and find out if we have the knowledge and experience to make it work. I’m going to write about many things here, including experiences of mine, perplexing and philosophical questions, methodologies, interesting occurrences, issues and resolutions. I hope some of what I do here will elicit responses and participation or all to share.
Posted by Jonathan Harris on Thursday, May 31, 2007 3:12 AM EDT
Leave a comment | View Comments (1)
|
