Case Study

Background

The application-under-test was a new release designed with better navigation, more attractive marketing, enhanced searching and purchasing capabilities. Accessible over the Internet, the current user hit rates were in the thousands hourly. The new site was scheduled to launch in one week and the team was experiencing unusually high response times under test conditions along with inconsistent test executions, RTTS was asked to work with the client team to analyze and validate the test results within the short timeframe.

The following goals were established:

• Identify weaknesses in the current testing processes, automation practices, and methodology and provide correction action
• Review and modify the existing test scripts to improve reliability and execution
• Optimize the automated testing tool, enabling it to emulate the required number of users in the test environment
• Assist in test execution and analysis

Challenges

The client was using IBM Rational Performance Tester to verify the production readiness and scalability of the new release site. An ideal choice for this challenge, this product determines whether an unreleased application, when under multi-user loads, performs as needed against specific response criteria.

Solution

Working in partnership with the client’s on-site test team, RTTS was able to meet these goals within 3 days, delivering rapid results and providing the required confidence to meet the release deadline.

• Day One: RTTS optimized the test tool driver configuration.
• Day Two: RTTS modified the automated test scripts for accurate duration, page and URL timings. RTTS also added test management code for more accurate control over test execution with the ability to ramp up and down the virtual users, as needed.
• Day Three: Using the modified test scripts, RTTS provided repeatable tests so that the test infrastructure could be analyzed to determine the cause of the poor performance. With the ability to run repeatable, well-defined tests, the client found major issues within their database and web servers.

Benefits

With a day to spare, the client was able to launch the new site with assurance that it would perform as expected, and the results showed it. The client reported the following record-breaking results to RTTS:

• Server response time in production averaged 2 seconds
• Average dynamic pages served ~60,000 pages per hour at an average of 1.7 seconds
• Web traffic reached over 80,000 unique visitors, 10% above the last all-time record
• At peak load, the site performance averaged under 2 seconds serving dynamic pages and forms
• Over 80,000 unique visitors were on the site in one 24-hour period, breaking previous records
• During one peak hour 10,00 unique visitors and 3,000 repeat visitors were served 100,000+ dynamic pages at an average of 1.7 seconds

RTTS was able to assist their client in quickly deploying a quality application by leveraging the features found in the Rational product suite. Furthermore, by solving problems in the test lab, the costs for diagnosing issues and tuning application performance are the lowest. The client was able to minimize expenses and provide a positive end-user experience.