Case Study

Background

As per its Thanksgiving and Christmas marketing initiatives, RTTS’ customer required the ability to support a peak load of 15,000+ customer visits per hour with a projected order conversion rate of 23% (i.e. approximately 3,500 orders per hour). Along with the anticipated customer traffic, RTTS’ customer was also implementing a new e‑commerce platform based on IBM’s B2C Web Commerce solution, using the Web 2.0 paradigm for optimizing the end-user experience with the web site.

Challenges

In order to prevent shopping cart abandonment and prevent negative customer experiences, the online retailer wanted to make sure that their website could handle the extra customer traffic generated during the holiday season, as well as determine if the new e‑commerce platform would scale proportionally. RTTS was engaged to lead the performance testing initiative with the goal of emulating concurrent customer visits and identifying areas for optimization and remediation.

Strategy

Using RTTS’ performance testing methodology, a thorough test strategy was created based off of the customer’s business and technical requirements. Various performance testing types were orchestrated via various performance testing tools and automated load testing solutions.

Solution

Using analytical tools, such as Coremetrics and Alexa, customer visit profiles were determined. Depending upon the performance testing type, these customer profiles were transformed into automated load testing scripts and/or used for transaction profiling, which was used to optimize the end-user experience. YSlow and Page Speed were used to optimize web page performance within the web browser. The following areas of improvement were identified:

1. Reduced transfer size by 73% by implementing GZIP compression on HTTP resources.
2. 10% reduction in transfer size by using Minify JavaScript resources.
3. Found multiple resources with identical content being served from different sources on a single page.
4. Optimized the use of Content Delivery Networks (CDN).
5. 92% reduction in the amount of HTTP 404 ​“Not Found” errors for numerous requests for resources that did not exist.

Based upon the customer’s marketing requirements, HP LoadRunner was used to engineer automated load tests, which emulated various customer e‑commerce profiles, such as browsing, cart abandonment and purchasing. Appropriate cross- sections of anonymous vs. registered users were reflected in order to accurately emulate various accesses to address books, billing information, new registrations, etc. Load testing was successful in identifying areas for remediation and tuning, which permitted the e‑commerce system to support its expected load. Some of the observations included the following:

1. At peak capacity the web site supported 15,000+ customer visits per hour and processed 3,700+ orders per hour. Web site reliability was 99.84%, with the majority of ​“errors” involving test data.
2. Web page response times were within 2 to 3 seconds for browsing related activities; checkout-related web pages were 4 to 6 seconds.
3. Web site capacity was limited by the CPU usage on its database server. Application server CPU usage was humming along around 65% during peak load.
4. Application code-level changes were implemented in order to optimize Ajax calls and limit redundant HTTP requests.
5. Business logic regarding delivery date calendars was implemented in order to reduce load on the database.
6. Apache web server configuration changes were optimized regarding timeouts and connection handlers.
7. WebSphere connection pools were tuned for various data sources.
8. Oracle database indexes were added in order to speed up certain SQL queries.

Benefits

At the completion of performance testing, expectations were clearly defined amongst the client’s marketing and IT departments. Using a combination of performance testing techniques, the following can be concluded:

• An alignment across the customer’s departments permitted effective marketing campaigns that were able to be supported by the customer’s e‑commerce web site.
• Via the efficient use of automated virtual users to emulate expected customer load, the costs of performance testing were offset by the potential loss of customer revenue, as well as limiting the need for customer support to manage customer issues.
• Web page response time was reduced by up to 30% by optimizing Ajax calls, omitting HTTP errors and optimizing business logic.
• System capacity was tuned through all of the application layers, including the web servers, application servers, and database server.