Case Study

Leading eTailer chooses Microsoft Visual Studio for load testing critical web services


RTTS’ client operated numerous heterogeneous customer-facing (i.e. revenue producing) web applications, as well as administrative and customer support applications, that relied on a common set of services, including customer account retrieval, delivery date availability, and address validation. Using IBM’s Integration Bus solution (formerly known as WebSphere Message Broker), RTTS’ client was launching an enterprise server bus (ESB), so that these individual applications (and future applications) could more easily scale in terms of business growth.


High availability and responsiveness were critical factors to the customer’s business model, since these factors directly affected order conversion for the customer-facing applications, as well as mean time to resolution regarding customer service inquiries. RTTS’s customer required a load testing solution that possessed a minimal total cost of ownership (TCO) that could scale in proportion to its business growth and the consumption of the web services provided by the ESB.

Licensing expenditures and load generation infrastructure costs were paramount in being able to meet budget constraints, while the ability to scale to the transaction rates and concurrency were imperative to the application and infrastructure teams. The short-term load testing goal was to be able to support the demands of at least one (1) of the customer’s primary brand’s e-retail web site using a single instance of the IBM Integration Bus. Peak usage had been observed to be approximately 7,000 online customers.


RTTS put forth a proposal that would assess the available load testing solutions. Proof-of-concepts were arranged that addressed the gamut of open source load testing solutions and commercial load testing solutions. The results of the proof-of-concepts were presented to the stakeholders in conjunction with the projected costs of the solution in order to determine the best return on investment. RTTS also took ownership of the rollout of the load testing solution by providing professional services that installed the testing products and engineered the baseline performance tests.


Several load testing solutions were evaluated using the criteria set forth by RTTS’ customer. Open source, commercial, and cloud-based load testing solutions were evaluated via proof-of-concepts against a subset of the ESB’s suite of web services. Load test solutions that were provided by open source testing tools were deemed a risk in terms of having product support. Cloud-based testing solutions turned out not to be a fit a the time due to concerns regarding security, proprietary scripting languages, and the client’s current corporate culture. However, RTTS’ client did express interest in having the flexibility to move to a cloud-based solution, especially regarding its load generation infrastructure. In the end Microsoft Visual Studio Ultimate was selected to be the best fit based upon its capabilities, familiarity, and cost.

The initial cost of Microsoft Visual Studio Ultimate was significantly less expensive when compared to some of the more established load testing solutions in the market. Microsoft’s licensing model permitted unlimited numbers of virtual users, so that testing could easily scale along with the growth of the customer’s business. Along with the ability to systematically test the ESB using a familiar Visual Studio IDE, the ability to add capacity to its load generation infrastructure was seamless. RTTS was able to incrementally add to the customer’s load generation infrastructure by simply spinning up additional virtual machines and adding them to the current Visual Studio Load Test Rig. If and when the need arises, these tests could be ported to the Windows Azure cloud platform for greater flexibility for implementing its load generation infrastructure.


RTTS successfully installed and configured the necessary Microsoft Visual Studio components and was able to jumpstart the load testing initiative by testing a subset of the ESB web services related to product availability, delivery dates, and shipping options for one (1) of the customer’s family of brands. Using a Visual Studio Load Test Rig of six (6) agents, load tests reaching 12,000 concurrent users were executed. Microsoft Visual Studio’s load testing capabilities were able to assist in discovering bottlenecks related to the threading model being used by the ESB’s database connection pool, as well as within the delivered HTTP configuration of the ESB’s web interface. Once the environment was optimized, 90% of all web service requests were returned in under one (1) second. The IBM Integration Bus was shown to support the processing of 55 web service requests per second with its CPU utilization averaging about 60%.

At the conclusion o the project RTTS was tasked with leading the testing effort by augmenting the collection of web services being tested, incorporating additional retail brands and back-office applications, and subsequently increasing the end-user consumption of the web services.



Solution & Services

  • Performance/Load Testing
  • Software Test Management
  • Test tool evaluation


  • Microsoft Visual Studio Ultimate

Are you interested in learning more or have additional questions?
Please fill out the form below and we will gladly assist you.