A brokerage firm utilizes TIBCO Enterprise Message Service (EMS) to support the sending and receiving of Financial Information eXchange (FIX) messages across their various electronic trading platforms. The goal of the software quality effort is to ensure that the messages are routed correctly, the trade data is correct and that the appropriate response messages are generated and sent to the proper applications.

• At the project start, the QA team lacked a direct mechanism for the test tool to interact with the TIBCO EMS queues
• For testing runs, specific market conditions needed to be simulated in order to provide realistic tests
  
• Developers made frequent changes in production without documenting the specific custom FIX tags that were added/removed from messages, so rapid regression coverage was needed
  
• Due to frequent code changes, a method was needed to quickly update the input data for test cases

The following strategy was employed:

• Implement a custom TIBCO EMS test client that the test tool could use to execute the necessary functions against the EMS queue.
• Harness admin commands in the applications under test to create specific market conditions.
• Continuously scrape production logs to determine which FIX messages changed.
• Scrape logs to automatically update test cases and test data.

A Custom .NET TIBCO EMS Client
All of the client application components were connected by TIBCO EMS queues. In order to interact with the EMS queues, RTTS implemented a .NET client to perform all of the necessary data entry and data capture functions required for testing. These functions included establishing and managing connections to the EMS server and sending, receiving and browsing messages on the EMS queues. A type library was created for the .NET client to serve as the bridge between the test tool and the custom .NET EMS client. Figure 1 shows the overall setup.

To ensure that the correct application-generated response message was being retrieved for the test results, messages were checked for content to verify whether or not they contained the desired order ID and other trade-specific data. When the desired response message was found, it was acknowledged, compared against the expected response message, and passes or fails were logged accordingly. Conditions such as the following constituted failures: extra tags in a message, missing tags from a message, or erroneous data in a message. If the desired response message was not found at all then the messages on all the EMS queues were dumped into a results file to allow for a thorough investigation of the failure. Examples of this situation included rejection of messages because they were sent after an exchange closed, or because the application component they were being routed to was not available. The batch dumping of messages proved quite useful as it helped catch defects in client application components which were causing all orders received to be rejected.

Simulating Market Conditions
As noted above, in order for the test cases to reflect realistic scenarios, specific market conditions needed to be simulated. Admin commands from the applications under test were used to spoof market conditions, by setting exchange times and creating market depth. Admin commands were automated at the sockets level, and controlled from the test tool at the beginning of each execution run.

Automated Updating of Test Cases
To work around the near-constant revision of custom FIX tags in the systems, logs were scraped for FIX data. Scripts were written and run to import current input messages into a database and to extract response messages to use as expected results in revised test cases. This provided an efficient solution to the need for ongoing generation and modification of test cases.