A database security firm researches new vulnerabilities as they are added to the CVE (Common Vulnerabilities and Exposures) List. This information is used to produce application monitoring and application probing products, including a real-time database intrusion detection tool and an application vulnerability auditing tool. The goal of the software quality effort is to provide an extensible framework to automate the execution and monitoring of vulnerabilities of each new build of the tools, with a variable target run of 20%-100% of the vulnerability test scripts per platform.

How can the software quality effort for a database security tool increase test coverage over multiple builds with a dynamic environment consisting of multiple platforms of various patch levels?

Implement an automation effort targeting high value processes in order to free up personnel for analysis and increase test coverage for each build with minimal automation script growth.

• RTTS used a hybrid approach creating a proprietary Keyword Automation Test Framework (KATF) to automate processes and SQL-based data comparisons between a baseline reference platform (BRP) and the application under test.

• Approximately 800 rules were validated utilizing approximately 3000 SQL test scripts against five major database platforms of varying configurations for the application monitoring application.

• Approximately 90% of the available exploit data was sampled in a given run of the application monitoring application.

• Several vulnerability-auditing tool processes were automated, allowing exact comparisons from backend to front-end of the application, covering approximately 10,000 vulnerabilities for each process.

• The KATF was used to vet the consistency of the system for each build of the system compared to a BRP

During the framework’s development, each additional test module increased test coverage and freed members of the software quality team to analyze higher priority issues.  Each additional module increased test coverage by 10,000 test cases.  Furthermore, increasing the number of Test Execution agents used in a full regression run increased test case execution throughput.

For users of the automated test suite, minimal training is needed due to the flexible procedure configuration built. A test procedure can be defined by selecting any function, and the procedure can then target multiple test cases by automatically adapting its configuration.