Prioritizing Test Cases for Regression Testing
IEEE Transactions on Software Engineering
Gregg Rothermel, Roland Untch, Chengyun Chu, and Mary Jean Harrold
Test case prioritization techniques schedule test cases
for execution in an order that attempts to increase
their effectiveness at meeting some performance goal.
Various goals are possible; one involves rate of fault detection
--- a measure of how quickly faults are detected within the testing process.
An improved rate of fault detection during testing can provide
faster feedback on the system under test and let software engineers
begin correcting faults earlier than might otherwise be possible.
One application of prioritization techniques involves regression
testing -- the retesting of software following modifications;
in this context, prioritization techniques can take advantage of
information gathered about the previous execution of test cases
to obtain test case orderings.
In this paper, we describe several techniques for using test
execution information to prioritize test cases for regression
testing, including: (1) techniques that order test cases
based on their total coverage of code components;
(2) techniques that order test cases based on their
coverage of code components not previously covered;
(3) techniques that order test cases based on their estimated
ability to reveal faults in the code components that they cover.
We report the results of several experiments in which we
applied these techniques to various test suites for various
programs and measured the rates of fault detection achieved
by the prioritized test suites, comparing those rates to the
rates achieved by untreated, randomly ordered, and optimally ordered suites.
Analysis of the data shows that each of the prioritization
techniques studied improved the rate of fault detection
of test suites, and this improvement occurred even with
the least expensive of those techniques.
The data also shows, however, that
considerable room remains for improvement.
The studies highlight several cost-benefits tradeoffs
among the techniques studied, as well as several
opportunities for future work.