An Empirical Investigation of the Correlation Between Fault-Revealing Test Behavior and Differences in Program Spectra

Abstract

Many software maintenance and testing tasks require comparisons of the behaviors of program versions. Program spectra have recently been proposed as a heuristic for use in performing such comparisons. To assess the potential usefulness of spectra in this context, we conducted an experiment that examined the relationship between differences in program spectra and the existence of faults in programs, and empirically compared several types of spectra. Our results reveal that certain types of spectra differences correlate with high frequency --- at least in one direction --- with the presence of faults. That is, when program failures exist on particular inputs, spectra differences are likely also to exist on those inputs, though the reverse is not true. Our results also suggest that several types of spectra that appear, analytically, to offer greater precision in predicting failure behavior than other cheaper spectra provide no greater precision in practice. These results are important in any consideration of the practical uses of program spectra.


Related research categories:
(1) Program Analysis
(2) Regression Testing
(3) Testing
(4) Empirical Studies

Go To Publications