Proceedings of the Third International Workshop on Software Quality Assurance (SOQUA '06)
(in conjunction with the 14th ACM SIGSOFT International Symposium on
Foundations of Software Engineering
(SIGSOFT 2006/FSE-14)), ACM Press, New York, NY, pp. 6-13 (2006)

An Empirical Comparison Between
Direct and Indirect Test Result Checking Approaches

P. Hu 2 , Z. Zhang 2 , W.K. Chan 3 , and T.H. Tse 4

[author-izer free download from ACM digital library]


An oracle in software testing is a mechanism for checking whether the system under test has behaved correctly for any executions. In some situations, oracles are unavailable or too expensive to apply. This is known as the oracle problem. It is crucial to develop techniques to address it, and metamorphic testing (MT) was one of such proposals. This paper conducts a controlled experiment to investigate the cost effectiveness of using MT by 38 testers on three open-source programs. The fault detection capability and time cost of MT are compared with the popular assertion checking method. Our results show that MT is cost-efficient and has potentials for detecting more faults than the assertion checking method.

Keywords: Metamorphic testing, test oracle, controlled experiment, empirical evaluation

1. This research is supported in part by a grant of the Research Grants Council of Hong Kong (project no. 714504), a grant of City University of Hong Kong and a grant of The University of Hong Kong.
2. Department of Computer Science, The University of Hong Kong, Pokfulam, Hong Kong.
3. Department of Computer Science, City University of Hong Kong, Tat Chee Avenue, Hong Kong.
4. (Corresponding author.)
Department of Computer Science, The University of Hong Kong, Pokfulam, Hong Kong.


  Cumulative visitor count