Software Testing, Verification and Reliability 22 (4): 221-243 (2012)

Automated Functional Testing of Online Search Services 1

Zhi Quan Zhou 2 , ShuJia Zhang 3 , Markus Hagenbuchner 3 , T.H. Tse 4 , Fei-Ching Kuo 5, 6 , and T.Y. Chen 5

[paper from Wiley Online Library | technical report TR-2010-06]


Search services are the main interface through which people discover information on the internet. A fundamental challenge in testing search services is the lack of oracles. The sheer volume of data on the internet prohibits testers from verifying the results. Furthermore, it is difficult to objectively assess the ranking quality because different assessors can have very different opinions on the relevance of a Web page to a query. This paper presents a novel method for automatically testing search services without the need of a human oracle. The experimental findings reveal that some commonly used search engines, including Google, Yahoo!, and Live Search, are not as reliable as what most users would expect. For example, they may fail to find pages that exist in their own repositories, or rank pages in a way that is logically inconsistent. Suggestions are made for search service providers to improve their service quality.

Keywords: software testing; metamorphic testing; verification and validation; internet search

1. This project is supported in part by a Virtual Earth Award from Microsoft Research, a Small Grant from University of Wollongong, the General Research Fund of the Research Grants Council of Hong Kong (project no. 717308), and a Discovery Grant of the Australian Research Council (project no. ARC DP0771733). Preliminary results of this research were presented by invitation at the Microsoft Research Virtual Earth and Location Summit, Redmond, WA, U.S.A. held on 30 April–1 May 2008. The techniques presented in this paper are currently in a patent application process.
2. (Corresponding author.)
School of Computer Science and Software Engineering, University of Wollongong, Wollongong, NSW 2522, Australia.
3. School of Computer Science and Software Engineering, University of Wollongong, Wollongong, NSW 2522, Australia.
4. Department of Computer Science, The University of Hong Kong, Pokfulam, Hong Kong.
5. Centre for Software Analysis and Testing, Swinburne University of Technology, Hawthorn 3122, Australia.
6. It is with deep regret and sadness that we report the passing of F.-C. Kuo on October 6, 2017.


  Cumulative visitor count