Repeatable and Reliable Semantic Search Evaluation

Roi Blanco, Harry Halpin, Daniel M. Herzig, Peter Mika, Jeffrey Pound, Henry S. Thompson, Thanh Tran


An increasing amount of structured data on the Web has attracted industry attention and renewed research interest  in what is collectively referred to as semantic search. These solutions exploit the explicit semantics captured in structured data such as RDF for enhancing document representation and retrieval, or for finding answers by directly searching over the data. These data have been used for dierent tasks and a wide range of corresponding semantic search solutions have been proposed in the past. However, it has been widely recognized that a standardized setting to evaluate and analyze the current state-of-the-art in semantic search is needed to monitor and stimulate further progress in the field. In this paper, we present an evaluation framework for semantic search, analyze the framework with regard to repeatability and reliability, and report on our experiences on applying it in the Semantic Search Challenge 2010 and 2011.

Full Text: PDF
Type of Paper: Research Paper
Keywords: Semantic search evaluation, semantic search, structured data, semantic data, Web data, RDF, Web search
Show BibTex format: BibTeX