An Overview of Semantic Search Evaluation Initiatives

Khadija M. Elbedweihy, Stuart N. Wrigley, Paul Clough, Fabio Ciravegna


Recent work on searching the Semantic Web has yielded a wide range of approaches with respect to the underlying search mechanisms, results management and presentation, and style of input. Each approach impacts upon the quality of the information retrieved and the user’s experience of the search process. However, despite the wealth of experience accumulated from evaluating Information Retrieval (IR) systems, the evaluation of Semantic Web search systems has largely been developed in isolation from mainstream IR evaluation with a far less unified approach to the design of evaluation activities. This has led to slow progress and
low interest when compared to other established evaluation series, such as TREC for IR or OAEI for Ontology Matching. In this paper, we review existing approaches to IR evaluation and analyse evaluation activities for SemanticWeb search systems. Through a discussion of these, we identify their weaknesses and highlight the future need for a more comprehensive evaluation framework that addresses current limitations.

Full Text: PDF
Type of Paper: Research Paper
Keywords: semantic search, usability, evaluation, benchmarking, performance, information retrieval
Show BibTex format: BibTeX