Evaluation measures act as objective functions to be optimized by Web search engines and other information retrieval systems. Such objective functions must accurately reflect user requirements, particularly when tuning systems and learning ranking functions. Ambiguity in queries and redundancy in retrieved documents represent two areas where current evaluation measures poorly reflect these requirements. In this talk, I will present a framework for evaluation that systematically rewards novelty and diversity. This framework is developed into a specific evaluation measure based on cumulative gain. The feasibility of the approach is demonstrated using a test collection based on the TREC question answering track.

Biography: Charles Clarke is an Associate Professor in the School of Computer Science at the University of Waterloo. His research interests cover a wide range of topics related to information search and retrieval, including question answering, XML, program analysis, filesystem search, user interfaces, and statistical NLP.