Evaluation Metrics and Learning to Rank for Information Retrieval

Add to your list(s) Download to your calendar using vCal

• Emine Yilmaz - University College London
• Wednesday 22 January 2014, 14:00-15:00
• Lecture Theatre 1, Computer Laboratory.

If you have a question about this talk, please contact David Greaves.

Please note new seminar start time of 2:00pm this term.

Most current information retrieval systems are machine learning algorithms that are designed to optimize for evaluation metrics measuring user satisfaction, a process referred to as learning to rank. Two important problems are raised during the learning to rank process: (1) Which evaluation metric should be used as the objective in optimization?, and (2) How to reduce the large number of judgments needed to create the training data for learning to rank?

In the first half of this talk, I will first focus on the effect of evaluation metrics used as objectives in learning to rank. I will first show that in contrast to the common belief, the target metric used in optimization is not necessarily the metric that evaluates user satisfaction. I will then describe an information theoretic framework that can be used to analyze the informativeness of evaluation metrics and show that more informative metrics should be used as objectives during learning to rank, independent of the measure that best captures user satisfaction.

In the second half of the talk, I will focus on techniques that can be used to reduce the number of judgments needed for training and evaluation of retrieval systems. I will describe a method based on sampling and statistical inference that can be used to reduce the number of judgments needed for training and evaluation of retrieval systems by 90%, enabling commercial search engine companies save millions of dollars and research laboratories build their own search engines. The sampling based methods I will describe have been adopted by the National Institute of Standards and Technology (NIST), and are used as standard methods by venues such as TREC (Text REtrieval Conference), CLEF (Cross Language Evaluation Forum) and INEX (Initiative for Evaluation of XML Retrieval).

This talk is part of the Wednesday Seminars - Department of Computer Science and Technology series.

This talk is included in these lists:

• All Talks (aka the CURE list)
• Cambridge talks
• Chris Davis' list
• Department of Computer Science and Technology talks and seminars
• Graduate-Seminars
• Guy Emerson's list
• Interested Talks
• Lecture Theatre 1, Computer Laboratory
• School of Technology
• Trust & Technology Initiative - interesting events
• Wednesday Seminars - Department of Computer Science and Technology
• bld31
• computer science
• se393's list

Note that ex-directory lists are not shown.