Program Overview

Session I: Efficiency/Effectiveness Trade-offs (Start time 9:00)

  • Introduction to LtR and aims of the tutorial. (30 min.)
    • Introduction on LtR, its historical evolution and main results and the illustration of the goals of the tutorial.
    • The role of LtR in modern Web search engines. Review of the main approaches of LtR: focus on tree-based models and artificial neural networks. Discussion of the quality vs. efficiency trade-off in the use of LtR models, and brief description of multi-stage ranking architectures
  • Efficiency in Learning to Rank (60 min.)
    • Detailed analysis of state-of-the-art solutions for improving the efficiency of LtR models along different dimensions.
  • Hands-on Session (30 min.)
    • We show how to develop state-of-the-art strategies to gain a more efficient ranking model without losing effectiveness. Given a model learnt with a state-of-the-art algorithm such as LambdaMART, we will show how to reduce its runtime cost by a factor larger than 18x.

Session II: Neural Learning to Rank using TensorFlow (Start time 11:30)

  • Introduction to Neural Ranking (30 min.)
    • Neural learning-to-rank primer
    • TensorFlow and Estimator framework overview
    • TensorFlow Ranking: components and APIs
  • Hand-on Session (90 min.)
    • Introduction to data formats and data sets and colaboratory setup
    • Demo A: TensorFlow Ranking for Search using the MSLR-Web30k data set
    • Demo B: TensorFlow Ranking for Passage Retrieval using the MSMARCO data set

Session III: Unbiased Learning to Rank (Start time 14:30)

  • Introduction to Learning from User Interactions (10 min.)
  • Counterfactual Learning to Rank (50 min.)
    • Counterfactual evaluation and Propensity-weighted Learning to Rank
    • Position bias estimation techniques — online and offline estimation and practical considerations
  • Online Learning to Rank (45 min.)
    • Interleaving and how it deals with position bias
    • Dueling Bandit Gradient Descent (DBGD): the method that defined a decade of Online Learning to Rank (OLTR) algorithms.
    • Extensions of DBGD and their limitations
    • Pairwise Differentiable Gradient Descent (PDGD)
    • An empirical and theoretical comparison between PDGD and DBGD
  • Conclusions & Future Directions (15 min.)

A detailed abstract for Session III is available here.

See here for more details about the program, and the material that will be covered.