Query Performance Prediction for Neural IR: Are We There Yet? - Sorbonne Université Access content directly
Conference Papers Year : 2023

Query Performance Prediction for Neural IR: Are We There Yet?

Abstract

Evaluation in Information Retrieval (IR) relies on post-hoc empirical procedures, which are time-consuming and expensive operations. To alleviate this, Query Performance Prediction (QPP) models have been developed to estimate the performance of a system without the need for human-made relevance judgements. Such models, usually relying on lexical features from queries and corpora, have been applied to traditional sparse IR methods – with various degrees of success. With the advent of neural IR and large Pre-trained Language Models, the retrieval paradigm has significantly shifted towards more semantic signals. In this work, we study and analyze to what extent current QPP models can predict the performance of such systems. Our experiments consider seven traditional bag-of-words and seven BERT-based IR approaches, as well as nineteen state-of-the-art QPPs evaluated on two collections, Deep Learning ’19 and Robust ’04. Our findings show that QPPs perform statistically significantly worse on neural IR systems. In settings where semantic signals are prominent (e.g., passage retrieval), their performance on neural models drops by as much as 10% compared to bag-of-words approaches. On top of that, in lexical-oriented scenarios, QPPs fail to predict performance for neural IR systems on those queries where they differ from traditional approaches the most.
Fichier principal
Vignette du fichier
paper_145.pdf (645.7 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-04290230 , version 1 (16-11-2023)

Identifiers

Cite

Guglielmo Faggioli, Thibault Formal, Stefano Marchesin, Stéphane Clinchant, Nicola Ferro, et al.. Query Performance Prediction for Neural IR: Are We There Yet?. European Conference on Information Retrieval, Apr 2023, Dublin (Ireland), Ireland. pp.232-248, ⟨10.1007/978-3-031-28244-7_15⟩. ⟨hal-04290230⟩
12 View
14 Download

Altmetric

Share

Gmail Mastodon Facebook X LinkedIn More