Our paper, accepted at CIKM 2025, addresses a foundational bottleneck in entity retrieval research: the scarcity of high-quality relevance annotations. We investigate whether large language models can serve as reliable relevance assessors in this setting, evaluating their judgments against both expert-labeled data and implicit behavioral signals derived from user clicks.

Background and Motivation

Entity retrieval, the task of returning structured real-world entities in response to a user query, underpins a large share of modern information access. Studies estimate that over 70% of queries on Bing and more than half on Semantic Scholar are entity-focused. Despite this, the field lacks the kind of large-scale, manually curated benchmark that has driven progress in passage retrieval. Resources like MS MARCO enabled a generation of neural ranking models; no equivalent exists for entities. The DBpedia-Entity v2 benchmark covers 485 queries and roughly 50,000 graded judgments, which is useful for evaluation but insufficient for data-intensive training. Silver-standard alternatives such as LaQuE address scale by mining click-through logs from ORCAS, producing over 2 million query-entity pairs, but click signals are inherently noisy: users click for reasons that do not always reflect topical relevance, including interface position, lexical surface similarity, and familiarity effects.

This tension motivates our core research question: can LLMs bridge the gap between expensive expert annotation and noisy behavioral data by acting as scalable, interpretable relevance judges for entity retrieval?

Methodology

We evaluate two open-weight models, Qwen3:8b and LLaMA4:Scout, under two input conditions: (1) query and entity title only, and (2) query, entity title, and DBpedia abstract. Relevance is elicited on a three-point graded scale (0 = irrelevant, 1 = relevant, 2 = highly relevant) and also in a collapsed binary format. Rather than applying the standard UMBRELLA prompt directly, we adapt it with a three-step reasoning chain tailored to entity retrieval: the model first estimates the likely user intent behind the query, then assesses factual or conceptual alignment between the query and the candidate entity, and finally produces a relevance judgment. This modification was necessary because entities tend to be short, semantically abstract, and dependent on factual grounding rather than topical elaboration, which causes direct application of passage-retrieval prompts to generalize poorly.

Agreement with expert annotations from DBpedia-Entity v2 is quantified using Cohen’s kappa across both graded and binary settings. Agreement with user behavior is assessed on a 15,000-query sample from LaQuE, treating each clicked entity as implicitly relevant and measuring what proportion the LLM also judges as relevant or highly relevant.

Results

On DBpedia-Entity v2, LLaMA4:Scout achieves a binary Cohen’s kappa of 0.46 using titles alone, rising to 0.52 when abstracts are included. Qwen3:8b reaches 0.39 and 0.46 respectively. These figures are consistent with agreement levels reported for LLM-based judges on TREC Deep Learning passage retrieval benchmarks, suggesting that entity retrieval is not categorically harder for LLM assessors despite its added ambiguity. In the graded setting, kappa values are lower across the board, reflecting the well-known difficulty of fine-grained annotation at grade boundaries. Adding abstracts produces a 37% increase in recall for grade-2 (highly relevant) entities but also introduces a 36% rise in false positives at that grade, and a 14% drop in correctly identified irrelevant entities. This precision-recall trade-off is consistent across both models and reflects how surface-level contextual information can both sharpen and blur relevance distinctions.

On LaQuE, LLMs achieve over 91% agreement with user click behavior using titles alone (91.93% for LLaMA4:Scout), with abstracts yielding a marginally similar rate of 91.79%. The near-identical performance across input conditions suggests that entity titles carry sufficient signal to replicate user behavior at scale, and that the marginal information in abstracts neither meaningfully helps nor hurts overall agreement. The slight drop with abstracts may reflect the model’s increased capacity to reject entities that received clicks due to superficial cues like lexical overlap with the query string rather than genuine semantic alignment.

Analyzing Click-Relevance Mismatches

A distinctive contribution of this work is our analysis of cases where user clicks and LLM relevance judgments systematically diverge. We isolate 420 query-entity pairs that were clicked by users but rated irrelevant by at least three of the four model-input configurations, ensuring high confidence in the irrelevance assessment. For each pair, we prompt an LLM to generate a plausible explanation for why a user might have clicked the entity despite its assessed irrelevance, grounding the hypotheses in known behavioral patterns from the search literature. The resulting explanations are then aggregated across the filtered set.

Prominent Result Bias emerges as the dominant factor, appearing in 95% of mismatch cases, consistent with well-established findings on the disproportionate influence of rank position and visual salience on click behavior. Name or lexical similarity accounts for 48% of cases, reflecting clicks on entities that match query surface forms but are topically unrelated, such as a user querying “Palm Springs Florida” and clicking “Palm Springs, California.” Category-level association appears in 23% of cases, familiarity and recognition bias in 25%, and geographic name confusion in 6%. These categories are not mutually exclusive; many mismatch cases are attributed to multiple overlapping factors.

Implications

These results establish LLMs as viable relevance assessors for entity retrieval across both explicit and implicit supervision regimes. The strong agreement with expert annotations suggests that LLM-generated labels could augment or partially replace manual annotation in low-resource evaluation settings. The high alignment with click behavior, achieved with minimal input, points toward practical utility in large-scale dataset construction pipelines where only behavioral signals are available. Perhaps most importantly, the mismatch analysis demonstrates that LLMs can do more than assign labels: they can generate interpretable explanations for the gap between observed user behavior and semantic relevance, a capability that has direct value for auditing silver-standard datasets and improving the reliability of implicit feedback in entity-centric retrieval systems.

LLM-as-a-Judge in Entity Retrieval: Assessing Explicit and Implicit Relevance
Mohammad Hossein Saliminabi, Negar Arabzadeh, Seyed Mohammad Hosseini, Dimitrios Androutsos, Morteza Zihayat, Ebrahim Bagheri
CIKM 2025, Seoul, Republic of Korea, November 10 to 14, 2025
Code, prompts, and data available on GitHub