This paper, accepted at SIGIR 2026 (Melbourne, Australia), introduces LearnDCG, a differentiable and learnable approximation of NDCG that eliminates the rigid, hand-designed components of existing surrogate losses and trains them jointly with the ranker in a single end-to-end pipeline. The work was carried out at Toronto Metropolitan University under the supervision of Professor Ebrahim Bagheri.

Motivation: The Fixed Surrogate Problem

Every learning-to-rank system is built on a contradiction. The metric we care about — Normalized Discounted Cumulative Gain (NDCG) — is non-differentiable. You cannot backpropagate through it. So we train on a surrogate: RankNet, LambdaLoss, ApproxNDCG. But the surrogate is not the metric. The gap between what we optimize during training and what we measure at evaluation time is not merely a technical inconvenience — it is a structural source of misalignment that accumulates across different datasets and relevance distributions.

The standard differentiable formulation of NDCG fixes two components a priori: the gain function g(y) = 2^y − 1 and the logarithmic discount d(r) = log₂(1 + r). These are mathematical conventions, not data-driven choices. They carry no information about whether a given dataset has three relevance grades or five, whether highly relevant documents are abundant or rare, or whether top-position accuracy matters more than mid-rank. I became interested in whether these components could be learned rather than fixed — and whether training them jointly with the ranker, on the actual target data, would resolve the mismatch.

Prior work had explored learning the loss function itself through NeuralLoss, which trains a transformer-based surrogate on synthetically generated relevance data before ranker training begins. The insight is sound: if the loss can be represented as a neural network, it can be trained. But the execution has a critical flaw. The synthetic pretraining data is drawn from a uniform distribution of relevance scores. Real-world ranking datasets are heavily skewed — highly relevant documents are rare, and the distribution varies substantially across benchmarks. The gap between pretraining distribution and target distribution undermines the gains from learnability, and this became especially visible on datasets like WEB10K with five-level relevance and pronounced imbalance.

The LearnDCG Formulation

LearnDCG keeps the position approximation structure of the standard differentiable NDCG — which has provably bounded error — but replaces the fixed gain, discount, and temperature components with learnable scalar parameters. The canonical differentiable NDCG approximates the discrete rank position π(x_i) with a smooth surrogate:

π̂(x_i) = 1 + Σ_j≠i σ(−α(s_i − s_j))

where σ is the sigmoid function and α controls the sharpness of the approximation. This yields a bounded approximation error: |π̂(x_i) − π(x_i)| < (n − 1) / (exp(αδ) + 1), where δ is the minimum pairwise score margin. LearnDCG inherits this guarantee unchanged. What it changes is everything around the position approximation: the gain function, the discount function, and α itself are all treated as learnable quantities.

Specifically, rather than fixing the exponential base at 2, I introduce a learnable base b_g > 1 such that g(y) = b_g^y − 1. The discount generalizes to d(r) = ln(1 + r) / ln(b_d) for a learnable b_d > 1. The temperature α, which previously required hand-tuning, is also learned. To ensure these parameters remain in their valid domains throughout gradient-based optimization, I apply a softplus reparameterization:

b_g = 1 + softplus(θ_g),   b_d = 1 + softplus(θ_d),   α = softplus(θ_α)

The training pipeline is then a single backpropagation pass that updates the ranker parameters θ and the loss parameters {b_g, b_d, α} jointly. No pretraining, no synthetic data, no staged optimization. The loss learns from the same data and the same gradients as the ranker, which means it adapts to the actual relevance distribution of the target dataset rather than an approximation of it. The additional computational overhead relative to the fixed formulation is negligible — three scalars, no additional network.

What the Learned Parameters Reveal

One of the more illuminating results in the paper is the behavior of the learned discount parameter θ_d across the five folds of WEB10K during training. Rather than converging to a single value, it converges to different values across folds — ranging from approximately 1.80 to 2.15. This is direct empirical evidence that no single fixed parameterization is universally optimal, even within the same dataset partitioned by fold. The fixed formulation imposes the same discount curvature everywhere. LearnDCG discovers that the right curvature is fold-specific, which translates into measurable gains at evaluation time.

Results: Consistent Gains Across Datasets and Architectures

I evaluated LearnDCG on four standard learning-to-rank benchmarks: MQ2007 and MQ2008 (three-level relevance, medium scale, from the TREC Million Query Track), WEB10K (five-level relevance, approximately 1.2 million query-document pairs from a commercial search engine), and YLTR (five-level relevance, ~710K pairs from the Yahoo Learning to Rank Challenge). Nine baselines span surrogate losses (RankNet, ListNet, ListMLE, RMSE), differentiable metric approximations (LambdaRank, LambdaLoss, ApproxNDCG, NeuralNDCG), and the learnable alternative (NeuralLoss). I also compare against a fixed-parameter variant of LearnDCG itself to isolate the contribution of joint optimization.

LearnDCG achieves the highest score on every metric across all four datasets, with all improvements statistically significant at p < 0.05 using paired t-test. The largest gains appear on WEB10K: the MLP ranker improves from 35.49 to 43.20 NDCG@10, and NeuralLoss collapses to 25.94 on the same benchmark — the most dramatic illustration of the distributional mismatch problem. On MQ2008, LearnDCG reaches 79.03 NDCG@10 against NeuralLoss’s 77.07 and NeuralNDCG’s 76.87. On YLTR, it reaches 76.82 against ListNet’s 75.75. Across all four ranker architectures tested (MLP, AttSets, Attention MIL, Context-Aware Ranker), the pattern holds: every architecture improves, with the gains largest on WEB10K where fixed parameterization is most limiting.

Efficiency

On the training efficiency front, LearnDCG requires no pretraining phase and adds negligible overhead to the standard differentiable NDCG pipeline. On the effectiveness-vs-cost plot for MQ2008 (NDCG@10 against training time in minutes on an RTX A6000), LearnDCG occupies the top-left corner — highest effectiveness at one of the lowest training costs. NeuralLoss achieves competitive accuracy but sits further to the right, reflecting the pretraining overhead. Classical surrogates cluster in the middle-bottom: faster to train, but clearly below LearnDCG in effectiveness. RankNet is worst on both dimensions.

This paper, accepted at SIGIR 2026 (Melbourne, Australia), studies query failure across the three main stages of a modern IR pipeline — retrieval, LLM-based generation, and LLM-based relevance judgment — asking whether the same queries that are hard to retrieve are also hard to generate answers for and hard to judge automatically. The work uses four years of TREC Deep Learning benchmarks (2019–2022) and introduces a unified operationalization of failure per task, a data-driven typology of failure causes, and a practical prompting intervention that converts failure analysis into measurable performance gains.

Motivation: Failure Is Not a Single Thing

Modern IR systems no longer end at retrieval. A query goes into a retriever, which selects documents; those documents condition an LLM that synthesizes an answer; and that answer (or the retrieved documents themselves) is scored by an automated LLM-based judge. When the system produces a bad outcome, attributing that failure to any single component is not straightforward. The retriever may have returned documents that appear relevant but misled the generator. The generator may have hallucinated despite retrieving the right passage. The judge may have diverged from human assessors even when generation succeeded. I became interested in whether these failure modes are correlated across queries — whether a query that is hard for retrieval is also hard for generation — or whether difficulty is fundamentally task-specific.

The existing literature treats these failure modes in isolation: Query Performance Prediction (QPP) addresses retrieval failure, Prompt Performance Prediction addresses generation failure, and a growing body of work examines LLM-as-judge reliability. None of these lines of work had examined whether the same queries systematically challenge all three components, or whether difficulty transfers across stages. That gap is what this paper addresses.

Operationalizing Failure Across Three Tasks

I define three complementary failure sets using TREC Deep Learning 2019–2022, which provide graded NIST relevance judgments on a four-point scale for queries over the MS MARCO v1 (DL 2019–2020) and MS MARCO v2 (DL 2021–2022) corpora.

Hard-to-Retrieve (H2R) queries are the bottom quartile of NDCG@10 under both BM25 (implemented via Anserini) and DistilBERT-TAS-B, a dense retriever fine-tuned on MS MARCO. Average NDCG@10 for this set is 0.186 under BM25 and 0.297 under the dense retriever — consistently low across both retrieval strategies.

Hard-to-Generate (H2G) queries are the bottom quartile of BERTScore F1, computed by comparing LLM-generated responses (from LLaMA3.2:3b and Qwen3:8b under zero-shot, instruction-style prompting) against all passages rated Perfectly or Highly relevant. BERTScore captures both lexical and semantic similarity, making it appropriate for this setting where paraphrase and surface variation are expected.

Hard-to-Judge (H2J) queries are the top quartile of LLM-vs-human disagreement, measured under two judgment setups: binary relevance (mapping grades 0–1 to not relevant and 2–3 to relevant) and graded relevance on a 0–3 scale following the UMBRELA reproduction of Bing assessor labels. Disagreement in the binary setting is counted as a label mismatch; in the graded setting, as any case where the absolute difference between LLM and human score exceeds one point.

Finding 1: Difficulty Does Not Transfer Across Tasks

The central empirical result is that fewer than 10% of queries are classified as hard across all three tasks simultaneously. Approximately half of each hard set is exclusive to that task — queries that are hard to retrieve are not disproportionately hard to generate for, and vice versa. The one consistent exception is the pairwise overlap between H2R and H2J, which is the highest among all three task pairs across all four TREC datasets. This is interpretable: the same retrieved documents that drive low retrieval effectiveness also feed the automatic judgment process, creating a structural dependency between those two failure types.

The interpretation I draw from this is that retrieval, generation, and judgment engage fundamentally distinct capacities. Retrieval relies on lexical and dense similarity matching, governed by document distributional properties. Generation depends on the model’s ability to conditionally synthesize accurate responses from its internal knowledge priors and the retrieved context. Judgment depends on calibration, evaluative alignment, and implicit relevance criteria that may not match the task a human assessor was performing. A query that strains one of these does not necessarily strain the others.

Finding 2: The Overlap Structure Is Stable Across Systems

I vary every component of the pipeline: retriever (BM25 vs. DistilBERT-TAS-B), generator (LLaMA3.2:3b vs. Qwen3:8b), and judgment setup (binary vs. graded). While the specific identities of hard queries shift with system configuration — different models fail on different individual queries — the structure of overlap between the three failure types remains stable. The proportion of queries overlapping between H2R and H2G is comparable under BM25 and the dense retriever. Switching between Qwen and LLaMA has little effect on generation–judgment overlap. Replacing binary with graded judgment produces only marginal deviation in pairwise overlaps.

This finding has a direct practical implication: mitigating failures through model substitution alone is unlikely to resolve the underlying structure of difficulty. Upgrading the retriever may shift which queries are hard to retrieve, but it does not reduce the hardness of those queries for the generator or judge. The difficulty is driven by properties of the queries themselves interacting with task-specific constraints — not by model weakness that a better architecture would fix.

Failure Typology: Why Queries Fail

Beyond identifying which queries fail, I wanted to understand why. I use an iterative LLM-guided reason extraction process: for each hard query, the model is given the query and its associated task context (retrieved documents, generated output, or relevance labels) and asked to generate a concise explanation for why the system may have failed. These candidate reasons are aggregated and passed through multiple refinement iterations, where the LLM merges overlapping items, generalizes overly narrow explanations, and adds novel causes observed in newly sampled queries. The process terminates when no substantive updates occur, yielding a converged typology of failure causes per task.

The top failure reasons differ clearly by task. For Hard-to-Retrieve queries, the leading causes are numerical data requirements, specific product terminology, and ambiguous subject references — properties that challenge both lexical matching and dense retrieval. For Hard-to-Generate queries, the dominant causes are requirements for more recent information (beyond the model’s knowledge), accurate technical knowledge, and risk of misinformation — reflecting the generative model’s sensitivity to knowledge gaps. For Hard-to-Judge queries, the top causes are quantitative data needs, specific and niche topics, queries unrelated to the provided relevance criteria, and absence of relevant passage context — reflecting misalignment between the LLM judge’s implicit criteria and the human assessor’s judgment framework.

I validated these LLM-generated reasons with two human annotators, each reviewing 50 randomly sampled queries and checking whether the assigned explanations were correct while tagging any additional applicable causes. Human and LLM judgments showed strong alignment, with 0.81 accuracy and 0.75 precision across all ten reason categories.

Difficulty-Aware Prompting

Having induced a typology of failure causes, the natural question is whether this analysis can be made actionable. I test a lightweight intervention: conditioning the generator on an explicit statement of the anticipated failure mode. In the baseline condition, the model receives the raw query. In the difficulty-aware condition, the model also receives a natural-language description of the relevant difficulty factor (e.g., “this query requires numerical data”) drawn from the typology. No retraining, no architecture change, one additional sentence in the prompt.

Generation quality is assessed using a pairwise comparison framework adjudicated by a state-of-the-art LLM-based judge, which selects the response that more accurately reflects the content of highly relevant passages. The difficulty-aware generation is preferred across all four TREC datasets: 64.56% preference rate on DL 2019, 75.40% on DL 2020, 52.77% on DL 2021, and 60.49% on DL 2022. The largest gain — DL 2020 at 75.40% — suggests that certain query difficulty profiles benefit substantially from explicit failure conditioning, even with this minimal intervention.

I interpret this result as initial evidence of a broader principle: making failure explicit enables systems to adapt. The typology is not just a diagnostic artifact — it encodes information that is actionable downstream. Extending this approach to retrieval and judgment conditioning is the natural next step.