reranker evaluation notes: cross-encoder vs BM25 on a financial-filing RAG pipeline

a cross-encoder and BM25 both went into a real financial-filing RAG system, and the project's own investigation found the reranker made results worse, for reasons that turned out to be about pruning, not the model

1. A cross-encoder reranker went into a real financial-filing RAG system. The project’s own investigation, alongside a look at what BM25 would have done, found it did not help and made the actual answers worse.
2. The obvious fix, reorder the context so the model reads the right part first, cannot work here: a fixed sorting step always re-sorts everything back into document order.
3. It is the cross-encoder’s scoring that is metadata-blind, not the underlying filings. Those filings carry year, section, and company fields throughout. The scoring model is simply never shown them.
4. A separate published study reranked a similar set of filings and found reranking helped, which is the counterexample this post has to answer rather than set aside.
5. The reranker’s own ranking ability held up under independent checking. The harm traces to the pruning policy, and pruning cannot be separated from reranking, because pruning is the only part of it that changes anything at all.

The system behind this post answers questions over a large set of financial regulatory filings, broken down into individual sentences and spanning many companies and years. This set of filings is referred to below as the corpus. Any system built this way has to solve a problem that has nothing to do with any particular model: pull the right fragments out of that data, from whichever filings and years are relevant, and arrange them into one block of text before a model ever reads it. This project built its own custom components to do that job. This post is about what happened when one more custom component, a reranker meant to sort those fragments by relevance before the model sees them, was added into that chain.

This shape appeared once before, in an earlier post about embeddings: a reranker that only reorders cannot change a concatenated context string, so reordering is free and only pruning does anything. That observation was borrowed from an outside source at the time. Here the reranker, the corpus, and the result all belong to this project, so the claim needed direct verification rather than another borrowing.

what a reranker actually does

A reranker looks like it does one job, put the best evidence first. It actually does two: score everything, then decide what survives. Whether the first job matters at all depends on whether anything downstream still cares about order once the scoring is done.

 score every block          pick survivors (top N)
      |                            |
      v                            v
 relevance 0..1            a fixed step re-sorts
                            back to document order
                                    |
                                    v
                     only the SURVIVING SET changes
                     the string the model actually reads

a custom-built sort step already settles it

The component in this project responsible for arranging fragments into text, referred to here as the assembler, was given one fixed rule. It re-sorts every surviving block before it ever reaches the model, unconditionally, by company, year, section, document, and position. No relevance score appears anywhere in that rule. Any similar system would need something playing this same role, whatever it happens to be called, since raw retrieved fragments do not arrive in an order a model should read them in.

# ContextAssembler._sort_sentences(), called unconditionally before formatting
return sorted(sentences, key=lambda s: (
    s.company_name, s.report_year, s.section_name, s.doc_id, s.sentence_pos
))

So a run with no reranker and a run that only scores, with no pruning, produce a byte-identical string sent to the model. This was checked directly rather than assumed: the two configurations kept an identical set of 166 sentences for a sample query, and the sort key above confirms no score ever survives past that line.

Score-only against pruned is the only fair comparison in that table, and it is flat. One qualification worth keeping in view: recall@5 does not move across any of the three arms, but that is close to a property of the metric rather than a finding, since the kept list is already sorted by score and the first five entries cannot change once anything at all is kept. The real result is narrower than it first looks. Reordering, by itself, is inert in this architecture. That is a property of a sorting step that discards order, not a property of rerankers in general.

a scorer that cannot see what the corpus already knows

The underlying data carries a filing year, a section label, and a company identifier on every row, none of it hidden. The cross-encoder is simply never given access to any of it. It scores text and nothing else. Financial filings repeat the same sentence, almost word for word, year after year, so the highest-scoring match for a question is frequently the right paragraph from the wrong year, because the scorer has no way to tell the years apart.

Across one sample: 45.2% of the blocks that survive at top-8 come from a year the question never asked about, against a 31.5% off-year rate in the unpruned pool. Off-year blocks are also longer and score higher on average, so pruning concentrates the problem instead of filtering it out. One live example, run on a question outside the fixed set used to check the system’s answers (called the gold set below): a question about a company’s 2025 manufacturing capacity returned the same disclosure sentence from two different filings at ranks one and two, with one number changed. The FY2022 filing estimated the commitment near $4.5 billion. The FY2023 filing restated it near $10 billion. The actual FY2025 block landed at rank seven. At a tighter budget of four blocks, the question would have received zero FY2025 evidence. At eight, the right year survives by a single position.

Not every question breaks this way. A separate query about a different company’s export controls, where the vocabulary is distinctive rather than repeated across years, ranked the correct year first and second without trouble. The failure is specific to language that repeats, not to every question the corpus can ask.

the ranking held up, the answers were mixed

Retrieval metrics are a proxy for whether an answer is actually good. Ten questions from the gold set were run through the full pipeline, retrieval followed by a language model writing an answer, at three different reranking settings, and the resulting answers were read by hand against the correct answer rather than scored automatically.

Cutting to top-8 dropped context by 61% and cost by 32%. Whether that also improved the answers depends on the question, which is why a human read every one rather than trusting the two score columns above. For each of the ten questions, the judge compared the answer produced at a given setting against the answer produced with no reranker at all, and marked it better, the same, or worse. At top-16: one answer improved, six were unchanged, and three got worse. At top-8: three improved, two were unchanged, and five got worse. So both settings lost more often than they won, and the losses cluster in one place: every question that needed evidence from several companies or several years in a single answer lost ground. The wins were single-fact questions, where removing distractor text let the model land on the one fact that mattered.

The clearest case is one where less context produced the more honest answer. At top-16, the model wrote a confident paragraph about a company’s 2010 risk disclosures. There is not one block from that company’s actual 2010 filing anywhere in the 44-block pool it was given, only near-identical filler from 2016 through 2025 filed under the same section heading. At top-8, with less material available, the model stated plainly that the year was not in context. More evidence produced a fabricated answer. Less evidence produced the true one.

The automated scores missed this distinction entirely. On two of the ten questions, ROUGE-L or cosine ranked the top-8 answer highest, on the same answer the human judge scored as covering fewer of the companies the question had actually asked about.

one ranking, one policy, and why they cannot be pulled apart

The reranker’s ranking ability was checked on its own terms, separately from the harm above, and it holds up. Gold evidence scores in the top 5% of the ranked list on average, and ranks first outright for ten of the twenty-five questions where the correct evidence is reachable at all. The model is doing real relevance work, not simply counting words: controlling for block length, the gap between gold and non-gold scores survives, and in the middle length range it grows rather than shrinks.

That is what makes the finding sharper than “the reranker is bad.” The harm measured above comes entirely from pruning, and pruning is not an optional add-on sitting beside the ranking. It is the only part of using this reranker that changes anything the model reads, established directly by the sort-order code earlier in this post. A reranker whose scores are never allowed to reorder anything can only ever act through what it throws away. So a correct ranking and a worse outcome are not two separate results here. They are the same mechanism, viewed from either end of it.

an external result worth answering directly

A separate published study, “Enhancing Financial Report Question-Answering: A RAG System with Reranking Analysis” (accepted at ICECET 2026), reranked a comparable corpus of financial filings using the FinDER benchmark, roughly 1,500 queries drawn from S&P 500 disclosures, and measured answer correctness directly rather than a retrieval proxy. Reranking raised the share of answers scored at least 8 out of 10 from 33.5% to 49.0%, and cut fully incorrect answers by more than a third.

The two results sit next to each other without actually contradicting, once the setups are compared. That study reranked a hybrid lexical-plus-semantic retrieval pool, not a purely score-ranked one. The pipeline examined here discards ranking outright once scoring finishes, and its corpus repeats near-identical sentences across two decades of filings in a way the cited study’s setup may not.

what these four add up to

The seed idea behind this post also named BM25 as a second method that would reward long, boilerplate-heavy text the same way the cross-encoder does. BM25 was never run against this corpus, and it normalises for document length by design, so it would not misbehave in that particular way even if it had been tried. What actually rewards boilerplate here is lexical overlap, a different mechanism entirely: 44.9% of sentences in the corpus are exact duplicates of another sentence, mostly the same company repeating the same disclosure across filing years. The project’s own lexical gold-label selector, built on that kind of matching, picked boilerplate over the real answer in at least five of thirty-one cases.

what this sample cannot establish

Thirty-one questions is not enough to detect a modest real effect. The smallest change this sample could reliably tell apart from noise is larger than the change actually observed, so “flat” here means the sample was too small to see past, not that nothing happened. The retrieval draws were also not perfectly reproducible from one run to the next under identical settings, so even the 64.5% ceiling, the share of gold evidence the retriever surfaces at all before any reranking happens, is probably an undercount rather than an exact figure.

The project shipped with reranking off by default, for the reason this post has been building toward: the reranker’s own ranking ability was fine on independent inspection. The pruning policy bundled with it was not, and the two cannot be separated in this design.