Rerank your RAG results

Reranking is a second pass over your retrieved chunks: a query-aware model re-scores the top candidates and reorders them, so the passage that answers the question sits first instead of fourth or tenth. You add it between retrieval and the answer, and it lifts answer quality for a small added latency.

This guide explains what reranking fixes, how a cross-encoderA model that reads the query and a candidate passage together and outputs a single relevance score for that pair. The query and passage attend to each other, so it judges relevance more precisely than comparing two separate embeddings. scores a passage, and how to wire reranking into a Claude Code workflow through an MCP retrieval server. It assumes you already have a retrieval pipeline. If you do not, read RAG fundamentals first.

What problem does reranking fix?

Vector search ranks chunks by embedding similarity. That score is a coarse proxy for relevance, computed by comparing two vectors that were each encoded without seeing the other. It is good enough to pull a broad set of candidates, and weak at deciding which of those candidates is the single best answer.

Three problems follow from that:

• The right chunk is present but not first. Models attend best to the start and end of a long context and worst to the middle. If the answer chunk lands at position seven, the model reads past it. (See feed Claude the right context.)
• Near-misses crowd the top. Passages that share vocabulary with the query but do not answer it score high on similarity. They push real answers down.
• You cannot just send more chunks. Sending the top fifty fills the window with noise, raises cost, and makes the answer chunk harder to find, not easier.

Reranking addresses all three by reordering the candidates with a model that reads the query and each passage together.

Two-stage retrieval

Reranking turns retrieval into two stages.

Stage one casts a wide net. Use vector search, or hybrid searchRetrieval that runs keyword search and vector search together, then merges the two ranked lists. Keyword search catches exact tokens like IDs and error codes; vector search catches meaning., and fetch a generous candidate set, around 50 passages. This stage optimises for recallThe share of all relevant passages that you managed to fetch. High recall means the answer is somewhere in your candidate set, even if not ranked first.: you want the answer to be somewhere in the set.

Stage two reorders for precision. Pass those candidates to a reranker, which scores each one against the query and sorts them. Keep the top handful, around five, and send only those to Claude.

The split matters because the two stages have different jobs. Stage one is fast and approximate so it can score thousands of chunks. Stage two is slower and more accurate, and it only runs on the few dozen survivors from stage one. You get the breadth of cheap retrieval and the judgement of an expensive model without paying the expensive cost on every chunk.

How does a cross-encoder score a passage?

The reranker model used here is a cross-encoder. The name describes the mechanism, and the contrast with embeddings is the whole point.

A bi-encoderA model that encodes one piece of text into a vector on its own, with no knowledge of the query. You compare two such vectors with cosine similarity. Cheap to precompute, less precise about relevance. (what your embedding model is) encodes the query and the passage separately, into two vectors, then measures the angle between them. Because each text is encoded alone, the model never sees how a specific word in the query relates to a specific word in the passage.

A cross-encoder feeds the query and the passage into the model together, as one input. Every token in the query can attend to every token in the passage. The model reads them as a pair and outputs one number: how relevant this passage is to this query. That joint reading is why it judges relevance more precisely.Common gotchaCross-encoders have a fixed input limit, often around 512 tokens for query plus passage combined.A chunk longer than that gets silently truncated before scoring, so an answer in the tail of a big chunk is never read and ranks low for no visible reason. Keep chunks within the reranker's window, or check its max-token spec before you trust the scores.

The trade-off is built in. You cannot precompute cross-encoder scores, because the score depends on the query you do not have yet. So a cross-encoder is too slow to rank a whole corpus, and well suited to rank a short candidate list. That is exactly the two-stage split above.

How many candidates should I rerank?

Two numbers control reranking: how many candidates stage one fetches, and how many survivors you send to Claude.

• Candidate count (stage one output). Larger sets give the reranker more chances to find a buried answer, at the cost of more pairs to score. Start at 50 and adjust against your evaluation set.
• Final count (what reaches Claude). Smaller is usually better. Once the reranker has put the answer first, sending three to five strong passages beats sending fifteen mixed ones. Fewer passages means less noise and lower cost.

A reasonable starting point: fetch 50 candidates, rerank them, send the top 5. Then move the numbers and watch your metrics, rather than guessing.Common gotchaThe quiet failure is fetching only 5 candidates and reranking those 5: the reranker can reorder them but cannot pull in the answer chunk that ranked 23rd, so it changes almost nothing and looks broken.The candidate count must be much larger than the final count (50 in, 5 out) for stage two to have buried answers to rescue. If reranking gives you no measurable lift, check this gap before you blame the reranker model.

How do I wire reranking into Claude Code?

You do not write model-calling code for this. You expose a retrieval tool to Claude Code through an MCP server, and reranking lives inside that tool. When Claude needs to search your data, it calls the tool, the tool retrieves and reranks, and Claude receives passages that are already in the right order.

Several community MCP servers already implement this two-stage shape. They expose a single search tool and run hybrid retrieval plus a reranker behind it, so the agent never sees the internals. Examples include code-search servers that index a repository and return reranked passages, and document servers that do the same for a markdown vault.

Register one the same way as any MCP server. In your project config:

{
  "mcpServers": {
    "docs-search": {
      "command": "npx",
      "args": ["-y", "your-rerank-search-server", "./docs"]
    }
  }
}

Then confirm Claude Code sees the tool:

claude mcp list

Once it is connected, you do not invoke reranking by hand. You ask a question and let Claude call the search tool. The reranking happens inside the tool call.

If the server reranks correctly, the passages Claude cites are the ones that directly describe the declined-card path, not passages that merely mention "checkout" or "card" somewhere in the text.

Reranking and hybrid search work together

Reranking and hybrid search solve different parts of the same problem, and pair well.

• Hybrid search widens stage one. It runs keyword search alongside vector search, so exact tokens like ERR_CARD_DECLINED or an order ID get caught even when the embedding misses them. This is about getting the right chunk into the candidate set.
• Reranking sharpens stage two. It reorders whatever stage one produced. This is about getting the right chunk to the top of the set.

A common arrangement: hybrid search returns 50 candidates, a reranker reorders them, and the top 5 go to Claude. Hybrid search owns recall; the reranker owns the final order.

Measure the gain, do not assume it

Reranking adds a model call to every search, so confirm it earns its latency before you keep it.

Evaluate the retrieval stage on its own as well as the final answer. A good model handed badly ordered context produces a wrong answer that looks like a model failure, so you have to measure ordering directly.

• Hit rate at the final count. Of your test questions, for how many is a relevant passage inside the top 5 you send to Claude? Run it with reranking off, then on, and compare.
• Position of the answer chunk. Track where the known-good passage lands before and after reranking. If reranking moves answers from position eight to position one, that is the win, made visible.
• Latency budget. Time a search with and without the rerank step. If your candidate set is large and the gain is small, lower the candidate count or rerank fewer items.

You can ask Claude Code to build this harness against a small labelled set:

When is reranking not the bottleneck?

Reranking reorders the candidate set. It cannot conjure a passage that stage one never fetched. If your evaluation shows the answer chunk is missing from the candidates entirely, reranking will not save you, and the fix lives upstream:

• The answer is not in the candidate set. Raise the candidate count, or add hybrid search so exact terms are not lost.
• Chunk boundaries split the fact. If the answer spans two chunks and neither is self-contained, fix chunking before touching the reranker. See RAG fundamentals for chunking.
• The index is stale. A reranker cannot reorder content that was never indexed. Re-index as the source changes.

Reranking is the last 20 percent: it makes a candidate set that already contains the answer reliably surface it. Get retrieval and chunking right first, then add reranking to sharpen the order. Next, read RAG for coding agents for how the standard recipe shifts when the corpus is a codebase.