The Problem: Retrieval Is a Cost Funnel
Kapa.ai builds AI assistants over large product knowledge bases. Their retrieval pipeline: a retriever finds candidate chunks, a reranker orders them, and the top ~15 reach the generator (the expensive LLM). The generator pays for every chunk it reads, even irrelevant ones. In their system, retrieved chunks account for about two-thirds of query cost — more than the answer, conversation history, and system prompt combined. Every chunk dropped cuts cost by ~4%.
But recall is sacred: drop a needed chunk, and the answer degrades. The ideal pruner maximizes compression per point of recall lost.
Why Naive Cutoffs Fail
Exposing reranker scores and cutting below a threshold doesn't work for two reasons:
-
Scores are not calibrated. Rerank scores are relative, not absolute. A chunk scoring 0.7 on one query might be noise, while on another it's essential. No fixed threshold works across queries.
-
Relevance is a property of the set, not individual chunks. Pointwise cross-encoders score each query-chunk pair alone. But a chunk may be useless by itself yet essential when combined with another. Example from production:
- Chunk A: "Audit logs capture user actions..."
- Chunk B: "...including login attempts and file access."
Chunk B never mentions audit logs, so it scores low alone. Yet it's half the answer. No pointwise score can capture that.
Anchor Documents: Elegant but Ineffective
Kapa tried anchor documents (Sinhababu et al.): synthetic chunks of known relevance are inserted into the ranking, and real chunks below a certain anchor are dropped. This fixes calibration but not the set-judgment problem. The reranker still placed partially relevant chunks below irrelevant ones, so to keep them, the anchor had to sit so low that almost nothing got pruned.
The Solution: Listwise LLM Grading
Kapa added a single LLM call between the reranker and generator. The LLM receives the question and all chunks, then grades each on a five-level scale:
| Score | Level | Meaning |
|---|---|---|
| 5 | ESSENTIAL | Answer cannot be produced without this chunk. |
| 4 | CONTRIBUTING | Does not answer alone but needed in combination with others. |
| 3 | SUPPORTING | On topic but answer likely complete without. |
| 2 | TANGENTIAL | Same domain but no concrete contribution. |
| 1 | UNRELATED | No meaningful connection. |
Chunks at or above a threshold survive. This solves both earlier failures: levels are defined in words, so a fixed threshold works; and the model sees all chunks together, enabling set-level judgment.
Three Knobs
- Model: Small, cheap, fast. Flagship models are ruled out because the pruner must cost less than it saves. They chose the fastest and cheapest at low reasoning effort.
- Threshold: The main dial between compression and recall.
- keep-top-k: The top few reranked chunks always pass, protecting against grading mistakes.
They also tested two simpler baselines: budget-select (keep top N, let LLM add at most M more) and direct ask ("which chunks to keep?" without a scale).
Results: 68% Compression at 96% Recall
Measured on a labeled set of production queries where ground-truth needed chunks were known. Every point on the graph is a configuration; x-axis is compression (% of chunks dropped), y-axis is recall (% of questions that still have all needed chunks). Dashed grey line is naive top-N truncation.
All LLM strategies beat truncation by a lot. At 98% recall, truncation drops ~7% of chunks; LLM strategies drop 30%+. The listwise scoring line dominates both simpler designs at every compression level.
They picked an aggressive point: ~96% recall preserved, ~68% of chunks dropped. One in 25 questions loses a needed chunk. In exchange, per-query cost drops ~34% (net of the pruner's own cost).
Latency Impact: ~0.7 Seconds Added
The pruner runs in the critical path. Their chosen configuration adds ~0.7 seconds per query. Heavier settings climb fast, so a small model at low reasoning effort is essential. The generator barely speeds up (fewer input tokens) — not enough to cancel the pruner's latency. So pruning trades a small fixed latency for significant compression.
Where It's Enabled
Kapa rolled it out first in agentic contexts where retrieval is one tool among many. Agents carry dozens of tools; every call adds to context. A documentation search returning two-thirds fewer chunks buys room for everything else. Lost recall is less dangerous because agents can search again if something is missing.
Pruning is now on by default in Kapa's Product Agent SDK knowledge base search, and optional in the retrieval API and MCP servers.
Key Takeaways
- Context pruning pays off. A cheap LLM grading chunks as a set beats reranker score cutoffs and anchor documents.
- Set-level judgment is essential. Pointwise scores miss partial and indirect relevance.
- Cost savings are real. 34% reduction in per-query cost at 96% recall.
- Latency is the tradeoff. ~0.7 seconds added is acceptable for agents but may be too much for latency-sensitive single-shot queries.
If you're building RAG pipelines and your generator is spending most of its budget on irrelevant chunks, consider inserting a pruner. Use a small, fast LLM, define a clear relevance scale, and always keep the top few chunks as a safety net.
