LLM reranker architecture — cross-encoder, candidates, relevance

Why architecture matters here

Reranker fails on latency + wrong model choice. Architecture matters because retriever + cross-encoder + LLM compose.

The architecture: every piece explained

The top strip is pipeline. Query. Retriever. Candidate list. Cross-encoder.

The middle row is output. Relevance score. Top-k final. LLM answer. Latency budget.

The lower rows are ops. Cache. Metrics. Ops — model choice + batch + hardware.

End-to-end flow

End-to-end: query in. Retriever gets top-100 in 30ms. Cross-encoder scores each pair in batch — 200ms. Top-5 pass to LLM. Total pre-LLM latency ~250ms.