Retrieval evaluation
Reproducible search-quality benchmark
A stable query set measures relevance, catastrophic topic drift, and server-reported latency against the live Space Frontiers search API. The harness, queries, rubric, and limitations are public.
Latest production checks
Results are not combined across judged and unjudged runs. A dash means that the metric was not measured in that run.
| Run | Configuration | nDCG@10 | P@5 | Canary miss | Server p50 | Server p95 |
|---|---|---|---|---|---|---|
| 2026-07-29 | Hybrid, Jina rerank on, deterministic safety run | — | — | 0% | 73 ms | 353 ms |
| 2026-07-28 | Hybrid, rerank off, gpt-4o-mini relevance judge | 0.947 | 0.939 | 0% | 172 ms | 352 ms |
The July 28 relevance run used the published gpt-4o-mini rubric with reranking disabled. The July 29 safety run used the current production Jina reranker and no LLM judge; it therefore reports deterministic canary and latency metrics only. Latencies come from server timings, excluding the SSH benchmark tunnel.
What is measured
- 36 stable queries across medicine, biology, physics, mathematics, computer science, history, fiction, and adversarial syntax.
- TREC-style 0–3 relevance grades over title, abstract, and matched passages.
- nDCG@10, nDCG@5, precision@5, MRR, topic drift, and server latency.
- A deterministic central-entity canary that does not require an LLM.
What is not claimed
- The query set is public and curated, not a universal workload.
- An LLM judge can drift; grades are cached and the rubric is versioned, but human review remains useful.
- Retrieval quality does not measure generated-answer correctness.
- These runs do not claim superiority over external search products.
Reproduce the benchmark
The repository includes the harness, query set, scoring rubric, production-safety notes, and machine-readable output. The deterministic run needs no judge key; the graded run uses an explicitly configured OpenRouter model.
uv run benchmarks/search-quality/bench.py \
--base http://127.0.0.1:18080 \
--modes hybrid \
--rerank \
--no-judge