npm run eval:search. Reproduce from a clean clone with
npm install && npm run eval:search.
Corpus: 27 records from
packages/index/src/seed.mjs, known-item.
Queries: 50 (48 judged, 2 no-match probes).
What these numbers are not
The corpus is the seeded demo catalog and the judgments were written by the same person who wrote the ranker. Both facts inflate confidence and neither is hidden: this is a known-item measurement, the honest floor for “does the ranker work at all”, and the Tranche 1 deliverable replaces it with 150–200 queries plus a rolling sample drawn from the live catalog, which nobody here authored.no-match silence is the fraction of deliberately unanswerable queries
(quantum teleportation as a service) for which the ranker returns nothing rather
than its best bad guess. A low number here is a real weakness, published on purpose:
BM25 will happily match a stray token, and the cold-start section of
SEARCH-QUALITY.md names the mitigations that are not built yet.
Weakest queries
Method
gain(rel) = 2^rel - 1, grades 0–3, judged ineval/golden.jsonl.- nDCG@10 against the ideal ordering of the judged set; queries with no judged document are excluded (nDCG is undefined for them) and counted under no-match instead.
- Recall@20 and MRR@10 count a document as an answer at grade >= 2. Grade 1 (“marginally related”) therefore contributes to nDCG and not to the binary metrics.
- Every query runs through
catalog.search, the same call/discovery/searchserves.
Regression gate
npm run eval:search compares against eval/baseline.json and exits non-zero if any
metric falls more than 0.02 below it. CI runs it on every push, so a ranking change
that quietly costs relevance fails the build instead of shipping.