Retrieval quality evaluation

How Kioku measures whether search_notes in keyword, semantic, and hybrid modes actually returns the right notes, and how to compare configurations (embedding model, thresholds, scoring changes) with numbers instead of gut feeling.

Scope

This evaluation harness measures retrieval quality only. Kioku can expose optional local generation workflows when the generation capability is enabled and a generation model is configured, but those workflows are outside this harness.

The metrics here score whether retrieval returns the right notes. Generation-side metrics such as RAGAS faithfulness, answer relevancy, or groundedness require a separate evaluation design and are not inferred from retrieval scores.

Metrics implemented in src/Kioku.Mcp.Server/Domain/RetrievalMetrics.cs:

The golden set

A golden set is a list of real queries annotated with the notes a good search should return, with a graded relevance (1 = somewhat relevant, 2 = relevant, 3 = exactly what the query asks for). Format (src/Kioku.Mcp.Server.Tests/Fixtures/golden-set.json):

{
  "queries": [
    { "id": "q01", "query": "notas sobre burnout laboral",
      "relevant": [ { "path": "Salud/Burnout Laboral.md", "grade": 3 } ] },
    { "id": "q23-no-answer", "query": "quantum entanglement research papers",
      "relevant": [] }
  ]
}

Authoring guidance:

• Use queries you actually type, including typos and paraphrases that share no keywords with the target note (those exercise the semantic leg).
• Include queries answered by tags, by title, by aliases, and by content buried deep in a long note.
• Include 2-3 queries with an empty relevant list (“no-answer probes”): a good configuration returns few or no results for them, so they measure noise/threshold quality.
• Paths are vault-relative with / separators; grades 1-3.

The checked-in fixture vault (src/Kioku.Mcp.Server.Tests/Fixtures/EvalVault/, 27 mixed Spanish/English notes) contains topic clusters, keyword distractors (same words, different meaning), semantic twins (same meaning, different words), alias-only matches and one very long note with a unique fact near the end (a truncation probe: whole-note embeddings get cut at the model context window — heading-aware chunking, see below, is what makes it findable by semantic/hybrid search too, not just keyword).

Running the evaluation

# Keyword only — works without Ollama
dotnet run --project scripts/Kioku.Eval -- --modes keyword --label baseline

# All modes — requires Ollama with the configured embedding model
dotnet run --project scripts/Kioku.Eval -- --label baseline-nomic

# Against your real vault with your own golden set
dotnet run --project scripts/Kioku.Eval -- \
  --vault ~/vault --golden ~/vault/.kioku/golden-set.json --min-score 0.4

# Compare embedding models (cache auto-invalidates on model change)
KIOKU_EMBEDDING_MODEL=qwen3-embedding:0.6b dotnet run --project scripts/Kioku.Eval -- --label qwen3

The runner boots the same VaultIndexService + EmbeddingService + HybridSearchService stack the MCP server uses (no transport), waits for the embedding backlog to drain, and prints one Markdown table per mode. Compare tables between runs with different --labels; only keep a change if Recall@10 / NDCG@10 improve or hold.

CI regression tests

src/Kioku.Mcp.Server.Tests/RetrievalRankingTests.cs runs the golden set through all three search paths on every dotnet test, without Ollama, using a deterministic fake embedder (DeterministicEmbeddingHandler.cs: hashed bag-of-words vectors, so cosine similarity correlates with lexical overlap). Assertions are floors and invariants — never exact orderings — so they catch ranking regressions without overfitting to the fake. Real-model quality is measured only with the runner above.

Baseline

Fixture vault (27 notes), 22 scored queries + 2 no-answer probes.

keyword — naive TF scoring (pre-BM25), 2026-07-12

No-answer probes: avg 5.0 results returned.

keyword — Okapi BM25 (k1=1.2, b=0.75, relative title/tag boosts), 2026-07-12

No-answer probes: avg 5.0 results returned.

Net effect of BM25: better early precision and ordering (P@5, R@5, MRR, NDCG@k all up); Recall@10 dips marginally because IDF demotes one weakly-relevant match. Kept per the “improve or hold” gate — rank quality is what the hybrid RRF fusion consumes.

Semantic threshold default

search_notes with mode='semantic' now defaults to min_score = 0.4 (explicit 0 disables the filter). The value is a conservative starting point for nomic-embed-text with task prefixes; validate it against your own golden set by sweeping --min-score with the runner and watching Precision@k versus the no-answer probes.

semantic / hybrid — nomic-embed-text, before chunking (whole-note embeddings), 2026-07-14

Fixture vault, 26/27 notes embedded — Referencias/Historia de la Computacion.md failed every attempt because its content exceeded the model’s context window (n_ctx_slot = 2048 tokens) and was excluded from semantic/hybrid results entirely (a known limitation of whole-note embeddings, fixed by chunking below). min_score = 0 (the runner’s default; the server itself defaults to 0.4, see above).

Hybrid:

semantic / hybrid — nomic-embed-text, with heading-aware chunking, 2026-07-14

Fixture vault, 27/27 notes embedded — the long note’s oversized sections now split into per-heading chunks (each under the context limit) instead of failing outright. Results still aggregate to one score per note (max-pooling across a note’s chunks), so the golden set — annotated at note level — needed no changes. min_score = 0.

Hybrid:

No-answer probes: avg 10.0 results returned for both semantic and hybrid, before and after (unaffected — min_score = 0; expected noise given no threshold, see above).

Net effect of chunking: every metric improves or holds except hybrid P@10, which dips within noise (0.177 → 0.173). Semantic MRR reaches a perfect 1.000 — the previously-missed q11-long-note-tail query (the fact buried at the end of the long note) is now found by both semantic and hybrid, not just keyword. Kept per the “improve or hold” gate.

dotnet run --project scripts/Kioku.Eval -- --label chunking

Design decisions (what was deliberately not built)

• LLM contextual enrichment (Anthropic contextual retrieval): adds an LLM call per chunk on every re-index. Obsidian vault notes change constantly, so the enrichment cost repeats forever; the deterministic breadcrumb prefix (note name + heading path) used by chunking above captures most of the benefit for free.
• Late chunking: needs token-level embeddings; Ollama’s embeddings API returns pooled vectors only. Not implementable against Ollama.
• Cross-encoder reranking: no local cross-encoder runtime available. Revisit only if eval numbers show Precision@5 is the bottleneck.
• ANN index: brute-force SIMD cosine over a whole vault is sub-10ms at typical vault sizes. Revisit above ~100k vectors.