Hybrid search was silently running vector-only on natural-language
queries — the FTS branch contributed zero candidates. Two causes in the
SQLite (and parallel Postgres) FTS query preparation:
1. Sentence punctuation forced phrase matching. A question like
"When did Melanie paint a sunrise?" reached FTS5 as the exact phrase
'"When did Melanie paint a sunrise?"*', which matches no document.
The FTS5 tokenizer ignores this punctuation in the index, so
stripping it from word edges loses nothing — but leaving it disabled
the entire FTS contribution. _prepare_single_term now strips
?!.,;: from word edges of multi-word queries (interior characters —
hyphens, slashes in permalinks/paths — untouched).
2. No relaxation when strict all-terms-AND matched nothing. Questions
rarely have every word in one document, so even after (1) the
strict AND returned zero rows. The hybrid path now retries once with
an OR-joined, stopword-filtered, content-term query when the strict
query is empty. bm25/ts_rank still rank multi-term matches first, and
fusion with the vector branch keeps relaxed lexical candidates from
dominating precision.
The relaxation is gated behind a new allow_relaxed=False parameter on
SearchRepositoryBase.search; only _search_hybrid opts in. Strict FTS
behavior (search_type=text, title, permalink, link resolution) is
unchanged — the service layer keeps its own conservative fallback.
No config flag, default-safe.
Discovered via the benchmark harness: two different fusion algorithms
produced byte-identical rankings across 1,986 queries (impossible with
two live sources), and instrumentation confirmed fts=0 on 40/40
sampled LoCoMo queries.
Benchmark impact (corrected LoCoMo, 1,986 queries, same index,
retrieval metrics — every category improves, no regression):
recall@5 0.745 -> 0.823 (+7.9)
MRR 0.618 -> 0.718 (+10.0)
headline r5 0.734 -> 0.801, MRR 0.621 -> 0.706
Largest gains on open_domain (+0.10 r5) and adversarial (+0.12 r5);
smallest on temporal (+0.003 r5 / +0.02 MRR).
Tests: punctuation no longer phrase-quotes; relaxation builds the
expected OR query and respects boolean/quoted/short-query intent; the
hybrid opt-in surfaces a partial-overlap document while the default
strict path still returns empty. Parallel coverage for Postgres. Full
SQLite unit suite green (2968 passed); ty + ruff clean.
Signed-off-by: Drew Cain <groksrc@gmail.com>
Root cause: entity, observation, and relation rows in search_index carry
ids from independent auto-increment sequences, so rows of different types
routinely share the same numeric id (guaranteed in young databases).
_search_vector_only parsed each vector hit's chunk_key (e.g. 'entity:4:0')
but discarded the type, and _fetch_search_index_rows_by_ids keyed its
result dict by bare row.id with no type discrimination. Whichever row the
database returned last clobbered the other in the dict; the clobbered hit
then hydrated against the wrong row or found None and was silently
dropped from results. The FTS-filter branch already guarded this with
(id, type) tuples, but the primary vector lookup path and the hybrid
fusion maps missed the same treatment.
Fix: introduce a SearchIndexKey = tuple[str, int] alias and key every map
in the vector/hybrid retrieval path by (type, id) — the similarity and
chunk maps in _search_vector_only, the _fetch_search_index_rows_by_ids
result, the FTS-filter allowed keys, and the rows/fts/vec/fused score
maps in _search_hybrid. The SQL stays unchanged; bare ids are deduped
before the IN query and rows are discriminated by row.type when building
dict keys.
Tests: end-to-end SQLite regression test indexes an entity row and a
relation row sharing id 7, syncs vectors for both, and asserts vector
search returns both rows (and that the entity survives a
search_item_types filter); a hybrid fusion unit test asserts an entity
and relation sharing id 1 stay distinct with single-source scores. Both
fail without the fix. Existing mocked vector tests updated for tuple keys.
Fixes#982
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Three changes to surface more answer text in search results:
- Populate matched_chunk_text for FTS-only hybrid results from content_snippet,
preventing fallback to truncated content when vector search has no match
- Increase TOP_CHUNKS_PER_RESULT from 3 to 5, catching answers in deeper chunks
for large notes (~2700 → ~4500 chars of matched context)
- Increase CONTENT_DISPLAY_LIMIT from 2000 to 4000, doubling the safety-net
content truncation for results without matched_chunk
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Two strategies to improve content hit rate when the right document is found:
1. Small notes (<=2000 chars): return full content_snippet as matched_chunk
so the answer is always present for correctly-retrieved small notes
2. Large notes: return top-3 chunks by similarity joined with \n---\n
instead of just the single best chunk (~2700 chars vs ~900 chars)
Also raises CONTENT_DISPLAY_LIMIT from 250 to 2000 for richer FTS results.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Drew Cain