Hybrid search was silently running vector-only on natural-language
queries — the FTS branch contributed zero candidates. Two causes in the
SQLite (and parallel Postgres) FTS query preparation:
1. Sentence punctuation forced phrase matching. A question like
"When did Melanie paint a sunrise?" reached FTS5 as the exact phrase
'"When did Melanie paint a sunrise?"*', which matches no document.
The FTS5 tokenizer ignores this punctuation in the index, so
stripping it from word edges loses nothing — but leaving it disabled
the entire FTS contribution. _prepare_single_term now strips
?!.,;: from word edges of multi-word queries (interior characters —
hyphens, slashes in permalinks/paths — untouched).
2. No relaxation when strict all-terms-AND matched nothing. Questions
rarely have every word in one document, so even after (1) the
strict AND returned zero rows. The hybrid path now retries once with
an OR-joined, stopword-filtered, content-term query when the strict
query is empty. bm25/ts_rank still rank multi-term matches first, and
fusion with the vector branch keeps relaxed lexical candidates from
dominating precision.
The relaxation is gated behind a new allow_relaxed=False parameter on
SearchRepositoryBase.search; only _search_hybrid opts in. Strict FTS
behavior (search_type=text, title, permalink, link resolution) is
unchanged — the service layer keeps its own conservative fallback.
No config flag, default-safe.
Discovered via the benchmark harness: two different fusion algorithms
produced byte-identical rankings across 1,986 queries (impossible with
two live sources), and instrumentation confirmed fts=0 on 40/40
sampled LoCoMo queries.
Benchmark impact (corrected LoCoMo, 1,986 queries, same index,
retrieval metrics — every category improves, no regression):
recall@5 0.745 -> 0.823 (+7.9)
MRR 0.618 -> 0.718 (+10.0)
headline r5 0.734 -> 0.801, MRR 0.621 -> 0.706
Largest gains on open_domain (+0.10 r5) and adversarial (+0.12 r5);
smallest on temporal (+0.003 r5 / +0.02 MRR).
Tests: punctuation no longer phrase-quotes; relaxation builds the
expected OR query and respects boolean/quoted/short-query intent; the
hybrid opt-in surfaces a partial-overlap document while the default
strict path still returns empty. Parallel coverage for Postgres. Full
SQLite unit suite green (2968 passed); ty + ruff clean.
Signed-off-by: Drew Cain <groksrc@gmail.com>
Drew Cain