Addresses Codex review on #994: the hybrid FTS branch opted into
OR-relaxation for every query shape, but SearchService's relaxed FTS
path deliberately rejects short queries and numeric identifiers because
OR-relaxing them over-broadens — and in hybrid the relaxed FTS-only rows
normalize to 1.0 and can outrank the vector result the user wanted
(e.g. "SPEC 16", "root note 1", "New Feature").
relaxed_query_words now enforces the same eligibility as
SearchService._is_relaxed_fts_fallback_eligible: tokenize on
[A-Za-z0-9]+ and return None when there are fewer than three tokens or
any token is a pure digit (in addition to the existing quoted/boolean
guards). Both the SQLite and Postgres relaxed retries route through this
helper, so the hybrid path now relaxes exactly the query shapes the
service does.
Tests updated for the tightened eligibility (short + numeric queries no
longer relax); SQLite + Postgres relaxation suites green, full
repository/search-service/link-resolver suite green (442 passed), ty
clean.
Signed-off-by: Drew Cain <groksrc@gmail.com>
CI Postgres shard caught two issues invisible to the local SQLite suite:
1. Postgres _prepare_single_term regression: the new edge-punctuation
strip ran after special-character cleaning, so an all-special-char
term ("()&!:") collapsed to empty and skipped the existing
NOSPECIALCHARS:* guard, emitting a malformed ":*". Folded the strip
into the word handlers so every guard survives, and added a
single-word empty guard.
2. Backend-specific test assumptions. Four tests in
test_search_repository.py (run under both backends via the
search_repository fixture) asserted SQLite FTS5 syntax and
SQLite-only strict-miss behavior. Postgres to_tsquery('english', ...)
auto-strips stopwords, so "When did Melanie paint a sunrise?" already
matches under strict AND. Made the four tests backend-aware via the
existing is_postgres_backend() helper, and switched the relaxation
integration test to a query with a word absent from the doc
("hiking") so the strict miss holds on both backends.
Reproduced and fixed against real Postgres (testcontainers): full
search test surface green on both backends (53 passed Postgres,
2968 SQLite), ruff + ty clean.
Signed-off-by: Drew Cain <groksrc@gmail.com>
Hybrid search was silently running vector-only on natural-language
queries — the FTS branch contributed zero candidates. Two causes in the
SQLite (and parallel Postgres) FTS query preparation:
1. Sentence punctuation forced phrase matching. A question like
"When did Melanie paint a sunrise?" reached FTS5 as the exact phrase
'"When did Melanie paint a sunrise?"*', which matches no document.
The FTS5 tokenizer ignores this punctuation in the index, so
stripping it from word edges loses nothing — but leaving it disabled
the entire FTS contribution. _prepare_single_term now strips
?!.,;: from word edges of multi-word queries (interior characters —
hyphens, slashes in permalinks/paths — untouched).
2. No relaxation when strict all-terms-AND matched nothing. Questions
rarely have every word in one document, so even after (1) the
strict AND returned zero rows. The hybrid path now retries once with
an OR-joined, stopword-filtered, content-term query when the strict
query is empty. bm25/ts_rank still rank multi-term matches first, and
fusion with the vector branch keeps relaxed lexical candidates from
dominating precision.
The relaxation is gated behind a new allow_relaxed=False parameter on
SearchRepositoryBase.search; only _search_hybrid opts in. Strict FTS
behavior (search_type=text, title, permalink, link resolution) is
unchanged — the service layer keeps its own conservative fallback.
No config flag, default-safe.
Discovered via the benchmark harness: two different fusion algorithms
produced byte-identical rankings across 1,986 queries (impossible with
two live sources), and instrumentation confirmed fts=0 on 40/40
sampled LoCoMo queries.
Benchmark impact (corrected LoCoMo, 1,986 queries, same index,
retrieval metrics — every category improves, no regression):
recall@5 0.745 -> 0.823 (+7.9)
MRR 0.618 -> 0.718 (+10.0)
headline r5 0.734 -> 0.801, MRR 0.621 -> 0.706
Largest gains on open_domain (+0.10 r5) and adversarial (+0.12 r5);
smallest on temporal (+0.003 r5 / +0.02 MRR).
Tests: punctuation no longer phrase-quotes; relaxation builds the
expected OR query and respects boolean/quoted/short-query intent; the
hybrid opt-in surfaces a partial-overlap document while the default
strict path still returns empty. Parallel coverage for Postgres. Full
SQLite unit suite green (2968 passed); ty + ruff clean.
Signed-off-by: Drew Cain <groksrc@gmail.com>
Root cause: entity, observation, and relation rows in search_index carry
ids from independent auto-increment sequences, so rows of different types
routinely share the same numeric id (guaranteed in young databases).
_search_vector_only parsed each vector hit's chunk_key (e.g. 'entity:4:0')
but discarded the type, and _fetch_search_index_rows_by_ids keyed its
result dict by bare row.id with no type discrimination. Whichever row the
database returned last clobbered the other in the dict; the clobbered hit
then hydrated against the wrong row or found None and was silently
dropped from results. The FTS-filter branch already guarded this with
(id, type) tuples, but the primary vector lookup path and the hybrid
fusion maps missed the same treatment.
Fix: introduce a SearchIndexKey = tuple[str, int] alias and key every map
in the vector/hybrid retrieval path by (type, id) — the similarity and
chunk maps in _search_vector_only, the _fetch_search_index_rows_by_ids
result, the FTS-filter allowed keys, and the rows/fts/vec/fused score
maps in _search_hybrid. The SQL stays unchanged; bare ids are deduped
before the IN query and rows are discriminated by row.type when building
dict keys.
Tests: end-to-end SQLite regression test indexes an entity row and a
relation row sharing id 7, syncs vectors for both, and asserts vector
search returns both rows (and that the entity survives a
search_item_types filter); a hybrid fusion unit test asserts an entity
and relation sharing id 1 stay distinct with single-source scores. Both
fail without the fix. Existing mocked vector tests updated for tuple keys.
Fixes#982
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Root cause of the test_sync_entity_circular_relations CI failure (#940,
len(entity_b.outgoing_relations) == 0): the in-memory SQLite URL
(sqlite+aiosqlite://) falls back to SQLAlchemy's StaticPool, which hands the
same DBAPI connection to every concurrently checked-out session. Concurrent
asyncio tasks therefore share one SQLite transaction scope. A rollback issued
through one session — scoped_session's exception handler, or the pool's
reset-on-return at connection checkin (~40 real ROLLBACKs per sync, measured)
— also rolls back any other task's executed-but-uncommitted statements.
During batch indexing, per-file tasks run concurrently (asyncio.gather bounded
by index_entity_max_concurrent). If a sibling session's checkin ROLLBACK lands
between one task's relation INSERT and its COMMIT, the relation row is silently
erased: no error is raised, the sync reports success. The final sync-level
resolve_relations() pass cannot heal this because it only re-queries rows with
to_id IS NULL — the destroyed INSERT leaves no row at all.
Fix: give MEMORY-type engines a single-connection AsyncAdaptedQueuePool
(pool_size=1, max_overflow=0). The lone connection keeps the in-memory
database alive for the engine's lifetime (the reason StaticPool was used),
while the blocking checkout serializes sessions at transaction granularity,
restoring the isolation the repositories assume. File-based SQLite and
Postgres engines are unchanged; production never uses MEMORY engines.
The regression test pins the invariant directly: a session that rolls back in
one task must never destroy another task's uncommitted writes. It fails
deterministically against StaticPool and passes with the serialized pool.
tests/repository/test_entity_repository.py held a scoped session open while
calling a repository method that opens its own session — tolerated on a shared
connection, a deadlock under a serialized pool — so the nested call moved out
of the session block.
Refs #940
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
L2-normalizes FastEmbed output vectors at the provider boundary so SQLite vector scoring keeps its unit-vector contract for custom FastEmbed models such as multilingual MiniLM variants.
Zero vectors are preserved as-is to avoid division errors, and the provider tests cover both non-unit vectors and zero-vector behavior.
Verification:
- uv run pytest tests/repository/test_fastembed_provider.py -q
- uv run ruff check src/basic_memory/repository/fastembed_provider.py tests/repository/test_fastembed_provider.py
- uv run ruff format --check src/basic_memory/repository/fastembed_provider.py tests/repository/test_fastembed_provider.py
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: tk-pkm111 <133480534+tk-pkm111@users.noreply.github.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Adds LiteLLM as a semantic embedding provider, including provider configuration, vector normalization, live-provider evaluation tooling, and documentation for OpenAI, Cohere, Azure Foundry, Azure OpenAI, and NVIDIA NIM-style cases.
Maintainer follow-up on this PR added provider hardening, asymmetric document/query embedding support, dimension-forwarding controls, SQLite/Postgres vector invalidation coverage, and the repeatable live LiteLLM harness.
Verification:
- Full base-repo Tests workflow passed for 3d4e092ceb: https://github.com/basicmachines-co/basic-memory/actions/runs/27072071785
- Live LiteLLM harness passed locally for OpenAI, Cohere, and Azure Foundry.
Co-authored-by: Aarish Alam <arishalam121@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: RheagalFire <arishalam121@gmail.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Add configurable cache_dir, threads, and parallel settings for FastEmbed
to support cloud deployments where defaults fail. Cache embedding providers
at the process level to avoid re-creating heavy ONNX model instances.
- Add semantic_embedding_cache_dir, semantic_embedding_threads, and
semantic_embedding_parallel config fields
- Thread-safe provider cache with double-checked locking in factory
- Forward runtime knobs through to TextEmbedding and embed() calls
- Fix if/elif chain in factory for correct error handling
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Three changes to surface more answer text in search results:
- Populate matched_chunk_text for FTS-only hybrid results from content_snippet,
preventing fallback to truncated content when vector search has no match
- Increase TOP_CHUNKS_PER_RESULT from 3 to 5, catching answers in deeper chunks
for large notes (~2700 → ~4500 chars of matched context)
- Increase CONTENT_DISPLAY_LIMIT from 2000 to 4000, doubling the safety-net
content truncation for results without matched_chunk
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Two strategies to improve content hit rate when the right document is found:
1. Small notes (<=2000 chars): return full content_snippet as matched_chunk
so the answer is always present for correctly-retrieved small notes
2. Large notes: return top-3 chunks by similarity joined with \n---\n
instead of just the single best chunk (~2700 chars vs ~900 chars)
Also raises CONTENT_DISPLAY_LIMIT from 250 to 2000 for richer FTS results.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
RRF compressed all fused scores to ~0.016, destroying ranking differentiation.
The new formula `max(vec, fts) + FUSION_BONUS * min(vec, fts)` preserves
dominant signals and rewards dual-source agreement.
Changes:
- Remove RRF_K constant; add FUSION_BONUS (0.3) and FTS_GATE_THRESHOLD (0.0)
- Use raw vector similarity scores instead of re-normalizing by vec_max
- Zero-score results now produce zero fused score (no 0.1 weight floor)
- Rename test_hybrid_rrf.py → test_hybrid_fusion.py with updated assertions
- Update docs and docstrings to reflect score-based fusion
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
sqlite-vec enforces k <= 4096 for nearest-neighbor queries. Projects with
>4096 vector chunks would crash all vector/hybrid search because
candidate_limit = max(100, (limit + offset) * 10) exceeded this hard limit.
Clamp the knn k in _run_vector_query while keeping the outer SQL LIMIT
unclamped. Only affects SQLite — pgvector has no such constraint.
Fixes#604
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Replace f-string interpolation with parameterized queries for note_types,
search_item_types, and metadata filter paths to prevent SQL injection.
Fixes#591
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Drew Cain