Two follow-up fixes on the same branch:
1. **Purge sqlite-vec embeddings during project delete** (Codex P2)
sqlite-vec stores vectors in a vec0 virtual table keyed by chunk rowid
with no cascade. The previous purge removed search_vector_chunks but
left the embeddings behind; `_run_vector_query` then keeps returning
stale vectors that crowd live results.
ProjectRepository.delete now deletes embeddings first (using the same
rowid-IN-chunks pattern as SQLiteSearchRepository.delete_project_vector_rows),
then the chunk rows. Both deletes are skipped if the underlying table
is absent on a given install. New test test_remove_project_purges_vector_embeddings
covers the happy path and skips cleanly when the embeddings table
isn't initialized.
2. **Fix search_all_projects=True on local installs**
`_search_all_projects` recurses into search_notes with both project=
and project_id= set. project_id (external UUID) routes through the
cloud v2 API path, which 401s on local installs because there's no
JWT to present — so the inner calls silently failed and the merged
result list stayed empty.
The fan-out now mirrors get_project_client's cloud_available composite
(factory mode OR explicit --cloud OR has_cloud_credentials). When that
composite is false we forward project= only and take the name-routed
local-ASGI path. Cloud disambiguation still works because the project
name in project_ref is already the workspace/project qualified_name.
The existing cloud-style fan-out tests now go through a cloud_routing
fixture that pins the three signals; a new local_routing test confirms
project_id is dropped when no cloud route is available.
Signed-off-by: phernandez <paul@basicmachines.co>
SQLite stores search_index as an FTS5 virtual table, which can't carry a
foreign key, so the ON DELETE CASCADE from search_index.project_id to
project.id only applies on Postgres. On SQLite, deleting a project left
its FTS rows behind — and when auto-increment handed the same id to a
new project, the leftover rows masqueraded as the new tenant's data and
leaked into searches scoped to that project.
- ProjectRepository.delete now explicitly purges search_index and
search_vector_chunks for the project id in the same session before
the ORM delete. Idempotent on Postgres (the cascade FK still runs).
- One-time cleanup migration sweeps two leftover shapes: rows whose
project_id is gone, and rows whose entity_id is gone (the larger
class from id reuse). Guarded by table-existence checks so fresh
SQLite installs — where search_index is created at runtime by
init_search_index, not by Alembic — don't fail the upgrade.
- Regression test seeds both derived tables, calls remove_project,
and asserts both come out clean. Verified red on pre-fix code.
Signed-off-by: phernandez <paul@basicmachines.co>
Test was polluted by other tests leaving rows in search_vector_chunks,
causing _needs_semantic_embedding_backfill to return False.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
The previous backfill trigger relied on Alembic revision tracking, but
alembic_version only stores the head revision — intermediate revisions
(like the backfill trigger) are invisible after a multi-step upgrade or
fresh DB creation.
Three changes fix this:
1. Replace Alembic revision check with a simple "entities exist but
embeddings are empty" check that works regardless of migration path
2. Generate embeddings during sync — after FTS indexing, batch-embed all
synced entities at the end of the sync operation
3. Add background backfill at MCP startup for the upgrade path (entities
already exist, no embeddings) without blocking server readiness
Also adds clear startup logging for semantic embedding status so issues
are easy to spot in the logs.
📋 Covers: fresh DB, upgrade from pre-embedding version, db reset,
interrupted backfill
Signed-off-by: Pedro Hernandez <pedro@basicmachines.co>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
🔧#640 — LinkResolver selects worst match instead of best
Replace `min(results, key=lambda x: x.score)` with `results[0]`.
Both SQLite and Postgres return results sorted best-first in SQL,
so using `results[0]` is backend-agnostic and correct.
🔧#641 — search_notes output_format="text" returns raw Pydantic model
Add `_format_search_markdown()` that formats SearchResponse as readable
markdown with title, permalink, score, and matched snippet per result.
Update prompts to use `output_format="json"` since they need structured
data for result counting and branching logic.
🔧#642 — metadata_filters with `note_type` key returns empty results
Add `_METADATA_KEY_ALIASES` mapping at the tool level that aliases
`note_type` → `type` before passing metadata_filters to the search query.
The frontmatter field is `type`, not `note_type`.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Embedding status tests were creating search_vector_chunks inline using
SQLite-only DDL (AUTOINCREMENT). Added Postgres DDL constants to
models/search.py and wired them into the test fixture so both backends
create the table at setup time — matching what the Alembic migration
does in production.
Also fixed stub search_vector_embeddings to use chunk_id (Postgres
column name) instead of rowid, and added inter-test cleanup to prevent
ordering-dependent failures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Add EmbeddingStatus model to project_info schemas and wire it into
ProjectInfoResponse. ProjectService.get_embedding_status() queries
vector tables for chunk/embedding counts, detects orphaned chunks
and missing embeddings, and recommends reindex when appropriate.
Handles both SQLite and Postgres backends. 🔍
Includes 6 unit tests covering: disabled search, missing vector tables,
entities without chunks, orphaned chunks, healthy state, and integration
with get_project_info().
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
build_context now falls back to LinkResolver when an exact permalink lookup
returns empty results. This reuses the same resolution pipeline as read_note
(permalink candidates, title match, file path, FTS) so callers no longer
get empty results for valid note identifiers.
Also changes ensure_frontmatter_on_sync default to True — frontmatter is
now added during sync by default. Tests updated accordingly.
🔧 ContextService accepts optional LinkResolver, wired via DI in all 3 factory variants
✅ 2027 unit + 278 integration tests passing
Closes#582
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Add a new config option to enforce frontmatter on markdown sync when missing, writing derived title/type/permalink and updating in-memory metadata before upsert. Add startup warning when this option is combined with disable_permalinks to make precedence explicit. Add config/sync/initialization tests for the new behavior, and stabilize project list CLI integration assertions by forcing a wide terminal in tests to avoid Rich truncation.
Signed-off-by: phernandez <paul@basicmachines.co>
phernandez