Test was polluted by other tests leaving rows in search_vector_chunks,
causing _needs_semantic_embedding_backfill to return False.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
The previous backfill trigger relied on Alembic revision tracking, but
alembic_version only stores the head revision — intermediate revisions
(like the backfill trigger) are invisible after a multi-step upgrade or
fresh DB creation.
Three changes fix this:
1. Replace Alembic revision check with a simple "entities exist but
embeddings are empty" check that works regardless of migration path
2. Generate embeddings during sync — after FTS indexing, batch-embed all
synced entities at the end of the sync operation
3. Add background backfill at MCP startup for the upgrade path (entities
already exist, no embeddings) without blocking server readiness
Also adds clear startup logging for semantic embedding status so issues
are easy to spot in the logs.
📋 Covers: fresh DB, upgrade from pre-embedding version, db reset,
interrupted backfill
Signed-off-by: Pedro Hernandez <pedro@basicmachines.co>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
- Add Annotated descriptions to note_types and entity_types parameters so
LLMs can distinguish frontmatter type filtering from knowledge graph item
type filtering (search.py, ui_sdk.py)
- Lowercase note_types values at filter time so "Chapter" matches stored
"chapter"
- Fix misleading entity_types references in schema.py guidance strings
(should be note_types)
- Add permalink pattern documentation note about full path matching
- Add test for note_types case-insensitive lowercasing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
In Postgres test mode, stale dependency_overrides on the module-level
FastAPI app allow _resolve_default_project_from_api() to query a live
database and return 'test-project' even when the test sets
default_project=None. Monkeypatch the async fallback in the three
affected tests to isolate config-based resolution from API leakage.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
In cloud mode, ConfigManager has no local config so default_project
is always None. Add API fallback in resolve_project_parameter that
queries /v2/projects/ for the default_project field. This fixes all
MCP tools that rely on project resolution (recent_activity, etc).
Removed discovery mode tests that simulated an invalid state by
clearing is_default — there must always be a default project.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
In cloud mode, ConfigManager has no local config file so
default_project always returned None. Add async
get_default_project_name() on ProjectService that falls back
to the database is_default flag when ConfigManager returns None.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Add configurable cache_dir, threads, and parallel settings for FastEmbed
to support cloud deployments where defaults fail. Cache embedding providers
at the process level to avoid re-creating heavy ONNX model instances.
- Add semantic_embedding_cache_dir, semantic_embedding_threads, and
semantic_embedding_parallel config fields
- Thread-safe provider cache with double-checked locking in factory
- Forward runtime knobs through to TextEmbedding and embed() calls
- Fix if/elif chain in factory for correct error handling
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Thread constructor no longer receives daemon=True, update mock
signatures to match. Also assert on the new "type": "event" field.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
🔧#640 — LinkResolver selects worst match instead of best
Replace `min(results, key=lambda x: x.score)` with `results[0]`.
Both SQLite and Postgres return results sorted best-first in SQL,
so using `results[0]` is backend-agnostic and correct.
🔧#641 — search_notes output_format="text" returns raw Pydantic model
Add `_format_search_markdown()` that formats SearchResponse as readable
markdown with title, permalink, score, and matched snippet per result.
Update prompts to use `output_format="json"` since they need structured
data for result counting and branching logic.
🔧#642 — metadata_filters with `note_type` key returns empty results
Add `_METADATA_KEY_ALIASES` mapping at the tool level that aliases
`note_type` → `type` before passing metadata_filters to the search query.
The frontmatter field is `type`, not `note_type`.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Update both the compact and extended AI assistant guides with v0.19.0 changes:
- 📝 write_note overwrite guard: callout, edit_note examples, overwrite=True
- 🔍 Expanded search section: all search types, tag: shorthand, filter-only
searches, metadata_filters operators, min_similarity
- ⚠️ "Note already exists" error handling pattern
- 📋 Tool quick reference: updated params, added list_workspaces
- 🔗 memory:// URL: added cross-project format
- ✏️ Best practice: prefer edit_note for updates
Fix tag: shorthand parsing to handle multiple tags anywhere in the query.
Old parser only handled queries starting with "tag:" and broke on
"tag:coffee AND tag:brewing". New parser uses re.findall to extract all
tag:value tokens, strips boolean connectors, and preserves remaining text.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
- Simplify `bm cloud status` output: remove verbose health check details
(status/version/timestamp), show simple "Cloud connected" / "Cloud not
connected" message instead
- Improve `bm reindex --project` error for cloud projects: distinguish
between "project not found" and "project is cloud-only" with a helpful
message explaining reindexing is a local operation
- Improve `bm project list` cloud error message: show the actual error
and soften the credentials suggestion
- Add tests for cloud status command (5 tests)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Add set_workspace_provider() injection point so the cloud MCP server can
list workspaces by querying its own database directly, instead of making
an HTTP round-trip to the control-plane API with credentials it doesn't have.
🔧 Mirrors the existing set_client_factory() pattern in async_client.py
🧪 Adds 3 tests for provider injection, fallback, and context caching
🩹 Updates build_context test assertions for v0.18 backward compat fields
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
When load_config() detects a legacy config format and resaves it, the old
config.json was overwritten in-place with no recovery path. Users switching
between dev and released versions would lose their config.
Now creates config.json.bak before the migration resave so users can revert
if needed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Add #634 (stale schema metadata) to bug fixes section.
Apply ruff formatting to schema.py, write_note.py, test files.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
schema_infer and schema_diff returned raw Pydantic models in text mode,
causing LLMs to render field names as "undefined". Add text formatters
(_format_inference_report, _format_drift_report) matching the existing
_format_validation_report pattern. CLI paths are unaffected — they
always use output_format="json".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
When semantic search is enabled (default), `search_notes(query="tag:security")`
failed because the HYBRID retrieval mode requires non-empty text, but the
service-layer tag: parser clears the text after the mode is already set.
Parse tag: prefix at the tool level before search mode selection, converting it
to a tags filter. This works with all search modes (text, hybrid, vector).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
search_notes was returning individual observations and relations as
separate top-level results, wasting the result limit and creating
confusing UX. Default entity_types to ["entity"] when the caller
doesn't specify it — the entity row already indexes full file content,
so no matches are lost. Users can still override with explicit
entity_types=["observation"] etc.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Fixes issues #29 and #33 from openclaw-basic-memory.
🔧 Identifier resolution (#33):
- Router used get_by_permalink() which only matched exact permalinks.
Replaced with link_resolver.resolve_link() so titles, paths, and
fuzzy matches work consistently with other tools like read_note.
- Set total_entities=1 and total_notes=len(results) on single-note
path for consistency with batch path.
- Guards for "no notes" and "no schema" now fire for identifier-based
validation too, not just note_type-based.
🎨 Text rendering (#29):
- Tool returned raw Pydantic model which LLMs rendered as
"undefined — invalid". Now returns pre-formatted markdown.
- Router uses entity.title (with permalink fallback) as note_identifier
for human-readable output in both text and JSON modes.
- JSON output (output_format="json") unchanged for CLI compatibility.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
frontmatter.Post.__init__ takes `content` and `handler` as positional
parameters. When user YAML frontmatter contains these as field names,
unpacking metadata via **kwargs causes "got multiple values for argument
'content'".
Replace frontmatter.loads() with frontmatter.parse() + Post() + update()
in entity_parser, and replace the **metadata unpacking in
entity_service.update_entity() with the same safe pattern.
Fixes basic-memory-cloud#375
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
YAML block sequence syntax can cause PyYAML to parse scalar fields like
`title` and `type` as lists instead of strings. Downstream code calls
.strip()/.casefold() on these values, crashing with
"'list' object has no attribute 'strip'".
Add _coerce_to_string() helper that joins list items with ", " and apply
it in entity_parser.parse_markdown_content() and
entity_service.fast_edit_entity() where these fields are extracted.
Fixes basic-memory-cloud#376
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
When append or prepend targets a non-existent note, the tool now
creates the file automatically instead of returning an error. This
eliminates silent failures for plugins (like openclaw) that use
edit_note(append) to build daily conversation notes — on the first
message of each day, the note didn't exist yet.
- find_replace and replace_section still require an existing note
- JSON output now includes `fileCreated: bool` in all responses
- Path traversal security check applied to auto-created directories
- Updated error messages to suggest append/prepend for missing notes
🧪 25 unit tests, 14 MCP integration tests, 10 CLI integration tests — all passing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
The picoschema enum-with-description syntax `[val1, val2], description`
is invalid YAML. Users must quote it so YAML parses it as a string.
This adds `_parse_enum_string()` to extract enum values and description
from the resulting string value (e.g., "[active, blocked], current state").
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Three changes to surface more answer text in search results:
- Populate matched_chunk_text for FTS-only hybrid results from content_snippet,
preventing fallback to truncated content when vector search has no match
- Increase TOP_CHUNKS_PER_RESULT from 3 to 5, catching answers in deeper chunks
for large notes (~2700 → ~4500 chars of matched context)
- Increase CONTENT_DISPLAY_LIMIT from 2000 to 4000, doubling the safety-net
content truncation for results without matched_chunk
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Two strategies to improve content hit rate when the right document is found:
1. Small notes (<=2000 chars): return full content_snippet as matched_chunk
so the answer is always present for correctly-retrieved small notes
2. Large notes: return top-3 chunks by similarity joined with \n---\n
instead of just the single best chunk (~2700 chars vs ~900 chars)
Also raises CONTENT_DISPLAY_LIMIT from 250 to 2000 for richer FTS results.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
RRF compressed all fused scores to ~0.016, destroying ranking differentiation.
The new formula `max(vec, fts) + FUSION_BONUS * min(vec, fts)` preserves
dominant signals and rewards dual-source agreement.
Changes:
- Remove RRF_K constant; add FUSION_BONUS (0.3) and FTS_GATE_THRESHOLD (0.0)
- Use raw vector similarity scores instead of re-normalizing by vec_max
- Zero-score results now produce zero fused score (no 0.1 weight floor)
- Rename test_hybrid_rrf.py → test_hybrid_fusion.py with updated assertions
- Update docs and docstrings to reflect score-based fusion
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
- Wrap isatty() in _is_interactive_session() with try/except ValueError
so MCP stdio transport shutdown no longer produces noisy tracebacks
- Check both search_vector_chunks AND search_vector_embeddings exist
before running JOIN queries in get_embedding_status(), fixing
OperationalError when only the chunks table is present
- Add test for closed-stream scenario
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Embedding status tests were creating search_vector_chunks inline using
SQLite-only DDL (AUTOINCREMENT). Added Postgres DDL constants to
models/search.py and wired them into the test fixture so both backends
create the table at setup time — matching what the Alembic migration
does in production.
Also fixed stub search_vector_embeddings to use chunk_id (Postgres
column name) instead of rowid, and added inter-test cleanup to prevent
ordering-dependent failures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
MCP clients may send explicit `null` for unused optional fields.
`expected_replacements: int = 1` caused FastMCP's JSON Schema validation
to reject null before the function body ran. Changed to `Optional[int] = None`
with an effective default resolved inside the function body.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Make `query` optional in `search_notes` so it becomes the single search tool.
Remove `search_by_metadata` entirely — it was unreleased and redundant since
`search_notes` already supports `metadata_filters`, `tags`, and `status`.
- 🔧 `query` param is now `Optional[str] = None`
- 🛡️ Added None guards for project detection and URL resolution
- ✅ Added `no_criteria()` validation with helpful error message
- 🗑️ Deleted `search_by_metadata` tool, imports, tests, and contract entry
- 📝 Updated docs, README, and v0.19.0 release notes
Closes#605
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Add EmbeddingStatus model to project_info schemas and wire it into
ProjectInfoResponse. ProjectService.get_embedding_status() queries
vector tables for chunk/embedding counts, detects orphaned chunks
and missing embeddings, and recommends reindex when appropriate.
Handles both SQLite and Postgres backends. 🔍
Includes 6 unit tests covering: disabled search, missing vector tables,
entities without chunks, orphaned chunks, healthy state, and integration
with get_project_info().
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
sqlite-vec enforces k <= 4096 for nearest-neighbor queries. Projects with
>4096 vector chunks would crash all vector/hybrid search because
candidate_limit = max(100, (limit + offset) * 10) exceeded this hard limit.
Clamp the knn k in _run_vector_query while keeping the outer SQL LIMIT
unclamped. Only affects SQLite — pgvector has no such constraint.
Fixes#604
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Bake in cloud.umami.is host and basic-memory-foss site ID so all
open-source installs send anonymous CLI events by default. Users
can still opt out with BASIC_MEMORY_NO_PROMOS=1 or override the
endpoint via env vars.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
- Prompts (search, continue_conversation) now call MCP tools directly
instead of going through API endpoints, matching the recent_activity
pattern and fixing #526 where prompts returned empty results
- sync_file catches SemanticDependenciesMissingError separately so
entities are still returned successfully when vector embedding fails,
with a clear warning instead of silent failure (#578)
- `bm status` and `bm doctor` default to local routing since they scan
the local filesystem — cloud routing returned Docker-internal paths
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
build_context now falls back to LinkResolver when an exact permalink lookup
returns empty results. This reuses the same resolution pipeline as read_note
(permalink candidates, title match, file path, FTS) so callers no longer
get empty results for valid note identifiers.
Also changes ensure_frontmatter_on_sync default to True — frontmatter is
now added during sync by default. Tests updated accordingly.
🔧 ContextService accepts optional LinkResolver, wired via DI in all 3 factory variants
✅ 2027 unit + 278 integration tests passing
Closes#582
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Drew Cain