Compare commits

...

53 Commits

Author SHA1 Message Date
phernandez 1c2120963a style: normalize spacing in project context tests
Signed-off-by: phernandez <paul@basicmachines.co>
2026-03-05 10:34:56 -06:00
phernandez 7c954ae509 feat: add graph intelligence and fcm contract slice
Signed-off-by: phernandez <paul@basicmachines.co>
2026-03-05 10:00:35 -06:00
phernandez f23dd0474b fix(test): patch API fallback in project_context tests for Postgres
In Postgres test mode, stale dependency_overrides on the module-level
FastAPI app allow _resolve_default_project_from_api() to query a live
database and return 'test-project' even when the test sets
default_project=None. Monkeypatch the async fallback in the three
affected tests to isolate config-based resolution from API leakage.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-03-04 15:34:35 -06:00
phernandez fced804438 fix(test): update integration test for DB default project fallback (#644)
The test previously asserted that write_note fails when ConfigManager
has no default_project. With the API fallback, it now correctly
resolves to the database is_default project. Updated the test to
verify this fallback behavior instead of expecting an error.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-03-04 13:27:03 -06:00
phernandez 1fdc9fdc69 fix: resolve_project_parameter falls back to projects API for default (#644)
In cloud mode, ConfigManager has no local config so default_project
is always None. Add API fallback in resolve_project_parameter that
queries /v2/projects/ for the default_project field. This fixes all
MCP tools that rely on project resolution (recent_activity, etc).

Removed discovery mode tests that simulated an invalid state by
clearing is_default — there must always be a default project.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-03-04 13:17:16 -06:00
phernandez 2feecdfaf7 fix: ChatGPT search/fetch tools broken in cloud mode (#644)
Both search() and fetch() read default_project from ConfigManager,
which returns None in cloud mode. Remove the manual ConfigManager
lookup and let the underlying search_notes/read_note resolve the
project via get_project_client(), which works in both modes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-03-04 12:40:40 -06:00
phernandez 195229f78e fix: resolve default_project returning null in cloud mode (#644)
In cloud mode, ConfigManager has no local config file so
default_project always returned None. Add async
get_default_project_name() on ProjectService that falls back
to the database is_default flag when ConfigManager returns None.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-03-04 11:57:15 -06:00
phernandez d15f6a8427 Add batched vector sync orchestration across repositories 2026-03-03 16:36:06 -06:00
phernandez b8a3a14ad2 Add semantic query timing and FastEmbed parallel guardrails
Signed-off-by: phernandez <paul@basicmachines.co>
2026-03-03 14:04:03 -06:00
phernandez 9b199c6dcb fix: add FastEmbed runtime tuning knobs and provider caching
Add configurable cache_dir, threads, and parallel settings for FastEmbed
to support cloud deployments where defaults fail. Cache embedding providers
at the process level to avoid re-creating heavy ONNX model instances.

- Add semantic_embedding_cache_dir, semantic_embedding_threads, and
  semantic_embedding_parallel config fields
- Thread-safe provider cache with double-checked locking in factory
- Forward runtime knobs through to TextEmbedding and embed() calls
- Fix if/elif chain in factory for correct error handling

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-03-03 10:26:47 -06:00
phernandez fe4a7b1622 fix: update analytics test mocks for non-daemon thread change
Thread constructor no longer receives daemon=True, update mock
signatures to match. Also assert on the new "type": "event" field.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-03-02 21:26:54 -06:00
phernandez 7d2012a82c fix(cli): fix Umami analytics event delivery
Three issues prevented CLI analytics from reaching the Umami dashboard:

1. Wrong API endpoint — cloud.umami.is rejects /api/send, the JS tracker
   uses api-gateway.umami.dev
2. Missing "type": "event" top-level field required by Umami v2 API
3. Non-browser User-Agent ("basic-memory-cli/...") triggers Umami's bot
   detection, which silently drops events with {"beep":"boop"} 🤖
4. Daemon thread was killed before HTTP request completed on fast commands

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-03-02 19:24:10 -06:00
phernandez 7f2d4d2a6f fix: three cloud-testing bugs (#640, #641, #642)
🔧 #640 — LinkResolver selects worst match instead of best
Replace `min(results, key=lambda x: x.score)` with `results[0]`.
Both SQLite and Postgres return results sorted best-first in SQL,
so using `results[0]` is backend-agnostic and correct.

🔧 #641 — search_notes output_format="text" returns raw Pydantic model
Add `_format_search_markdown()` that formats SearchResponse as readable
markdown with title, permalink, score, and matched snippet per result.
Update prompts to use `output_format="json"` since they need structured
data for result counting and branching logic.

🔧 #642 — metadata_filters with `note_type` key returns empty results
Add `_METADATA_KEY_ALIASES` mapping at the tool level that aliases
`note_type` → `type` before passing metadata_filters to the search query.
The frontmatter field is `type`, not `note_type`.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-03-02 12:58:22 -06:00
phernandez 8b7f39ee9b docs: update AI assistant guides for v0.19.0, fix multi-tag search parsing
Update both the compact and extended AI assistant guides with v0.19.0 changes:
- 📝 write_note overwrite guard: callout, edit_note examples, overwrite=True
- 🔍 Expanded search section: all search types, tag: shorthand, filter-only
  searches, metadata_filters operators, min_similarity
- ⚠️ "Note already exists" error handling pattern
- 📋 Tool quick reference: updated params, added list_workspaces
- 🔗 memory:// URL: added cross-project format
- ✏️ Best practice: prefer edit_note for updates

Fix tag: shorthand parsing to handle multiple tags anywhere in the query.
Old parser only handled queries starting with "tag:" and broke on
"tag:coffee AND tag:brewing". New parser uses re.findall to extract all
tag:value tokens, strips boolean connectors, and preserves remaining text.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-03-02 09:50:22 -06:00
phernandez a368d06fd2 fix: improve cloud CLI status and error messages
- Simplify `bm cloud status` output: remove verbose health check details
  (status/version/timestamp), show simple "Cloud connected" / "Cloud not
  connected" message instead
- Improve `bm reindex --project` error for cloud projects: distinguish
  between "project not found" and "project is cloud-only" with a helpful
  message explaining reindexing is a local operation
- Improve `bm project list` cloud error message: show the actual error
  and soften the credentials suggestion
- Add tests for cloud status command (5 tests)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-03-01 20:24:11 -06:00
phernandez 3a2b80b7e9 fix: remove broken CI coverage infrastructure
The coverage collection had multiple issues:
- Wrong pytest markers caused 0 tests to run
- Postgres jobs silently skipped artifact uploads
- Coverage Summary job failed when artifacts were missing
- uv venv picked wrong Python version for coverage jobs

Simplify: every job just runs tests via `just` recipes. No more
dual code paths, artifact uploads, or summary aggregation job.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-03-01 13:18:51 -06:00
phernandez bd5099120f fix: CI Postgres coverage jobs used wrong pytest marker (-m postgres)
The coverage code paths for Postgres unit and integration jobs filtered
with `-m postgres`, but no tests use that marker. Postgres is selected
via BASIC_MEMORY_TEST_POSTGRES=1 env var. This caused 0 tests to run
on Python 3.12 (the only version with coverage: true).

- Postgres unit: remove `-m postgres` (matches `just test-unit-postgres`)
- Postgres integration: use `-m "not semantic"` (matches `just test-int-postgres`)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-03-01 12:47:38 -06:00
phernandez 8e64caf784 fix: list_workspaces bypasses factory pattern on cloud MCP server (#636)
Add set_workspace_provider() injection point so the cloud MCP server can
list workspaces by querying its own database directly, instead of making
an HTTP round-trip to the control-plane API with credentials it doesn't have.

🔧 Mirrors the existing set_client_factory() pattern in async_client.py
🧪 Adds 3 tests for provider injection, fallback, and context caching
🩹 Updates build_context test assertions for v0.18 backward compat fields

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-03-01 12:08:25 -06:00
phernandez ccb5740920 fix: create backup before config migration overwrites old format (#637)
When load_config() detects a legacy config format and resaves it, the old
config.json was overwritten in-place with no recovery path. Users switching
between dev and released versions would lose their config.

Now creates config.json.bak before the migration resave so users can revert
if needed.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-03-01 12:08:25 -06:00
phernandez 373893a7ee fix: restore API backward compatibility for v0.18.x clients (#638)
v0.18.5 clients (homebrew) fail against v0.19.0 servers because:
- `entity_type` was renamed to `note_type` in EntityResponse
- 13 fields gained `Field(exclude=True)` and vanished from JSON

🔧 EntityResponse: add `entity_type` computed_field mirroring `note_type`
🔧 memory.py: remove `exclude=True` from 13 fields across EntitySummary,
   RelationSummary, ObservationSummary, and MemoryMetadata
🔧 Add v0.18 backward-compat contract tests

Closes #638

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-03-01 12:08:25 -06:00
phernandez 1987581ee3 docs: update v0.19.0 release notes and apply formatting fixes
Add #634 (stale schema metadata) to bug fixes section.
Apply ruff formatting to schema.py, write_note.py, test files.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-03-01 12:08:25 -06:00
Drew Cain 59134affa4 fix: read schema definitions from file instead of stale database metadata (#635)
Signed-off-by: Drew Cain <groksrc@gmail.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: phernandez <paul@basicmachines.co>
2026-03-01 09:41:22 -06:00
phernandez c7d97decd6 docs: update v0.19.0 release notes with recent commits
Add write_note overwrite guard (#632) to new capabilities, 8 bug fixes
(#631, #630, #30, #31, #28, plus schema_validate/Post/frontmatter fixes),
and write_note idempotency breaking change to upgrade notes. Bump commit
count from 80+ to 90+.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-28 13:53:29 -06:00
phernandez 5d5efa02a5 fix: format schema_infer and schema_diff as markdown text (#28)
schema_infer and schema_diff returned raw Pydantic models in text mode,
causing LLMs to render field names as "undefined". Add text formatters
(_format_inference_report, _format_drift_report) matching the existing
_format_validation_report pattern. CLI paths are unaffected — they
always use output_format="json".

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-28 12:24:38 -06:00
phernandez 4f12182e28 fix: parse tag: prefix at MCP tool level to avoid hybrid search failure (#30)
When semantic search is enabled (default), `search_notes(query="tag:security")`
failed because the HYBRID retrieval mode requires non-empty text, but the
service-layer tag: parser clears the text after the mode is already set.

Parse tag: prefix at the tool level before search mode selection, converting it
to a tags filter. This works with all search modes (text, hybrid, vector).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-28 11:53:05 -06:00
phernandez 2c5b63c4a2 fix: default search_notes to entity-level results (#31)
search_notes was returning individual observations and relations as
separate top-level results, wasting the result limit and creating
confusing UX. Default entity_types to ["entity"] when the caller
doesn't specify it — the entity row already indexes full file content,
so no matches are lost. Users can still override with explicit
entity_types=["observation"] etc.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-28 11:28:51 -06:00
phernandez c4c9f842ea fix: schema_validate identifier resolution and text rendering
Fixes issues #29 and #33 from openclaw-basic-memory.

🔧 Identifier resolution (#33):
- Router used get_by_permalink() which only matched exact permalinks.
  Replaced with link_resolver.resolve_link() so titles, paths, and
  fuzzy matches work consistently with other tools like read_note.
- Set total_entities=1 and total_notes=len(results) on single-note
  path for consistency with batch path.
- Guards for "no notes" and "no schema" now fire for identifier-based
  validation too, not just note_type-based.

🎨 Text rendering (#29):
- Tool returned raw Pydantic model which LLMs rendered as
  "undefined — invalid". Now returns pre-formatted markdown.
- Router uses entity.title (with permalink fallback) as note_identifier
  for human-readable output in both text and JSON modes.
- JSON output (output_format="json") unchanged for CLI compatibility.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-28 10:40:01 -06:00
phernandez dc0ca71c18 fix: avoid Post(**metadata) crash when frontmatter contains 'content' or 'handler' keys
frontmatter.Post.__init__ takes `content` and `handler` as positional
parameters. When user YAML frontmatter contains these as field names,
unpacking metadata via **kwargs causes "got multiple values for argument
'content'".

Replace frontmatter.loads() with frontmatter.parse() + Post() + update()
in entity_parser, and replace the **metadata unpacking in
entity_service.update_entity() with the same safe pattern.

Fixes basic-memory-cloud#375

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-27 22:52:16 -06:00
phernandez 236ae268aa fix: coerce list frontmatter values to strings for title and type fields
YAML block sequence syntax can cause PyYAML to parse scalar fields like
`title` and `type` as lists instead of strings. Downstream code calls
.strip()/.casefold() on these values, crashing with
"'list' object has no attribute 'strip'".

Add _coerce_to_string() helper that joins list items with ", " and apply
it in entity_parser.parse_markdown_content() and
entity_service.fast_edit_entity() where these fields are extracted.

Fixes basic-memory-cloud#376

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-27 22:45:09 -06:00
Paul Hernandez bd5923a370 feat: add overwrite guard to write_note tool (#632)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 16:12:38 -06:00
jope-bm 4ea5396ddd fix: skip workspace resolution when client factory is active (#630)
Signed-off-by: Joe P <joe@basicmemory.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 14:40:04 -06:00
Paul Hernandez e97eafa55a fix: build_context related_results schema validation failure (#631)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 12:30:44 -06:00
bm-clawd 254e30423d docs: update v0.19.0 release notes with post-draft changes (#626)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 09:36:27 -06:00
phernandez 496af07ced fix: resolve pyright possibly-unbound errors in edit_note
Initialize entity_id and result before the try/except block and
add assertion before the formatting section to help pyright prove
result is always bound on all code paths.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-26 20:10:35 -06:00
bm-clawd b5667f9b55 docs: Add UTM tracking to README links (#611) 2026-02-26 19:55:33 -06:00
phernandez 54b968b93c fix: reduce excessive log volume by demoting per-request noise to DEBUG (#613)
Demote high-frequency per-request/per-item logs from INFO to DEBUG:
- 🔇 Client routing decisions (async_client.py) — logged every MCP tool call
- 🔇 DB migration checks (db.py) — logged every ASGI client creation
- 🔇 Vector table ensure/ready (sqlite + postgres search repos) — logged every search
- 🔇 Per-entity search index start/complete (search_service.py) — logged every file sync
- 🔇 Incremental scan details + per-file permalink updates (sync_service.py)
- 🔇 MCP search tool params and no-results (search.py)
- 📉 Log retention: "10 days" → 5 files (~50MB cap)

API v2 request/response logs remain at INFO for observability.

Closes #613

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-26 19:48:24 -06:00
phernandez e59b5cb6d9 feat: edit_note append/prepend auto-create note if not found (#614)
When append or prepend targets a non-existent note, the tool now
creates the file automatically instead of returning an error. This
eliminates silent failures for plugins (like openclaw) that use
edit_note(append) to build daily conversation notes — on the first
message of each day, the note didn't exist yet.

- find_replace and replace_section still require an existing note
- JSON output now includes `fileCreated: bool` in all responses
- Path traversal security check applied to auto-created directories
- Updated error messages to suggest append/prepend for missing notes

🧪 25 unit tests, 14 MCP integration tests, 10 CLI integration tests — all passing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-26 19:00:34 -06:00
phernandez f0335b998e fix: handle quoted picoschema enum strings in YAML frontmatter (#612)
The picoschema enum-with-description syntax `[val1, val2], description`
is invalid YAML. Users must quote it so YAML parses it as a string.
This adds `_parse_enum_string()` to extract enum values and description
from the resulting string value (e.g., "[active, blocked], current state").

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-26 18:10:48 -06:00
phernandez 66effb04a7 fix: upgrade cryptography and python-multipart for security advisories
- cryptography 46.0.3 → 46.0.5 (subgroup attack on SECT curves)
- python-multipart 0.0.21 → 0.0.22 (arbitrary file write via non-default config)
- pillow alert dismissed — blocked by fastembed 0.7.4 pinning <12.0 (tracking qdrant/fastembed#606)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-26 14:42:20 -06:00
phernandez 9331126ba1 feat: improve content hit rate in search results (#609)
Three changes to surface more answer text in search results:

- Populate matched_chunk_text for FTS-only hybrid results from content_snippet,
  preventing fallback to truncated content when vector search has no match
- Increase TOP_CHUNKS_PER_RESULT from 3 to 5, catching answers in deeper chunks
  for large notes (~2700 → ~4500 chars of matched context)
- Increase CONTENT_DISPLAY_LIMIT from 2000 to 4000, doubling the safety-net
  content truncation for results without matched_chunk

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-26 11:55:38 -06:00
phernandez 3bbb44af0b feat: add --json output to CLI commands for scripting and CI
Add machine-readable JSON output to five CLI commands:
- `bm status --json` — sync report
- `bm project list --json` — structured project list
- `bm schema validate --json` — validation report
- `bm schema infer --json` — inference report
- `bm schema diff --json` — drift report

Refactored `run_status()` to return data instead of printing directly,
improving testability. Follows the established `bm project info --json`
pattern using `print()` for clean JSON (no Rich markup).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-26 10:33:00 -06:00
phernandez f9b2a075a9 feat: return richer content context in search results (#609)
Two strategies to improve content hit rate when the right document is found:

1. Small notes (<=2000 chars): return full content_snippet as matched_chunk
   so the answer is always present for correctly-retrieved small notes
2. Large notes: return top-3 chunks by similarity joined with \n---\n
   instead of just the single best chunk (~2700 chars vs ~900 chars)

Also raises CONTENT_DISPLAY_LIMIT from 250 to 2000 for richer FTS results.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-26 08:36:54 -06:00
phernandez 0a3f3f07f8 fix: run coverage instead of tests on 3.12/ubuntu, not in addition to
Previous commit still ran tests twice on the coverage matrix entry.
Now the 3.12/ubuntu/main combo runs with coverage directly, and all
other matrix entries run without. No duplicate work.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-25 23:36:46 -06:00
phernandez 184ea6d9fa fix: collect coverage from test jobs instead of re-running all tests
The coverage summary job was re-running the entire SQLite + Postgres test
suite from scratch (~60 min), duplicating work already done by upstream jobs.
It consistently timed out.

Now each test job collects coverage data and uploads it as an artifact.
The coverage job just downloads, combines, and reports — should take <1 min.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-25 23:33:15 -06:00
phernandez f5a0e942b0 fix: replace RRF with score-based fusion in hybrid search (#577)
RRF compressed all fused scores to ~0.016, destroying ranking differentiation.
The new formula `max(vec, fts) + FUSION_BONUS * min(vec, fts)` preserves
dominant signals and rewards dual-source agreement.

Changes:
- Remove RRF_K constant; add FUSION_BONUS (0.3) and FTS_GATE_THRESHOLD (0.0)
- Use raw vector similarity scores instead of re-normalizing by vec_max
- Zero-score results now produce zero fused score (no 0.1 weight floor)
- Rename test_hybrid_rrf.py → test_hybrid_fusion.py with updated assertions
- Update docs and docstrings to reflect score-based fusion

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-25 22:53:00 -06:00
phernandez e46555bf2c fix: guard against closed streams in promo and missing vector tables (#579, #607)
- Wrap isatty() in _is_interactive_session() with try/except ValueError
  so MCP stdio transport shutdown no longer produces noisy tracebacks
- Check both search_vector_chunks AND search_vector_embeddings exist
  before running JOIN queries in get_embedding_status(), fixing
  OperationalError when only the chunks table is present
- Add test for closed-stream scenario

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-25 18:01:59 -06:00
phernandez 74e6afdc0f fix: create search_vector_chunks in test fixtures for Postgres compatibility
Embedding status tests were creating search_vector_chunks inline using
SQLite-only DDL (AUTOINCREMENT). Added Postgres DDL constants to
models/search.py and wired them into the test fixture so both backends
create the table at setup time — matching what the Alembic migration
does in production.

Also fixed stub search_vector_embeddings to use chunk_id (Postgres
column name) instead of rowid, and added inter-test cleanup to prevent
ordering-dependent failures.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-25 17:30:35 -06:00
phernandez aa635b8a8b fix: accept null for expected_replacements in edit_note (#606)
MCP clients may send explicit `null` for unused optional fields.
`expected_replacements: int = 1` caused FastMCP's JSON Schema validation
to reject null before the function body ran. Changed to `Optional[int] = None`
with an effective default resolved inside the function body.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-25 14:18:53 -06:00
phernandez 1856d4b462 fix: update test_project_info assertions for dashboard format
The htop-inspired dashboard (3004d0d1) changed the project info output
from "Basic Memory Project Info" / "Statistics" to project name title
with "Knowledge Graph" section. Update test assertions to match.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-25 13:38:01 -06:00
phernandez 73413486bc feat: merge search_by_metadata into search_notes with optional query
Make `query` optional in `search_notes` so it becomes the single search tool.
Remove `search_by_metadata` entirely — it was unreleased and redundant since
`search_notes` already supports `metadata_filters`, `tags`, and `status`.

- 🔧 `query` param is now `Optional[str] = None`
- 🛡️ Added None guards for project detection and URL resolution
-  Added `no_criteria()` validation with helpful error message
- 🗑️ Deleted `search_by_metadata` tool, imports, tests, and contract entry
- 📝 Updated docs, README, and v0.19.0 release notes

Closes #605

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-25 13:30:42 -06:00
phernandez b09eca1698 feat: add EmbeddingStatus schema and get_embedding_status() service method
Add EmbeddingStatus model to project_info schemas and wire it into
ProjectInfoResponse. ProjectService.get_embedding_status() queries
vector tables for chunk/embedding counts, detects orphaned chunks
and missing embeddings, and recommends reindex when appropriate.
Handles both SQLite and Postgres backends. 🔍

Includes 6 unit tests covering: disabled search, missing vector tables,
entities without chunks, orphaned chunks, healthy state, and integration
with get_project_info().

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-25 11:44:50 -06:00
phernandez 3004d0d1fe feat: replace project info with htop-inspired dashboard
Replace Layout-based display (expanded to terminal width) with a compact
Panel using Table.grid(expand=False). Add horizontal bar charts for note
types (top 5), embedding coverage bar with Unicode blocks, and colored
status dots. Removes verbose sections (most connected, recent activity,
available projects) in favor of a dense, visually engaging dashboard. 📊

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-25 11:11:58 -06:00
phernandez b6369d3d14 fix: cap sqlite-vec knn k parameter at 4096 limit
sqlite-vec enforces k <= 4096 for nearest-neighbor queries. Projects with
>4096 vector chunks would crash all vector/hybrid search because
candidate_limit = max(100, (limit + offset) * 10) exceeded this hard limit.

Clamp the knn k in _run_vector_query while keeping the outer SQL LIMIT
unclamped. Only affects SQLite — pgvector has no such constraint.

Fixes #604

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-24 23:23:15 -06:00
121 changed files with 8562 additions and 1318 deletions
+13 -67
View File
@@ -92,7 +92,7 @@ jobs:
run: |
uv pip install -e ".[dev]"
- name: Run tests (SQLite Unit)
- name: Run tests
run: |
just test-unit-sqlite
@@ -139,7 +139,7 @@ jobs:
run: |
uv pip install -e ".[dev]"
- name: Run tests (SQLite Integration)
- name: Run tests
run: |
just test-int-sqlite
@@ -150,7 +150,10 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: [ "3.12", "3.13", "3.14" ]
include:
- python-version: "3.12"
- python-version: "3.13"
- python-version: "3.14"
runs-on: ubuntu-latest
# Note: No services section needed - testcontainers handles Postgres in Docker
@@ -180,7 +183,7 @@ jobs:
run: |
uv pip install -e ".[dev]"
- name: Run tests (Postgres Unit)
- name: Run tests
run: |
just test-unit-postgres
@@ -191,7 +194,10 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: [ "3.12", "3.13", "3.14" ]
include:
- python-version: "3.12"
- python-version: "3.13"
- python-version: "3.14"
runs-on: ubuntu-latest
# Note: No services section needed - testcontainers handles Postgres in Docker
@@ -221,7 +227,7 @@ jobs:
run: |
uv pip install -e ".[dev]"
- name: Run tests (Postgres Integration)
- name: Run tests
run: |
just test-int-postgres
@@ -256,66 +262,6 @@ jobs:
run: |
uv pip install -e ".[dev]"
- name: Run tests (Semantic)
- name: Run tests
run: |
just test-semantic
coverage:
name: Coverage Summary (combined, Python 3.12)
timeout-minutes: 60
needs:
- static-checks
- test-sqlite-unit
- test-sqlite-integration
- test-postgres-unit
- test-postgres-integration
- test-semantic
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python 3.12
uses: actions/setup-python@v4
with:
python-version: "3.12"
cache: "pip"
- name: Install uv
run: |
pip install uv
- uses: extractions/setup-just@v3
- name: Create virtual env
run: |
uv venv
- name: Install dependencies
run: |
uv pip install -e ".[dev]"
- name: Run combined coverage (SQLite + Postgres)
run: |
just coverage
- name: Add coverage report to job summary
if: always()
run: |
{
echo "## Coverage"
echo ""
echo '```'
uv run coverage report -m
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
- name: Upload HTML coverage report
if: always()
uses: actions/upload-artifact@v4
with:
name: htmlcov
path: htmlcov/
+12 -12
View File
@@ -13,7 +13,7 @@
- **Cloud is optional.** The local-first open-source workflow continues as always.
- **OSS discount:** use code `BMFOSS` for 20% off for 3 months.
[Sign up now →](https://basicmemory.com)
[Sign up now →](https://basicmemory.com?utm_source=github&utm_medium=referral&utm_campaign=readme)
with a 7 day free trial
@@ -23,8 +23,9 @@ Basic Memory lets you build persistent knowledge through natural conversations w
Claude, while keeping everything in simple Markdown files on your computer. It uses the Model Context Protocol (MCP) to
enable any compatible LLM to read and write to your local knowledge base.
- Website: https://basicmemory.com
- Documentation: https://docs.basicmemory.com
- Website: [basicmemory.com](https://basicmemory.com?utm_source=github&utm_medium=referral&utm_campaign=readme)
- Documentation: [docs.basicmemory.com](https://docs.basicmemory.com?utm_source=github&utm_medium=referral&utm_campaign=readme)
- Community: [Discord](https://discord.gg/tyvKNccgqN?utm_source=github&utm_medium=referral&utm_campaign=readme)
## Pick up your conversation right where you left off
@@ -438,8 +439,7 @@ list_directory(dir_name, depth) - Browse directory contents with filtering
**Search & Discovery:**
```
search(query, page, page_size) - Search across your knowledge base
search_notes(query, page, page_size, search_type, types, entity_types, after_date, metadata_filters, tags, status, project) - Search with filters
search_by_metadata(filters, limit, offset, project) - Structured frontmatter search
search_notes(query, page, page_size, search_type, types, entity_types, after_date, metadata_filters, tags, status, project) - Search with filters (query is optional for filter-only searches)
```
**Project Management:**
@@ -476,13 +476,13 @@ canvas(nodes, edges, title, folder) - Generate knowledge visualizations
## Futher info
See the [Documentation](https://docs.basicmemory.com) for more info, including:
See the [Documentation](https://docs.basicmemory.com?utm_source=github&utm_medium=referral&utm_campaign=readme) for more info, including:
- [Complete User Guide](https://docs.basicmemory.com/user-guide/)
- [CLI tools](https://docs.basicmemory.com/guides/cli-reference/)
- [Cloud CLI and Sync](https://docs.basicmemory.com/guides/cloud-cli/)
- [Managing multiple Projects](https://docs.basicmemory.com/guides/cli-reference/#project)
- [Importing data from OpenAI/Claude Projects](https://docs.basicmemory.com/guides/cli-reference/#import)
- [Complete User Guide](https://docs.basicmemory.com/user-guide/?utm_source=github&utm_medium=referral&utm_campaign=readme)
- [CLI tools](https://docs.basicmemory.com/guides/cli-reference/?utm_source=github&utm_medium=referral&utm_campaign=readme)
- [Cloud CLI and Sync](https://docs.basicmemory.com/guides/cloud-cli/?utm_source=github&utm_medium=referral&utm_campaign=readme)
- [Managing multiple Projects](https://docs.basicmemory.com/guides/cli-reference/?utm_source=github&utm_medium=referral&utm_campaign=readme#project)
- [Importing data from OpenAI/Claude Projects](https://docs.basicmemory.com/guides/cli-reference/?utm_source=github&utm_medium=referral&utm_campaign=readme#import)
## Telemetry
@@ -635,4 +635,4 @@ and submitting PRs.
</picture>
</a>
Built with ♥️ by Basic Machines
Built with ♥️ by [Basic Machines](https://basicmachines.co?utm_source=github&utm_medium=referral&utm_campaign=readme)
+98 -26
View File
@@ -427,6 +427,8 @@ await write_note(
)
```
> **Important**: `write_note` errors if the note already exists. Use `edit_note` for incremental changes, or pass `overwrite=True` to replace.
**Well-structured note**:
```python
@@ -760,6 +762,9 @@ notes = await read_note(
identifier="memory://specs/*",
project="main"
)
# Cross-project URL (auto-routes to the correct project)
note = await read_note(identifier="memory://research/specs/api-design")
```
```python
@@ -1060,16 +1065,19 @@ results = await search_notes(
project="main"
)
# Metadata-only search
results = await search_by_metadata(
filters={"type": "spec", "status": "in-progress"},
# Metadata-only search (no query needed)
results = await search_notes(
metadata_filters={"type": "spec", "status": "in-progress"},
project="main"
)
```
### Search Types
**Text search (default)**:
Available types: `"text"`, `"title"`, `"permalink"`, `"vector"`/`"semantic"`, `"hybrid"`.
Default is `"hybrid"` when semantic search is enabled, `"text"` otherwise.
**Text search**:
```python
# Full-text search across all content
@@ -1080,15 +1088,50 @@ results = await search_notes(
)
```
**Semantic search**:
**Title and permalink search**:
```python
# Search by title only
results = await search_notes(query="API Design", search_type="title", project="main")
# Search by permalink
results = await search_notes(query="specs/api-design", search_type="permalink", project="main")
```
**Semantic/vector search**:
```python
# Semantic/vector search (if enabled)
results = await search_notes(
query="user login security",
search_type="semantic",
search_type="semantic", # or "vector"
project="main"
)
# Override similarity threshold
results = await search_notes(
query="user login security",
search_type="semantic",
min_similarity=0.5,
project="main"
)
```
**Hybrid search** (combines text + semantic):
```python
results = await search_notes(
query="authentication best practices",
search_type="hybrid",
project="main"
)
```
**Tag shorthand in query**:
```python
# Use tag: prefix as shorthand
results = await search_notes(query="tag:security", project="main")
```
### Search Response
@@ -2161,6 +2204,31 @@ active_project = projects[0]["name"]
results = await search_notes(query="test", project=active_project)
```
### Note Already Exists
**Error**: `write_note` called for a note that already exists
**Solution**:
```python
# Preferred: use edit_note for incremental updates
await edit_note(
identifier="Existing Topic",
operation="append",
content="\n- [update] new information",
project="main"
)
# Alternative: replace the entire note
await write_note(
title="Existing Topic",
content="# Existing Topic\n...",
folder="notes",
overwrite=True,
project="main"
)
```
### Entity Not Found
**Error**: Note doesn't exist
@@ -2716,14 +2784,15 @@ await write_note(
### Content Management
**write_note(title, content, folder, tags, note_type, project)**
- Create or update markdown notes
**write_note(title, content, folder, tags, note_type, overwrite, project)**
- Create new markdown notes (errors if note already exists unless overwrite=True)
- Parameters:
- `title` (required): Note title
- `content` (required): Markdown content
- `folder` (required): Destination folder
- `tags` (optional): List of tags
- `note_type` (optional): Type of note (stored in frontmatter). Can be "note", "person", "meeting", "guide", etc.
- `overwrite` (optional): Set to True to replace an existing note (default: error if exists)
- `project` (required unless default_project_mode): Target project
- Returns: Created/updated entity with permalink
- Example:
@@ -2890,19 +2959,20 @@ contents = await list_directory(
### Search & Discovery
**search_notes(query, page, page_size, search_type, types, entity_types, after_date, metadata_filters, tags, status, project)**
**search_notes(query, page, page_size, search_type, types, entity_types, after_date, metadata_filters, tags, status, min_similarity, project)**
- Search across knowledge base
- Parameters:
- `query` (required): Search query
- `query` (optional): Search query (not required for filter-only searches)
- `page` (optional): Page number (default: 1)
- `page_size` (optional): Results per page (default: 10)
- `search_type` (optional): "text" or "semantic"
- `search_type` (optional): "text", "title", "permalink", "vector"/"semantic", "hybrid" (default: "hybrid" when semantic enabled, "text" otherwise)
- `types` (optional): Entity type filter
- `entity_types` (optional): Observation category filter
- `after_date` (optional): Date filter (ISO format)
- `metadata_filters` (optional): Structured frontmatter filters (dict)
- `tags` (optional): Frontmatter tags filter (list)
- `metadata_filters` (optional): Structured frontmatter filters (dict, supports `$in`, `$gt`, `$gte`, `$lt`, `$lte`, `$between` operators)
- `tags` (optional): Frontmatter tags filter (list); also available via `tag:` query shorthand
- `status` (optional): Frontmatter status filter (string)
- `min_similarity` (optional): Override similarity threshold for vector/hybrid search
- `project` (required unless default_project_mode): Target project
- Returns: Matching entities with scores
- Example:
@@ -2915,18 +2985,11 @@ results = await search_notes(
)
```
**search_by_metadata(filters, limit, offset, project)**
- Metadata-only search using structured frontmatter
- Parameters:
- `filters` (required): Dict of field -> value (supports $in, $gt/$gte/$lt/$lte, $between)
- `limit` (optional): Max results (default: 20)
- `offset` (optional): Pagination offset (default: 0)
- `project` (required unless default_project_mode): Target project
- Returns: Matching entities
- Example:
**Metadata-only search (via search_notes)**
- Use `search_notes` with `metadata_filters` and no `query` for metadata-only searches:
```python
results = await search_by_metadata(
filters={"type": "spec", "status": "in-progress"},
results = await search_notes(
metadata_filters={"type": "spec", "status": "in-progress"},
project="main"
)
```
@@ -2978,6 +3041,15 @@ await delete_project(project_name="old-project")
status = await sync_status(project="main")
```
**list_workspaces()**
- List available workspaces (cloud)
- Parameters: None
- Returns: List of workspaces with metadata
- Example:
```python
workspaces = await list_workspaces()
```
### Visualization
**canvas(nodes, edges, title, folder, project)**
@@ -3246,8 +3318,8 @@ await edit_note(
project="main"
)
# Avoid: Complete rewrite
# (unless necessary for major restructuring)
# When full rewrite is needed, use overwrite=True
await write_note(title="Note", content="...", folder="notes", overwrite=True)
```
### 14. Tagging Strategy
+13 -47
View File
@@ -2,14 +2,9 @@
Basic Memory automatically indexes custom frontmatter fields so you can query them with structured filters. Any YAML key in a note's frontmatter beyond the standard set (`title`, `type`, `tags`, `permalink`, `schema`) is stored as `entity_metadata` and becomes searchable.
## Two Ways to Query
## Querying with `search_notes`
| Tool | Use When |
|------|----------|
| `search_by_metadata` | You only need metadata filters (no text query) |
| `search_notes` | You want to combine a text query with metadata filters |
Both tools accept the same filter syntax.
`search_notes` is the single search tool for all queries — text, metadata filters, or both. The `query` parameter is optional, so you can use metadata filters alone without passing an empty string.
## Filter Syntax
@@ -88,45 +83,16 @@ This queries the `version` key inside a `schema` object in frontmatter.
- `$in` and array-contains require non-empty lists.
- `$between` requires exactly two values `[min, max]`.
## MCP Tools
## MCP Tool — `search_notes`
### `search_by_metadata` — metadata-only search
Searches entities by structured frontmatter metadata without a text query. Results are scoped to entity-level items.
**Parameters:**
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `filters` | dict | Yes | Metadata filter dictionary (see syntax above) |
| `project` | string | No | Project to search in (uses default if omitted) |
| `limit` | int | No | Max results (default 20) |
| `offset` | int | No | Skip N results for pagination (default 0) |
**Example:**
```python
# Find all notes with status "in-progress"
await search_by_metadata({"status": "in-progress"})
# Find high-priority specs in the research project
await search_by_metadata(
{"type": "spec", "priority": {"$in": ["high", "critical"]}},
project="research",
limit=10,
)
```
### `search_notes` with metadata — combined text + metadata
The `search_notes` tool accepts `metadata_filters`, `tags`, and `status` parameters alongside the text `query`. This lets you combine full-text search with structured filtering.
`search_notes` is the single search tool for text queries, metadata filters, or both. The `query` parameter is optional.
**Relevant parameters:**
| Parameter | Type | Description |
|-----------|------|-------------|
| `query` | string | Text search query (can be empty when using only filters) |
| `metadata_filters` | dict | Structured filter dict (same syntax as `search_by_metadata`) |
| `query` | string (optional) | Text search query. Omit for filter-only searches. |
| `metadata_filters` | dict | Structured filter dict (see syntax above) |
| `tags` | list[str] | Convenience shorthand — merged into `metadata_filters["tags"]` |
| `status` | string | Convenience shorthand — merged into `metadata_filters["status"]` |
@@ -138,8 +104,8 @@ The `search_notes` tool accepts `metadata_filters`, `tags`, and `status` paramet
# Text search filtered by metadata
await search_notes("authentication", metadata_filters={"status": "draft"})
# Filter-only search (empty query)
await search_notes("", metadata_filters={"type": "spec"})
# Filter-only search (no query needed)
await search_notes(metadata_filters={"type": "spec"})
# Combine text, tags shortcut, and metadata
await search_notes(
@@ -150,7 +116,7 @@ await search_notes(
# Convenience shortcuts
await search_notes("planning", status="active")
await search_notes("", tags=["tier1", "alpha"])
await search_notes(tags=["tier1", "alpha"])
```
## Tag Search Shortcuts
@@ -260,19 +226,19 @@ confidence: 0.6
```python
# Find all in-progress specs
await search_by_metadata({"status": "in-progress", "type": "spec"})
await search_notes(metadata_filters={"status": "in-progress", "type": "spec"})
# → Auth Design
# Find high-confidence specs
await search_by_metadata({"confidence": {"$gt": 0.7}})
await search_notes(metadata_filters={"confidence": {"$gt": 0.7}})
# → Auth Design (confidence: 0.85)
# Find specs with priority high or medium
await search_by_metadata({"priority": {"$in": ["high", "medium"]}})
await search_notes(metadata_filters={"priority": {"$in": ["high", "medium"]}})
# → Auth Design, Search Redesign
# Find specs in a confidence range
await search_by_metadata({"confidence": {"$between": [0.5, 0.9]}})
await search_notes(metadata_filters={"confidence": {"$between": [0.5, 0.9]}})
# → Auth Design (0.85), Search Redesign (0.6)
# Find notes tagged with security
+89 -14
View File
@@ -4,7 +4,7 @@
v0.19.0 is a major release that introduces semantic vector search, a schema validation system,
project-prefixed permalinks, per-project cloud routing, and a significant upgrade to FastMCP 3.0.
It includes 66 commits since v0.18.0 spanning new features, architectural improvements, and
It includes 90+ commits since v0.18.0 spanning new features, architectural improvements, and
stability fixes across both SQLite and Postgres backends.
---
@@ -16,6 +16,7 @@ stability fixes across both SQLite and Postgres backends.
Full vector and hybrid search for SQLite (via sqlite-vec) and Postgres (via pgvector).
- **Hybrid search mode** combines full-text search (FTS) with vector similarity for best results
- **Score-based fusion** replaces RRF for hybrid ranking — `max(vec, fts) + 0.3 * min(vec, fts)` preserves dominant signals and rewards dual-source agreement (#577)
- **Default search mode** is now `hybrid` when semantic search is enabled, `text` when disabled
- Embedding providers: FastEmbed (local, default) or OpenAI API
- Configurable similarity threshold via `semantic_min_similarity` (default 0.55)
@@ -23,6 +24,7 @@ Full vector and hybrid search for SQLite (via sqlite-vec) and Postgres (via pgve
- Auto-backfill: existing entities get embeddings generated on first startup
- Backend-specific distance-to-similarity conversion (cosine for SQLite, inner product for Postgres)
- FTS fallback: if semantic dependencies are missing, search gracefully degrades to text-only
- sqlite-vec knn `k` parameter capped at 4096 to prevent backend errors
**Configuration:**
```json
@@ -84,6 +86,29 @@ Cloud projects can target specific workspaces for multi-tenant environments.
## New Tools and Capabilities
### Dashboard (`bm project info`)
`bm project info` now displays an htop-inspired compact dashboard with:
- Horizontal bar charts for note types (top 5)
- Embedding coverage bar with Unicode block characters
- Colored status dots for at-a-glance health
- `EmbeddingStatus` schema and `get_embedding_status()` service method for programmatic access
### Unified Metadata Search
`search_by_metadata` has been merged into `search_notes` — one tool for all searches.
`query` is now optional, so you can search purely by frontmatter metadata.
```
search_notes(metadata_filters={"status": "in-progress"})
search_notes(metadata_filters={"tags": ["security", "oauth"]})
search_notes(metadata_filters={"priority": {"$in": ["high", "critical"]}})
search_notes(metadata_filters={"schema.confidence": {"$gt": 0.7}})
search_notes(tags=["security"]) # convenience shorthand
search_notes(status="draft") # convenience shorthand
```
### JSON Output Mode
All MCP tools now support `output_format="json"` for machine-readable responses.
@@ -92,17 +117,6 @@ All MCP tools now support `output_format="json"` for machine-readable responses.
- `build_context` defaults to `"json"` with slimmed payloads (redundant fields stripped)
- CLI tool commands support `--format json` flag
### Structured Metadata Search
New `search_by_metadata` tool for searching by frontmatter fields.
```
search_by_metadata({"status": "in-progress"})
search_by_metadata({"tags": ["security", "oauth"]})
search_by_metadata({"priority": {"$in": ["high", "critical"]}})
search_by_metadata({"schema.confidence": {"$gt": 0.7}})
```
### `tag:` Search Shorthand
Search by tag using convenient shorthand syntax.
@@ -116,14 +130,32 @@ search_notes("tag:coffee AND tag:brewing")
Entities now track `created_by` and `last_updated_by` fields for attribution.
### Matched Chunk Text in Search
### Improved Search Result Content (#609)
Search results now include `matched_chunk` field showing the specific text that matched.
Search results now surface more relevant context:
- `matched_chunk_text` populated for FTS-only hybrid results (no more fallback to truncated content)
- `TOP_CHUNKS_PER_RESULT` increased from 3 to 5, catching answers deeper in large notes (~2700 → ~4500 chars)
- `CONTENT_DISPLAY_LIMIT` doubled from 2000 to 4000 chars for results without matched chunks
### `write_note` Overwrite Guard (#632)
`write_note` is now non-idempotent by default. If a note already exists, the tool returns an
error instead of silently overwriting. Pass `overwrite=True` to replace, or use `edit_note`
for incremental updates. Config option `write_note_overwrite_default` restores the old upsert
behavior.
---
## Architecture Changes
### Score-Based Hybrid Fusion (#577)
RRF (Reciprocal Rank Fusion) compressed all fused scores to ~0.016, destroying ranking
differentiation. The new formula `max(vec, fts) + FUSION_BONUS * min(vec, fts)` preserves
dominant signals and rewards dual-source agreement. Zero-score results now produce zero
fused score instead of receiving a 0.1 weight floor.
### FastMCP 3.0 Upgrade
Upgraded from FastMCP 2.12.3 to 3.0.1.
@@ -176,6 +208,19 @@ Docker-internal paths that don't exist locally.
All `bm tool` subcommands support `--format json` for machine-readable output, enabling
integration with scripts and plugins.
### `--json` for Top-Level CLI Commands
Five additional CLI commands now support `--json` for machine-readable output:
- `bm status --json` — sync report with new/modified/deleted/moved files and skipped files
- `bm project list --json` — structured project list with name, paths, routing mode, and defaults
- `bm schema validate --json` — validation report with per-note pass/fail, warnings, and errors
- `bm schema infer --json` — field frequency analysis and suggested schema definition
- `bm schema diff --json` — drift report with new fields, dropped fields, and cardinality changes
This complements the existing `bm project info --json` and `bm tool --format json` support,
making all major CLI commands scriptable for CI pipelines and automation.
### Cloud Promo and Analytics
- Cloud promo panel shown on first run or version bump with OSS discount code
@@ -188,6 +233,7 @@ integration with scripts and plugins.
## Bug Fixes
- **#577**: RRF fusion compressed all hybrid scores to ~0.016, destroying ranking differentiation
- **#582**: build_context returns empty results on valid note identifiers
- **#575**: Remove hardcoded "main" default from default_project
- **#595**: recent_activity dedup and pagination across MCP tools
@@ -200,6 +246,19 @@ integration with scripts and plugins.
- **#533**: Fix recent_activity prompt defaults
- **#530**: Prevent spurious `metadata: {}` in frontmatter output
- **#601**: Return matched chunk text in search results
- **#606**: Accept `null` for `expected_replacements` in `edit_note`
- **#579, #607**: Guard against closed streams in promo panel and missing vector tables on shutdown
- **#609**: FTS-only hybrid results missing `matched_chunk_text`; content limits too conservative
- **#631**: `build_context` related_results schema validation failure — replaced fragile `_slim_context()` stripping with Pydantic `exclude=True` field config
- **#630**: Skip workspace resolution when client factory is active — prevents 401 errors in cloud MCP server mode
- **#30**: `tag:` prefix query fails with hybrid search — moved tag prefix parsing to MCP tool level so it works with all search modes
- **#31**: `search_notes` returns cluttered observation/relation-level results — now defaults to entity-level results
- **#28**: `schema_infer` and `schema_diff` return raw Pydantic models as "undefined" in LLM output — added markdown formatters
- Fix `schema_validate` identifier resolution (now uses LinkResolver) and text rendering (markdown formatter)
- **#634**: `schema_validate` and `schema_diff` use stale database metadata instead of reading schema definitions from file — now reads frontmatter directly from the file with fallback to database metadata
- Fix `Post(**metadata)` crash when frontmatter contains `content` or `handler` keys
- Fix list-valued frontmatter fields (`title`, `type`) crashing on `.strip()` — now coerced to strings
- Cap sqlite-vec knn `k` parameter at 4096 to prevent backend errors
- Parameterize SQL queries in search repository type filters
- Double-default display in project list
- `ensure_frontmatter_on_sync` default changed to `True`
@@ -208,6 +267,13 @@ integration with scripts and plugins.
---
## Security
- Upgrade `cryptography` for CVE advisory
- Upgrade `python-multipart` for security advisory
---
## Internal / Developer
- **#598**: Upgrade FastMCP 2.12.3 → 3.0.1 with tool annotations
@@ -217,6 +283,8 @@ integration with scripts and plugins.
- **#596**: Fix CLI runtime defects and audit regressions
- CLI refactoring and workspace-aware cloud project listing
- Split and speed up PR test matrix in CI
- Fix CI: collect coverage from test jobs instead of re-running all tests
- Create `search_vector_chunks` in test fixtures for Postgres compatibility
---
@@ -235,9 +303,16 @@ integration with scripts and plugins.
- **Semantic search dependencies** are now included by default. If sqlite-vec fails to load,
search gracefully falls back to FTS. Run `bm reindex --embeddings` to generate embeddings
for existing content.
- **Hybrid search scoring** has changed from RRF to score-based fusion. Search result ordering
may differ — results should be more accurate with better score differentiation.
- **`search_by_metadata`** is removed as a standalone tool. Use `search_notes` with
`metadata_filters` instead (same parameters, same behavior).
- **Project-prefixed permalinks** are enabled by default. Existing notes keep their current
permalinks until modified. Set `permalinks_include_project: false` to disable.
- **Frontmatter on sync** is now enabled by default. Files without frontmatter will have it
added on next sync. Set `ensure_frontmatter_on_sync: false` to preserve old behavior.
- **Config migration** runs automatically for cloud projects with bisync — `local_sync_path`
is promoted to `path` so filesystem operations work correctly.
- **`write_note` is no longer idempotent** — calls to `write_note` for existing notes now
return an error unless `overwrite=True` is passed. Use `edit_note` for incremental changes,
or set `write_note_overwrite_default: true` in config to restore the old behavior.
+7 -7
View File
@@ -165,13 +165,13 @@ Returns results ranked by cosine similarity. Individual observations and relatio
### `hybrid`
Combines FTS and vector results using reciprocal rank fusion (RRF). This is generally the best mode when you want both keyword precision and semantic recall.
Combines FTS and vector results using score-based fusion. This is generally the best mode when you want both keyword precision and semantic recall.
```python
search_notes("authentication security", search_type="hybrid")
```
RRF merges the two ranked lists so that items appearing in both get a score boost, while items found by only one method still appear.
Score-based fusion uses the formula `max(vec, fts) + bonus * min(vec, fts)` to preserve the dominant signal while rewarding results found by both methods.
### When to Use Which
@@ -236,14 +236,14 @@ Each chunk has a `source_hash` (SHA-256 of the chunk text). On re-sync, unchange
### Hybrid Fusion
Hybrid search uses reciprocal rank fusion (RRF) to merge FTS and vector results:
Hybrid search uses score-based fusion to merge FTS and vector results:
1. Run FTS search to get keyword-ranked results
2. Run vector search to get similarity-ranked results
3. For each result, compute: `score = 1/(k + fts_rank) + 1/(k + vector_rank)` where `k = 60`
1. Run FTS search to get keyword-ranked results; normalize scores to [0, 1]
2. Run vector search to get similarity-ranked results (already [0, 1])
3. For each result, compute: `fused = max(vec_score, fts_score) + 0.3 * min(vec_score, fts_score)`
4. Sort by fused score
Items found by both methods get a natural score boost. Items found by only one method still appear but rank lower.
The dominant signal (whichever source scored higher) is preserved, and dual-source agreement adds a bonus. Unlike rank-based fusion, this approach retains score magnitude — a strong vector match stays strong even without an FTS hit.
### Observation-Level Results
+18
View File
@@ -69,6 +69,24 @@ testmon *args:
test-smoke:
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m smoke test-int/mcp/test_smoke_integration.py
# Run graph intelligence API contract tests only
test-graph-intel-api:
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov tests/api/v2/test_graph_intelligence_router.py
# Run graph intelligence MCP tests only
test-graph-intel-mcp:
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov tests/mcp/clients/test_graph_clients.py tests/mcp/test_tool_graph_intelligence.py tests/mcp/test_tool_contracts.py
# Run graph intelligence CLI passthrough tests only
test-graph-intel-cli:
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov tests/cli/test_cli_tool_graph_intelligence_json_output.py
# Run the full graph intelligence fast iteration slice
test-graph-intel:
just test-graph-intel-api
just test-graph-intel-mcp
just test-graph-intel-cli
# Fast local loop: lint, format, typecheck, impacted tests
fast-check:
just fix
@@ -22,10 +22,7 @@ def table_exists(connection, table_name: str) -> bool:
"""Check if a table exists (idempotent migration support)."""
if connection.dialect.name == "postgresql":
result = connection.execute(
text(
"SELECT 1 FROM information_schema.tables "
"WHERE table_name = :table_name"
),
text("SELECT 1 FROM information_schema.tables WHERE table_name = :table_name"),
{"table_name": table_name},
)
return result.fetchone() is not None
+4
View File
@@ -19,6 +19,8 @@ from basic_memory.api.v2.routers import (
prompt_router as v2_prompt,
importer_router as v2_importer,
schema_router as v2_schema,
graph_router as v2_graph,
fcm_router as v2_fcm,
)
from basic_memory.api.v2.routers.project_router import (
add_project,
@@ -86,6 +88,8 @@ app.include_router(v2_directory, prefix="/v2/projects/{project_id}")
app.include_router(v2_prompt, prefix="/v2/projects/{project_id}")
app.include_router(v2_importer, prefix="/v2/projects/{project_id}")
app.include_router(v2_schema, prefix="/v2/projects/{project_id}")
app.include_router(v2_graph, prefix="/v2/projects/{project_id}")
app.include_router(v2_fcm, prefix="/v2/projects/{project_id}")
app.include_router(v2_project, prefix="/v2")
# Legacy web app proxy paths (compat with /proxy/projects/projects)
+4
View File
@@ -21,6 +21,8 @@ from basic_memory.api.v2.routers import (
directory_router,
prompt_router,
importer_router,
graph_router,
fcm_router,
)
__all__ = [
@@ -32,4 +34,6 @@ __all__ = [
"directory_router",
"prompt_router",
"importer_router",
"graph_router",
"fcm_router",
]
@@ -9,6 +9,8 @@ from basic_memory.api.v2.routers.directory_router import router as directory_rou
from basic_memory.api.v2.routers.prompt_router import router as prompt_router
from basic_memory.api.v2.routers.importer_router import router as importer_router
from basic_memory.api.v2.routers.schema_router import router as schema_router
from basic_memory.api.v2.routers.graph_router import router as graph_router
from basic_memory.api.v2.routers.fcm_router import router as fcm_router
__all__ = [
"knowledge_router",
@@ -20,4 +22,6 @@ __all__ = [
"prompt_router",
"importer_router",
"schema_router",
"graph_router",
"fcm_router",
]
@@ -0,0 +1,61 @@
"""V2 router for FCM simulation and interop endpoints."""
from fastapi import APIRouter
from basic_memory.deps import FCMServiceV2ExternalDep, ProjectExternalIdPathDep
from basic_memory.schemas.graph_intelligence import (
FCMExportRequest,
FCMExportResponse,
FCMImportRequest,
FCMImportResponse,
FCMRankActionsRequest,
FCMRankActionsResponse,
FCMSimulateRequest,
FCMSimulateResponse,
)
router = APIRouter(prefix="/fcm", tags=["fcm-v2"])
@router.post("/simulate", response_model=FCMSimulateResponse)
async def fcm_simulate(
request: FCMSimulateRequest,
fcm_service: FCMServiceV2ExternalDep,
project_id: ProjectExternalIdPathDep,
) -> FCMSimulateResponse:
"""Run an FCM scenario simulation."""
_ = project_id
return await fcm_service.simulate(request)
@router.post("/rank-actions", response_model=FCMRankActionsResponse)
async def fcm_rank_actions(
request: FCMRankActionsRequest,
fcm_service: FCMServiceV2ExternalDep,
project_id: ProjectExternalIdPathDep,
) -> FCMRankActionsResponse:
"""Rank action candidates toward a goal."""
_ = project_id
return await fcm_service.rank_actions(request)
@router.post("/import", response_model=FCMImportResponse)
async def fcm_import(
request: FCMImportRequest,
fcm_service: FCMServiceV2ExternalDep,
project_id: ProjectExternalIdPathDep,
) -> FCMImportResponse:
"""Import an FCM model using a supported interchange format."""
_ = project_id
return await fcm_service.import_model(request)
@router.post("/export", response_model=FCMExportResponse)
async def fcm_export(
request: FCMExportRequest,
fcm_service: FCMServiceV2ExternalDep,
project_id: ProjectExternalIdPathDep,
) -> FCMExportResponse:
"""Export an FCM model using a supported interchange format."""
_ = project_id
return await fcm_service.export_model(request)
@@ -0,0 +1,71 @@
"""V2 router for graph intelligence endpoints."""
from fastapi import APIRouter, Query
from basic_memory.deps import (
GraphIntelligenceServiceV2ExternalDep,
ProjectExternalIdPathDep,
TaskSchedulerDep,
)
from basic_memory.schemas.graph_intelligence import (
GraphHealthResponse,
GraphImpactRequest,
GraphImpactResponse,
GraphLineageRequest,
GraphLineageResponse,
GraphReindexRequest,
GraphReindexResponse,
)
router = APIRouter(prefix="/graph", tags=["graph-v2"])
@router.post("/lineage", response_model=GraphLineageResponse)
async def graph_lineage(
request: GraphLineageRequest,
graph_service: GraphIntelligenceServiceV2ExternalDep,
project_id: ProjectExternalIdPathDep,
) -> GraphLineageResponse:
"""Build lineage paths from a start node toward an optional goal."""
_ = project_id
return await graph_service.lineage(request)
@router.post("/impact", response_model=GraphImpactResponse)
async def graph_impact(
request: GraphImpactRequest,
graph_service: GraphIntelligenceServiceV2ExternalDep,
project_id: ProjectExternalIdPathDep,
) -> GraphImpactResponse:
"""Compute impact radius from a target node."""
_ = project_id
return await graph_service.impact(request)
@router.get("/health", response_model=GraphHealthResponse)
async def graph_health(
graph_service: GraphIntelligenceServiceV2ExternalDep,
project_id: ProjectExternalIdPathDep,
scope: str | None = Query(default=None),
timeframe: str | None = Query(default=None),
) -> GraphHealthResponse:
"""Report graph quality metrics and issue candidates."""
_ = project_id
return await graph_service.health(scope=scope, timeframe=timeframe)
@router.post("/reindex", response_model=GraphReindexResponse)
async def graph_reindex(
request: GraphReindexRequest,
graph_service: GraphIntelligenceServiceV2ExternalDep,
task_scheduler: TaskSchedulerDep,
project_id: ProjectExternalIdPathDep,
) -> GraphReindexResponse:
"""Queue a graph reindex operation for the current project."""
task_scheduler.schedule(
"reindex_graph_project",
project_id=project_id,
mode=request.mode,
reason=request.reason,
)
return await graph_service.start_reindex_job()
@@ -130,7 +130,7 @@ async def resolve_identifier(
resolution_method=resolution_method,
)
logger.info(
logger.debug(
f"API v2 response: resolved '{data.identifier}' to external_id={result.external_id} via {resolution_method}"
)
@@ -48,7 +48,7 @@ async def list_projects(
A list of all projects with metadata
"""
projects = await project_service.list_projects()
default_project = project_service.default_project
default_project = await project_service.get_default_project_name()
project_items = [
ProjectItem(
@@ -10,9 +10,15 @@ Flow: Entity loaded with eager observations/relations -> convert to tuples -> co
from pathlib import Path as FilePath
import frontmatter
from fastapi import APIRouter, Path, Query
from loguru import logger
from basic_memory.deps import EntityRepositoryV2ExternalDep
from basic_memory.deps import (
EntityRepositoryV2ExternalDep,
FileServiceV2ExternalDep,
LinkResolverV2ExternalDep,
)
from basic_memory.models.knowledge import Entity
from basic_memory.schemas.schema import (
ValidationReport,
@@ -67,11 +73,54 @@ def _entity_to_note_data(entity: Entity) -> NoteData:
def _entity_frontmatter(entity: Entity) -> dict:
"""Build a frontmatter dict from an entity for schema resolution."""
frontmatter = dict(entity.entity_metadata) if entity.entity_metadata else {}
"""Build a frontmatter dict from an entity's database metadata.
Used for the notes being validated — their type and schema ref are
unlikely to change between syncs.
"""
fm = dict(entity.entity_metadata) if entity.entity_metadata else {}
if entity.note_type:
frontmatter.setdefault("type", entity.note_type)
return frontmatter
fm.setdefault("type", entity.note_type)
return fm
async def _schema_frontmatter_from_file(
file_service: FileServiceV2ExternalDep,
entity: Entity,
) -> dict:
"""Read a schema entity's frontmatter directly from its file.
Schema definitions (field declarations, validation mode) are the source
of truth for validation. Reading from the file ensures schema-validate
always uses the latest settings, even when the file watcher hasn't
synced changes to entity_metadata in the database.
"""
try:
content = await file_service.read_file_content(entity.file_path)
post = frontmatter.loads(content)
metadata = dict(post.metadata)
# Trigger: file is mid-edit and missing required schema fields
# Why: parse_schema_note() raises ValueError for missing entity/schema,
# which would turn validation into a 500 response
# Outcome: fall back to last-known-good database metadata
if not metadata.get("entity") or not isinstance(metadata.get("schema"), dict):
logger.warning(
"Schema file has incomplete frontmatter, falling back to database metadata",
file_path=entity.file_path,
)
return _entity_frontmatter(entity)
return metadata
except Exception:
# Trigger: file is missing, unreadable, or has malformed frontmatter
# Why: fall back to database metadata rather than failing validation entirely
# Outcome: behaves like before this change — uses potentially stale data
logger.warning(
"Failed to read schema file, falling back to database metadata",
file_path=entity.file_path,
)
return _entity_frontmatter(entity)
# --- Validation ---
@@ -80,6 +129,8 @@ def _entity_frontmatter(entity: Entity) -> dict:
@router.post("/schema/validate", response_model=ValidationReport)
async def validate_schema(
entity_repository: EntityRepositoryV2ExternalDep,
file_service: FileServiceV2ExternalDep,
link_resolver: LinkResolverV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
note_type: str | None = Query(None, description="Note type to validate"),
identifier: str | None = Query(None, description="Specific note identifier"),
@@ -88,14 +139,20 @@ async def validate_schema(
Validates a specific note (by identifier) or all notes of a given type.
Returns warnings/errors based on the schema's validation mode.
Schema definitions are read directly from their files to ensure the
latest settings (validation mode, field declarations) are always used,
even when file changes haven't been synced to the database yet.
"""
results: list[NoteValidationResponse] = []
# --- Single note validation ---
if identifier:
entity = await entity_repository.get_by_permalink(identifier)
# Resolve identifier flexibly (permalink, title, path, fuzzy)
# to match how read_note and other tools resolve identifiers
entity = await link_resolver.resolve_link(identifier)
if not entity:
return ValidationReport(note_type=note_type, total_notes=0, results=[])
return ValidationReport(note_type=note_type, total_notes=0, total_entities=0)
frontmatter = _entity_frontmatter(entity)
schema_ref = frontmatter.get("schema")
@@ -106,12 +163,12 @@ async def validate_schema(
query,
allow_reference_match=isinstance(schema_ref, str) and query == schema_ref,
)
return [_entity_frontmatter(e) for e in entities]
return [await _schema_frontmatter_from_file(file_service, e) for e in entities]
schema_def = await resolve_schema(frontmatter, search_fn)
if schema_def:
result = validate_note(
entity.permalink or identifier,
entity.title or entity.permalink or identifier,
schema_def,
_entity_observations(entity),
_entity_relations(entity),
@@ -121,7 +178,8 @@ async def validate_schema(
return ValidationReport(
note_type=note_type or entity.note_type,
total_notes=1,
total_notes=len(results),
total_entities=1,
valid_count=1 if (results and results[0].passed) else 0,
warning_count=sum(len(r.warnings) for r in results),
error_count=sum(len(r.errors) for r in results),
@@ -141,12 +199,12 @@ async def validate_schema(
query,
allow_reference_match=isinstance(schema_ref, str) and query == schema_ref,
)
return [_entity_frontmatter(e) for e in entities]
return [await _schema_frontmatter_from_file(file_service, e) for e in entities]
schema_def = await resolve_schema(frontmatter, search_fn)
if schema_def:
result = validate_note(
entity.permalink or entity.file_path,
entity.title or entity.permalink or entity.file_path,
schema_def,
_entity_observations(entity),
_entity_relations(entity),
@@ -215,6 +273,7 @@ async def infer_schema_endpoint(
@router.get("/schema/diff/{note_type}", response_model=DriftReport)
async def diff_schema_endpoint(
entity_repository: EntityRepositoryV2ExternalDep,
file_service: FileServiceV2ExternalDep,
note_type: str = Path(..., description="Note type to check for drift"),
project_id: str = Path(..., description="Project external UUID"),
):
@@ -227,7 +286,7 @@ async def diff_schema_endpoint(
async def search_fn(query: str) -> list[dict]:
entities = await _find_schema_entities(entity_repository, query)
return [_entity_frontmatter(e) for e in entities]
return [await _schema_frontmatter_from_file(file_service, e) for e in entities]
# Resolve schema by note type
schema_frontmatter = {"type": note_type}
+11 -4
View File
@@ -25,7 +25,7 @@ import basic_memory
# Configuration — defaults baked in, overridable via environment
# ---------------------------------------------------------------------------
_DEFAULT_UMAMI_HOST = "https://cloud.umami.is"
_DEFAULT_UMAMI_HOST = "https://api-gateway.umami.dev"
_DEFAULT_UMAMI_SITE_ID = "f6479898-ebaf-4e60-bce2-6dc60a3f6c5c"
@@ -76,7 +76,9 @@ def track(event_name: str, data: Optional[dict] = None) -> None:
host = _umami_host()
site_id = _umami_site_id()
# Umami v2 /api/send requires "type" at top level alongside "payload"
payload = {
"type": "event",
"payload": {
"hostname": "cli.basicmemory.com",
"language": "en",
@@ -87,7 +89,7 @@ def track(event_name: str, data: Optional[dict] = None) -> None:
"version": basic_memory.__version__,
**(data or {}),
},
}
},
}
def _send():
@@ -97,11 +99,16 @@ def track(event_name: str, data: Optional[dict] = None) -> None:
data=json.dumps(payload).encode("utf-8"),
headers={
"Content-Type": "application/json",
"User-Agent": f"basic-memory-cli/{basic_memory.__version__}",
# Umami's bot detection rejects non-browser User-Agents
"User-Agent": "Mozilla/5.0 (compatible; BasicMemoryCLI/"
f"{basic_memory.__version__})",
},
)
urllib.request.urlopen(req, timeout=3)
except Exception:
pass # Never break the CLI for analytics
threading.Thread(target=_send, daemon=True).start()
# Non-daemon so the process waits for the request to complete.
# The 3s urllib timeout caps the worst-case exit delay.
t = threading.Thread(target=_send)
t.start()
@@ -85,13 +85,13 @@ def logout():
@cloud_app.command("status")
def status() -> None:
"""Check cloud authentication state and cloud instance health."""
"""Check cloud authentication and connection status."""
config_manager = ConfigManager()
config = config_manager.load_config()
auth = CLIAuth(client_id=config.cloud_client_id, authkit_domain=config.cloud_domain)
tokens = auth.load_tokens()
console.print("[bold blue]Cloud Authentication Status[/bold blue]")
console.print("[bold blue]Cloud Status[/bold blue]")
console.print(f" Host: {config.cloud_host}")
console.print(
f" API Key: {'[green]configured[/green]' if config.cloud_api_key else '[yellow]not set[/yellow]'}"
@@ -99,17 +99,12 @@ def status() -> None:
oauth_status = "[yellow]not logged in[/yellow]"
if tokens:
oauth_status = (
"[green]token valid[/green]"
if auth.is_token_valid(tokens)
else "[yellow]token expired[/yellow]"
)
if auth.is_token_valid(tokens):
oauth_status = "[green]token valid[/green]"
else:
oauth_status = "[yellow]token expired[/yellow]"
console.print(f" OAuth: {oauth_status}")
# Get cloud configuration
_, _, host_url = get_cloud_config()
host_url = host_url.rstrip("/")
has_credentials = bool(config.cloud_api_key) or tokens is not None
if not has_credentials:
console.print(
@@ -117,33 +112,20 @@ def status() -> None:
)
return
# Quick connection check — just verify we can reach the cloud
_, _, host_url = get_cloud_config()
host_url = host_url.rstrip("/")
try:
console.print("\n[blue]Checking cloud instance health...[/blue]")
# Make API request to check health
response = run_with_cleanup(make_api_request(method="GET", url=f"{host_url}/proxy/health"))
health_data = response.json()
console.print("[green]Cloud instance is healthy[/green]")
# Display status details
if "status" in health_data:
console.print(f" Status: {health_data['status']}")
if "version" in health_data:
console.print(f" Version: {health_data['version']}")
if "timestamp" in health_data:
console.print(f" Timestamp: {health_data['timestamp']}")
console.print("\n[dim]To sync projects, use: bm project bisync --name <project>[/dim]")
except CloudAPIError as e:
console.print(f"[yellow]Cloud health check failed: {e}[/yellow]")
run_with_cleanup(make_api_request(method="GET", url=f"{host_url}/proxy/health"))
console.print("\n[green]Cloud connected[/green]")
except CloudAPIError:
console.print("\n[yellow]Cloud not connected[/yellow]")
console.print(
"[dim]Try re-authenticating with 'bm cloud login' or setting API key with 'bm cloud api-key save'.[/dim]"
"[dim]Try re-authenticating with 'bm cloud login' or 'bm cloud api-key save'.[/dim]"
)
except Exception as e:
console.print(f"[yellow]Unexpected health check error: {e}[/yellow]")
except Exception:
console.print("\n[yellow]Cloud not connected[/yellow]")
@cloud_app.command("setup")
+11 -2
View File
@@ -11,7 +11,7 @@ from sqlalchemy.exc import OperationalError
from basic_memory import db
from basic_memory.cli.app import app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.config import ConfigManager
from basic_memory.config import ConfigManager, ProjectMode
from basic_memory.repository import ProjectRepository
from basic_memory.services.initialization import reconcile_projects_with_config
from basic_memory.sync.sync_service import get_sync_service
@@ -169,7 +169,16 @@ async def _reindex(app_config, search: bool, embeddings: bool, project: str | No
if project:
projects = [p for p in projects if p.name == project]
if not projects:
console.print(f"[red]Project '{project}' not found.[/red]")
# Check if it's a cloud-only project — those can't be reindexed locally
project_mode = app_config.get_project_mode(project)
if project_mode == ProjectMode.CLOUD:
console.print(
f"[yellow]Project '{project}' is a cloud project.[/yellow]\n"
"Reindexing is a local operation — cloud projects are "
"indexed on the server."
)
else:
console.print(f"[red]Project '{project}' not found.[/red]")
raise typer.Exit(1)
for proj in projects:
+151 -104
View File
@@ -6,9 +6,10 @@ from datetime import datetime
from pathlib import Path
import typer
from rich.console import Console
from rich.console import Console, Group
from rich.panel import Panel
from rich.table import Table
from rich.text import Text
from basic_memory.cli.app import app
from basic_memory.cli.auth import CLIAuth
@@ -44,11 +45,23 @@ def format_path(path: str) -> str:
return path
def make_bar(value: int, max_value: int, width: int = 40) -> Text:
"""Create a horizontal bar chart element using Unicode blocks."""
if max_value == 0:
return Text("" * width, style="dim")
filled = max(1, round(value / max_value * width)) if value > 0 else 0
bar = Text()
bar.append("" * filled, style="cyan")
bar.append("" * (width - filled), style="dim")
return bar
@project_app.command("list")
def list_projects(
local: bool = typer.Option(False, "--local", help="Force local routing for this command"),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
workspace: str = typer.Option(None, "--workspace", help="Cloud workspace name or tenant_id"),
json_output: bool = typer.Option(False, "--json", help="Output in JSON format"),
) -> None:
"""List Basic Memory projects from local and (when available) cloud."""
try:
@@ -84,13 +97,9 @@ def list_projects(
if _has_cloud_credentials(config):
try:
with console.status(
"[bold blue]Fetching cloud projects...", spinner="dots"
):
with console.status("[bold blue]Fetching cloud projects...", spinner="dots"):
with force_routing(cloud=True):
cloud_result = run_with_cleanup(
_list_projects(effective_workspace)
)
cloud_result = run_with_cleanup(_list_projects(effective_workspace))
except Exception as exc: # pragma: no cover
cloud_error = exc
@@ -101,9 +110,7 @@ def list_projects(
try:
from basic_memory.mcp.project_context import get_available_workspaces
with console.status(
"[bold blue]Resolving workspace...", spinner="dots"
):
with console.status("[bold blue]Resolving workspace...", spinner="dots"):
workspaces = run_with_cleanup(get_available_workspaces())
matched = next(
(ws for ws in workspaces if ws.tenant_id == effective_workspace),
@@ -141,6 +148,8 @@ def list_projects(
project_names_by_permalink[permalink] = project.name
cloud_projects_by_permalink[permalink] = project
# --- Build unified project list ---
project_rows: list[dict] = []
for permalink in sorted(project_names_by_permalink):
project_name = project_names_by_permalink[permalink]
local_project = local_projects_by_permalink.get(permalink)
@@ -170,9 +179,9 @@ def list_projects(
else:
cli_route = ProjectMode.LOCAL.value
is_default = "[X]" if config.default_project == project_name else ""
is_default = config.default_project == project_name
has_sync = "[X]" if entry and entry.local_sync_path else ""
has_sync = bool(entry and entry.local_sync_path)
mcp_stdio_target = "local" if local_project is not None else "n/a"
# Show workspace name (type) for cloud-sourced projects
@@ -180,24 +189,48 @@ def list_projects(
if cloud_project is not None and cloud_ws_name:
ws_label = f"{cloud_ws_name} ({cloud_ws_type})" if cloud_ws_type else cloud_ws_name
row = [
project_name,
local_path,
cloud_path,
ws_label,
cli_route,
mcp_stdio_target,
has_sync,
is_default,
]
row_data = {
"name": project_name,
"permalink": permalink,
"local_path": local_path,
"cloud_path": cloud_path,
"cli_route": cli_route,
"mcp_stdio": mcp_stdio_target,
"sync": has_sync,
"is_default": is_default,
}
if ws_label:
row_data["workspace"] = cloud_ws_name or ""
if cloud_ws_type:
row_data["workspace_type"] = cloud_ws_type
table.add_row(*row)
project_rows.append(row_data)
# --- JSON output ---
if json_output:
print(json.dumps({"projects": project_rows}, indent=2, default=str))
return
# --- Rich table output ---
for row_data in project_rows:
table.add_row(
row_data["name"],
row_data["local_path"],
row_data["cloud_path"],
row_data.get("workspace", "")
+ (f" ({row_data['workspace_type']})" if row_data.get("workspace_type") else ""),
row_data["cli_route"],
row_data["mcp_stdio"],
"[X]" if row_data["sync"] else "",
"[X]" if row_data["is_default"] else "",
)
console.print(table)
if cloud_error is not None:
console.print(f"[yellow]Cloud project discovery failed: {cloud_error}[/yellow]")
console.print(
"[yellow]Cloud project discovery failed. "
"Showing local projects only. Run 'bm cloud login' or 'bm cloud api-key save <key>'.[/yellow]"
"[dim]Showing local projects only. "
"Run 'bm cloud login' or 'bm cloud api-key save <key>' if this is a credentials issue.[/dim]"
)
except Exception as e:
console.print(f"[red]Error listing projects: {str(e)}[/red]")
@@ -758,99 +791,113 @@ def display_project_info(
# Convert to JSON and print
print(json.dumps(info.model_dump(), indent=2, default=str))
else:
# Project configuration section
console.print(
Panel(
f"Basic Memory version: [bold green]{info.system.version}[/bold green]\n"
f"[bold]Project:[/bold] {info.project_name}\n"
f"[bold]Path:[/bold] {info.project_path}\n"
f"[bold]Default Project:[/bold] {info.default_project}\n",
title="Basic Memory Project Info",
expand=False,
)
)
# --- Left column: Knowledge Graph stats ---
left = Table.grid(padding=(0, 2))
left.add_column("metric", style="cyan")
left.add_column("value", style="green", justify="right")
# Statistics section
stats_table = Table(title="Statistics")
stats_table.add_column("Metric", style="cyan")
stats_table.add_column("Count", style="green")
left.add_row("[bold]Knowledge Graph[/bold]", "")
left.add_row("Entities", str(info.statistics.total_entities))
left.add_row("Observations", str(info.statistics.total_observations))
left.add_row("Relations", str(info.statistics.total_relations))
left.add_row("Unresolved", str(info.statistics.total_unresolved_relations))
left.add_row("Isolated", str(info.statistics.isolated_entities))
stats_table.add_row("Entities", str(info.statistics.total_entities))
stats_table.add_row("Observations", str(info.statistics.total_observations))
stats_table.add_row("Relations", str(info.statistics.total_relations))
stats_table.add_row(
"Unresolved Relations", str(info.statistics.total_unresolved_relations)
)
stats_table.add_row("Isolated Entities", str(info.statistics.isolated_entities))
# --- Right column: Embeddings ---
right = Table.grid(padding=(0, 2))
right.add_column("property", style="cyan")
right.add_column("value", style="green")
console.print(stats_table)
right.add_row("[bold]Embeddings[/bold]", "")
if info.embedding_status:
es = info.embedding_status
if not es.semantic_search_enabled:
right.add_row("[green]●[/green] Semantic Search", "Disabled")
else:
right.add_row("[green]●[/green] Semantic Search", "Enabled")
if es.embedding_provider:
right.add_row(" Provider", es.embedding_provider)
if es.embedding_model:
right.add_row(" Model", es.embedding_model)
# Embedding coverage bar
if es.total_indexed_entities > 0:
coverage_bar = make_bar(
es.total_entities_with_chunks,
es.total_indexed_entities,
width=20,
)
count_text = Text(
f" {es.total_entities_with_chunks}/{es.total_indexed_entities}",
style="green",
)
bar_with_count = Text.assemble(" Indexed ", coverage_bar, count_text)
right.add_row(bar_with_count, "")
right.add_row(" Chunks", str(es.total_chunks))
if es.reindex_recommended:
right.add_row(
"[yellow]●[/yellow] Status",
"[yellow]Reindex recommended[/yellow]",
)
if es.reindex_reason:
right.add_row(" Reason", f"[yellow]{es.reindex_reason}[/yellow]")
else:
right.add_row("[green]●[/green] Status", "[green]Up to date[/green]")
# Note types
# --- Compose two-column layout (content-sized, NOT Layout) ---
columns = Table.grid(padding=(0, 4), expand=False)
columns.add_row(left, right)
# --- Note Types bar chart (top 5 by count) ---
bars_section = None
if info.statistics.note_types:
note_types_table = Table(title="Note Types")
note_types_table.add_column("Type", style="blue")
note_types_table.add_column("Count", style="green")
sorted_types = sorted(
info.statistics.note_types.items(), key=lambda x: x[1], reverse=True
)
top_types = sorted_types[:5]
max_count = top_types[0][1] if top_types else 1
for note_type, count in info.statistics.note_types.items():
note_types_table.add_row(note_type, str(count))
bars = Table.grid(padding=(0, 2), expand=False)
bars.add_column("type", style="cyan", width=16, justify="right")
bars.add_column("bar")
bars.add_column("count", style="green", justify="right")
console.print(note_types_table)
for note_type, count in top_types:
bars.add_row(note_type, make_bar(count, max_count), str(count))
# Most connected entities
if info.statistics.most_connected_entities: # pragma: no cover
connected_table = Table(title="Most Connected Entities")
connected_table.add_column("Title", style="blue")
connected_table.add_column("Permalink", style="cyan")
connected_table.add_column("Relations", style="green")
remaining = len(sorted_types) - len(top_types)
bars_section = Group(
"[bold]Note Types[/bold]",
bars,
f"[dim]+{remaining} more types[/dim]" if remaining > 0 else "",
)
for entity in info.statistics.most_connected_entities:
connected_table.add_row(
entity["title"], entity["permalink"], str(entity["relation_count"])
)
console.print(connected_table)
# Recent activity
if info.activity.recently_updated: # pragma: no cover
recent_table = Table(title="Recent Activity")
recent_table.add_column("Title", style="blue")
recent_table.add_column("Type", style="cyan")
recent_table.add_column("Last Updated", style="green")
for entity in info.activity.recently_updated[:5]: # Show top 5
updated_at = (
datetime.fromisoformat(entity["updated_at"])
if isinstance(entity["updated_at"], str)
else entity["updated_at"]
)
recent_table.add_row(
entity["title"],
entity["note_type"],
updated_at.strftime("%Y-%m-%d %H:%M"),
)
console.print(recent_table)
# Available projects
projects_table = Table(title="Available Projects")
projects_table.add_column("Name", style="blue")
projects_table.add_column("Path", style="cyan")
projects_table.add_column("Default", style="green")
for name, proj_info in info.available_projects.items():
is_default = name == info.default_project
project_path = proj_info["path"]
projects_table.add_row(name, project_path, "[X]" if is_default else "")
console.print(projects_table)
# Timestamp
# --- Footer ---
current_time = (
datetime.fromisoformat(str(info.system.timestamp))
if isinstance(info.system.timestamp, str)
else info.system.timestamp
)
console.print(f"\nTimestamp: [cyan]{current_time.strftime('%Y-%m-%d %H:%M:%S')}[/cyan]")
footer = (
f"[dim]{format_path(info.project_path)} "
f"default: {info.default_project} "
f"{current_time.strftime('%Y-%m-%d %H:%M')}[/dim]"
)
# --- Assemble dashboard ---
parts: list = [columns, ""]
if bars_section:
parts.extend([bars_section, ""])
parts.append(footer)
body = Group(*parts)
console.print(
Panel(
body,
title=f"[bold]{info.project_name}[/bold]",
subtitle=f"Basic Memory {info.system.version}",
expand=False,
)
)
except typer.Exit:
raise
+36 -7
View File
@@ -173,6 +173,7 @@ def validate(
typer.Option(help="The project name."),
] = None,
strict: bool = typer.Option(False, "--strict", help="Exit with error on validation failures"),
json_output: bool = typer.Option(False, "--json", help="Output in JSON format"),
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
@@ -183,6 +184,7 @@ def validate(
TARGET can be a note path (e.g., people/ada-lovelace.md) or a note type
(e.g., person). If omitted, validates all notes that have schemas.
Use --json for machine-readable output.
Use --strict to exit with error code 1 if any validation errors are found.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
@@ -211,12 +213,19 @@ def validate(
# Handle error responses
if isinstance(result, dict) and "error" in result:
console.print(f"[yellow]{result['error']}[/yellow]")
if json_output:
print(json.dumps(result, indent=2, default=str))
else:
console.print(f"[yellow]{result['error']}[/yellow]")
return
# output_format="json" guarantees a dict return
assert isinstance(result, dict)
_render_validate_table(result)
if json_output:
print(json.dumps(result, indent=2, default=str))
else:
_render_validate_table(result)
if strict and result.get("error_count", 0) > 0:
raise typer.Exit(1)
@@ -245,6 +254,7 @@ def infer(
0.25, "--threshold", help="Minimum frequency for optional fields (0-1)"
),
save: bool = typer.Option(False, "--save", help="Save inferred schema to schema/ directory"),
json_output: bool = typer.Option(False, "--json", help="Output in JSON format"),
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
@@ -258,6 +268,7 @@ def infer(
Fields present in 95%+ of notes become required. Fields above the
threshold (default 25%) become optional. Fields below threshold are excluded.
Use --json for machine-readable output.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
@@ -277,7 +288,10 @@ def infer(
# Handle error responses
if isinstance(result, dict) and "error" in result:
console.print(f"[yellow]{result['error']}[/yellow]")
if json_output:
print(json.dumps(result, indent=2, default=str))
else:
console.print(f"[yellow]{result['error']}[/yellow]")
return
# output_format="json" guarantees a dict return
@@ -285,10 +299,16 @@ def infer(
# Handle zero notes
if result.get("notes_analyzed", 0) == 0:
console.print(f"[yellow]No notes found with type: {note_type}[/yellow]")
if json_output:
print(json.dumps(result, indent=2, default=str))
else:
console.print(f"[yellow]No notes found with type: {note_type}[/yellow]")
return
_render_infer_table(result)
if json_output:
print(json.dumps(result, indent=2, default=str))
else:
_render_infer_table(result)
if save:
console.print(
@@ -316,6 +336,7 @@ def diff(
Optional[str],
typer.Option(help="The project name."),
] = None,
json_output: bool = typer.Option(False, "--json", help="Output in JSON format"),
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
@@ -327,6 +348,7 @@ def diff(
are actually structured. Identifies new fields,
dropped fields, and cardinality changes.
Use --json for machine-readable output.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
@@ -345,12 +367,19 @@ def diff(
# Handle error responses
if isinstance(result, dict) and "error" in result:
console.print(f"[yellow]{result['error']}[/yellow]")
if json_output:
print(json.dumps(result, indent=2, default=str))
else:
console.print(f"[yellow]{result['error']}[/yellow]")
return
# output_format="json" guarantees a dict return
assert isinstance(result, dict)
_render_diff_output(result)
if json_output:
print(json.dumps(result, indent=2, default=str))
else:
_render_diff_output(result)
except ValueError as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
+31 -16
View File
@@ -1,5 +1,6 @@
"""Status command for basic-memory CLI."""
import json
from typing import Set, Dict
from typing import Annotated, Optional
@@ -141,21 +142,20 @@ def display_changes(
console.print(Panel(tree, expand=False))
async def run_status(project: Optional[str] = None, verbose: bool = False): # pragma: no cover
"""Check sync status of files vs database."""
async def run_status(
project: Optional[str] = None,
) -> tuple[str, SyncReportResponse]:
"""Fetch sync status of files vs database.
Returns (project_name, sync_report) for the caller to render.
"""
# Resolve default project so get_client() can route per-project
project = project or ConfigManager().default_project
try:
async with get_client(project_name=project) as client:
project_item = await get_active_project(client, project, None)
sync_report = await ProjectClient(client).get_status(project_item.external_id)
display_changes(project_item.name, "Status", sync_report, verbose)
except (ValueError, ToolError) as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
async with get_client(project_name=project) as client:
project_item = await get_active_project(client, project, None)
sync_report = await ProjectClient(client).get_status(project_item.external_id)
return project_item.name, sync_report
@app.command()
@@ -165,6 +165,7 @@ def status(
typer.Option(help="The project name."),
] = None,
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed file information"),
json_output: bool = typer.Option(False, "--json", help="Output in JSON format"),
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
@@ -172,6 +173,7 @@ def status(
):
"""Show sync status between files and database.
Use --json for machine-readable output.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
@@ -187,11 +189,24 @@ def status(
if not local and not cloud:
local = True
with force_routing(local=local, cloud=cloud):
run_with_cleanup(run_status(project, verbose)) # pragma: no cover
except ValueError as e:
console.print(f"[red]Error: {e}[/red]")
project_name, sync_report = run_with_cleanup(run_status(project))
if json_output:
print(json.dumps(sync_report.model_dump(mode="json"), indent=2, default=str))
else:
display_changes(project_name, "Status", sync_report, verbose)
except (ValueError, ToolError) as e:
if json_output:
print(json.dumps({"error": str(e)}, indent=2))
else:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(code=1)
except typer.Exit:
raise
except Exception as e:
logger.error(f"Error checking status: {e}")
typer.echo(f"Error checking status: {e}", err=True)
if json_output:
print(json.dumps({"error": str(e)}, indent=2))
else:
typer.echo(f"Error checking status: {e}", err=True)
raise typer.Exit(code=1) # pragma: no cover
+385 -1
View File
@@ -16,6 +16,13 @@ from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.cli.commands.routing import force_routing, validate_routing_flags
from basic_memory.mcp.tools import build_context as mcp_build_context
from basic_memory.mcp.tools import edit_note as mcp_edit_note
from basic_memory.mcp.tools import fcm_export_model as mcp_fcm_export_model
from basic_memory.mcp.tools import fcm_import_model as mcp_fcm_import_model
from basic_memory.mcp.tools import fcm_rank_actions as mcp_fcm_rank_actions
from basic_memory.mcp.tools import fcm_simulate as mcp_fcm_simulate
from basic_memory.mcp.tools import graph_health as mcp_graph_health
from basic_memory.mcp.tools import graph_impact as mcp_graph_impact
from basic_memory.mcp.tools import graph_lineage as mcp_graph_lineage
from basic_memory.mcp.tools import list_memory_projects as mcp_list_projects
from basic_memory.mcp.tools import list_workspaces as mcp_list_workspaces
from basic_memory.mcp.tools import read_note as mcp_read_note
@@ -40,6 +47,17 @@ def _print_json(result: Any) -> None:
print(json.dumps(result, indent=2, ensure_ascii=True, default=str))
def _parse_json_option(raw_value: Optional[str], option_name: str) -> Any:
"""Parse a JSON CLI option with deterministic error handling."""
if raw_value is None:
return None
try:
return json.loads(raw_value)
except json.JSONDecodeError as exc:
typer.echo(f"Invalid JSON for {option_name}: {exc}", err=True)
raise typer.Exit(1)
# --- Commands ---
@@ -366,6 +384,372 @@ def recent_activity(
raise
@tool_app.command("graph-lineage")
def graph_lineage(
start: Annotated[str, typer.Argument(help="Start node identifier or memory:// reference")],
goal: Annotated[
Optional[str],
typer.Option("--goal", help="Optional goal node identifier for targeted lineage"),
] = None,
max_hops: int = typer.Option(4, "--max-hops", help="Maximum traversal hops (1-6)"),
relation_filters: Annotated[
Optional[List[str]],
typer.Option("--relation-filter", help="Relation filters (repeatable)"),
] = None,
project: Annotated[
Optional[str],
typer.Option(help="The project to use. If not provided, the default project will be used."),
] = None,
workspace: Annotated[
Optional[str],
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
] = None,
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Get graph lineage paths from a start node."""
try:
validate_routing_flags(local, cloud)
with force_routing(local=local, cloud=cloud):
result = run_with_cleanup(
mcp_graph_lineage(
start=start,
goal=goal,
max_hops=max_hops,
relation_filters=relation_filters or [],
project=project,
workspace=workspace,
output_format="json",
)
)
_print_json(result)
except ValueError as e:
typer.echo(f"Error: {e}", err=True)
raise typer.Exit(1)
except Exception as e: # pragma: no cover
if not isinstance(e, typer.Exit):
typer.echo(f"Error during graph_lineage: {e}", err=True)
raise typer.Exit(1)
raise
@tool_app.command("graph-impact")
def graph_impact(
target: Annotated[str, typer.Argument(help="Target node identifier or memory:// reference")],
horizon: int = typer.Option(2, "--horizon", help="Impact horizon in hops (1-4)"),
relation_filters: Annotated[
Optional[List[str]],
typer.Option("--relation-filter", help="Relation filters (repeatable)"),
] = None,
include_reasons: bool = typer.Option(
True,
"--include-reasons/--no-include-reasons",
help="Include reason strings in impact output",
),
project: Annotated[
Optional[str],
typer.Option(help="The project to use. If not provided, the default project will be used."),
] = None,
workspace: Annotated[
Optional[str],
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
] = None,
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Get impact radius for a target node."""
try:
validate_routing_flags(local, cloud)
with force_routing(local=local, cloud=cloud):
result = run_with_cleanup(
mcp_graph_impact(
target=target,
horizon=horizon,
relation_filters=relation_filters or [],
include_reasons=include_reasons,
project=project,
workspace=workspace,
output_format="json",
)
)
_print_json(result)
except ValueError as e:
typer.echo(f"Error: {e}", err=True)
raise typer.Exit(1)
except Exception as e: # pragma: no cover
if not isinstance(e, typer.Exit):
typer.echo(f"Error during graph_impact: {e}", err=True)
raise typer.Exit(1)
raise
@tool_app.command("graph-health")
def graph_health(
scope: Annotated[Optional[str], typer.Option("--scope", help="Optional scope prefix")] = None,
timeframe: Annotated[
Optional[str], typer.Option("--timeframe", help="Optional timeframe filter")
] = None,
project: Annotated[
Optional[str],
typer.Option(help="The project to use. If not provided, the default project will be used."),
] = None,
workspace: Annotated[
Optional[str],
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
] = None,
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Get graph health metrics and issue candidates."""
try:
validate_routing_flags(local, cloud)
with force_routing(local=local, cloud=cloud):
result = run_with_cleanup(
mcp_graph_health(
scope=scope,
timeframe=timeframe,
project=project,
workspace=workspace,
output_format="json",
)
)
_print_json(result)
except ValueError as e:
typer.echo(f"Error: {e}", err=True)
raise typer.Exit(1)
except Exception as e: # pragma: no cover
if not isinstance(e, typer.Exit):
typer.echo(f"Error during graph_health: {e}", err=True)
raise typer.Exit(1)
raise
@tool_app.command("fcm-simulate")
def fcm_simulate(
actions_json: Annotated[
str,
typer.Option(
"--actions-json",
help='JSON array of actions, e.g. [{"node_id":"n1","delta":0.2}]',
),
],
scenario_json: Annotated[
Optional[str],
typer.Option("--scenario-json", help="Optional JSON scenario object"),
] = None,
clamp_rules_json: Annotated[
Optional[str],
typer.Option("--clamp-rules-json", help="Optional JSON array of clamp rules"),
] = None,
project: Annotated[
Optional[str],
typer.Option(help="The project to use. If not provided, the default project will be used."),
] = None,
workspace: Annotated[
Optional[str],
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
] = None,
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Run an FCM simulation."""
actions = _parse_json_option(actions_json, "--actions-json")
scenario = _parse_json_option(scenario_json, "--scenario-json")
clamp_rules = _parse_json_option(clamp_rules_json, "--clamp-rules-json")
if not isinstance(actions, list):
typer.echo("Invalid JSON for --actions-json: expected a JSON array", err=True)
raise typer.Exit(1)
if scenario is not None and not isinstance(scenario, dict):
typer.echo("Invalid JSON for --scenario-json: expected a JSON object", err=True)
raise typer.Exit(1)
if clamp_rules is not None and not isinstance(clamp_rules, list):
typer.echo("Invalid JSON for --clamp-rules-json: expected a JSON array", err=True)
raise typer.Exit(1)
try:
validate_routing_flags(local, cloud)
with force_routing(local=local, cloud=cloud):
result = run_with_cleanup(
mcp_fcm_simulate(
actions=actions,
scenario=scenario,
clamp_rules=clamp_rules,
project=project,
workspace=workspace,
output_format="json",
)
)
_print_json(result)
except ValueError as e:
typer.echo(f"Error: {e}", err=True)
raise typer.Exit(1)
except Exception as e: # pragma: no cover
if not isinstance(e, typer.Exit):
typer.echo(f"Error during fcm_simulate: {e}", err=True)
raise typer.Exit(1)
raise
@tool_app.command("fcm-rank-actions")
def fcm_rank_actions(
goal: Annotated[str, typer.Argument(help="Goal node identifier")],
constraints_json: Annotated[
Optional[str],
typer.Option("--constraints-json", help="Optional JSON object of ranking constraints"),
] = None,
top_k: int = typer.Option(10, "--top-k", help="Number of recommendations to return"),
project: Annotated[
Optional[str],
typer.Option(help="The project to use. If not provided, the default project will be used."),
] = None,
workspace: Annotated[
Optional[str],
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
] = None,
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Rank intervention actions for an FCM goal."""
constraints = _parse_json_option(constraints_json, "--constraints-json")
if constraints is not None and not isinstance(constraints, dict):
typer.echo("Invalid JSON for --constraints-json: expected a JSON object", err=True)
raise typer.Exit(1)
try:
validate_routing_flags(local, cloud)
with force_routing(local=local, cloud=cloud):
result = run_with_cleanup(
mcp_fcm_rank_actions(
goal=goal,
constraints=constraints,
top_k=top_k,
project=project,
workspace=workspace,
output_format="json",
)
)
_print_json(result)
except ValueError as e:
typer.echo(f"Error: {e}", err=True)
raise typer.Exit(1)
except Exception as e: # pragma: no cover
if not isinstance(e, typer.Exit):
typer.echo(f"Error during fcm_rank_actions: {e}", err=True)
raise typer.Exit(1)
raise
@tool_app.command("fcm-import-model")
def fcm_import_model(
source: Annotated[str, typer.Argument(help="Source path or URI for import payload")],
format: Annotated[
str,
typer.Option("--format", help="Import format (currently csv_bundle_v1)"),
] = "csv_bundle_v1",
merge_mode: Annotated[
str,
typer.Option("--merge-mode", help="Merge strategy: replace or upsert"),
] = "upsert",
project: Annotated[
Optional[str],
typer.Option(help="The project to use. If not provided, the default project will be used."),
] = None,
workspace: Annotated[
Optional[str],
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
] = None,
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Import an FCM model."""
try:
validate_routing_flags(local, cloud)
with force_routing(local=local, cloud=cloud):
result = run_with_cleanup(
mcp_fcm_import_model(
source=source,
format=format, # pyright: ignore[reportArgumentType]
merge_mode=merge_mode, # pyright: ignore[reportArgumentType]
project=project,
workspace=workspace,
output_format="json",
)
)
_print_json(result)
except ValueError as e:
typer.echo(f"Error: {e}", err=True)
raise typer.Exit(1)
except Exception as e: # pragma: no cover
if not isinstance(e, typer.Exit):
typer.echo(f"Error during fcm_import_model: {e}", err=True)
raise typer.Exit(1)
raise
@tool_app.command("fcm-export-model")
def fcm_export_model(
format: Annotated[
str,
typer.Option("--format", help="Export format (currently csv_bundle_v1)"),
] = "csv_bundle_v1",
selection_json: Annotated[
Optional[str],
typer.Option("--selection-json", help="Optional JSON object selection payload"),
] = None,
project: Annotated[
Optional[str],
typer.Option(help="The project to use. If not provided, the default project will be used."),
] = None,
workspace: Annotated[
Optional[str],
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
] = None,
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Export an FCM model."""
selection = _parse_json_option(selection_json, "--selection-json")
if selection is not None and not isinstance(selection, dict):
typer.echo("Invalid JSON for --selection-json: expected a JSON object", err=True)
raise typer.Exit(1)
try:
validate_routing_flags(local, cloud)
with force_routing(local=local, cloud=cloud):
result = run_with_cleanup(
mcp_fcm_export_model(
format=format, # pyright: ignore[reportArgumentType]
selection=selection,
project=project,
workspace=workspace,
output_format="json",
)
)
_print_json(result)
except ValueError as e:
typer.echo(f"Error: {e}", err=True)
raise typer.Exit(1)
except Exception as e: # pragma: no cover
if not isinstance(e, typer.Exit):
typer.echo(f"Error during fcm_export_model: {e}", err=True)
raise typer.Exit(1)
raise
@tool_app.command("search-notes")
def search_notes(
query: Annotated[
@@ -485,7 +869,7 @@ def search_notes(
with force_routing(local=local, cloud=cloud):
result = run_with_cleanup(
mcp_search(
query=query or "",
query=query or None,
project=project,
workspace=workspace,
search_type=search_type,
+7 -1
View File
@@ -24,7 +24,13 @@ def _promos_disabled_by_env() -> bool:
def _is_interactive_session() -> bool:
"""Return whether stdin/stdout are interactive terminals."""
return sys.stdin.isatty() and sys.stdout.isatty()
try:
return sys.stdin.isatty() and sys.stdout.isatty()
except ValueError:
# Trigger: stdin/stdout already closed (e.g., MCP stdio transport shutdown)
# Why: isatty() raises ValueError on closed file descriptors
# Outcome: treat as non-interactive, suppressing promo output
return False
def _build_cloud_promo_message() -> str:
+34 -1
View File
@@ -3,6 +3,7 @@
import importlib.util
import json
import os
import shutil
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
@@ -172,6 +173,25 @@ class BasicMemoryConfig(BaseSettings):
description="Batch size for embedding generation.",
gt=0,
)
semantic_embedding_sync_batch_size: int = Field(
default=64,
description="Batch size for vector sync orchestration flushes.",
gt=0,
)
semantic_embedding_cache_dir: str | None = Field(
default=None,
description="Optional cache directory for FastEmbed model artifacts.",
)
semantic_embedding_threads: int | None = Field(
default=None,
description="Optional FastEmbed runtime thread count override.",
gt=0,
)
semantic_embedding_parallel: int | None = Field(
default=None,
description="Optional FastEmbed embed() parallelism override.",
gt=0,
)
semantic_vector_k: int = Field(
default=100,
description="Vector candidate count for vector and hybrid retrieval.",
@@ -245,6 +265,16 @@ class BasicMemoryConfig(BaseSettings):
description="Disable automatic permalink generation in frontmatter. When enabled, new notes won't have permalinks added and sync won't update permalinks. Existing permalinks will still work for reading.",
)
write_note_overwrite_default: bool = Field(
default=False,
description=(
"Default value for write_note's overwrite parameter. "
"When False (default), write_note errors if note already exists. "
"Set to True to restore pre-v0.20 upsert behavior. "
"Env: BASIC_MEMORY_WRITE_NOTE_OVERWRITE_DEFAULT"
),
)
ensure_frontmatter_on_sync: bool = Field(
default=True,
description="Ensure markdown files have frontmatter during sync by adding derived title/type/permalink when missing. When combined with disable_permalinks=True, this setting takes precedence for missing-frontmatter files and still writes permalinks.",
@@ -695,7 +725,10 @@ class ConfigManager:
# Re-save to normalize legacy config into current format
if needs_resave:
logger.info("Migrating config to current format")
# Create backup before overwriting so users can revert if needed
backup_path = self.config_file.with_suffix(".json.bak")
shutil.copy2(self.config_file, backup_path)
logger.info(f"Migrating config to current format (backup: {backup_path})")
save_basic_memory_config(self.config_file, _CONFIG_CACHE)
return _CONFIG_CACHE
+9 -4
View File
@@ -128,8 +128,13 @@ async def _run_semantic_embedding_backfill(
project_id=project_id,
app_config=app_config,
)
for entity_id in entity_ids:
await search_repository.sync_entity_vectors(entity_id)
batch_result = await search_repository.sync_entity_vectors_batch(entity_ids)
if batch_result.entities_failed > 0:
logger.warning(
"Automatic semantic embedding backfill encountered entity failures: "
f"project={project_name}, failed={batch_result.entities_failed}, "
f"failed_entity_ids={batch_result.failed_entity_ids}"
)
logger.info(
"Automatic semantic embedding backfill complete: "
@@ -475,7 +480,7 @@ async def run_migrations(
Note: Alembic tracks which migrations have been applied via the alembic_version table,
so it's safe to call this multiple times - it will only run pending migrations.
"""
logger.info("Running database migrations...")
logger.debug("Running database migrations...")
temp_engine: AsyncEngine | None = None
try:
revisions_before_upgrade: set[str] = set()
@@ -514,7 +519,7 @@ async def run_migrations(
config.set_main_option("sqlalchemy.url", db_url)
command.upgrade(config, "head")
logger.info("Migrations completed successfully")
logger.debug("Migrations completed successfully")
# Get session maker - ensure we don't trigger recursive migration calls
if _session_maker is None:
+8
View File
@@ -131,6 +131,10 @@ from basic_memory.deps.services import (
DirectoryServiceV2Dep,
get_directory_service_v2_external,
DirectoryServiceV2ExternalDep,
get_graph_intelligence_service_v2_external,
GraphIntelligenceServiceV2ExternalDep,
get_fcm_service_v2_external,
FCMServiceV2ExternalDep,
)
from basic_memory.deps.importers import (
@@ -269,6 +273,10 @@ __all__ = [
"DirectoryServiceV2Dep",
"get_directory_service_v2_external",
"DirectoryServiceV2ExternalDep",
"get_graph_intelligence_service_v2_external",
"GraphIntelligenceServiceV2ExternalDep",
"get_fcm_service_v2_external",
"FCMServiceV2ExternalDep",
# Importers
"get_chatgpt_importer",
"ChatGPTImporterDep",
+44
View File
@@ -39,6 +39,8 @@ from basic_memory.deps.repositories import (
from basic_memory.markdown import EntityParser
from basic_memory.markdown.markdown_processor import MarkdownProcessor
from basic_memory.services import EntityService, ProjectService
from basic_memory.services.fcm_service import FCMService
from basic_memory.services.graph_intelligence_service import GraphIntelligenceService
from basic_memory.services.context_service import ContextService
from basic_memory.services.directory_service import DirectoryService
from basic_memory.services.file_service import FileService
@@ -358,6 +360,30 @@ async def get_context_service_v2_external(
ContextServiceV2ExternalDep = Annotated[ContextService, Depends(get_context_service_v2_external)]
# --- Graph Intelligence Service ---
async def get_graph_intelligence_service_v2_external() -> GraphIntelligenceService:
"""Create GraphIntelligenceService for v2 API (uses external_id routing)."""
return GraphIntelligenceService()
GraphIntelligenceServiceV2ExternalDep = Annotated[
GraphIntelligenceService, Depends(get_graph_intelligence_service_v2_external)
]
# --- FCM Service ---
async def get_fcm_service_v2_external() -> FCMService:
"""Create FCMService for v2 API (uses external_id routing)."""
return FCMService()
FCMServiceV2ExternalDep = Annotated[FCMService, Depends(get_fcm_service_v2_external)]
# --- Sync Service ---
@@ -535,6 +561,21 @@ async def get_task_scheduler(
async def _reindex_project(**_: Any) -> None:
await search_service.reindex_all()
async def _sync_graph_entity(entity_id: int, **extra_payload: Any) -> None:
# Trigger: graph-entity sync task is scheduled from graph lifecycle hooks.
# Why: keep scheduler contract stable while graph index provider work lands in later phases.
# Outcome: no-op in phase 1; task name remains valid for API and tool contracts.
del entity_id, extra_payload
async def _sync_graph_project(force_full: bool = False, **_: Any) -> None:
await _sync_project(force_full=force_full)
async def _reindex_graph_project(**_: Any) -> None:
# Trigger: graph reindex requested.
# Why: phase 1 has no dedicated graph index worker yet.
# Outcome: run project sync path so writes stay coherent while graph provider ships.
await _sync_project(force_full=True)
scheduler = LocalTaskScheduler(
{
"reindex_entity": _reindex_entity,
@@ -542,6 +583,9 @@ async def get_task_scheduler(
"sync_entity_vectors": _sync_entity_vectors,
"sync_project": _sync_project,
"reindex_project": _reindex_project,
"sync_graph_entity": _sync_graph_entity,
"sync_graph_project": _sync_graph_project,
"reindex_graph_project": _reindex_graph_project,
},
test_mode=app_config.is_test_env,
)
+32 -3
View File
@@ -88,6 +88,22 @@ def normalize_frontmatter_value(value: Any) -> Any:
return value
def _coerce_to_string(value: Any) -> str:
"""Coerce a frontmatter value to a string.
YAML can parse scalar-looking fields as lists when the author uses block
sequence syntax. For fields like ``title`` and ``type`` that *must* be
strings, this helper converts lists to a comma-separated string and any
other non-string type via ``str()``.
"""
if isinstance(value, str):
return value
if isinstance(value, list):
# Join list items, converting each to string first
return ", ".join(str(item) for item in value)
return str(value)
def normalize_frontmatter_metadata(metadata: dict) -> dict:
"""Normalize all values in frontmatter metadata dict.
@@ -233,9 +249,15 @@ class EntityParser:
content = strip_bom(content)
# Parse frontmatter with proper error handling for malformed YAML
# Parse frontmatter with proper error handling for malformed YAML.
# We use frontmatter.parse() instead of frontmatter.loads() because
# loads() does Post(content, handler, **metadata), which crashes when
# the YAML contains reserved keys like 'content' or 'handler'.
# See basic-memory-cloud#375.
try:
post = frontmatter.loads(content)
fm_metadata, fm_content = frontmatter.parse(content)
post = frontmatter.Post(fm_content)
post.metadata.update(fm_metadata)
except yaml.YAMLError as e:
logger.warning(
f"Failed to parse YAML frontmatter in {file_path}: {e}. "
@@ -248,14 +270,21 @@ class EntityParser:
# Normalize frontmatter values
metadata = normalize_frontmatter_metadata(post.metadata)
# Ensure required fields have defaults
# Ensure required string fields are always strings.
# YAML can parse these as lists when authors use block sequence syntax
# (e.g. "title:\n - My Title"), causing 'list' has no attribute 'strip'
# downstream. See basic-memory-cloud#376.
title = metadata.get("title")
if title is not None:
title = _coerce_to_string(title)
if not title or title == "None":
metadata["title"] = file_path.stem
else:
metadata["title"] = title
note_type = metadata.get("type")
if note_type is not None:
note_type = _coerce_to_string(note_type)
metadata["type"] = note_type if note_type is not None else "note"
tags = parse_tags(metadata.get("tags", [])) # pyright: ignore
+5 -5
View File
@@ -154,13 +154,13 @@ async def get_client(
# Outcome: route strictly based on explicit flag.
if _explicit_routing():
if _force_local_mode():
logger.info("Explicit local routing enabled - using ASGI client")
logger.debug("Explicit local routing enabled - using ASGI client")
async with _asgi_client(timeout) as client:
yield client
return
if _force_cloud_mode():
logger.info("Explicit cloud routing enabled - using cloud proxy client")
logger.debug("Explicit cloud routing enabled - using cloud proxy client")
async with _cloud_client(config, timeout, workspace=workspace) as client:
yield client
return
@@ -172,7 +172,7 @@ async def get_client(
if project_name is not None and not _explicit_routing():
project_mode = config.get_project_mode(project_name)
if project_mode == ProjectMode.CLOUD:
logger.info(f"Project '{project_name}' is cloud mode - using cloud proxy client")
logger.debug(f"Project '{project_name}' is cloud mode - using cloud proxy client")
try:
async with _cloud_client(config, timeout, workspace=workspace) as client:
yield client
@@ -183,13 +183,13 @@ async def get_client(
) from exc
return
logger.info(f"Project '{project_name}' is local mode - using ASGI client")
logger.debug(f"Project '{project_name}' is local mode - using ASGI client")
async with _asgi_client(timeout) as client:
yield client
return
# --- Default fallback ---
logger.info("Default routing - using ASGI client for local Basic Memory API")
logger.debug("Default routing - using ASGI client for local Basic Memory API")
async with _asgi_client(timeout) as client:
yield client
+4
View File
@@ -18,6 +18,8 @@ from basic_memory.mcp.clients.directory import DirectoryClient
from basic_memory.mcp.clients.resource import ResourceClient
from basic_memory.mcp.clients.project import ProjectClient
from basic_memory.mcp.clients.schema import SchemaClient
from basic_memory.mcp.clients.graph import GraphClient
from basic_memory.mcp.clients.fcm import FCMClient
__all__ = [
"KnowledgeClient",
@@ -27,4 +29,6 @@ __all__ = [
"ResourceClient",
"ProjectClient",
"SchemaClient",
"GraphClient",
"FCMClient",
]
+56
View File
@@ -0,0 +1,56 @@
"""Typed client for FCM API operations."""
from httpx import AsyncClient
from basic_memory.mcp.tools.utils import call_post
from basic_memory.schemas.graph_intelligence import (
FCMExportRequest,
FCMExportResponse,
FCMImportRequest,
FCMImportResponse,
FCMRankActionsRequest,
FCMRankActionsResponse,
FCMSimulateRequest,
FCMSimulateResponse,
)
class FCMClient:
"""Typed client for FCM operations."""
def __init__(self, http_client: AsyncClient, project_id: str):
self.http_client = http_client
self.project_id = project_id
self._base_path = f"/v2/projects/{project_id}/fcm"
async def simulate(self, request: FCMSimulateRequest) -> FCMSimulateResponse:
response = await call_post(
self.http_client,
f"{self._base_path}/simulate",
json=request.model_dump(mode="json"),
)
return FCMSimulateResponse.model_validate(response.json())
async def rank_actions(self, request: FCMRankActionsRequest) -> FCMRankActionsResponse:
response = await call_post(
self.http_client,
f"{self._base_path}/rank-actions",
json=request.model_dump(mode="json"),
)
return FCMRankActionsResponse.model_validate(response.json())
async def import_model(self, request: FCMImportRequest) -> FCMImportResponse:
response = await call_post(
self.http_client,
f"{self._base_path}/import",
json=request.model_dump(mode="json"),
)
return FCMImportResponse.model_validate(response.json())
async def export_model(self, request: FCMExportRequest) -> FCMExportResponse:
response = await call_post(
self.http_client,
f"{self._base_path}/export",
json=request.model_dump(mode="json"),
)
return FCMExportResponse.model_validate(response.json())
+62
View File
@@ -0,0 +1,62 @@
"""Typed client for graph intelligence API operations."""
from httpx import AsyncClient
from basic_memory.mcp.tools.utils import call_get, call_post
from basic_memory.schemas.graph_intelligence import (
GraphHealthResponse,
GraphImpactRequest,
GraphImpactResponse,
GraphLineageRequest,
GraphLineageResponse,
GraphReindexRequest,
GraphReindexResponse,
)
class GraphClient:
"""Typed client for graph intelligence operations."""
def __init__(self, http_client: AsyncClient, project_id: str):
self.http_client = http_client
self.project_id = project_id
self._base_path = f"/v2/projects/{project_id}/graph"
async def lineage(self, request: GraphLineageRequest) -> GraphLineageResponse:
response = await call_post(
self.http_client,
f"{self._base_path}/lineage",
json=request.model_dump(mode="json"),
)
return GraphLineageResponse.model_validate(response.json())
async def impact(self, request: GraphImpactRequest) -> GraphImpactResponse:
response = await call_post(
self.http_client,
f"{self._base_path}/impact",
json=request.model_dump(mode="json"),
)
return GraphImpactResponse.model_validate(response.json())
async def health(
self, scope: str | None = None, timeframe: str | None = None
) -> GraphHealthResponse:
params: dict[str, str] = {}
if scope is not None:
params["scope"] = scope
if timeframe is not None:
params["timeframe"] = timeframe
response = await call_get(
self.http_client,
f"{self._base_path}/health",
params=params,
)
return GraphHealthResponse.model_validate(response.json())
async def reindex(self, request: GraphReindexRequest) -> GraphReindexResponse:
response = await call_post(
self.http_client,
f"{self._base_path}/reindex",
json=request.model_dump(mode="json"),
)
return GraphReindexResponse.model_validate(response.json())
+68 -2
View File
@@ -9,7 +9,7 @@ compatibility with existing MCP tools.
"""
from contextlib import asynccontextmanager
from typing import AsyncIterator, Optional, List, Tuple
from typing import AsyncIterator, Awaitable, Callable, Optional, List, Tuple
from httpx import AsyncClient
from httpx._types import (
@@ -27,6 +27,41 @@ from basic_memory.schemas.v2 import ProjectResolveResponse
from basic_memory.schemas.memory import memory_url_path
from basic_memory.utils import generate_permalink, normalize_project_reference
# --- Workspace provider injection ---
# Mirrors the set_client_factory() pattern in async_client.py.
# The cloud MCP server sets a provider that queries its own database directly,
# avoiding the control-plane HTTP round-trip that requires local credentials.
_workspace_provider: Optional[Callable[[], Awaitable[list[WorkspaceInfo]]]] = None
def set_workspace_provider(provider: Callable[[], Awaitable[list[WorkspaceInfo]]]) -> None:
"""Override workspace discovery (for cloud app, testing, etc)."""
global _workspace_provider
_workspace_provider = provider
async def _resolve_default_project_from_api() -> Optional[str]:
"""Query the projects API for the default project.
Used as a fallback when ConfigManager has no local config (cloud mode).
"""
from basic_memory.mcp.async_client import get_client
try:
async with get_client() as client:
response = await client.get("/v2/projects/")
if response.status_code == 200:
project_list = ProjectList.model_validate(response.json())
if project_list.default_project:
return project_list.default_project
# Fallback: find project with is_default=True
for p in project_list.projects:
if p.is_default:
return p.name
except Exception:
pass
return None
async def resolve_project_parameter(
project: Optional[str] = None,
@@ -54,11 +89,16 @@ async def resolve_project_parameter(
Returns:
Resolved project name or None if no resolution possible
"""
# Load config for any values not explicitly provided
# Load config for any values not explicitly provided.
# ConfigManager reads from the local config file, which doesn't exist in cloud mode.
# When it returns None, fall back to querying the projects API for the is_default flag.
if default_project is None:
config = ConfigManager().config
default_project = config.default_project
if default_project is None:
default_project = await _resolve_default_project_from_api()
# Create resolver with configuration and resolve
resolver = ProjectResolver.from_env(
default_project=default_project,
@@ -103,6 +143,19 @@ async def get_available_workspaces(context: Optional[Context] = None) -> list[Wo
if isinstance(cached_raw, list):
return [WorkspaceInfo.model_validate(item) for item in cached_raw]
# Trigger: workspace provider was injected (e.g., by cloud MCP server)
# Why: the cloud server IS the cloud — it can query its own database
# directly instead of making an HTTP round-trip that requires local credentials
# Outcome: use provider result, cache in context, skip control-plane client
if _workspace_provider is not None:
workspaces = await _workspace_provider()
if context:
await context.set_state(
"available_workspaces",
[ws.model_dump() for ws in workspaces],
)
return workspaces
from basic_memory.mcp.async_client import get_cloud_control_plane_client
from basic_memory.mcp.tools.utils import call_get
@@ -419,6 +472,7 @@ async def get_project_client(
_explicit_routing,
_force_local_mode,
get_client,
is_factory_mode,
)
# Step 1: Resolve project name from config (no network call)
@@ -433,6 +487,18 @@ async def get_project_client(
f"Available projects: {project_names}"
)
# Step 1b: Factory injection (in-process cloud server)
# Trigger: set_client_factory() was called (e.g., by cloud MCP server)
# Why: the transport layer already resolved workspace and tenant context;
# attempting cloud workspace resolution here would call the production
# control-plane API with no valid credentials and fail with 401
# Outcome: use the factory client directly, skip workspace resolution
if is_factory_mode():
async with get_client() as client:
active_project = await get_active_project(client, resolved_project, context)
yield client, active_project
return
# Step 2: Check explicit routing BEFORE workspace resolution
# Trigger: CLI passed --local or --cloud
# Why: explicit flags must be deterministic — skip workspace entirely for --local
@@ -13,7 +13,6 @@ from pydantic import Field
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.recent_activity import recent_activity
from basic_memory.mcp.tools.search import search_notes
from basic_memory.schemas.search import SearchResponse
@mcp.prompt(
@@ -42,15 +41,12 @@ async def continue_conversation(
logger.info(f"Continuing session, topic: {topic}, timeframe: {timeframe}")
if topic:
# Search for the topic using the search tool directly
result = await search_notes(query=topic, after_date=timeframe)
# Use json format to get structured data for result counting and branching
result = await search_notes(query=topic, after_date=timeframe, output_format="json")
if isinstance(result, SearchResponse):
context_text = _format_continuation_results(result, topic)
result_count = len(result.results)
elif isinstance(result, dict):
if isinstance(result, dict):
results = result.get("results", [])
context_text = str(result)
context_text = _format_continuation_results(results, topic)
result_count = len(results)
else:
# Error string
@@ -111,23 +107,24 @@ async def continue_conversation(
return prompt
def _format_continuation_results(result: SearchResponse, topic: str) -> str:
"""Format search results for conversation continuation context."""
if not result.results:
def _format_continuation_results(results: list[dict], topic: str) -> str:
"""Format search result dicts for conversation continuation context."""
if not results:
return f"No previous context found for '{topic}'."
lines = [f"## Previous Context for '{topic}'\n"]
for item in result.results:
title = item.title or "Untitled"
permalink = item.permalink or ""
for item in results:
title = item.get("title", "Untitled")
permalink = item.get("permalink", "")
lines.append(f"### {title}")
if permalink:
lines.append(f"permalink: {permalink}")
lines.append(f"Read with: `read_note(\"{permalink}\")`")
if item.content:
content = item.content[:300] + "..." if len(item.content) > 300 else item.content
lines.append(f'Read with: `read_note("{permalink}")`')
content = item.get("content")
if content:
content = content[:300] + "..." if len(content) > 300 else content
lines.append(f"\n{content}")
lines.append("")
+17 -23
View File
@@ -11,7 +11,6 @@ from pydantic import Field
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.search import search_notes
from basic_memory.schemas.search import SearchResponse
@mcp.prompt(
@@ -39,18 +38,14 @@ async def search_prompt(
"""
logger.info(f"Searching knowledge base, query: {query}, timeframe: {timeframe}")
# Call the search tool directly — it returns SearchResponse, dict, or error string
result = await search_notes(query=query, after_date=timeframe)
# Use json format to get structured data for result counting and formatting
result = await search_notes(query=query, after_date=timeframe, output_format="json")
# Format the tool output into a prompt with guidance
if isinstance(result, SearchResponse):
result_count = len(result.results)
result_text = _format_search_results(result, query)
elif isinstance(result, dict):
# json output format
if isinstance(result, dict):
results = result.get("results", [])
result_count = len(results)
result_text = str(result)
result_text = _format_search_results(results, query)
else:
# Error string from search tool
result_count = 0
@@ -76,28 +71,27 @@ async def search_prompt(
""")
def _format_search_results(result: SearchResponse, query: str) -> str:
"""Format SearchResponse into readable markdown."""
if not result.results:
def _format_search_results(results: list[dict], query: str) -> str:
"""Format search result dicts into readable markdown."""
if not results:
return f"No results found for '{query}'."
lines = [f"Found {len(result.results)} results:\n"]
lines = [f"Found {len(results)} results:\n"]
for item in result.results:
title = item.title or "Untitled"
permalink = item.permalink or ""
score = f" (score: {item.score:.2f})" if item.score else ""
for item in results:
title = item.get("title", "Untitled")
permalink = item.get("permalink", "")
score = item.get("score")
score_text = f" (score: {score:.2f})" if score else ""
lines.append(f"- **{title}**{score}")
lines.append(f"- **{title}**{score_text}")
if permalink:
lines.append(f" permalink: {permalink}")
if item.content:
content = item.get("content")
if content:
# Truncate content snippet
content = item.content[:200] + "..." if len(item.content) > 200 else item.content
content = content[:200] + "..." if len(content) > 200 else content
lines.append(f" {content}")
lines.append("")
if result.has_more:
lines.append("*More results available. Use page=2 to see next page.*")
return "\n".join(lines)
@@ -57,6 +57,16 @@ await write_note(
)
```
> **Important**: `write_note` errors if the note already exists. Use `edit_note` for incremental changes, or pass `overwrite=True` to replace.
```python
# Preferred: update an existing note incrementally
await edit_note(identifier="Topic", operation="append", content="\n- [category] new fact")
# Alternative: replace the entire note
await write_note(title="Topic", content="...", folder="notes", overwrite=True)
```
### Reading Knowledge
```python
@@ -70,11 +80,27 @@ content = await read_note("memory://folder/topic", project="main")
### Searching
```python
# Basic text search
results = await search_notes(query="authentication", project="main")
# Search types: "text" (default), "title", "permalink", "vector"/"semantic", "hybrid"
# Default is "hybrid" when semantic search is enabled, "text" otherwise
results = await search_notes(query="auth flow", search_type="hybrid")
# Tag shorthand in query (multiple tags: "tag:x AND tag:y" or "tag:x tag:y")
results = await search_notes(query="tag:security")
results = await search_notes(query="tag:coffee AND tag:brewing")
# Filter-only search (no query needed)
results = await search_notes(tags=["security", "auth"], status="active")
# Metadata filters with operators: $in, $gt, $gte, $lt, $lte, $between
results = await search_notes(
query="authentication",
project="main",
page_size=10
metadata_filters={"priority": {"$in": ["high", "critical"]}}
)
# Override similarity threshold for vector/hybrid search
results = await search_notes(query="auth", search_type="hybrid", min_similarity=0.5)
```
### Building Context
@@ -162,6 +188,8 @@ activity = await recent_activity(project="main")
- 2-3 relations per note
- Meaningful categories and relation types
**Prefer `edit_note` for updates** — use `write_note` only for new notes.
**Search before creating:**
```python
# Find existing entities to reference
@@ -201,6 +229,14 @@ except:
results = await search_notes(query="test", project=projects[0].name)
```
**Note already exists:**
```python
# write_note returns an error if the note exists — use edit_note or overwrite
await edit_note(identifier="Existing Topic", operation="append", content="\n- [update] new info")
# Or replace entirely:
await write_note(title="Existing Topic", content="...", folder="notes", overwrite=True)
```
**Forward references:**
```python
# Check response for unresolved relations
@@ -256,13 +292,14 @@ context = await build_context(url=f"memory://{results[0].permalink}", project="m
| Tool | Purpose | Key Params |
|------|---------|------------|
| `write_note` | Create/update | title, content, folder, project |
| `write_note` | Create new | title, content, folder, project, overwrite |
| `read_note` | Read content | identifier, project |
| `edit_note` | Modify existing | identifier, operation, content, project |
| `search_notes` | Find notes | query, project |
| `search_notes` | Find notes | query, search_type, tags, metadata_filters, project |
| `build_context` | Graph traversal | url, depth, project |
| `recent_activity` | Recent changes | timeframe, project |
| `list_memory_projects` | Show projects | (none) |
| `list_workspaces` | Show workspaces | (none) |
## memory:// URL Format
@@ -270,6 +307,7 @@ context = await build_context(url=f"memory://{results[0].permalink}", project="m
- `memory://folder/title` - By folder + title
- `memory://permalink` - By permalink
- `memory://folder/*` - All in folder
- `memory://project-name/folder/title` - Cross-project (auto-routes to the correct project)
For full documentation: https://docs.basicmemory.com
+19 -2
View File
@@ -18,12 +18,22 @@ from basic_memory.mcp.tools.view_note import view_note
from basic_memory.mcp.tools.write_note import write_note
from basic_memory.mcp.tools.cloud_info import cloud_info
from basic_memory.mcp.tools.release_notes import release_notes
from basic_memory.mcp.tools.search import search_notes, search_by_metadata
from basic_memory.mcp.tools.search import search_notes
from basic_memory.mcp.tools.canvas import canvas
from basic_memory.mcp.tools.list_directory import list_directory
from basic_memory.mcp.tools.edit_note import edit_note
from basic_memory.mcp.tools.move_note import move_note
from basic_memory.mcp.tools.workspaces import list_workspaces
from basic_memory.mcp.tools.graph_intelligence import (
graph_lineage,
graph_impact,
graph_health,
graph_reindex,
fcm_simulate,
fcm_rank_actions,
fcm_import_model,
fcm_export_model,
)
from basic_memory.mcp.tools.project_management import (
list_memory_projects,
create_memory_project,
@@ -44,7 +54,15 @@ __all__ = [
"delete_note",
"delete_project",
"edit_note",
"fcm_export_model",
"fcm_import_model",
"fcm_rank_actions",
"fcm_simulate",
"fetch",
"graph_health",
"graph_impact",
"graph_lineage",
"graph_reindex",
"list_directory",
"list_memory_projects",
"list_workspaces",
@@ -58,7 +76,6 @@ __all__ = [
"schema_infer",
"schema_validate",
"search",
"search_by_metadata",
"search_notes",
# "search_notes_ui",
"view_note",
+4 -72
View File
@@ -22,74 +22,6 @@ from basic_memory.schemas.memory import (
RelationSummary,
)
# --- Fields to strip from each model (redundant with parent entity) ---
_OBSERVATION_STRIP = {
"observation_id",
"entity_id",
"entity_external_id",
"title",
"file_path",
"created_at",
}
_RELATION_STRIP = {
"relation_id",
"entity_id",
"from_entity_id",
"from_entity_external_id",
"to_entity_id",
"to_entity_external_id",
"title",
"file_path",
"created_at",
}
_ENTITY_STRIP = {"entity_id", "created_at"}
_METADATA_STRIP = {"total_results", "generated_at"}
def _slim_summary(summary: EntitySummary | RelationSummary | ObservationSummary) -> dict:
"""Strip redundant fields from a summary model based on its type."""
if isinstance(summary, ObservationSummary):
strip = _OBSERVATION_STRIP
elif isinstance(summary, RelationSummary):
strip = _RELATION_STRIP
else:
strip = _ENTITY_STRIP
data = summary.model_dump()
for key in strip:
data.pop(key, None)
return data
def _slim_context(graph: GraphContext) -> dict:
"""Transform GraphContext into a slimmed dict, stripping redundant fields.
Reduces payload size ~40% by removing fields on nested objects that
duplicate information already present on the parent entity (IDs,
timestamps, file paths).
"""
slimmed_results = []
for result in graph.results:
slimmed_results.append(
{
"primary_result": _slim_summary(result.primary_result),
"observations": [_slim_summary(obs) for obs in result.observations],
"related_results": [_slim_summary(rel) for rel in result.related_results],
}
)
metadata = graph.metadata.model_dump()
for key in _METADATA_STRIP:
metadata.pop(key, None)
return {
"results": slimmed_results,
"metadata": metadata,
"page": graph.page,
"page_size": graph.page_size,
}
def _format_entity_block(result: ContextResult) -> str:
"""Format a single context result as a markdown block."""
@@ -194,7 +126,7 @@ def _format_context_markdown(graph: GraphContext, project: str) -> str:
- Or standard formats like "7d", "24h"
Format options:
- "json" (default): Slimmed JSON with redundant fields removed
- "json" (default): Structured JSON with internal fields excluded
- "text": Compact markdown text for LLM consumption
""",
annotations={"readOnlyHint": True, "openWorldHint": False},
@@ -231,12 +163,12 @@ async def build_context(
page: Page number of results to return (default: 1)
page_size: Number of results to return per page (default: 10)
max_related: Maximum number of related results to return (default: 10)
output_format: Response format - "json" for slimmed JSON dict,
output_format: Response format - "json" for structured JSON dict,
"text" for compact markdown text
context: Optional FastMCP context for performance caching.
Returns:
dict (output_format="json"): Slimmed JSON with redundant fields removed
dict (output_format="json"): Structured JSON with internal fields excluded
str (output_format="text"): Compact markdown representation
Examples:
@@ -292,4 +224,4 @@ async def build_context(
if output_format == "text":
return _format_context_markdown(graph, active_project.name)
return _slim_context(graph)
return graph.model_dump()
+9 -17
View File
@@ -7,13 +7,13 @@ a list containing a single `{"type": "text", "text": "{...json...}"}` item.
import json
from typing import Any, Dict, List, Optional
from loguru import logger
from fastmcp import Context
from loguru import logger
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.search import search_notes
from basic_memory.mcp.tools.read_note import read_note
from basic_memory.config import ConfigManager
from basic_memory.mcp.tools.search import search_notes
from basic_memory.schemas.search import SearchResponse, SearchResult
@@ -113,16 +113,12 @@ async def search(
logger.info(f"ChatGPT search request: query='{query}'")
try:
# ChatGPT tools don't expose project parameter, so use default project
config = ConfigManager().config
default_project = config.default_project
# Call underlying search_notes with sensible defaults for ChatGPT
# Let search_notes resolve the default project via get_project_client(),
# which works in both local mode (ConfigManager) and cloud mode (database).
results = await search_notes(
query=query,
project=default_project, # Use default project for ChatGPT
page=1,
page_size=10, # Reasonable default for ChatGPT consumption
page_size=10,
output_format="json",
context=context,
)
@@ -180,17 +176,13 @@ async def fetch(
logger.info(f"ChatGPT fetch request: id='{id}'")
try:
# ChatGPT tools don't expose project parameter, so use default project
config = ConfigManager().config
default_project = config.default_project
# Call underlying read_note function (default output_format="text" returns str)
# Let read_note resolve the default project via get_project_client(),
# which works in both local mode (ConfigManager) and cloud mode (database).
content = str(
await read_note(
identifier=id,
project=default_project, # Use default project for ChatGPT
page=1,
page_size=10, # Default pagination
page_size=10,
context=context,
)
)
+161 -46
View File
@@ -7,6 +7,36 @@ from fastmcp import Context
from basic_memory.mcp.project_context import get_project_client, add_project_metadata
from basic_memory.mcp.server import mcp
from basic_memory.schemas.base import Entity
from basic_memory.schemas.response import EntityResponse
from basic_memory.utils import validate_project_path
def _parse_identifier_to_title_and_directory(identifier: str) -> tuple[str, str]:
"""Parse an identifier into (title, directory) for creating a new note.
Strips memory:// prefix if present, then splits on the last '/' to
separate the directory path from the note title.
Examples:
"conversations/my-note" ("my-note", "conversations")
"my-note" ("my-note", "")
"a/b/c/my-note" ("my-note", "a/b/c")
"memory://a/b/note" ("note", "a/b")
"""
cleaned = identifier
if cleaned.startswith("memory://"):
cleaned = cleaned[len("memory://") :]
if "/" in cleaned:
last_slash = cleaned.rfind("/")
directory = cleaned[:last_slash]
title = cleaned[last_slash + 1 :]
else:
directory = ""
title = cleaned
return title, directory
def _format_error_response(
@@ -19,15 +49,19 @@ def _format_error_response(
) -> str:
"""Format helpful error responses for edit_note failures that guide the AI to retry successfully."""
# Entity not found errors
# Entity not found errors — only reachable for find_replace/replace_section
# because append/prepend auto-create the note when it doesn't exist
if "Entity not found" in error_message or "entity not found" in error_message.lower():
return f"""# Edit Failed - Note Not Found
The note with identifier '{identifier}' could not be found. Edit operations require an exact match (no fuzzy matching).
The note with identifier '{identifier}' could not be found. The `find_replace` and `replace_section` operations require an existing note with content to modify.
**Tip:** `append` and `prepend` operations automatically create the note if it doesn't exist.
## Suggestions to try:
1. **Search for the note first**: Use `search_notes("{project or "project-name"}", "{identifier.split("/")[-1]}")` to find similar notes with exact identifiers
2. **Try different exact identifier formats**:
1. **Use append/prepend instead**: These operations will create the note automatically if it doesn't exist
2. **Search for the note first**: Use `search_notes("{project or "project-name"}", "{identifier.split("/")[-1]}")` to find similar notes with exact identifiers
3. **Try different exact identifier formats**:
- If you used a permalink like "folder/note-title", try the exact title: "{identifier.split("/")[-1].replace("-", " ").title()}"
- If you used a title, try the exact permalink format: "{identifier.lower().replace(" ", "-")}"
- Use `read_note("{project or "project-name"}", "{identifier}")` first to verify the note exists and get the exact identifier
@@ -135,7 +169,7 @@ async def edit_note(
workspace: Optional[str] = None,
section: Optional[str] = None,
find_text: Optional[str] = None,
expected_replacements: int = 1,
expected_replacements: Optional[int] = None,
output_format: Literal["text", "json"] = "text",
context: Context | None = None,
) -> str | dict:
@@ -152,10 +186,10 @@ async def edit_note(
Must be an exact match - fuzzy matching is not supported for edit operations.
Use search_notes() or read_note() first to find the correct identifier if uncertain.
operation: The editing operation to perform:
- "append": Add content to the end of the note
- "prepend": Add content to the beginning of the note
- "find_replace": Replace occurrences of find_text with content
- "replace_section": Replace content under a specific markdown header
- "append": Add content to the end of the note (creates the note if it doesn't exist)
- "prepend": Add content to the beginning of the note (creates the note if it doesn't exist)
- "find_replace": Replace occurrences of find_text with content (note must exist)
- "replace_section": Replace content under a specific markdown header (note must exist)
content: The content to add or use for replacement
project: Project name to edit in. Optional - server will resolve using hierarchy.
If unknown, use list_memory_projects() to discover available projects.
@@ -216,6 +250,9 @@ async def edit_note(
search_notes() first to find the correct identifier. The tool provides detailed
error messages with suggestions if operations fail.
"""
# Resolve effective default: allow MCP clients to send null for optional int field
effective_replacements = expected_replacements if expected_replacements is not None else 1
async with get_project_client(project, workspace, context) as (client, active_project):
logger.info("MCP tool call", tool="edit_note", identifier=identifier, operation=operation)
@@ -240,48 +277,118 @@ async def edit_note(
# Use typed KnowledgeClient for API calls
knowledge_client = KnowledgeClient(client, active_project.external_id)
# Resolve identifier to entity ID
entity_id = await knowledge_client.resolve_entity(identifier)
file_created = False
entity_id = ""
result: EntityResponse | None = None
# Prepare the edit request data
edit_data = {
"operation": operation,
"content": content,
}
# Try to resolve the entity; for append/prepend, create it if not found
try:
entity_id = await knowledge_client.resolve_entity(identifier)
except Exception as resolve_error:
# Trigger: entity does not exist yet
# Why: append/prepend can meaningfully create a new note from the content,
# while find_replace/replace_section require existing content to modify
# Outcome: note is created via the same path as write_note
error_msg = str(resolve_error).lower()
is_not_found = "entity not found" in error_msg or "not found" in error_msg
# Add optional parameters
if section:
edit_data["section"] = section
if find_text:
edit_data["find_text"] = find_text
if expected_replacements != 1: # Only send if different from default
edit_data["expected_replacements"] = str(expected_replacements)
if is_not_found and operation in ("append", "prepend"):
title, directory = _parse_identifier_to_title_and_directory(identifier)
# Call the PATCH endpoint
result = await knowledge_client.patch_entity(entity_id, edit_data, fast=False)
# Validate directory path (same security check as write_note)
project_path = active_project.home
if directory and not validate_project_path(directory, project_path):
logger.warning(
"Attempted path traversal attack blocked",
directory=directory,
project=active_project.name,
)
if output_format == "json":
return {
"title": title,
"permalink": None,
"file_path": None,
"checksum": None,
"operation": operation,
"fileCreated": False,
"error": "SECURITY_VALIDATION_ERROR",
}
return f"# Error\n\nDirectory path '{directory}' is not allowed - paths must stay within project boundaries"
# Format summary
summary = [
f"# Edited note ({operation})",
f"project: {active_project.name}",
f"file_path: {result.file_path}",
f"permalink: {result.permalink}",
f"checksum: {result.checksum[:8] if result.checksum else 'unknown'}",
]
entity = Entity(
title=title,
directory=directory,
content_type="text/markdown",
content=content,
)
# Add operation-specific details
if operation == "append":
logger.info(
"Creating note via edit_note auto-create",
title=title,
directory=directory,
operation=operation,
)
result = await knowledge_client.create_entity(entity.model_dump(), fast=False)
file_created = True
else:
# find_replace/replace_section require existing content — re-raise
raise resolve_error
# --- Standard edit path (entity already existed) ---
if not file_created:
# Prepare the edit request data
edit_data = {
"operation": operation,
"content": content,
}
# Add optional parameters
if section:
edit_data["section"] = section
if find_text:
edit_data["find_text"] = find_text
if effective_replacements != 1: # Only send if different from default
edit_data["expected_replacements"] = str(effective_replacements)
# Call the PATCH endpoint
result = await knowledge_client.patch_entity(entity_id, edit_data, fast=False)
# --- Format response ---
# result is always set: either by create_entity (auto-create) or patch_entity (edit)
assert result is not None
if file_created:
summary = [
f"# Created note ({operation})",
f"project: {active_project.name}",
f"file_path: {result.file_path}",
f"permalink: {result.permalink}",
f"checksum: {result.checksum[:8] if result.checksum else 'unknown'}",
"fileCreated: true",
]
lines_added = len(content.split("\n"))
summary.append(f"operation: Added {lines_added} lines to end of note")
elif operation == "prepend":
lines_added = len(content.split("\n"))
summary.append(f"operation: Added {lines_added} lines to beginning of note")
elif operation == "find_replace":
# For find_replace, we can't easily count replacements from here
# since we don't have the original content, but the server handled it
summary.append("operation: Find and replace operation completed")
elif operation == "replace_section":
summary.append(f"operation: Replaced content under section '{section}'")
summary.append(f"operation: Created note with {lines_added} lines")
else:
summary = [
f"# Edited note ({operation})",
f"project: {active_project.name}",
f"file_path: {result.file_path}",
f"permalink: {result.permalink}",
f"checksum: {result.checksum[:8] if result.checksum else 'unknown'}",
]
# Add operation-specific details
if operation == "append":
lines_added = len(content.split("\n"))
summary.append(f"operation: Added {lines_added} lines to end of note")
elif operation == "prepend":
lines_added = len(content.split("\n"))
summary.append(f"operation: Added {lines_added} lines to beginning of note")
elif operation == "find_replace":
# For find_replace, we can't easily count replacements from here
# since we don't have the original content, but the server handled it
summary.append("operation: Find and replace operation completed")
elif operation == "replace_section":
summary.append(f"operation: Replaced content under section '{section}'")
# Count observations by category (reuse logic from write_note)
categories = {}
@@ -313,6 +420,7 @@ async def edit_note(
permalink=result.permalink,
observations_count=len(result.observations),
relations_count=len(result.relations),
file_created=file_created,
)
if output_format == "json":
@@ -322,6 +430,7 @@ async def edit_note(
"file_path": result.file_path,
"checksum": result.checksum,
"operation": operation,
"fileCreated": file_created,
}
summary_result = "\n".join(summary)
@@ -336,8 +445,14 @@ async def edit_note(
"file_path": None,
"checksum": None,
"operation": operation,
"fileCreated": False,
"error": str(e),
}
return _format_error_response(
str(e), operation, identifier, find_text, expected_replacements, active_project.name
str(e),
operation,
identifier,
find_text,
effective_replacements,
active_project.name,
)
@@ -0,0 +1,271 @@
"""MCP tools for graph intelligence and FCM contracts."""
from typing import Any, Literal
from fastmcp import Context
from basic_memory.mcp.project_context import get_project_client
from basic_memory.mcp.server import mcp
from basic_memory.schemas.graph_intelligence import (
FCMExportRequest,
FCMImportRequest,
FCMRankActionsRequest,
FCMSimulateRequest,
GraphImpactRequest,
GraphLineageRequest,
GraphReindexRequest,
)
def _format_lineage_text(result: dict[str, Any]) -> str:
root = result["root"]["title"]
path_count = len(result.get("paths", []))
return f"# Graph Lineage\n\nRoot: {root}\nPaths: {path_count}"
def _format_impact_text(result: dict[str, Any]) -> str:
target = result["target"]["title"]
affected = len(result.get("affected", []))
return f"# Graph Impact\n\nTarget: {target}\nAffected: {affected}"
def _format_health_text(result: dict[str, Any]) -> str:
metrics = result["metrics"]
return (
"# Graph Health\n\n"
f"- orphan_rate: {metrics['orphan_rate']}\n"
f"- stale_central_nodes: {metrics['stale_central_nodes']}\n"
f"- overloaded_hubs: {metrics['overloaded_hubs']}\n"
f"- contradiction_candidates: {metrics['contradiction_candidates']}"
)
def _format_fcm_simulate_text(result: dict[str, Any]) -> str:
deltas = len(result.get("deltas", []))
converged = result["stability"]["converged"]
return f"# FCM Simulation\n\nDeltas: {deltas}\nConverged: {converged}"
def _format_fcm_rank_text(result: dict[str, Any]) -> str:
goal = result["goal"]["label"]
count = len(result.get("recommendations", []))
return f"# FCM Action Ranking\n\nGoal: {goal}\nRecommendations: {count}"
@mcp.tool(annotations={"readOnlyHint": True, "openWorldHint": False})
async def graph_lineage(
start: str,
goal: str | None = None,
max_hops: int = 4,
relation_filters: list[str] | None = None,
project: str | None = None,
workspace: str | None = None,
output_format: Literal["json", "text"] = "json",
context: Context | None = None,
) -> dict[str, Any] | str:
"""Get lineage paths from a start node toward an optional goal."""
from basic_memory.mcp.clients import GraphClient
request = GraphLineageRequest(
start=start,
goal=goal,
max_hops=max_hops,
relation_filters=relation_filters or [],
)
async with get_project_client(project, workspace, context) as (client, active_project):
graph_client = GraphClient(client, active_project.external_id)
result = await graph_client.lineage(request)
payload = result.model_dump(mode="json")
if output_format == "text":
return _format_lineage_text(payload)
return payload
@mcp.tool(annotations={"readOnlyHint": True, "openWorldHint": False})
async def graph_impact(
target: str,
horizon: int,
relation_filters: list[str] | None = None,
include_reasons: bool = True,
project: str | None = None,
workspace: str | None = None,
output_format: Literal["json", "text"] = "json",
context: Context | None = None,
) -> dict[str, Any] | str:
"""Get impact radius from a target node."""
from basic_memory.mcp.clients import GraphClient
request = GraphImpactRequest(
target=target,
horizon=horizon,
relation_filters=relation_filters or [],
include_reasons=include_reasons,
)
async with get_project_client(project, workspace, context) as (client, active_project):
graph_client = GraphClient(client, active_project.external_id)
result = await graph_client.impact(request)
payload = result.model_dump(mode="json")
if output_format == "text":
return _format_impact_text(payload)
return payload
@mcp.tool(annotations={"readOnlyHint": True, "openWorldHint": False})
async def graph_health(
scope: str | None = None,
timeframe: str | None = None,
project: str | None = None,
workspace: str | None = None,
output_format: Literal["json", "text"] = "json",
context: Context | None = None,
) -> dict[str, Any] | str:
"""Get graph health metrics and issues."""
from basic_memory.mcp.clients import GraphClient
async with get_project_client(project, workspace, context) as (client, active_project):
graph_client = GraphClient(client, active_project.external_id)
result = await graph_client.health(scope=scope, timeframe=timeframe)
payload = result.model_dump(mode="json")
if output_format == "text":
return _format_health_text(payload)
return payload
@mcp.tool(annotations={"readOnlyHint": False, "openWorldHint": False})
async def fcm_simulate(
actions: list[dict[str, Any]],
scenario: dict[str, Any] | None = None,
clamp_rules: list[dict[str, Any]] | None = None,
project: str | None = None,
workspace: str | None = None,
output_format: Literal["json", "text"] = "json",
context: Context | None = None,
) -> dict[str, Any] | str:
"""Run an FCM simulation with optional scenario controls."""
from basic_memory.mcp.clients import FCMClient
request = FCMSimulateRequest.model_validate(
{
"actions": actions,
"scenario": scenario or {},
"clamp_rules": clamp_rules or [],
}
)
async with get_project_client(project, workspace, context) as (client, active_project):
fcm_client = FCMClient(client, active_project.external_id)
result = await fcm_client.simulate(request)
payload = result.model_dump(mode="json")
if output_format == "text":
return _format_fcm_simulate_text(payload)
return payload
@mcp.tool(annotations={"readOnlyHint": True, "openWorldHint": False})
async def fcm_rank_actions(
goal: str,
constraints: dict[str, Any] | None = None,
top_k: int = 10,
project: str | None = None,
workspace: str | None = None,
output_format: Literal["json", "text"] = "json",
context: Context | None = None,
) -> dict[str, Any] | str:
"""Rank intervention actions for an FCM goal node."""
from basic_memory.mcp.clients import FCMClient
request = FCMRankActionsRequest.model_validate(
{
"goal": goal,
"constraints": constraints or {},
"top_k": top_k,
}
)
async with get_project_client(project, workspace, context) as (client, active_project):
fcm_client = FCMClient(client, active_project.external_id)
result = await fcm_client.rank_actions(request)
payload = result.model_dump(mode="json")
if output_format == "text":
return _format_fcm_rank_text(payload)
return payload
@mcp.tool(annotations={"readOnlyHint": False, "openWorldHint": False})
async def fcm_import_model(
source: str,
format: Literal["csv_bundle_v1"] = "csv_bundle_v1",
merge_mode: Literal["replace", "upsert"] = "upsert",
project: str | None = None,
workspace: str | None = None,
output_format: Literal["json", "text"] = "json",
context: Context | None = None,
) -> dict[str, Any] | str:
"""Import an FCM model from an external source."""
from basic_memory.mcp.clients import FCMClient
request = FCMImportRequest(source=source, format=format, merge_mode=merge_mode)
async with get_project_client(project, workspace, context) as (client, active_project):
fcm_client = FCMClient(client, active_project.external_id)
result = await fcm_client.import_model(request)
payload = result.model_dump(mode="json")
if output_format == "text":
return (
"# FCM Import\n\n"
f"Import ID: {payload['import_id']}\n"
f"Nodes Loaded: {payload['nodes_loaded']}\n"
f"Edges Loaded: {payload['edges_loaded']}"
)
return payload
@mcp.tool(annotations={"readOnlyHint": True, "openWorldHint": False})
async def fcm_export_model(
format: Literal["csv_bundle_v1"] = "csv_bundle_v1",
selection: dict[str, Any] | None = None,
project: str | None = None,
workspace: str | None = None,
output_format: Literal["json", "text"] = "json",
context: Context | None = None,
) -> dict[str, Any] | str:
"""Export an FCM model selection."""
from basic_memory.mcp.clients import FCMClient
request = FCMExportRequest.model_validate(
{
"format": format,
"selection": selection or {},
}
)
async with get_project_client(project, workspace, context) as (client, active_project):
fcm_client = FCMClient(client, active_project.external_id)
result = await fcm_client.export_model(request)
payload = result.model_dump(mode="json")
if output_format == "text":
return (
"# FCM Export\n\n"
f"Export ID: {payload['export_id']}\n"
f"Node Count: {payload['node_count']}\n"
f"Edge Count: {payload['edge_count']}"
)
return payload
@mcp.tool(annotations={"readOnlyHint": False, "openWorldHint": False})
async def graph_reindex(
mode: Literal["full", "incremental"] = "incremental",
reason: str | None = None,
project: str | None = None,
workspace: str | None = None,
output_format: Literal["json", "text"] = "json",
context: Context | None = None,
) -> dict[str, Any] | str:
"""Queue a graph reindex for the active project."""
from basic_memory.mcp.clients import GraphClient
request = GraphReindexRequest(mode=mode, reason=reason)
async with get_project_client(project, workspace, context) as (client, active_project):
graph_client = GraphClient(client, active_project.external_id)
result = await graph_client.reindex(request)
payload = result.model_dump(mode="json")
if output_format == "text":
return f"# Graph Reindex\n\nJob ID: {payload['job_id']}\nStatus: {payload['status']}"
return payload
+147 -12
View File
@@ -11,7 +11,141 @@ from fastmcp import Context
from basic_memory.mcp.project_context import get_project_client
from basic_memory.mcp.server import mcp
from basic_memory.schemas.schema import ValidationReport, InferenceReport, DriftReport
from basic_memory.schemas.schema import DriftReport, InferenceReport, ValidationReport
def _format_validation_report(report: ValidationReport) -> str:
"""Render a ValidationReport as readable markdown.
Produces output the LLM can display directly instead of trying to
interpret raw JSON, which leads to "undefined — invalid" rendering.
"""
lines: list[str] = []
# --- Header ---
type_label = report.note_type or "all"
lines.append(f"# Schema Validation: {type_label}")
lines.append("")
lines.append(
f"Notes: {report.total_notes} | Valid: {report.valid_count} "
f"| Warnings: {report.warning_count} | Errors: {report.error_count}"
)
lines.append("")
# --- Per-note results ---
for r in report.results:
status = "valid" if r.passed else "INVALID"
lines.append(f"- **{r.note_identifier}** — {status}")
for w in r.warnings:
lines.append(f" - warning: {w}")
for e in r.errors:
lines.append(f" - error: {e}")
return "\n".join(lines)
def _format_inference_report(report: InferenceReport) -> str:
"""Render an InferenceReport as readable markdown.
Without this formatter the LLM receives raw JSON and renders
field names as "undefined".
"""
lines: list[str] = []
# --- Header ---
lines.append(f"# Schema Inference: {report.note_type}")
lines.append("")
lines.append(f"Notes analyzed: {report.notes_analyzed}")
lines.append("")
# --- Suggested schema YAML ---
if report.suggested_schema:
lines.append("## Suggested Schema")
lines.append("")
lines.append("```yaml")
lines.append("---")
lines.append(f"title: {report.note_type.title()}")
lines.append("type: schema")
lines.append(f"entity: {report.note_type}")
lines.append("version: 1")
lines.append("schema:")
for field_name, field_def in report.suggested_schema.items():
lines.append(f" {field_name}: {field_def}")
lines.append("---")
lines.append("```")
lines.append("")
# --- Field frequency table ---
if report.field_frequencies:
lines.append("## Field Frequencies")
lines.append("")
for f in report.field_frequencies:
pct = f"{f.percentage:.0%}"
req_marker = "required" if f.name in report.suggested_required else "optional"
samples = ", ".join(f.sample_values[:3]) if f.sample_values else ""
sample_str = f" (e.g. {samples})" if samples else ""
lines.append(
f"- **{f.name}** ({f.source}) — {pct} ({f.count}/{f.total}) "
f"[{req_marker}]{sample_str}"
)
lines.append("")
# --- Excluded fields ---
if report.excluded:
lines.append("## Excluded (below threshold)")
lines.append("")
for name in report.excluded:
lines.append(f"- {name}")
lines.append("")
return "\n".join(lines)
def _format_drift_report(report: DriftReport) -> str:
"""Render a DriftReport as readable markdown.
Without this formatter the LLM receives raw JSON and renders
field names as "undefined".
"""
lines: list[str] = []
# --- Header ---
lines.append(f"# Schema Drift: {report.note_type}")
lines.append("")
has_drift = report.new_fields or report.dropped_fields or report.cardinality_changes
if not has_drift:
lines.append("No drift detected — schema matches actual usage.")
return "\n".join(lines)
# --- New fields ---
if report.new_fields:
lines.append("## New Fields (in notes but not in schema)")
lines.append("")
for f in report.new_fields:
pct = f"{f.percentage:.0%}"
lines.append(f"- **{f.name}** ({f.source}) — {pct} ({f.count}/{f.total})")
lines.append("")
# --- Dropped fields ---
if report.dropped_fields:
lines.append("## Dropped Fields (in schema but rare in notes)")
lines.append("")
for f in report.dropped_fields:
pct = f"{f.percentage:.0%}"
lines.append(f"- **{f.name}** ({f.source}) — {pct} ({f.count}/{f.total})")
lines.append("")
# --- Cardinality changes ---
if report.cardinality_changes:
lines.append("## Cardinality Changes")
lines.append("")
for change in report.cardinality_changes:
lines.append(f"- {change}")
lines.append("")
return "\n".join(lines)
def _no_notes_guidance(note_type: str, tool_name: str) -> str:
@@ -142,24 +276,25 @@ async def schema_validate(
# Trigger: no entities of this type exist in the project
# Why: can't validate notes that don't exist yet
# Outcome: return guidance on creating notes of this type
if note_type and result.total_entities == 0:
effective_type = note_type or result.note_type or "unknown"
if result.total_entities == 0:
if output_format == "json":
return {"error": f"No notes found of type '{note_type}'"}
return _no_notes_guidance(note_type, "schema_validate")
return {"error": f"No notes found of type '{effective_type}'"}
return _no_notes_guidance(effective_type, "schema_validate")
# --- No schema guard ---
# Trigger: entities exist but none were validated (no schema found)
# Why: notes of this type exist but no schema was found, so none were validated
# Outcome: return guidance on how to create a schema
if note_type and result.total_notes == 0:
if result.total_notes == 0:
if output_format == "json":
return {"error": f"No schema found for type '{note_type}'"}
return _no_schema_guidance(note_type, "schema_validate")
return {"error": f"No schema found for type '{effective_type}'"}
return _no_schema_guidance(effective_type, "schema_validate")
if output_format == "json":
return result.model_dump(mode="json", exclude_none=True)
return result
return _format_validation_report(result)
except Exception as e:
logger.error(f"Schema validation failed: {e}, project: {active_project.name}")
@@ -186,7 +321,7 @@ async def schema_infer(
workspace: Optional[str] = None,
output_format: Literal["text", "json"] = "text",
context: Context | None = None,
) -> InferenceReport | str | dict:
) -> str | dict:
"""Analyze existing notes and suggest a schema definition.
Examines observation categories and relation types across all notes
@@ -274,7 +409,7 @@ async def schema_infer(
if output_format == "json":
return result.model_dump(mode="json", exclude_none=True)
return result
return _format_inference_report(result)
except Exception as e:
logger.error(f"Schema inference failed: {e}, project: {active_project.name}")
@@ -300,7 +435,7 @@ async def schema_diff(
workspace: Optional[str] = None,
output_format: Literal["text", "json"] = "text",
context: Context | None = None,
) -> DriftReport | str | dict:
) -> str | dict:
"""Detect drift between a schema definition and actual note usage.
Compares the existing schema for a note type against how notes of
@@ -362,7 +497,7 @@ async def schema_diff(
if output_format == "json":
return result.model_dump(mode="json", exclude_none=True)
return result
return _format_drift_report(result)
except Exception as e:
logger.error(f"Schema diff failed: {e}, project: {active_project.name}")
+138 -120
View File
@@ -1,5 +1,6 @@
"""Search tools for Basic Memory MCP server."""
import re
from textwrap import dedent
from typing import List, Optional, Dict, Any, Literal
@@ -250,6 +251,46 @@ Error searching for '{query}': {error_message}
- **Patterns**: `tag:example`, `category:observation`"""
def _format_search_markdown(result: SearchResponse, project: str, query: str | None) -> str:
"""Format SearchResponse as compact markdown text.
Produces a human-readable markdown representation suitable for LLM
consumption when structured data isn't needed.
"""
if not result.results:
return f"No results found for '{query or ''}' in project '{project}'."
parts = []
# --- Header ---
if query:
parts.append(f"# Search Results: {query}")
else:
parts.append("# Search Results")
parts.append(f"*project: {project}*")
parts.append("")
# --- Result blocks ---
for r in result.results:
parts.append(f"### {r.title}")
parts.append(f"- permalink: {r.permalink}")
parts.append(f"- score: {r.score:.4f}")
if r.matched_chunk:
parts.append(f"- match: {r.matched_chunk[:200]}")
parts.append("")
# --- Footer with pagination ---
parts.append("---")
count = len(result.results)
parts.append(
f"*{count} result{'s' if count != 1 else ''}"
f" | page {result.current_page}, page_size {result.page_size}"
f"{' | more available' if result.has_more else ''}*"
)
return "\n".join(parts)
@mcp.tool(
description="Search across all content in the knowledge base with advanced syntax support.",
# TODO: re-enable once MCP client rendering is working
@@ -257,7 +298,7 @@ Error searching for '{query}': {error_message}
annotations={"readOnlyHint": True, "openWorldHint": False},
)
async def search_notes(
query: str,
query: Optional[str] = None,
project: Optional[str] = None,
workspace: Optional[str] = None,
page: int = 1,
@@ -272,7 +313,7 @@ async def search_notes(
status: Optional[str] = None,
min_similarity: Optional[float] = None,
context: Context | None = None,
) -> SearchResponse | dict | str:
) -> dict | str:
"""Search across all content in the knowledge base with comprehensive syntax support.
This tool searches the knowledge base using full-text search, pattern matching,
@@ -302,9 +343,9 @@ async def search_notes(
- `search_notes("work-project", "category:observation")` - Filter by observation categories
- `search_notes("team-docs", "author:username")` - Find content by author (if metadata available)
**Note:** `tag:` shorthand requires `search_type="text"` when semantic search is enabled
(the default is hybrid). Alternatively, use the `tags` parameter for tag filtering with
any search type: `search_notes("project", "query", tags=["my-tag"])`
**Note:** `tag:` shorthand is automatically converted to a `tags` filter, so it works
with any search type (text, hybrid, vector). You can also use the `tags` parameter
directly: `search_notes("project", "query", tags=["my-tag"])`
### Search Type Examples
- `search_notes("my-project", "Meeting", search_type="title")` - Search only in titles
@@ -333,8 +374,10 @@ async def search_notes(
- Nested keys use dot notation (e.g., `"schema.confidence"`).
### Filter-only Searches
You can pass an empty query string when only using structured filters:
- `search_notes("my-project", "", metadata_filters={"type": "spec"})`
Omit `query` (or pass None) when only using structured filters:
- `search_notes(metadata_filters={"type": "spec"}, project="my-project")`
- `search_notes(tags=["security"], project="my-project")`
- `search_notes(status="draft", project="my-project")`
### Convenience Filters
`tags` and `status` are shorthand for metadata_filters. If the same key exists in
@@ -347,7 +390,8 @@ async def search_notes(
- `search_notes("archive", "docs/2024-*", search_type="permalink")` - Year-based permalink search
Args:
query: The search query string (supports boolean operators, phrases, patterns)
query: Optional search query string (supports boolean operators, phrases, patterns).
Omit or pass None for filter-only searches using metadata_filters, tags, or status.
project: Project name to search in. Optional - server will resolve using hierarchy.
If unknown, use list_memory_projects() to discover available projects.
page: The page number of results to return (default 1)
@@ -369,7 +413,8 @@ async def search_notes(
context: Optional FastMCP context for performance caching.
Returns:
SearchResponse with results and pagination info, or helpful error guidance if search fails
Formatted markdown text (output_format="text"), dict (output_format="json"),
or helpful error guidance string if search fails
Examples:
# Basic text search
@@ -435,48 +480,77 @@ async def search_notes(
note_types = note_types or []
entity_types = entity_types or []
# Parse tag:<value> shorthand at tool level so it works with all search modes.
# Handles "tag:security", "tag:coffee tag:brewing", "tag:coffee AND tag:brewing".
# Without this, hybrid/vector modes fail because they require non-empty text,
# but the service-layer tag: parser clears the text after the mode is set.
if query and "tag:" in query.lower():
# Extract tag values, splitting comma-separated lists (e.g. "tag:coffee,brewing")
raw_values = re.findall(r"tag:(\S+)", query, flags=re.IGNORECASE)
tag_values = [v for raw in raw_values for v in raw.split(",") if v]
if tag_values:
# Merge with any explicitly provided tags
tags = list(set((tags or []) + tag_values))
# Remove tag: tokens and boolean connectors, keep remaining text as query
remainder = re.sub(r"tag:\S+", "", query, flags=re.IGNORECASE)
remainder = re.sub(r"\b(AND|OR|NOT)\b", "", remainder).strip()
query = remainder or None
# Detect project from memory URL prefix before routing
if project is None:
if project is None and query is not None:
detected = detect_project_from_url_prefix(query, ConfigManager().config)
if detected:
project = detected
async with get_project_client(project, workspace, context) as (client, active_project):
# Handle memory:// URLs by resolving to permalink search
_, resolved_query, is_memory_url = await resolve_project_and_path(
client, query, project, context
)
is_memory_url = False
if query is not None:
_, resolved_query, is_memory_url = await resolve_project_and_path(
client, query, project, context
)
if is_memory_url:
query = resolved_query
effective_search_type = search_type or _default_search_type()
if is_memory_url:
query = resolved_query
effective_search_type = "permalink"
try:
# Create a SearchQuery object based on the parameters
search_query = SearchQuery()
# Map search_type to the appropriate query field and retrieval mode
valid_search_types = {"text", "title", "permalink", "vector", "semantic", "hybrid"}
if effective_search_type == "text":
search_query.text = query
search_query.retrieval_mode = SearchRetrievalMode.FTS
elif effective_search_type in ("vector", "semantic"):
search_query.text = query
search_query.retrieval_mode = SearchRetrievalMode.VECTOR
elif effective_search_type == "hybrid":
search_query.text = query
search_query.retrieval_mode = SearchRetrievalMode.HYBRID
elif effective_search_type == "title":
search_query.title = query
elif effective_search_type == "permalink" and "*" in query:
search_query.permalink_match = query
elif effective_search_type == "permalink":
search_query.permalink = query
else:
raise ValueError(
f"Invalid search_type '{effective_search_type}'. "
f"Valid options: {', '.join(sorted(valid_search_types))}"
)
# Only map search_type to query fields when there is an actual query string.
# When query is None/empty, skip the search mode block — filters-only path.
effective_query = (query or "").strip()
if effective_query:
valid_search_types = {
"text",
"title",
"permalink",
"vector",
"semantic",
"hybrid",
}
if effective_search_type == "text":
search_query.text = effective_query
search_query.retrieval_mode = SearchRetrievalMode.FTS
elif effective_search_type in ("vector", "semantic"):
search_query.text = effective_query
search_query.retrieval_mode = SearchRetrievalMode.VECTOR
elif effective_search_type == "hybrid":
search_query.text = effective_query
search_query.retrieval_mode = SearchRetrievalMode.HYBRID
elif effective_search_type == "title":
search_query.title = effective_query
elif effective_search_type == "permalink" and "*" in effective_query:
search_query.permalink_match = effective_query
elif effective_search_type == "permalink":
search_query.permalink = effective_query
else:
raise ValueError(
f"Invalid search_type '{effective_search_type}'. "
f"Valid options: {', '.join(sorted(valid_search_types))}"
)
# Add optional filters if provided (empty lists are treated as no filter)
if entity_types:
@@ -486,6 +560,13 @@ async def search_notes(
if after_date:
search_query.after_date = after_date
if metadata_filters:
# Alias common column/model names to their frontmatter key equivalents.
# Users often pass "note_type" (the entity model column) when the
# frontmatter field is actually "type".
_METADATA_KEY_ALIASES = {"note_type": "type"}
metadata_filters = {
_METADATA_KEY_ALIASES.get(k, k): v for k, v in metadata_filters.items()
}
search_query.metadata_filters = metadata_filters
if tags:
search_query.tags = tags
@@ -494,7 +575,22 @@ async def search_notes(
if min_similarity is not None:
search_query.min_similarity = min_similarity
logger.info(f"Searching for {search_query} in project {active_project.name}")
# Reject searches with no criteria at all
if search_query.no_criteria():
return (
"# No Search Criteria\n\n"
"Please provide at least one of: `query`, `metadata_filters`, "
"`tags`, `status`, `note_types`, `entity_types`, or `after_date`."
)
# Default to entity-level results to avoid returning individual
# observations/relations as separate search results (see issue #31).
# Applied after no_criteria() so that the implicit default doesn't
# mask a truly empty search request.
if not search_query.entity_types:
search_query.entity_types = [SearchItemType("entity")]
logger.debug(f"Searching for {search_query} in project {active_project.name}")
# Import here to avoid circular import (tools → clients → utils → tools)
from basic_memory.mcp.clients import SearchClient
@@ -508,7 +604,7 @@ async def search_notes(
# Check if we got no results and provide helpful guidance
if not result.results:
logger.info(
logger.debug(
f"Search returned no results for query: {query} in project {active_project.name}"
)
# Don't treat this as an error, but the user might want guidance
@@ -517,91 +613,13 @@ async def search_notes(
if output_format == "json":
return result.model_dump(mode="json", exclude_none=True)
return result
except Exception as e:
logger.error(f"Search failed for query '{query}': {e}, project: {active_project.name}")
# Return formatted error message as string for better user experience
return _format_search_error_response(
active_project.name, str(e), query, effective_search_type
)
@mcp.tool(
description="Search entities by structured frontmatter metadata.",
annotations={"readOnlyHint": True, "openWorldHint": False},
)
async def search_by_metadata(
filters: Dict[str, Any],
project: Optional[str] = None,
workspace: Optional[str] = None,
limit: int = 20,
offset: int = 0,
context: Context | None = None,
) -> SearchResponse | str:
"""Search entities by structured frontmatter metadata.
Args:
filters: Dictionary of metadata filters (e.g., {"status": "in-progress"})
project: Project name to search in. Optional - server will resolve using hierarchy.
limit: Maximum number of results to return
offset: Number of results to skip (for pagination)
context: Optional FastMCP context for performance caching.
Returns:
SearchResponse with results, or helpful error guidance if search fails
"""
if limit <= 0:
return "# Error\n\n`limit` must be greater than 0."
# Build a structured-only search query
search_query = SearchQuery()
search_query.metadata_filters = filters
search_query.entity_types = [SearchItemType.ENTITY]
# Convert offset/limit to page/page_size (API uses paging)
page_size = limit
page = (offset // limit) + 1
offset_within_page = offset % limit
async with get_project_client(project, workspace, context) as (client, active_project):
logger.info(
f"Structured search in project {active_project.name} filters={filters} limit={limit} offset={offset}"
)
try:
from basic_memory.mcp.clients import SearchClient
search_client = SearchClient(client, active_project.external_id)
result = await search_client.search(
search_query.model_dump(),
page=page,
page_size=page_size,
)
# Apply offset within page, fetch next page if needed
if offset_within_page:
remaining = result.results[offset_within_page:]
if len(remaining) < limit:
next_page = page + 1
extra = await search_client.search(
search_query.model_dump(),
page=next_page,
page_size=page_size,
)
remaining.extend(extra.results[: max(0, limit - len(remaining))])
result = SearchResponse(
results=remaining[:limit],
current_page=page,
page_size=page_size,
)
return result
return _format_search_markdown(result, active_project.name, query)
except Exception as e:
logger.error(
f"Metadata search failed for filters '{filters}': {e}, project: {active_project.name}"
f"Search failed for query '{query or ''}': {e}, project: {active_project.name}"
)
# Return formatted error message as string for better user experience
return _format_search_error_response(
active_project.name, str(e), str(filters), "metadata"
active_project.name, str(e), query or "", effective_search_type
)
+57 -5
View File
@@ -1,9 +1,11 @@
"""Write note tool for Basic Memory MCP server."""
import textwrap
from typing import List, Union, Optional, Literal
from loguru import logger
from basic_memory.config import ConfigManager
from basic_memory.mcp.project_context import get_project_client, add_project_metadata
from basic_memory.mcp.server import mcp
from fastmcp import Context
@@ -15,8 +17,8 @@ TagType = Union[List[str], str, None]
@mcp.tool(
description="Create or update a markdown note. Returns a markdown formatted summary of the semantic content.",
annotations={"destructiveHint": False, "idempotentHint": True, "openWorldHint": False},
description="Create a markdown note. If the note already exists, returns an error by default — pass overwrite=True to replace.",
annotations={"destructiveHint": True, "idempotentHint": False, "openWorldHint": False},
)
async def write_note(
title: str,
@@ -27,12 +29,15 @@ async def write_note(
tags: list[str] | str | None = None,
note_type: str = "note",
metadata: dict | None = None,
overwrite: bool | None = None,
output_format: Literal["text", "json"] = "text",
context: Context | None = None,
) -> str | dict:
"""Write a markdown note to the knowledge base.
Creates or updates a markdown note with semantic observations and relations.
Creates a markdown note with semantic observations and relations.
If the note already exists, returns an error by default. Pass overwrite=True
to replace the existing note. For incremental updates, use edit_note instead.
Project Resolution:
Server resolves projects using a unified priority chain (same in local and cloud modes):
@@ -74,6 +79,8 @@ async def write_note(
metadata: Optional dict of extra frontmatter fields merged into entity_metadata.
Useful for schema notes or any note that needs custom YAML frontmatter
beyond title/type/tags. Nested dicts are supported.
overwrite: If True, replace existing note on conflict. If False, error on conflict.
If None (default), consult write_note_overwrite_default config setting.
output_format: "text" returns the existing markdown summary. "json" returns
machine-readable metadata.
context: Optional FastMCP context for performance caching.
@@ -106,12 +113,13 @@ async def write_note(
note_type="guide"
)
# Update existing note (same title/directory)
# Overwrite an existing note explicitly
write_note(
project="my-research",
title="Meeting Notes",
directory="meetings",
content="# Weekly Standup\\n\\n- [decision] Use PostgreSQL instead #tech"
content="# Weekly Standup\\n\\n- [decision] Use PostgreSQL instead #tech",
overwrite=True
)
# Create a schema note with custom frontmatter via metadata
@@ -132,6 +140,13 @@ async def write_note(
HTTPError: If project doesn't exist or is inaccessible
SecurityError: If directory path attempts path traversal
"""
# Resolve overwrite flag: explicit parameter > config default
# Trigger: caller omitted the parameter (None)
# Why: lets users set a global default without breaking per-call overrides
effective_overwrite = (
overwrite if overwrite is not None else ConfigManager().config.write_note_overwrite_default
)
async with get_project_client(project, workspace, context) as (client, active_project):
logger.info(
f"MCP tool call tool=write_note project={active_project.name} directory={directory}, title={title}, tags={tags}"
@@ -199,6 +214,23 @@ async def write_note(
or "conflict" in str(e).lower()
or "already exists" in str(e).lower()
):
# Guard: block overwrite unless explicitly enabled
if not effective_overwrite:
logger.warning(
f"write_note blocked: note already exists (overwrite not enabled) "
f"permalink={entity.permalink}"
)
if output_format == "json":
return {
"title": title,
"permalink": entity.permalink,
"file_path": None,
"checksum": None,
"action": "conflict",
"error": "NOTE_ALREADY_EXISTS",
}
return _format_overwrite_error(title, entity.permalink, active_project.name)
logger.debug(f"Entity exists, updating instead permalink={entity.permalink}")
try:
if not entity.permalink:
@@ -270,3 +302,23 @@ async def write_note(
summary_result = "\n".join(summary)
return add_project_metadata(summary_result, active_project.name)
def _format_overwrite_error(title: str, permalink: str | None, project_name: str) -> str:
"""Format a helpful error when write_note is blocked by the overwrite guard."""
return textwrap.dedent(f"""\
# Error: Note already exists
**"{title}"** already exists (permalink: `{permalink}`).
`write_note` does not overwrite by default. Choose an option:
| Goal | Action |
|------|--------|
| Append content | `edit_note("{permalink}", operation="append", content="...")` |
| Prepend content | `edit_note("{permalink}", operation="prepend", content="...")` |
| Replace a section | `edit_note("{permalink}", operation="replace_section", section="...", content="...")` |
| Full replace | `write_note("{title}", ..., overwrite=True)` |
| Inspect first | `read_note("{permalink}")` |
Project: {project_name}""")
+21
View File
@@ -93,6 +93,27 @@ CREATE VIRTUAL TABLE IF NOT EXISTS search_index USING fts5(
);
""")
# Postgres semantic chunk metadata table.
# Matches the Alembic migration (h1b2c3d4e5f6) schema.
# Used by tests to create the table without running full migrations.
CREATE_POSTGRES_SEARCH_VECTOR_CHUNKS_TABLE = DDL("""
CREATE TABLE IF NOT EXISTS search_vector_chunks (
id BIGSERIAL PRIMARY KEY,
entity_id INTEGER NOT NULL,
project_id INTEGER NOT NULL,
chunk_key TEXT NOT NULL,
chunk_text TEXT NOT NULL,
source_hash TEXT NOT NULL,
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
UNIQUE (project_id, entity_id, chunk_key)
)
""")
CREATE_POSTGRES_SEARCH_VECTOR_CHUNKS_INDEX = DDL("""
CREATE INDEX IF NOT EXISTS idx_search_vector_chunks_project_entity
ON search_vector_chunks (project_id, entity_id)
""")
# Local semantic chunk metadata table for SQLite.
# Embedding vectors live in sqlite-vec virtual table keyed by this table rowid.
CREATE_SQLITE_SEARCH_VECTOR_CHUNKS = DDL("""
@@ -1,8 +1,34 @@
"""Factory for creating configured semantic embedding providers."""
from threading import Lock
from basic_memory.config import BasicMemoryConfig
from basic_memory.repository.embedding_provider import EmbeddingProvider
type ProviderCacheKey = tuple[str, str, int | None, int, str | None, int | None, int | None]
_EMBEDDING_PROVIDER_CACHE: dict[ProviderCacheKey, EmbeddingProvider] = {}
_EMBEDDING_PROVIDER_CACHE_LOCK = Lock()
def _provider_cache_key(app_config: BasicMemoryConfig) -> ProviderCacheKey:
"""Build a stable cache key from provider-relevant semantic embedding config."""
return (
app_config.semantic_embedding_provider.strip().lower(),
app_config.semantic_embedding_model,
app_config.semantic_embedding_dimensions,
app_config.semantic_embedding_batch_size,
app_config.semantic_embedding_cache_dir,
app_config.semantic_embedding_threads,
app_config.semantic_embedding_parallel,
)
def reset_embedding_provider_cache() -> None:
"""Clear process-level embedding provider cache (used by tests)."""
with _EMBEDDING_PROVIDER_CACHE_LOCK:
_EMBEDDING_PROVIDER_CACHE.clear()
def create_embedding_provider(app_config: BasicMemoryConfig) -> EmbeddingProvider:
"""Create an embedding provider based on semantic config.
@@ -10,32 +36,50 @@ def create_embedding_provider(app_config: BasicMemoryConfig) -> EmbeddingProvide
When semantic_embedding_dimensions is set in config, it overrides
the provider's default dimensions (384 for FastEmbed, 1536 for OpenAI).
"""
cache_key = _provider_cache_key(app_config)
with _EMBEDDING_PROVIDER_CACHE_LOCK:
if cached_provider := _EMBEDDING_PROVIDER_CACHE.get(cache_key):
return cached_provider
provider_name = app_config.semantic_embedding_provider.strip().lower()
extra_kwargs: dict = {}
if app_config.semantic_embedding_dimensions is not None:
extra_kwargs["dimensions"] = app_config.semantic_embedding_dimensions
provider: EmbeddingProvider
if provider_name == "fastembed":
# Deferred import: fastembed (and its onnxruntime dep) may not be installed
from basic_memory.repository.fastembed_provider import FastEmbedEmbeddingProvider
return FastEmbedEmbeddingProvider(
if app_config.semantic_embedding_cache_dir is not None:
extra_kwargs["cache_dir"] = app_config.semantic_embedding_cache_dir
if app_config.semantic_embedding_threads is not None:
extra_kwargs["threads"] = app_config.semantic_embedding_threads
if app_config.semantic_embedding_parallel is not None:
extra_kwargs["parallel"] = app_config.semantic_embedding_parallel
provider = FastEmbedEmbeddingProvider(
model_name=app_config.semantic_embedding_model,
batch_size=app_config.semantic_embedding_batch_size,
**extra_kwargs,
)
if provider_name == "openai":
elif provider_name == "openai":
# Deferred import: openai may not be installed
from basic_memory.repository.openai_provider import OpenAIEmbeddingProvider
model_name = app_config.semantic_embedding_model or "text-embedding-3-small"
if model_name == "bge-small-en-v1.5":
model_name = "text-embedding-3-small"
return OpenAIEmbeddingProvider(
provider = OpenAIEmbeddingProvider(
model_name=model_name,
batch_size=app_config.semantic_embedding_batch_size,
**extra_kwargs,
)
else:
raise ValueError(f"Unsupported semantic embedding provider: {provider_name}")
raise ValueError(f"Unsupported semantic embedding provider: {provider_name}")
with _EMBEDDING_PROVIDER_CACHE_LOCK:
if cached_provider := _EMBEDDING_PROVIDER_CACHE.get(cache_key):
return cached_provider
_EMBEDDING_PROVIDER_CACHE[cache_key] = provider
return provider
@@ -5,6 +5,8 @@ from __future__ import annotations
import asyncio
from typing import TYPE_CHECKING
from loguru import logger
from basic_memory.repository.embedding_provider import EmbeddingProvider
from basic_memory.repository.semantic_errors import SemanticDependenciesMissingError
@@ -19,16 +21,25 @@ class FastEmbedEmbeddingProvider(EmbeddingProvider):
"bge-small-en-v1.5": "BAAI/bge-small-en-v1.5",
}
def _effective_parallel(self) -> int | None:
return self.parallel if self.parallel is not None and self.parallel > 1 else None
def __init__(
self,
model_name: str = "bge-small-en-v1.5",
*,
batch_size: int = 64,
dimensions: int = 384,
cache_dir: str | None = None,
threads: int | None = None,
parallel: int | None = None,
) -> None:
self.model_name = model_name
self.dimensions = dimensions
self.batch_size = batch_size
self.cache_dir = cache_dir
self.threads = threads
self.parallel = parallel
self._model: TextEmbedding | None = None
self._model_lock = asyncio.Lock()
@@ -52,9 +63,29 @@ class FastEmbedEmbeddingProvider(EmbeddingProvider):
"pip install -U basic-memory"
) from exc
resolved_model_name = self._MODEL_ALIASES.get(self.model_name, self.model_name)
if self.cache_dir is not None and self.threads is not None:
return TextEmbedding(
model_name=resolved_model_name,
cache_dir=self.cache_dir,
threads=self.threads,
)
if self.cache_dir is not None:
return TextEmbedding(model_name=resolved_model_name, cache_dir=self.cache_dir)
if self.threads is not None:
return TextEmbedding(model_name=resolved_model_name, threads=self.threads)
return TextEmbedding(model_name=resolved_model_name)
self._model = await asyncio.to_thread(_create_model)
logger.info(
"FastEmbed model loaded: model_name={model_name} batch_size={batch_size} "
"threads={threads} configured_parallel={configured_parallel} "
"effective_parallel={effective_parallel}",
model_name=self._MODEL_ALIASES.get(self.model_name, self.model_name),
batch_size=self.batch_size,
threads=self.threads,
configured_parallel=self.parallel,
effective_parallel=self._effective_parallel(),
)
return self._model
async def embed_documents(self, texts: list[str]) -> list[list[float]]:
@@ -62,9 +93,23 @@ class FastEmbedEmbeddingProvider(EmbeddingProvider):
return []
model = await self._load_model()
effective_parallel = self._effective_parallel()
logger.debug(
"FastEmbed embed_documents call: text_count={text_count} batch_size={batch_size} "
"threads={threads} configured_parallel={configured_parallel} "
"effective_parallel={effective_parallel}",
text_count=len(texts),
batch_size=self.batch_size,
threads=self.threads,
configured_parallel=self.parallel,
effective_parallel=effective_parallel,
)
def _embed_batch() -> list[list[float]]:
vectors = list(model.embed(texts, batch_size=self.batch_size))
embed_kwargs: dict[str, int] = {"batch_size": self.batch_size}
if effective_parallel is not None:
embed_kwargs["parallel"] = effective_parallel
vectors = list(model.embed(texts, **embed_kwargs))
normalized: list[list[float]] = []
for vector in vectors:
values = vector.tolist() if hasattr(vector, "tolist") else vector
@@ -2,9 +2,10 @@
from typing import Dict, List, Sequence
from sqlalchemy import select
from sqlalchemy.ext.asyncio import async_sessionmaker
from sqlalchemy.orm import selectinload
from sqlalchemy.orm.interfaces import LoaderOption
from basic_memory.models import Observation
from basic_memory.repository.repository import Repository
@@ -22,6 +23,10 @@ class ObservationRepository(Repository[Observation]):
"""
super().__init__(session_maker, Observation, project_id=project_id)
def get_load_options(self) -> List[LoaderOption]:
"""Eager-load parent entity to prevent N+1 if obs.entity is accessed."""
return [selectinload(Observation.entity)]
async def find_by_entity(self, entity_id: int) -> Sequence[Observation]:
"""Find all observations for a specific entity."""
query = select(Observation).filter(Observation.entity_id == entity_id)
@@ -58,6 +58,9 @@ class PostgresSearchRepository(SearchRepositoryBase):
self._semantic_enabled = self._app_config.semantic_search_enabled
self._semantic_vector_k = self._app_config.semantic_vector_k
self._semantic_min_similarity = self._app_config.semantic_min_similarity
self._semantic_embedding_sync_batch_size = (
self._app_config.semantic_embedding_sync_batch_size
)
self._embedding_provider = embedding_provider
self._vector_dimensions = 384
self._vector_tables_initialized = False
@@ -267,7 +270,7 @@ class PostgresSearchRepository(SearchRepositoryBase):
if self._vector_tables_initialized:
return
logger.info("Ensuring Postgres vector tables exist for semantic search")
logger.debug("Ensuring Postgres vector tables exist for semantic search")
async with self._vector_tables_lock:
if self._vector_tables_initialized:
@@ -358,7 +361,7 @@ class PostgresSearchRepository(SearchRepositoryBase):
)
await session.commit()
logger.info(f"Postgres vector tables ready (dimensions={self._vector_dimensions})")
logger.debug(f"Postgres vector tables ready (dimensions={self._vector_dimensions})")
self._vector_tables_initialized = True
async def _get_existing_embedding_dims(self, session: AsyncSession) -> int | None:
@@ -689,9 +692,7 @@ class PostgresSearchRepository(SearchRepositoryBase):
for idx, note_type in enumerate(note_types):
param_name = f"note_type_{idx}"
params[param_name] = json.dumps({"note_type": note_type})
type_conditions.append(
f"search_index.metadata @> CAST(:{param_name} AS jsonb)"
)
type_conditions.append(f"search_index.metadata @> CAST(:{param_name} AS jsonb)")
conditions.append(f"({' OR '.join(type_conditions)})")
# Handle date filter
@@ -41,7 +41,7 @@ class SearchIndexRow:
# Matched chunk text from vector search (the actual content that matched the query)
matched_chunk_text: Optional[str] = None
CONTENT_DISPLAY_LIMIT = 250
CONTENT_DISPLAY_LIMIT = 4000
@property
def content(self):
@@ -7,7 +7,7 @@ The actual repository implementations are backend-specific:
"""
from datetime import datetime
from typing import List, Optional, Protocol
from typing import Any, Callable, List, Optional, Protocol
from sqlalchemy import Result
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
@@ -15,6 +15,7 @@ from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from basic_memory.config import BasicMemoryConfig, ConfigManager, DatabaseBackend
from basic_memory.repository.postgres_search_repository import PostgresSearchRepository
from basic_memory.repository.search_index_row import SearchIndexRow
from basic_memory.repository.search_repository_base import VectorSyncBatchResult
from basic_memory.repository.sqlite_search_repository import SQLiteSearchRepository
from basic_memory.schemas.search import SearchItemType, SearchRetrievalMode
@@ -69,6 +70,14 @@ class SearchRepository(Protocol):
"""Sync semantic vector chunks for an entity."""
...
async def sync_entity_vectors_batch(
self,
entity_ids: list[int],
progress_callback: Optional[Callable[[int, int, int], Any]] = None,
) -> VectorSyncBatchResult:
"""Sync semantic vector chunks for a batch of entities."""
...
async def execute_query(self, query, params: dict) -> Result:
"""Execute a raw SQL query."""
...
@@ -5,9 +5,9 @@ import json
import re
import time
from abc import ABC, abstractmethod
from dataclasses import replace
from dataclasses import dataclass, field, replace
from datetime import datetime
from typing import Any, Dict, List, Optional
from typing import Any, Callable, Dict, List, Optional
from loguru import logger
from sqlalchemy import Executable, Result, text
@@ -25,20 +25,67 @@ from basic_memory.schemas.search import SearchItemType, SearchRetrievalMode
# --- Semantic search constants ---
VECTOR_FILTER_SCAN_LIMIT = 50000
RRF_K = 60
FUSION_BONUS = 0.3
FTS_GATE_THRESHOLD = 0.0
MAX_VECTOR_CHUNK_CHARS = 900
VECTOR_CHUNK_OVERLAP_CHARS = 120
TOP_CHUNKS_PER_RESULT = 5
SMALL_NOTE_CONTENT_LIMIT = 2000
HEADER_LINE_PATTERN = re.compile(r"^\s*#{1,6}\s+")
BULLET_PATTERN = re.compile(r"^[\-\*]\s+")
@dataclass
class VectorSyncBatchResult:
"""Aggregate result for batched semantic vector sync runs."""
entities_total: int
entities_synced: int
entities_failed: int
failed_entity_ids: list[int] = field(default_factory=list)
embedding_jobs_total: int = 0
embed_seconds_total: float = 0.0
write_seconds_total: float = 0.0
@dataclass
class _PreparedEntityVectorSync:
"""Prepared chunk mutations + embedding jobs for one entity."""
entity_id: int
sync_start: float
source_rows_count: int
embedding_jobs: list[tuple[int, str]]
@dataclass
class _PendingEmbeddingJob:
"""Pending embedding write entry with entity ownership metadata."""
entity_id: int
chunk_row_id: int
chunk_text: str
@dataclass
class _EntitySyncRuntime:
"""Per-entity runtime counters used while flushes are in flight."""
sync_start: float
source_rows_count: int
embedding_jobs_count: int
remaining_jobs: int
embed_seconds: float = 0.0
write_seconds: float = 0.0
class SearchRepositoryBase(ABC):
"""Abstract base class for backend-specific search repository implementations.
This class defines the common interface that all search repositories must implement,
regardless of whether they use SQLite FTS5 or Postgres tsvector for full-text search.
Shared semantic search logic (chunking, embedding orchestration, hybrid RRF fusion)
Shared semantic search logic (chunking, embedding orchestration, hybrid score-based fusion)
lives here. Backend-specific operations are delegated to abstract hooks.
Concrete implementations:
@@ -51,6 +98,7 @@ class SearchRepositoryBase(ABC):
_semantic_vector_k: int
_semantic_min_similarity: float
_embedding_provider: Optional[EmbeddingProvider]
_semantic_embedding_sync_batch_size: int
_vector_dimensions: int
_vector_tables_initialized: bool
@@ -560,15 +608,205 @@ class SearchRepositoryBase(ABC):
# ------------------------------------------------------------------
async def sync_entity_vectors(self, entity_id: int) -> None:
"""Sync semantic chunk rows + embeddings for a single entity.
"""Sync semantic chunk rows + embeddings for a single entity."""
await self._sync_entity_vectors_internal(
[entity_id],
progress_callback=None,
continue_on_error=False,
)
This is the shared orchestration logic. Backend-specific SQL operations
are delegated to abstract hooks (_delete_entity_chunks, _write_embeddings, etc.).
"""
async def sync_entity_vectors_batch(
self,
entity_ids: list[int],
progress_callback: Optional[Callable[[int, int, int], Any]] = None,
) -> VectorSyncBatchResult:
"""Sync semantic chunk rows + embeddings for a batch of entities."""
return await self._sync_entity_vectors_internal(
entity_ids,
progress_callback=progress_callback,
continue_on_error=True,
)
async def _sync_entity_vectors_internal(
self,
entity_ids: list[int],
progress_callback: Optional[Callable[[int, int, int], Any]],
continue_on_error: bool,
) -> VectorSyncBatchResult:
"""Run shared vector sync orchestration for one or many entities."""
self._assert_semantic_available()
await self._ensure_vector_tables()
assert self._embedding_provider is not None
total_entities = len(entity_ids)
result = VectorSyncBatchResult(
entities_total=total_entities,
entities_synced=0,
entities_failed=0,
)
if total_entities == 0:
return result
logger.info(
"Vector batch sync start: project_id={project_id} entities_total={entities_total} "
"sync_batch_size={sync_batch_size}",
project_id=self.project_id,
entities_total=total_entities,
sync_batch_size=self._semantic_embedding_sync_batch_size,
)
pending_jobs: list[_PendingEmbeddingJob] = []
entity_runtime: dict[int, _EntitySyncRuntime] = {}
failed_entity_ids: set[int] = set()
synced_entity_ids: set[int] = set()
for index, entity_id in enumerate(entity_ids):
if progress_callback is not None:
progress_callback(entity_id, index, total_entities)
try:
prepared = await self._prepare_entity_vector_jobs(entity_id)
except Exception as exc:
if not continue_on_error:
raise
failed_entity_ids.add(entity_id)
logger.warning(
"Vector batch sync entity prepare failed: project_id={project_id} "
"entity_id={entity_id} error={error}",
project_id=self.project_id,
entity_id=entity_id,
error=str(exc),
)
continue
embedding_jobs_count = len(prepared.embedding_jobs)
result.embedding_jobs_total += embedding_jobs_count
if embedding_jobs_count == 0:
synced_entity_ids.add(entity_id)
total_seconds = time.perf_counter() - prepared.sync_start
self._log_vector_sync_complete(
entity_id=entity_id,
total_seconds=total_seconds,
embed_seconds=0.0,
write_seconds=0.0,
source_rows_count=prepared.source_rows_count,
embedding_jobs_count=0,
)
continue
entity_runtime[entity_id] = _EntitySyncRuntime(
sync_start=prepared.sync_start,
source_rows_count=prepared.source_rows_count,
embedding_jobs_count=embedding_jobs_count,
remaining_jobs=embedding_jobs_count,
)
pending_jobs.extend(
_PendingEmbeddingJob(
entity_id=entity_id, chunk_row_id=row_id, chunk_text=chunk_text
)
for row_id, chunk_text in prepared.embedding_jobs
)
while len(pending_jobs) >= self._semantic_embedding_sync_batch_size:
flush_jobs = pending_jobs[: self._semantic_embedding_sync_batch_size]
pending_jobs = pending_jobs[self._semantic_embedding_sync_batch_size :]
try:
embed_seconds, write_seconds = await self._flush_embedding_jobs(
flush_jobs=flush_jobs,
entity_runtime=entity_runtime,
synced_entity_ids=synced_entity_ids,
)
result.embed_seconds_total += embed_seconds
result.write_seconds_total += write_seconds
except Exception as exc:
if not continue_on_error:
raise
affected_entity_ids = sorted({job.entity_id for job in flush_jobs})
failed_entity_ids.update(affected_entity_ids)
for failed_entity_id in affected_entity_ids:
entity_runtime.pop(failed_entity_id, None)
logger.warning(
"Vector batch sync flush failed: project_id={project_id} "
"affected_entities={affected_entities} chunk_count={chunk_count} error={error}",
project_id=self.project_id,
affected_entities=affected_entity_ids,
chunk_count=len(flush_jobs),
error=str(exc),
)
if pending_jobs:
flush_jobs = list(pending_jobs)
pending_jobs = []
try:
embed_seconds, write_seconds = await self._flush_embedding_jobs(
flush_jobs=flush_jobs,
entity_runtime=entity_runtime,
synced_entity_ids=synced_entity_ids,
)
result.embed_seconds_total += embed_seconds
result.write_seconds_total += write_seconds
except Exception as exc:
if not continue_on_error:
raise
affected_entity_ids = sorted({job.entity_id for job in flush_jobs})
failed_entity_ids.update(affected_entity_ids)
for failed_entity_id in affected_entity_ids:
entity_runtime.pop(failed_entity_id, None)
logger.warning(
"Vector batch sync final flush failed: project_id={project_id} "
"affected_entities={affected_entities} chunk_count={chunk_count} error={error}",
project_id=self.project_id,
affected_entities=affected_entity_ids,
chunk_count=len(flush_jobs),
error=str(exc),
)
# Trigger: this should never happen after all flushes succeed.
# Why: remaining jobs mean runtime tracking drifted from queued jobs.
# Outcome: fail-safe marks these entities as failed to avoid false positives.
if entity_runtime:
orphan_runtime_entities = sorted(entity_runtime.keys())
failed_entity_ids.update(orphan_runtime_entities)
logger.warning(
"Vector batch sync left unfinished entities after flushes: "
"project_id={project_id} unfinished_entities={unfinished_entities}",
project_id=self.project_id,
unfinished_entities=orphan_runtime_entities,
)
# Keep result counters aligned with successful/failed terminal states.
synced_entity_ids.difference_update(failed_entity_ids)
result.failed_entity_ids = sorted(failed_entity_ids)
result.entities_failed = len(result.failed_entity_ids)
result.entities_synced = len(synced_entity_ids)
logger.info(
"Vector batch sync complete: project_id={project_id} entities_total={entities_total} "
"entities_synced={entities_synced} entities_failed={entities_failed} "
"embedding_jobs_total={embedding_jobs_total} embed_seconds_total={embed_seconds_total:.3f} "
"write_seconds_total={write_seconds_total:.3f}",
project_id=self.project_id,
entities_total=result.entities_total,
entities_synced=result.entities_synced,
entities_failed=result.entities_failed,
embedding_jobs_total=result.embedding_jobs_total,
embed_seconds_total=result.embed_seconds_total,
write_seconds_total=result.write_seconds_total,
)
return result
async def _prepare_entity_vector_jobs(self, entity_id: int) -> _PreparedEntityVectorSync:
"""Prepare chunk mutations and embedding jobs for one entity."""
sync_start = time.perf_counter()
logger.info(
"Vector sync start: project_id={project_id} entity_id={entity_id}",
project_id=self.project_id,
entity_id=entity_id,
)
async with db.scoped_session(self.session_maker) as session:
await self._prepare_vector_session(session)
@@ -594,18 +832,49 @@ class SearchRepositoryBase(ABC):
},
)
rows = row_result.fetchall()
source_rows_count = len(rows)
built_chunk_records_count = 0
# No search_index rows → delete all chunk/embedding data for this entity.
if not rows:
logger.info(
"Vector sync source prepared: project_id={project_id} entity_id={entity_id} "
"source_rows_count={source_rows_count} "
"built_chunk_records_count={built_chunk_records_count}",
project_id=self.project_id,
entity_id=entity_id,
source_rows_count=source_rows_count,
built_chunk_records_count=built_chunk_records_count,
)
await self._delete_entity_chunks(session, entity_id)
await session.commit()
return
return _PreparedEntityVectorSync(
entity_id=entity_id,
sync_start=sync_start,
source_rows_count=source_rows_count,
embedding_jobs=[],
)
chunk_records = self._build_chunk_records(rows)
built_chunk_records_count = len(chunk_records)
logger.info(
"Vector sync source prepared: project_id={project_id} entity_id={entity_id} "
"source_rows_count={source_rows_count} "
"built_chunk_records_count={built_chunk_records_count}",
project_id=self.project_id,
entity_id=entity_id,
source_rows_count=source_rows_count,
built_chunk_records_count=built_chunk_records_count,
)
if not chunk_records:
await self._delete_entity_chunks(session, entity_id)
await session.commit()
return
return _PreparedEntityVectorSync(
entity_id=entity_id,
sync_start=sync_start,
source_rows_count=source_rows_count,
embedding_jobs=[],
)
# --- Diff existing chunks against incoming ---
existing_rows_result = await session.execute(
@@ -617,6 +886,7 @@ class SearchRepositoryBase(ABC):
{"project_id": self.project_id, "entity_id": entity_id},
)
existing_by_key = {row.chunk_key: row for row in existing_rows_result.fetchall()}
existing_chunks_count = len(existing_by_key)
incoming_hashes = {
record["chunk_key"]: record["source_hash"] for record in chunk_records
}
@@ -625,6 +895,7 @@ class SearchRepositoryBase(ABC):
for chunk_key, row in existing_by_key.items()
if chunk_key not in incoming_hashes
]
stale_chunks_count = len(stale_ids)
if stale_ids:
await self._delete_stale_chunks(session, stale_ids, entity_id)
@@ -638,6 +909,8 @@ class SearchRepositoryBase(ABC):
{"project_id": self.project_id, "entity_id": entity_id},
)
orphan_rows = orphan_result.fetchall()
orphan_ids = {int(row.id) for row in orphan_rows}
orphan_chunks_count = len(orphan_ids)
# --- Upsert changed / new chunks, collect embedding jobs ---
timestamp_expr = self._timestamp_now_expr()
@@ -648,7 +921,7 @@ class SearchRepositoryBase(ABC):
# Trigger: chunk exists and hash matches (no content change)
# but chunk has no embedding (orphan from crash).
# Outcome: schedule re-embedding without touching chunk metadata.
is_orphan = current and any(o.id == current.id for o in orphan_rows)
is_orphan = current and int(current.id) in orphan_ids
if current and current.source_hash == record["source_hash"] and not is_orphan:
continue
@@ -691,20 +964,141 @@ class SearchRepositoryBase(ABC):
row_id = int(inserted.scalar_one())
embedding_jobs.append((row_id, record["chunk_text"]))
logger.info(
"Vector sync diff complete: project_id={project_id} entity_id={entity_id} "
"existing_chunks_count={existing_chunks_count} "
"stale_chunks_count={stale_chunks_count} "
"orphan_chunks_count={orphan_chunks_count} "
"embedding_jobs_count={embedding_jobs_count}",
project_id=self.project_id,
entity_id=entity_id,
existing_chunks_count=existing_chunks_count,
stale_chunks_count=stale_chunks_count,
orphan_chunks_count=orphan_chunks_count,
embedding_jobs_count=len(embedding_jobs),
)
await session.commit()
if not embedding_jobs:
return
return _PreparedEntityVectorSync(
entity_id=entity_id,
sync_start=sync_start,
source_rows_count=source_rows_count,
embedding_jobs=embedding_jobs,
)
texts = [t for _, t in embedding_jobs]
async def _flush_embedding_jobs(
self,
flush_jobs: list[_PendingEmbeddingJob],
entity_runtime: dict[int, _EntitySyncRuntime],
synced_entity_ids: set[int],
) -> tuple[float, float]:
"""Embed and persist one queued flush chunk."""
if not flush_jobs:
return 0.0, 0.0
assert self._embedding_provider is not None
embed_start = time.perf_counter()
texts = [job.chunk_text for job in flush_jobs]
embeddings = await self._embedding_provider.embed_documents(texts)
if len(embeddings) != len(embedding_jobs):
embed_seconds = time.perf_counter() - embed_start
embed_rate = (len(flush_jobs) / embed_seconds) if embed_seconds > 0 else 0.0
logger.info(
"Vector batch embed flush: project_id={project_id} chunk_count={chunk_count} "
"embed_seconds={embed_seconds:.3f} embed_rate_chunks_per_second={embed_rate:.2f}",
project_id=self.project_id,
chunk_count=len(flush_jobs),
embed_seconds=embed_seconds,
embed_rate=embed_rate,
)
if len(embeddings) != len(flush_jobs):
raise RuntimeError("Embedding provider returned an unexpected number of vectors.")
write_start = time.perf_counter()
async with db.scoped_session(self.session_maker) as session:
await self._prepare_vector_session(session)
await self._write_embeddings(session, embedding_jobs, embeddings)
write_jobs = [(job.chunk_row_id, job.chunk_text) for job in flush_jobs]
await self._write_embeddings(session, write_jobs, embeddings)
await session.commit()
write_seconds = time.perf_counter() - write_start
write_rate = (len(flush_jobs) / write_seconds) if write_seconds > 0 else 0.0
logger.info(
"Vector batch write flush: project_id={project_id} row_count={row_count} "
"write_seconds={write_seconds:.3f} write_rate_rows_per_second={write_rate:.2f}",
project_id=self.project_id,
row_count=len(flush_jobs),
write_seconds=write_seconds,
write_rate=write_rate,
)
flush_size = len(flush_jobs)
entity_job_counts: dict[int, int] = {}
for job in flush_jobs:
entity_job_counts[job.entity_id] = entity_job_counts.get(job.entity_id, 0) + 1
for entity_id, entity_job_count in entity_job_counts.items():
runtime = entity_runtime.get(entity_id)
if runtime is None:
continue
runtime.remaining_jobs -= entity_job_count
# Attribute flush wall-clock to entities in proportion to rows written.
flush_share = entity_job_count / flush_size
runtime.embed_seconds += embed_seconds * flush_share
runtime.write_seconds += write_seconds * flush_share
if runtime.remaining_jobs <= 0:
synced_entity_ids.add(entity_id)
total_seconds = time.perf_counter() - runtime.sync_start
self._log_vector_sync_complete(
entity_id=entity_id,
total_seconds=total_seconds,
embed_seconds=runtime.embed_seconds,
write_seconds=runtime.write_seconds,
source_rows_count=runtime.source_rows_count,
embedding_jobs_count=runtime.embedding_jobs_count,
)
entity_runtime.pop(entity_id, None)
return embed_seconds, write_seconds
def _log_vector_sync_complete(
self,
*,
entity_id: int,
total_seconds: float,
embed_seconds: float,
write_seconds: float,
source_rows_count: int,
embedding_jobs_count: int,
) -> None:
"""Log completion and slow-entity warnings with a consistent format."""
logger.info(
"Vector sync complete: project_id={project_id} entity_id={entity_id} "
"total_seconds={total_seconds:.3f} embed_seconds={embed_seconds:.3f} "
"write_seconds={write_seconds:.3f} source_rows_count={source_rows_count} "
"embedding_jobs_count={embedding_jobs_count}",
project_id=self.project_id,
entity_id=entity_id,
total_seconds=total_seconds,
embed_seconds=embed_seconds,
write_seconds=write_seconds,
source_rows_count=source_rows_count,
embedding_jobs_count=embedding_jobs_count,
)
if total_seconds > 10:
logger.warning(
"Vector sync slow entity: project_id={project_id} entity_id={entity_id} "
"total_seconds={total_seconds:.3f} embed_seconds={embed_seconds:.3f} "
"write_seconds={write_seconds:.3f} source_rows_count={source_rows_count} "
"embedding_jobs_count={embedding_jobs_count}",
project_id=self.project_id,
entity_id=entity_id,
total_seconds=total_seconds,
embed_seconds=embed_seconds,
write_seconds=write_seconds,
source_rows_count=source_rows_count,
embedding_jobs_count=embedding_jobs_count,
)
async def _prepare_vector_session(self, session: AsyncSession) -> None:
"""Hook for per-session setup (e.g. loading sqlite-vec extension).
@@ -849,6 +1243,7 @@ class SearchRepositoryBase(ABC):
min_similarity: Optional[float] = None,
limit: int,
offset: int,
_emit_observability_log: bool = True,
) -> List[SearchIndexRow]:
"""Run vector-only search returning chunk-level results.
@@ -859,21 +1254,70 @@ class SearchRepositoryBase(ABC):
self._assert_semantic_available()
await self._ensure_vector_tables()
assert self._embedding_provider is not None
query_embedding = await self._embedding_provider.embed_query(search_text.strip())
query_text = search_text.strip()
candidate_limit = max(self._semantic_vector_k, (limit + offset) * 10)
query_start = time.perf_counter()
embed_start = time.perf_counter()
query_embedding = await self._embedding_provider.embed_query(query_text)
embed_ms = (time.perf_counter() - embed_start) * 1000
vector_query_start = time.perf_counter()
async with db.scoped_session(self.session_maker) as session:
await self._prepare_vector_session(session)
vector_rows = await self._run_vector_query(session, query_embedding, candidate_limit)
vector_query_ms = (time.perf_counter() - vector_query_start) * 1000
vector_row_count = len(vector_rows)
hydrate_ms = 0.0
def _log_vector_summary() -> None:
if not _emit_observability_log:
return
total_ms = (time.perf_counter() - query_start) * 1000
logger.info(
"Semantic query timing: project_id={project_id} retrieval_mode={retrieval_mode} "
"query_length={query_length} candidate_limit={candidate_limit} "
"vector_row_count={vector_row_count} embed_ms={embed_ms:.2f} "
"vector_query_ms={vector_query_ms:.2f} hydrate_ms={hydrate_ms:.2f} "
"total_ms={total_ms:.2f}",
project_id=self.project_id,
retrieval_mode="vector",
query_length=len(query_text),
candidate_limit=candidate_limit,
vector_row_count=vector_row_count,
embed_ms=embed_ms,
vector_query_ms=vector_query_ms,
hydrate_ms=hydrate_ms,
total_ms=total_ms,
)
if total_ms > 2000:
logger.warning(
"[SEMANTIC_SLOW_QUERY] Semantic query timing: project_id={project_id} "
"retrieval_mode={retrieval_mode} query_length={query_length} "
"candidate_limit={candidate_limit} vector_row_count={vector_row_count} "
"embed_ms={embed_ms:.2f} vector_query_ms={vector_query_ms:.2f} "
"hydrate_ms={hydrate_ms:.2f} total_ms={total_ms:.2f}",
project_id=self.project_id,
retrieval_mode="vector",
query_length=len(query_text),
candidate_limit=candidate_limit,
vector_row_count=vector_row_count,
embed_ms=embed_ms,
vector_query_ms=vector_query_ms,
hydrate_ms=hydrate_ms,
total_ms=total_ms,
)
if not vector_rows:
_log_vector_summary()
return []
hydrate_start = time.perf_counter()
# Build per-search_index_row similarity scores from chunk-level results.
# Each chunk_key encodes the search_index row type and id.
# Keep the best similarity (and its chunk text) per search_index row id.
# Track the best similarity per row (for ranking) and all chunks (for context).
similarity_by_si_id: dict[int, float] = {}
best_chunk_by_si_id: dict[int, str] = {}
chunks_by_si_id: dict[int, list[tuple[float, str]]] = {}
for row in vector_rows:
chunk_key = row.get("chunk_key", "")
distance = float(row["best_distance"])
@@ -887,9 +1331,11 @@ class SearchRepositoryBase(ABC):
current = similarity_by_si_id.get(si_id)
if current is None or similarity > current:
similarity_by_si_id[si_id] = similarity
best_chunk_by_si_id[si_id] = chunk_text
chunks_by_si_id.setdefault(si_id, []).append((similarity, chunk_text))
if not similarity_by_si_id:
hydrate_ms = (time.perf_counter() - hydrate_start) * 1000
_log_vector_summary()
return []
# Filter out results below the minimum similarity threshold.
@@ -902,6 +1348,8 @@ class SearchRepositoryBase(ABC):
k: v for k, v in similarity_by_si_id.items() if v >= effective_min_similarity
}
if not similarity_by_si_id:
hydrate_ms = (time.perf_counter() - hydrate_start) * 1000
_log_vector_summary()
return []
# Fetch the actual search_index rows
@@ -947,15 +1395,29 @@ class SearchRepositoryBase(ABC):
row = search_index_rows.get(si_id)
if row is None:
continue
# Small notes: return full content so the answer is always present.
# Large notes: return top-N most relevant chunks for richer context.
content_snippet = row.content_snippet or ""
if content_snippet and len(content_snippet) <= SMALL_NOTE_CONTENT_LIMIT:
matched_chunk_text = content_snippet
else:
si_chunks = chunks_by_si_id.get(si_id, [])
si_chunks.sort(key=lambda c: c[0], reverse=True)
top_texts = [text for _, text in si_chunks[:TOP_CHUNKS_PER_RESULT]]
matched_chunk_text = "\n---\n".join(top_texts) if top_texts else None
ranked_rows.append(
replace(
row,
score=similarity,
matched_chunk_text=best_chunk_by_si_id.get(si_id),
matched_chunk_text=matched_chunk_text,
)
)
ranked_rows.sort(key=lambda item: item.score or 0.0, reverse=True)
hydrate_ms = (time.perf_counter() - hydrate_start) * 1000
_log_vector_summary()
return ranked_rows[offset : offset + limit]
async def _fetch_entity_rows_by_ids(self, entity_ids: list[int]) -> dict[int, SearchIndexRow]:
@@ -1053,7 +1515,7 @@ class SearchRepositoryBase(ABC):
return result
# ------------------------------------------------------------------
# Shared semantic search: hybrid RRF fusion
# Shared semantic search: hybrid score-based fusion
# ------------------------------------------------------------------
async def _search_hybrid(
@@ -1071,13 +1533,17 @@ class SearchRepositoryBase(ABC):
limit: int,
offset: int,
) -> List[SearchIndexRow]:
"""Fuse FTS and vector rankings using reciprocal rank fusion (RRF).
"""Fuse FTS and vector results using score-based fusion.
Uses entity_id as the fusion key (not permalink) to correctly handle
entities with NULL permalinks.
Uses search_index row id as the fusion key. The formula
``max(vec, fts) + FUSION_BONUS * min(vec, fts)`` preserves
the dominant signal and rewards dual-source agreement.
"""
self._assert_semantic_available()
query_text = search_text.strip()
query_start = time.perf_counter()
candidate_limit = max(self._semantic_vector_k, (limit + offset) * 10)
fts_start = time.perf_counter()
fts_results = await self.search(
search_text=search_text,
permalink=permalink,
@@ -1091,6 +1557,8 @@ class SearchRepositoryBase(ABC):
limit=candidate_limit,
offset=0,
)
fts_ms = (time.perf_counter() - fts_start) * 1000
vector_start = time.perf_counter()
vector_results = await self._search_vector_only(
search_text=search_text,
permalink=permalink,
@@ -1103,12 +1571,14 @@ class SearchRepositoryBase(ABC):
min_similarity=min_similarity,
limit=candidate_limit,
offset=0,
_emit_observability_log=False,
)
vector_ms = (time.perf_counter() - vector_start) * 1000
fusion_start = time.perf_counter()
# Score-weighted RRF fusion keyed on search_index row id.
# Multiplies the standard 1/(k+rank) score by the normalized original score
# so that high-confidence matches contribute more than weak ones at the same rank.
fused_scores: dict[int, float] = {}
# --- Score-based fusion keyed on search_index row id ---
# FTS scores are normalized to [0, 1] (BM25 is unbounded).
# Vector scores are used raw — already calibrated [0, 1] by _distance_to_similarity().
rows_by_id: dict[int, SearchIndexRow] = {}
# Normalize FTS scores to [0, 1] — handles both SQLite (negative bm25)
@@ -1116,27 +1586,80 @@ class SearchRepositoryBase(ABC):
fts_abs = [abs(row.score or 0.0) for row in fts_results]
fts_max = max(fts_abs) if fts_abs else 1.0
for rank, row in enumerate(fts_results, start=1):
fts_scores: dict[int, float] = {}
for row in fts_results:
if row.id is None:
continue
norm = abs(row.score or 0.0) / fts_max if fts_max > 0 else 0.0
weight = max(norm, 0.1) # floor preserves RRF stability
fused_scores[row.id] = fused_scores.get(row.id, 0.0) + weight * (1.0 / (RRF_K + rank))
# Gate: FTS scores below threshold contribute zero
if norm < FTS_GATE_THRESHOLD:
norm = 0.0
fts_scores[row.id] = norm
rows_by_id[row.id] = row
# Vector scores already in [0, 1] from the similarity formula
vec_max = max((row.score or 0.0) for row in vector_results) if vector_results else 1.0
for rank, row in enumerate(vector_results, start=1):
vec_scores: dict[int, float] = {}
for row in vector_results:
if row.id is None:
continue
norm = (row.score or 0.0) / vec_max if vec_max > 0 else 0.0
weight = max(norm, 0.1) # floor preserves RRF stability
fused_scores[row.id] = fused_scores.get(row.id, 0.0) + weight * (1.0 / (RRF_K + rank))
# Trigger: no re-normalization by vec_max
# Why: vector similarity is already calibrated [0, 1]; re-normalizing
# inflates weak matches when the entire result set is mediocre
vec_scores[row.id] = row.score or 0.0
rows_by_id[row.id] = row
# Fuse: max(v, f) + FUSION_BONUS * min(v, f)
# Preserves the dominant signal; bonus rewards dual-source agreement.
# Output range: [0, 1.3] for dual-source, [0, 1.0] for single-source.
fused_scores: dict[int, float] = {}
for row_id in fts_scores.keys() | vec_scores.keys():
v = vec_scores.get(row_id, 0.0)
f = fts_scores.get(row_id, 0.0)
fused_scores[row_id] = max(v, f) + FUSION_BONUS * min(v, f)
ranked = sorted(fused_scores.items(), key=lambda item: item[1], reverse=True)
output: list[SearchIndexRow] = []
for row_id, fused_score in ranked[offset : offset + limit]:
output.append(replace(rows_by_id[row_id], score=fused_score))
row = rows_by_id[row_id]
# Trigger: FTS-only results have no matched_chunk_text from vector search.
# Why: without chunk text, API falls back to truncated content, losing answer text.
# Outcome: FTS-only results get full content_snippet as matched_chunk.
if row.matched_chunk_text is None and row.content_snippet:
row = replace(row, matched_chunk_text=row.content_snippet)
output.append(replace(row, score=fused_score))
fusion_ms = (time.perf_counter() - fusion_start) * 1000
total_ms = (time.perf_counter() - query_start) * 1000
logger.info(
"Semantic query timing: project_id={project_id} retrieval_mode={retrieval_mode} "
"query_length={query_length} candidate_limit={candidate_limit} "
"fts_count={fts_count} vector_count={vector_count} fts_ms={fts_ms:.2f} "
"vector_ms={vector_ms:.2f} fusion_ms={fusion_ms:.2f} total_ms={total_ms:.2f}",
project_id=self.project_id,
retrieval_mode="hybrid",
query_length=len(query_text),
candidate_limit=candidate_limit,
fts_count=len(fts_results),
vector_count=len(vector_results),
fts_ms=fts_ms,
vector_ms=vector_ms,
fusion_ms=fusion_ms,
total_ms=total_ms,
)
if total_ms > 2500:
logger.warning(
"[SEMANTIC_SLOW_QUERY] Semantic query timing: project_id={project_id} "
"retrieval_mode={retrieval_mode} query_length={query_length} "
"candidate_limit={candidate_limit} fts_count={fts_count} "
"vector_count={vector_count} fts_ms={fts_ms:.2f} vector_ms={vector_ms:.2f} "
"fusion_ms={fusion_ms:.2f} total_ms={total_ms:.2f}",
project_id=self.project_id,
retrieval_mode="hybrid",
query_length=len(query_text),
candidate_limit=candidate_limit,
fts_count=len(fts_results),
vector_count=len(vector_results),
fts_ms=fts_ms,
vector_ms=vector_ms,
fusion_ms=fusion_ms,
total_ms=total_ms,
)
return output
@@ -52,6 +52,9 @@ class SQLiteSearchRepository(SearchRepositoryBase):
self._semantic_enabled = self._app_config.semantic_search_enabled
self._semantic_vector_k = self._app_config.semantic_vector_k
self._semantic_min_similarity = self._app_config.semantic_min_similarity
self._semantic_embedding_sync_batch_size = (
self._app_config.semantic_embedding_sync_batch_size
)
self._embedding_provider = embedding_provider
self._sqlite_vec_lock = asyncio.Lock()
self._vector_tables_initialized = False
@@ -79,7 +82,7 @@ class SQLiteSearchRepository(SearchRepositoryBase):
across server restarts. Also creates vector tables when semantic search
is enabled so missing dependencies are caught at startup, not first query.
"""
logger.info("Initializing SQLite FTS5 search index")
logger.debug("Initializing SQLite FTS5 search index")
try:
async with db.scoped_session(self.session_maker) as session:
# Create FTS5 virtual table if it doesn't exist
@@ -378,7 +381,7 @@ class SQLiteSearchRepository(SearchRepositoryBase):
if self._vector_tables_initialized:
return
logger.info("Ensuring SQLite vector tables exist for semantic search")
logger.debug("Ensuring SQLite vector tables exist for semantic search")
async with db.scoped_session(self.session_maker) as session:
await self._ensure_sqlite_vec_loaded(session)
@@ -431,19 +434,24 @@ class SQLiteSearchRepository(SearchRepositoryBase):
await session.execute(create_sqlite_search_vector_embeddings(self._vector_dimensions))
await session.commit()
logger.info(f"SQLite vector tables ready (dimensions={self._vector_dimensions})")
logger.debug(f"SQLite vector tables ready (dimensions={self._vector_dimensions})")
self._vector_tables_initialized = True
async def _prepare_vector_session(self, session: AsyncSession) -> None:
"""Load sqlite-vec extension for the session."""
await self._ensure_sqlite_vec_loaded(session)
# sqlite-vec hard limit for knn k parameter
SQLITE_VEC_MAX_K = 4096
async def _run_vector_query(
self,
session: AsyncSession,
query_embedding: list[float],
candidate_limit: int,
) -> list[dict]:
# Constraint: sqlite-vec enforces k <= 4096 for knn queries
vector_k = min(candidate_limit, self.SQLITE_VEC_MAX_K)
query_embedding_json = json.dumps(query_embedding)
vector_result = await session.execute(
text(
@@ -458,12 +466,13 @@ class SQLiteSearchRepository(SearchRepositoryBase):
"JOIN search_vector_chunks c ON c.id = vector_matches.rowid "
"WHERE c.project_id = :project_id "
"ORDER BY best_distance ASC "
"LIMIT :vector_k"
"LIMIT :candidate_limit"
),
{
"query_embedding": query_embedding_json,
"project_id": self.project_id,
"vector_k": candidate_limit,
"vector_k": vector_k,
"candidate_limit": candidate_limit,
},
)
return [dict(row) for row in vector_result.mappings().all()]
+37 -4
View File
@@ -14,6 +14,7 @@ Syntax reference:
EntityName as type (capitalized) # entity reference
"""
import re
from dataclasses import dataclass, field
@@ -124,6 +125,31 @@ def _is_entity_ref_type(type_str: str) -> bool:
return len(type_str) > 0 and type_str[0].isupper()
# --- Enum String Parsing ---
def _parse_enum_string(value: str) -> tuple[list[str], str | None]:
"""Parse a string-typed enum value into enum values and optional description.
When picoschema enum values are quoted in YAML frontmatter (required when a
description follows the list), YAML parses the whole thing as a string. This
function extracts the enum values and description from that string.
Examples:
"[active, blocked, done], current state" -> (['active', 'blocked', 'done'], 'current state')
"[active, blocked]" -> (['active', 'blocked'], None)
"active" -> (['active'], None)
"""
# Match bracketed list with optional trailing description
m = re.match(r"\[([^\]]+)\](?:\s*,\s*(.+))?", value)
if m:
items = [item.strip() for item in m.group(1).split(",")]
description = m.group(2).strip() if m.group(2) else None
return items, description
# Plain string — single enum value
return [value.strip()], None
# --- Main Parser ---
@@ -147,18 +173,25 @@ def parse_picoschema(yaml_dict: dict) -> list[SchemaField]:
name, required, is_array, is_enum, is_object = _parse_field_key(key)
# --- Enum fields ---
# Trigger: value is a list (e.g., [active, inactive])
# Why: enums declare allowed values directly as a YAML list
# Trigger: value is a list or a string containing bracketed enum values
# Why: enums declare allowed values directly as a YAML list, or as a quoted
# string when a description follows (e.g., "[a, b], desc" must be quoted
# in YAML to avoid parse errors)
# Outcome: SchemaField with is_enum=True and enum_values populated
if is_enum:
enum_values = value if isinstance(value, list) else [str(value)]
description = None
if isinstance(value, list):
enum_values = [str(v) for v in value]
else:
enum_values, description = _parse_enum_string(str(value))
fields.append(
SchemaField(
name=name,
type="enum",
required=required,
is_enum=True,
enum_values=[str(v) for v in enum_values],
enum_values=enum_values,
description=description,
)
)
continue
+2
View File
@@ -41,6 +41,7 @@ from basic_memory.schemas.project_info import (
ProjectStatistics,
ActivityMetrics,
SystemStatus,
EmbeddingStatus,
ProjectInfoResponse,
)
@@ -78,6 +79,7 @@ __all__ = [
"ProjectStatistics",
"ActivityMetrics",
"SystemStatus",
"EmbeddingStatus",
"ProjectInfoResponse",
# Directory
"DirectoryNode",
@@ -0,0 +1,318 @@
"""Schemas for Local+ graph intelligence and FCM contracts."""
from datetime import datetime
from typing import Any, Literal
from pydantic import BaseModel, Field
# --- Graph contracts ---
class GraphLineageRequest(BaseModel):
"""Request contract for graph lineage queries."""
start: str
goal: str | None = None
max_hops: int = Field(default=4, ge=1, le=6)
relation_filters: list[str] = Field(default_factory=list)
class GraphNodeRef(BaseModel):
"""Minimal graph node descriptor."""
id: str
title: str
permalink: str | None = None
class GraphPathEdge(BaseModel):
"""Edge descriptor for lineage paths."""
relation: str
direction: Literal["outgoing", "incoming"]
class GraphLineagePath(BaseModel):
"""Single lineage path with scores and provenance."""
path_id: str
nodes: list[GraphNodeRef] = Field(default_factory=list)
edges: list[GraphPathEdge] = Field(default_factory=list)
deterministic_path_score: float
confidence: float
evidence_refs: list[str] = Field(default_factory=list)
class GraphLineageResponse(BaseModel):
"""Response contract for graph lineage queries."""
root: GraphNodeRef
paths: list[GraphLineagePath] = Field(default_factory=list)
generated_at: datetime
class GraphImpactRequest(BaseModel):
"""Request contract for impact-radius queries."""
target: str
horizon: int = Field(ge=1, le=4)
relation_filters: list[str] = Field(default_factory=list)
include_reasons: bool = True
class GraphImpactTarget(BaseModel):
"""Impact response target descriptor."""
id: str
title: str
class GraphImpactItem(BaseModel):
"""Affected node entry for impact responses."""
id: str
title: str
distance: int
impact_score: float
confidence: float
reasons: list[str] = Field(default_factory=list)
evidence_refs: list[str] = Field(default_factory=list)
class GraphImpactSummary(BaseModel):
"""Summary counters for impact responses."""
total_considered: int
total_returned: int
class GraphImpactResponse(BaseModel):
"""Response contract for impact-radius queries."""
target: GraphImpactTarget
affected: list[GraphImpactItem] = Field(default_factory=list)
summary: GraphImpactSummary
class GraphHealthMetrics(BaseModel):
"""Top-level graph health metrics."""
orphan_rate: float
stale_central_nodes: int
overloaded_hubs: int
contradiction_candidates: int
class GraphHealthIssue(BaseModel):
"""Actionable graph-health issue entry."""
issue_type: Literal[
"orphan",
"stale_central",
"overloaded_hub",
"contradiction_candidate",
]
entity_id: str
severity: Literal["low", "medium", "high"]
reason: str
suggested_action: str
confidence: float | None = None
class GraphHealthResponse(BaseModel):
"""Response contract for health checks."""
metrics: GraphHealthMetrics
issues: list[GraphHealthIssue] = Field(default_factory=list)
computed_at: datetime
class GraphReindexRequest(BaseModel):
"""Request contract for graph reindex scheduling."""
mode: Literal["full", "incremental"] = "incremental"
reason: str | None = None
class GraphReindexResponse(BaseModel):
"""Response contract for graph reindex scheduling."""
job_id: str
status: Literal["queued", "running", "completed", "failed"]
scheduled_at: datetime
# --- FCM contracts ---
class FCMAction(BaseModel):
"""Action delta for simulation input."""
node_id: str
delta: float
class FCMScenario(BaseModel):
"""Simulation runtime configuration."""
steps: int = 12
activation: Literal["tanh", "sigmoid", "bounded_linear"] = "tanh"
decay: float = 0.05
class FCMClampRule(BaseModel):
"""Clamp bounds for selected nodes."""
node_id: str
min: float
max: float
class FCMSimulateRequest(BaseModel):
"""Request contract for FCM simulation."""
actions: list[FCMAction]
scenario: FCMScenario = Field(default_factory=FCMScenario)
clamp_rules: list[FCMClampRule] = Field(default_factory=list)
class FCMNodeState(BaseModel):
"""Node state in baseline/projected vectors."""
node_id: str
state: float
class FCMNodeDelta(BaseModel):
"""Node delta entry in simulation output."""
node_id: str
delta: float
class FCMStability(BaseModel):
"""Simulation stability metadata."""
converged: bool
iterations_used: int
residual: float
class FCMInfluencer(BaseModel):
"""Top influencer entry for explanation payload."""
source: str
weight: float
class FCMExplanation(BaseModel):
"""Per-node explanation payload."""
node_id: str
top_influencers: list[FCMInfluencer] = Field(default_factory=list)
class FCMSimulateResponse(BaseModel):
"""Response contract for FCM simulation."""
baseline: list[FCMNodeState] = Field(default_factory=list)
projected: list[FCMNodeState] = Field(default_factory=list)
deltas: list[FCMNodeDelta] = Field(default_factory=list)
stability: FCMStability
confidence: float
explanations: list[FCMExplanation] = Field(default_factory=list)
evidence_refs: list[str] = Field(default_factory=list)
class FCMRankConstraints(BaseModel):
"""Constraint set for action ranking."""
max_negative_impact: float | None = None
required_tags: list[str] = Field(default_factory=list)
disallowed_nodes: list[str] = Field(default_factory=list)
class FCMRankActionsRequest(BaseModel):
"""Request contract for FCM action ranking."""
goal: str
constraints: FCMRankConstraints = Field(default_factory=FCMRankConstraints)
top_k: int = Field(default=10, ge=1, le=25)
class FCMGoalRef(BaseModel):
"""Goal descriptor for ranking output."""
node_id: str
label: str
class FCMRecommendation(BaseModel):
"""Ranked intervention candidate."""
action_node_id: str
expected_goal_delta: float
risk_penalty: float
net_score: float
confidence: float
rationale: list[str] = Field(default_factory=list)
evidence_refs: list[str] = Field(default_factory=list)
class FCMRankActionsResponse(BaseModel):
"""Response contract for action ranking."""
goal: FCMGoalRef
recommendations: list[FCMRecommendation] = Field(default_factory=list)
class FCMImportRequest(BaseModel):
"""Request contract for model import."""
source: str
format: Literal["csv_bundle_v1"] = "csv_bundle_v1"
merge_mode: Literal["replace", "upsert"] = "upsert"
class FCMImportResponse(BaseModel):
"""Response contract for model import."""
import_id: str
nodes_loaded: int
edges_loaded: int
warnings: list[str] = Field(default_factory=list)
errors: list[str] = Field(default_factory=list)
class FCMExportSelection(BaseModel):
"""Scope selection for model export."""
scope: Literal["all", "tag", "subgraph"] = "all"
tag: str | None = None
seed_nodes: list[str] = Field(default_factory=list)
class FCMExportRequest(BaseModel):
"""Request contract for model export."""
format: Literal["csv_bundle_v1"] = "csv_bundle_v1"
selection: FCMExportSelection = Field(default_factory=FCMExportSelection)
class FCMExportFile(BaseModel):
"""Single file descriptor in an export response."""
name: str
path: str
class FCMExportResponse(BaseModel):
"""Response contract for model export."""
export_id: str
format: Literal["csv_bundle_v1"]
files: list[FCMExportFile] = Field(default_factory=list)
node_count: int
edge_count: int
metadata: dict[str, Any] | None = None
+21 -19
View File
@@ -125,7 +125,8 @@ class EntitySummary(BaseModel):
type: Literal["entity"] = "entity"
external_id: str # UUID for v2 API routing
entity_id: int # Database ID for v2 API consistency
# COMPAT(v0.18): old clients expect these fields in JSON
entity_id: Optional[int] = None
permalink: Optional[str]
title: str
content: Optional[str] = None
@@ -143,18 +144,19 @@ class RelationSummary(BaseModel):
"""Simplified relation representation."""
type: Literal["relation"] = "relation"
relation_id: int # Database ID for v2 API consistency
entity_id: Optional[int] = None # ID of the entity this relation belongs to
# COMPAT(v0.18): old clients expect these fields in JSON
relation_id: Optional[int] = None
entity_id: Optional[int] = None
title: str
file_path: str
permalink: str
relation_type: str
from_entity: Optional[str] = None
from_entity_id: Optional[int] = None # ID of source entity
from_entity_external_id: Optional[str] = None # UUID of source entity for v2 API routing
from_entity_id: Optional[int] = None
from_entity_external_id: Optional[str] = None
to_entity: Optional[str] = None
to_entity_id: Optional[int] = None # ID of target entity
to_entity_external_id: Optional[str] = None # UUID of target entity for v2 API routing
to_entity_id: Optional[int] = None
to_entity_external_id: Optional[str] = None
created_at: Annotated[
datetime, Field(json_schema_extra={"type": "string", "format": "date-time"})
]
@@ -168,10 +170,11 @@ class ObservationSummary(BaseModel):
"""Simplified observation representation."""
type: Literal["observation"] = "observation"
observation_id: int # Database ID for v2 API consistency
entity_id: Optional[int] = None # ID of the entity this observation belongs to
entity_external_id: Optional[str] = None # UUID of parent entity for v2 API routing
title: str
# COMPAT(v0.18): old clients expect these fields in JSON
observation_id: Optional[int] = None
entity_id: Optional[int] = None
entity_external_id: Optional[str] = None
title: Optional[str] = None
file_path: str
permalink: str
category: str
@@ -192,18 +195,17 @@ class MemoryMetadata(BaseModel):
types: Optional[List[SearchItemType]] = None
depth: int
timeframe: Optional[str] = None
generated_at: Annotated[
datetime, Field(json_schema_extra={"type": "string", "format": "date-time"})
]
primary_count: Optional[int] = None # Changed field name
related_count: Optional[int] = None # Changed field name
total_results: Optional[int] = None # For backward compatibility
# COMPAT(v0.18): old clients expect generated_at and total_results in JSON
generated_at: Optional[datetime] = None
primary_count: Optional[int] = None
related_count: Optional[int] = None
total_results: Optional[int] = None
total_relations: Optional[int] = None
total_observations: Optional[int] = None
@field_serializer("generated_at")
def serialize_generated_at(self, dt: datetime) -> str:
return dt.isoformat()
def serialize_generated_at(self, dt: Optional[datetime]) -> Optional[str]:
return dt.isoformat() if dt else None
class ContextResult(BaseModel):
+27
View File
@@ -79,6 +79,28 @@ class SystemStatus(BaseModel):
timestamp: datetime = Field(description="Timestamp when the information was collected")
class EmbeddingStatus(BaseModel):
"""Embedding/vector index status for a project."""
# Config
semantic_search_enabled: bool
embedding_provider: Optional[str] = None
embedding_model: Optional[str] = None
embedding_dimensions: Optional[int] = None
# Counts
total_indexed_entities: int = 0
total_entities_with_chunks: int = 0
total_chunks: int = 0
total_embeddings: int = 0
orphaned_chunks: int = 0
vector_tables_exist: bool = False
# Derived
reindex_recommended: bool = False
reindex_reason: Optional[str] = None
class ProjectInfoResponse(BaseModel):
"""Response for the project_info tool."""
@@ -99,6 +121,11 @@ class ProjectInfoResponse(BaseModel):
# System status
system: SystemStatus = Field(description="System and service status information")
# Embedding status
embedding_status: Optional[EmbeddingStatus] = Field(
default=None, description="Embedding/vector index status"
)
class ProjectInfoRequest(BaseModel):
"""Request model for switching projects."""
+8 -1
View File
@@ -14,7 +14,7 @@ Key Features:
from datetime import datetime
from typing import List, Optional, Dict
from pydantic import BaseModel, ConfigDict, Field, model_validator
from pydantic import BaseModel, ConfigDict, Field, computed_field, model_validator
from basic_memory.schemas.base import Relation, Permalink, NoteType, ContentType, Observation
@@ -192,6 +192,13 @@ class EntityResponse(SQLAlchemyModel):
title: str
file_path: str
note_type: NoteType
# COMPAT(v0.18): old clients expect entity_type; remove when no longer needed
@computed_field # type: ignore[prop-decorator]
@property
def entity_type(self) -> str:
return self.note_type
entity_metadata: Optional[Dict] = None
checksum: Optional[str] = None
content_type: ContentType
+13 -5
View File
@@ -19,7 +19,11 @@ from basic_memory.file_utils import (
dump_frontmatter,
)
from basic_memory.markdown import EntityMarkdown
from basic_memory.markdown.entity_parser import EntityParser, normalize_frontmatter_metadata
from basic_memory.markdown.entity_parser import (
EntityParser,
_coerce_to_string,
normalize_frontmatter_metadata,
)
from basic_memory.markdown.utils import entity_model_from_markdown, schema_to_markdown
from basic_memory.models import Entity as EntityModel
from basic_memory.models import Observation, Relation
@@ -357,8 +361,11 @@ class EntityService(BaseService[EntityModel]):
# in the existing file. Setting it unconditionally preserves the correct value.
existing_markdown.frontmatter.metadata["permalink"] = new_permalink
# Create a new post with merged metadata
merged_post = frontmatter.Post(post.content, **existing_markdown.frontmatter.metadata)
# Create a new post with merged metadata.
# Avoid **metadata unpacking — user frontmatter may contain reserved keys
# like 'content' or 'handler' that conflict with Post.__init__ (cloud#375).
merged_post = frontmatter.Post(post.content)
merged_post.metadata.update(existing_markdown.frontmatter.metadata)
# write file
final_content = dump_frontmatter(merged_post)
@@ -501,10 +508,11 @@ class EntityService(BaseService[EntityModel]):
if has_frontmatter(new_content):
content_frontmatter = parse_frontmatter(new_content)
# Coerce to string — YAML may parse these as lists (cloud#376)
if "title" in content_frontmatter:
update_data["title"] = content_frontmatter["title"]
update_data["title"] = _coerce_to_string(content_frontmatter["title"])
if "type" in content_frontmatter:
update_data["note_type"] = content_frontmatter["type"]
update_data["note_type"] = _coerce_to_string(content_frontmatter["type"])
if "permalink" in content_frontmatter:
content_markdown = self._build_frontmatter_markdown(
+96
View File
@@ -0,0 +1,96 @@
"""Service layer for FCM contract endpoints."""
from uuid import uuid4
from basic_memory.schemas.graph_intelligence import (
FCMExportFile,
FCMExportRequest,
FCMExportResponse,
FCMGoalRef,
FCMImportRequest,
FCMImportResponse,
FCMNodeDelta,
FCMNodeState,
FCMRankActionsRequest,
FCMRankActionsResponse,
FCMRecommendation,
FCMSimulateRequest,
FCMSimulateResponse,
FCMStability,
)
class FCMService:
"""FCM contract service.
Phase 1 keeps deterministic behavior so API and tool surfaces stabilize
before introducing advanced simulation engines.
"""
async def simulate(self, request: FCMSimulateRequest) -> FCMSimulateResponse:
"""Return deterministic baseline/projected state vectors."""
baseline = [FCMNodeState(node_id=action.node_id, state=0.0) for action in request.actions]
projected = [
FCMNodeState(node_id=action.node_id, state=action.delta) for action in request.actions
]
deltas = [
FCMNodeDelta(node_id=action.node_id, delta=action.delta) for action in request.actions
]
return FCMSimulateResponse(
baseline=baseline,
projected=projected,
deltas=deltas,
stability=FCMStability(
converged=True,
iterations_used=min(request.scenario.steps, 5),
residual=0.0,
),
confidence=0.5,
explanations=[],
evidence_refs=[],
)
async def rank_actions(self, request: FCMRankActionsRequest) -> FCMRankActionsResponse:
"""Return deterministic ranked actions for a target goal."""
recommendations = [
FCMRecommendation(
action_node_id=f"{request.goal}:action:{idx + 1}",
expected_goal_delta=0.25 - (idx * 0.01),
risk_penalty=0.05 + (idx * 0.005),
net_score=0.20 - (idx * 0.015),
confidence=0.5,
rationale=["Contract skeleton recommendation"],
evidence_refs=[],
)
for idx in range(min(request.top_k, 3))
]
return FCMRankActionsResponse(
goal=FCMGoalRef(node_id=request.goal, label=request.goal),
recommendations=recommendations,
)
async def import_model(self, request: FCMImportRequest) -> FCMImportResponse:
"""Return deterministic import metadata."""
_ = request
return FCMImportResponse(
import_id=str(uuid4()),
nodes_loaded=0,
edges_loaded=0,
warnings=[],
errors=[],
)
async def export_model(self, request: FCMExportRequest) -> FCMExportResponse:
"""Return deterministic export metadata and file descriptors."""
scope = request.selection.scope
return FCMExportResponse(
export_id=str(uuid4()),
format=request.format,
files=[
FCMExportFile(name="nodes.csv", path=f"/tmp/{scope}-nodes.csv"),
FCMExportFile(name="edges.csv", path=f"/tmp/{scope}-edges.csv"),
],
node_count=0,
edge_count=0,
metadata={"scope": scope},
)
@@ -0,0 +1,122 @@
"""Service layer for graph intelligence contract endpoints."""
from datetime import datetime, timezone
from uuid import uuid4
from basic_memory.schemas.graph_intelligence import (
GraphHealthMetrics,
GraphHealthResponse,
GraphImpactItem,
GraphImpactRequest,
GraphImpactResponse,
GraphImpactSummary,
GraphImpactTarget,
GraphLineagePath,
GraphLineageRequest,
GraphLineageResponse,
GraphNodeRef,
GraphPathEdge,
GraphReindexResponse,
)
def _normalize_memory_ref(value: str) -> str:
"""Normalize user input into a memory:// reference string."""
if value.startswith("memory://"):
return value
return f"memory://{value}"
def _normalize_node_id(value: str) -> str:
"""Return a stable node id for contract skeleton outputs."""
return value.removeprefix("memory://")
class GraphIntelligenceService:
"""Graph intelligence contract service.
Phase 1 behavior is intentionally deterministic and lightweight so routing,
clients, and contract tests can ship before deeper traversal engines.
"""
async def lineage(self, request: GraphLineageRequest) -> GraphLineageResponse:
"""Return a deterministic lineage payload for the requested root/goal."""
root_ref = _normalize_memory_ref(request.start)
root = GraphNodeRef(
id=_normalize_node_id(root_ref),
title=_normalize_node_id(root_ref),
permalink=_normalize_node_id(root_ref),
)
nodes = [root]
edges: list[GraphPathEdge] = []
if request.goal:
goal_ref = _normalize_memory_ref(request.goal)
nodes.append(
GraphNodeRef(
id=_normalize_node_id(goal_ref),
title=_normalize_node_id(goal_ref),
permalink=_normalize_node_id(goal_ref),
)
)
edges.append(GraphPathEdge(relation="related_to", direction="outgoing"))
path = GraphLineagePath(
path_id=f"path-{uuid4()}",
nodes=nodes,
edges=edges,
deterministic_path_score=1.0 if request.goal else 0.5,
confidence=0.5,
evidence_refs=[root_ref],
)
return GraphLineageResponse(
root=root,
paths=[path],
generated_at=datetime.now(timezone.utc),
)
async def impact(self, request: GraphImpactRequest) -> GraphImpactResponse:
"""Return a deterministic impact preview payload."""
target_id = _normalize_node_id(_normalize_memory_ref(request.target))
affected = [
GraphImpactItem(
id=f"{target_id}:neighbor:1",
title=f"{target_id} dependent",
distance=min(request.horizon, 1),
impact_score=0.55,
confidence=0.5,
reasons=["Connected via typed relation in contract skeleton"],
evidence_refs=[_normalize_memory_ref(request.target)],
)
]
if not request.include_reasons:
affected[0].reasons = []
return GraphImpactResponse(
target=GraphImpactTarget(id=target_id, title=target_id),
affected=affected,
summary=GraphImpactSummary(total_considered=1, total_returned=1),
)
async def health(self, scope: str | None, timeframe: str | None) -> GraphHealthResponse:
"""Return deterministic baseline health metrics."""
_ = (scope, timeframe)
return GraphHealthResponse(
metrics=GraphHealthMetrics(
orphan_rate=0.0,
stale_central_nodes=0,
overloaded_hubs=0,
contradiction_candidates=0,
),
issues=[],
computed_at=datetime.now(timezone.utc),
)
async def start_reindex_job(self) -> GraphReindexResponse:
"""Create reindex job metadata for queued responses."""
return GraphReindexResponse(
job_id=str(uuid4()),
status="queued",
scheduled_at=datetime.now(timezone.utc),
)
+4 -2
View File
@@ -301,8 +301,10 @@ class LinkResolver:
)
if results:
# Look for best match
best_match = min(results, key=lambda x: x.score) # pyright: ignore
# Both SQLite and Postgres return results sorted best-first in SQL
# (SQLite: ORDER BY score ASC for negative BM25, Postgres: ORDER BY score DESC
# for positive ts_rank). Using results[0] is backend-agnostic and correct.
best_match = results[0]
logger.trace(
f"Selected best match from {len(results)} results: {best_match.permalink}"
)
@@ -16,6 +16,7 @@ from basic_memory.models import Project
from basic_memory.repository.project_repository import ProjectRepository
from basic_memory.schemas import (
ActivityMetrics,
EmbeddingStatus,
ProjectInfoResponse,
ProjectStatistics,
SystemStatus,
@@ -81,6 +82,21 @@ class ProjectService:
"""
return self.config_manager.default_project
async def get_default_project_name(self) -> str:
"""Get the default project name, falling back to the database.
ConfigManager reads from the local config file, which doesn't exist
in cloud mode. When it returns None, fall back to the is_default
flag stored in the database.
"""
default = self.config_manager.default_project
if default is not None:
return default
db_default = await self.repository.get_default_project()
if db_default is not None:
return db_default.name
raise ValueError("No default project configured")
@property
def current_project(self) -> Optional[str]:
"""Get the name of the currently active project.
@@ -597,6 +613,9 @@ class ProjectService:
# Get activity metrics for the specified project
activity = await self.get_activity_metrics(db_project.id)
# Get embedding status for the specified project
embedding_status = await self.get_embedding_status(db_project.id)
# Get system status
system = self.get_system_status()
@@ -650,6 +669,7 @@ class ProjectService:
statistics=statistics,
activity=activity,
system=system,
embedding_status=embedding_status,
)
async def get_statistics(self, project_id: int) -> ProjectStatistics:
@@ -918,6 +938,156 @@ class ProjectService:
monthly_growth=monthly_growth,
)
async def get_embedding_status(self, project_id: int) -> EmbeddingStatus:
"""Get embedding/vector index status for the specified project.
Reports config, counts, and whether a reindex is recommended.
"""
config = self.config_manager.config
semantic_enabled = config.semantic_search_enabled
# When semantic search is disabled, return minimal status
if not semantic_enabled:
return EmbeddingStatus(semantic_search_enabled=False)
provider = config.semantic_embedding_provider
model = config.semantic_embedding_model
dimensions = config.semantic_embedding_dimensions
is_postgres = config.database_backend == DatabaseBackend.POSTGRES
# --- Check vector table existence ---
# Both search_vector_chunks and search_vector_embeddings must exist
# for the detailed stats queries (JOINs between them) to work.
if is_postgres:
table_check_sql = text(
"SELECT COUNT(*) FROM information_schema.tables "
"WHERE table_name IN ('search_vector_chunks', 'search_vector_embeddings')"
)
else:
table_check_sql = text(
"SELECT COUNT(*) FROM sqlite_master "
"WHERE type = 'table' AND name IN ('search_vector_chunks', 'search_vector_embeddings')"
)
table_result = await self.repository.execute_query(table_check_sql, {})
vector_tables_exist = (table_result.scalar() or 0) == 2
if not vector_tables_exist:
# Count distinct entities in search index for the recommendation message
si_result = await self.repository.execute_query(
text(
"SELECT COUNT(DISTINCT entity_id) FROM search_index "
"WHERE project_id = :project_id"
),
{"project_id": project_id},
)
total_indexed_entities = si_result.scalar() or 0
return EmbeddingStatus(
semantic_search_enabled=True,
embedding_provider=provider,
embedding_model=model,
embedding_dimensions=dimensions,
total_indexed_entities=total_indexed_entities,
vector_tables_exist=False,
reindex_recommended=True,
reindex_reason=("Vector tables not initialized — run: bm reindex --embeddings"),
)
# --- Count queries (tables exist) ---
si_result = await self.repository.execute_query(
text(
"SELECT COUNT(DISTINCT entity_id) FROM search_index WHERE project_id = :project_id"
),
{"project_id": project_id},
)
total_indexed_entities = si_result.scalar() or 0
chunks_result = await self.repository.execute_query(
text("SELECT COUNT(*) FROM search_vector_chunks WHERE project_id = :project_id"),
{"project_id": project_id},
)
total_chunks = chunks_result.scalar() or 0
entities_with_chunks_result = await self.repository.execute_query(
text(
"SELECT COUNT(DISTINCT entity_id) FROM search_vector_chunks "
"WHERE project_id = :project_id"
),
{"project_id": project_id},
)
total_entities_with_chunks = entities_with_chunks_result.scalar() or 0
# Embeddings count — join pattern differs between SQLite and Postgres
if is_postgres:
embeddings_sql = text(
"SELECT COUNT(*) FROM search_vector_chunks c "
"JOIN search_vector_embeddings e ON e.chunk_id = c.id "
"WHERE c.project_id = :project_id"
)
else:
embeddings_sql = text(
"SELECT COUNT(*) FROM search_vector_chunks c "
"JOIN search_vector_embeddings e ON e.rowid = c.id "
"WHERE c.project_id = :project_id"
)
embeddings_result = await self.repository.execute_query(
embeddings_sql, {"project_id": project_id}
)
total_embeddings = embeddings_result.scalar() or 0
# Orphaned chunks (chunks without embeddings — indicates interrupted indexing)
if is_postgres:
orphan_sql = text(
"SELECT COUNT(*) FROM search_vector_chunks c "
"LEFT JOIN search_vector_embeddings e ON e.chunk_id = c.id "
"WHERE c.project_id = :project_id AND e.chunk_id IS NULL"
)
else:
orphan_sql = text(
"SELECT COUNT(*) FROM search_vector_chunks c "
"LEFT JOIN search_vector_embeddings e ON e.rowid = c.id "
"WHERE c.project_id = :project_id AND e.rowid IS NULL"
)
orphan_result = await self.repository.execute_query(orphan_sql, {"project_id": project_id})
orphaned_chunks = orphan_result.scalar() or 0
# --- Reindex recommendation logic (priority order) ---
reindex_recommended = False
reindex_reason = None
if total_indexed_entities > 0 and total_chunks == 0:
reindex_recommended = True
reindex_reason = "Embeddings have never been built — run: bm reindex --embeddings"
elif orphaned_chunks > 0:
reindex_recommended = True
reindex_reason = (
f"{orphaned_chunks} orphaned chunks found (interrupted indexing) "
"— run: bm reindex --embeddings"
)
elif total_indexed_entities > total_entities_with_chunks:
missing = total_indexed_entities - total_entities_with_chunks
reindex_recommended = True
reindex_reason = f"{missing} entities missing embeddings — run: bm reindex --embeddings"
return EmbeddingStatus(
semantic_search_enabled=True,
embedding_provider=provider,
embedding_model=model,
embedding_dimensions=dimensions,
total_indexed_entities=total_indexed_entities,
total_entities_with_chunks=total_entities_with_chunks,
total_chunks=total_chunks,
total_embeddings=total_embeddings,
orphaned_chunks=orphaned_chunks,
vector_tables_exist=True,
reindex_recommended=reindex_recommended,
reindex_reason=reindex_reason,
)
def get_system_status(self) -> SystemStatus:
"""Get system status information."""
import basic_memory
+31 -13
View File
@@ -13,7 +13,11 @@ from sqlalchemy import text
from basic_memory.models import Entity
from basic_memory.repository import EntityRepository
from basic_memory.repository.search_repository import SearchRepository, SearchIndexRow
from basic_memory.repository.search_repository import (
SearchIndexRow,
SearchRepository,
VectorSyncBatchResult,
)
from basic_memory.schemas.search import SearchQuery, SearchItemType, SearchRetrievalMode
from basic_memory.services import FileService
@@ -347,7 +351,7 @@ class SearchService:
entity: Entity,
content: str | None = None,
) -> None:
logger.info(
logger.debug(
f"[BackgroundTask] Starting search index for entity_id={entity.id} "
f"permalink={entity.permalink} project_id={entity.project_id}"
)
@@ -360,7 +364,7 @@ class SearchService:
entity, content
) if entity.is_markdown else await self.index_entity_file(entity)
logger.info(
logger.debug(
f"[BackgroundTask] Completed search index for entity_id={entity.id} "
f"permalink={entity.permalink}"
)
@@ -377,6 +381,17 @@ class SearchService:
"""Refresh vector chunks for one entity in repositories that support semantic indexing."""
await self.repository.sync_entity_vectors(entity_id)
async def sync_entity_vectors_batch(
self,
entity_ids: list[int],
progress_callback=None,
) -> VectorSyncBatchResult:
"""Refresh vector chunks for a batch of entities."""
return await self.repository.sync_entity_vectors_batch(
entity_ids,
progress_callback=progress_callback,
)
async def reindex_vectors(self, progress_callback=None) -> dict:
"""Rebuild vector embeddings for all entities.
@@ -387,17 +402,20 @@ class SearchService:
dict with stats: total_entities, embedded, skipped, errors
"""
entities = await self.entity_repository.find_all()
stats = {"total_entities": len(entities), "embedded": 0, "skipped": 0, "errors": 0}
entity_ids = [entity.id for entity in entities]
batch_result = await self.repository.sync_entity_vectors_batch(
entity_ids,
progress_callback=progress_callback,
)
stats = {
"total_entities": batch_result.entities_total,
"embedded": batch_result.entities_synced,
"skipped": 0,
"errors": batch_result.entities_failed,
}
for i, entity in enumerate(entities):
if progress_callback:
progress_callback(entity.id, i, len(entities))
try:
await self.repository.sync_entity_vectors(entity.id)
stats["embedded"] += 1
except Exception as e:
logger.warning(f"Failed to embed entity {entity.id} ({entity.permalink}): {e}")
stats["errors"] += 1
for failed_entity_id in batch_result.failed_entity_ids:
logger.warning(f"Failed to embed entity {failed_entity_id}")
return stats
+3 -3
View File
@@ -437,13 +437,13 @@ class SyncService:
elif project.last_scan_timestamp is not None:
# Incremental scan: only files modified since last scan
scan_type = "incremental"
logger.info(
logger.debug(
f"Running incremental scan for files modified since {project.last_scan_timestamp}"
)
file_paths_to_scan = await self._scan_directory_modified_since(
directory, project.last_scan_timestamp
)
logger.info(
logger.debug(
f"Incremental scan found {len(file_paths_to_scan)} potentially changed files"
)
@@ -705,7 +705,7 @@ class SyncService:
# If permalink changed, update the file
if permalink != entity_markdown.frontmatter.permalink:
logger.info(
logger.debug(
f"Updating permalink for path: {path}, old_permalink: {entity_markdown.frontmatter.permalink}, new_permalink: {permalink}"
)
+1 -1
View File
@@ -281,7 +281,7 @@ def setup_logging(
str(log_path),
level=log_level,
rotation="10 MB",
retention="10 days",
retention=5,
backtrace=True,
diagnose=True,
enqueue=False,
@@ -211,6 +211,34 @@ def test_edit_note_replace_section_fails_without_section(
assert "section parameter is required for replace_section operation" in result.output
def test_edit_note_append_creates_nonexistent_note_cli(
app, app_config, test_project, config_manager
):
"""append to a non-existent note via CLI should auto-create and include fileCreated."""
result = runner.invoke(
cli_app,
[
"tool",
"edit-note",
"cli-tests/auto-created-note",
"--operation",
"append",
"--content",
"# Auto Created\n\nCreated via CLI append.",
],
)
assert result.exit_code == 0, result.output
data = json.loads(result.stdout)
assert data["fileCreated"] is True
assert data["operation"] == "append"
assert data["title"] is not None
# Verify the note is readable
read_data = _read_note(data["permalink"])
assert "Auto Created" in read_data["content"]
def test_edit_note_json_format_contract(app, app_config, test_project, config_manager):
"""JSON output returns metadata keys required by contract."""
note = _write_note(
@@ -234,8 +262,16 @@ def test_edit_note_json_format_contract(app, app_config, test_project, config_ma
assert result.exit_code == 0, result.output
data = json.loads(result.stdout)
assert set(data.keys()) == {"title", "permalink", "file_path", "operation", "checksum"}
assert set(data.keys()) == {
"title",
"permalink",
"file_path",
"operation",
"checksum",
"fileCreated",
}
assert data["operation"] == "append"
assert data["fileCreated"] is False
assert data["title"] == "Edit JSON Note"
@@ -33,9 +33,8 @@ def test_project_info(app, app_config, test_project, config_manager):
print(f"STDOUT: {result.stdout}")
print(f"STDERR: {result.stderr}")
assert result.exit_code == 0
assert "Basic Memory Project Info" in result.stdout
assert "test-project" in result.stdout
assert "Statistics" in result.stdout
assert "Knowledge Graph" in result.stdout
def test_project_info_json(app, app_config, test_project, config_manager):
@@ -93,32 +93,33 @@ async def test_explicit_project_overrides_default(
@pytest.mark.asyncio
async def test_no_default_project_requires_project(mcp_server, app, test_project):
"""Test that tools require project parameter when no default_project is configured."""
async def test_no_config_default_falls_back_to_db(mcp_server, app, test_project):
"""When ConfigManager has no default_project, tools fall back to the database is_default flag."""
mock_config = BasicMemoryConfig(
default_project=None, # No default
default_project=None, # No config default
projects={test_project.name: test_project.path},
)
# test_project has is_default=True in the database, so write_note should
# resolve to it via the API fallback in resolve_project_parameter.
with patch.object(ConfigManager, "config", mock_config):
async with Client(mcp_server) as client:
with pytest.raises(Exception) as exc_info:
await client.call_tool(
"write_note",
{
"title": "Should Fail",
"directory": "test",
"content": "# Should Fail\n\nThis should fail because no project specified.",
},
)
error_message = str(exc_info.value)
assert (
"No project specified" in error_message
or "project parameter" in error_message.lower()
result = await client.call_tool(
"write_note",
{
"title": "DB Fallback Test",
"directory": "test",
"content": "# DB Fallback Test\n\nShould resolve to the database default project.",
},
)
assert len(result.content) == 1
response_text = result.content[0].text # pyright: ignore [reportAttributeAccessIssue]
assert f"project: {test_project.name}" in response_text
assert "# Created note" in response_text
@pytest.mark.asyncio
async def test_cli_constraint_overrides_default_project(
+4 -10
View File
@@ -105,11 +105,8 @@ async def test_delete_note_by_permalink(mcp_server, app, test_project):
},
)
# Should have no results
assert (
'"results": []' in search_result.content[0].text
or '"results":[]' in search_result.content[0].text
)
# Default text format returns "No results found" when empty
assert "No results found" in search_result.content[0].text
@pytest.mark.asyncio
@@ -387,11 +384,8 @@ async def test_delete_multiple_notes_sequentially(mcp_server, app, test_project)
},
)
# Should have no results
assert (
'"results": []' in search_result.content[0].text
or '"results":[]' in search_result.content[0].text
)
# Default text format returns "No results found" when empty
assert "No results found" in search_result.content[0].text
@pytest.mark.asyncio
+77 -4
View File
@@ -323,17 +323,18 @@ Current endpoints include user management."""
@pytest.mark.asyncio
async def test_edit_note_error_handling_note_not_found(mcp_server, app, test_project):
"""Test error handling when trying to edit a non-existent note."""
"""Test error handling when using find_replace on a non-existent note."""
async with Client(mcp_server) as client:
# Try to edit a note that doesn't exist
# find_replace on a non-existent note should still error
edit_result = await client.call_tool(
"edit_note",
{
"project": test_project.name,
"identifier": "Non-existent Note",
"operation": "append",
"content": "Some content to add",
"operation": "find_replace",
"content": "replacement",
"find_text": "old text",
},
)
@@ -345,6 +346,78 @@ async def test_edit_note_error_handling_note_not_found(mcp_server, app, test_pro
assert "search_notes(" in error_text
@pytest.mark.asyncio
async def test_edit_note_append_creates_nonexistent_note(mcp_server, app, test_project):
"""append to a non-existent note should auto-create it and make it readable."""
async with Client(mcp_server) as client:
# Append to a note that doesn't exist yet
edit_result = await client.call_tool(
"edit_note",
{
"project": test_project.name,
"identifier": "conversations/daily-log",
"operation": "append",
"content": "# Daily Log\n\nFirst entry for today.",
},
)
# Should return a "Created note" summary
assert len(edit_result.content) == 1
edit_text = edit_result.content[0].text
assert "Created note (append)" in edit_text
assert "fileCreated: true" in edit_text
# The note should now be readable
read_result = await client.call_tool(
"read_note",
{
"project": test_project.name,
"identifier": "conversations/daily-log",
},
)
content = read_result.content[0].text
assert "Daily Log" in content
assert "First entry for today." in content
@pytest.mark.asyncio
async def test_edit_note_prepend_creates_nonexistent_note(mcp_server, app, test_project):
"""prepend to a non-existent note should auto-create it and make it readable."""
async with Client(mcp_server) as client:
# Prepend to a note that doesn't exist yet
edit_result = await client.call_tool(
"edit_note",
{
"project": test_project.name,
"identifier": "notes/quick-thought",
"operation": "prepend",
"content": "# Quick Thought\n\nSomething important.",
},
)
# Should return a "Created note" summary
assert len(edit_result.content) == 1
edit_text = edit_result.content[0].text
assert "Created note (prepend)" in edit_text
assert "fileCreated: true" in edit_text
# The note should now be readable
read_result = await client.call_tool(
"read_note",
{
"project": test_project.name,
"identifier": "notes/quick-thought",
},
)
content = read_result.content[0].text
assert "Quick Thought" in content
assert "Something important." in content
@pytest.mark.asyncio
async def test_edit_note_error_handling_text_not_found(mcp_server, app, test_project):
"""Test error handling when find_text is not found in the note."""
+6 -5
View File
@@ -362,9 +362,9 @@ async def test_search_pagination(mcp_server, app, test_project):
)
result_text = search_result.content[0].text
# Should contain 5 results and pagination info
assert '"current_page":1' in result_text
assert '"page_size":5' in result_text
# Text format includes pagination info in footer
assert "page 1" in result_text
assert "page_size 5" in result_text
# Search page 2
search_result = await client.call_tool(
@@ -378,7 +378,7 @@ async def test_search_pagination(mcp_server, app, test_project):
)
result_text = search_result.content[0].text
assert '"current_page":2' in result_text
assert "page 2" in result_text
@pytest.mark.asyncio
@@ -407,8 +407,9 @@ async def test_search_no_results(mcp_server, app, test_project):
},
)
# Default text format returns "No results found" when empty
result_text = search_result.content[0].text
assert '"results": []' in result_text or '"results":[]' in result_text
assert "No results found" in result_text
@pytest.mark.asyncio
+48 -1
View File
@@ -88,7 +88,7 @@ async def test_write_note_update_existing(mcp_server, app, test_project):
assert "# Created note" in result1.content[0].text # pyright: ignore [reportAttributeAccessIssue]
# Update the same note
# Update the same note (explicit overwrite)
result2 = await client.call_tool(
"write_note",
{
@@ -97,6 +97,7 @@ async def test_write_note_update_existing(mcp_server, app, test_project):
"directory": "test",
"content": "# Update Test\n\nUpdated content with changes.",
"tags": "updated,modified",
"overwrite": True,
},
)
@@ -475,3 +476,49 @@ async def test_write_note_project_path_validation(mcp_server, app, test_project)
# Should successfully create without path validation errors
assert "# Created note" in response_text
assert "not allowed" not in response_text
@pytest.mark.asyncio
async def test_write_note_overwrite_guard_via_mcp_client(mcp_server, app, test_project):
"""End-to-end test: overwrite guard works through the MCP Client protocol."""
async with Client(mcp_server) as client:
# Create initial note
result1 = await client.call_tool(
"write_note",
{
"project": test_project.name,
"title": "MCP Guard Test",
"directory": "guard",
"content": "# MCP Guard Test\n\nOriginal content via MCP.",
},
)
assert "# Created note" in result1.content[0].text # pyright: ignore [reportAttributeAccessIssue]
# Second write without overwrite should be blocked
result2 = await client.call_tool(
"write_note",
{
"project": test_project.name,
"title": "MCP Guard Test",
"directory": "guard",
"content": "# MCP Guard Test\n\nReplacement content via MCP.",
},
)
response_text = result2.content[0].text # pyright: ignore [reportAttributeAccessIssue]
assert "# Error: Note already exists" in response_text
assert "edit_note" in response_text
# Overwrite with explicit flag should succeed
result3 = await client.call_tool(
"write_note",
{
"project": test_project.name,
"title": "MCP Guard Test",
"directory": "guard",
"content": "# MCP Guard Test\n\nReplacement content via MCP.",
"overwrite": True,
},
)
response_text3 = result3.content[0].text # pyright: ignore [reportAttributeAccessIssue]
assert "# Updated note" in response_text3
+5 -6
View File
@@ -3,7 +3,7 @@
These tests isolate specific problems with the search pipeline:
1. Similarity score compression cosine distances map to a narrow similarity band
2. Observation noise context-free observations match too broadly
3. RRF fusion behavior how FTS and vector scores interact
3. Hybrid fusion behavior how FTS and vector scores interact
4. Min-similarity threshold effectiveness
"""
@@ -235,17 +235,16 @@ async def test_observation_noise_vs_entity(sqlite_engine_factory, tmp_path):
print(f" {r.permalink}: {r.score:.4f}")
# --- Test: RRF fusion — vector vs hybrid comparison ---
# --- Test: Score-based fusion — vector vs hybrid comparison ---
@pytest.mark.asyncio
@pytest.mark.semantic
@pytest.mark.benchmark
async def test_rrf_fusion_preserves_strong_vector_match(sqlite_engine_factory, tmp_path):
async def test_score_fusion_preserves_strong_vector_match(sqlite_engine_factory, tmp_path):
"""When vector gives a strong match and FTS doesn't, hybrid should still surface it.
This is the core claim of issue #577 — that RRF dilutes strong vector scores.
Let's verify with a controlled corpus.
Score-based fusion preserves dominant signals instead of compressing them.
"""
skip_if_needed(DIAG_COMBO)
provider = _create_fastembed_provider()
@@ -296,7 +295,7 @@ async def test_rrf_fusion_preserves_strong_vector_match(sqlite_engine_factory, t
# Auth should still be in the top 3 in hybrid mode
assert hybrid_auth_rank <= 3, (
f"Hybrid pushed auth from vector rank 1 to hybrid rank {hybrid_auth_rank}. "
f"RRF dilution confirmed."
f"Fusion diluted strong vector match."
)
else:
print("\n WARNING: Auth found by vector but missing from hybrid results entirely!")
+3 -3
View File
@@ -82,10 +82,10 @@ async def test_postgres_vector_table_setup_and_query(postgres_engine_factory, tm
@pytest.mark.semantic
@pytest.mark.benchmark
async def test_postgres_hybrid_search(postgres_engine_factory, tmp_path):
"""Exercise the hybrid (RRF fusion) code path on Postgres.
"""Exercise the hybrid (score-based fusion) code path on Postgres.
This covers the full _search_hybrid path including both FTS and vector
retrieval with reciprocal rank fusion.
retrieval with score-based fusion.
"""
skip_if_needed(PG_FASTEMBED)
if postgres_engine_factory is None:
@@ -98,7 +98,7 @@ async def test_postgres_hybrid_search(postgres_engine_factory, tmp_path):
await seed_benchmark_notes(search_service, note_count=20)
# Hybrid search — exercises _search_hybrid RRF fusion
# Hybrid search — exercises _search_hybrid score-based fusion
results = await search_service.search(
SearchQuery(
text="database migration schema",
@@ -0,0 +1,120 @@
"""Tests for v2 graph intelligence and FCM routers."""
import pytest
from httpx import AsyncClient
@pytest.mark.asyncio
async def test_graph_lineage_contract(client: AsyncClient, v2_project_url: str):
response = await client.post(
f"{v2_project_url}/graph/lineage",
json={"start": "memory://specs/search"},
)
assert response.status_code == 200
data = response.json()
assert set(["root", "paths", "generated_at"]).issubset(data.keys())
assert data["root"]["id"] == "specs/search"
assert isinstance(data["paths"], list)
@pytest.mark.asyncio
async def test_graph_impact_contract(client: AsyncClient, v2_project_url: str):
response = await client.post(
f"{v2_project_url}/graph/impact",
json={"target": "memory://specs/search", "horizon": 2},
)
assert response.status_code == 200
data = response.json()
assert set(["target", "affected", "summary"]).issubset(data.keys())
assert data["summary"]["total_considered"] >= data["summary"]["total_returned"]
@pytest.mark.asyncio
async def test_graph_health_contract(client: AsyncClient, v2_project_url: str):
response = await client.get(
f"{v2_project_url}/graph/health",
params={"scope": "specs", "timeframe": "30d"},
)
assert response.status_code == 200
data = response.json()
assert set(["metrics", "issues", "computed_at"]).issubset(data.keys())
assert "orphan_rate" in data["metrics"]
@pytest.mark.asyncio
async def test_graph_reindex_schedules_task(
client: AsyncClient,
v2_project_url: str,
task_scheduler_spy: list[dict[str, object]],
):
response = await client.post(
f"{v2_project_url}/graph/reindex",
json={"mode": "full", "reason": "contract test"},
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "queued"
assert data["job_id"]
assert task_scheduler_spy
last = task_scheduler_spy[-1]
assert last["task_name"] == "reindex_graph_project"
assert last["payload"]["mode"] == "full"
assert last["payload"]["reason"] == "contract test"
@pytest.mark.asyncio
async def test_fcm_simulate_contract(client: AsyncClient, v2_project_url: str):
response = await client.post(
f"{v2_project_url}/fcm/simulate",
json={"actions": [{"node_id": "test-node", "delta": 0.2}]},
)
assert response.status_code == 200
data = response.json()
assert set(["baseline", "projected", "deltas", "stability", "confidence"]).issubset(data.keys())
assert data["stability"]["converged"] is True
@pytest.mark.asyncio
async def test_fcm_rank_actions_contract(client: AsyncClient, v2_project_url: str):
response = await client.post(
f"{v2_project_url}/fcm/rank-actions",
json={"goal": "reduce-regressions", "top_k": 2},
)
assert response.status_code == 200
data = response.json()
assert set(["goal", "recommendations"]).issubset(data.keys())
assert len(data["recommendations"]) <= 2
@pytest.mark.asyncio
async def test_fcm_import_contract(client: AsyncClient, v2_project_url: str):
response = await client.post(
f"{v2_project_url}/fcm/import",
json={"source": "/tmp/model.csv", "format": "csv_bundle_v1"},
)
assert response.status_code == 200
data = response.json()
assert set(["import_id", "nodes_loaded", "edges_loaded", "warnings", "errors"]).issubset(
data.keys()
)
@pytest.mark.asyncio
async def test_fcm_export_contract(client: AsyncClient, v2_project_url: str):
response = await client.post(
f"{v2_project_url}/fcm/export",
json={"format": "csv_bundle_v1", "selection": {"scope": "all"}},
)
assert response.status_code == 200
data = response.json()
assert set(["export_id", "format", "files", "node_count", "edge_count"]).issubset(data.keys())
assert len(data["files"]) == 2
+1 -3
View File
@@ -836,9 +836,7 @@ async def test_delete_directory_v2_nested_structure(client: AsyncClient, v2_proj
@pytest.mark.asyncio
async def test_entity_response_includes_user_tracking_fields(
client: AsyncClient, v2_project_url
):
async def test_entity_response_includes_user_tracking_fields(client: AsyncClient, v2_project_url):
"""EntityResponseV2 includes created_by and last_updated_by fields (null for local)."""
entity_data = {
"title": "UserTrackingTest",
+18 -2
View File
@@ -11,6 +11,21 @@ from basic_memory.schemas.project_info import ProjectItem, ProjectStatusResponse
from basic_memory.schemas.v2 import ProjectResolveResponse
@pytest.mark.asyncio
async def test_list_projects(client: AsyncClient, test_project: Project, v2_projects_url):
"""Test listing projects returns default_project from the database."""
response = await client.get(f"{v2_projects_url}/")
assert response.status_code == 200
data = response.json()
# default_project must be populated from the is_default flag in the database
assert data["default_project"] == test_project.name
project_names = [p["name"] for p in data["projects"]]
assert test_project.name in project_names
@pytest.mark.asyncio
async def test_get_project_by_id(client: AsyncClient, test_project: Project, v2_projects_url):
"""Test getting a project by its external_id UUID."""
@@ -361,9 +376,10 @@ async def test_legacy_v1_list_projects_endpoint(client: AsyncClient, test_projec
assert response.status_code == 200
data = response.json()
assert "projects" in data
assert "default_project" in data
# Verify the test project is in the list
# default_project must be populated, not null
assert data["default_project"] == test_project.name
project_names = [p["name"] for p in data["projects"]]
assert test_project.name in project_names
+275
View File
@@ -7,6 +7,7 @@ Note: EntityType uses BeforeValidator(to_snake_case) so "Person" becomes "person
in the database. All query params must use the stored (snake_case) form.
"""
from pathlib import Path
from textwrap import dedent
import pytest
@@ -14,6 +15,7 @@ from httpx import AsyncClient
from basic_memory.models import Project
from basic_memory.schemas.base import Entity as EntitySchema
from basic_memory.services.file_service import FileService
# --- Helpers ---
@@ -624,3 +626,276 @@ async def test_diff_with_schema_note(
assert isinstance(data["new_fields"], list)
assert isinstance(data["dropped_fields"], list)
assert isinstance(data["cardinality_changes"], list)
# --- File-based schema frontmatter tests ---
@pytest.mark.asyncio
async def test_validate_reads_schema_from_file_not_database(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
entity_service,
search_service,
file_service: FileService,
):
"""Validate uses schema frontmatter from the file, not stale database metadata.
Simulates the core bug from #634: user edits a schema file to change
validation mode from 'warn' to 'strict', but the file watcher hasn't
synced. The database still has 'warn', but validation should use 'strict'
from the file.
"""
# Create schema entity — DB gets validation=warn
schema_entity, _ = await entity_service.create_or_update_entity(
EntitySchema(
title="Editable Schema",
directory="schemas",
note_type="schema",
entity_metadata={
"entity": "editable_type",
"schema": {"name": "string", "role": "string"},
"settings": {"validation": "warn"},
},
content=dedent("""\
## Observations
- [note] Schema that will be edited on disk
"""),
)
)
await search_service.index_entity(schema_entity)
# Overwrite the file on disk with validation=strict
file_path = Path(file_service.base_path) / schema_entity.file_path
file_path.write_text(
dedent("""\
---
title: Editable Schema
permalink: schemas/editable-schema
type: schema
entity: editable_type
schema:
name: string
role: string
settings:
validation: strict
---
# Editable Schema
## Observations
- [note] Schema that will be edited on disk
""")
)
# Create a note missing "role" — strict mode should produce errors, not warnings
note_entity, _ = await entity_service.create_or_update_entity(
EntitySchema(
title="TestNote",
directory="notes",
note_type="editable_type",
content=dedent("""\
## Observations
- [name] Test Person
"""),
)
)
await search_service.index_entity(note_entity)
response = await client.post(
f"{v2_project_url}/schema/validate",
params={"identifier": note_entity.permalink},
)
assert response.status_code == 200
data = response.json()
assert data["total_notes"] == 1
result = data["results"][0]
# strict mode: missing required field is an error, not a warning
assert result["passed"] is False
assert any("role" in e for e in result["errors"])
@pytest.mark.asyncio
async def test_validate_falls_back_to_db_on_incomplete_frontmatter(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
entity_service,
search_service,
file_service: FileService,
):
"""Validate falls back to database metadata when file has incomplete frontmatter.
Simulates a mid-edit state where the user has removed the 'schema' key
from the file. The validator should use the last-known-good metadata
from the database rather than failing with a 500.
"""
schema_entity, _ = await entity_service.create_or_update_entity(
EntitySchema(
title="Incomplete Schema",
directory="schemas",
note_type="schema",
entity_metadata={
"entity": "incomplete_type",
"schema": {"name": "string"},
},
content=dedent("""\
## Observations
- [note] Schema that will have incomplete frontmatter
"""),
)
)
await search_service.index_entity(schema_entity)
# Overwrite file with frontmatter missing the 'schema' key
file_path = Path(file_service.base_path) / schema_entity.file_path
file_path.write_text(
dedent("""\
---
title: Incomplete Schema
permalink: schemas/incomplete-schema
type: schema
entity: incomplete_type
---
# Incomplete Schema
## Observations
- [note] Mid-edit state
""")
)
# Create a note to validate against this schema
note_entity, _ = await entity_service.create_or_update_entity(
EntitySchema(
title="FallbackNote",
directory="notes",
note_type="incomplete_type",
content=dedent("""\
## Observations
- [name] Test Fallback
"""),
)
)
await search_service.index_entity(note_entity)
response = await client.post(
f"{v2_project_url}/schema/validate",
params={"note_type": "incomplete_type"},
)
# Should not 500 — falls back to DB metadata and validates successfully
assert response.status_code == 200
data = response.json()
assert data["total_notes"] == 1
result = data["results"][0]
assert result["passed"] is True
@pytest.mark.asyncio
async def test_validate_falls_back_to_db_on_missing_file(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
entity_service,
search_service,
file_service: FileService,
):
"""Validate falls back to database metadata when schema file is missing.
Simulates a race condition where the file has been deleted but the
database still has the entity. The validator should use DB metadata
rather than failing entirely.
"""
schema_entity, _ = await entity_service.create_or_update_entity(
EntitySchema(
title="Missing File Schema",
directory="schemas",
note_type="schema",
entity_metadata={
"entity": "missing_file_type",
"schema": {"name": "string"},
},
content=dedent("""\
## Observations
- [note] Schema whose file will be deleted
"""),
)
)
await search_service.index_entity(schema_entity)
# Delete the schema file from disk
file_path = Path(file_service.base_path) / schema_entity.file_path
file_path.unlink()
# Create a note to validate
note_entity, _ = await entity_service.create_or_update_entity(
EntitySchema(
title="OrphanNote",
directory="notes",
note_type="missing_file_type",
content=dedent("""\
## Observations
- [name] Test Orphan
"""),
)
)
await search_service.index_entity(note_entity)
response = await client.post(
f"{v2_project_url}/schema/validate",
params={"note_type": "missing_file_type"},
)
# Should not 500 — falls back to DB metadata and validates
assert response.status_code == 200
data = response.json()
assert data["total_notes"] == 1
result = data["results"][0]
assert result["passed"] is True
@pytest.mark.asyncio
async def test_diff_falls_back_to_db_on_missing_file(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
entity_service,
search_service,
file_service: FileService,
):
"""Diff endpoint falls back to DB metadata when schema file is missing."""
schema_entity, _ = await entity_service.create_or_update_entity(
EntitySchema(
title="Diff Missing Schema",
directory="schemas",
note_type="schema",
entity_metadata={
"entity": "diff_missing_type",
"schema": {"name": "string", "role": "string"},
},
content=dedent("""\
## Observations
- [note] Schema for diff fallback test
"""),
)
)
await search_service.index_entity(schema_entity)
# Delete the schema file
file_path = Path(file_service.base_path) / schema_entity.file_path
file_path.unlink()
# Create person entities
await create_person_entities(entity_service, search_service)
response = await client.get(
f"{v2_project_url}/schema/diff/diff_missing_type",
)
# Should not 500 — falls back to DB metadata for schema resolution
assert response.status_code == 200
data = response.json()
assert data["note_type"] == "diff_missing_type"
+4 -3
View File
@@ -76,7 +76,7 @@ class TestTrack:
captured_target = None
def fake_thread(target, daemon):
def fake_thread(target):
nonlocal captured_target
captured_target = target
mock = MagicMock()
@@ -103,7 +103,7 @@ class TestTrack:
with patch("basic_memory.cli.analytics.urllib.request.urlopen", fake_urlopen):
with patch("basic_memory.cli.analytics.threading.Thread") as mock_thread:
# Capture the target function and call it directly
def run_target(target, daemon):
def run_target(target):
target() # Execute synchronously
return MagicMock()
@@ -113,6 +113,7 @@ class TestTrack:
assert captured_request is not None
assert captured_request.full_url == "https://analytics.example.com/api/send"
body = json.loads(captured_request.data)
assert body["type"] == "event"
assert body["payload"]["name"] == "cli-cloud-login-started"
assert body["payload"]["website"] == "test-site-id"
assert body["payload"]["hostname"] == "cli.basicmemory.com"
@@ -129,7 +130,7 @@ class TestTrack:
with patch("basic_memory.cli.analytics.urllib.request.urlopen", fake_urlopen):
with patch("basic_memory.cli.analytics.threading.Thread") as mock_thread:
def run_target(target, daemon):
def run_target(target):
target() # Should not raise
return MagicMock()
@@ -0,0 +1,184 @@
"""Tests for graph/FCM CLI tool JSON passthrough commands."""
import json
from unittest.mock import AsyncMock, patch
from typer.testing import CliRunner
from basic_memory.cli.main import app as cli_app
runner = CliRunner()
@patch(
"basic_memory.cli.commands.tool.mcp_graph_lineage",
new_callable=AsyncMock,
return_value={
"root": {"id": "specs/search"},
"paths": [],
"generated_at": "2026-03-05T00:00:00Z",
},
)
def test_graph_lineage_json_output(mock_tool):
result = runner.invoke(cli_app, ["tool", "graph-lineage", "memory://specs/search"])
assert result.exit_code == 0, f"CLI failed: {result.output}"
data = json.loads(result.output)
assert data["root"]["id"] == "specs/search"
assert mock_tool.call_args.kwargs["output_format"] == "json"
@patch(
"basic_memory.cli.commands.tool.mcp_graph_impact",
new_callable=AsyncMock,
return_value={
"target": {"id": "specs/search", "title": "specs/search"},
"affected": [],
"summary": {"total_considered": 0, "total_returned": 0},
},
)
def test_graph_impact_passthrough(mock_tool):
result = runner.invoke(
cli_app,
[
"tool",
"graph-impact",
"memory://specs/search",
"--horizon",
"3",
"--relation-filter",
"depends_on",
],
)
assert result.exit_code == 0, f"CLI failed: {result.output}"
assert mock_tool.call_args.kwargs["horizon"] == 3
assert mock_tool.call_args.kwargs["relation_filters"] == ["depends_on"]
assert mock_tool.call_args.kwargs["output_format"] == "json"
@patch(
"basic_memory.cli.commands.tool.mcp_graph_health",
new_callable=AsyncMock,
return_value={
"metrics": {
"orphan_rate": 0.0,
"stale_central_nodes": 0,
"overloaded_hubs": 0,
"contradiction_candidates": 0,
},
"issues": [],
"computed_at": "2026-03-05T00:00:00Z",
},
)
def test_graph_health_json_output(mock_tool):
result = runner.invoke(
cli_app,
["tool", "graph-health", "--scope", "specs", "--timeframe", "30d"],
)
assert result.exit_code == 0, f"CLI failed: {result.output}"
data = json.loads(result.output)
assert "metrics" in data
assert mock_tool.call_args.kwargs["scope"] == "specs"
assert mock_tool.call_args.kwargs["timeframe"] == "30d"
@patch(
"basic_memory.cli.commands.tool.mcp_fcm_simulate",
new_callable=AsyncMock,
return_value={
"baseline": [],
"projected": [],
"deltas": [],
"stability": {"converged": True, "iterations_used": 1, "residual": 0.0},
"confidence": 0.5,
},
)
def test_fcm_simulate_json_output(mock_tool):
result = runner.invoke(
cli_app,
[
"tool",
"fcm-simulate",
"--actions-json",
'[{"node_id":"n1","delta":0.2}]',
"--scenario-json",
'{"steps":8}',
],
)
assert result.exit_code == 0, f"CLI failed: {result.output}"
assert mock_tool.call_args.kwargs["actions"] == [{"node_id": "n1", "delta": 0.2}]
assert mock_tool.call_args.kwargs["scenario"] == {"steps": 8}
def test_fcm_simulate_invalid_actions_json():
result = runner.invoke(
cli_app,
["tool", "fcm-simulate", "--actions-json", '{"node_id":"n1","delta":0.2}'],
)
assert result.exit_code == 1
assert "expected a JSON array" in result.output
@patch(
"basic_memory.cli.commands.tool.mcp_fcm_rank_actions",
new_callable=AsyncMock,
return_value={"goal": {"node_id": "g1", "label": "g1"}, "recommendations": []},
)
def test_fcm_rank_actions_passthrough(mock_tool):
result = runner.invoke(
cli_app,
["tool", "fcm-rank-actions", "g1", "--constraints-json", '{"required_tags":["risk"]}'],
)
assert result.exit_code == 0, f"CLI failed: {result.output}"
assert mock_tool.call_args.kwargs["constraints"] == {"required_tags": ["risk"]}
assert mock_tool.call_args.kwargs["output_format"] == "json"
@patch(
"basic_memory.cli.commands.tool.mcp_fcm_import_model",
new_callable=AsyncMock,
return_value={
"import_id": "imp-1",
"nodes_loaded": 0,
"edges_loaded": 0,
"warnings": [],
"errors": [],
},
)
def test_fcm_import_model_json_output(mock_tool):
result = runner.invoke(
cli_app,
["tool", "fcm-import-model", "/tmp/model.csv", "--format", "csv_bundle_v1"],
)
assert result.exit_code == 0, f"CLI failed: {result.output}"
data = json.loads(result.output)
assert data["import_id"] == "imp-1"
assert mock_tool.call_args.kwargs["output_format"] == "json"
@patch(
"basic_memory.cli.commands.tool.mcp_fcm_export_model",
new_callable=AsyncMock,
return_value={
"export_id": "exp-1",
"format": "csv_bundle_v1",
"files": [],
"node_count": 0,
"edge_count": 0,
},
)
def test_fcm_export_model_json_output(mock_tool):
result = runner.invoke(
cli_app,
[
"tool",
"fcm-export-model",
"--format",
"csv_bundle_v1",
"--selection-json",
'{"scope":"all"}',
],
)
assert result.exit_code == 0, f"CLI failed: {result.output}"
data = json.loads(result.output)
assert data["export_id"] == "exp-1"
assert mock_tool.call_args.kwargs["selection"] == {"scope": "all"}
+15
View File
@@ -8,6 +8,7 @@ from typer.testing import CliRunner
from basic_memory.cli.app import app
import basic_memory
from basic_memory.cli.promo import (
_is_interactive_session,
maybe_show_cloud_promo,
maybe_show_init_line,
)
@@ -294,3 +295,17 @@ def test_cloud_promo_command_on_clears_opt_out(monkeypatch):
assert "Cloud promo messages enabled" in result.stdout
assert len(instances) == 1
assert instances[0].saved_config.cloud_promo_opt_out is False
# --- _is_interactive_session tests ---
def test_is_interactive_session_returns_false_when_streams_closed(monkeypatch):
"""isatty() raises ValueError on closed file descriptors (e.g., MCP shutdown)."""
class ClosedStream:
def isatty(self):
raise ValueError("I/O operation on closed file")
monkeypatch.setattr("sys.stdin", ClosedStream())
assert _is_interactive_session() is False
+146
View File
@@ -0,0 +1,146 @@
"""Tests for cloud status command."""
from __future__ import annotations
import time
import httpx
import pytest
from typer.testing import CliRunner
from basic_memory.cli.app import app
from basic_memory.cli.commands.cloud.api_client import CloudAPIError
# --- status command integration tests ---
class _FakeTokens:
"""Provides canned token data for CLIAuth stubs."""
@classmethod
def valid(cls) -> dict:
return {
"access_token": "fake-access-token",
"refresh_token": "rt_test",
"expires_at": int(time.time()) + 3600,
}
@classmethod
def expired(cls) -> dict:
return {
"access_token": "fake-access-token",
"refresh_token": "rt_test",
"expires_at": int(time.time()) - 3600,
}
def _patch_status_deps(monkeypatch, *, tokens=None, api_side_effect=None):
"""Patch ConfigManager and CLIAuth for the status command."""
class FakeConfig:
cloud_client_id = "cid"
cloud_domain = "https://auth.example.com"
cloud_host = "https://cloud.example.com"
cloud_api_key = "bmc_test123"
class FakeConfigManager:
config = FakeConfig()
def load_config(self):
return self.config
class FakeAuth:
def __init__(self, **_kwargs):
pass
def load_tokens(self):
return tokens
def is_token_valid(self, t):
return t.get("expires_at", 0) > time.time()
monkeypatch.setattr(
"basic_memory.cli.commands.cloud.core_commands.ConfigManager", FakeConfigManager
)
monkeypatch.setattr("basic_memory.cli.commands.cloud.core_commands.CLIAuth", FakeAuth)
monkeypatch.setattr(
"basic_memory.cli.commands.cloud.core_commands.get_cloud_config",
lambda: ("cid", "domain", "https://cloud.example.com"),
)
if api_side_effect is None:
# Default: cloud is reachable
async def _ok(*_a, **_kw):
return httpx.Response(200, json={"status": "ok"})
api_side_effect = _ok
monkeypatch.setattr(
"basic_memory.cli.commands.cloud.core_commands.make_api_request", api_side_effect
)
class TestStatusCommand:
def test_status_connected(self, monkeypatch):
_patch_status_deps(monkeypatch, tokens=_FakeTokens.valid())
runner = CliRunner()
result = runner.invoke(app, ["cloud", "status"])
assert result.exit_code == 0
assert "Cloud Status" in result.stdout
assert "cloud.example.com" in result.stdout
assert "token valid" in result.stdout
assert "Cloud connected" in result.stdout
def test_status_expired_token(self, monkeypatch):
_patch_status_deps(monkeypatch, tokens=_FakeTokens.expired())
runner = CliRunner()
result = runner.invoke(app, ["cloud", "status"])
assert result.exit_code == 0
assert "token expired" in result.stdout
def test_status_no_credentials(self, monkeypatch):
_patch_status_deps(monkeypatch, tokens=None)
# Also clear the API key so there are no credentials at all
class FakeConfig:
cloud_client_id = "cid"
cloud_domain = "https://auth.example.com"
cloud_host = "https://cloud.example.com"
cloud_api_key = ""
class FakeConfigManager:
config = FakeConfig()
def load_config(self):
return self.config
monkeypatch.setattr(
"basic_memory.cli.commands.cloud.core_commands.ConfigManager", FakeConfigManager
)
runner = CliRunner()
result = runner.invoke(app, ["cloud", "status"])
assert result.exit_code == 0
assert "No cloud credentials found" in result.stdout
@pytest.mark.parametrize(
"exc",
[
CloudAPIError("connection refused"),
Exception("network timeout"),
],
)
def test_status_cloud_not_connected(self, monkeypatch, exc):
async def _fail(*_a, **_kw):
raise exc
_patch_status_deps(monkeypatch, tokens=_FakeTokens.valid(), api_side_effect=_fail)
runner = CliRunner()
result = runner.invoke(app, ["cloud", "status"])
assert result.exit_code == 0
assert "Cloud not connected" in result.stdout
+418
View File
@@ -0,0 +1,418 @@
"""Tests for --json output across CLI commands.
Each test verifies:
- Exit code 0 (or 1 for strict mode)
- Output is valid json.loads()-able
- Expected keys present in the parsed data
"""
import json
from contextlib import asynccontextmanager
from datetime import datetime
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from typer.testing import CliRunner
from basic_memory.cli.main import app as cli_app
from basic_memory.mcp.clients.project import ProjectClient
from basic_memory.schemas.project_info import ProjectList
from basic_memory.schemas.sync_report import SyncReportResponse
# Importing registers subcommands on the shared app instance.
import basic_memory.cli.commands.project as project_cmd # noqa: F401
runner = CliRunner()
def _parse_json_output(output: str) -> dict:
"""Extract and parse the JSON object from CLI output.
The CliRunner may capture log lines before the JSON payload.
We find the first '{' and parse from there.
"""
start = output.index("{")
return json.loads(output[start:])
# ---------------------------------------------------------------------------
# Shared mock helpers
# ---------------------------------------------------------------------------
def _mock_config_manager():
"""Create a mock ConfigManager that avoids reading real config."""
mock_cm = MagicMock()
mock_cm.config = MagicMock()
mock_cm.default_project = "test-project"
mock_cm.get_project.return_value = ("test-project", "/tmp/test")
return mock_cm
SYNC_REPORT_WITH_CHANGES = SyncReportResponse(
new={"notes/new-file.md"},
modified={"notes/existing.md"},
deleted={"notes/old.md"},
moves={"notes/moved-from.md": "notes/moved-to.md"},
checksums={"notes/new-file.md": "abc12345", "notes/existing.md": "def67890"},
skipped_files=[],
total=4,
)
SYNC_REPORT_EMPTY = SyncReportResponse(
new=set(),
modified=set(),
deleted=set(),
moves={},
checksums={},
skipped_files=[],
total=0,
)
SYNC_REPORT_WITH_SKIPPED = SyncReportResponse(
new=set(),
modified=set(),
deleted=set(),
moves={},
checksums={},
skipped_files=[
{
"path": "bad/file.md",
"reason": "parse error",
"failure_count": 3,
"first_failed": datetime(2025, 6, 15, 12, 0, 0),
}
],
total=0,
)
VALIDATE_REPORT = {
"note_type": "person",
"total_notes": 2,
"total_entities": 2,
"valid_count": 1,
"warning_count": 1,
"error_count": 1,
"results": [
{
"note_identifier": "people/alice",
"schema_entity": "person",
"passed": True,
"warnings": [],
"errors": [],
},
{
"note_identifier": "people/bob",
"schema_entity": "person",
"passed": False,
"warnings": ["Missing optional field: role"],
"errors": ["Missing required field: name"],
},
],
}
INFER_REPORT = {
"note_type": "person",
"notes_analyzed": 5,
"field_frequencies": [
{"name": "name", "source": "observation", "count": 5, "total": 5, "percentage": 1.0},
{"name": "role", "source": "observation", "count": 3, "total": 5, "percentage": 0.6},
],
"suggested_schema": {"name": "string, full name", "role?": "string, job title"},
"suggested_required": ["name"],
"suggested_optional": ["role"],
"excluded": [],
}
DIFF_REPORT_WITH_DRIFT = {
"note_type": "person",
"schema_found": True,
"new_fields": [
{"name": "email", "source": "observation", "count": 3, "total": 5, "percentage": 0.6}
],
"dropped_fields": [
{"name": "phone", "source": "observation", "count": 0, "total": 5, "percentage": 0.0}
],
"cardinality_changes": ["role: single -> array"],
}
# ---------------------------------------------------------------------------
# Status --json
# ---------------------------------------------------------------------------
_MOCK_PROJECT_ITEM = MagicMock()
_MOCK_PROJECT_ITEM.name = "test-project"
_MOCK_PROJECT_ITEM.external_id = "11111111-1111-1111-1111-111111111111"
@patch("basic_memory.cli.commands.status.ConfigManager")
@patch("basic_memory.cli.commands.status.get_active_project", new_callable=AsyncMock)
@patch("basic_memory.cli.commands.status.get_client")
def test_status_json_outputs_sync_report(mock_get_client, mock_get_active, mock_config_cls):
"""bm status --json outputs a valid JSON sync report with changes."""
mock_config_cls.return_value = _mock_config_manager()
mock_get_active.return_value = _MOCK_PROJECT_ITEM
mock_project_client = AsyncMock()
mock_project_client.get_status.return_value = SYNC_REPORT_WITH_CHANGES
@asynccontextmanager
async def fake_get_client(project_name=None):
yield MagicMock()
mock_get_client.side_effect = fake_get_client
with patch.object(ProjectClient, "get_status", mock_project_client.get_status):
result = runner.invoke(cli_app, ["status", "--json"])
assert result.exit_code == 0, f"CLI failed: {result.output}"
data = _parse_json_output(result.output)
assert data["total"] == 4
assert "new" in data
assert "modified" in data
assert "deleted" in data
assert "moves" in data
@patch("basic_memory.cli.commands.status.ConfigManager")
@patch("basic_memory.cli.commands.status.get_active_project", new_callable=AsyncMock)
@patch("basic_memory.cli.commands.status.get_client")
def test_status_json_no_changes(mock_get_client, mock_get_active, mock_config_cls):
"""bm status --json with empty report outputs total: 0."""
mock_config_cls.return_value = _mock_config_manager()
mock_get_active.return_value = _MOCK_PROJECT_ITEM
mock_project_client = AsyncMock()
mock_project_client.get_status.return_value = SYNC_REPORT_EMPTY
@asynccontextmanager
async def fake_get_client(project_name=None):
yield MagicMock()
mock_get_client.side_effect = fake_get_client
with patch.object(ProjectClient, "get_status", mock_project_client.get_status):
result = runner.invoke(cli_app, ["status", "--json"])
assert result.exit_code == 0, f"CLI failed: {result.output}"
data = _parse_json_output(result.output)
assert data["total"] == 0
assert data["new"] == []
assert data["modified"] == []
@patch("basic_memory.cli.commands.status.ConfigManager")
@patch("basic_memory.cli.commands.status.get_active_project", new_callable=AsyncMock)
@patch("basic_memory.cli.commands.status.get_client")
def test_status_json_with_skipped_files(mock_get_client, mock_get_active, mock_config_cls):
"""bm status --json serializes skipped_files with datetime fields."""
mock_config_cls.return_value = _mock_config_manager()
mock_get_active.return_value = _MOCK_PROJECT_ITEM
mock_project_client = AsyncMock()
mock_project_client.get_status.return_value = SYNC_REPORT_WITH_SKIPPED
@asynccontextmanager
async def fake_get_client(project_name=None):
yield MagicMock()
mock_get_client.side_effect = fake_get_client
with patch.object(ProjectClient, "get_status", mock_project_client.get_status):
result = runner.invoke(cli_app, ["status", "--json"])
assert result.exit_code == 0, f"CLI failed: {result.output}"
data = _parse_json_output(result.output)
assert len(data["skipped_files"]) == 1
assert data["skipped_files"][0]["path"] == "bad/file.md"
# datetime should be serialized as ISO string via mode="json"
assert "2025-06-15" in data["skipped_files"][0]["first_failed"]
# ---------------------------------------------------------------------------
# Schema validate --json
# ---------------------------------------------------------------------------
@patch("basic_memory.cli.commands.schema.ConfigManager")
@patch(
"basic_memory.cli.commands.schema.mcp_schema_validate",
new_callable=AsyncMock,
return_value=VALIDATE_REPORT,
)
def test_schema_validate_json(mock_mcp, mock_config_cls):
"""bm schema validate person --json outputs the validation report as JSON."""
mock_config_cls.return_value = _mock_config_manager()
result = runner.invoke(cli_app, ["schema", "validate", "person", "--json"])
assert result.exit_code == 0, f"CLI failed: {result.output}"
data = _parse_json_output(result.output)
assert data["note_type"] == "person"
assert data["total_notes"] == 2
assert len(data["results"]) == 2
@patch("basic_memory.cli.commands.schema.ConfigManager")
@patch(
"basic_memory.cli.commands.schema.mcp_schema_validate",
new_callable=AsyncMock,
return_value={"error": "No schema found for type 'person'"},
)
def test_schema_validate_json_error(mock_mcp, mock_config_cls):
"""bm schema validate --json with error dict outputs the error as JSON."""
mock_config_cls.return_value = _mock_config_manager()
result = runner.invoke(cli_app, ["schema", "validate", "person", "--json"])
assert result.exit_code == 0, f"CLI failed: {result.output}"
data = _parse_json_output(result.output)
assert "error" in data
@patch("basic_memory.cli.commands.schema.ConfigManager")
@patch(
"basic_memory.cli.commands.schema.mcp_schema_validate",
new_callable=AsyncMock,
return_value=VALIDATE_REPORT,
)
def test_schema_validate_json_strict_exit(mock_mcp, mock_config_cls):
"""bm schema validate --json --strict exits 1 when errors present."""
mock_config_cls.return_value = _mock_config_manager()
result = runner.invoke(cli_app, ["schema", "validate", "person", "--json", "--strict"])
assert result.exit_code == 1
# JSON should still be valid in stdout
data = _parse_json_output(result.output)
assert data["error_count"] == 1
# ---------------------------------------------------------------------------
# Schema infer --json
# ---------------------------------------------------------------------------
@patch("basic_memory.cli.commands.schema.ConfigManager")
@patch(
"basic_memory.cli.commands.schema.mcp_schema_infer",
new_callable=AsyncMock,
return_value=INFER_REPORT,
)
def test_schema_infer_json(mock_mcp, mock_config_cls):
"""bm schema infer person --json outputs the inference report as JSON."""
mock_config_cls.return_value = _mock_config_manager()
result = runner.invoke(cli_app, ["schema", "infer", "person", "--json"])
assert result.exit_code == 0, f"CLI failed: {result.output}"
data = _parse_json_output(result.output)
assert data["note_type"] == "person"
assert data["notes_analyzed"] == 5
assert "suggested_schema" in data
# ---------------------------------------------------------------------------
# Schema diff --json
# ---------------------------------------------------------------------------
@patch("basic_memory.cli.commands.schema.ConfigManager")
@patch(
"basic_memory.cli.commands.schema.mcp_schema_diff",
new_callable=AsyncMock,
return_value=DIFF_REPORT_WITH_DRIFT,
)
def test_schema_diff_json(mock_mcp, mock_config_cls):
"""bm schema diff person --json outputs the drift report as JSON."""
mock_config_cls.return_value = _mock_config_manager()
result = runner.invoke(cli_app, ["schema", "diff", "person", "--json"])
assert result.exit_code == 0, f"CLI failed: {result.output}"
data = _parse_json_output(result.output)
assert data["note_type"] == "person"
assert len(data["new_fields"]) == 1
assert len(data["dropped_fields"]) == 1
# ---------------------------------------------------------------------------
# Project list --json
# ---------------------------------------------------------------------------
@pytest.fixture
def write_config(tmp_path, monkeypatch):
"""Write config.json under a temporary HOME and return the file path."""
def _write(config_data: dict):
from basic_memory import config as config_module
config_module._CONFIG_CACHE = None
config_dir = tmp_path / ".basic-memory"
config_dir.mkdir(parents=True, exist_ok=True)
config_file = config_dir / "config.json"
config_file.write_text(json.dumps(config_data, indent=2))
monkeypatch.setenv("HOME", str(tmp_path))
return config_file
return _write
@pytest.fixture
def mock_client(monkeypatch):
"""Mock get_client with a no-op async context manager."""
@asynccontextmanager
async def fake_get_client(workspace=None):
yield object()
monkeypatch.setattr(project_cmd, "get_client", fake_get_client)
def test_project_list_json_outputs_projects(write_config, mock_client, tmp_path, monkeypatch):
"""project list --json --local outputs structured JSON with project data."""
alpha_local = (tmp_path / "alpha-local").as_posix()
write_config(
{
"env": "dev",
"projects": {
"alpha": {"path": alpha_local, "mode": "local"},
},
"default_project": "alpha",
}
)
local_payload = {
"projects": [
{
"id": 1,
"external_id": "11111111-1111-1111-1111-111111111111",
"name": "alpha",
"path": alpha_local,
"is_default": True,
}
],
"default_project": "alpha",
}
async def fake_list_projects(self):
return ProjectList.model_validate(local_payload)
monkeypatch.setattr(ProjectClient, "list_projects", fake_list_projects)
result = runner.invoke(cli_app, ["project", "list", "--json", "--local"])
assert result.exit_code == 0, f"CLI failed: {result.output}"
data = _parse_json_output(result.output)
assert "projects" in data
assert len(data["projects"]) == 1
proj = data["projects"][0]
assert proj["name"] == "alpha"
assert proj["is_default"] is True
assert "local_path" in proj
assert "cli_route" in proj
assert "mcp_stdio" in proj
+13 -1
View File
@@ -188,7 +188,12 @@ async def engine_factory(
Uses parameterized db_backend fixture to run tests against both backends.
"""
from basic_memory.models.search import CREATE_SEARCH_INDEX
from basic_memory.models.search import (
CREATE_SEARCH_INDEX,
CREATE_SQLITE_SEARCH_VECTOR_CHUNKS,
CREATE_SQLITE_SEARCH_VECTOR_CHUNKS_PROJECT_ENTITY,
CREATE_SQLITE_SEARCH_VECTOR_CHUNKS_UNIQUE,
)
if db_backend == "postgres":
# Postgres mode using testcontainers
@@ -221,6 +226,8 @@ async def engine_factory(
CREATE_POSTGRES_SEARCH_INDEX_FTS,
CREATE_POSTGRES_SEARCH_INDEX_METADATA,
CREATE_POSTGRES_SEARCH_INDEX_PERMALINK,
CREATE_POSTGRES_SEARCH_VECTOR_CHUNKS_TABLE,
CREATE_POSTGRES_SEARCH_VECTOR_CHUNKS_INDEX,
)
# Drop and recreate all tables for test isolation
@@ -235,6 +242,8 @@ async def engine_factory(
await conn.execute(CREATE_POSTGRES_SEARCH_INDEX_FTS)
await conn.execute(CREATE_POSTGRES_SEARCH_INDEX_METADATA)
await conn.execute(CREATE_POSTGRES_SEARCH_INDEX_PERMALINK)
await conn.execute(CREATE_POSTGRES_SEARCH_VECTOR_CHUNKS_TABLE)
await conn.execute(CREATE_POSTGRES_SEARCH_VECTOR_CHUNKS_INDEX)
# Mark migrations as already applied for this test-created schema.
#
@@ -269,6 +278,9 @@ async def engine_factory(
async with engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
await conn.execute(CREATE_SEARCH_INDEX)
await conn.execute(CREATE_SQLITE_SEARCH_VECTOR_CHUNKS)
await conn.execute(CREATE_SQLITE_SEARCH_VECTOR_CHUNKS_PROJECT_ENTITY)
await conn.execute(CREATE_SQLITE_SEARCH_VECTOR_CHUNKS_UNIQUE)
# Yield after setup is complete
yield engine, session_maker
+184 -59
View File
@@ -5,7 +5,10 @@ in YAML frontmatter are automatically parsed as datetime.date objects by PyYAML,
but later code expects strings and calls .strip() on them, causing AttributeError.
"""
from textwrap import dedent
import pytest
from basic_memory.markdown.entity_parser import EntityParser
@@ -13,20 +16,22 @@ from basic_memory.markdown.entity_parser import EntityParser
def test_file_with_date(tmp_path):
"""Create a test file with date fields in frontmatter."""
test_file = tmp_path / "test_note.md"
content = """---
title: Test Note
date: 2025-10-24
created: 2025-10-24
tags:
- python
- testing
---
test_file.write_text(
dedent("""\
---
title: Test Note
date: 2025-10-24
created: 2025-10-24
tags:
- python
- testing
---
# Test Content
# Test Content
This file has date fields in frontmatter that PyYAML will parse as datetime.date objects.
"""
test_file.write_text(content)
This file has date fields in frontmatter that PyYAML will parse as datetime.date objects.
""")
)
return test_file
@@ -34,16 +39,18 @@ This file has date fields in frontmatter that PyYAML will parse as datetime.date
def test_file_with_date_in_tags(tmp_path):
"""Create a test file with a date value in tags (edge case)."""
test_file = tmp_path / "test_note_date_tags.md"
content = """---
title: Test Note with Date Tags
tags: 2025-10-24
---
test_file.write_text(
dedent("""\
---
title: Test Note with Date Tags
tags: 2025-10-24
---
# Test Content
# Test Content
This file has a date value as tags, which will be parsed as datetime.date.
"""
test_file.write_text(content)
This file has a date value as tags, which will be parsed as datetime.date.
""")
)
return test_file
@@ -51,19 +58,21 @@ This file has a date value as tags, which will be parsed as datetime.date.
def test_file_with_dates_in_tag_list(tmp_path):
"""Create a test file with dates in a tag list (edge case)."""
test_file = tmp_path / "test_note_dates_in_list.md"
content = """---
title: Test Note with Dates in Tags List
tags:
- valid-tag
- 2025-10-24
- another-tag
---
test_file.write_text(
dedent("""\
---
title: Test Note with Dates in Tags List
tags:
- valid-tag
- 2025-10-24
- another-tag
---
# Test Content
# Test Content
This file has date values mixed into tags list.
"""
test_file.write_text(content)
This file has date values mixed into tags list.
""")
)
return test_file
@@ -129,6 +138,87 @@ async def test_parse_file_with_dates_in_tag_list(test_file_with_dates_in_tag_lis
assert "2025-10-24" in tags
@pytest.mark.asyncio
async def test_parse_file_with_list_frontmatter_fields(tmp_path):
"""Test that list values in expected-string frontmatter fields are coerced to strings.
Reproduces basic-memory-cloud#376 where a markdown file has YAML list values
in frontmatter fields like 'title' or 'type' that downstream code expects
to be strings, causing 'list' object has no attribute 'strip'.
"""
test_file = tmp_path / "test_list_fields.md"
test_file.write_text(
dedent("""\
---
title:
- Week 2 Discussion Post
- Alternate Title
tags:
- coursework
- sie-571
type:
- note
- assignment
some_field:
- item1
- item2
---
# Content
Some body text.
""")
)
parser = EntityParser(tmp_path)
entity_markdown = await parser.parse_file(test_file)
# title must always be a string, even when YAML parses it as a list
title = entity_markdown.frontmatter.title
assert isinstance(title, str), f"Expected str, got {type(title)}"
assert "Week 2 Discussion Post" in title
# type must always be a string
note_type = entity_markdown.frontmatter.type
assert isinstance(note_type, str), f"Expected str, got {type(note_type)}"
# tags should still be a list (they're explicitly handled)
tags = entity_markdown.frontmatter.tags
assert isinstance(tags, list)
assert "coursework" in tags
# arbitrary list fields in metadata are preserved as lists
some_field = entity_markdown.frontmatter.metadata.get("some_field")
assert isinstance(some_field, list)
assert some_field == ["item1", "item2"]
# Verify title is safe for .strip() and .casefold() (the actual crash sites)
assert title.strip().casefold()
@pytest.mark.asyncio
async def test_parse_file_with_list_title_single_item(tmp_path):
"""Test that a single-item list title is coerced to a plain string."""
test_file = tmp_path / "test_single_list_title.md"
test_file.write_text(
dedent("""\
---
title:
- My Single Title
---
# Content
""")
)
parser = EntityParser(tmp_path)
entity_markdown = await parser.parse_file(test_file)
title = entity_markdown.frontmatter.title
assert isinstance(title, str)
assert title == "My Single Title"
@pytest.mark.asyncio
async def test_parse_file_with_various_yaml_types(tmp_path):
"""Test that various YAML types in frontmatter don't cause errors.
@@ -138,24 +228,26 @@ async def test_parse_file_with_various_yaml_types(tmp_path):
when code expects strings and calls .strip().
"""
test_file = tmp_path / "test_yaml_types.md"
content = """---
title: Test YAML Types
date: 2025-10-24
priority: 1
completed: true
tags:
- python
- testing
metadata:
author: Test User
version: 1.0
---
test_file.write_text(
dedent("""\
---
title: Test YAML Types
date: 2025-10-24
priority: 1
completed: true
tags:
- python
- testing
metadata:
author: Test User
version: 1.0
---
# Test Content
# Test Content
This file has various YAML types that need to be normalized.
"""
test_file.write_text(content)
This file has various YAML types that need to be normalized.
""")
)
parser = EntityParser(tmp_path)
entity_markdown = await parser.parse_file(test_file)
@@ -202,20 +294,19 @@ async def test_parse_file_with_datetime_objects(tmp_path):
with time components (as parsed by PyYAML), ensuring they're converted to ISO format strings.
"""
test_file = tmp_path / "test_datetime.md"
test_file.write_text(
dedent("""\
---
title: Test Datetime
created_at: 2025-10-24 14:30:00
updated_at: 2025-10-24T00:00:00
---
# YAML datetime strings that PyYAML will parse as datetime objects
# Format: YYYY-MM-DD HH:MM:SS or YYYY-MM-DDTHH:MM:SS
content = """---
title: Test Datetime
created_at: 2025-10-24 14:30:00
updated_at: 2025-10-24T00:00:00
---
# Test Content
# Test Content
This file has datetime values in frontmatter that PyYAML will parse as datetime objects.
"""
test_file.write_text(content)
This file has datetime values in frontmatter that PyYAML will parse as datetime objects.
""")
)
parser = EntityParser(tmp_path)
entity_markdown = await parser.parse_file(test_file)
@@ -234,3 +325,37 @@ This file has datetime values in frontmatter that PyYAML will parse as datetime
assert "2025-10-24" in updated_at and "00:00:00" in updated_at, (
f"Datetime at midnight should be normalized to ISO format, got: {updated_at}"
)
@pytest.mark.asyncio
async def test_parse_file_with_reserved_frontmatter_field_content(tmp_path):
"""Test that a 'content' field in frontmatter doesn't break parsing.
Reproduces basic-memory-cloud#375 where frontmatter containing a field named
'content' causes frontmatter.Post.__init__() to receive multiple values for
the 'content' positional argument.
"""
test_file = tmp_path / "topic-note-template.md"
test_file.write_text(
dedent("""\
---
title: Topic Note Template
content: Template for topic notes
handler: some-handler-value
---
# Template Body
Actual body content here.
""")
)
parser = EntityParser(tmp_path)
entity_markdown = await parser.parse_file(test_file)
assert entity_markdown.frontmatter.title == "Topic Note Template"
# The 'content' and 'handler' fields should be preserved in metadata
assert entity_markdown.frontmatter.metadata.get("content") == "Template for topic notes"
assert entity_markdown.frontmatter.metadata.get("handler") == "some-handler-value"
# The actual body content should be parsed correctly
assert "Template Body" in entity_markdown.content
+216
View File
@@ -0,0 +1,216 @@
"""Tests for graph and FCM typed clients."""
from unittest.mock import MagicMock
import pytest
from basic_memory.mcp.clients import FCMClient, GraphClient
class TestGraphClient:
def test_init(self):
mock_http = MagicMock()
client = GraphClient(mock_http, "project-123")
assert client.http_client is mock_http
assert client.project_id == "project-123"
assert client._base_path == "/v2/projects/project-123/graph"
@pytest.mark.asyncio
async def test_lineage(self, monkeypatch):
from basic_memory.mcp.clients import graph as graph_mod
from basic_memory.schemas.graph_intelligence import GraphLineageRequest
mock_response = MagicMock()
mock_response.json.return_value = {
"root": {"id": "specs/search", "title": "specs/search", "permalink": "specs/search"},
"paths": [],
"generated_at": "2026-03-05T00:00:00+00:00",
}
async def mock_call_post(client, url, **kwargs):
assert "/v2/projects/proj-123/graph/lineage" in url
return mock_response
monkeypatch.setattr(graph_mod, "call_post", mock_call_post)
client = GraphClient(MagicMock(), "proj-123")
result = await client.lineage(GraphLineageRequest(start="memory://specs/search"))
assert result.root.id == "specs/search"
@pytest.mark.asyncio
async def test_impact(self, monkeypatch):
from basic_memory.mcp.clients import graph as graph_mod
from basic_memory.schemas.graph_intelligence import GraphImpactRequest
mock_response = MagicMock()
mock_response.json.return_value = {
"target": {"id": "specs/search", "title": "specs/search"},
"affected": [],
"summary": {"total_considered": 0, "total_returned": 0},
}
async def mock_call_post(client, url, **kwargs):
assert "/v2/projects/proj-123/graph/impact" in url
return mock_response
monkeypatch.setattr(graph_mod, "call_post", mock_call_post)
client = GraphClient(MagicMock(), "proj-123")
result = await client.impact(GraphImpactRequest(target="memory://specs/search", horizon=2))
assert result.summary.total_returned == 0
@pytest.mark.asyncio
async def test_health(self, monkeypatch):
from basic_memory.mcp.clients import graph as graph_mod
mock_response = MagicMock()
mock_response.json.return_value = {
"metrics": {
"orphan_rate": 0.0,
"stale_central_nodes": 0,
"overloaded_hubs": 0,
"contradiction_candidates": 0,
},
"issues": [],
"computed_at": "2026-03-05T00:00:00+00:00",
}
async def mock_call_get(client, url, **kwargs):
assert "/v2/projects/proj-123/graph/health" in url
assert kwargs["params"]["scope"] == "specs"
return mock_response
monkeypatch.setattr(graph_mod, "call_get", mock_call_get)
client = GraphClient(MagicMock(), "proj-123")
result = await client.health(scope="specs", timeframe="30d")
assert result.metrics.orphan_rate == 0.0
@pytest.mark.asyncio
async def test_reindex(self, monkeypatch):
from basic_memory.mcp.clients import graph as graph_mod
from basic_memory.schemas.graph_intelligence import GraphReindexRequest
mock_response = MagicMock()
mock_response.json.return_value = {
"job_id": "job-123",
"status": "queued",
"scheduled_at": "2026-03-05T00:00:00+00:00",
}
async def mock_call_post(client, url, **kwargs):
assert "/v2/projects/proj-123/graph/reindex" in url
return mock_response
monkeypatch.setattr(graph_mod, "call_post", mock_call_post)
client = GraphClient(MagicMock(), "proj-123")
result = await client.reindex(GraphReindexRequest(mode="full"))
assert result.status == "queued"
class TestFCMClient:
def test_init(self):
mock_http = MagicMock()
client = FCMClient(mock_http, "project-123")
assert client.http_client is mock_http
assert client.project_id == "project-123"
assert client._base_path == "/v2/projects/project-123/fcm"
@pytest.mark.asyncio
async def test_simulate(self, monkeypatch):
from basic_memory.mcp.clients import fcm as fcm_mod
from basic_memory.schemas.graph_intelligence import FCMSimulateRequest
mock_response = MagicMock()
mock_response.json.return_value = {
"baseline": [{"node_id": "n1", "state": 0.0}],
"projected": [{"node_id": "n1", "state": 0.2}],
"deltas": [{"node_id": "n1", "delta": 0.2}],
"stability": {"converged": True, "iterations_used": 3, "residual": 0.0},
"confidence": 0.5,
"explanations": [],
"evidence_refs": [],
}
async def mock_call_post(client, url, **kwargs):
assert "/v2/projects/proj-123/fcm/simulate" in url
return mock_response
monkeypatch.setattr(fcm_mod, "call_post", mock_call_post)
request = FCMSimulateRequest(actions=[{"node_id": "n1", "delta": 0.2}])
result = await FCMClient(MagicMock(), "proj-123").simulate(request)
assert result.stability.converged is True
@pytest.mark.asyncio
async def test_rank_actions(self, monkeypatch):
from basic_memory.mcp.clients import fcm as fcm_mod
from basic_memory.schemas.graph_intelligence import FCMRankActionsRequest
mock_response = MagicMock()
mock_response.json.return_value = {
"goal": {"node_id": "g1", "label": "g1"},
"recommendations": [],
}
async def mock_call_post(client, url, **kwargs):
assert "/v2/projects/proj-123/fcm/rank-actions" in url
return mock_response
monkeypatch.setattr(fcm_mod, "call_post", mock_call_post)
request = FCMRankActionsRequest(goal="g1")
result = await FCMClient(MagicMock(), "proj-123").rank_actions(request)
assert result.goal.node_id == "g1"
@pytest.mark.asyncio
async def test_import_model(self, monkeypatch):
from basic_memory.mcp.clients import fcm as fcm_mod
from basic_memory.schemas.graph_intelligence import FCMImportRequest
mock_response = MagicMock()
mock_response.json.return_value = {
"import_id": "imp-1",
"nodes_loaded": 0,
"edges_loaded": 0,
"warnings": [],
"errors": [],
}
async def mock_call_post(client, url, **kwargs):
assert "/v2/projects/proj-123/fcm/import" in url
return mock_response
monkeypatch.setattr(fcm_mod, "call_post", mock_call_post)
request = FCMImportRequest(source="/tmp/model.csv")
result = await FCMClient(MagicMock(), "proj-123").import_model(request)
assert result.import_id == "imp-1"
@pytest.mark.asyncio
async def test_export_model(self, monkeypatch):
from basic_memory.mcp.clients import fcm as fcm_mod
from basic_memory.schemas.graph_intelligence import FCMExportRequest
mock_response = MagicMock()
mock_response.json.return_value = {
"export_id": "exp-1",
"format": "csv_bundle_v1",
"files": [
{"name": "nodes.csv", "path": "/tmp/nodes.csv"},
{"name": "edges.csv", "path": "/tmp/edges.csv"},
],
"node_count": 0,
"edge_count": 0,
}
async def mock_call_post(client, url, **kwargs):
assert "/v2/projects/proj-123/fcm/export" in url
return mock_response
monkeypatch.setattr(fcm_mod, "call_post", mock_call_post)
request = FCMExportRequest()
result = await FCMClient(MagicMock(), "proj-123").export_model(request)
assert result.format == "csv_bundle_v1"
@@ -143,6 +143,7 @@ async def test_write_note_update_preserves_yaml_format(app, project_config, test
directory="test",
content="Updated content",
tags=["updated", "new-tag", "format"],
overwrite=True,
)
# Should be an update, not a new creation
@@ -168,6 +168,7 @@ async def test_notes_with_similar_titles_maintain_separate_files(app, test_proje
title=title,
directory=folder,
content=f"# {title}\n\nUnique content for {title}",
overwrite=True,
)
permalink = None
+99 -1
View File
@@ -31,17 +31,34 @@ async def test_returns_none_when_no_default_and_no_project(config_manager, monke
config_manager.save_config(cfg)
monkeypatch.delenv("BASIC_MEMORY_MCP_PROJECT", raising=False)
# Prevent API fallback from returning a project via stale dependency overrides
async def _no_api_fallback():
return None
monkeypatch.setattr(
"basic_memory.mcp.project_context._resolve_default_project_from_api",
_no_api_fallback,
)
assert await resolve_project_parameter(project=None, allow_discovery=False) is None
@pytest.mark.asyncio
async def test_allows_discovery_when_enabled(config_manager):
async def test_allows_discovery_when_enabled(config_manager, monkeypatch):
from basic_memory.mcp.project_context import resolve_project_parameter
cfg = config_manager.load_config()
cfg.default_project = None
config_manager.save_config(cfg)
# Prevent API fallback from returning a project via stale dependency overrides
async def _no_api_fallback():
return None
monkeypatch.setattr(
"basic_memory.mcp.project_context._resolve_default_project_from_api",
_no_api_fallback,
)
assert await resolve_project_parameter(project=None, allow_discovery=True) is None
@@ -101,6 +118,15 @@ async def test_returns_none_when_no_default(config_manager, monkeypatch):
config_manager.save_config(cfg)
monkeypatch.delenv("BASIC_MEMORY_MCP_PROJECT", raising=False)
# Prevent API fallback from returning a project via stale dependency overrides
async def _no_api_fallback():
return None
monkeypatch.setattr(
"basic_memory.mcp.project_context._resolve_default_project_from_api",
_no_api_fallback,
)
assert await resolve_project_parameter(project=None) is None
@@ -475,3 +501,75 @@ class TestGetProjectClientRoutingOrder:
assert "resolve_workspace_parameter should not be called" not in error_msg
# Should not get a local ASGI routing error
assert "no project found" not in error_msg
@pytest.mark.asyncio
async def test_factory_mode_skips_workspace_resolution(self, config_manager, monkeypatch):
"""When a client factory is set (in-process cloud server), skip workspace resolution.
The cloud MCP server calls set_client_factory() so that get_client() routes
requests through TenantASGITransport. In this mode, workspace and tenant context
are already resolved by the transport layer. Attempting cloud workspace resolution
would call the production control-plane API and fail with 401.
"""
from contextlib import asynccontextmanager
from basic_memory.mcp import async_client
from basic_memory.mcp.project_context import get_project_client
from basic_memory.config import ProjectEntry, ProjectMode
config = config_manager.load_config()
config.projects["cloud-proj"] = ProjectEntry(
path=str(config_manager.config_dir.parent / "cloud-proj"),
mode=ProjectMode.CLOUD,
)
config_manager.save_config(config)
# Set up a factory (simulates what cloud MCP server does)
@asynccontextmanager
async def fake_factory():
from httpx import ASGITransport, AsyncClient
from basic_memory.api.app import app as fastapi_app
async with AsyncClient(
transport=ASGITransport(app=fastapi_app),
base_url="http://test",
) as client:
yield client
original_factory = async_client._client_factory
async_client.set_client_factory(fake_factory)
# Patch workspace resolution to fail if called — factory mode should skip it
async def fail_if_called(**kwargs): # pragma: no cover
raise AssertionError("resolve_workspace_parameter must not be called in factory mode")
monkeypatch.setattr(
"basic_memory.mcp.project_context.resolve_workspace_parameter",
fail_if_called,
)
# Patch get_cloud_control_plane_client to fail if called
@asynccontextmanager
async def fail_control_plane(): # pragma: no cover
raise AssertionError(
"get_cloud_control_plane_client must not be called in factory mode"
)
monkeypatch.setattr(
"basic_memory.mcp.async_client.get_cloud_control_plane_client",
fail_control_plane,
)
try:
# Will fail at project validation (no real project in DB), but proves
# workspace resolution and control-plane calls were skipped
with pytest.raises(Exception) as exc_info:
async with get_project_client(project="cloud-proj"):
pass
error_msg = str(exc_info.value).lower()
assert "resolve_workspace_parameter must not be called" not in error_msg
assert "get_cloud_control_plane_client must not be called" not in error_msg
finally:
# Restore original factory to avoid polluting other tests
async_client._client_factory = original_factory
+30 -27
View File
@@ -9,7 +9,6 @@ import pytest
from basic_memory.mcp.prompts.search import search_prompt
from basic_memory.mcp.prompts.continue_conversation import continue_conversation
from basic_memory.schemas.search import SearchResponse, SearchResult
# --- search_prompt ---
@@ -20,19 +19,20 @@ async def test_search_prompt_delegates_to_search_notes(monkeypatch):
"""Search prompt should call search_notes tool and wrap output."""
captured_kwargs = {}
fake_result = SearchResponse(
results=[
SearchResult(
type="entity",
title="Test Note",
permalink="test-note",
file_path="test-note.md",
score=0.95,
)
# Prompts use output_format="json", so mock returns a dict
fake_result = {
"results": [
{
"type": "entity",
"title": "Test Note",
"permalink": "test-note",
"file_path": "test-note.md",
"score": 0.95,
}
],
current_page=1,
page_size=10,
)
"current_page": 1,
"page_size": 10,
}
async def fake_search_notes(**kwargs):
captured_kwargs.update(kwargs)
@@ -45,6 +45,7 @@ async def test_search_prompt_delegates_to_search_notes(monkeypatch):
# Verify delegation
assert captured_kwargs["query"] == "my query"
assert captured_kwargs["after_date"] == "1w"
assert captured_kwargs["output_format"] == "json"
# Verify output wrapping
assert 'Search Results: "my query"' in out
@@ -55,7 +56,7 @@ async def test_search_prompt_delegates_to_search_notes(monkeypatch):
@pytest.mark.asyncio
async def test_search_prompt_handles_no_results(monkeypatch):
"""Search prompt should handle empty results gracefully."""
fake_result = SearchResponse(results=[], current_page=1, page_size=10)
fake_result = {"results": [], "current_page": 1, "page_size": 10}
async def fake_search_notes(**kwargs):
return fake_result
@@ -91,19 +92,20 @@ async def test_continue_conversation_delegates_to_search_notes(monkeypatch):
"""Continue conversation with topic should call search_notes."""
captured_kwargs = {}
fake_result = SearchResponse(
results=[
SearchResult(
type="entity",
title="Previous Discussion",
permalink="discussions/previous",
file_path="discussions/previous.md",
score=0.9,
)
# Prompts use output_format="json", so mock returns a dict
fake_result = {
"results": [
{
"type": "entity",
"title": "Previous Discussion",
"permalink": "discussions/previous",
"file_path": "discussions/previous.md",
"score": 0.9,
}
],
current_page=1,
page_size=10,
)
"current_page": 1,
"page_size": 10,
}
async def fake_search_notes(**kwargs):
captured_kwargs.update(kwargs)
@@ -117,6 +119,7 @@ async def test_continue_conversation_delegates_to_search_notes(monkeypatch):
assert captured_kwargs["query"] == "my topic"
assert captured_kwargs["after_date"] == "3d"
assert captured_kwargs["output_format"] == "json"
assert "'my topic'" in out
assert "Previous Discussion" in out
@@ -164,7 +167,7 @@ async def test_continue_conversation_no_topic_default_timeframe(monkeypatch):
@pytest.mark.asyncio
async def test_continue_conversation_no_results_for_topic(monkeypatch):
"""Continue conversation should show capture opportunity when no results found."""
fake_result = SearchResponse(results=[], current_page=1, page_size=10)
fake_result = {"results": [], "current_page": 1, "page_size": 10}
async def fake_search_notes(**kwargs):
return fake_result
+32 -11
View File
@@ -9,7 +9,7 @@ from basic_memory.mcp.tools import build_context
@pytest.mark.asyncio
async def test_get_basic_discussion_context(client, test_graph, test_project):
"""Test getting basic discussion context returns slimmed JSON dict."""
"""Test getting basic discussion context returns JSON dict with expected fields."""
result = await build_context(project=test_project.name, url="memory://test/root")
assert isinstance(result, dict)
@@ -19,25 +19,46 @@ async def test_get_basic_discussion_context(client, test_graph, test_project):
assert primary["permalink"] == f"{test_project.name}/test/root"
assert len(result["results"][0]["related_results"]) > 0
# Verify metadata — stripped fields should be absent
# Verify metadata fields
meta = result["metadata"]
assert meta["uri"] == f"{test_project.name}/test/root"
assert meta["depth"] == 1 # default depth
assert meta["timeframe"] is not None
assert meta["primary_count"] == 1
assert "generated_at" not in meta
assert "total_results" not in meta
# COMPAT(v0.18): generated_at and total_results restored for old clients
assert "generated_at" in meta
assert "total_results" in meta
# Verify entity-level stripped fields
assert "entity_id" not in primary
assert "created_at" not in primary
# Entity fields present
assert "entity_id" in primary
assert "created_at" in primary
# Verify observation-level stripped fields
# Verify observation-level fields
if result["results"][0]["observations"]:
obs = result["results"][0]["observations"][0]
assert "observation_id" not in obs
assert "entity_id" not in obs
assert "file_path" not in obs
assert "observation_id" in obs
assert "entity_id" in obs
assert "file_path" in obs
assert "created_at" in obs
assert "permalink" in obs
assert "category" in obs
assert "content" in obs
# Verify related_results item structure — entities have identifying fields
for related in result["results"][0]["related_results"]:
item_type = related["type"]
if item_type == "entity":
assert "title" in related
assert "file_path" in related
assert "created_at" in related
assert "entity_id" in related
elif item_type == "relation":
assert "relation_type" in related
assert "title" in related
assert "file_path" in related
assert "created_at" in related
assert "relation_id" in related
assert "entity_id" in related
@pytest.mark.asyncio
+33 -2
View File
@@ -35,7 +35,31 @@ EXPECTED_TOOL_SIGNATURES: dict[str, list[str]] = {
"expected_replacements",
"output_format",
],
"fcm_export_model": ["format", "selection", "project", "workspace", "output_format"],
"fcm_import_model": ["source", "format", "merge_mode", "project", "workspace", "output_format"],
"fcm_rank_actions": ["goal", "constraints", "top_k", "project", "workspace", "output_format"],
"fcm_simulate": ["actions", "scenario", "clamp_rules", "project", "workspace", "output_format"],
"fetch": ["id"],
"graph_health": ["scope", "timeframe", "project", "workspace", "output_format"],
"graph_impact": [
"target",
"horizon",
"relation_filters",
"include_reasons",
"project",
"workspace",
"output_format",
],
"graph_lineage": [
"start",
"goal",
"max_hops",
"relation_filters",
"project",
"workspace",
"output_format",
],
"graph_reindex": ["mode", "reason", "project", "workspace", "output_format"],
"list_directory": ["dir_name", "depth", "file_name_glob", "project", "workspace"],
"list_memory_projects": ["output_format", "workspace"],
"list_workspaces": ["output_format"],
@@ -73,7 +97,6 @@ EXPECTED_TOOL_SIGNATURES: dict[str, list[str]] = {
"schema_infer": ["note_type", "threshold", "project", "workspace", "output_format"],
"schema_validate": ["note_type", "identifier", "project", "workspace", "output_format"],
"search": ["query"],
"search_by_metadata": ["filters", "project", "workspace", "limit", "offset"],
"search_notes": [
"query",
"project",
@@ -100,6 +123,7 @@ EXPECTED_TOOL_SIGNATURES: dict[str, list[str]] = {
"tags",
"note_type",
"metadata",
"overwrite",
"output_format",
],
}
@@ -113,7 +137,15 @@ TOOL_FUNCTIONS: dict[str, object] = {
"delete_note": tools.delete_note,
"delete_project": tools.delete_project,
"edit_note": tools.edit_note,
"fcm_export_model": tools.fcm_export_model,
"fcm_import_model": tools.fcm_import_model,
"fcm_rank_actions": tools.fcm_rank_actions,
"fcm_simulate": tools.fcm_simulate,
"fetch": tools.fetch,
"graph_health": tools.graph_health,
"graph_impact": tools.graph_impact,
"graph_lineage": tools.graph_lineage,
"graph_reindex": tools.graph_reindex,
"list_directory": tools.list_directory,
"list_memory_projects": tools.list_memory_projects,
"list_workspaces": tools.list_workspaces,
@@ -126,7 +158,6 @@ TOOL_FUNCTIONS: dict[str, object] = {
"schema_infer": tools.schema_infer,
"schema_validate": tools.schema_validate,
"search": tools.search,
"search_by_metadata": tools.search_by_metadata,
"search_notes": tools.search_notes,
"view_note": tools.view_note,
"write_note": tools.write_note,
+160 -5
View File
@@ -120,19 +120,141 @@ async def test_edit_note_replace_section_operation(client, test_project):
@pytest.mark.asyncio
async def test_edit_note_nonexistent_note(client, test_project):
"""Test editing a note that doesn't exist - should return helpful guidance."""
async def test_edit_note_nonexistent_note_find_replace(client, test_project):
"""Test find_replace on a note that doesn't exist - should return helpful guidance."""
result = await edit_note(
project=test_project.name,
identifier="nonexistent/note",
operation="append",
content="Some content",
operation="find_replace",
content="replacement",
find_text="old text",
)
assert isinstance(result, str)
assert "# Edit Failed" in result
assert "search_notes" in result # Should suggest searching
assert "read_note" in result # Should suggest reading to verify
assert "append" in result # Should suggest using append/prepend instead
@pytest.mark.asyncio
async def test_edit_note_nonexistent_note_replace_section(client, test_project):
"""Test replace_section on a note that doesn't exist - should return helpful guidance."""
result = await edit_note(
project=test_project.name,
identifier="nonexistent/note",
operation="replace_section",
content="new section content",
section="## Missing Section",
)
assert isinstance(result, str)
assert "# Edit Failed" in result
assert "search_notes" in result # Should suggest searching
@pytest.mark.asyncio
async def test_edit_note_append_creates_note_if_not_found(client, test_project):
"""append to a non-existent note should create it automatically."""
result = await edit_note(
project=test_project.name,
identifier="auto-created-note",
operation="append",
content="# New Note\n\nCreated via append.",
)
assert isinstance(result, str)
assert "Created note (append)" in result
assert "fileCreated: true" in result
assert f"project: {test_project.name}" in result
@pytest.mark.asyncio
async def test_edit_note_prepend_creates_note_if_not_found(client, test_project):
"""prepend to a non-existent note should create it automatically."""
result = await edit_note(
project=test_project.name,
identifier="auto-created-prepend",
operation="prepend",
content="# Prepended Note\n\nCreated via prepend.",
)
assert isinstance(result, str)
assert "Created note (prepend)" in result
assert "fileCreated: true" in result
assert f"project: {test_project.name}" in result
@pytest.mark.asyncio
async def test_edit_note_append_creates_with_directory_from_identifier(client, test_project):
"""Identifier 'conversations/my-note' should create in conversations/ directory."""
result = await edit_note(
project=test_project.name,
identifier="conversations/my-note",
operation="append",
content="# My Note\n\nCreated in conversations directory.",
)
assert isinstance(result, str)
assert "Created note (append)" in result
assert "fileCreated: true" in result
assert "conversations/" in result
@pytest.mark.asyncio
async def test_edit_note_append_creates_at_root_when_no_directory(client, test_project):
"""Identifier 'my-note' (no slash) should create at project root."""
result = await edit_note(
project=test_project.name,
identifier="root-level-note",
operation="append",
content="# Root Note\n\nCreated at root.",
)
assert isinstance(result, str)
assert "Created note (append)" in result
assert "fileCreated: true" in result
@pytest.mark.asyncio
async def test_edit_note_append_creates_json_format(client, test_project):
"""JSON output should include fileCreated: true when note is auto-created."""
result = await edit_note(
project=test_project.name,
identifier="json-auto-create",
operation="append",
content="# JSON Test\n\nAuto-created.",
output_format="json",
)
assert isinstance(result, dict)
assert result["fileCreated"] is True
assert result["title"] is not None
assert result["operation"] == "append"
@pytest.mark.asyncio
async def test_edit_note_existing_note_json_includes_file_created_false(client, test_project):
"""JSON output for editing an existing note should include fileCreated: false."""
# Create the note first
await write_note(
project=test_project.name,
title="Existing JSON Note",
directory="test",
content="# Existing Note\nOriginal content.",
)
result = await edit_note(
project=test_project.name,
identifier="test/existing-json-note",
operation="append",
content="\nAppended content.",
output_format="json",
)
assert isinstance(result, dict)
assert result["fileCreated"] is False
assert result["title"] == "Existing JSON Note"
assert result["operation"] == "append"
@pytest.mark.asyncio
@@ -414,6 +536,39 @@ async def test_edit_note_find_replace_empty_find_text(client, test_project):
# Should contain helpful guidance about the error
@pytest.mark.asyncio
async def test_edit_note_append_with_null_optional_fields(client, test_project):
"""Regression test: MCP clients may send explicit null for unused optional fields.
When an MCP client sends find_text=None, section=None, expected_replacements=None
for an append operation, the tool should accept them without validation errors.
"""
# Create initial note
await write_note(
project=test_project.name,
title="Null Fields Test",
directory="test",
content="# Null Fields Test\nOriginal content.",
)
# Call edit_note with explicit None for all optional fields (simulates MCP null)
result = await edit_note(
project=test_project.name,
identifier="test/null-fields-test",
operation="append",
content="\nAppended content.",
find_text=None,
section=None,
expected_replacements=None,
)
assert isinstance(result, str)
assert "Edited note (append)" in result
assert f"project: {test_project.name}" in result
assert "file_path: test/Null Fields Test.md" in result
assert f"[Session: Using project '{test_project.name}']" in result
@pytest.mark.asyncio
async def test_edit_note_preserves_permalink_when_frontmatter_missing(client, test_project):
"""Test that editing a note preserves the permalink when frontmatter doesn't contain one.
+114
View File
@@ -0,0 +1,114 @@
"""Tests for graph intelligence MCP tools."""
import pytest
from basic_memory.mcp.tools import (
fcm_export_model,
fcm_import_model,
fcm_rank_actions,
fcm_simulate,
graph_health,
graph_impact,
graph_lineage,
graph_reindex,
)
@pytest.mark.asyncio
async def test_graph_lineage_json_and_text_modes(app, test_project):
json_result = await graph_lineage(
start="memory://specs/search",
project=test_project.name,
output_format="json",
)
assert isinstance(json_result, dict)
assert set(["root", "paths", "generated_at"]).issubset(json_result.keys())
text_result = await graph_lineage(
start="memory://specs/search",
project=test_project.name,
output_format="text",
)
assert isinstance(text_result, str)
assert "Graph Lineage" in text_result
@pytest.mark.asyncio
async def test_graph_impact_and_health(app, test_project):
impact = await graph_impact(
target="memory://specs/search",
horizon=2,
project=test_project.name,
output_format="json",
)
assert isinstance(impact, dict)
assert set(["target", "affected", "summary"]).issubset(impact.keys())
health = await graph_health(
scope="specs",
timeframe="30d",
project=test_project.name,
output_format="json",
)
assert isinstance(health, dict)
assert set(["metrics", "issues", "computed_at"]).issubset(health.keys())
@pytest.mark.asyncio
async def test_graph_reindex(app, test_project):
result = await graph_reindex(project=test_project.name, output_format="json")
assert isinstance(result, dict)
assert result["status"] == "queued"
@pytest.mark.asyncio
async def test_fcm_simulate_and_rank_actions(app, test_project):
simulation = await fcm_simulate(
actions=[{"node_id": "n1", "delta": 0.2}],
project=test_project.name,
output_format="json",
)
assert isinstance(simulation, dict)
assert set(["baseline", "projected", "deltas", "stability", "confidence"]).issubset(
simulation.keys()
)
ranking = await fcm_rank_actions(
goal="reduce-regressions",
top_k=2,
project=test_project.name,
output_format="json",
)
assert isinstance(ranking, dict)
assert set(["goal", "recommendations"]).issubset(ranking.keys())
assert len(ranking["recommendations"]) <= 2
@pytest.mark.asyncio
async def test_fcm_import_export_json_and_text(app, test_project):
imported = await fcm_import_model(
source="/tmp/model.csv",
format="csv_bundle_v1",
project=test_project.name,
output_format="json",
)
assert isinstance(imported, dict)
assert "import_id" in imported
exported_json = await fcm_export_model(
format="csv_bundle_v1",
selection={"scope": "all"},
project=test_project.name,
output_format="json",
)
assert isinstance(exported_json, dict)
assert set(["export_id", "files", "node_count", "edge_count"]).issubset(exported_json.keys())
exported_text = await fcm_export_model(
format="csv_bundle_v1",
selection={"scope": "all"},
project=test_project.name,
output_format="text",
)
assert isinstance(exported_text, str)
assert "FCM Export" in exported_text
+1
View File
@@ -38,6 +38,7 @@ async def test_write_note_text_and_json_modes(app, test_project):
directory="mode-tests",
content="# Mode Write Note\n\nupdated",
output_format="json",
overwrite=True,
)
assert isinstance(json_result, dict)
assert json_result["title"] == "Mode Write Note"
+6 -59
View File
@@ -128,69 +128,16 @@ async def test_recent_activity_type_invalid(client, test_project, test_graph):
@pytest.mark.asyncio
async def test_recent_activity_discovery_mode(client, test_project, test_graph, config_manager):
"""Test that recent_activity discovery mode works without project parameter."""
# Clear default_project to test discovery mode
cfg = config_manager.load_config()
cfg.default_project = None
config_manager.save_config(cfg)
# Test discovery mode (no project parameter)
async def test_recent_activity_uses_default_project(client, test_project, test_graph):
"""When no project parameter is given, recent_activity uses the default project."""
# Call without explicit project — should resolve to the default
result = await recent_activity()
assert result is not None
assert isinstance(result, str)
# Check that we get a formatted summary
assert "Recent Activity Summary" in result
assert "Most Active Project:" in result or "Other Active Projects:" in result
assert "Summary:" in result
assert "active projects" in result
# Should contain project discovery guidance
assert "Suggested project:" in result or "Multiple active projects" in result
assert "Session reminder:" in result
@pytest.mark.asyncio
async def test_recent_activity_discovery_mode_no_activity(client, test_project, config_manager):
"""If there is no activity in any project, discovery mode should say so."""
# Clear default_project to test discovery mode
cfg = config_manager.load_config()
cfg.default_project = None
config_manager.save_config(cfg)
result = await recent_activity()
assert "Recent Activity Summary" in result
assert "No recent activity found in any project." in result
@pytest.mark.asyncio
async def test_recent_activity_discovery_mode_multiple_active_projects(
app, client, test_project, tmp_path_factory, config_manager
):
"""Discovery mode should use the multi-project guidance when multiple projects have activity."""
# Clear default_project to test discovery mode
cfg = config_manager.load_config()
cfg.default_project = None
config_manager.save_config(cfg)
from basic_memory.mcp.tools import create_memory_project, write_note
second_root = tmp_path_factory.mktemp("second-project-home")
result = await create_memory_project(
project_name="second-project",
project_path=str(second_root),
set_default=False,
)
assert result.startswith("")
await write_note(project=test_project.name, title="One", directory="notes", content="one")
await write_note(project="second-project", title="Two", directory="notes", content="two")
out = await recent_activity()
assert "Recent Activity Summary" in out
assert "or would you prefer a different project" in out
# Should return project-specific output for the default project
assert "Recent Activity:" in result
assert "Activity Summary:" in result
def test_recent_activity_format_relative_time_and_truncate_helpers():
+116 -15
View File
@@ -12,7 +12,6 @@ import pytest
from basic_memory.mcp.tools.schema import schema_validate, schema_infer, schema_diff
from basic_memory.mcp.tools.write_note import write_note
from basic_memory.schemas.schema import ValidationReport, InferenceReport, DriftReport
# --- Helpers ---
@@ -82,8 +81,39 @@ async def test_schema_validate_by_type(app, test_project, sync_service):
project=test_project.name,
)
assert isinstance(result, ValidationReport)
assert result.total_notes >= 1
assert isinstance(result, str)
assert "Schema Validation: person" in result
assert "Notes: 1" in result
assert "**Alice**" in result
assert "valid" in result
@pytest.mark.asyncio
async def test_schema_validate_json_output(app, test_project, sync_service):
"""JSON output returns a dict with full structured data."""
project_path = Path(test_project.path)
_write_schema_file(project_path, "schemas/Person.md", PERSON_SCHEMA)
_write_schema_file(
project_path,
"people/Alice.md",
PERSON_NOTE.format(name="Alice", permalink="alice"),
)
await sync_service.sync(project_path)
result = await schema_validate(
note_type="person",
project=test_project.name,
output_format="json",
)
assert isinstance(result, dict)
assert result["total_notes"] == 1
assert result["valid_count"] == 1
assert len(result["results"]) == 1
assert result["results"][0]["note_identifier"] == "Alice"
assert result["results"][0]["passed"] is True
@pytest.mark.asyncio
@@ -105,8 +135,70 @@ async def test_schema_validate_by_identifier(app, test_project, sync_service):
project=test_project.name,
)
assert isinstance(result, ValidationReport)
assert result.total_notes >= 1
assert isinstance(result, str)
assert "**Alice**" in result
assert "valid" in result
@pytest.mark.asyncio
async def test_schema_validate_by_title(app, test_project, sync_service):
"""Validate a specific note by title (not permalink).
Regression test for issue #33: schema_validate(identifier="Note Title")
returned 0 notes because the router only searched by permalink.
"""
project_path = Path(test_project.path)
_write_schema_file(project_path, "schemas/Person.md", PERSON_SCHEMA)
_write_schema_file(
project_path,
"people/Alice.md",
PERSON_NOTE.format(name="Alice", permalink="alice"),
)
await sync_service.sync(project_path)
# Use the title "Alice" instead of the permalink "people/alice"
result = await schema_validate(
identifier="Alice",
project=test_project.name,
)
assert isinstance(result, str)
assert "**Alice**" in result
assert "Notes: 1" in result
assert "valid" in result
@pytest.mark.asyncio
async def test_schema_validate_identifier_no_schema_returns_guidance(
app, test_project, sync_service
):
"""When a note exists but no schema is defined, return guidance.
Regression test for issue #33: when validating a single note by identifier
and no schema exists, the tool should return guidance instead of an empty report.
"""
project_path = Path(test_project.path)
# Create a person note but no schema note
_write_schema_file(
project_path,
"people/Alice.md",
PERSON_NOTE.format(name="Alice", permalink="alice"),
)
await sync_service.sync(project_path)
result = await schema_validate(
identifier="Alice",
project=test_project.name,
)
# Should return guidance string about missing schema
assert isinstance(result, str)
assert "No Schema Found" in result
assert "person" in result
@pytest.mark.asyncio
@@ -128,9 +220,12 @@ async def test_schema_infer(app, test_project, sync_service):
project=test_project.name,
)
assert isinstance(result, InferenceReport)
assert result.note_type == "person"
assert result.notes_analyzed >= 3
assert isinstance(result, str)
assert "Schema Inference: person" in result
assert "Notes analyzed: 3" in result
assert "Field Frequencies" in result
assert "**name**" in result
assert "**role**" in result
@pytest.mark.asyncio
@@ -167,8 +262,10 @@ permalink: people/dave
project=test_project.name,
)
assert isinstance(result, DriftReport)
assert result.note_type == "person"
assert isinstance(result, str)
assert "Schema Drift: person" in result
# Dave has a "hobby" field not in the schema, so drift should be detected
assert "**hobby**" in result
# --- write_note metadata → schema workflow ---
@@ -214,8 +311,9 @@ async def test_write_note_metadata_creates_schema_note(app, test_project, sync_s
project=test_project.name,
)
assert isinstance(result, ValidationReport)
assert result.total_notes >= 2
assert isinstance(result, str)
assert "Schema Validation: person" in result
assert "valid" in result
@pytest.mark.asyncio
@@ -279,10 +377,13 @@ permalink: employees/{name.lower()}
project=test_project.name,
)
assert isinstance(result, ValidationReport)
assert result.total_notes == 2
assert isinstance(result, str)
assert "Schema Validation: employee" in result
assert "Notes: 2" in result
assert "Valid: 2" in result
# Both notes have name + department, schema requires name and optionally department
assert result.valid_count == 2
assert "**Alice**" in result
assert "**Bob**" in result
# --- Empty schema guard ---

Some files were not shown because too many files have changed in this diff Show More