Compare commits

..

39 Commits

Author SHA1 Message Date
phernandez 2d42ba16e3 fix(core): preserve sqlite rollback semantics on windows
Signed-off-by: phernandez <paul@basicmachines.co>
2026-06-14 17:25:12 -05:00
phernandez 82d42268c8 fix(core): flush note content deletes
Signed-off-by: phernandez <paul@basicmachines.co>
2026-06-14 17:02:54 -05:00
phernandez fef6f61ffa fix(core): preserve explicit-session write state
Signed-off-by: phernandez <paul@basicmachines.co>
2026-06-14 16:50:19 -05:00
phernandez a6ad20a0b7 fix(core): address explicit-session review fallout
Signed-off-by: phernandez <paul@basicmachines.co>
2026-06-14 16:31:46 -05:00
phernandez bb65be817b test(core): update repository session coverage
Signed-off-by: phernandez <paul@basicmachines.co>
2026-06-14 14:49:11 -05:00
phernandez dcb4fe927f refactor(core): use explicit sessions in core repositories
Signed-off-by: phernandez <paul@basicmachines.co>
2026-06-14 14:49:03 -05:00
Paul Hernandez b5f13d6903 Update README.md
add devin deep wiki badge for weekly updates

Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
2026-06-14 11:08:45 -05:00
Drew Cain 6d06c4a4e7 docs(ci): fix stale basicmachines.co post-release step
The marketing site moved to basicmemory.com (repo basicmachines-co/basicmemory.com,
now Astro + React) and no longer renders a hardcoded version number anywhere in
its UI — src/components/sections/hero.tsx has no version string. The post-release
instruction to bump the version there is obsolete. Replace it across the runbook
(release.md), AGENTS.md post-release tasks, and the release/beta justfile reminders
with the current reality: no version bump; optionally add a dated blog post under
src/content/blog/ for significant releases; skip for patches.

Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-13 14:53:52 -05:00
Drew Cain 16869867da fix(core): gate hybrid FTS relaxation with the service's eligibility rules
Addresses Codex review on #994: the hybrid FTS branch opted into
OR-relaxation for every query shape, but SearchService's relaxed FTS
path deliberately rejects short queries and numeric identifiers because
OR-relaxing them over-broadens — and in hybrid the relaxed FTS-only rows
normalize to 1.0 and can outrank the vector result the user wanted
(e.g. "SPEC 16", "root note 1", "New Feature").

relaxed_query_words now enforces the same eligibility as
SearchService._is_relaxed_fts_fallback_eligible: tokenize on
[A-Za-z0-9]+ and return None when there are fewer than three tokens or
any token is a pure digit (in addition to the existing quoted/boolean
guards). Both the SQLite and Postgres relaxed retries route through this
helper, so the hybrid path now relaxes exactly the query shapes the
service does.

Tests updated for the tightened eligibility (short + numeric queries no
longer relax); SQLite + Postgres relaxation suites green, full
repository/search-service/link-resolver suite green (442 passed), ty
clean.

Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-13 14:50:15 -05:00
Drew Cain 18da6194f9 fix(core): Postgres parity for FTS punctuation/relaxation (CI green)
CI Postgres shard caught two issues invisible to the local SQLite suite:

1. Postgres _prepare_single_term regression: the new edge-punctuation
   strip ran after special-character cleaning, so an all-special-char
   term ("()&!:") collapsed to empty and skipped the existing
   NOSPECIALCHARS:* guard, emitting a malformed ":*". Folded the strip
   into the word handlers so every guard survives, and added a
   single-word empty guard.

2. Backend-specific test assumptions. Four tests in
   test_search_repository.py (run under both backends via the
   search_repository fixture) asserted SQLite FTS5 syntax and
   SQLite-only strict-miss behavior. Postgres to_tsquery('english', ...)
   auto-strips stopwords, so "When did Melanie paint a sunrise?" already
   matches under strict AND. Made the four tests backend-aware via the
   existing is_postgres_backend() helper, and switched the relaxation
   integration test to a query with a word absent from the doc
   ("hiking") so the strict miss holds on both backends.

Reproduced and fixed against real Postgres (testcontainers): full
search test surface green on both backends (53 passed Postgres,
2968 SQLite), ruff + ty clean.

Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-13 14:50:15 -05:00
Drew Cain a6d0784335 fix(core): repair FTS half of hybrid search for natural-language queries
Hybrid search was silently running vector-only on natural-language
queries — the FTS branch contributed zero candidates. Two causes in the
SQLite (and parallel Postgres) FTS query preparation:

1. Sentence punctuation forced phrase matching. A question like
   "When did Melanie paint a sunrise?" reached FTS5 as the exact phrase
   '"When did Melanie paint a sunrise?"*', which matches no document.
   The FTS5 tokenizer ignores this punctuation in the index, so
   stripping it from word edges loses nothing — but leaving it disabled
   the entire FTS contribution. _prepare_single_term now strips
   ?!.,;: from word edges of multi-word queries (interior characters —
   hyphens, slashes in permalinks/paths — untouched).

2. No relaxation when strict all-terms-AND matched nothing. Questions
   rarely have every word in one document, so even after (1) the
   strict AND returned zero rows. The hybrid path now retries once with
   an OR-joined, stopword-filtered, content-term query when the strict
   query is empty. bm25/ts_rank still rank multi-term matches first, and
   fusion with the vector branch keeps relaxed lexical candidates from
   dominating precision.

The relaxation is gated behind a new allow_relaxed=False parameter on
SearchRepositoryBase.search; only _search_hybrid opts in. Strict FTS
behavior (search_type=text, title, permalink, link resolution) is
unchanged — the service layer keeps its own conservative fallback.
No config flag, default-safe.

Discovered via the benchmark harness: two different fusion algorithms
produced byte-identical rankings across 1,986 queries (impossible with
two live sources), and instrumentation confirmed fts=0 on 40/40
sampled LoCoMo queries.

Benchmark impact (corrected LoCoMo, 1,986 queries, same index,
retrieval metrics — every category improves, no regression):
  recall@5  0.745 -> 0.823  (+7.9)
  MRR       0.618 -> 0.718  (+10.0)
  headline  r5 0.734 -> 0.801, MRR 0.621 -> 0.706
Largest gains on open_domain (+0.10 r5) and adversarial (+0.12 r5);
smallest on temporal (+0.003 r5 / +0.02 MRR).

Tests: punctuation no longer phrase-quotes; relaxation builds the
expected OR query and respects boolean/quoted/short-query intent; the
hybrid opt-in surfaces a partial-overlap document while the default
strict path still returns empty. Parallel coverage for Postgres. Full
SQLite unit suite green (2968 passed); ty + ruff clean.

Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-13 14:50:15 -05:00
Drew Cain 232f469065 chore: update version to 0.22.1 for v0.22.1 release
Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-12 22:34:04 -05:00
Drew Cain 5a08cfd9ac docs(core): add v0.22.1 changelog entry
Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-12 22:33:34 -05:00
Drew Cain b997d858cd fix(sync): also exclude orphan DB projects absent from config (#949)
Address Codex review feedback. The previous path-only filter dropped an
implicit protection: get_project_mode() defaults projects missing from
config to CLOUD, so the old mode-based guard skipped stale DB rows that
had been removed from config. With a path-only check, an orphan row with
an absolute path would pass and background sync/watch could still mutate
a directory the user already removed from config (config is the source of
truth) if reconciliation was skipped or failed.

Introduce BasicMemoryConfig.is_locally_syncable(name, path), which
requires both config membership and an absolute path, and use it from
both the background sync selection and the watch cycle so the two paths
cannot diverge. Add direct unit tests for the helper plus an orphan-row
regression test for the watch selection.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-12 18:02:18 -05:00
Drew Cain 598965c389 test(sync): use OS-absolute paths in watch selection tests
The absolute-path sync guard from the previous commit now checks every
project's path, not just cloud-mode ones. Three existing watch-selection
tests hardcoded POSIX paths like "/tmp/alpha", which are absolute on
Linux/macOS but not on Windows (no drive letter), so the guard filtered
them out and the tests failed on windows-latest.

Build the project paths from the tmp_path fixture so they are absolute on
every platform. Production paths are unaffected: real local projects are
always resolved to OS-absolute paths at creation.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-12 18:02:18 -05:00
Drew Cain 8dd6451dfe fix(sync): skip projects without an absolute local path (#949)
A project entry in config.json with an empty path (e.g. `{"path": ""}`)
caused background sync and the watch service to adopt the process cwd as
the project root, injecting Basic Memory frontmatter into unrelated
markdown files.

The existing guards only skipped a project when get_project_mode()
returned CLOUD. But ProjectEntry.mode defaults to LOCAL, so an empty- or
relative-path entry without an explicit mode slipped through, and
Path("") resolves against the current working directory.

Gate local sync and watching on the path itself: any project whose path
is not absolute is excluded, regardless of mode. Legitimate local
projects are always resolved to absolute paths at creation, and cloud
projects with a real local bisync copy keep their absolute path and are
still synced/watched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-12 18:02:18 -05:00
Drew Cain d46c68806e test(mcp): add workspace to write_note expected signature contract
The signature contract test pins exact tool parameters; update it for
the new workspace parameter.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-12 12:19:56 -05:00
Drew Cain 1d9ff3c94f feat(mcp): add workspace parameter to write_note for parity with edit_note
write_note now accepts workspace= alongside project= and project_id=,
matching the parameter surface and docstring guidance already present in
edit_note. The _compose_workspace_project_route helper is added locally
(mirroring the edit_note pattern) so agents can route writes to same-named
projects across workspaces using workspace/project qualified syntax.

Closes #882

Other write-path tools that share the same gap (move_note, delete_note)
are noted here but left for a separate PR to keep this change focused.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-12 12:19:56 -05:00
phernandez 515b2c8365 fix(api): point fresh installs at project setup when resolve finds an empty projects table
On a brand-new config dir, model_post_init bootstraps a 'main' default in
config.json but the one-shot CLI path never runs the server-lifespan
reconciliation that would create its database row, so the first read fails
with a bare "Project not found: 'main'" — which reads as a broken install
rather than a missing first-run step. When resolution misses and the
projects table is empty, the 404 now names the setup command.

Follow-up to #974/#985/#987 (which repair the default during project add
but cannot help when the first action is a read). Refs #974.

Signed-off-by: phernandez <paul@basicmachines.co>
2026-06-12 10:27:16 -05:00
phernandez 32a1c208b3 fix(core): preserve an existing database default when repairing a missing config default
Follow-up to #985 (the #974 fix by @rudi193-cmd), addressing the codex P2
raised on #987: when config's default_project has no database row but the
database still holds a valid default of its own, promoting the just-added
project would silently steal that default. The repair now repoints config
at a surviving database default — matching synchronize_projects, which
treats the database default as authoritative — and promotes the added
project only when no usable database default exists (no default at all,
or one unknown to config, which set_default_project rejects and
reconciliation deletes).

Also removes an unused ProjectConfig import from the #985 regression test
(strict ruff failure; CI's --fix lint masked it).

Refs #974

Signed-off-by: phernandez <paul@basicmachines.co>
2026-06-12 09:44:21 -05:00
rudi193-cmd 25732b2fe2 fix: return promoted default state from project create API
When add_project promotes a new project because the configured default is missing from the database, return the persisted default flag instead of echoing the request flag.

Signed-off-by: rudi193-cmd <rudi193@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-12 09:03:36 -05:00
rudi193-cmd b92b0340d5 fix: promote first project when config default is missing from DB
Fresh CLI installs auto-create main in config but never sync it to the
projects table; the first project add now becomes default when the
configured default has no DB row. Fixes #974.

Signed-off-by: rudi193-cmd <rudi193@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-12 09:03:36 -05:00
phernandez 0247ef0ead fix(cli): defer FastAPI and app imports out of CLI startup
Every basic-memory CLI invocation paid roughly 2 seconds of module-import
cost before any work started, which blew the Claude Code plugin's
SessionStart hook budget on cold machines (#886). The cost came from
module-level imports that pulled the entire server stack into CLI startup:

- mcp/async_client.py imported FastAPI at module level, so every consumer
  of get_client() loaded FastAPI even for cloud-routed or help-only paths.
- mcp/clients/*.py imported call_* helpers from basic_memory.mcp.tools.utils,
  which executes the whole tools package __init__ — every MCP tool module
  plus fastmcp and the mcp SDK.
- mcp/project_context.py imported fastmcp.Context and ToolError eagerly.
- CLI command modules (tool, ci, schema) imported MCP tool functions at
  module level; db and the import_* commands pulled SQLAlchemy/Alembic and
  the markdown/file-service stack; status/doctor/orphans/command_utils
  imported ToolError (the mcp SDK) and basic_memory.db.
- schemas/base.py imported dateparser (~0.13s) for one helper function.

The fix only defers imports to the point of use (no behavior changes):
FastAPI now loads inside _resolve_local_asgi_database alongside the
existing lazy api.app import, so it is only paid when a request actually
routes through the in-process ASGI transport; the typed clients import
call_* per method; project_context uses PEP 563 annotations with Context
under TYPE_CHECKING; the CLI command modules import their heavy
dependencies inside the command bodies. Tests that patched the old
module-level aliases now patch the source modules instead.

Measured on a warm cache (python -X importtime / wall time):
- import basic_memory.cli.main: 1.92s -> 0.45s
- bm --help: 2.40s -> 0.52s
- bm tool search-notes --help: 2.40s -> 0.86s

A regression test asserts that importing the CLI entry module with full
command registration leaves fastapi, sqlalchemy, alembic, fastmcp, mcp,
basic_memory.api.app, basic_memory.db, basic_memory.markdown,
basic_memory.mcp.tools, and basic_memory.services out of sys.modules.

Fixes #886

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-06-12 09:03:08 -05:00
phernandez 253e240d68 fix(core): use (type, id) keys in vector search hydration to prevent id collisions
Root cause: entity, observation, and relation rows in search_index carry
ids from independent auto-increment sequences, so rows of different types
routinely share the same numeric id (guaranteed in young databases).
_search_vector_only parsed each vector hit's chunk_key (e.g. 'entity:4:0')
but discarded the type, and _fetch_search_index_rows_by_ids keyed its
result dict by bare row.id with no type discrimination. Whichever row the
database returned last clobbered the other in the dict; the clobbered hit
then hydrated against the wrong row or found None and was silently
dropped from results. The FTS-filter branch already guarded this with
(id, type) tuples, but the primary vector lookup path and the hybrid
fusion maps missed the same treatment.

Fix: introduce a SearchIndexKey = tuple[str, int] alias and key every map
in the vector/hybrid retrieval path by (type, id) — the similarity and
chunk maps in _search_vector_only, the _fetch_search_index_rows_by_ids
result, the FTS-filter allowed keys, and the rows/fts/vec/fused score
maps in _search_hybrid. The SQL stays unchanged; bare ids are deduped
before the IN query and rows are discriminated by row.type when building
dict keys.

Tests: end-to-end SQLite regression test indexes an entity row and a
relation row sharing id 7, syncs vectors for both, and asserts vector
search returns both rows (and that the entity survives a
search_item_types filter); a hybrid fusion unit test asserts an entity
and relation sharing id 1 stay distinct with single-source scores. Both
fail without the fix. Existing mocked vector tests updated for tuple keys.

Fixes #982

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-06-12 09:03:04 -05:00
phernandez b3bdd5914f fix(cli): write config.json atomically and isolate auto-update in mcp routing tests
Root cause of the test_mcp_sse_forces_local KeyError('FORCE_LOCAL') flake
(#940, Python 3.14 leg): the stdio variants of the mcp routing tests invoke
`bm mcp`, which starts a real background auto-update daemon thread before
mcp_server.run (the tests only mock run). That thread hits PyPI and then
rewrites config.json (auto_update_last_checked_at) with an in-place
Path.write_text, which truncates the file before writing. If that write lands
while a later test's CLI invocation is reading config.json (the app callback's
CliContainer.create()), load_config() sees empty/partial JSON and raises
SystemExit. CliRunner.invoke swallows it, the mocked mcp_server.run never
executes, and the test dies with KeyError on env_at_run['FORCE_LOCAL'] —
exactly the observed CI failure shape. Nothing is 3.14-specific; that leg
only shifted the timing.

The same torn write is user-visible in production: the MCP stdio server
re-reads config.json on mtime change and load_config() exits the process on
invalid JSON if it races a CLI save.

Fix: save_basic_memory_config writes a per-process/per-thread sibling temp
file and publishes it with os.replace, so readers always observe either the
old or the new complete document. The regression test injects an interrupted
write and asserts the published config stays untouched; it fails against the
old in-place write.

Test hardening: the stdio routing tests stub run_auto_update so no PyPI call
or config write leaks across tests, and all four transport tests now assert
result.exit_code == 0 so a future pre-run failure surfaces its real error
instead of a KeyError.

Refs #940

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-06-12 09:03:00 -05:00
phernandez 85c701b8c2 fix(sync): serialize in-memory SQLite sessions so concurrent rollbacks cannot destroy writes
Root cause of the test_sync_entity_circular_relations CI failure (#940,
len(entity_b.outgoing_relations) == 0): the in-memory SQLite URL
(sqlite+aiosqlite://) falls back to SQLAlchemy's StaticPool, which hands the
same DBAPI connection to every concurrently checked-out session. Concurrent
asyncio tasks therefore share one SQLite transaction scope. A rollback issued
through one session — scoped_session's exception handler, or the pool's
reset-on-return at connection checkin (~40 real ROLLBACKs per sync, measured)
— also rolls back any other task's executed-but-uncommitted statements.

During batch indexing, per-file tasks run concurrently (asyncio.gather bounded
by index_entity_max_concurrent). If a sibling session's checkin ROLLBACK lands
between one task's relation INSERT and its COMMIT, the relation row is silently
erased: no error is raised, the sync reports success. The final sync-level
resolve_relations() pass cannot heal this because it only re-queries rows with
to_id IS NULL — the destroyed INSERT leaves no row at all.

Fix: give MEMORY-type engines a single-connection AsyncAdaptedQueuePool
(pool_size=1, max_overflow=0). The lone connection keeps the in-memory
database alive for the engine's lifetime (the reason StaticPool was used),
while the blocking checkout serializes sessions at transaction granularity,
restoring the isolation the repositories assume. File-based SQLite and
Postgres engines are unchanged; production never uses MEMORY engines.

The regression test pins the invariant directly: a session that rolls back in
one task must never destroy another task's uncommitted writes. It fails
deterministically against StaticPool and passes with the serialized pool.

tests/repository/test_entity_repository.py held a scoped session open while
calling a repository method that opens its own session — tolerated on a shared
connection, a deadlock under a serialized pool — so the nested call moved out
of the session block.

Refs #940

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-06-12 09:03:00 -05:00
Paul Hernandez 1ad3a350ad fix(mcp): close out the #952 manual verification findings (#981)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 17:59:35 -05:00
Paul Hernandez a148e72f56 feat(plugins): manual-pages flow — manpage seed schema, flow docs, verification fixes (#971)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 16:27:30 -05:00
Drew Cain 33e741fd29 fix(mcp): resolve workspace display names and tenant ids in qualified project routes
The edit_note docstring (and write_note in PR #964) advertise that the
workspace segment of a "workspace/project" route may be a slug, name, or
tenant_id, but resolve_workspace_project_identifier() only matched against
WorkspaceInfo.slug. Cloud routes using a display name or tenant UUID failed
with "Workspace ... was not found" despite the documented contract.

Extend the first-segment matching (only the matching logic; the overall
resolution flow is unchanged) to honor, in priority order:

1. slug (casefold) — unchanged, checked first so working routes keep meaning
2. tenant_id — exact match on the opaque id
3. display name (casefold) — fails fast on collisions, listing candidate slugs

A name that collides with another workspace's slug resolves to the slug owner.
Unknown identifiers raise a not-found error naming the forms that were tried.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-11 15:28:19 -05:00
Drew Cain c9770375e9 feat(mcp): add title and tags annotations to all MCP tool decorators (issue #826 phase 1)
Adds human-readable `title` and categorization `tags` to all ~24 @mcp.tool
decorators across the MCP tools package. FastMCP 3.3.1 (pinned in pyproject.toml)
supports both fields natively. Tags used: notes, search, projects, cloud, schema,
navigation, canvas, ui.

Extends test_tool_contracts.py with an async test that asserts every registered
tool has a non-empty title and at least one tag to prevent future regressions.

output_schema is explicitly deferred as a follow-up (phase 2): it requires
per-tool design decisions about which tools reliably return structured JSON and
how to handle tools that return str|dict depending on output_format.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-11 15:21:25 -05:00
Drew Cain 0811c48252 fix(mcp): normalize note_types/entity_types/categories on direct call path and reject non-string list elements
Codex review of PR #962 identified two real issues:

1. CLI bypass: the BeforeValidator(parse_str_list) on note_types, entity_types, and
   categories only fires through MCP/Pydantic validation. The CLI path in
   cli/commands/tool.py calls search_notes() directly, so `bm tool search-notes
   --type note,task` arrived as note_types=["note,task"] and matched nothing.
   Fix: add in-body parse_str_list() normalization for all three params (mirroring
   the existing parse_tags() call for tags on the same code path).

2. Silent stringify: parse_str_list used str(raw) in the list branch, so [42] became
   ["42"] before Pydantic saw it, accepting invalid input as a no-result search
   instead of rejecting it. Fix: guard against non-string list elements and return
   the original value unchanged so Pydantic rejects it with a clear error.

Tests added: annotation-level split tests for note_types/entity_types/categories,
non-string-element rejection tests, async direct-call regression for note_types,
and unit-level parse_str_list non-string list tests in test_coerce.py.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-11 14:38:24 -05:00
Drew Cain 747e64e5e9 feat(mcp): comma-split note_types/entity_types/categories in search_notes (#930)
Add `parse_str_list` to utils.py — like `parse_tags` but without stripping
'#' — and wire it as the BeforeValidator for note_types, entity_types, and
categories in search_notes. This makes passing "note,task" or
'["note","task"]' work correctly instead of being wrapped as a single literal
value by coerce_list.

coerce_list is left unchanged; canvas and other callers that depend on its
wrap-single-string behaviour are unaffected.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-11 14:38:24 -05:00
Drew Cain be00df27c7 fix(api): allow setting default project when none is currently set
The v2 set_default_project_by_id handler fetched the current default
solely to echo it as old_project in the response, and raised 404
"No default project is currently set" when none existed. That guard
(marked # pragma: no cover, never tested) made the bootstrap/recovery
case impossible: with no default row in the DB, `bm project default
<name>` always failed -- yet that is exactly the command you reach for
when no default is set.

ProjectStatusResponse.old_project is already Optional, so the guard
served no schema requirement. Remove it and build old_project only
when a previous default exists, else pass None.

Add a regression test that clears is_default, sets a default via the
endpoint, and asserts 200, old_project is None, the new project is
default, and a follow-up read-back returns it.

Closes #975

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-11 14:17:35 -05:00
Drew Cain 3ce42de57e test(mcp): narrow read_note return before string assertions for ty
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-11 13:30:16 -05:00
Drew Cain a143072d35 fix(core): require line-anchored frontmatter fences in file_utils
has_frontmatter(), parse_frontmatter(), and remove_frontmatter() detected
frontmatter by substring/split (`content.startswith("---")` plus
`content.split("---", 2)`) rather than by line-anchored fences. A single-line
string that merely starts with `---` — e.g. `---\nstatus: active\n---\nBody`
where `\n` are literal backslash-n characters, a common CLI/agent input shape —
was misread as frontmatter: yaml parsed `\nstatus` as a key that got merged into
the note's YAML on disk, and the body was silently transformed.

Introduce a shared `_split_frontmatter()` helper that anchors both fences to
their own lines (`^---[ \t]*$`), tolerating leading blank lines so dedented
heredoc-style content still parses. All three public helpers now delegate to it,
preserving existing behavior for valid frontmatter (BOM stripping, empty -> {},
non-dict -> ParseError, and the existing ParseError messages callers assert on).

Closes #972

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-11 13:30:16 -05:00
Drew Cain 497a4e0a43 fix(cli): preload deferred rich/typer modules before in-place upgrade
brew upgrade / uv tool upgrade delete the running install's files while the old process is still alive. rich defers importing rich._emoji_codes until print time and typer defers typer.rich_utils until excepthook time, so printing the post-upgrade status message crashed the exiting process with ModuleNotFoundError (and the traceback renderer crashed too). Import both before launching the upgrade subprocess, while the files still exist.

Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-11 12:55:04 -05:00
Drew Cain ec5fac8d76 chore(ci): wait for the release PR merge to land before tagging
Addresses Codex review on PR #961: gh pr merge may not complete synchronously if merge gates exist (or while GitHub computes mergeability), so the recipes now fall back to queueing auto-merge and poll origin/main for the rebased bump commit (up to 5 minutes) before tagging, with manual finish instructions on timeout.

Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-11 01:35:34 -05:00
Drew Cain 05bfd0f04e chore(ci): remove user-specific absolute paths from release runbook
Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-11 01:35:34 -05:00
Drew Cain 338f357f55 chore(ci): route release recipes through PRs and refresh release runbook
main's ruleset now rejects direct pushes, which broke 'just release' mid-flight for v0.22.0. The release/beta recipes now land the version bump via a rebase-merged release PR and tag the rebased commit (found by commit subject). Adds a changelog-on-main pre-flight check, and updates AGENTS.md and the release runbook to match, including the docs.basicmemory.com steps which still described the old src/pages site.

Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-11 01:35:34 -05:00
204 changed files with 9748 additions and 6164 deletions
+2 -2
View File
@@ -6,14 +6,14 @@
},
"metadata": {
"description": "Official Basic Memory plugins from the canonical basic-memory repository",
"version": "0.22.0"
"version": "0.22.1"
},
"plugins": [
{
"name": "basic-memory",
"source": "./plugins/claude-code",
"description": "The bridge between Claude's working memory and Basic Memory's durable knowledge graph \u2014 session briefings, pre-compaction checkpoints, and capture reflexes",
"version": "0.22.0",
"version": "0.22.1",
"author": {
"name": "Basic Machines"
},
+50 -36
View File
@@ -28,7 +28,10 @@ You are an expert release manager for the Basic Memory project. When the user ru
#### Documentation Validation
1. **Changelog Check**
- CHANGELOG.md contains entry for target version
- CHANGELOG.md contains entry for target version **already landed on `main`**
(main only accepts changes via PR, so the changelog entry must go through
its own PR before running the release; the recipe pre-flight-checks for a
`## vX.Y.Z` heading)
- Entry includes all major features and fixes
- Breaking changes are documented
@@ -41,11 +44,15 @@ just release <version>
The justfile target handles:
- ✅ Version format validation
- ✅ Git status and branch checks
-Quality checks (`just check` - lint, format, type-check, tests)
-Changelog entry check (must already be on `main`)
- ✅ Quality checks (`just lint` + `just typecheck`)
- ✅ Version update across all consolidated manifests via `just set-version` (Python
package + Claude Code plugin/marketplaces + Codex plugin + Hermes + OpenClaw)
-Automatic commit with proper message
- ✅ Tag creation and pushing to GitHub
-Release PR: commits the bump on a `release/vX.Y.Z` branch, opens a PR
(`chore(core): release vX.Y.Z`), and rebase-merges it — the `main` ruleset
rejects direct pushes and the repo disallows merge commits
- ✅ Tags the rebased bump commit on `main` (found by commit subject, since
the rebase rewrites the SHA) and pushes the tag
- ✅ Release workflow trigger (automatic on tag push)
The GitHub Actions workflow (`.github/workflows/release.yml`) then:
@@ -89,7 +96,7 @@ After PyPI release is published, update the MCP registry:
2. **Publish to MCP Registry**
```bash
cd /Users/drew/code/basic-memory
# from the basic-memory repo root
mcp-publisher publish
```
@@ -109,43 +116,50 @@ After PyPI release is published, update the MCP registry:
#### Website Updates
**1. basicmachines.co** (`/Users/drew/code/basicmachines.co`)
- **Goal**: Update version number displayed on the homepage
- **Location**: Search for "Basic Memory v0." in the codebase to find version displays
- **What to update**:
- Hero section heading that shows "Basic Memory v{VERSION}"
- "What's New in v{VERSION}" section heading
- Feature highlights array (look for array of features with title/description)
- **Process**:
**1. basicmemory.com** (sibling `basicmemory.com` repo —
`basicmachines-co/basicmemory.com`, formerly `basicmachines.co`)
- **No version bump needed.** The marketing site is an Astro + React app and
carries **no hardcoded Basic Memory version number** anywhere in its UI
(`hero.tsx` and the rest of the site have no version string). The old
instruction to bump `src/components/sections/hero.tsx` is obsolete — that
file no longer holds a version. Release announcements are dated blog posts,
not an in-place edit.
- **Skip entirely for patch releases.**
- **Significant releases only — optional announcement post**:
1. Pull latest from GitHub: `git pull origin main`
2. Create release branch: `git checkout -b release/v{VERSION}`
3. Search codebase for current version number (e.g., "v0.16.1")
4. Update version numbers to new release version
5. Update feature highlights with 3-5 key features from this release (extract from CHANGELOG.md)
6. Commit changes: `git commit -m "chore: update to v{VERSION}"`
7. Push branch: `git push origin release/v{VERSION}`
- **Deploy**: Follow deployment process for basicmachines.co
3. Add a dated post under `src/content/blog/` modeled on an existing
release post (e.g. `basic-memory-v0-19-0-release.md`), summarizing 35
headline features from `CHANGELOG.md`
4. Commit (`git commit -s -m "..."`), push, and open a PR against
`basicmachines-co/basicmemory.com`
- **Deploy**: follow that repo's deployment process.
**2. docs.basicmemory.com** (`/Users/drew/code/docs.basicmemory.com`)
- **Goal**: Add new release notes section to the latest-releases page
- **File**: `src/pages/latest-releases.mdx`
**2. docs.basicmemory.com** (sibling `docs.basicmemory.com` repo)
- **Goal**: Add a What's New page for the release and bump the homepage badge
- **Site shape**: Nuxt/Docus content site. The changelog page
(`content/2.whats-new/*.changelog.md`) auto-fetches GitHub releases — no
manual changelog update needed. See that repo's CLAUDE.md "Version Bump
Checklist".
- **What to do**:
1. Pull latest from GitHub: `git pull origin main`
2. Create release branch: `git checkout -b release/v{VERSION}`
3. Read the existing file to understand the format and structure
4. Read `/Users/drew/code/basic-memory/CHANGELOG.md` to get release content
5. Add new release section **at the top** (after MDX imports, before other releases)
6. Follow the existing pattern:
- Heading: `## [v{VERSION}](github-link) — YYYY-MM-DD`
- Focus statement if applicable
- `<Info>` block with highlights (3-5 key items)
- Sections for Features, Bug Fixes, Breaking Changes, etc.
- Link to full changelog at the end
- Separator `---` between releases
7. Commit changes: `git commit -m "docs: add v{VERSION} release notes"`
8. Push branch: `git push origin release/v{VERSION}`
- **Source content**: Extract and format sections from CHANGELOG.md for this version
- **Deploy**: Follow deployment process for docs.basicmemory.com
3. Read `CHANGELOG.md` in the `basic-memory` repo to get release content
4. **New minor/major release**: add `content/2.whats-new/1.v{VERSION}.md`
modeled on the previous version page (frontmatter title/description,
headline feature first, then sections, then an Upgrading note) and
renumber the existing what's-new pages down one slot (URLs don't
change — Nuxt strips the numeric prefixes)
5. **Patch release**: append a short note to the current version's page
instead of creating a new one
6. Update the homepage version badge in `content/index.md` (the
`v0.XX →` button text and its `to: /whats-new/v{VERSION}` link)
7. If the release adds user-facing features, update the relevant guide
and reference pages (`content/3.cloud/`, `content/9.reference/`)
8. Commit: `git commit -s -m "docs: add v{VERSION} release notes"`
9. Push branch and open a PR; merge after the release is tagged
- **Deploy**: push to main auto-deploys to development; production requires
manual workflow dispatch via GitHub Actions
**4. Announce Release**
- Post to Discord community if significant changes
+11 -4
View File
@@ -300,7 +300,9 @@ See SPEC-16 for full context manager refactor details.
### Release Process
Releases are driven by `just release` / `just beta` — never by a bare `git tag`. The recipes bump version metadata, run pre-flight checks, commit, tag, and push. GitHub Actions then publishes to PyPI and updates the Homebrew formula.
Releases are driven by `just release` / `just beta` — never by a bare `git tag`. The recipes bump version metadata, run pre-flight checks, land the bump on `main` through a release PR, tag, and push the tag. GitHub Actions then publishes to PyPI and updates the Homebrew formula.
**Main requires PRs.** The `main` ruleset rejects direct pushes ("Changes must be made through a pull request") and the repo disallows merge commits, so the recipes push a `release/vX.Y.Z` branch, open a PR titled `chore(core): release vX.Y.Z`, rebase-merge it with `gh pr merge --rebase`, then tag the rebased bump commit on `main` (located by its commit subject, since rebasing rewrites the SHA) and push the tag. The CHANGELOG entry for the version must already be on `main` — land it via a normal PR before running the recipe (it pre-flight-checks for a `## vX.Y.Z` heading).
**Stable release:**
@@ -308,7 +310,7 @@ Releases are driven by `just release` / `just beta` — never by a bare `git tag
just release v0.21.3
```
The recipe runs `just lint` + `just typecheck`, then updates every release manifest through `scripts/update_versions.py`: `src/basic_memory/__init__.py`, `server.json`, the root Claude marketplace, the Claude Code plugin manifest and local marketplace, the Hermes `plugin.yaml`, and the OpenClaw `package.json`. It commits as `chore: update version to X.Y.Z for vX.Y.Z release`, creates the `vX.Y.Z` tag, and pushes both the commit and the tag to `origin/main`. After the tag lands, the `Release` workflow builds the Python package, publishes to PyPI, creates the GitHub release with auto-generated notes, publishes the OpenClaw npm package, and updates the Homebrew formula. The recipe finishes by printing the post-release tasks the workflow doesn't cover.
The recipe runs `just lint` + `just typecheck`, then updates every release manifest through `scripts/update_versions.py`: `src/basic_memory/__init__.py`, `server.json`, the root Claude marketplace, the Claude Code plugin manifest and local marketplace, the Hermes `plugin.yaml`, and the OpenClaw `package.json`. It commits as `chore: update version to X.Y.Z for vX.Y.Z release` on a `release/vX.Y.Z` branch, lands it on `main` via a rebase-merged PR, then tags the rebased commit and pushes the tag. After the tag lands, the `Release` workflow builds the Python package, publishes to PyPI, creates the GitHub release with auto-generated notes, publishes the OpenClaw npm package, and updates the Homebrew formula. The recipe finishes by printing the post-release tasks the workflow doesn't cover.
**Beta release:** `just beta v0.21.3b1` — same flow with a beta-suffixed tag. PyPI consumers install with `pip install basic-memory --pre`.
@@ -319,8 +321,13 @@ The recipe runs `just lint` + `just typecheck`, then updates every release manif
**Do not tag releases by hand.** A bare `git tag vX.Y.Z` skips the in-code version bump. Package metadata is still correct (uv-dynamic-versioning derives it from the git tag) but `basic-memory --version` reports the previous release, which is what happened with v0.21.2 → v0.21.3.
**Post-release tasks** the recipe surfaces but doesn't run:
- `docs.basicmemory.com` — add notes to `src/pages/latest-releases.mdx`
- `basicmachines.co`bump version in `src/components/sections/hero.tsx`
- `docs.basicmemory.com` — add a What's New page under `content/2.whats-new/` and bump the version badge in `content/index.md` (the changelog page auto-fetches GitHub releases; see that repo's CLAUDE.md version-bump checklist)
- `basicmemory.com`the marketing site (Astro + React, repo
`basicmachines-co/basicmemory.com`, formerly `basicmachines.co`) carries **no
hardcoded version number** in its UI, so there is nothing to bump. For a
significant release, optionally add a dated announcement post under
`src/content/blog/` (model it on an existing `basic-memory-vX-Y-Z-release.md`).
Skip entirely for routine patch releases.
- MCP Registry — `mcp-publisher publish` from the repo root
See `.claude/commands/release/release.md` (and `beta.md`, `release-check.md`, `changelog.md` alongside it) for the full release + post-release runbook, including the slash commands.
+44
View File
@@ -1,5 +1,49 @@
# CHANGELOG
## v0.22.1 (2026-06-12)
Follow-up patch to v0.22.0. Fixes project and default-project resolution on
fresh installs, MCP workspace routing, sync project selection, and CLI startup
latency, plus a few MCP parity additions.
### Features
- Added a `workspace` parameter to `write_note` for parity with `edit_note`.
- **#826**: Added `title` and `tags` annotations to all MCP tool decorators
(phase 1).
- **#930**: `search_notes` now comma-splits `note_types`, `entity_types`, and
`categories`.
- **#971**: Added the manual-pages flow — manpage seed schema, flow docs, and
verification fixes.
### Bug Fixes
- Fresh installs no longer fail when the projects table is empty: resolve now
points them at project setup, the first project is promoted to default when
the config default is missing from the database, the promoted default state
is returned from the project-create API, and a default can be set when none
is currently set. An existing database default is preserved when repairing a
missing config default.
- **#949**: Sync skips projects without an absolute local path and excludes
orphan DB projects that are absent from config.
- **#952 / #981**: Resolved workspace display names and tenant ids in qualified
project routes, closing out the manual verification findings.
- `note_types`/`entity_types`/`categories` are normalized on the direct-call
path, with non-string list elements rejected.
- Vector-search hydration keys on `(type, id)` to prevent id collisions.
- `file_utils` requires line-anchored frontmatter fences.
- CLI startup is faster: FastAPI and app imports are deferred out of CLI
startup, and rich/typer modules are preloaded before an in-place upgrade.
- `config.json` is written atomically.
- In-memory SQLite sessions are serialized so concurrent rollbacks cannot
destroy writes.
### Maintenance
- Release recipes route through PRs and wait for the release PR merge to land
before tagging; the release runbook is refreshed and stripped of
user-specific absolute paths.
## v0.22.0 (2026-06-11)
Team-safe cloud sync. New additive `bm cloud push` and `bm cloud pull`
+2 -1
View File
@@ -6,6 +6,7 @@
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
![](https://badge.mcpx.dev?type=server 'MCP Server')
![](https://badge.mcpx.dev?type=dev 'MCP Dev')
[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/basicmachines-co/basic-memory)
## Skip the install — try Basic Memory in the cloud
@@ -590,7 +591,7 @@ retention).
| `BASIC_MEMORY_IMPORT_UPLOAD_MAX_BYTES` | `104857600` | Max uploaded import size |
```bash
BASIC_MEMORY_LOG_LEVEL=DEBUG basic-memory sync
BASIC_MEMORY_LOG_LEVEL=DEBUG basic-memory reindex
tail -f ~/.basic-memory/basic-memory.log
```
+3 -3
View File
@@ -111,7 +111,7 @@ You can run Basic Memory CLI commands inside the container using `docker exec`:
docker exec basic-memory-server basic-memory status
# Sync files
docker exec basic-memory-server basic-memory sync
docker exec basic-memory-server basic-memory reindex
# Show help
docker exec basic-memory-server basic-memory --help
@@ -137,7 +137,7 @@ When using Docker volumes, you'll need to configure projects to point to your mo
3. **Sync the new project:**
```bash
docker exec basic-memory-server basic-memory sync
docker exec basic-memory-server basic-memory reindex
```
### Example: Setting up an Obsidian Vault
@@ -157,7 +157,7 @@ docker exec basic-memory-server basic-memory project create obsidian /app/data
docker exec basic-memory-server basic-memory project set-default obsidian
# Sync to index all files
docker exec basic-memory-server basic-memory sync
docker exec basic-memory-server basic-memory reindex
```
### Environment Variables
+2 -2
View File
@@ -184,8 +184,8 @@ finance/ (lowercase f)
Use Basic Memory's built-in conflict detection:
```bash
# Sync will report conflicts
basic-memory sync
# Index local file changes (conflicts are handled during the scan)
basic-memory reindex
# Check sync status for warnings
basic-memory status
+183
View File
@@ -0,0 +1,183 @@
# Manual Pages
Basic Memory's manual is written in the style of Unix man pages — and
implemented as Basic Memory notes ([#952](https://github.com/basicmachines-co/basic-memory/issues/952)).
Every page is a markdown note conforming to the `Manpage` schema, `SEE ALSO`
entries are real knowledge-graph relations, and every example on every page
was executed against a live project before the page shipped. The manual
documents the tools; the tools verify the manual.
## Where it lives
The canonical manual is the **`manual` project in the Basic Memory team
workspace** (cloud, shared). Anyone can build their own: the schema ships as
an opt-in seed at `plugins/claude-code/schemas/manpage.md` — copy it into any
project's folder and start writing pages against it.
Layout:
```
manual/
├── schemas/Manpage.md # the manpage schema (type: schema)
├── man1/ # CLI commands bm(1), bm-status(1), ...
├── man3/ # MCP tools write-note(3), search-notes(3), ...
├── man5/ # file formats bm-note(5), bm-observation(5), ...
├── man7/ # concepts basic-memory(7), semantic-memory(7), ...
├── playground/ # scratch notes for destructive examples
└── diagrams/ # canvas visualizations of the manual graph
```
### Why "man1", "man3", "man5"?
The folder names are Unix's, unchanged since 1971. The manual is divided
into numbered **sections**, pages physically live in directories named
after them (`/usr/share/man/man1`, `man5`, ...), and the number tells you
what *kind* of thing is documented — not importance, not reading order:
- **1** — user commands (`ls`, `grep`)
- **2** — system calls
- **3** — library functions / APIs (`printf(3)`)
- **4** — devices
- **5** — file formats and config files (`crontab(5)`, `passwd(5)`)
- **6** — games (really)
- **7** — miscellanea: concepts, conventions, overviews (`regex(7)`, `signal(7)`)
- **8** — system administration
That's also why man page names carry the parenthesized number —
`crontab(1)` is the command, `crontab(5)` is the file format, same name in
two sections. `man 5 crontab` picks the section explicitly.
This manual copies that layout with the sections that have a Basic Memory
analog:
- **man1/** — `bm` CLI commands → `bm-status(1)`
- **man3/** — MCP tools, our equivalent of the "library API" section → `write-note(3)`
- **man5/** — file formats: note syntax, observations, relations, schemas → `bm-note(5)`
- **man7/** — concepts → `basic-memory(7)`, `semantic-memory(7)`
- **8** is reserved for admin/cloud operations but has no pages yet; 2, 4,
and 6 have no analog (no system calls, no devices, and no games — yet)
When a page says `see_also [[bm-note(5)]]`, the `(5)` reads "the
file-format page," exactly the way a Unix manual cross-references — except
here it's a traversable relation in the graph instead of a typographic
convention. The manual explains its own conventions in `man-pages(7)`
fittingly, the same page name Linux uses for this, and that almost nobody
ever reads.
## Page anatomy
Pages use the classic headers where applicable: `NAME`, `SYNOPSIS`,
`DESCRIPTION`, `PARAMETERS`, `MCP USAGE`, `CLI EQUIVALENT`, `EXAMPLES`,
`GOTCHAS`, `SEE ALSO`. Frontmatter (validated by the schema):
```yaml
type: manpage
section: 3 # 1 | 3 | 5 | 7 | 8
name: write-note # page name without section suffix
summary: create or overwrite a markdown note in the knowledge base
generated: hand # hand | registry | typer (regeneration ownership)
tool: write_note # section-3 pages: the MCP tool documented
command: basic-memory status # section-1 pages: the CLI command documented
verified: 0.21.6 mcp+cli # version + path(s) that proved the page
```
Field knowledge accumulates as observations — `[gotcha]`, `[bug]` (with issue
links), `[pattern]` — and `SEE ALSO` entries are `see_also` relations, so the
manual is a navigable graph, not a folder of files.
## How to use it
Man-style reads (any MCP client or the CLI):
```bash
# read a page
bm tool read-note "man3/write-note-3" --project manual
# apropos — find pages by section, tool, or text
bm tool search-notes --project manual # then filter, or via MCP:
# search_notes(project="manual", metadata_filters={"type": "manpage", "section": 3})
# search_notes(project="manual", metadata_filters={"type": "manpage", "tool": "write_note"})
# traverse SEE ALSO from any page
# build_context(url="man3/write-note-3", project="manual")
```
A future `bm man <topic>` command is thin sugar over exactly these calls.
And for the real thing — `man bm` in an actual terminal:
```bash
bm man install # copies bundled groff pages to ~/.local/share/man
man bm # the overview page, rendered by man(1)
man basic-memory # same page via its alias
```
`bm man install` warns with a one-line `MANPATH` fix if the install root
isn't searched by your `man`. Agents with shell access can use `man bm` as
an offline quick reference; the full per-tool detail stays in the manual
project's section-3 pages.
## The verification discipline
Two rules make the manual trustworthy:
1. **Examples must have run.** An `EXAMPLES` (or `MCP USAGE` / `CLI
EQUIVALENT`) block contains only commands that actually executed against
the manual project. Destructive operations (`delete_note`, `move_note`,
destructive `edit_note`) run only against `playground/` notes — never
against pages. The `verified:` field records the version and which path
proved the page: `mcp` (live service), `cli` (dev checkout), or both.
2. **The schema is the linter.** Validate the whole manual any time:
```bash
bm tool schema-validate manpage --project manual
# → {"total_notes": 38, "valid_count": 38, "warning_count": 0, ...}
```
`bm orphans --project manual` confirms every page is connected to the
graph, and `schema_diff`/`schema_infer` report drift between the schema
and how pages are actually written.
Because verification exercises real tool calls against the live service,
building the manual doubles as an end-to-end smoke test. The initial build
found six bugs in one pass (#954#959) — including the verification rule
catching a test that asserted a bug as expected output (#958).
## Adding or updating a page
1. Run the commands you intend to document; keep the actual output.
2. Write the page with `write_note`, passing frontmatter through the
`metadata` parameter (nested YAML in content frontmatter is unreliable on
some clients):
```
write_note(title="my-tool(3)", directory="man3", project="manual",
note_type="manpage",
metadata={"section": 3, "name": "my-tool",
"summary": "...", "generated": "hand",
"tool": "my_tool", "verified": "<version> mcp"})
```
3. Link related pages in `SEE ALSO` with `see_also [[other-page(3)]]`.
Forward references to pages that don't exist yet are fine — they resolve
automatically when the target is written.
4. Validate: `bm tool schema-validate manpage --project manual`.
For mechanical updates to generated sections, prefer `edit_note` with
`replace_section` / `insert_after_section` so curated content (EXAMPLES,
GOTCHAS, SEE ALSO, observations) survives — that ownership split is what the
`generated:` field declares.
## Roadmap
- **Registry generator** — section-3 SYNOPSIS/PARAMETERS generated from the
MCP tool registry (docstrings + pydantic schemas), section-1 from Typer
help; the hand-written corpus is the template spec. Regenerate-and-diff in
CI becomes the drift gate.
- **`bm man <topic>`** — CLI sugar over `read_note` + metadata search.
(`bm man install` + a hand-written `bm.1` already ship — the first slice
of [#610](https://github.com/basicmachines-co/basic-memory/issues/610);
the generator will produce per-command pages from the same extraction.)
- **Docs site** — the notes remain canonical for sections 5 and 7, code is
canonical for 1 and 3; both render to the hosted docs site.
+1 -1
View File
@@ -38,7 +38,7 @@ from typing import Any, Callable
from agent.memory_provider import MemoryProvider
from tools.registry import tool_error
__version__ = "0.22.0"
__version__ = "0.22.1"
logger = logging.getLogger("hermes.memory.basic-memory")
+1 -1
View File
@@ -1,5 +1,5 @@
name: basic-memory
version: 0.22.0
version: 0.22.1
description: "Basic Memory — persistent knowledge graph backed by the basic-memory MCP server"
pip_dependencies:
- mcp
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@basicmemory/openclaw-basic-memory",
"version": "0.22.0",
"version": "0.22.1",
"type": "module",
"main": "./dist/index.js",
"types": "./dist/index.d.ts",
+118 -26
View File
@@ -383,17 +383,31 @@ release version:
echo "❌ Tag {{version}} already exists"
exit 1
fi
# Changelog must already be on main (land it via a normal PR first)
if ! grep -q "^## {{version}} " CHANGELOG.md; then
echo "❌ CHANGELOG.md has no entry for {{version}}. Land one via PR first."
exit 1
fi
# Run quality checks
echo "🔍 Running lint checks..."
just lint
just typecheck
# Update all package manifests to the one Basic Memory product version.
echo "📝 Updating consolidated package versions..."
just set-version "{{version}}"
# Commit version update
# Trigger: main's ruleset rejects direct pushes ("Changes must be made
# through a pull request").
# Why: the version bump must land on main before the tag is cut, so it
# rides a release PR that is rebase-merged (the repo disallows merge
# commits).
# Outcome: the bump commit gets a new SHA on main; the tag is created on
# that rebased commit, found by its commit subject.
COMMIT_SUBJECT="chore: update version to $VERSION_NUM for {{version}} release"
git checkout -b "release/{{version}}"
git add \
src/basic_memory/__init__.py \
server.json \
@@ -404,23 +418,58 @@ release version:
integrations/hermes/plugin.yaml \
integrations/hermes/__init__.py \
integrations/openclaw/package.json
git commit -s -m "chore: update version to $VERSION_NUM for {{version}} release"
# Create and push tag
echo "🏷️ Creating tag {{version}}..."
git tag "{{version}}"
echo "📤 Pushing to GitHub..."
git push origin main
git commit -s -m "$COMMIT_SUBJECT"
echo "📤 Opening release PR..."
git push -u origin "release/{{version}}"
gh pr create --title "chore(core): release {{version}}" \
--body "Version bump for {{version}}. See CHANGELOG.md for release notes."
# Trigger: the PR may not be mergeable synchronously (merge gates,
# required checks added later, or GitHub still computing mergeability).
# Why: the tag must point at the bump commit on main, so the recipe
# cannot tag until the merge has actually landed.
# Outcome: try a direct rebase-merge, fall back to queueing auto-merge,
# then poll main for the rebased bump commit before tagging.
if ! gh pr merge "release/{{version}}" --rebase --delete-branch; then
echo "⚠️ Direct merge did not complete (merge gates pending?). Queueing auto-merge..."
gh pr merge "release/{{version}}" --rebase --delete-branch --auto
fi
echo "⏳ Waiting for the bump commit to land on main..."
TAG_COMMIT=""
for _ in $(seq 1 60); do
git fetch origin main --quiet
TAG_COMMIT=$(git log FETCH_HEAD --fixed-strings --grep "$COMMIT_SUBJECT" --format='%H' -1)
[[ -n "$TAG_COMMIT" ]] && break
sleep 5
done
if [[ -z "$TAG_COMMIT" ]]; then
echo "❌ Bump commit not on main after 5 minutes (merge still pending?)."
echo " Once the release PR merges, finish the release manually:"
echo " git fetch origin main"
echo " git tag {{version}} \$(git log FETCH_HEAD --fixed-strings --grep \"$COMMIT_SUBJECT\" --format='%H' -1)"
echo " git push origin {{version}}"
exit 1
fi
git checkout main
git pull --ff-only origin main
git branch -D "release/{{version}}" 2>/dev/null || true
echo "🏷️ Creating tag {{version}} at $TAG_COMMIT..."
git tag "{{version}}" "$TAG_COMMIT"
git push origin "{{version}}"
echo "✅ Release {{version}} created successfully!"
echo "📦 GitHub Actions will build and publish to PyPI"
echo "🔗 Monitor at: https://github.com/basicmachines-co/basic-memory/actions"
echo ""
echo "📝 REMINDER: Post-release tasks:"
echo " 1. docs.basicmemory.com - Add release notes to src/pages/latest-releases.mdx"
echo " 2. basicmachines.co - Update version in src/components/sections/hero.tsx"
echo " 1. docs.basicmemory.com - Add a What's New page under content/2.whats-new/"
echo " and bump the badge in content/index.md (see that repo's CLAUDE.md)"
echo " 2. basicmemory.com - No version number in the site UI; for a significant"
echo " release optionally add a post under src/content/blog/. Skip for patches."
echo " 3. MCP Registry - Run: mcp-publisher publish"
echo " See: .claude/commands/release/release.md for detailed instructions"
@@ -467,7 +516,15 @@ beta version:
echo "📝 Updating consolidated package versions..."
just set-version "{{version}}"
# Commit version update
# Trigger: main's ruleset rejects direct pushes ("Changes must be made
# through a pull request").
# Why: the version bump must land on main before the tag is cut, so it
# rides a release PR that is rebase-merged (the repo disallows merge
# commits).
# Outcome: the bump commit gets a new SHA on main; the tag is created on
# that rebased commit, found by its commit subject.
COMMIT_SUBJECT="chore: update version to $VERSION_NUM for {{version}} beta release"
git checkout -b "release/{{version}}"
git add \
src/basic_memory/__init__.py \
server.json \
@@ -478,24 +535,59 @@ beta version:
integrations/hermes/plugin.yaml \
integrations/hermes/__init__.py \
integrations/openclaw/package.json
git commit -s -m "chore: update version to $VERSION_NUM for {{version}} beta release"
# Create and push tag
echo "🏷️ Creating tag {{version}}..."
git tag "{{version}}"
echo "📤 Pushing to GitHub..."
git push origin main
git commit -s -m "$COMMIT_SUBJECT"
echo "📤 Opening release PR..."
git push -u origin "release/{{version}}"
gh pr create --title "chore(core): release {{version}}" \
--body "Version bump for {{version}} beta."
# Trigger: the PR may not be mergeable synchronously (merge gates,
# required checks added later, or GitHub still computing mergeability).
# Why: the tag must point at the bump commit on main, so the recipe
# cannot tag until the merge has actually landed.
# Outcome: try a direct rebase-merge, fall back to queueing auto-merge,
# then poll main for the rebased bump commit before tagging.
if ! gh pr merge "release/{{version}}" --rebase --delete-branch; then
echo "⚠️ Direct merge did not complete (merge gates pending?). Queueing auto-merge..."
gh pr merge "release/{{version}}" --rebase --delete-branch --auto
fi
echo "⏳ Waiting for the bump commit to land on main..."
TAG_COMMIT=""
for _ in $(seq 1 60); do
git fetch origin main --quiet
TAG_COMMIT=$(git log FETCH_HEAD --fixed-strings --grep "$COMMIT_SUBJECT" --format='%H' -1)
[[ -n "$TAG_COMMIT" ]] && break
sleep 5
done
if [[ -z "$TAG_COMMIT" ]]; then
echo "❌ Bump commit not on main after 5 minutes (merge still pending?)."
echo " Once the release PR merges, finish the release manually:"
echo " git fetch origin main"
echo " git tag {{version}} \$(git log FETCH_HEAD --fixed-strings --grep \"$COMMIT_SUBJECT\" --format='%H' -1)"
echo " git push origin {{version}}"
exit 1
fi
git checkout main
git pull --ff-only origin main
git branch -D "release/{{version}}" 2>/dev/null || true
echo "🏷️ Creating tag {{version}} at $TAG_COMMIT..."
git tag "{{version}}" "$TAG_COMMIT"
git push origin "{{version}}"
echo "✅ Beta release {{version}} created successfully!"
echo "📦 GitHub Actions will build and publish to PyPI as pre-release"
echo "🔗 Monitor at: https://github.com/basicmachines-co/basic-memory/actions"
echo "📥 Install with: uv tool install basic-memory --pre"
echo ""
echo "📝 REMINDER: For stable releases, update documentation sites:"
echo " 1. docs.basicmemory.com - Add release notes to src/pages/latest-releases.mdx"
echo " 2. basicmachines.co - Update version in src/components/sections/hero.tsx"
echo " 1. docs.basicmemory.com - Add a What's New page under content/2.whats-new/"
echo " and bump the badge in content/index.md (see that repo's CLAUDE.md)"
echo " 2. basicmemory.com - No version number in the site UI; for a significant"
echo " release optionally add a post under src/content/blog/. Skip for patches."
echo " See: .claude/commands/release/release.md for detailed instructions"
# List all available recipes
@@ -6,14 +6,14 @@
},
"metadata": {
"description": "Official plugins from Basic Machines for knowledge management and AI-assisted development",
"version": "0.22.0"
"version": "0.22.1"
},
"plugins": [
{
"name": "basic-memory",
"source": "./",
"description": "The bridge between Claude's working memory and Basic Memory's durable knowledge graph \u2014 session briefings, pre-compaction checkpoints, and capture reflexes",
"version": "0.22.0",
"version": "0.22.1",
"author": {
"name": "Basic Machines"
},
@@ -1,7 +1,7 @@
{
"name": "basic-memory",
"description": "The bridge between Claude's working memory and Basic Memory's durable knowledge graph \u2014 session briefings, pre-compaction checkpoints, and capture reflexes",
"version": "0.22.0",
"version": "0.22.1",
"author": {
"name": "Basic Machines"
},
+55
View File
@@ -0,0 +1,55 @@
---
title: Manpage
type: schema
entity: Manpage
version: 1
schema:
gotcha?(array): string, sharp edges and surprising behavior learned from live verification
example?(array): string, worked examples beyond the generated synopsis
pattern?(array): string, recommended idioms and usage patterns
bug?(array): string, known defects affecting this surface, with issue links
see_also?(array): Entity, related manual pages — the SEE ALSO graph
settings:
validation: warn
frontmatter:
section(enum, Unix manual section number): [1, 3, 5, 7, 8]
name: string, page name without section suffix (e.g. write-note)
summary: string, one-line NAME description
generated?(enum, who owns the mechanical sections): [registry, typer, hand]
tool?: string, MCP tool this page documents (section 3 pages)
command?: string, CLI command this page documents (section 1 pages)
verified?: string, version and path that verified this page (e.g. 0.21.6 mcp+cli)
since?: string, version this surface first appeared
---
# Manpage
A **ManpageNote** is one page of a Unix-style manual implemented as Basic
Memory notes (issue #952): commands in section 1, MCP tools in section 3,
file formats in section 5, concepts in section 7, admin in section 8. The
manual becomes a knowledge graph — `SEE ALSO` entries are typed relations,
and pages are found by structured recall:
`search_notes(metadata_filters={"type": "manpage", "section": 3})`.
This schema is an opt-in seed for documentation projects; the canonical
manual lives in the Basic Memory team workspace `manual` project.
## What makes a good ManpageNote
- **NAME / SYNOPSIS / DESCRIPTION** — classic man-page structure, with
PARAMETERS, MCP USAGE, CLI EQUIVALENT, EXAMPLES, GOTCHAS, SEE ALSO where
applicable.
- **Verified examples** — EXAMPLES contain only commands that actually ran;
the `verified` field records the version and path (mcp, cli, or both).
- **generated** — declares regeneration ownership: `registry` (from the MCP
tool registry) and `typer` (from CLI help) pages get mechanical sections
rewritten; curated sections (EXAMPLES, GOTCHAS, SEE ALSO, observations)
are never overwritten.
- **gotcha / bug observations** — field knowledge accumulates on pages
without being clobbered by regeneration; bugs link their tracking issues.
## Frontmatter
`type: manpage` plus `section` makes the manual queryable like `man -k`:
by section, by `tool`, by `command`, or by missing/stale `verified` stamps.
Validation is `warn`, never blocking.
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "codex",
"version": "0.22.0",
"version": "0.22.1",
"description": "A Codex-native bridge to Basic Memory for durable engineering context, decisions, and resumable checkpoints.",
"author": {
"name": "Basic Machines",
+2 -2
View File
@@ -6,12 +6,12 @@
"url": "https://github.com/basicmachines-co/basic-memory.git",
"source": "github"
},
"version": "0.22.0",
"version": "0.22.1",
"packages": [
{
"registryType": "pypi",
"identifier": "basic-memory",
"version": "0.22.0",
"version": "0.22.1",
"runtimeHint": "uvx",
"runtimeArguments": [
{
+1 -1
View File
@@ -1,7 +1,7 @@
"""basic-memory - Local-first knowledge management combining Zettelkasten with knowledge graphs"""
# Package version - updated by release automation
__version__ = "0.22.0"
__version__ = "0.22.1"
# API version for FastAPI - independent of package version
__api_version__ = "v0"
@@ -17,6 +17,7 @@ from fastapi import APIRouter, HTTPException, Response, Path
from loguru import logger
import logfire
from basic_memory import db
from basic_memory.ignore_utils import (
IGNORED_PATH_REJECTION_DETAIL,
load_gitignore_patterns,
@@ -34,6 +35,8 @@ from basic_memory.deps import (
ProjectExternalIdPathDep,
SyncServiceV2ExternalDep,
TaskSchedulerDep,
SessionDep,
SessionMakerDep,
)
from basic_memory.schemas import DeleteEntitiesResponse
from basic_memory.schemas.base import Entity
@@ -81,6 +84,7 @@ async def get_graph(
project_id: ProjectExternalIdPathDep,
entity_repository: EntityRepositoryV2ExternalDep,
relation_repository: RelationRepositoryV2ExternalDep,
session: SessionDep,
) -> GraphResponse:
"""Return all entities and resolved relations for knowledge graph visualization.
@@ -96,7 +100,7 @@ async def get_graph(
logger.info("API v2 request: get_graph")
# Fetch all entities for this project
entities = await entity_repository.find_all(use_load_options=False)
entities = await entity_repository.find_all(session, use_load_options=False)
nodes = [
GraphNode(
external_id=entity.external_id,
@@ -108,7 +112,7 @@ async def get_graph(
]
# Fetch all resolved relations (to_id is not null) with eager-loaded entities
relations = await relation_repository.find_all()
relations = await relation_repository.find_all(session)
edges = [
GraphEdge(
from_id=relation.from_entity.external_id,
@@ -130,6 +134,7 @@ async def get_graph(
async def get_orphan_entities(
project_id: ProjectExternalIdPathDep,
entity_repository: EntityRepositoryV2ExternalDep,
session: SessionDep,
) -> OrphanEntitiesResponse:
"""Return entities that have no incoming or outgoing relations."""
with logfire.span(
@@ -140,7 +145,7 @@ async def get_orphan_entities(
):
logger.info("API v2 request: get_orphan_entities")
entities = await entity_repository.find_without_relations()
entities = await entity_repository.find_without_relations(session)
nodes = [
GraphNode(
external_id=entity.external_id,
@@ -165,6 +170,7 @@ async def resolve_identifier(
link_resolver: LinkResolverV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
project_repository: ProjectRepositoryDep,
session: SessionDep,
) -> EntityResolveResponse:
"""Resolve a string identifier (external_id, permalink, title, or path) to entity info.
@@ -203,12 +209,15 @@ async def resolve_identifier(
):
logger.info(f"API v2 request: resolve_identifier for '{data.identifier}'")
entity = await entity_repository.get_by_external_id(data.identifier)
entity = await entity_repository.get_by_external_id(session, data.identifier)
resolution_method = "external_id" if entity else "search"
if not entity:
entity = await link_resolver.resolve_link(
data.identifier, source_path=data.source_path, strict=data.strict
data.identifier,
source_path=data.source_path,
strict=data.strict,
session=session,
)
if entity:
if entity.permalink == data.identifier:
@@ -223,7 +232,7 @@ async def resolve_identifier(
if not entity:
raise HTTPException(status_code=404, detail=f"Entity not found: '{data.identifier}'")
owner_project = await project_repository.get_by_id(entity.project_id)
owner_project = await project_repository.get_by_id(session, entity.project_id)
if not owner_project: # pragma: no cover
raise HTTPException(
status_code=500,
@@ -304,6 +313,7 @@ async def sync_file(
project_config: ProjectConfigV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
app_config: AppConfigDep,
session_maker: SessionMakerDep,
) -> EntityResponseV2:
"""Index a single markdown file that exists on disk but is not indexed yet.
@@ -406,7 +416,8 @@ async def sync_file(
# Trigger: the file may already have a DB row (e.g. modified on disk after indexing)
# Why: the indexer needs to know whether to insert or update the entity
# Outcome: new is computed from the database instead of assumed by the caller
existing = await sync_service.entity_repository.get_by_file_path(file_path)
async with db.scoped_session(session_maker) as session:
existing = await sync_service.entity_repository.get_by_file_path(session, file_path)
synced = await sync_service.sync_one_markdown_file(
file_path, new=existing is None, index_search=True
)
@@ -435,6 +446,7 @@ async def sync_file(
async def get_entity_by_id(
project_id: ProjectExternalIdPathDep,
entity_repository: EntityRepositoryV2ExternalDep,
session: SessionDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
) -> EntityResponseV2:
"""Get an entity by its external ID (UUID).
@@ -459,7 +471,7 @@ async def get_entity_by_id(
):
logger.info(f"API v2 request: get_entity_by_id entity_id={entity_id}")
entity = await entity_repository.get_by_external_id(entity_id)
entity = await entity_repository.get_by_external_id(session, entity_id)
if not entity:
raise HTTPException(
status_code=404, detail=f"Entity with external_id '{entity_id}' not found"
@@ -536,6 +548,7 @@ async def update_entity_by_id(
entity_repository: EntityRepositoryV2ExternalDep,
task_scheduler: TaskSchedulerDep,
app_config: AppConfigDep,
session_maker: SessionMakerDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
) -> EntityResponseV2:
"""Update an entity by external ID.
@@ -557,7 +570,8 @@ async def update_entity_by_id(
):
logger.info(f"API v2 request: update_entity_by_id entity_id={entity_id}")
existing = await entity_repository.get_by_external_id(entity_id)
async with db.scoped_session(session_maker) as session:
existing = await entity_repository.get_by_external_id(session, entity_id)
created = existing is None
if existing:
@@ -568,10 +582,12 @@ async def update_entity_by_id(
write_result = await entity_service.create_entity_with_content(data)
entity = write_result.entity
if entity.external_id != entity_id:
entity = await entity_repository.update(
entity.id,
{"external_id": entity_id},
)
async with db.scoped_session(session_maker) as session:
entity = await entity_repository.update(
session,
entity.id,
{"external_id": entity_id},
)
# external_id fixup only changes the DB row. The file content is unchanged,
# so the markdown captured during the write remains valid downstream.
if not entity:
@@ -607,6 +623,7 @@ async def edit_entity_by_id(
entity_repository: EntityRepositoryV2ExternalDep,
task_scheduler: TaskSchedulerDep,
app_config: AppConfigDep,
session_maker: SessionMakerDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
) -> EntityResponseV2:
"""Edit an existing entity by external ID using operations like append, prepend, etc.
@@ -631,7 +648,8 @@ async def edit_entity_by_id(
f"API v2 request: edit_entity_by_id entity_id={entity_id}, operation='{data.operation}'"
)
entity = await entity_repository.get_by_external_id(entity_id)
async with db.scoped_session(session_maker) as session:
entity = await entity_repository.get_by_external_id(session, entity_id)
if not entity: # pragma: no cover
raise HTTPException(
status_code=404, detail=f"Entity with external_id '{entity_id}' not found"
@@ -678,6 +696,7 @@ async def delete_entity_by_id(
project_id: ProjectExternalIdPathDep,
entity_service: EntityServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
session_maker: SessionMakerDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
) -> DeleteEntitiesResponse:
"""Delete an entity by external ID.
@@ -698,7 +717,8 @@ async def delete_entity_by_id(
):
logger.info(f"API v2 request: delete_entity_by_id entity_id={entity_id}")
entity = await entity_repository.get_by_external_id(entity_id)
async with db.scoped_session(session_maker) as session:
entity = await entity_repository.get_by_external_id(session, entity_id)
if entity is None:
logger.info(f"API v2 response: external_id={entity_id} not found, deleted=False")
return DeleteEntitiesResponse(deleted=False)
@@ -724,6 +744,7 @@ async def move_entity(
app_config: AppConfigDep,
search_service: SearchServiceV2ExternalDep,
task_scheduler: TaskSchedulerDep,
session_maker: SessionMakerDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
) -> EntityResponseV2:
"""Move an entity to a new file location.
@@ -751,7 +772,8 @@ async def move_entity(
try:
# First, get the entity by external_id to verify it exists
entity = await entity_repository.get_by_external_id(entity_id)
async with db.scoped_session(session_maker) as session:
entity = await entity_repository.get_by_external_id(session, entity_id)
if not entity: # pragma: no cover
raise HTTPException(
status_code=404, detail=f"Entity with external_id '{entity_id}' not found"
@@ -766,9 +788,11 @@ async def move_entity(
)
# Reindex at new location
reindexed_entity = await entity_service.link_resolver.resolve_link(
data.destination_path
)
async with db.scoped_session(session_maker) as session:
reindexed_entity = await entity_service.link_resolver.resolve_link(
data.destination_path,
session=session,
)
if reindexed_entity:
await search_service.index_entity(reindexed_entity)
_schedule_vector_sync_if_enabled(
@@ -805,6 +829,7 @@ async def move_directory(
app_config: AppConfigDep,
search_service: SearchServiceV2ExternalDep,
task_scheduler: TaskSchedulerDep,
session_maker: SessionMakerDep,
) -> DirectoryMoveResult:
"""Move all entities in a directory to a new location.
@@ -840,7 +865,10 @@ async def move_directory(
# Reindex moved entities
for file_path in result.moved_files:
entity = await entity_service.link_resolver.resolve_link(file_path)
async with db.scoped_session(session_maker) as session:
entity = await entity_service.link_resolver.resolve_link(
file_path, session=session
)
if entity:
await search_service.index_entity(entity)
_schedule_vector_sync_if_enabled(
@@ -10,7 +10,12 @@ from fastapi import APIRouter, Query, Path
from loguru import logger
import logfire
from basic_memory.deps import ContextServiceV2ExternalDep, EntityRepositoryV2ExternalDep
from basic_memory import db
from basic_memory.deps import (
ContextServiceV2ExternalDep,
EntityRepositoryV2ExternalDep,
SessionMakerDep,
)
from basic_memory.schemas.base import TimeFrame, parse_timeframe
from basic_memory.schemas.memory import (
GraphContext,
@@ -27,6 +32,7 @@ router = APIRouter(tags=["memory"])
async def recent(
context_service: ContextServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
session_maker: SessionMakerDep,
project_id: str = Path(..., description="Project external UUID"),
type: Annotated[list[SearchItemType] | None, Query()] = None,
depth: int = 1,
@@ -95,9 +101,14 @@ async def recent(
phase="shape_response",
result_count=len(context.results),
):
recent_context = await to_graph_context(
context, entity_repository=entity_repository, page=page, page_size=page_size
)
async with db.scoped_session(session_maker) as session:
recent_context = await to_graph_context(
context,
entity_repository=entity_repository,
session=session,
page=page,
page_size=page_size,
)
logger.debug(f"V2 Recent context: {recent_context.model_dump_json()}")
return recent_context
@@ -109,6 +120,7 @@ async def recent(
async def get_memory_context(
context_service: ContextServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
session_maker: SessionMakerDep,
uri: str,
project_id: str = Path(..., description="Project external UUID"),
depth: int = 1,
@@ -177,6 +189,11 @@ async def get_memory_context(
phase="shape_response",
result_count=len(context.results),
):
return await to_graph_context(
context, entity_repository=entity_repository, page=page, page_size=page_size
)
async with db.scoped_session(session_maker) as session:
return await to_graph_context(
context,
entity_repository=entity_repository,
session=session,
page=page,
page_size=page_size,
)
@@ -16,6 +16,7 @@ from typing import Literal, Optional
from fastapi import APIRouter, HTTPException, Body, Query, Path
from loguru import logger
from basic_memory import db
from basic_memory.deps import (
ProjectServiceDep,
ProjectRepositoryDep,
@@ -23,6 +24,8 @@ from basic_memory.deps import (
SyncServiceV2ExternalDep,
TaskSchedulerDep,
ProjectExternalIdPathDep,
SessionDep,
SessionMakerDep,
)
from basic_memory.schemas import SyncReportResponse
from basic_memory.models import Project
@@ -54,21 +57,22 @@ def _split_qualified_project_identifier(identifier: str) -> tuple[str | None, st
async def _resolve_project_identifier_candidate(
session: SessionDep,
project_repository: ProjectRepository,
identifier: str,
) -> tuple[Project | None, ProjectResolveMethod]:
"""Resolve one project identifier candidate and report the matching method."""
identifier_permalink = generate_permalink(identifier)
project = await project_repository.get_by_external_id(identifier)
project = await project_repository.get_by_external_id(session, identifier)
if project:
return project, "external_id"
project = await project_repository.get_by_permalink(identifier_permalink)
project = await project_repository.get_by_permalink(session, identifier_permalink)
if project:
return project, "permalink"
project = await project_repository.get_by_name_case_insensitive(identifier)
project = await project_repository.get_by_name_case_insensitive(session, identifier)
if project:
return project, "name" # pragma: no cover
@@ -76,11 +80,13 @@ async def _resolve_project_identifier_candidate(
async def _resolve_project_identifier(
session: SessionDep,
project_repository: ProjectRepository,
identifier: str,
) -> tuple[Project | None, ProjectResolveMethod]:
"""Resolve exact identifiers first, then accepted workspace-qualified forms."""
project, resolution_method = await _resolve_project_identifier_candidate(
session,
project_repository,
identifier,
)
@@ -96,6 +102,7 @@ async def _resolve_project_identifier(
# only needs the project segment to validate the active project.
# Outcome: models can follow the hint verbatim instead of looping on a 404.
project, resolution_method = await _resolve_project_identifier_candidate(
session,
project_repository,
project_identifier,
)
@@ -193,7 +200,7 @@ async def add_project(
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message=f"Project '{new_project.name}' added successfully",
status="success",
default=project_data.set_default,
default=new_project.is_default or False,
new_project=ProjectItem(
id=new_project.id,
external_id=new_project.external_id,
@@ -277,6 +284,7 @@ async def get_project_status(
@router.post("/resolve", response_model=ProjectResolveResponse)
async def resolve_project_identifier(
data: ProjectResolveRequest,
session: SessionDep,
project_repository: ProjectRepositoryDep,
) -> ProjectResolveResponse:
"""Resolve a project identifier (name, permalink, or external_id) to project info.
@@ -315,12 +323,26 @@ async def resolve_project_identifier(
logger.info(f"API v2 request: resolve_project_identifier for '{data.identifier}'")
project, resolution_method = await _resolve_project_identifier(
session,
project_repository,
data.identifier,
)
if not project:
raise HTTPException(status_code=404, detail=f"Project not found: '{data.identifier}'")
detail = f"Project not found: '{data.identifier}'"
# Trigger: resolution missed and the projects table is empty.
# Why: a fresh install bootstraps config.json's default project before any
# reconciliation has created database rows (the one-shot CLI never runs
# the server lifespan), so the first read fails on the configured
# default and the bare not-found message reads as a broken install
# rather than a missing first-run step (#974 follow-up).
# Outcome: the error names the setup command instead.
if not await project_repository.find_all(session, limit=1, use_load_options=False):
detail = (
f"{detail}. No projects are set up yet — run "
"'basic-memory project add <name> <path>' to create one."
)
raise HTTPException(status_code=404, detail=detail)
return ProjectResolveResponse(
external_id=project.external_id,
@@ -336,6 +358,7 @@ async def resolve_project_identifier(
@router.get("/{project_id}", response_model=ProjectItem)
async def get_project_by_id(
session: SessionDep,
project_repository: ProjectRepositoryDep,
project_id: str = Path(..., description="Project external ID (UUID)"),
) -> ProjectItem:
@@ -358,7 +381,7 @@ async def get_project_by_id(
"""
logger.info(f"API v2 request: get_project_by_id for project_id={project_id}")
project = await project_repository.get_by_external_id(project_id)
project = await project_repository.get_by_external_id(session, project_id)
if not project:
raise HTTPException(
status_code=404, detail=f"Project with external_id '{project_id}' not found"
@@ -376,12 +399,14 @@ async def get_project_by_id(
@router.get("/{project_id}/info", response_model=ProjectInfoResponse)
async def get_project_info_by_id(
project_service: ProjectServiceDep,
session_maker: SessionMakerDep,
project_repository: ProjectRepositoryDep,
project_id: str = Path(..., description="Project external ID (UUID)"),
) -> ProjectInfoResponse:
"""Get detailed project information by external ID."""
logger.info(f"API v2 request: get_project_info_by_id for project_id={project_id}")
project = await project_repository.get_by_external_id(project_id)
async with db.scoped_session(session_maker) as session:
project = await project_repository.get_by_external_id(session, project_id)
if not project:
raise HTTPException(
status_code=404, detail=f"Project with external_id '{project_id}' not found"
@@ -392,6 +417,7 @@ async def get_project_info_by_id(
@router.patch("/{project_id}", response_model=ProjectStatusResponse)
async def update_project_by_id(
project_service: ProjectServiceDep,
session_maker: SessionMakerDep,
project_repository: ProjectRepositoryDep,
project_id: str = Path(..., description="Project external ID (UUID)"),
path: Optional[str] = Body(None, description="New absolute path for the project"),
@@ -422,7 +448,8 @@ async def update_project_by_id(
raise HTTPException(status_code=400, detail="Path must be absolute")
# Get original project info for the response
old_project = await project_repository.get_by_external_id(project_id)
async with db.scoped_session(session_maker) as session:
old_project = await project_repository.get_by_external_id(session, project_id)
if not old_project:
raise HTTPException(
status_code=404, detail=f"Project with external_id '{project_id}' not found"
@@ -443,7 +470,8 @@ async def update_project_by_id(
await project_service.update_project(old_project.name, is_active=is_active)
# Get updated project info (use the same external_id)
updated_project = await project_repository.get_by_external_id(project_id)
async with db.scoped_session(session_maker) as session:
updated_project = await project_repository.get_by_external_id(session, project_id)
if not updated_project: # pragma: no cover
raise HTTPException(
status_code=404,
@@ -470,6 +498,7 @@ async def update_project_by_id(
@router.delete("/{project_id}", response_model=ProjectStatusResponse)
async def delete_project_by_id(
project_service: ProjectServiceDep,
session_maker: SessionMakerDep,
project_repository: ProjectRepositoryDep,
project_id: str = Path(..., description="Project external ID (UUID)"),
delete_notes: bool = Query(
@@ -496,7 +525,8 @@ async def delete_project_by_id(
)
try:
old_project = await project_repository.get_by_external_id(project_id)
async with db.scoped_session(session_maker) as session:
old_project = await project_repository.get_by_external_id(session, project_id)
if not old_project:
raise HTTPException(
status_code=404, detail=f"Project with external_id '{project_id}' not found"
@@ -539,6 +569,7 @@ async def delete_project_by_id(
@router.put("/{project_id}/default", response_model=ProjectStatusResponse)
async def set_default_project_by_id(
project_service: ProjectServiceDep,
session_maker: SessionMakerDep,
project_repository: ProjectRepositoryDep,
project_id: str = Path(..., description="Project external ID (UUID)"),
) -> ProjectStatusResponse:
@@ -559,15 +590,14 @@ async def set_default_project_by_id(
logger.info(f"API v2 request: set_default_project_by_id for project_id={project_id}")
try:
# Get the old default project from database
default_project = await project_repository.get_default_project()
if not default_project:
raise HTTPException( # pragma: no cover
status_code=404, detail="No default project is currently set"
)
# Get the old default project from database. It may be absent during
# bootstrap/recovery (no default row yet); that is a valid state, not an
# error, so we only echo it back when one exists.
async with db.scoped_session(session_maker) as session:
default_project = await project_repository.get_default_project(session)
# Get the new default project by external_id
new_default_project = await project_repository.get_by_external_id(project_id)
# Get the new default project by external_id
new_default_project = await project_repository.get_by_external_id(session, project_id)
if not new_default_project:
raise HTTPException(
status_code=404, detail=f"Project with external_id '{project_id}' not found"
@@ -576,17 +606,27 @@ async def set_default_project_by_id(
# Set as default using project name (service layer still uses names internally)
await project_service.set_default_project(new_default_project.name)
return ProjectStatusResponse(
message=f"Project '{new_default_project.name}' set as default successfully",
status="success",
default=True,
old_project=ProjectItem(
# Trigger: a previous default existed
# Why: ProjectStatusResponse.old_project is Optional; the no-default
# bootstrap case must succeed with old_project=None
# Outcome: response echoes the prior default only when there was one
old_project = (
ProjectItem(
id=default_project.id,
external_id=default_project.external_id,
name=default_project.name,
path=default_project.path,
is_default=False,
),
)
if default_project
else None
)
return ProjectStatusResponse(
message=f"Project '{new_default_project.name}' set as default successfully",
status="success",
default=True,
old_project=old_project,
new_project=ProjectItem(
id=new_default_project.id,
external_id=new_default_project.external_id,
@@ -10,6 +10,7 @@ from typing import Any
from fastapi import APIRouter, HTTPException, status, Path
from loguru import logger
from basic_memory import db
from basic_memory.api.v2.utils import to_graph_context, to_search_results
from basic_memory.api.template_loader import template_loader
from basic_memory.schemas.base import parse_timeframe
@@ -18,6 +19,7 @@ from basic_memory.deps import (
EntityRepositoryV2ExternalDep,
SearchServiceV2ExternalDep,
EntityServiceV2ExternalDep,
SessionMakerDep,
)
from basic_memory.schemas.prompt import (
ContinueConversationRequest,
@@ -36,6 +38,7 @@ async def continue_conversation(
entity_service: EntityServiceV2ExternalDep,
context_service: ContextServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
session_maker: SessionMakerDep,
request: ContinueConversationRequest,
project_id: str = Path(..., description="Project external UUID"),
) -> PromptResponse:
@@ -82,9 +85,10 @@ async def continue_conversation(
)
# Process results into the schema format
graph_context = await to_graph_context(
context_result, entity_repository=entity_repository
)
async with db.scoped_session(session_maker) as session:
graph_context = await to_graph_context(
context_result, entity_repository=entity_repository, session=session
)
# Add results to our collection (limit to top results for each permalink)
if graph_context.results:
@@ -109,7 +113,10 @@ async def continue_conversation(
max_related=request.related_items_limit,
include_observations=True,
)
recent_context = await to_graph_context(context_result, entity_repository=entity_repository)
async with db.scoped_session(session_maker) as session:
recent_context = await to_graph_context(
context_result, entity_repository=entity_repository, session=session
)
hierarchical_results = recent_context.results[:5] # Limit to top 5 recent items
@@ -16,11 +16,14 @@ from fastapi import APIRouter, HTTPException, Response, Path
from loguru import logger
import logfire
from basic_memory import db
from basic_memory.deps import (
ProjectConfigV2ExternalDep,
FileServiceV2ExternalDep,
EntityRepositoryV2ExternalDep,
SearchServiceV2ExternalDep,
SessionDep,
SessionMakerDep,
)
from basic_memory.models.knowledge import Entity as EntityModel
from basic_memory.schemas.v2.resource import (
@@ -38,6 +41,7 @@ async def get_resource_content(
config: ProjectConfigV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
file_service: FileServiceV2ExternalDep,
session: SessionDep,
project_id: str = Path(..., description="Project external UUID"),
entity_id: str = Path(..., description="Entity external UUID"),
) -> Response:
@@ -70,7 +74,7 @@ async def get_resource_content(
action="get_content",
phase="load_entity",
):
entity = await entity_repository.get_by_external_id(entity_id)
entity = await entity_repository.get_by_external_id(session, entity_id)
if not entity:
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
@@ -121,6 +125,7 @@ async def create_resource(
file_service: FileServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
session_maker: SessionMakerDep,
project_id: str = Path(..., description="Project external UUID"),
) -> ResourceResponse:
"""Create a new resource file.
@@ -158,13 +163,14 @@ async def create_resource(
"Path must be relative and stay within project boundaries.",
)
existing_entity = await entity_repository.get_by_file_path(data.file_path)
if existing_entity:
raise HTTPException(
status_code=409,
detail=f"Resource already exists at {data.file_path} with entity_id {existing_entity.external_id}. "
f"Use PUT /resource/{existing_entity.external_id} to update it.",
)
async with db.scoped_session(session_maker) as session:
existing_entity = await entity_repository.get_by_file_path(session, data.file_path)
if existing_entity:
raise HTTPException(
status_code=409,
detail=f"Resource already exists at {data.file_path} with entity_id {existing_entity.external_id}. "
f"Use PUT /resource/{existing_entity.external_id} to update it.",
)
with logfire.span(
"api.resource.create.write_file",
@@ -203,7 +209,8 @@ async def create_resource(
action="create",
phase="upsert_entity",
):
entity = await entity_repository.add(entity)
async with db.scoped_session(session_maker) as session:
entity = await entity_repository.add(session, entity)
with logfire.span(
"api.resource.create.search_index",
@@ -236,6 +243,7 @@ async def update_resource(
file_service: FileServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
session_maker: SessionMakerDep,
project_id: str = Path(..., description="Project external UUID"),
entity_id: str = Path(..., description="Entity external UUID"),
) -> ResourceResponse:
@@ -265,7 +273,8 @@ async def update_resource(
action="update",
):
try:
entity = await entity_repository.get_by_external_id(entity_id)
async with db.scoped_session(session_maker) as session:
entity = await entity_repository.get_by_external_id(session, entity_id)
if not entity:
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
@@ -315,17 +324,19 @@ async def update_resource(
action="update",
phase="update_entity",
):
updated_entity = await entity_repository.update(
entity.id,
{
"title": file_name,
"note_type": note_type,
"content_type": content_type,
"file_path": target_file_path,
"checksum": checksum,
"updated_at": file_metadata.modified_at,
},
)
async with db.scoped_session(session_maker) as session:
updated_entity = await entity_repository.update(
session,
entity.id,
{
"title": file_name,
"note_type": note_type,
"content_type": content_type,
"file_path": target_file_path,
"checksum": checksum,
"updated_at": file_metadata.modified_at,
},
)
if updated_entity is None:
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
@@ -13,11 +13,13 @@ from pathlib import Path as FilePath
import frontmatter
from fastapi import APIRouter, Path, Query
from loguru import logger
from sqlalchemy.ext.asyncio import AsyncSession
from basic_memory.deps import (
EntityRepositoryV2ExternalDep,
FileServiceV2ExternalDep,
LinkResolverV2ExternalDep,
SessionDep,
)
from basic_memory.models.knowledge import Entity
from basic_memory.schemas.schema import (
@@ -131,6 +133,7 @@ async def validate_schema(
entity_repository: EntityRepositoryV2ExternalDep,
file_service: FileServiceV2ExternalDep,
link_resolver: LinkResolverV2ExternalDep,
session: SessionDep,
project_id: str = Path(..., description="Project external UUID"),
note_type: str | None = Query(None, description="Note type to validate"),
identifier: str | None = Query(None, description="Specific note identifier"),
@@ -150,7 +153,7 @@ async def validate_schema(
if identifier:
# Resolve identifier flexibly (permalink, title, path, fuzzy)
# to match how read_note and other tools resolve identifiers
entity = await link_resolver.resolve_link(identifier)
entity = await link_resolver.resolve_link(identifier, session=session)
if not entity:
return ValidationReport(note_type=note_type, total_notes=0, total_entities=0)
@@ -159,6 +162,7 @@ async def validate_schema(
async def search_fn(query: str) -> list[dict]:
entities = await _find_schema_entities(
session,
entity_repository,
query,
allow_reference_match=isinstance(schema_ref, str) and query == schema_ref,
@@ -187,7 +191,7 @@ async def validate_schema(
)
# --- Batch validation by note type ---
entities = await _find_by_note_type(entity_repository, note_type) if note_type else []
entities = await _find_by_note_type(session, entity_repository, note_type) if note_type else []
for entity in entities:
frontmatter = _entity_frontmatter(entity)
@@ -195,6 +199,7 @@ async def validate_schema(
async def search_fn(query: str) -> list[dict]:
entities = await _find_schema_entities(
session,
entity_repository,
query,
allow_reference_match=isinstance(schema_ref, str) and query == schema_ref,
@@ -230,6 +235,7 @@ async def validate_schema(
@router.post("/schema/infer", response_model=InferenceReport)
async def infer_schema_endpoint(
entity_repository: EntityRepositoryV2ExternalDep,
session: SessionDep,
project_id: str = Path(..., description="Project external UUID"),
note_type: str = Query(..., description="Note type to analyze"),
threshold: float = Query(0.25, description="Minimum frequency for optional fields"),
@@ -239,7 +245,7 @@ async def infer_schema_endpoint(
Examines observation categories and relation types across all notes
of the given type. Returns frequency analysis and suggested Picoschema.
"""
entities = await _find_by_note_type(entity_repository, note_type)
entities = await _find_by_note_type(session, entity_repository, note_type)
notes_data = [_entity_to_note_data(entity) for entity in entities]
result = infer_schema(note_type, notes_data, optional_threshold=threshold)
@@ -274,6 +280,7 @@ async def infer_schema_endpoint(
async def diff_schema_endpoint(
entity_repository: EntityRepositoryV2ExternalDep,
file_service: FileServiceV2ExternalDep,
session: SessionDep,
note_type: str = Path(..., description="Note type to check for drift"),
project_id: str = Path(..., description="Project external UUID"),
):
@@ -285,7 +292,7 @@ async def diff_schema_endpoint(
"""
async def search_fn(query: str) -> list[dict]:
entities = await _find_schema_entities(entity_repository, query)
entities = await _find_schema_entities(session, entity_repository, query)
return [await _schema_frontmatter_from_file(file_service, e) for e in entities]
# Resolve schema by note type
@@ -296,7 +303,7 @@ async def diff_schema_endpoint(
return DriftReport(note_type=note_type, schema_found=False)
# Collect all notes of this type
entities = await _find_by_note_type(entity_repository, note_type)
entities = await _find_by_note_type(session, entity_repository, note_type)
notes_data = [_entity_to_note_data(entity) for entity in entities]
result = diff_schema(schema_def, notes_data)
@@ -331,16 +338,18 @@ async def diff_schema_endpoint(
async def _find_by_note_type(
session: AsyncSession,
entity_repository: EntityRepositoryV2ExternalDep,
note_type: str,
) -> list[Entity]:
"""Find all entities of a given type using the repository's select pattern."""
query = entity_repository.select().where(Entity.note_type == note_type)
result = await entity_repository.execute_query(query)
result = await entity_repository.execute_query(session, query)
return list(result.scalars().all())
async def _find_schema_entities(
session: AsyncSession,
entity_repository: EntityRepositoryV2ExternalDep,
target_note_type: str,
*,
@@ -355,7 +364,7 @@ async def _find_schema_entities(
exact reference matching by title/permalink (explicit schema references)
"""
query = entity_repository.select().where(Entity.note_type == "schema")
result = await entity_repository.execute_query(query)
result = await entity_repository.execute_query(session, query)
entities = list(result.scalars().all())
normalized_target = generate_permalink(target_note_type)
+9 -2
View File
@@ -1,6 +1,7 @@
from typing import Any, Protocol, Optional, List, Sequence
import logfire
from sqlalchemy.ext.asyncio import AsyncSession
from basic_memory.repository.search_repository import SearchIndexRow
from basic_memory.schemas.memory import (
EntitySummary,
@@ -19,7 +20,11 @@ from basic_memory.services.context_service import (
class EntityBatchLookup(Protocol):
async def find_by_ids_for_hydration(
self, ids: List[int], *, include_cross_project: bool = False
self,
session: AsyncSession,
ids: List[int],
*,
include_cross_project: bool = False,
) -> Sequence[Any]: ...
@@ -42,6 +47,7 @@ def _search_item_type(value: str | SearchItemType) -> SearchItemType:
async def to_graph_context(
context_result: ServiceContextResult,
entity_repository: EntityBatchLookup,
session: AsyncSession,
page: Optional[int] = None,
page_size: Optional[int] = None,
) -> GraphContext:
@@ -90,7 +96,7 @@ async def to_graph_context(
result_count=len(entity_ids_needed),
):
entities = await entity_repository.find_by_ids_for_hydration(
list(entity_ids_needed), include_cross_project=True
session, list(entity_ids_needed), include_cross_project=True
)
for e in entities:
entity_title_lookup[e.id] = e.title
@@ -147,6 +153,7 @@ async def to_graph_context(
from_entity_id=item.from_id,
from_entity_external_id=from_ext_id,
to_entity=to_title,
to_name=item.to_name,
to_entity_id=item.to_id,
to_entity_external_id=to_ext_id,
created_at=item.created_at,
+3
View File
@@ -83,8 +83,11 @@ def app_callback(
# Skip for 'mcp' command - it has its own lifespan that handles initialization
# Skip for API-using commands (status, sync, etc.) - they handle initialization via deps.py
# Skip for 'reset' command - it manages its own database lifecycle
# Skip for 'man' - it only copies packaged files; a broken local database
# must not block installing the offline docs
skip_init_commands = {
"doctor",
"man",
"mcp",
"status",
"sync",
+15
View File
@@ -168,6 +168,20 @@ def _manual_update_hint(source: InstallSource) -> str:
)
def _preload_lazy_console_modules() -> None:
"""Import modules the post-upgrade output path defers until print time.
Trigger: an in-place upgrade is about to replace this installation on disk.
Why: rich and typer defer some imports until print/excepthook time; once
`brew upgrade` / `uv tool upgrade` removes the running version's files,
those imports raise ModuleNotFoundError and the final status message
crashes the exiting process.
Outcome: import the deferred modules now, while the files still exist.
"""
import rich._emoji_codes # noqa: F401
import typer.rich_utils # noqa: F401
def _save_last_checked_timestamp(config_manager: ConfigManager, checked_at: datetime) -> None:
"""Persist the timestamp for the most recent attempted update check."""
config = config_manager.load_config()
@@ -288,6 +302,7 @@ def run_auto_update(
else BREW_UPGRADE_TIMEOUT_SECONDS
)
_preload_lazy_console_modules()
install_result = _run_subprocess(
command,
timeout_seconds=timeout,
@@ -4,6 +4,7 @@ from . import ci, status, db, doctor, import_memory_json, mcp, import_claude_con
from . import (
import_claude_projects,
import_chatgpt,
man,
tool,
project,
format,
@@ -29,4 +30,5 @@ __all__ = [
"schema",
"update",
"workspace",
"man",
]
+12 -2
View File
@@ -34,8 +34,10 @@ from basic_memory.ci.project_updates import (
from basic_memory.cli.app import app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.cli.commands.routing import force_routing, validate_routing_flags
from basic_memory.mcp.tools import search_notes as mcp_search_notes
from basic_memory.mcp.tools import write_note as mcp_write_note
# MCP tool functions are imported inside the async helpers below: importing
# basic_memory.mcp.tools loads the entire tool stack (fastmcp, mcp SDK,
# SQLAlchemy), which would slow every CLI invocation, including --help (#886).
console = Console()
@@ -252,6 +254,10 @@ async def seed_project_update_schemas(
refresh: bool = False,
) -> list[str]:
"""Seed Auto BM schema notes without overwriting customized schemas."""
# Deferred: loading the MCP tool stack at module import slows CLI startup (#886).
from basic_memory.mcp.tools import search_notes as mcp_search_notes
from basic_memory.mcp.tools import write_note as mcp_write_note
seeded: list[str] = []
routed_project = _routed_project(project=project, project_id=project_id, workspace=workspace)
for spec in schema_seed_specs():
@@ -295,6 +301,10 @@ async def publish_project_update_note(
note: ProjectUpdateNote,
) -> dict[str, Any]:
"""Search by idempotency key and then upsert the deterministic note path."""
# Deferred: loading the MCP tool stack at module import slows CLI startup (#886).
from basic_memory.mcp.tools import search_notes as mcp_search_notes
from basic_memory.mcp.tools import write_note as mcp_write_note
routed_project = _routed_project(
project=config.project,
project_id=config.project_id,
@@ -11,11 +11,9 @@ from basic_memory.cli.commands.cloud.project_sync import * # noqa: F401,F403
# Register snapshot sub-command group
from basic_memory.cli.commands.cloud.snapshot import snapshot_app
from basic_memory.cli.commands.cloud.workspace import workspace_app
from basic_memory.cli.commands.cloud.shares import share_app
cloud_app.add_typer(snapshot_app, name="snapshot")
cloud_app.add_typer(workspace_app, name="workspace")
cloud_app.add_typer(share_app, name="share")
# Register restore command (directly on cloud_app via decorator)
from basic_memory.cli.commands.cloud.restore import restore # noqa: F401, E402
@@ -1,533 +0,0 @@
"""Public share CLI commands for Basic Memory Cloud.
Surfaces the cloud `/api/shares` endpoints so users can manage public share
links for notes without leaving the terminal:
- POST /api/shares -> create
- GET /api/shares -> list
- PATCH /api/shares/{token} -> update (enable/disable, set expiration)
- DELETE /api/shares/{token} -> revoke
Auth, config lookup, and error handling reuse the shared `make_api_request()`
helper, matching the `snapshot.py` command group.
"""
import asyncio
from datetime import datetime
from typing import Optional
from urllib.parse import urlencode
from uuid import UUID
import typer
from rich.console import Console
from rich.table import Table
from basic_memory.cli.commands.cloud.api_client import (
CloudAPIError,
SubscriptionRequiredError,
make_api_request,
)
from basic_memory.config import ConfigManager
from basic_memory.mcp.async_client import resolve_configured_workspace
from basic_memory.schemas.cloud import WorkspaceInfo
console = Console()
share_app = typer.Typer(help="Manage public share links for notes")
# Header the cloud uses to route a request to a specific tenant's workspace.
# Mirrors basic_memory.cli.commands.cloud.cloud_utils._workspace_headers: the
# cloud /api/shares endpoints resolve the workspace from X-Workspace-ID (see
# resolve_workspace in basic-memory-cloud deps.py), so without it a team
# workspace project would be evaluated against the caller's default tenant.
WORKSPACE_ID_HEADER = "X-Workspace-ID"
def _is_uuid(value: str) -> bool:
"""Return True when value parses as a UUID in any standard textual form."""
try:
UUID(value)
except ValueError:
return False
return True
def _match_workspace_identifier(
workspaces: list[WorkspaceInfo], identifier: str
) -> Optional[WorkspaceInfo]:
"""Match a human workspace identifier with slug > tenant_id > name precedence.
Mirrors project_context._match_workspace_identifier (PR #979): try the stable
slug first (case-insensitive), then the exact tenant_id, then the display name
(case-insensitive). The first tier that yields any match wins, so a display
name colliding with another workspace's slug never shadows the slug match.
"""
slug_matches = [ws for ws in workspaces if ws.slug.casefold() == identifier.casefold()]
if slug_matches:
return slug_matches[0] if len(slug_matches) == 1 else _ambiguous(slug_matches, identifier)
tenant_matches = [ws for ws in workspaces if ws.tenant_id == identifier]
if tenant_matches:
return tenant_matches[0]
name_matches = [ws for ws in workspaces if ws.name.casefold() == identifier.casefold()]
if name_matches:
return name_matches[0] if len(name_matches) == 1 else _ambiguous(name_matches, identifier)
return None
def _ambiguous(matches: list[WorkspaceInfo], identifier: str) -> WorkspaceInfo:
"""Fail with a clear, copyable error when an identifier matches >1 workspace.
Trigger: a display name (or slug, defensively) resolves to multiple workspaces.
Why: silently picking one would route a share to the wrong tenant.
Outcome: list candidate slugs/tenant_ids and exit non-zero so the user re-runs
with an unambiguous slug or tenant_id.
"""
candidates = "\n".join(f" - {ws.slug} (tenant_id: {ws.tenant_id})" for ws in matches)
console.print(
f"[red]Workspace '{identifier}' is ambiguous; it matches multiple workspaces.[/red]\n"
"[yellow]Re-run with a unique workspace slug or tenant_id:[/yellow]\n"
f"{candidates}"
)
raise typer.Exit(1)
async def _resolve_workspace_to_tenant_id(identifier: str) -> str:
"""Resolve a human workspace identifier (slug/name/tenant_id) to a tenant UUID.
Constraint: the cloud's X-Workspace-ID resolver only accepts a workspace/tenant
UUID, but users see slugs and display names (in list-workspaces output and
memory:// URLs). So when --workspace (or the configured default) is not already
a UUID, fetch the caller's workspaces once and map it to the tenant_id here.
get_available_workspaces is the same workspace-fetch seam project.py uses for
CLI workspace resolution; it is awaited directly here because the share
commands already run inside their own event loop.
"""
from basic_memory.mcp.project_context import get_available_workspaces
workspaces = await get_available_workspaces()
match = _match_workspace_identifier(workspaces, identifier)
if match is None:
available = "\n".join(f" - {ws.slug}" for ws in workspaces)
console.print(
f"[red]Workspace '{identifier}' was not found.[/red]\n"
"[yellow]Use one of these workspace slugs (or a tenant_id):[/yellow]\n"
f"{available}"
)
raise typer.Exit(1)
return match.tenant_id
async def _workspace_headers(
*,
project_name: Optional[str] = None,
workspace: Optional[str] = None,
) -> dict[str, str]:
"""Resolve the target workspace and build the routing header, if any.
Resolution chain (see resolve_configured_workspace): explicit --workspace,
then the project's configured workspace_id, then the global default. Returns
an empty dict when nothing resolves so the request falls back to the
caller's default tenant exactly as before.
The cloud's X-Workspace-ID resolver only accepts a workspace/tenant UUID. A
UUID is forwarded verbatim (covers per-project config workspace_id values and
the default chain, zero extra API calls); any other value is treated as a
human identifier and resolved to the tenant UUID via one workspace lookup.
"""
resolved = resolve_configured_workspace(project_name=project_name, workspace=workspace)
if resolved is None:
return {}
if _is_uuid(resolved):
return {WORKSPACE_ID_HEADER: resolved}
tenant_id = await _resolve_workspace_to_tenant_id(resolved)
return {WORKSPACE_ID_HEADER: tenant_id}
def _format_timestamp(iso_timestamp: Optional[str]) -> str:
"""Format an ISO timestamp to a human-readable form, or '-' when absent."""
if not iso_timestamp:
return "-"
try:
dt = datetime.fromisoformat(iso_timestamp.replace("Z", "+00:00"))
return dt.strftime("%Y-%m-%d %H:%M:%S")
except (ValueError, AttributeError):
return iso_timestamp
def _parse_expires_at(value: str) -> str:
"""Validate an --expires-at value and normalize it to an ISO 8601 string.
Accepts either a full ISO timestamp ("2025-12-31T23:59:00") or a bare date
("2025-12-31"). Exits with a clear error on anything we can't parse so the
server never sees a malformed payload.
"""
try:
dt = datetime.fromisoformat(value)
except ValueError:
console.print(
f"[red]Invalid --expires-at value '{value}'. "
"Use ISO format, e.g. 2025-12-31 or 2025-12-31T23:59:00.[/red]"
)
raise typer.Exit(1)
return dt.isoformat()
def _print_share_details(data: dict) -> None:
"""Print a single share's fields in the snapshot-style detail layout."""
console.print(f" Token: {data.get('token', 'unknown')}")
console.print(f" URL: [blue underline]{data.get('share_url', '-')}[/blue underline]")
console.print(f" Project: {data.get('project_name', '-')}")
console.print(f" Note: {data.get('note_permalink', '-')}")
console.print(f" Enabled: {'yes' if data.get('enabled', False) else 'no'}")
console.print(f" Expires: {_format_timestamp(data.get('expires_at'))}")
console.print(f" Views: {data.get('view_count', 0)}")
console.print(f" Created: {_format_timestamp(data.get('created_at'))}")
@share_app.command("create")
def create(
project: str = typer.Argument(
...,
help="Name of the project the note belongs to",
),
permalink: str = typer.Argument(
...,
help="Permalink of the note to share",
),
expires_at: Optional[str] = typer.Option(
None,
"--expires-at",
"-e",
help="Optional expiration date/time (ISO 8601, e.g. 2025-12-31)",
),
workspace: Optional[str] = typer.Option(
None,
"--workspace",
help="Workspace to route to: a workspace slug, display name, or tenant ID",
),
) -> None:
"""Create a public share link for a note.
Examples:
bm cloud share create my-project notes/my-idea
bm cloud share create my-project notes/my-idea --expires-at 2025-12-31
bm cloud share create my-project notes/my-idea --workspace acme
"""
# Validate --expires-at before any async/API work so a parse error surfaces
# a single clean message and exits, rather than being re-wrapped by the broad
# handler below as "Unexpected error: 1" (typer.Exit subclasses Exception).
payload: dict = {
"project_name": project,
"note_permalink": permalink,
}
if expires_at is not None:
payload["expires_at"] = _parse_expires_at(expires_at)
async def _create():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
console.print("[blue]Creating share link...[/blue]")
response = await make_api_request(
method="POST",
url=f"{host_url}/api/shares",
json_data=payload,
headers=await _workspace_headers(project_name=project, workspace=workspace),
)
data = response.json()
console.print("[green]Share link created successfully[/green]")
_print_share_details(data)
except typer.Exit:
raise
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
if e.status_code == 404:
console.print(f"[red]Note not found: {permalink} (project: {project})[/red]")
else:
console.print(f"[red]Failed to create share link: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_create())
@share_app.command("list")
def list_shares(
project: Optional[str] = typer.Option(
None,
"--project",
"-p",
help="Filter shares by project name",
),
workspace: Optional[str] = typer.Option(
None,
"--workspace",
help="Workspace to route to: a workspace slug, display name, or tenant ID",
),
) -> None:
"""List public share links.
Examples:
bm cloud share list
bm cloud share list --project my-project
bm cloud share list --project my-project --workspace acme
"""
async def _list():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
url = f"{host_url}/api/shares"
if project:
# Encode the filter so project names with query-reserved
# characters (&, +, #, spaces) reach the server intact rather
# than being parsed as extra query parameters.
url += f"?{urlencode({'project_name': project})}"
console.print("[blue]Fetching share links...[/blue]")
response = await make_api_request(
method="GET",
url=url,
headers=await _workspace_headers(project_name=project, workspace=workspace),
)
data = response.json()
shares = data.get("shares", [])
total = data.get("total", len(shares))
if not shares:
console.print("[yellow]No share links found[/yellow]")
console.print(
"\n[dim]Create a share with: bm cloud share create <project> <permalink>[/dim]"
)
return
table = Table(title=f"Public Shares ({total} total)")
table.add_column("Token", style="cyan", no_wrap=True)
table.add_column("Project", style="yellow")
table.add_column("Note", style="white")
table.add_column("Enabled", style="green")
table.add_column("Expires", style="green")
table.add_column("Views", style="magenta", justify="right")
table.add_column("URL", style="blue", overflow="fold")
for share in shares:
table.add_row(
share.get("token", "unknown"),
share.get("project_name", "-"),
share.get("note_permalink", "-"),
"yes" if share.get("enabled", False) else "no",
_format_timestamp(share.get("expires_at")),
str(share.get("view_count", 0)),
share.get("share_url", "-"),
)
console.print(table)
# Re-raise typer.Exit before the broad handler below: workspace resolution
# raises typer.Exit (a subclass of Exception) for ambiguous/unknown
# identifiers, and that must not be re-wrapped as "Unexpected error: 1".
except typer.Exit:
raise
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
console.print(f"[red]Failed to list share links: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_list())
@share_app.command("update")
def update(
token: str = typer.Argument(
...,
help="The token of the share to update",
),
enable: bool = typer.Option(
False,
"--enable",
help="Enable the share link",
),
disable: bool = typer.Option(
False,
"--disable",
help="Disable the share link without deleting it",
),
expires_at: Optional[str] = typer.Option(
None,
"--expires-at",
"-e",
help="New expiration date/time (ISO 8601). Use 'none' to clear it.",
),
workspace: Optional[str] = typer.Option(
None,
"--workspace",
help="Workspace the share belongs to: a workspace slug, display name, or tenant ID",
),
) -> None:
"""Update a share link: enable/disable it or change its expiration.
Examples:
bm cloud share update abc123 --disable
bm cloud share update abc123 --enable
bm cloud share update abc123 --expires-at 2026-01-01
bm cloud share update abc123 --expires-at none
bm cloud share update abc123 --disable --workspace acme
"""
async def _update():
try:
# --- Validate flags ---
# Trigger: both toggles passed, or neither toggle and no expiry change.
# Why: PATCH needs at least one concrete field, and enable/disable
# conflict; reject up front so we don't send an empty/ambiguous body.
if enable and disable:
console.print("[red]Cannot use --enable and --disable together[/red]")
raise typer.Exit(1)
if not enable and not disable and expires_at is None:
console.print(
"[red]Nothing to update. Pass --enable, --disable, or --expires-at.[/red]"
)
raise typer.Exit(1)
payload: dict = {}
if enable:
payload["enabled"] = True
if disable:
payload["enabled"] = False
if expires_at is not None:
# "none" clears the expiration; anything else is parsed as a date.
payload["expires_at"] = (
None if expires_at.lower() == "none" else _parse_expires_at(expires_at)
)
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
console.print("[blue]Updating share link...[/blue]")
response = await make_api_request(
method="PATCH",
url=f"{host_url}/api/shares/{token}",
json_data=payload,
headers=await _workspace_headers(workspace=workspace),
)
data = response.json()
console.print("[green]Share link updated successfully[/green]")
_print_share_details(data)
except typer.Exit:
raise
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
if e.status_code == 404:
console.print(f"[red]Share not found: {token}[/red]")
else:
console.print(f"[red]Failed to update share link: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_update())
@share_app.command("revoke")
def revoke(
token: str = typer.Argument(
...,
help="The token of the share to revoke",
),
force: bool = typer.Option(
False,
"--force",
"-f",
help="Skip confirmation prompt",
),
workspace: Optional[str] = typer.Option(
None,
"--workspace",
help="Workspace the share belongs to: a workspace slug, display name, or tenant ID",
),
) -> None:
"""Revoke (delete) a public share link.
Examples:
bm cloud share revoke abc123
bm cloud share revoke abc123 --force
bm cloud share revoke abc123 --force --workspace acme
"""
async def _revoke():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
if not force:
confirmed = typer.confirm(f"Are you sure you want to revoke share '{token}'?")
if not confirmed:
console.print("[yellow]Revocation cancelled[/yellow]")
raise typer.Exit(0)
console.print("[blue]Revoking share link...[/blue]")
await make_api_request(
method="DELETE",
url=f"{host_url}/api/shares/{token}",
headers=await _workspace_headers(workspace=workspace),
)
console.print(f"[green]Share {token} revoked successfully[/green]")
except typer.Exit:
raise
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
if e.status_code == 404:
console.print(f"[red]Share not found: {token}[/red]")
else:
console.print(f"[red]Failed to revoke share link: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_revoke())
@@ -3,12 +3,10 @@
import asyncio
from typing import Optional, TypeVar, Coroutine, Any
from mcp.server.fastmcp.exceptions import ToolError
import typer
from rich.console import Console
from basic_memory import db
from basic_memory.config import ConfigManager
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.clients import ProjectClient
@@ -31,6 +29,9 @@ def run_with_cleanup(coro: Coroutine[Any, Any, T]) -> T:
Returns:
The result of the coroutine
"""
# Deferred: basic_memory.db pulls SQLAlchemy + Alembic, which must not load
# at CLI import time — only when a command actually runs (#886).
from basic_memory import db
async def _with_cleanup() -> T:
try:
@@ -53,6 +54,8 @@ async def run_sync(
force_full: If True, force a full scan bypassing watermark optimization
run_in_background: If True, return immediately; if False, wait for completion
"""
# Deferred: ToolError lives in the mcp SDK, which must not load at CLI startup (#886).
from mcp.server.fastmcp.exceptions import ToolError
# Resolve default project so get_client() can route per-project
project = project or ConfigManager().default_project
@@ -86,6 +89,9 @@ async def run_sync(
async def get_project_info(project: str):
"""Get project information via API endpoint."""
# Deferred: ToolError lives in the mcp SDK, which must not load at CLI startup (#886).
from mcp.server.fastmcp.exceptions import ToolError
try:
async with get_client(project_name=project) as client:
project_item = await get_active_project(client, project, None)
+41 -13
View File
@@ -1,24 +1,27 @@
"""Database management commands."""
# PEP 563 lazy annotations let signatures reference IndexProgress without importing
# the indexing stack at module load; reset/reindex import their heavy database and
# sync dependencies at call time so CLI startup stays fast (#886).
from __future__ import annotations
import os
from dataclasses import dataclass
from pathlib import Path, PurePosixPath, PureWindowsPath
from typing import TYPE_CHECKING
import psutil
import typer
from loguru import logger
from rich.console import Console
from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn, TaskProgressColumn
from sqlalchemy.exc import OperationalError
from basic_memory import db
from basic_memory.cli.app import app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.config import ConfigManager, ProjectMode
from basic_memory.indexing import IndexProgress
from basic_memory.repository import ProjectRepository
from basic_memory.services.initialization import reconcile_projects_with_config
from basic_memory.sync.sync_service import get_sync_service
if TYPE_CHECKING:
from basic_memory.indexing import IndexProgress
console = Console()
@@ -159,6 +162,13 @@ async def _reindex_projects(app_config):
This ensures all database operations use the same event loop,
and proper cleanup happens when the function completes.
"""
# Deferred: SQLAlchemy, repositories, and the sync stack load only when a
# reindex actually runs, not on every CLI start (#886).
from basic_memory import db
from basic_memory.repository import ProjectRepository
from basic_memory.services.initialization import reconcile_projects_with_config
from basic_memory.sync.sync_service import get_sync_service
try:
await reconcile_projects_with_config(app_config)
@@ -167,8 +177,9 @@ async def _reindex_projects(app_config):
db_path=app_config.database_path,
db_type=db.DatabaseType.FILESYSTEM,
)
project_repository = ProjectRepository(session_maker)
projects = await project_repository.get_active_projects()
project_repository = ProjectRepository()
async with db.scoped_session(session_maker) as session:
projects = await project_repository.get_active_projects(session)
for project in projects:
console.print(f" Indexing [cyan]{project.name}[/cyan]...")
@@ -197,6 +208,12 @@ def reset(
),
): # pragma: no cover
"""Reset database (drop all tables and recreate)."""
# Deferred: SQLAlchemy and the db module load only when a reset actually
# runs, not on every CLI start (#886).
from sqlalchemy.exc import OperationalError
from basic_memory import db
console.print(
"[yellow]Note:[/yellow] This only deletes the index database. "
"Your markdown note files will not be affected.\n"
@@ -320,10 +337,15 @@ async def _reindex(
project: str | None,
):
"""Run reindex operations."""
from basic_memory.repository import EntityRepository
# Deferred: SQLAlchemy, repositories, and the sync stack load only when a
# reindex actually runs, not on every CLI start (#886).
from basic_memory import db
from basic_memory.repository import EntityRepository, ProjectRepository
from basic_memory.repository.search_repository import create_search_repository
from basic_memory.services.initialization import reconcile_projects_with_config
from basic_memory.services.search_service import SearchService
from basic_memory.services.file_service import FileService
from basic_memory.sync.sync_service import get_sync_service
from basic_memory.markdown.markdown_processor import MarkdownProcessor
from basic_memory.markdown.entity_parser import EntityParser
@@ -334,8 +356,9 @@ async def _reindex(
db_path=app_config.database_path,
db_type=db.DatabaseType.FILESYSTEM,
)
project_repository = ProjectRepository(session_maker)
projects = await project_repository.get_active_projects()
project_repository = ProjectRepository()
async with db.scoped_session(session_maker) as session:
projects = await project_repository.get_active_projects(session)
if project:
projects = [p for p in projects if p.name == project]
@@ -397,7 +420,7 @@ async def _reindex(
console.print(
f" Building vector embeddings ([cyan]{embedding_mode_label}[/cyan])..."
)
entity_repository = EntityRepository(session_maker, project_id=proj.id)
entity_repository = EntityRepository(project_id=proj.id)
search_repository = create_search_repository(
session_maker, project_id=proj.id, app_config=app_config
)
@@ -405,7 +428,12 @@ async def _reindex(
entity_parser = EntityParser(project_path)
markdown_processor = MarkdownProcessor(entity_parser, app_config=app_config)
file_service = FileService(project_path, markdown_processor, app_config=app_config)
search_service = SearchService(search_repository, entity_repository, file_service)
search_service = SearchService(
search_repository,
entity_repository,
file_service,
session_maker=session_maker,
)
with Progress(
SpinnerColumn(),
+68 -5
View File
@@ -7,16 +7,12 @@ import uuid
from pathlib import Path
from loguru import logger
from mcp.server.fastmcp.exceptions import ToolError
from rich.console import Console
import typer
from basic_memory.cli.app import app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.cli.commands.routing import force_routing, validate_routing_flags
from basic_memory.markdown.entity_parser import EntityParser
from basic_memory.markdown.markdown_processor import MarkdownProcessor
from basic_memory.markdown.schemas import EntityFrontmatter, EntityMarkdown
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.clients import KnowledgeClient, ProjectClient, SearchClient
from basic_memory.schemas.base import Entity
@@ -27,8 +23,72 @@ from basic_memory.schemas import SyncReportResponse
console = Console()
def _is_default_project_delete_error(error: Exception) -> bool:
"""Return True only for the API guard that blocks deleting the default project."""
error_text = str(error)
return "Cannot delete default project" in error_text
async def _delete_doctor_project_locally(project_name: str, project_id: str) -> None:
"""Remove the generated doctor project when the public API guard blocks cleanup."""
from basic_memory import db
from basic_memory.config import ConfigManager
from basic_memory.repository import ProjectRepository
config_manager = ConfigManager()
repository = ProjectRepository()
_, session_maker = await db.get_or_create_db(
db_path=config_manager.config.database_path,
db_type=db.DatabaseType.FILESYSTEM,
)
async with db.scoped_session(session_maker) as session:
project = await repository.get_by_external_id(session, project_id)
if project is None:
raise ValueError(f"Doctor cleanup project '{project_id}' not found")
if project.name != project_name:
raise ValueError(
f"Doctor cleanup expected project '{project_name}', found '{project.name}'"
)
await repository.delete(session, project.id)
config = config_manager.load_config()
if project_name in config.projects:
del config.projects[project_name]
if config.default_project == project_name:
config.default_project = next(iter(config.projects), None)
config_manager.save_config(config)
async def _delete_doctor_project(
project_client: ProjectClient, project_name: str, project_id: str
) -> None:
"""Delete the generated doctor project without weakening the public API guard."""
# Deferred: ToolError lives in the mcp SDK, which must not load at CLI startup (#886).
from mcp.server.fastmcp.exceptions import ToolError
try:
await project_client.delete_project(project_id)
except ToolError as exc:
if not _is_default_project_delete_error(exc):
raise
# Trigger: fresh local configs can promote the generated doctor project
# to default because the placeholder default has no DB row.
# Why: the project is disposable doctor-owned state, while the public API
# must keep rejecting default-project deletion for normal callers.
# Outcome: cleanup removes only the exact doctor project it created.
await _delete_doctor_project_locally(project_name, project_id)
async def run_doctor() -> None:
"""Run local consistency checks for file <-> database flows."""
# Deferred: the markdown parsing stack is only needed while the checks run,
# and importing it at module level slows every CLI invocation (#886).
from basic_memory.markdown.entity_parser import EntityParser
from basic_memory.markdown.markdown_processor import MarkdownProcessor
from basic_memory.markdown.schemas import EntityFrontmatter, EntityMarkdown
console.print("[blue]Running Basic Memory doctor checks...[/blue]")
project_name = f"doctor-{uuid.uuid4().hex[:8]}"
@@ -127,7 +187,7 @@ async def run_doctor() -> None:
finally:
if project_id:
await project_client.delete_project(project_id)
await _delete_doctor_project(project_client, project_name, project_id)
console.print("[green]Doctor checks passed.[/green]")
@@ -140,6 +200,9 @@ def doctor(
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
) -> None:
"""Run local consistency checks to verify file/database sync."""
# Deferred: ToolError lives in the mcp SDK, which must not load at CLI startup (#886).
from mcp.server.fastmcp.exceptions import ToolError
try:
validate_routing_flags(local, cloud)
# Doctor runs local filesystem checks — always default to local routing
@@ -1,25 +1,34 @@
"""Import command for ChatGPT conversations."""
# PEP 563 lazy annotations keep heavy importer types out of module import (#886).
from __future__ import annotations
import json
from pathlib import Path
from typing import Annotated, Tuple
from typing import TYPE_CHECKING, Annotated, Tuple
import typer
from basic_memory.cli.app import import_app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.config import ConfigManager, get_project_config
from basic_memory.importers import ChatGPTImporter
from basic_memory.markdown import EntityParser, MarkdownProcessor
from basic_memory.services.file_service import FileService
from loguru import logger
from rich.console import Console
from rich.panel import Panel
if TYPE_CHECKING:
from basic_memory.markdown import MarkdownProcessor
from basic_memory.services.file_service import FileService
console = Console()
async def get_importer_dependencies() -> Tuple[MarkdownProcessor, FileService]:
"""Get MarkdownProcessor and FileService instances for importers."""
# Deferred: the markdown/file-service stack pulls SQLAlchemy and must load
# only when an import actually runs, not on every CLI start (#886).
from basic_memory.markdown import EntityParser, MarkdownProcessor
from basic_memory.services.file_service import FileService
config = get_project_config()
app_config = ConfigManager().config
entity_parser = EntityParser(config.home)
@@ -60,6 +69,9 @@ def import_chatgpt(
console.print(f"\nImporting chats from {conversations_json}...writing to {base_path}")
# Create importer and run import
# Deferred: importer stack loads at import-command run time only (#886).
from basic_memory.importers import ChatGPTImporter
importer = ChatGPTImporter(
config.home, markdown_processor, file_service, project_name=config.name
)
@@ -1,25 +1,34 @@
"""Import command for basic-memory CLI to import chat data from conversations2.json format."""
# PEP 563 lazy annotations keep heavy importer types out of module import (#886).
from __future__ import annotations
import json
from pathlib import Path
from typing import Annotated, Tuple
from typing import TYPE_CHECKING, Annotated, Tuple
import typer
from basic_memory.cli.app import claude_app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.config import ConfigManager, get_project_config
from basic_memory.importers.claude_conversations_importer import ClaudeConversationsImporter
from basic_memory.markdown import EntityParser, MarkdownProcessor
from basic_memory.services.file_service import FileService
from loguru import logger
from rich.console import Console
from rich.panel import Panel
if TYPE_CHECKING:
from basic_memory.markdown import MarkdownProcessor
from basic_memory.services.file_service import FileService
console = Console()
async def get_importer_dependencies() -> Tuple[MarkdownProcessor, FileService]:
"""Get MarkdownProcessor and FileService instances for importers."""
# Deferred: the markdown/file-service stack pulls SQLAlchemy and must load
# only when an import actually runs, not on every CLI start (#886).
from basic_memory.markdown import EntityParser, MarkdownProcessor
from basic_memory.services.file_service import FileService
config = get_project_config()
app_config = ConfigManager().config
entity_parser = EntityParser(config.home)
@@ -57,6 +66,9 @@ def import_claude(
markdown_processor, file_service = run_with_cleanup(get_importer_dependencies())
# Create the importer
# Deferred: importer stack loads at import-command run time only (#886).
from basic_memory.importers.claude_conversations_importer import ClaudeConversationsImporter
importer = ClaudeConversationsImporter(
config.home, markdown_processor, file_service, project_name=config.name
)
@@ -1,25 +1,34 @@
"""Import command for basic-memory CLI to import project data from Claude.ai."""
# PEP 563 lazy annotations keep heavy importer types out of module import (#886).
from __future__ import annotations
import json
from pathlib import Path
from typing import Annotated, Tuple
from typing import TYPE_CHECKING, Annotated, Tuple
import typer
from basic_memory.cli.app import claude_app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.config import ConfigManager, get_project_config
from basic_memory.importers.claude_projects_importer import ClaudeProjectsImporter
from basic_memory.markdown import EntityParser, MarkdownProcessor
from basic_memory.services.file_service import FileService
from loguru import logger
from rich.console import Console
from rich.panel import Panel
if TYPE_CHECKING:
from basic_memory.markdown import MarkdownProcessor
from basic_memory.services.file_service import FileService
console = Console()
async def get_importer_dependencies() -> Tuple[MarkdownProcessor, FileService]:
"""Get MarkdownProcessor and FileService instances for importers."""
# Deferred: the markdown/file-service stack pulls SQLAlchemy and must load
# only when an import actually runs, not on every CLI start (#886).
from basic_memory.markdown import EntityParser, MarkdownProcessor
from basic_memory.services.file_service import FileService
config = get_project_config()
app_config = ConfigManager().config
entity_parser = EntityParser(config.home)
@@ -56,6 +65,9 @@ def import_projects(
markdown_processor, file_service = run_with_cleanup(get_importer_dependencies())
# Create the importer
# Deferred: importer stack loads at import-command run time only (#886).
from basic_memory.importers.claude_projects_importer import ClaudeProjectsImporter
importer = ClaudeProjectsImporter(
config.home, markdown_processor, file_service, project_name=config.name
)
@@ -1,25 +1,34 @@
"""Import command for basic-memory CLI to import from JSON memory format."""
# PEP 563 lazy annotations keep heavy importer types out of module import (#886).
from __future__ import annotations
import json
from pathlib import Path
from typing import Annotated, Tuple
from typing import TYPE_CHECKING, Annotated, Tuple
import typer
from basic_memory.cli.app import import_app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.config import ConfigManager, get_project_config
from basic_memory.importers.memory_json_importer import MemoryJsonImporter
from basic_memory.markdown import EntityParser, MarkdownProcessor
from basic_memory.services.file_service import FileService
from loguru import logger
from rich.console import Console
from rich.panel import Panel
if TYPE_CHECKING:
from basic_memory.markdown import MarkdownProcessor
from basic_memory.services.file_service import FileService
console = Console()
async def get_importer_dependencies() -> Tuple[MarkdownProcessor, FileService]:
"""Get MarkdownProcessor and FileService instances for importers."""
# Deferred: the markdown/file-service stack pulls SQLAlchemy and must load
# only when an import actually runs, not on every CLI start (#886).
from basic_memory.markdown import EntityParser, MarkdownProcessor
from basic_memory.services.file_service import FileService
config = get_project_config()
app_config = ConfigManager().config
entity_parser = EntityParser(config.home)
@@ -55,6 +64,9 @@ def memory_json(
markdown_processor, file_service = run_with_cleanup(get_importer_dependencies())
# Create the importer
# Deferred: importer stack loads at import-command run time only (#886).
from basic_memory.importers.memory_json_importer import MemoryJsonImporter
importer = MemoryJsonImporter(
config.home, markdown_processor, file_service, project_name=config.name
)
+76
View File
@@ -0,0 +1,76 @@
"""Install the bundled man pages so `man bm` works."""
import shutil
import subprocess
from pathlib import Path
from typing import Annotated, Optional
import typer
from rich.console import Console
from basic_memory.cli.app import app
console = Console()
man_app = typer.Typer(help="Manage the bm man pages.")
app.add_typer(man_app, name="man")
# Bundled groff sources ship inside the package (src/basic_memory/man).
_MAN_SOURCE_DIR = Path(__file__).parent.parent.parent / "man"
def _default_man_root() -> Path:
# Why ~/.local/share/man: manpath(1) derives man directories from PATH
# entries on both man-db (Linux) and BSD man (macOS), so ~/.local/bin on
# PATH — the pipx/uv tool layout — makes this root searchable without any
# MANPATH configuration.
return Path.home() / ".local" / "share" / "man"
def _man_root_on_manpath(man_root: Path) -> Optional[bool]:
"""Best-effort check whether man(1) will search man_root; None if unknown."""
try:
result = subprocess.run(["manpath"], capture_output=True, text=True, timeout=5)
except (FileNotFoundError, subprocess.TimeoutExpired):
return None
if result.returncode != 0:
return None
paths = [entry.rstrip("/") for entry in result.stdout.strip().split(":") if entry]
return str(man_root).rstrip("/") in paths
@man_app.command()
def install(
directory: Annotated[
Optional[Path],
typer.Option(
"--dir",
help="Man root to install into (default: ~/.local/share/man)",
),
] = None,
) -> None:
"""Install the bm man pages, then try `man bm`."""
man_root = (directory or _default_man_root()).expanduser()
man1 = man_root / "man1"
man1.mkdir(parents=True, exist_ok=True)
pages = sorted(_MAN_SOURCE_DIR.glob("*.1"))
if not pages: # pragma: no cover - broken packaging, not a runtime state
console.print("[red]No bundled man pages found — broken installation[/red]")
raise typer.Exit(1)
for page in pages:
shutil.copyfile(page, man1 / page.name)
console.print(f"installed {man1 / page.name}")
# Trigger: the chosen root is provably absent from manpath output.
# Why: a silent install into an unsearched directory looks like success
# but `man bm` still fails; say so and hand over the one-line fix.
# Outcome: actionable hint; unknown (None) stays quiet to avoid false alarms.
if _man_root_on_manpath(man_root) is False:
console.print(
f"\n[yellow]{man_root} is not on your manpath.[/yellow] Add it with:\n"
f' export MANPATH="{man_root}:$MANPATH"'
)
console.print("\nTry: [bold]man bm[/bold]")
+3 -1
View File
@@ -5,7 +5,6 @@ from typing import Annotated, Optional
import typer
from loguru import logger
from mcp.server.fastmcp.exceptions import ToolError
from rich.console import Console
from rich.table import Table
@@ -50,6 +49,9 @@ def orphans(
"""
from basic_memory.cli.commands.command_utils import run_with_cleanup
# Deferred: ToolError lives in the mcp SDK, which must not load at CLI startup (#886).
from mcp.server.fastmcp.exceptions import ToolError
try:
validate_routing_flags(local, cloud)
with force_routing(local=local, cloud=cloud):
+23 -20
View File
@@ -1050,12 +1050,13 @@ async def _detach_local_project_row(app_config: BasicMemoryConfig, name: str) ->
db_type=db.DatabaseType.FILESYSTEM,
)
try:
repo = ProjectRepository(session_maker)
existing = await repo.get_by_name(name)
if existing is None:
return False
await repo.delete(existing.id)
return True
repo = ProjectRepository()
async with db.scoped_session(session_maker) as session:
existing = await repo.get_by_name(session, name)
if existing is None:
return False
await repo.delete(session, existing.id)
return True
finally:
# CLI-only: safe to tear down the global DB singleton here since
# set-cloud/set-local never run inside a long-lived MCP/API server.
@@ -1082,20 +1083,22 @@ async def _attach_local_project_row(app_config: BasicMemoryConfig, name: str, pa
db_type=db.DatabaseType.FILESYSTEM,
)
try:
repo = ProjectRepository(session_maker)
existing = await repo.get_by_name(name)
if existing is None:
await repo.create(
{
"name": name,
"path": path,
"permalink": generate_permalink(name),
"is_active": True,
}
)
return
if existing.path != path:
await repo.update_path(existing.id, path)
repo = ProjectRepository()
async with db.scoped_session(session_maker) as session:
existing = await repo.get_by_name(session, name)
if existing is None:
await repo.create(
session,
{
"name": name,
"path": path,
"permalink": generate_permalink(name),
"is_active": True,
},
)
return
if existing.path != path:
await repo.update_path(session, existing.id, path)
finally:
# CLI-only: safe to tear down the global DB singleton here since
# set-cloud/set-local never run inside a long-lived MCP/API server.
+13 -3
View File
@@ -20,9 +20,10 @@ from basic_memory.cli.app import app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.cli.commands.routing import force_routing, validate_routing_flags
from basic_memory.config import ConfigManager
from basic_memory.mcp.tools import schema_diff as mcp_schema_diff
from basic_memory.mcp.tools import schema_infer as mcp_schema_infer
from basic_memory.mcp.tools import schema_validate as mcp_schema_validate
# MCP tool functions are imported inside each command: importing
# basic_memory.mcp.tools loads the entire tool stack (fastmcp, mcp SDK,
# SQLAlchemy), which would slow every CLI invocation, including --help (#886).
console = Console()
@@ -189,6 +190,9 @@ def validate(
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
# Deferred: loading the MCP tool stack at module import slows CLI startup (#886).
from basic_memory.mcp.tools import schema_validate as mcp_schema_validate
try:
validate_routing_flags(local, cloud)
project_name = _resolve_project_name(project)
@@ -272,6 +276,9 @@ def infer(
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
# Deferred: loading the MCP tool stack at module import slows CLI startup (#886).
from basic_memory.mcp.tools import schema_infer as mcp_schema_infer
try:
validate_routing_flags(local, cloud)
project_name = _resolve_project_name(project)
@@ -352,6 +359,9 @@ def diff(
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
# Deferred: loading the MCP tool stack at module import slows CLI startup (#886).
from basic_memory.mcp.tools import schema_diff as mcp_schema_diff
try:
validate_routing_flags(local, cloud)
project_name = _resolve_project_name(project)
+11 -2
View File
@@ -5,7 +5,6 @@ import json
import time
from typing import Annotated, Dict, Optional, Set
from mcp.server.fastmcp.exceptions import ToolError
import typer
from loguru import logger
from rich.console import Console
@@ -190,9 +189,16 @@ async def run_status(
if sync_report.total == 0:
return project_item.name, sync_report
if time.monotonic() >= deadline:
# Why the hint: indexing is done by the sync coordinator, which
# only runs inside a live server (bm mcp / hosted API). In a
# CLI-only session nothing will ever drain the pending count,
# so this wait cannot succeed — point at the command that
# actually indexes (#959).
raise StatusTimeout(
f"Timed out after {timeout:g}s waiting for '{project_item.name}' "
f"to finish indexing ({sync_report.total} pending change(s) remaining)."
f"to finish indexing ({sync_report.total} pending change(s) remaining). "
f"If no Basic Memory server is running, pending changes are never "
f"indexed — run 'bm reindex --project {project_item.name}' instead."
)
await asyncio.sleep(poll_interval)
@@ -224,6 +230,9 @@ def status(
"""
from basic_memory.cli.commands.command_utils import run_with_cleanup
# Deferred: ToolError lives in the mcp SDK, which must not load at CLI startup (#886).
from mcp.server.fastmcp.exceptions import ToolError
# Trigger: --wait with a negative --timeout
# Why: a negative deadline times out on the very first poll, producing a confusing
# "Timed out after -5s" message instead of flagging the bad input. Raised
+40 -12
View File
@@ -14,18 +14,10 @@ from loguru import logger
from basic_memory.cli.app import app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.cli.commands.routing import force_routing, validate_routing_flags
from basic_memory.mcp.tools import build_context as mcp_build_context
from basic_memory.mcp.tools import delete_note as mcp_delete_note
from basic_memory.mcp.tools import edit_note as mcp_edit_note
from basic_memory.mcp.tools import list_memory_projects as mcp_list_projects
from basic_memory.mcp.tools import list_workspaces as mcp_list_workspaces
from basic_memory.mcp.tools import read_note as mcp_read_note
from basic_memory.mcp.tools import recent_activity as mcp_recent_activity
from basic_memory.mcp.tools import schema_diff as mcp_schema_diff
from basic_memory.mcp.tools import schema_infer as mcp_schema_infer
from basic_memory.mcp.tools import schema_validate as mcp_schema_validate
from basic_memory.mcp.tools import search_notes as mcp_search
from basic_memory.mcp.tools import write_note as mcp_write_note
# MCP tool functions are imported inside each command: importing
# basic_memory.mcp.tools loads the entire tool stack (fastmcp, mcp SDK,
# SQLAlchemy), which would slow every CLI invocation, including --help (#886).
tool_app = typer.Typer()
app.add_typer(tool_app, name="tool", help="Access to MCP tools via CLI")
@@ -120,6 +112,9 @@ def write_note(
bm tool write-note --title "My Note" --folder "notes" --overwrite
bm tool write-note --title "My Note" --folder "notes" --local
"""
# Deferred: loading the MCP tool stack at module import slows CLI startup (#886).
from basic_memory.mcp.tools import write_note as mcp_write_note
try:
validate_routing_flags(local, cloud)
@@ -206,6 +201,9 @@ def read_note(
bm tool read-note my-note
bm tool read-note my-note --include-frontmatter
"""
# Deferred: loading the MCP tool stack at module import slows CLI startup (#886).
from basic_memory.mcp.tools import read_note as mcp_read_note
try:
validate_routing_flags(local, cloud)
@@ -272,6 +270,9 @@ def delete_note(
bm tool delete-note notes/old-draft
bm tool delete-note docs/archive --is-directory
"""
# Deferred: loading the MCP tool stack at module import slows CLI startup (#886).
from basic_memory.mcp.tools import delete_note as mcp_delete_note
try:
validate_routing_flags(local, cloud)
@@ -346,6 +347,9 @@ def edit_note(
bm tool edit-note my-note --operation find_replace --find-text "old" --content "new"
bm tool edit-note my-note --operation replace_section --section "## Notes" --content "updated"
"""
# Deferred: loading the MCP tool stack at module import slows CLI startup (#886).
from basic_memory.mcp.tools import edit_note as mcp_edit_note
try:
validate_routing_flags(local, cloud)
@@ -413,6 +417,9 @@ def build_context(
bm tool build-context memory://specs/search
bm tool build-context specs/search --depth 2 --timeframe 30d
"""
# Deferred: loading the MCP tool stack at module import slows CLI startup (#886).
from basic_memory.mcp.tools import build_context as mcp_build_context
try:
validate_routing_flags(local, cloud)
@@ -476,6 +483,9 @@ def recent_activity(
bm tool recent-activity --timeframe 30d --page-size 20
bm tool recent-activity --type entity --type observation
"""
# Deferred: loading the MCP tool stack at module import slows CLI startup (#886).
from basic_memory.mcp.tools import recent_activity as mcp_recent_activity
try:
validate_routing_flags(local, cloud)
@@ -582,6 +592,9 @@ def search_notes(
bm tool search-notes --meta status=draft
bm tool search-notes "auth" --entity-type observation --category requirement
"""
# Deferred: loading the MCP tool stack at module import slows CLI startup (#886).
from basic_memory.mcp.tools import search_notes as mcp_search
try:
validate_routing_flags(local, cloud)
@@ -687,6 +700,9 @@ def list_projects(
bm tool list-projects
bm tool list-projects --local
"""
# Deferred: loading the MCP tool stack at module import slows CLI startup (#886).
from basic_memory.mcp.tools import list_memory_projects as mcp_list_projects
try:
validate_routing_flags(local, cloud)
@@ -720,6 +736,9 @@ def list_workspaces(
bm tool list-workspaces
bm tool list-workspaces --cloud
"""
# Deferred: loading the MCP tool stack at module import slows CLI startup (#886).
from basic_memory.mcp.tools import list_workspaces as mcp_list_workspaces
try:
validate_routing_flags(local, cloud)
@@ -772,6 +791,9 @@ def schema_validate(
bm tool schema-validate people/ada-lovelace.md
bm tool schema-validate --project research
"""
# Deferred: loading the MCP tool stack at module import slows CLI startup (#886).
from basic_memory.mcp.tools import schema_validate as mcp_schema_validate
try:
validate_routing_flags(local, cloud)
@@ -840,6 +862,9 @@ def schema_infer(
bm tool schema-infer meeting --threshold 0.5
bm tool schema-infer person --project research
"""
# Deferred: loading the MCP tool stack at module import slows CLI startup (#886).
from basic_memory.mcp.tools import schema_infer as mcp_schema_infer
try:
validate_routing_flags(local, cloud)
@@ -896,6 +921,9 @@ def schema_diff(
bm tool schema-diff person
bm tool schema-diff person --project research
"""
# Deferred: loading the MCP tool stack at module import slows CLI startup (#886).
from basic_memory.mcp.tools import schema_diff as mcp_schema_diff
try:
validate_routing_flags(local, cloud)
+1
View File
@@ -24,6 +24,7 @@ if not _version_only_invocation(sys.argv[1:]):
import_claude_conversations,
import_claude_projects,
import_memory_json,
man,
mcp,
orphans,
project,
+36 -2
View File
@@ -4,6 +4,7 @@ import importlib.util
import json
import os
import shutil
import threading
from dataclasses import dataclass
from datetime import datetime
from enum import Enum
@@ -657,6 +658,26 @@ class BasicMemoryConfig(BaseSettings):
entry = self.projects.get(project_name)
return entry.mode if entry else ProjectMode.CLOUD
def is_locally_syncable(self, project_name: str, project_path: str) -> bool:
"""Whether a project should be synced/watched on the local filesystem.
Both conditions are required (issue #949):
* The project is present in config. Config is the source of truth, so a
stale database row that was removed from config but whose deletion
has not yet been reconciled, or whose reconciliation failed must
not be synced even though it still has a real directory on disk.
* Its path is absolute. An empty or relative path resolves against the
process cwd, so syncing it would adopt whatever directory the server
was launched from as the project root and mutate unrelated files.
Cloud-only projects (empty/slug path) and cloud projects with a real
local bisync copy (absolute path) are handled correctly by these two
conditions, so no separate mode check is needed.
"""
entry = self.projects.get(project_name)
return entry is not None and Path(project_path).is_absolute()
def set_project_mode(self, project_name: str, mode: ProjectMode) -> None:
"""Set the routing mode for a project.
@@ -1099,8 +1120,21 @@ def save_basic_memory_config(file_path: Path, config: BasicMemoryConfig) -> None
_secure_config_dir(file_path.parent)
# Use model_dump with mode='json' to serialize datetime objects properly
config_dict = config.model_dump(mode="json")
file_path.write_text(json.dumps(config_dict, indent=2))
_secure_config_file(file_path)
# Trigger: long-lived readers (MCP stdio server config reload, background
# auto-update threads) re-read config.json whenever its mtime changes,
# concurrently with CLI commands saving it.
# Why: writing the destination in place truncates it first, so a concurrent
# reader can observe empty/partial JSON and load_config() exits the process.
# Outcome: write a sibling temp file (unique per process/thread so parallel
# savers cannot interleave) and publish atomically via os.replace — readers
# always see either the old or the new complete document. (#940)
tmp_path = file_path.parent / f"{file_path.name}.{os.getpid()}.{threading.get_ident()}.tmp"
try:
tmp_path.write_text(json.dumps(config_dict, indent=2))
_secure_config_file(tmp_path)
os.replace(tmp_path, file_path)
finally:
tmp_path.unlink(missing_ok=True)
except Exception as e: # pragma: no cover
logger.error(f"Failed to save config: {e}")
+26 -15
View File
@@ -19,7 +19,7 @@ from sqlalchemy.ext.asyncio import (
AsyncEngine,
async_scoped_session,
)
from sqlalchemy.pool import NullPool
from sqlalchemy.pool import AsyncAdaptedQueuePool, NullPool
from basic_memory.repository.postgres_search_repository import PostgresSearchRepository
from basic_memory.repository.sqlite_search_repository import SQLiteSearchRepository
@@ -206,29 +206,40 @@ def _create_sqlite_engine(db_url: str, db_type: DatabaseType) -> AsyncEngine:
Configured async engine for SQLite
"""
# Configure connection args with Windows-specific settings
connect_args: dict[str, bool | float | None] = {"check_same_thread": False}
connect_args: dict[str, bool | float] = {"check_same_thread": False}
# Add Windows-specific parameters to improve reliability
if os.name == "nt": # Windows
connect_args.update(
{
"timeout": 30.0, # Increase timeout to 30 seconds for Windows
"isolation_level": None, # Use autocommit mode
}
)
if db_type == DatabaseType.MEMORY:
# Trigger: an in-memory SQLite URL would default to StaticPool, which hands the
# same DBAPI connection to every concurrently checked-out session.
# Why: concurrent asyncio tasks then share one transaction scope — a rollback
# issued by one session (scoped_session exception handling or the pool's
# reset-on-return) silently destroys another session's uncommitted writes (#940).
# Outcome: a single-connection blocking queue pool keeps the in-memory database
# alive for the engine's lifetime while serializing sessions at transaction
# granularity, restoring the isolation the repositories assume.
engine = create_async_engine(
db_url,
connect_args=connect_args,
poolclass=AsyncAdaptedQueuePool,
pool_size=1,
max_overflow=0,
)
elif os.name == "nt":
# Use NullPool for Windows filesystem databases to avoid connection pooling issues
# Important: Do NOT use NullPool for in-memory databases as it will destroy the database
# between connections
if db_type == DatabaseType.FILESYSTEM:
engine = create_async_engine(
db_url,
connect_args=connect_args,
poolclass=NullPool, # Disable connection pooling on Windows
echo=False,
)
else:
# In-memory databases need connection pooling to maintain state
engine = create_async_engine(db_url, connect_args=connect_args)
engine = create_async_engine(
db_url,
connect_args=connect_args,
poolclass=NullPool, # Disable connection pooling on Windows
echo=False,
)
else:
engine = create_async_engine(db_url, connect_args=connect_args)
+4
View File
@@ -25,6 +25,8 @@ from basic_memory.deps.db import (
EngineFactoryDep,
get_session_maker,
SessionMakerDep,
get_session,
SessionDep,
)
from basic_memory.deps.projects import (
@@ -169,6 +171,8 @@ __all__ = [
"EngineFactoryDep",
"get_session_maker",
"SessionMakerDep",
"get_session",
"SessionDep",
# Projects
"get_project_repository",
"ProjectRepositoryDep",
+10
View File
@@ -5,6 +5,7 @@ This module provides database-related dependencies:
- Session dependencies for request handling
"""
from collections.abc import AsyncGenerator
from typing import Annotated
from fastapi import Depends, Request
@@ -54,3 +55,12 @@ async def get_session_maker(engine_factory: EngineFactoryDep) -> async_sessionma
SessionMakerDep = Annotated[async_sessionmaker, Depends(get_session_maker)]
async def get_session(session_maker: SessionMakerDep) -> AsyncGenerator[AsyncSession, None]:
"""Yield a request-scoped SQLAlchemy session."""
async with db.scoped_session(session_maker) as session:
yield session
SessionDep = Annotated[AsyncSession, Depends(get_session)]
+51 -35
View File
@@ -12,6 +12,7 @@ from typing import Annotated
from fastapi import Depends, HTTPException, Path, status
from basic_memory import db
from basic_memory.config import ProjectConfig
from basic_memory.deps.db import SessionMakerDep
from basic_memory.repository.project_repository import ProjectRepository
@@ -21,11 +22,9 @@ from basic_memory.utils import generate_permalink
# --- Project Repository ---
async def get_project_repository(
session_maker: SessionMakerDep,
) -> ProjectRepository:
async def get_project_repository() -> ProjectRepository:
"""Get the project repository."""
return ProjectRepository(session_maker)
return ProjectRepository()
ProjectRepositoryDep = Annotated[ProjectRepository, Depends(get_project_repository)]
@@ -41,6 +40,7 @@ ProjectPathDep = Annotated[str, Path()]
async def get_project_id(
session_maker: SessionMakerDep,
project_repository: ProjectRepositoryDep,
project: ProjectPathDep,
) -> int:
@@ -61,14 +61,17 @@ async def get_project_id(
"""
# Convert project name to permalink for lookup
project_permalink = generate_permalink(str(project))
project_obj = await project_repository.get_by_permalink(project_permalink)
if project_obj:
return project_obj.id
async with db.scoped_session(session_maker) as session:
project_obj = await project_repository.get_by_permalink(session, project_permalink)
if project_obj:
return project_obj.id
# Try by name if permalink lookup fails
project_obj = await project_repository.get_by_name(str(project)) # pragma: no cover
if project_obj: # pragma: no cover
return project_obj.id
# Try by name if permalink lookup fails
project_obj = await project_repository.get_by_name(
session, str(project)
) # pragma: no cover
if project_obj: # pragma: no cover
return project_obj.id
# Not found
raise HTTPException( # pragma: no cover
@@ -83,7 +86,9 @@ ProjectIdDep = Annotated[int, Depends(get_project_id)]
async def get_project_config(
project: ProjectPathDep, project_repository: ProjectRepositoryDep
session_maker: SessionMakerDep,
project: ProjectPathDep,
project_repository: ProjectRepositoryDep,
) -> ProjectConfig: # pragma: no cover
"""Get the current project referenced from request state.
@@ -99,9 +104,10 @@ async def get_project_config(
"""
# Convert project name to permalink for lookup
project_permalink = generate_permalink(str(project))
project_obj = await project_repository.get_by_permalink(project_permalink)
if project_obj:
return ProjectConfig(name=project_obj.name, home=pathlib.Path(project_obj.path))
async with db.scoped_session(session_maker) as session:
project_obj = await project_repository.get_by_permalink(session, project_permalink)
if project_obj:
return ProjectConfig(name=project_obj.name, home=pathlib.Path(project_obj.path))
# Not found
raise HTTPException( # pragma: no cover
@@ -116,6 +122,7 @@ ProjectConfigDep = Annotated[ProjectConfig, Depends(get_project_config)]
async def validate_project_id(
session_maker: SessionMakerDep,
project_id: int,
project_repository: ProjectRepositoryDep,
) -> int:
@@ -134,12 +141,13 @@ async def validate_project_id(
Raises:
HTTPException: If project with that ID is not found
"""
project_obj = await project_repository.get_by_id(project_id)
if not project_obj:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail=f"Project with ID {project_id} not found.",
)
async with db.scoped_session(session_maker) as session:
project_obj = await project_repository.get_by_id(session, project_id)
if not project_obj:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail=f"Project with ID {project_id} not found.",
)
return project_id
@@ -147,7 +155,9 @@ ProjectIdPathDep = Annotated[int, Depends(validate_project_id)]
async def get_project_config_v2(
project_id: ProjectIdPathDep, project_repository: ProjectRepositoryDep
session_maker: SessionMakerDep,
project_id: ProjectIdPathDep,
project_repository: ProjectRepositoryDep,
) -> ProjectConfig: # pragma: no cover
"""Get the project config for v2 API (uses integer project_id from path).
@@ -161,9 +171,10 @@ async def get_project_config_v2(
Raises:
HTTPException: If project is not found
"""
project_obj = await project_repository.get_by_id(project_id)
if project_obj:
return ProjectConfig(name=project_obj.name, home=pathlib.Path(project_obj.path))
async with db.scoped_session(session_maker) as session:
project_obj = await project_repository.get_by_id(session, project_id)
if project_obj:
return ProjectConfig(name=project_obj.name, home=pathlib.Path(project_obj.path))
# Not found (this should not happen since ProjectIdPathDep already validates existence)
raise HTTPException( # pragma: no cover
@@ -178,6 +189,7 @@ ProjectConfigV2Dep = Annotated[ProjectConfig, Depends(get_project_config_v2)]
async def validate_project_external_id(
session_maker: SessionMakerDep,
project_id: str,
project_repository: ProjectRepositoryDep,
) -> int:
@@ -196,12 +208,13 @@ async def validate_project_external_id(
Raises:
HTTPException: If project with that external_id is not found
"""
project_obj = await project_repository.get_by_external_id(project_id)
if not project_obj:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail=f"Project with external_id '{project_id}' not found.",
)
async with db.scoped_session(session_maker) as session:
project_obj = await project_repository.get_by_external_id(session, project_id)
if not project_obj:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail=f"Project with external_id '{project_id}' not found.",
)
return project_obj.id
@@ -209,7 +222,9 @@ ProjectExternalIdPathDep = Annotated[int, Depends(validate_project_external_id)]
async def get_project_config_v2_external(
project_id: ProjectExternalIdPathDep, project_repository: ProjectRepositoryDep
session_maker: SessionMakerDep,
project_id: ProjectExternalIdPathDep,
project_repository: ProjectRepositoryDep,
) -> ProjectConfig: # pragma: no cover
"""Get the project config for v2 API (uses external_id UUID from path).
@@ -223,9 +238,10 @@ async def get_project_config_v2_external(
Raises:
HTTPException: If project is not found
"""
project_obj = await project_repository.get_by_id(project_id)
if project_obj:
return ProjectConfig(name=project_obj.name, home=pathlib.Path(project_obj.path))
async with db.scoped_session(session_maker) as session:
project_obj = await project_repository.get_by_id(session, project_id)
if project_obj:
return ProjectConfig(name=project_obj.name, home=pathlib.Path(project_obj.path))
# Not found (this should not happen since ProjectExternalIdPathDep already validates)
raise HTTPException( # pragma: no cover
+9 -18
View File
@@ -30,33 +30,30 @@ from basic_memory.repository.search_repository import SearchRepository, create_s
async def get_entity_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
) -> EntityRepository:
"""Create an EntityRepository instance for the current project."""
return EntityRepository(session_maker, project_id=project_id)
return EntityRepository(project_id=project_id)
EntityRepositoryDep = Annotated[EntityRepository, Depends(get_entity_repository)]
async def get_entity_repository_v2( # pragma: no cover
session_maker: SessionMakerDep,
project_id: ProjectIdPathDep,
) -> EntityRepository:
"""Create an EntityRepository instance for v2 API (uses integer project_id from path)."""
return EntityRepository(session_maker, project_id=project_id)
return EntityRepository(project_id=project_id)
EntityRepositoryV2Dep = Annotated[EntityRepository, Depends(get_entity_repository_v2)]
async def get_entity_repository_v2_external(
session_maker: SessionMakerDep,
project_id: ProjectExternalIdPathDep,
) -> EntityRepository:
"""Create an EntityRepository instance for v2 API (uses external_id from path)."""
return EntityRepository(session_maker, project_id=project_id)
return EntityRepository(project_id=project_id)
EntityRepositoryV2ExternalDep = Annotated[
@@ -68,22 +65,20 @@ EntityRepositoryV2ExternalDep = Annotated[
async def get_observation_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
) -> ObservationRepository:
"""Create an ObservationRepository instance for the current project."""
return ObservationRepository(session_maker, project_id=project_id)
return ObservationRepository(project_id=project_id)
ObservationRepositoryDep = Annotated[ObservationRepository, Depends(get_observation_repository)]
async def get_observation_repository_v2( # pragma: no cover
session_maker: SessionMakerDep,
project_id: ProjectIdPathDep,
) -> ObservationRepository:
"""Create an ObservationRepository instance for v2 API."""
return ObservationRepository(session_maker, project_id=project_id)
return ObservationRepository(project_id=project_id)
ObservationRepositoryV2Dep = Annotated[
@@ -92,11 +87,10 @@ ObservationRepositoryV2Dep = Annotated[
async def get_observation_repository_v2_external(
session_maker: SessionMakerDep,
project_id: ProjectExternalIdPathDep,
) -> ObservationRepository:
"""Create an ObservationRepository instance for v2 API (uses external_id)."""
return ObservationRepository(session_maker, project_id=project_id)
return ObservationRepository(project_id=project_id)
ObservationRepositoryV2ExternalDep = Annotated[
@@ -108,33 +102,30 @@ ObservationRepositoryV2ExternalDep = Annotated[
async def get_relation_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
) -> RelationRepository:
"""Create a RelationRepository instance for the current project."""
return RelationRepository(session_maker, project_id=project_id)
return RelationRepository(project_id=project_id)
RelationRepositoryDep = Annotated[RelationRepository, Depends(get_relation_repository)]
async def get_relation_repository_v2( # pragma: no cover
session_maker: SessionMakerDep,
project_id: ProjectIdPathDep,
) -> RelationRepository:
"""Create a RelationRepository instance for v2 API."""
return RelationRepository(session_maker, project_id=project_id)
return RelationRepository(project_id=project_id)
RelationRepositoryV2Dep = Annotated[RelationRepository, Depends(get_relation_repository_v2)]
async def get_relation_repository_v2_external(
session_maker: SessionMakerDep,
project_id: ProjectExternalIdPathDep,
) -> RelationRepository:
"""Create a RelationRepository instance for v2 API (uses external_id)."""
return RelationRepository(session_maker, project_id=project_id)
return RelationRepository(project_id=project_id)
RelationRepositoryV2ExternalDep = Annotated[
+59 -10
View File
@@ -16,6 +16,7 @@ from fastapi import Depends
from loguru import logger
from basic_memory.deps.config import AppConfigDep
from basic_memory.deps.db import SessionMakerDep
from basic_memory.deps.projects import (
ProjectConfigDep,
ProjectConfigV2Dep,
@@ -159,9 +160,10 @@ async def get_search_service(
search_repository: SearchRepositoryDep,
entity_repository: EntityRepositoryDep,
file_service: FileServiceDep,
session_maker: SessionMakerDep,
) -> SearchService:
"""Create SearchService with dependencies."""
return SearchService(search_repository, entity_repository, file_service)
return SearchService(search_repository, entity_repository, file_service, session_maker)
SearchServiceDep = Annotated[SearchService, Depends(get_search_service)]
@@ -171,9 +173,10 @@ async def get_search_service_v2( # pragma: no cover
search_repository: SearchRepositoryV2Dep,
entity_repository: EntityRepositoryV2Dep,
file_service: FileServiceV2Dep,
session_maker: SessionMakerDep,
) -> SearchService:
"""Create SearchService for v2 API."""
return SearchService(search_repository, entity_repository, file_service)
return SearchService(search_repository, entity_repository, file_service, session_maker)
SearchServiceV2Dep = Annotated[SearchService, Depends(get_search_service_v2)]
@@ -183,9 +186,10 @@ async def get_search_service_v2_external(
search_repository: SearchRepositoryV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
file_service: FileServiceV2ExternalDep,
session_maker: SessionMakerDep,
) -> SearchService:
"""Create SearchService for v2 API (uses external_id)."""
return SearchService(search_repository, entity_repository, file_service)
return SearchService(search_repository, entity_repository, file_service, session_maker)
SearchServiceV2ExternalDep = Annotated[SearchService, Depends(get_search_service_v2_external)]
@@ -195,27 +199,45 @@ SearchServiceV2ExternalDep = Annotated[SearchService, Depends(get_search_service
async def get_link_resolver(
entity_repository: EntityRepositoryDep, search_service: SearchServiceDep
entity_repository: EntityRepositoryDep,
search_service: SearchServiceDep,
session_maker: SessionMakerDep,
) -> LinkResolver:
return LinkResolver(entity_repository=entity_repository, search_service=search_service)
return LinkResolver(
entity_repository=entity_repository,
search_service=search_service,
session_maker=session_maker,
)
LinkResolverDep = Annotated[LinkResolver, Depends(get_link_resolver)]
async def get_link_resolver_v2( # pragma: no cover
entity_repository: EntityRepositoryV2Dep, search_service: SearchServiceV2Dep
entity_repository: EntityRepositoryV2Dep,
search_service: SearchServiceV2Dep,
session_maker: SessionMakerDep,
) -> LinkResolver:
return LinkResolver(entity_repository=entity_repository, search_service=search_service)
return LinkResolver(
entity_repository=entity_repository,
search_service=search_service,
session_maker=session_maker,
)
LinkResolverV2Dep = Annotated[LinkResolver, Depends(get_link_resolver_v2)]
async def get_link_resolver_v2_external(
entity_repository: EntityRepositoryV2ExternalDep, search_service: SearchServiceV2ExternalDep
entity_repository: EntityRepositoryV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
session_maker: SessionMakerDep,
) -> LinkResolver:
return LinkResolver(entity_repository=entity_repository, search_service=search_service)
return LinkResolver(
entity_repository=entity_repository,
search_service=search_service,
session_maker=session_maker,
)
LinkResolverV2ExternalDep = Annotated[LinkResolver, Depends(get_link_resolver_v2_external)]
@@ -232,6 +254,7 @@ async def get_entity_service(
file_service: FileServiceDep,
link_resolver: LinkResolverDep,
search_service: SearchServiceDep,
session_maker: SessionMakerDep,
app_config: AppConfigDep,
) -> EntityService:
"""Create EntityService with repository."""
@@ -242,6 +265,7 @@ async def get_entity_service(
entity_parser=entity_parser,
file_service=file_service,
link_resolver=link_resolver,
session_maker=session_maker,
search_service=search_service,
app_config=app_config,
)
@@ -258,6 +282,7 @@ async def get_entity_service_v2( # pragma: no cover
file_service: FileServiceV2Dep,
link_resolver: LinkResolverV2Dep,
search_service: SearchServiceV2Dep,
session_maker: SessionMakerDep,
app_config: AppConfigDep,
) -> EntityService:
"""Create EntityService for v2 API."""
@@ -268,6 +293,7 @@ async def get_entity_service_v2( # pragma: no cover
entity_parser=entity_parser,
file_service=file_service,
link_resolver=link_resolver,
session_maker=session_maker,
search_service=search_service,
app_config=app_config,
)
@@ -284,6 +310,7 @@ async def get_entity_service_v2_external(
file_service: FileServiceV2ExternalDep,
link_resolver: LinkResolverV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
session_maker: SessionMakerDep,
app_config: AppConfigDep,
) -> EntityService:
"""Create EntityService for v2 API (uses external_id)."""
@@ -294,6 +321,7 @@ async def get_entity_service_v2_external(
entity_parser=entity_parser,
file_service=file_service,
link_resolver=link_resolver,
session_maker=session_maker,
search_service=search_service,
app_config=app_config,
)
@@ -310,12 +338,14 @@ async def get_context_service(
entity_repository: EntityRepositoryDep,
observation_repository: ObservationRepositoryDep,
link_resolver: LinkResolverDep,
session_maker: SessionMakerDep,
) -> ContextService:
return ContextService(
search_repository=search_repository,
entity_repository=entity_repository,
observation_repository=observation_repository,
link_resolver=link_resolver,
session_maker=session_maker,
)
@@ -327,6 +357,7 @@ async def get_context_service_v2( # pragma: no cover
entity_repository: EntityRepositoryV2Dep,
observation_repository: ObservationRepositoryV2Dep,
link_resolver: LinkResolverV2Dep,
session_maker: SessionMakerDep,
) -> ContextService:
"""Create ContextService for v2 API."""
return ContextService(
@@ -334,6 +365,7 @@ async def get_context_service_v2( # pragma: no cover
entity_repository=entity_repository,
observation_repository=observation_repository,
link_resolver=link_resolver,
session_maker=session_maker,
)
@@ -345,6 +377,7 @@ async def get_context_service_v2_external(
entity_repository: EntityRepositoryV2ExternalDep,
observation_repository: ObservationRepositoryV2ExternalDep,
link_resolver: LinkResolverV2ExternalDep,
session_maker: SessionMakerDep,
) -> ContextService:
"""Create ContextService for v2 API (uses external_id)."""
return ContextService(
@@ -352,6 +385,7 @@ async def get_context_service_v2_external(
entity_repository=entity_repository,
observation_repository=observation_repository,
link_resolver=link_resolver,
session_maker=session_maker,
)
@@ -370,6 +404,7 @@ async def get_sync_service(
project_repository: ProjectRepositoryDep,
search_service: SearchServiceDep,
file_service: FileServiceDep,
session_maker: SessionMakerDep,
) -> SyncService: # pragma: no cover
return SyncService(
app_config=app_config,
@@ -380,6 +415,7 @@ async def get_sync_service(
project_repository=project_repository,
search_service=search_service,
file_service=file_service,
session_maker=session_maker,
)
@@ -395,6 +431,7 @@ async def get_sync_service_v2(
project_repository: ProjectRepositoryDep,
search_service: SearchServiceV2Dep,
file_service: FileServiceV2Dep,
session_maker: SessionMakerDep,
) -> SyncService: # pragma: no cover
"""Create SyncService for v2 API."""
return SyncService(
@@ -406,6 +443,7 @@ async def get_sync_service_v2(
project_repository=project_repository,
search_service=search_service,
file_service=file_service,
session_maker=session_maker,
)
@@ -421,6 +459,7 @@ async def get_sync_service_v2_external(
project_repository: ProjectRepositoryDep,
search_service: SearchServiceV2ExternalDep,
file_service: FileServiceV2ExternalDep,
session_maker: SessionMakerDep,
) -> SyncService: # pragma: no cover
"""Create SyncService for v2 API (uses external_id)."""
return SyncService(
@@ -432,6 +471,7 @@ async def get_sync_service_v2_external(
project_repository=project_repository,
search_service=search_service,
file_service=file_service,
session_maker=session_maker,
)
@@ -535,6 +575,7 @@ TaskSchedulerDep = Annotated[TaskScheduler, Depends(get_task_scheduler)]
async def get_project_service(
project_repository: ProjectRepositoryDep,
session_maker: SessionMakerDep,
app_config: AppConfigDep,
) -> ProjectService:
"""Create ProjectService with repository and a system-level FileService for directory operations."""
@@ -543,7 +584,9 @@ async def get_project_service(
entity_parser = EntityParser(Path.home())
markdown_processor = MarkdownProcessor(entity_parser, app_config=app_config)
file_service = FileService(Path.home(), markdown_processor, app_config=app_config)
return ProjectService(repository=project_repository, file_service=file_service)
return ProjectService(
repository=project_repository, session_maker=session_maker, file_service=file_service
)
ProjectServiceDep = Annotated[ProjectService, Depends(get_project_service)]
@@ -554,10 +597,12 @@ ProjectServiceDep = Annotated[ProjectService, Depends(get_project_service)]
async def get_directory_service(
entity_repository: EntityRepositoryDep,
session_maker: SessionMakerDep,
) -> DirectoryService:
"""Create DirectoryService with dependencies."""
return DirectoryService(
entity_repository=entity_repository,
session_maker=session_maker,
)
@@ -566,10 +611,12 @@ DirectoryServiceDep = Annotated[DirectoryService, Depends(get_directory_service)
async def get_directory_service_v2( # pragma: no cover
entity_repository: EntityRepositoryV2Dep,
session_maker: SessionMakerDep,
) -> DirectoryService:
"""Create DirectoryService for v2 API (uses integer project_id from path)."""
return DirectoryService(
entity_repository=entity_repository,
session_maker=session_maker,
)
@@ -578,10 +625,12 @@ DirectoryServiceV2Dep = Annotated[DirectoryService, Depends(get_directory_servic
async def get_directory_service_v2_external(
entity_repository: EntityRepositoryV2ExternalDep,
session_maker: SessionMakerDep,
) -> DirectoryService:
"""Create DirectoryService for v2 API (uses external_id from path)."""
return DirectoryService(
entity_repository=entity_repository,
session_maker=session_maker,
)
+68 -22
View File
@@ -302,26 +302,76 @@ async def format_file(
return None
# A frontmatter fence is a line containing exactly `---`, optionally followed by
# trailing horizontal whitespace. Anchoring to a full line (rather than a bare
# substring/`startswith`) prevents single-line content like
# `---\nstatus: active\n---\nBody` — where `\n` is a literal backslash-n, not a
# newline — from being misread as frontmatter. See issue #972.
_FENCE_RE = re.compile(r"^---[ \t]*$")
def _split_frontmatter(content: str) -> Optional[tuple[str, str]]:
"""Split content into (yaml_block, body) when it opens with a line-anchored fence.
The opening fence must be the very first line and the closing fence must be a
later line, each matching exactly `---` (with optional trailing whitespace).
Returns:
A `(yaml_block, body)` tuple when a complete fenced block is present, or
``None`` when the content does not open with a frontmatter fence.
Raises:
ParseError: If the content opens with a fence but has no closing fence.
"""
lines = content.splitlines(keepends=True)
# Skip leading blank lines: a document may begin with whitespace before the
# opening fence (e.g. a heredoc/dedented string starting with a newline). This
# does NOT relax line-anchoring — the opening fence must still be the first
# non-blank line, all on its own, so a single-line `---\\nstatus...` (literal
# backslash-n) is still rejected. See issue #972.
start = 0
while start < len(lines) and lines[start].strip() == "":
start += 1
if start >= len(lines) or not _FENCE_RE.match(lines[start].rstrip("\r\n")):
return None
# Find the closing fence on its own line somewhere after the opening fence.
for index in range(start + 1, len(lines)):
if _FENCE_RE.match(lines[index].rstrip("\r\n")):
yaml_block = "".join(lines[start + 1 : index])
body = "".join(lines[index + 1 :])
return yaml_block, body
raise ParseError("Invalid frontmatter format")
def has_frontmatter(content: str) -> bool:
"""
Check if content contains valid YAML frontmatter.
Frontmatter requires `---` fences on their own lines; an inline `---` (such as
a single-line string that merely starts with the characters `---`) is not
frontmatter.
Args:
content: Content to check
Returns:
True if content has valid frontmatter markers (---), False otherwise
True if content has line-anchored frontmatter fences, False otherwise
"""
if not content:
return False
# Strip BOM before checking for frontmatter markers
content = strip_bom(content).strip()
if not content.startswith("---"):
content = strip_bom(content)
try:
return _split_frontmatter(content) is not None
except ParseError:
# An opening fence with no closing fence is not usable frontmatter.
return False
return "---" in content[3:]
def parse_frontmatter(content: str) -> Dict[str, Any]:
"""
@@ -339,17 +389,14 @@ def parse_frontmatter(content: str) -> Dict[str, Any]:
try:
# Strip BOM before parsing frontmatter
content = strip_bom(content)
if not content.strip().startswith("---"):
split = _split_frontmatter(content)
if split is None:
raise ParseError("Content has no frontmatter")
# Split on first two occurrences of ---
parts = content.split("---", 2)
if len(parts) < 3:
raise ParseError("Invalid frontmatter format")
yaml_block, _ = split
# Parse YAML
try:
frontmatter = yaml.safe_load(parts[1])
frontmatter = yaml.safe_load(yaml_block)
# Handle empty frontmatter (None from yaml.safe_load)
if frontmatter is None:
return {}
@@ -381,18 +428,17 @@ def remove_frontmatter(content: str) -> str:
ParseError: If content starts with frontmatter marker but is malformed
"""
# Strip BOM before processing
content = strip_bom(content).strip()
content = strip_bom(content)
# Return as-is if no frontmatter marker
if not content.startswith("---"):
return content
split = _split_frontmatter(content)
# Trigger: content does not open with a line-anchored fence
# Why: inline `---` is ordinary content, not frontmatter (issue #972)
# Outcome: return the content untouched (stripped to preserve prior behavior)
if split is None:
return content.strip()
# Split on first two occurrences of ---
parts = content.split("---", 2)
if len(parts) < 3:
raise ParseError("Invalid frontmatter format")
return parts[2].strip()
_, body = split
return body.strip()
def dump_frontmatter(post: frontmatter.Post) -> str:
+109 -70
View File
@@ -3,15 +3,18 @@
from __future__ import annotations
import asyncio
from contextlib import asynccontextmanager
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from typing import Awaitable, Callable, Mapping, TypeVar
from typing import AsyncIterator, Awaitable, Callable, Mapping, TypeVar
from loguru import logger
from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
import logfire
from basic_memory import db
from basic_memory.config import BasicMemoryConfig
from basic_memory.file_utils import compute_checksum, has_frontmatter, remove_frontmatter
from basic_memory.markdown.schemas import EntityMarkdown
@@ -69,6 +72,7 @@ class BatchIndexer:
relation_repository: RelationRepository,
search_service: SearchService,
file_writer: IndexFileWriter,
session_maker: async_sessionmaker[AsyncSession],
) -> None:
self.app_config = app_config
self.entity_service = entity_service
@@ -76,6 +80,19 @@ class BatchIndexer:
self.relation_repository = relation_repository
self.search_service = search_service
self.file_writer = file_writer
self.session_maker = session_maker
@asynccontextmanager
async def _session_scope(
self, session: AsyncSession | None = None
) -> AsyncIterator[AsyncSession]:
"""Use the caller's session or open a service-owned transaction."""
if session is not None:
yield session
return
async with db.scoped_session(self.session_maker) as owned_session:
yield owned_session
async def index_files(
self,
@@ -149,9 +166,10 @@ class BatchIndexer:
max_concurrent=max_concurrent,
)
refreshed_entities = await self.entity_repository.find_by_ids(
[prepared.entity_id for prepared in prepared_entities.values()]
)
async with self._session_scope() as session:
refreshed_entities = await self.entity_repository.find_by_ids(
session, [prepared.entity_id for prepared in prepared_entities.values()]
)
entities_by_id = {entity.id: entity for entity in refreshed_entities}
refreshed, refresh_errors = await self._run_bounded(
@@ -196,12 +214,7 @@ class BatchIndexer:
prepared = await self._prepare_markdown_file(file)
if existing_permalink_by_path is None:
with logfire.span("index.markdown_file.load_permalink_map", path=file.path):
existing_permalink_by_path = {
path: permalink
for path, permalink in (
await self.entity_repository.get_file_path_to_permalink_map()
).items()
}
existing_permalink_by_path = await self._get_file_path_to_permalink_map()
reserved_permalinks = {
permalink
@@ -226,7 +239,8 @@ class BatchIndexer:
path=file.path,
entity_id=persisted.entity.id,
):
refreshed = await self.entity_repository.find_by_ids([persisted.entity.id])
async with self._session_scope() as session:
refreshed = await self.entity_repository.find_by_ids(session, [persisted.entity.id])
if len(refreshed) != 1: # pragma: no cover
raise ValueError(f"Failed to reload indexed entity for {file.path}")
entity = refreshed[0]
@@ -249,6 +263,17 @@ class BatchIndexer:
markdown_content=prepared_entity.markdown_content,
)
async def _get_file_path_to_permalink_map(self) -> dict[str, str | None]:
"""Load current file-path to permalink mappings in a service-owned session."""
async with self._session_scope() as session:
permalink_by_path: dict[str, str | None] = {
path: permalink
for path, permalink in (
await self.entity_repository.get_file_path_to_permalink_map(session)
).items()
}
return permalink_by_path
# --- Preparation ---
async def _prepare_markdown_file(self, file: IndexInputFile) -> _PreparedMarkdownFile:
@@ -283,12 +308,7 @@ class BatchIndexer:
return {}, {}
if existing_permalink_by_path is None:
existing_permalink_by_path = {
path: permalink
for path, permalink in (
await self.entity_repository.get_file_path_to_permalink_map()
).items()
}
existing_permalink_by_path = await self._get_file_path_to_permalink_map()
batch_paths = set(prepared_markdown)
reserved_permalinks = {
@@ -406,7 +426,10 @@ class BatchIndexer:
async def _upsert_regular_file(self, file: IndexInputFile) -> _PreparedEntity:
checksum = await self._resolve_checksum(file)
existing = await self.entity_repository.get_by_file_path(file.path, load_relations=False)
async with self._session_scope() as session:
existing = await self.entity_repository.get_by_file_path(
session, file.path, load_relations=False
)
is_new_entity = existing is None
if existing is None:
@@ -424,7 +447,8 @@ class BatchIndexer:
)
try:
created = await self.entity_repository.add(entity)
async with self._session_scope() as session:
created = await self.entity_repository.add(session, entity)
entity_id = created.id
except IntegrityError as exc:
message = str(exc)
@@ -436,10 +460,12 @@ class BatchIndexer:
and "file_path" in message
)
):
existing = await self.entity_repository.get_by_file_path(
file.path,
load_relations=False,
)
async with self._session_scope() as session:
existing = await self.entity_repository.get_by_file_path(
session,
file.path,
load_relations=False,
)
if existing is None:
raise ValueError(
f"Entity not found after file_path conflict: {file.path}"
@@ -450,10 +476,12 @@ class BatchIndexer:
else:
entity_id = existing.id
updated = await self.entity_repository.update(
entity_id,
self._entity_metadata_updates(file, checksum, include_created_at=is_new_entity),
)
async with self._session_scope() as session:
updated = await self.entity_repository.update(
session,
entity_id,
self._entity_metadata_updates(file, checksum, include_created_at=is_new_entity),
)
if updated is None:
raise ValueError(f"Failed to update file entity metadata for {file.path}")
@@ -476,10 +504,7 @@ class BatchIndexer:
max_concurrent: int,
) -> tuple[int, int]:
unresolved_relation_lists = await asyncio.gather(
*(
self.relation_repository.find_unresolved_relations_for_entity(entity_id)
for entity_id in entity_ids
)
*(self._find_unresolved_relations_for_entity(entity_id) for entity_id in entity_ids)
)
unresolved_relations = [
relation for relation_list in unresolved_relation_lists for relation in relation_list
@@ -499,22 +524,26 @@ class BatchIndexer:
# link text, mismatching this with the sync_service forward-reference
# path and producing confidently-wrong graph edges. See
# sync_service.resolve_forward_references for the same change.
resolved_entity = await self.entity_service.link_resolver.resolve_link(
relation.to_name, strict=True
)
async with self._session_scope() as session:
resolved_entity = await self.entity_service.link_resolver.resolve_link(
relation.to_name, strict=True, session=session
)
if resolved_entity is None or resolved_entity.id == relation.from_id:
return 0
try:
await self.relation_repository.update(
relation.id,
{
"to_id": resolved_entity.id,
"to_name": resolved_entity.title,
},
)
async with self._session_scope() as session:
await self.relation_repository.update(
session,
relation.id,
{
"to_id": resolved_entity.id,
"to_name": resolved_entity.title,
},
)
except IntegrityError:
await self.relation_repository.delete(relation.id)
async with self._session_scope() as session:
await self.relation_repository.delete(session, relation.id)
return 1
except Exception as exc: # pragma: no cover - defensive logging
logger.warning(
@@ -531,15 +560,19 @@ class BatchIndexer:
)
remaining_relation_lists = await asyncio.gather(
*(
self.relation_repository.find_unresolved_relations_for_entity(entity_id)
for entity_id in entity_ids
)
*(self._find_unresolved_relations_for_entity(entity_id) for entity_id in entity_ids)
)
remaining_unresolved = sum(len(relations) for relations in remaining_relation_lists)
return sum(resolved_counts), remaining_unresolved
async def _find_unresolved_relations_for_entity(self, entity_id: int):
"""Load unresolved relations for one entity in a service-owned session."""
async with self._session_scope() as session:
return await self.relation_repository.find_unresolved_relations_for_entity(
session, entity_id
)
# --- Search refresh ---
async def _refresh_search_index(
@@ -565,30 +598,36 @@ class BatchIndexer:
resolve_relations: bool = True,
reload_entity: bool = True,
) -> _PersistedMarkdownFile:
existing = await self.entity_repository.get_by_file_path(
prepared.file.path,
load_relations=False,
)
if is_new is None:
is_new = existing is None
entity = await self.entity_service.upsert_entity_from_markdown(
Path(prepared.file.path),
prepared.markdown,
is_new=is_new,
existing_entity=existing,
resolve_relations=resolve_relations,
reload_entity=reload_entity,
)
prepared = await self._reconcile_persisted_permalink(prepared, entity)
metadata_updates = self._entity_metadata_updates(prepared.file, prepared.final_checksum)
updated = await self.entity_repository.update_fields(
entity.id,
metadata_updates,
)
if not updated:
raise ValueError(f"Failed to update markdown entity metadata for {prepared.file.path}")
self._apply_entity_metadata_updates(entity, metadata_updates)
return _PersistedMarkdownFile(prepared=prepared, entity=entity)
async with self._session_scope() as session:
existing = await self.entity_repository.get_by_file_path(
session,
prepared.file.path,
load_relations=False,
)
if is_new is None:
is_new = existing is None
entity = await self.entity_service.upsert_entity_from_markdown(
Path(prepared.file.path),
prepared.markdown,
is_new=is_new,
existing_entity=existing,
resolve_relations=resolve_relations,
reload_entity=reload_entity,
session=session,
)
prepared = await self._reconcile_persisted_permalink(prepared, entity)
metadata_updates = self._entity_metadata_updates(prepared.file, prepared.final_checksum)
updated = await self.entity_repository.update_fields(
session,
entity.id,
metadata_updates,
)
if not updated:
raise ValueError(
f"Failed to update markdown entity metadata for {prepared.file.path}"
)
self._apply_entity_metadata_updates(entity, metadata_updates)
return _PersistedMarkdownFile(prepared=prepared, entity=entity)
async def _reconcile_persisted_permalink(
self,
+1
View File
@@ -0,0 +1 @@
.so man1/bm.1
+134
View File
@@ -0,0 +1,134 @@
.TH BM 1 "2026-06-11" "basic-memory" "Basic Memory Manual"
.SH NAME
bm \- local-first knowledge base for humans and AI agents
.SH SYNOPSIS
.B bm
.I COMMAND
.RI [ ARGS ]...
.br
.B basic-memory
.I COMMAND
.RI [ ARGS ]...
.SH DESCRIPTION
.B bm
manages Basic Memory projects: plain markdown files that form a knowledge
graph. Files are the source of truth; SQLite provides indexing and
full-text search; the same operations are exposed to AI agents over the
Model Context Protocol (MCP) and to humans and scripts through this CLI.
.PP
Notes use semantic markdown: observations
.RB ( "\- [category] text #tag" )
and relations
.RB ( "\- relation_type [[Target]]" )
become queryable graph structure. Projects route independently to the
local API or to Basic Memory Cloud.
.SH COMMANDS
Knowledge operations:
.TP
.B bm tool
CLI access to the MCP tools (write-note, read-note, search-notes,
build-context, ...). These emit JSON and are the scriptable surface.
.TP
.B bm status
Show sync status between files and the database.
.TP
.B bm reindex
Index local file changes and rebuild search/embeddings. This is the manual
sync trigger when no MCP server is running.
.TP
.B bm doctor
Run end-to-end file/database consistency checks.
.TP
.B bm orphans
List entities with no relations in the knowledge graph.
.TP
.B bm format
Run configured formatters over note files.
.PP
Projects and schemas:
.TP
.B bm project
Add, remove, list projects; set the default; flip a project between local
and cloud routing.
.TP
.B bm schema
List, validate, infer, and drift-check Picoschema note-type contracts.
.PP
Data and cloud:
.TP
.B bm import
Import from ChatGPT, Claude, or memory.json exports. Imports write files;
run
.B bm reindex
afterwards.
.TP
.B bm cloud
Authenticate, sync (push/pull/sync/bisync), snapshots, and team workspace
administration.
.PP
Infrastructure:
.TP
.B bm mcp
Run the MCP server (hosts the live file watcher).
.TP
.B bm man
Manage these man pages
.RB ( "bm man install" ).
.TP
.B bm reset
Drop and recreate the database (destructive).
.PP
Most commands accept
.B \-\-project
.I NAME
to target a project and
.BR \-\-local / \-\-cloud
to override routing.
.SH EXAMPLES
Write and find a note from the shell:
.PP
.nf
.RS
echo "# Standup notes" | bm tool write-note \\
\-\-title "Standup" \-\-folder notes
bm tool search-notes "standup"
.RE
.fi
.PP
Pick up files created outside the tools:
.PP
.nf
.RS
bm status # shows pending changes
bm reindex # indexes them
.RE
.fi
.SH FILES
.TP
.I ~/.basic-memory/config.json
Projects, default project, per-project routing modes, cloud settings.
.TP
.I ~/.basic-memory/memory.db
SQLite index (derived; safe to rebuild with bm reindex).
.SH ENVIRONMENT
.TP
.B BASIC_MEMORY_FORCE_LOCAL
Force local routing regardless of cloud mode.
.TP
.B BASIC_MEMORY_LOG_LEVEL
Logging verbosity (e.g. DEBUG).
.SH SEE ALSO
Full manual (machine-readable, agent-traversable): the
.I manual
Basic Memory project \(em see docs/manual-pages.md in the repository.
Documentation: https://docs.basicmemory.com
.PP
For AI agents: the complete tool reference lives in the manual project as
section\-3 pages (write-note(3), search-notes(3), ...), queryable via
.B search_notes
with
.BR "metadata_filters={\(dqtype\(dq: \(dqmanpage\(dq}" .
.SH BUGS
https://github.com/basicmachines-co/basic-memory/issues
.SH AUTHORS
Basic Machines (https://basicmachines.co)
+38 -13
View File
@@ -5,14 +5,16 @@ from dataclasses import dataclass
from threading import RLock
from typing import TYPE_CHECKING, Annotated, Any, AsyncIterator, Callable, Optional
from fastapi import Depends, FastAPI, Request
from httpx import ASGITransport, AsyncClient, Timeout
from loguru import logger
import logfire
from basic_memory.config import ConfigManager, ProjectMode
from basic_memory.config import ConfigManager, ProjectMode, has_cloud_credentials
if TYPE_CHECKING:
# FastAPI is only needed when a request routes through the local ASGI
# transport; importing it at module level costs ~0.1s on every CLI start (#886).
from fastapi import FastAPI
from sqlalchemy.ext.asyncio import AsyncEngine, AsyncSession, async_sessionmaker
LocalDatabaseState = tuple["AsyncEngine", "async_sessionmaker[AsyncSession]"]
@@ -28,8 +30,8 @@ class _PreparedLocalAsgiDatabase:
_prepared_local_asgi_database_lock = RLock()
_prepared_local_asgi_database_prepare_locks: dict[FastAPI, Lock] = {}
_prepared_local_asgi_databases: dict[FastAPI, _PreparedLocalAsgiDatabase] = {}
_prepared_local_asgi_database_prepare_locks: dict["FastAPI", Lock] = {}
_prepared_local_asgi_databases: dict["FastAPI", _PreparedLocalAsgiDatabase] = {}
def _force_local_mode() -> bool:
@@ -57,7 +59,7 @@ def _build_timeout() -> Timeout:
)
def _build_asgi_client(app: FastAPI, timeout: Timeout) -> AsyncClient:
def _build_asgi_client(app: "FastAPI", timeout: Timeout) -> AsyncClient:
"""Create a local ASGI client for an already-prepared FastAPI app."""
from basic_memory.workspace_context import workspace_permalink_headers
@@ -71,7 +73,7 @@ def _build_asgi_client(app: FastAPI, timeout: Timeout) -> AsyncClient:
)
def _get_prepared_local_asgi_database_prepare_lock(app: FastAPI) -> Lock:
def _get_prepared_local_asgi_database_prepare_lock(app: "FastAPI") -> Lock:
"""Get the async lock that serializes first-time DB preparation for an app."""
with _prepared_local_asgi_database_lock:
prepare_lock = _prepared_local_asgi_database_prepare_locks.get(app)
@@ -82,8 +84,10 @@ def _get_prepared_local_asgi_database_prepare_lock(app: FastAPI) -> Lock:
@asynccontextmanager
async def _resolve_local_asgi_database(app: FastAPI) -> AsyncIterator[LocalDatabaseState]:
async def _resolve_local_asgi_database(app: "FastAPI") -> AsyncIterator[LocalDatabaseState]:
"""Resolve database state for a local ASGI request."""
# Imported on first local-ASGI use so CLI startup never pays for FastAPI (#886).
from fastapi import Depends, Request
from fastapi.dependencies.utils import get_dependant, solve_dependencies
from basic_memory.deps import get_engine_factory
@@ -127,7 +131,7 @@ async def _resolve_local_asgi_database(app: FastAPI) -> AsyncIterator[LocalDatab
yield await resolve_database_state(**solved.values)
def _retain_prepared_local_asgi_database(app: FastAPI) -> bool:
def _retain_prepared_local_asgi_database(app: "FastAPI") -> bool:
"""Retain an active local ASGI database preparation if one exists."""
with _prepared_local_asgi_database_lock:
active = _prepared_local_asgi_databases.get(app)
@@ -139,7 +143,7 @@ def _retain_prepared_local_asgi_database(app: FastAPI) -> bool:
def _install_prepared_local_asgi_database(
app: FastAPI,
app: "FastAPI",
database_state: LocalDatabaseState,
dependency_context: AbstractAsyncContextManager[LocalDatabaseState],
) -> None:
@@ -163,7 +167,7 @@ def _install_prepared_local_asgi_database(
)
def _restore_local_asgi_state_attribute(app: FastAPI, name: str, previous_value: object) -> None:
def _restore_local_asgi_state_attribute(app: "FastAPI", name: str, previous_value: object) -> None:
"""Restore a FastAPI app.state attribute captured before local ASGI preparation."""
if previous_value is _MISSING_STATE_VALUE:
if hasattr(app.state, name):
@@ -173,7 +177,7 @@ def _restore_local_asgi_state_attribute(app: FastAPI, name: str, previous_value:
def _release_prepared_local_asgi_database(
app: FastAPI,
app: "FastAPI",
) -> AbstractAsyncContextManager[LocalDatabaseState] | None:
"""Release local ASGI database state after a client context exits."""
with _prepared_local_asgi_database_lock:
@@ -196,7 +200,7 @@ def _release_prepared_local_asgi_database(
@asynccontextmanager
async def _prepared_local_asgi_database(app: FastAPI) -> AsyncIterator[None]:
async def _prepared_local_asgi_database(app: "FastAPI") -> AsyncIterator[None]:
"""Initialize local ASGI database state before the first request."""
prepare_lock = _get_prepared_local_asgi_database_prepare_lock(app)
async with prepare_lock:
@@ -368,7 +372,8 @@ async def get_client(
1. Factory injection.
2. Explicit routing flags (--local/--cloud).
3. Per-project mode routing when project_name is provided.
4. Local ASGI transport by default.
4. Cloud routing when a workspace selector is provided.
5. Local ASGI transport by default.
"""
if _client_factory:
async with _client_factory(workspace=workspace) as client:
@@ -428,6 +433,26 @@ async def get_client(
yield client
return
# --- Workspace-selector routing ---
# Trigger: caller passed a cloud workspace selector and nothing above routed.
# Why: a workspace names a cloud tenant — silently serving the request from
# the local ASGI app sent writes to the wrong destination (#954: a cloud
# project create either failed on the cloud-style path or landed locally).
# Outcome: route to the cloud proxy when credentials exist; without
# credentials fail fast instead of pretending the operation succeeded.
if workspace is not None:
if not has_cloud_credentials(config):
raise RuntimeError(
f"A cloud workspace was requested ('{workspace}') but no cloud "
"credentials were found. Run 'bm cloud login' or "
"'bm cloud set-key <key>' first, or omit the workspace selector "
"for a local operation."
)
logger.debug(f"Workspace selector '{workspace}' provided - using cloud proxy client")
async with _cloud_client(config, timeout, workspace=workspace) as client:
yield client
return
# --- Default fallback ---
logger.debug("Default routing - using ASGI client for local Basic Memory API")
async with _asgi_client(timeout) as client:
+5 -1
View File
@@ -7,7 +7,9 @@ from typing import Optional, Any
from httpx import AsyncClient
from basic_memory.mcp.tools.utils import call_get
# call_* helpers live in basic_memory.mcp.tools.utils; importing that at module
# level executes the whole tools package (fastmcp + mcp SDK) during CLI startup,
# so each method defers the import to call time instead (#886).
class DirectoryClient:
@@ -55,6 +57,8 @@ class DirectoryClient:
Raises:
ToolError: If the request fails
"""
from basic_memory.mcp.tools.utils import call_get
params: dict = {
"dir_name": dir_name,
"depth": depth,
+26 -1
View File
@@ -8,7 +8,10 @@ from typing import Any
from httpx import AsyncClient
import logfire
from basic_memory.mcp.tools.utils import call_get, call_post, call_put, call_patch, call_delete
# call_* helpers live in basic_memory.mcp.tools.utils; importing that at module
# level executes the whole tools package (fastmcp + mcp SDK) during CLI startup,
# so each method defers the import to call time instead (#886).
from basic_memory.schemas.response import (
EntityResponse,
DeleteEntitiesResponse,
@@ -57,6 +60,8 @@ class KnowledgeClient:
Raises:
ToolError: If the request fails
"""
from basic_memory.mcp.tools.utils import call_post
with logfire.span(
"mcp.client.knowledge.create_entity",
client_name="knowledge",
@@ -89,6 +94,8 @@ class KnowledgeClient:
Raises:
ToolError: If the request fails
"""
from basic_memory.mcp.tools.utils import call_put
with logfire.span(
"mcp.client.knowledge.update_entity",
client_name="knowledge",
@@ -116,6 +123,8 @@ class KnowledgeClient:
Raises:
ToolError: If the entity is not found or request fails
"""
from basic_memory.mcp.tools.utils import call_get
with logfire.span(
"mcp.client.knowledge.get_entity",
client_name="knowledge",
@@ -147,6 +156,8 @@ class KnowledgeClient:
Raises:
ToolError: If the request fails
"""
from basic_memory.mcp.tools.utils import call_patch
with logfire.span(
"mcp.client.knowledge.patch_entity",
client_name="knowledge",
@@ -174,6 +185,8 @@ class KnowledgeClient:
Raises:
ToolError: If the entity is not found or request fails
"""
from basic_memory.mcp.tools.utils import call_delete
with logfire.span(
"mcp.client.knowledge.delete_entity",
client_name="knowledge",
@@ -201,6 +214,8 @@ class KnowledgeClient:
Raises:
ToolError: If the request fails
"""
from basic_memory.mcp.tools.utils import call_put
with logfire.span(
"mcp.client.knowledge.move_entity",
client_name="knowledge",
@@ -231,6 +246,8 @@ class KnowledgeClient:
Raises:
ToolError: If the request fails
"""
from basic_memory.mcp.tools.utils import call_post
with logfire.span(
"mcp.client.knowledge.move_directory",
client_name="knowledge",
@@ -261,6 +278,8 @@ class KnowledgeClient:
Raises:
ToolError: If the request fails
"""
from basic_memory.mcp.tools.utils import call_post
with logfire.span(
"mcp.client.knowledge.delete_directory",
client_name="knowledge",
@@ -290,6 +309,8 @@ class KnowledgeClient:
Raises:
ToolError: If the file does not exist on disk or indexing fails
"""
from basic_memory.mcp.tools.utils import call_post
with logfire.span(
"mcp.client.knowledge.sync_file",
client_name="knowledge",
@@ -309,6 +330,8 @@ class KnowledgeClient:
async def get_orphans(self) -> list[GraphNode]:
"""Get entities that have no incoming or outgoing relations."""
from basic_memory.mcp.tools.utils import call_get
with logfire.span(
"mcp.client.knowledge.get_orphans",
client_name="knowledge",
@@ -338,6 +361,8 @@ class KnowledgeClient:
Raises:
ToolError: If the identifier cannot be resolved
"""
from basic_memory.mcp.tools.utils import call_post
with logfire.span(
"mcp.client.knowledge.resolve_entity",
client_name="knowledge",
+8 -1
View File
@@ -8,7 +8,10 @@ from typing import Optional
from httpx import AsyncClient
import logfire
from basic_memory.mcp.tools.utils import call_get
# call_* helpers live in basic_memory.mcp.tools.utils; importing that at module
# level executes the whole tools package (fastmcp + mcp SDK) during CLI startup,
# so each method defers the import to call time instead (#886).
from basic_memory.schemas.memory import GraphContext
@@ -63,6 +66,8 @@ class MemoryClient:
Raises:
ToolError: If the request fails
"""
from basic_memory.mcp.tools.utils import call_get
params: dict = {
"depth": depth,
"page": page,
@@ -113,6 +118,8 @@ class MemoryClient:
Raises:
ToolError: If the request fails
"""
from basic_memory.mcp.tools.utils import call_get
params: dict = {
"timeframe": timeframe,
"depth": depth,
+21 -7
View File
@@ -7,13 +7,9 @@ from typing import Any
from httpx import AsyncClient
from basic_memory.mcp.tools.utils import (
call_delete,
call_get,
call_patch,
call_post,
call_put,
)
# call_* helpers live in basic_memory.mcp.tools.utils; importing that at module
# level executes the whole tools package (fastmcp + mcp SDK) during CLI startup,
# so each method defers the import to call time instead (#886).
from basic_memory.schemas import ProjectInfoResponse, SyncReportResponse
from basic_memory.schemas.project_info import ProjectList, ProjectStatusResponse
from basic_memory.schemas.v2 import ProjectResolveResponse
@@ -53,6 +49,8 @@ class ProjectClient:
Raises:
ToolError: If the request fails
"""
from basic_memory.mcp.tools.utils import call_get
response = await call_get(
self.http_client,
"/v2/projects/",
@@ -71,6 +69,8 @@ class ProjectClient:
Raises:
ToolError: If the request fails
"""
from basic_memory.mcp.tools.utils import call_post
response = await call_post(
self.http_client,
"/v2/projects/",
@@ -93,6 +93,8 @@ class ProjectClient:
Raises:
ToolError: If the request fails
"""
from basic_memory.mcp.tools.utils import call_delete
url = f"/v2/projects/{project_external_id}"
if delete_notes:
url += "?delete_notes=true"
@@ -114,6 +116,8 @@ class ProjectClient:
Raises:
ToolError: If the request fails
"""
from basic_memory.mcp.tools.utils import call_post
response = await call_post(
self.http_client,
"/v2/projects/resolve",
@@ -133,6 +137,8 @@ class ProjectClient:
Raises:
ToolError: If the request fails
"""
from basic_memory.mcp.tools.utils import call_put
response = await call_put(
self.http_client,
f"/v2/projects/{project_external_id}/default",
@@ -154,6 +160,8 @@ class ProjectClient:
Raises:
ToolError: If the request fails
"""
from basic_memory.mcp.tools.utils import call_patch
response = await call_patch(
self.http_client,
f"/v2/projects/{project_external_id}",
@@ -181,6 +189,8 @@ class ProjectClient:
Raises:
ToolError: If the request fails
"""
from basic_memory.mcp.tools.utils import call_post
url = f"/v2/projects/{project_external_id}/sync"
params = []
if force_full:
@@ -204,6 +214,8 @@ class ProjectClient:
Raises:
ToolError: If the request fails
"""
from basic_memory.mcp.tools.utils import call_post
response = await call_post(
self.http_client,
f"/v2/projects/{project_external_id}/status",
@@ -222,6 +234,8 @@ class ProjectClient:
Raises:
ToolError: If the request fails
"""
from basic_memory.mcp.tools.utils import call_get
response = await call_get(
self.http_client,
f"/v2/projects/{project_external_id}/info",
+5 -1
View File
@@ -6,7 +6,9 @@ Encapsulates all /v2/projects/{project_id}/resource/* endpoints.
from httpx import AsyncClient, Response
import logfire
from basic_memory.mcp.tools.utils import call_get
# call_* helpers live in basic_memory.mcp.tools.utils; importing that at module
# level executes the whole tools package (fastmcp + mcp SDK) during CLI startup,
# so each method defers the import to call time instead (#886).
class ResourceClient:
@@ -49,6 +51,8 @@ class ResourceClient:
Raises:
ToolError: If the resource is not found or request fails
"""
from basic_memory.mcp.tools.utils import call_get
with logfire.span(
"mcp.client.resource.read",
client_name="resource",
+9 -1
View File
@@ -5,7 +5,9 @@ Encapsulates all /v2/projects/{project_id}/schema/* endpoints.
from httpx import AsyncClient
from basic_memory.mcp.tools.utils import call_post, call_get
# call_* helpers live in basic_memory.mcp.tools.utils; importing that at module
# level executes the whole tools package (fastmcp + mcp SDK) during CLI startup,
# so each method defers the import to call time instead (#886).
from basic_memory.schemas.schema import (
ValidationReport,
InferenceReport,
@@ -56,6 +58,8 @@ class SchemaClient:
Raises:
ToolError: If the request fails
"""
from basic_memory.mcp.tools.utils import call_post
params: dict[str, str] = {}
if note_type:
params["note_type"] = note_type
@@ -87,6 +91,8 @@ class SchemaClient:
Raises:
ToolError: If the request fails
"""
from basic_memory.mcp.tools.utils import call_post
response = await call_post(
self.http_client,
f"{self._base_path}/infer",
@@ -106,6 +112,8 @@ class SchemaClient:
Raises:
ToolError: If the request fails
"""
from basic_memory.mcp.tools.utils import call_get
response = await call_get(
self.http_client,
f"{self._base_path}/diff/{note_type}",
+6 -1
View File
@@ -8,7 +8,10 @@ from typing import Any
from httpx import AsyncClient
import logfire
from basic_memory.mcp.tools.utils import call_post
# call_* helpers live in basic_memory.mcp.tools.utils; importing that at module
# level executes the whole tools package (fastmcp + mcp SDK) during CLI startup,
# so each method defers the import to call time instead (#886).
from basic_memory.schemas.search import SearchResponse
@@ -57,6 +60,8 @@ class SearchClient:
Raises:
ToolError: If the request fails
"""
from basic_memory.mcp.tools.utils import call_post
with logfire.span(
"mcp.client.search.search",
client_name="search",
+164 -23
View File
@@ -8,10 +8,24 @@ The resolve_project_parameter function is a thin wrapper for backwards
compatibility with existing MCP tools.
"""
# PEP 563 lazy annotations keep `Context` usable in signatures without importing
# fastmcp at module load — the fastmcp/mcp stack costs ~0.5s of CLI startup (#886).
from __future__ import annotations
import asyncio
from contextlib import asynccontextmanager, nullcontext
from typing import (
TYPE_CHECKING,
AsyncIterator,
Awaitable,
Callable,
List,
Optional,
Sequence,
Tuple,
cast,
)
from dataclasses import dataclass, field
from typing import AsyncIterator, Awaitable, Callable, List, Optional, Sequence, Tuple, cast
from uuid import UUID
from httpx import AsyncClient
@@ -19,8 +33,6 @@ from httpx._types import (
HeaderTypes,
)
from loguru import logger
from fastmcp import Context
from mcp.server.fastmcp.exceptions import ToolError
import logfire
from basic_memory.config import BasicMemoryConfig, ConfigManager, ProjectMode, has_cloud_credentials
@@ -46,6 +58,9 @@ from basic_memory.workspace_context import (
workspace_permalink_context,
)
if TYPE_CHECKING:
from fastmcp import Context
# --- Workspace provider injection ---
# Mirrors the set_client_factory() pattern in async_client.py.
# The cloud MCP server sets a provider that queries its own database directly,
@@ -54,6 +69,14 @@ _workspace_provider: Optional[Callable[[], Awaitable[list[WorkspaceInfo]]]] = No
_WORKSPACE_PROJECT_INDEX_STATE_KEY = "workspace_project_index"
class WorkspaceProjectLookupMiss(ValueError):
"""A project was absent from the workspace index (as opposed to ambiguous).
Misses are retried once against a freshly rebuilt index, because the
session cache may simply predate an out-of-band project creation (#956).
"""
@dataclass(frozen=True)
class WorkspaceProjectEntry:
"""A cloud project resolved together with the workspace that owns it."""
@@ -460,6 +483,17 @@ def _canonical_memory_path_for_workspace(
# Outcome: lookups preserve the complete workspace/project canonical permalink.
if not normalized_remainder:
normalized_remainder = project_permalink
# Same index-form rule as _canonical_memory_path_for_active_route (#957):
# without an active workspace permalink context, stored permalinks are
# project-qualified and a workspace-prefixed pattern cannot match.
if "*" in normalized_remainder and current_workspace_permalink_context() is None:
return build_qualified_permalink_reference(
project_permalink,
normalized_remainder,
include_project=True,
)
return build_qualified_permalink_reference(
project_permalink,
normalized_remainder,
@@ -477,6 +511,24 @@ def _canonical_memory_path_for_active_route(
) -> str:
"""Return the canonical permalink path for the currently routed project/workspace."""
project_prefix = active_project.permalink
# Trigger: the path contains a glob wildcard (folder/*) and no server-side
# workspace permalink context is active.
# Why: patterns match raw against the search index, so they must mirror the
# stored permalink form. The contextvar is what qualified permalinks at
# write time — when it is absent, stored rows are project-qualified and a
# workspace prefix (from the client's cached_workspace display state)
# guarantees zero matches (#957). Direct lookups keep full qualification
# because the link resolver understands it; patterns have no fallback.
# Outcome: without the contextvar, qualify patterns with the project prefix
# only; with it, fall through to normal workspace canonicalization.
if "*" in path and current_workspace_permalink_context() is None:
if not include_project:
return path
if path == project_prefix or path.startswith(f"{project_prefix}/"):
return path
return f"{project_prefix}/{path}"
workspace_remainder = path
if include_project and (path == project_prefix or path.startswith(f"{project_prefix}/")):
# Trigger: the memory URL already names the active project root/prefix
@@ -722,9 +774,15 @@ async def _fetch_workspace_project_entries(
async def _ensure_workspace_project_index(
context: Optional[Context] = None,
*,
force_refresh: bool = False,
) -> WorkspaceProjectIndex:
"""Build or load the session-local workspace/project lookup index."""
if context:
"""Build or load the session-local workspace/project lookup index.
force_refresh bypasses the cached index and rebuilds from discovery
used by resolve_workspace_project_identifier when a lookup misses (#956).
"""
if context and not force_refresh:
cached_raw = await context.get_state(_WORKSPACE_PROJECT_INDEX_STATE_KEY)
cached_index = _workspace_project_index_from_state(cached_raw)
if cached_index is not None:
@@ -798,13 +856,99 @@ async def ensure_workspace_project_index(
return await _ensure_workspace_project_index(context=context)
def _match_workspace_identifier(
workspaces: tuple[WorkspaceInfo, ...],
workspace_identifier: str,
) -> WorkspaceInfo:
"""Resolve the first segment of a qualified route to a single workspace.
The edit_note/write_note contract advertises that the workspace segment may be a
slug, tenant_id, or display name. We honor those forms in a fixed priority order so
that adding tenant_id/name support never changes the meaning of an identifier that
already resolves today:
1. slug (casefold) existing behavior, checked first so working routes are stable.
2. tenant_id exact match against the opaque id (no casefolding, mirroring the
precedent in ``workspace_matches_exact_identifier``).
3. display name (casefold) names are not guaranteed unique, so a name that matches
multiple workspaces is rejected rather than silently picking one.
"""
# Trigger: identifier equals a workspace slug (casefold).
# Why: slug is the canonical routing key; resolving it first guarantees a workspace
# whose display name collides with another workspace's slug yields to the slug owner.
# Outcome: return the slug owner before tenant_id/name are considered.
slug_matches = [
workspace
for workspace in workspaces
if workspace.slug.casefold() == workspace_identifier.casefold()
]
if slug_matches:
return slug_matches[0]
# Trigger: identifier exactly equals a workspace tenant_id (an opaque id).
# Why: tenant_ids are unique, so an exact hit is unambiguous and needs no tie-break.
tenant_matches = [
workspace for workspace in workspaces if workspace.tenant_id == workspace_identifier
]
if tenant_matches:
return tenant_matches[0]
# Trigger: identifier matches one or more workspace display names (casefold).
# Why: names are not guaranteed unique; failing fast on collisions keeps routing
# deterministic and tells the caller exactly how to disambiguate.
name_matches = [
workspace
for workspace in workspaces
if workspace.name.casefold() == workspace_identifier.casefold()
]
if len(name_matches) > 1:
candidates = ", ".join(workspace.slug for workspace in name_matches)
raise ValueError(
f"Workspace name '{workspace_identifier}' matched multiple workspaces "
f"(slugs: {candidates}). Use the workspace slug or tenant_id to disambiguate."
)
if name_matches:
return name_matches[0]
available = ", ".join(workspace.slug for workspace in workspaces)
raise ValueError(
f"Workspace '{workspace_identifier}' was not found by slug, tenant_id, or name. "
f"Available workspace slugs: {available}"
)
async def resolve_workspace_project_identifier(
project: str,
context: Optional[Context] = None,
) -> WorkspaceProjectEntry:
"""Resolve a project by external_id (UUID), qualified name, or unqualified name."""
index = await _ensure_workspace_project_index(context=context)
try:
return await _resolve_workspace_project_from_index(index, project, context)
except WorkspaceProjectLookupMiss:
# Trigger: the lookup missed the session-cached index.
# Why: a miss is exactly the signal the cache may be stale — projects
# created out-of-band (CLI, a teammate in a shared workspace) post-date
# the index built at session start (#956).
# Outcome: rebuild the index once and retry; a second miss is authoritative
# and its error (with the refreshed project list) propagates.
logger.info(
f"Workspace project lookup missed for '{project}'; refreshing index and retrying"
)
refreshed = await _ensure_workspace_project_index(context=context, force_refresh=True)
return await _resolve_workspace_project_from_index(refreshed, project, context)
async def _resolve_workspace_project_from_index(
index: WorkspaceProjectIndex,
project: str,
context: Optional[Context] = None,
) -> WorkspaceProjectEntry:
"""Resolve a project against one concrete index snapshot.
Raises WorkspaceProjectLookupMiss for absent projects (retryable via index
refresh) and plain ValueError for ambiguity, which a refresh cannot fix.
"""
# Fast path: direct lookup by external_id when the identifier is a UUID
# Canonicalize via str(UUID(...)) so uppercase, brace-wrapped, or urn:uuid forms
# all hash to the same lowercase-hyphenated key as the stored external_ids.
@@ -816,23 +960,14 @@ async def resolve_workspace_project_identifier(
except ValueError:
pass
workspace_slug, project_identifier = _split_qualified_project_identifier(project)
workspace_identifier, project_identifier = _split_qualified_project_identifier(project)
project_permalink = generate_permalink(project_identifier)
if workspace_slug:
workspace_matches = [
workspace
for workspace in index.workspaces
if workspace.slug.casefold() == workspace_slug.casefold()
]
if not workspace_matches:
available = ", ".join(workspace.slug for workspace in index.workspaces)
raise ValueError(
f"Workspace '{workspace_slug}' was not found. "
f"Available workspace slugs: {available}"
)
workspace = workspace_matches[0]
if workspace_identifier:
# Honor the documented "slug, name, or tenant_id" contract for the workspace
# segment; _match_workspace_identifier raises a clear error on ambiguous names
# and unknown identifiers, listing what forms were tried.
workspace = _match_workspace_identifier(index.workspaces, workspace_identifier)
matches = [
entry
for entry in index.entries_by_permalink.get(project_permalink, ())
@@ -843,7 +978,7 @@ async def resolve_workspace_project_identifier(
failed_workspace.tenant_id == workspace.tenant_id
for failed_workspace in index.failed_workspaces
):
raise ValueError(
raise WorkspaceProjectLookupMiss(
f"Projects for workspace '{workspace.name}' ({workspace.slug}) "
"could not be loaded. Retry after workspace discovery recovers."
)
@@ -852,7 +987,7 @@ async def resolve_workspace_project_identifier(
for entry in index.entries
if entry.workspace.tenant_id == workspace.tenant_id
)
raise ValueError(
raise WorkspaceProjectLookupMiss(
f"Project '{project_identifier}' was not found in workspace "
f"'{workspace.name}' ({workspace.slug}). Available projects: {available}"
)
@@ -877,7 +1012,7 @@ async def resolve_workspace_project_identifier(
"retry or use a qualified project from an indexed workspace."
)
available = ", ".join(entry.qualified_name for entry in index.entries)
raise ValueError(
raise WorkspaceProjectLookupMiss(
f"Project '{project}' was not found in indexed cloud workspaces. "
f"Available projects: {available}.{failed_note}"
)
@@ -1212,6 +1347,9 @@ async def resolve_project_and_path(
# Why: allow project-scoped memory URLs without requiring a separate project parameter
# Outcome: attempt to resolve the prefix as a project and route to it
if project_prefix:
# Deferred: ToolError lives in the mcp SDK, which must not load at CLI startup (#886).
from mcp.server.fastmcp.exceptions import ToolError
if cached_project and _project_matches_identifier(cached_project, project_prefix):
resolved_project = await resolve_project_parameter(project_prefix, context=context)
if resolved_project and generate_permalink(resolved_project) != generate_permalink(
@@ -1438,6 +1576,9 @@ async def get_project_client(
is_factory_mode,
)
# Deferred: ToolError lives in the mcp SDK, which must not load at CLI startup (#886).
from mcp.server.fastmcp.exceptions import ToolError
# When project_id (UUID) is provided, prefer it as the resolution identifier.
# external_id is unambiguous across workspaces; project name can collide.
project_identifier = project_id if project_id else project
@@ -0,0 +1,17 @@
"""Shared loader for the bundled cloud-discovery markdown resources."""
from pathlib import Path
def load_discovery_resource(filename: str) -> str:
"""Read a bundled discovery markdown file with promo placeholders rendered.
The markdown carries a {{OSS_DISCOUNT_CODE}} placeholder so the promo code
has one source of truth (cli.promo); substitute before it reaches users.
"""
# Import here to avoid pulling CLI promo machinery (analytics, rich, config)
# into the MCP server import graph at module load.
from basic_memory.cli.promo import OSS_DISCOUNT_CODE
content = (Path(__file__).parent / filename).read_text(encoding="utf-8")
return content.replace("{{OSS_DISCOUNT_CODE}}", OSS_DISCOUNT_CODE)
+6 -1
View File
@@ -54,7 +54,10 @@ def _format_entity_block(result: ContextResult) -> str:
lines.append("")
lines.append("### Relations")
for rel in relation_items:
lines.append(f"- {rel.relation_type} [[{rel.to_entity}]]")
# Unresolved forward references have no resolved entity yet; fall back
# to the literal target text instead of rendering [[None]] (#955)
target = rel.to_entity or rel.to_name
lines.append(f"- {rel.relation_type} [[{target}]]")
# --- Related entities (non-relation related results) ---
related_entities: list[EntitySummary | ObservationSummary] = [
@@ -112,6 +115,7 @@ def _format_context_markdown(graph: GraphContext, project: str) -> str:
@mcp.tool(
title="Build Context",
description="""Build context from a memory:// URI to continue conversations naturally.
Use this to follow up on previous discussions or explore related topics.
@@ -131,6 +135,7 @@ def _format_context_markdown(graph: GraphContext, project: str) -> str:
- "json" (default): Structured JSON with internal fields excluded
- "text": Compact markdown text for LLM consumption
""",
tags={"navigation", "notes"},
annotations={"readOnlyHint": True, "openWorldHint": False},
)
async def build_context(
+2
View File
@@ -17,7 +17,9 @@ from basic_memory.mcp.tools.utils import call_put, call_post, resolve_entity_id
@mcp.tool(
title="Create Canvas",
description="Create an Obsidian canvas file to visualize concepts and connections.",
tags={"canvas", "notes"},
annotations={"destructiveHint": False, "idempotentHint": True, "openWorldHint": False},
)
async def canvas(
@@ -105,7 +105,9 @@ def _format_document_for_chatgpt(
@mcp.tool(
title="Search Knowledge Base",
description="Search for content across the knowledge base",
tags={"search"},
annotations={"readOnlyHint": True, "openWorldHint": False},
)
async def search(
@@ -167,7 +169,9 @@ async def search(
@mcp.tool(
title="Fetch Document",
description="Fetch the full contents of a search result document",
tags={"search", "notes"},
annotations={"readOnlyHint": True, "openWorldHint": False},
)
async def fetch(
+4 -4
View File
@@ -1,15 +1,15 @@
"""Cloud information MCP tool."""
from pathlib import Path
from basic_memory.mcp.resources.discovery import load_discovery_resource
from basic_memory.mcp.server import mcp
@mcp.tool(
"cloud_info",
title="Cloud Info",
tags={"cloud"},
annotations={"readOnlyHint": True, "openWorldHint": False},
)
def cloud_info() -> str:
"""Return optional Basic Memory Cloud information and setup guidance."""
content_path = Path(__file__).parent.parent / "resources" / "cloud_info.md"
return content_path.read_text(encoding="utf-8")
return load_discovery_resource("cloud_info.md")
@@ -182,7 +182,9 @@ def _directory_path_for_delete(
@mcp.tool(
title="Delete Note",
description="Delete a note or directory by title, permalink, or path",
tags={"notes"},
annotations={"destructiveHint": True, "openWorldHint": False},
)
async def delete_note(
+2
View File
@@ -304,7 +304,9 @@ Error editing note '{identifier}': {error_message}
@mcp.tool(
title="Edit Note",
description="Edit an existing markdown note using various operations like append, prepend, find_replace, replace_section, insert_before_section, or insert_after_section.",
tags={"notes"},
annotations={"destructiveHint": False, "openWorldHint": False},
)
async def edit_note(
@@ -11,7 +11,9 @@ from basic_memory.mcp.server import mcp
@mcp.tool(
title="List Directory",
description="List directory contents with filtering and depth control.",
tags={"navigation", "notes"},
annotations={"readOnlyHint": True, "openWorldHint": False},
)
async def list_directory(
+2
View File
@@ -357,7 +357,9 @@ delete_note("{identifier}")
@mcp.tool(
title="Move Note",
description="Move a note or directory to a new location, updating database and maintaining links.",
tags={"notes"},
annotations={"destructiveHint": False, "openWorldHint": False},
)
async def move_note(
@@ -357,6 +357,8 @@ def _format_project_list_json(
@mcp.tool(
"list_memory_projects",
title="List Memory Projects",
tags={"projects"},
annotations={"readOnlyHint": True, "openWorldHint": False},
)
async def list_memory_projects(
@@ -475,10 +477,14 @@ async def _resolve_workspace_routing(
if workspace is None:
return None
explicit_cloud_routing = _explicit_routing() and not _force_local_mode()
forced_local = _explicit_routing() and _force_local_mode()
config = ConfigManager().config
# Resolve whenever credentials make workspace discovery possible — not only
# under explicit --cloud. A workspace selector implies cloud routing
# (get_client routes it to the cloud proxy, #954), and the transport needs
# the tenant id in X-Workspace-ID, not a slug or display name.
should_resolve_workspace = is_factory_mode() or (
explicit_cloud_routing and has_cloud_credentials(config)
has_cloud_credentials(config) and not forced_local
)
if not should_resolve_workspace:
return workspace
@@ -495,6 +501,8 @@ async def _resolve_workspace_routing(
@mcp.tool(
"create_memory_project",
title="Create Memory Project",
tags={"projects"},
annotations={"destructiveHint": False, "openWorldHint": False},
)
async def create_memory_project(
@@ -517,8 +525,9 @@ async def create_memory_project(
workspace: Optional cloud workspace selector to create the project in. Slug is
preferred for AI callers, but tenant_id and unique name are also accepted.
When omitted, the connection's default workspace is used. Discover values
via `list_workspaces`. In local mode the selector is passed through
without slug resolution.
via `list_workspaces`. A workspace selector implies cloud routing:
without cloud credentials the call fails fast instead of silently
creating a local project (#954).
output_format: "text" returns the existing human-readable result text.
"json" returns structured project creation metadata.
context: Optional FastMCP context for progress/status logging.
@@ -636,6 +645,8 @@ async def create_memory_project(
@mcp.tool(
title="Delete Project",
tags={"projects"},
annotations={"destructiveHint": True, "openWorldHint": False},
)
async def delete_project(
@@ -654,8 +665,9 @@ async def delete_project(
workspace: Optional cloud workspace selector to delete the project from.
Slug is preferred for AI callers, but tenant_id and unique name are
also accepted. When omitted, the connection's default workspace is
used. In local mode the selector is passed through without slug
resolution, matching create_memory_project behavior.
used. A workspace selector implies cloud routing: without cloud
credentials the call fails fast, matching create_memory_project
behavior (#954).
Returns:
Confirmation message about project deletion
@@ -155,7 +155,9 @@ def optimize_image(img, content_length, max_output_bytes=350000):
@mcp.tool(
title="Read Content",
description="Read a file's raw content by path or permalink",
tags={"notes"},
annotations={"readOnlyHint": True, "openWorldHint": False},
)
async def read_content(
+2
View File
@@ -74,7 +74,9 @@ def _parse_opening_frontmatter(content: str) -> tuple[str, dict | None]:
@mcp.tool(
title="Read Note",
description="Read a markdown note by title or permalink.",
tags={"notes"},
# TODO: re-enable once MCP client rendering is working
# meta={"ui/resourceUri": "ui://basic-memory/note-preview"},
annotations={"readOnlyHint": True, "openWorldHint": False},
@@ -26,6 +26,7 @@ from basic_memory.schemas.search import SearchItemType
@mcp.tool(
title="Recent Activity",
description="""Get recent activity for a project or across all projects.
Timeframe supports natural language formats like:
@@ -36,6 +37,7 @@ from basic_memory.schemas.search import SearchItemType
- "3 weeks ago"
Or standard formats like "7d"
""",
tags={"navigation", "notes"},
annotations={"readOnlyHint": True, "openWorldHint": False},
)
async def recent_activity(
+4 -4
View File
@@ -1,15 +1,15 @@
"""Release notes MCP tool."""
from pathlib import Path
from basic_memory.mcp.resources.discovery import load_discovery_resource
from basic_memory.mcp.server import mcp
@mcp.tool(
"release_notes",
title="Release Notes",
tags={"cloud"},
annotations={"readOnlyHint": True, "openWorldHint": False},
)
def release_notes() -> str:
"""Return the latest product release notes for optional user review."""
content_path = Path(__file__).parent.parent / "resources" / "release_notes.md"
return content_path.read_text(encoding="utf-8")
return load_discovery_resource("release_notes.md")
+6
View File
@@ -204,7 +204,9 @@ def _no_schema_guidance(note_type: str, tool_name: str) -> str:
@mcp.tool(
title="Validate Schema",
description="Validate notes against their Picoschema definitions.",
tags={"schema"},
annotations={"readOnlyHint": True, "openWorldHint": False},
)
async def schema_validate(
@@ -317,7 +319,9 @@ async def schema_validate(
@mcp.tool(
title="Infer Schema",
description="Analyze existing notes and suggest a Picoschema definition.",
tags={"schema"},
annotations={"readOnlyHint": True, "openWorldHint": False},
)
async def schema_infer(
@@ -438,7 +442,9 @@ async def schema_infer(
@mcp.tool(
title="Schema Diff",
description="Detect drift between a schema definition and actual note usage.",
tags={"schema"},
annotations={"readOnlyHint": True, "openWorldHint": False},
)
async def schema_diff(
+29 -5
View File
@@ -14,7 +14,7 @@ from basic_memory.config import ConfigManager, has_cloud_credentials
from basic_memory.utils import (
build_canonical_permalink,
coerce_dict,
coerce_list,
parse_str_list,
parse_tags,
strict_search_tags,
)
@@ -612,7 +612,9 @@ async def _search_all_projects(
@mcp.tool(
title="Search Notes",
description="Search across all content in the knowledge base with advanced syntax support.",
tags={"search"},
# TODO: re-enable once MCP client rendering is working
# meta={"ui/resourceUri": "ui://basic-memory/search-results"},
annotations={"readOnlyHint": True, "openWorldHint": False},
@@ -649,24 +651,30 @@ async def search_notes(
# Plural-vs-singular trips models constantly. Accept the singular too.
note_types: Annotated[
List[str] | None,
BeforeValidator(coerce_list),
# parse_str_list, not coerce_list: "note,task" must split into ["note", "task"]
# consistent with how tags are handled (#910/#930). coerce_list wraps the whole
# comma string as the single literal type ["note,task"], which matches nothing.
BeforeValidator(parse_str_list),
Field(default=None, validation_alias=AliasChoices("note_types", "note_type", "types")),
"Filter by the 'type' field in note frontmatter (e.g. 'note', 'chapter', 'person'). "
"Accepts a list, a comma-separated string (e.g. 'note,task'), or a JSON-array string. "
"Case-insensitive.",
] = None,
entity_types: Annotated[
List[str] | None,
BeforeValidator(coerce_list),
BeforeValidator(parse_str_list),
Field(default=None, validation_alias=AliasChoices("entity_types", "entity_type")),
"Filter by knowledge graph item type: 'entity' (whole notes), 'observation', or "
"'relation'. Defaults to 'entity'. Do NOT pass schema/frontmatter types like "
"'Chapter' here — use note_types instead.",
"'Chapter' here — use note_types instead. "
"Accepts a list, a comma-separated string (e.g. 'entity,observation'), or a JSON-array string.",
] = None,
categories: Annotated[
List[str] | None,
BeforeValidator(coerce_list),
BeforeValidator(parse_str_list),
Field(default=None, validation_alias=AliasChoices("categories", "category")),
"Filter observation results to these exact categories (e.g. ['requirement']). "
"Accepts a list, a comma-separated string (e.g. 'requirement,decision'), or a JSON-array string. "
"Pair with entity_types=['observation'] to return only observations whose "
"category matches exactly — not every row mentioning the word.",
] = None,
@@ -820,6 +828,11 @@ async def search_notes(
Formatted markdown text (output_format="text"), dict (output_format="json"),
or helpful error guidance string if search fails
Pagination note: `total` is exact only for text/title/permalink searches.
Vector and hybrid searches skip the count query (it would cost a second
semantic retrieval pass) and report `total: 0` even when results are
returned use `has_more` for pagination in those modes.
Examples:
# Basic text search
results = await search_notes("project planning")
@@ -893,6 +906,17 @@ async def search_notes(
if page_size < 1:
raise ValueError(f"page_size must be >= 1, got {page_size}")
# Trigger: list params arrived via a direct function call instead of the MCP layer.
# Why: the BeforeValidator annotations only run through MCP/Pydantic validation; direct
# callers (e.g. `bm tool search-notes --type note,task` in cli/commands/tool.py,
# which Typer collects as the one-element list ["note,task"]) would otherwise
# forward the comma string as one literal type that matches nothing (#930).
# Outcome: comma-split/list normalization applies on every path; parse_str_list is
# idempotent, so MCP-validated input passes through unchanged.
note_types = parse_str_list(note_types) if note_types is not None else []
entity_types = parse_str_list(entity_types) if entity_types is not None else []
categories = parse_str_list(categories) if categories is not None else []
# Avoid mutable-default-argument footguns. Treat None as "no filter".
# Lowercase note_types so "Chapter" matches the stored "chapter".
note_types = [t.lower() for t in note_types] if note_types else []
+4
View File
@@ -18,7 +18,9 @@ def _text_block(message: str) -> List[ContentBlock]:
@mcp.tool(
title="Search Notes (UI)",
description="Search notes and return an embedded MCP-UI resource (raw HTML).",
tags={"search", "ui"},
output_schema=None,
annotations={"readOnlyHint": True, "openWorldHint": False},
)
@@ -92,7 +94,9 @@ async def search_notes_ui(
@mcp.tool(
title="Read Note (UI)",
description="Read a note and return an embedded MCP-UI resource (raw HTML).",
tags={"notes", "ui"},
output_schema=None,
annotations={"readOnlyHint": True, "openWorldHint": False},
)
+2
View File
@@ -11,7 +11,9 @@ from basic_memory.mcp.tools.read_note import read_note
@mcp.tool(
title="View Note",
description="View a note as a formatted artifact for better readability.",
tags={"notes"},
annotations={"readOnlyHint": True, "openWorldHint": False},
)
async def view_note(
+2
View File
@@ -45,7 +45,9 @@ def _workspace_list_response(workspaces: list[WorkspaceInfo]) -> WorkspaceListRe
@mcp.tool(
title="List Workspaces",
description="List available cloud workspaces (tenant_id, type, role, and name).",
tags={"cloud", "projects"},
annotations={"readOnlyHint": True, "openWorldHint": False},
)
async def list_workspaces(
+39
View File
@@ -25,8 +25,39 @@ from basic_memory.workspace_context import current_workspace_permalink_context
TagType = Union[List[str], str, None]
def _compose_workspace_project_route(
*,
workspace: Optional[str],
project: Optional[str],
project_id: Optional[str],
) -> Optional[str]:
"""Return the explicit project route requested by workspace/project args."""
if workspace is None:
return project
cleaned_workspace = workspace.strip().strip("/")
if not cleaned_workspace:
raise ValueError("workspace must not be empty when provided")
if "/" in cleaned_workspace:
raise ValueError("workspace must be a single workspace slug, name, or tenant_id")
if project_id is not None:
raise ValueError("workspace cannot be combined with project_id; use project_id alone")
if project is None or not project.strip().strip("/"):
raise ValueError("workspace requires an explicit project argument")
cleaned_project = project.strip().strip("/")
if "/" in cleaned_project:
raise ValueError(
"Use either workspace='workspace' with project='project', "
"or project='workspace/project', not both"
)
return f"{cleaned_workspace}/{cleaned_project}"
@mcp.tool(
title="Write Note",
description="Create a markdown note. If the note already exists, returns an error by default — pass overwrite=True to replace.",
tags={"notes"},
annotations={"destructiveHint": True, "idempotentHint": False, "openWorldHint": False},
)
async def write_note(
@@ -38,6 +69,7 @@ async def write_note(
Field(validation_alias=AliasChoices("directory", "folder", "dir", "path")),
],
project: Optional[str] = None,
workspace: Optional[str] = None,
project_id: Optional[str] = None,
tags: list[str] | str | None = None,
note_type: str = "note",
@@ -93,6 +125,8 @@ async def write_note(
form (or project_id) to disambiguate. If unknown, use
list_memory_projects() to discover available projects and their
qualified names.
workspace: Workspace slug, name, or tenant_id. When provided with `project`,
routes as `workspace/project`. Cannot be combined with `project_id`.
project_id: Project external_id (UUID). Prefer this over `project` when known
it routes to the exact project regardless of name collisions across cloud
workspaces. Takes precedence over `project`. Get from list_memory_projects().
@@ -170,6 +204,11 @@ async def write_note(
effective_overwrite = (
overwrite if overwrite is not None else ConfigManager().config.write_note_overwrite_default
)
project = _compose_workspace_project_route(
workspace=workspace,
project=project,
project_id=project_id,
)
with logfire.span(
"mcp.tool.write_note",
+133 -133
View File
@@ -7,12 +7,11 @@ from typing import List, Optional, Sequence, Union, Any
from loguru import logger
from sqlalchemy import exists, func, select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import load_only, selectinload
from sqlalchemy.orm.interfaces import LoaderOption
from sqlalchemy.engine import Row
from basic_memory import db
from basic_memory.models.knowledge import Entity, Observation, Relation
from basic_memory.repository.repository import Repository
@@ -24,16 +23,17 @@ class EntityRepository(Repository[Entity]):
to strings before passing to repository methods.
"""
def __init__(self, session_maker: async_sessionmaker[AsyncSession], project_id: int):
"""Initialize with session maker and project_id filter.
def __init__(self, project_id: int):
"""Initialize with project_id filter.
Args:
session_maker: SQLAlchemy session maker
project_id: Project ID to filter all operations by
"""
super().__init__(session_maker, Entity, project_id=project_id)
super().__init__(Entity, project_id=project_id)
async def get_by_id(self, entity_id: int, *, load_relations: bool = True) -> Optional[Entity]:
async def get_by_id(
self, session: AsyncSession, entity_id: int, *, load_relations: bool = True
) -> Optional[Entity]:
"""Get entity by numeric ID.
Args:
@@ -42,23 +42,24 @@ class EntityRepository(Repository[Entity]):
Returns:
Entity if found, None otherwise
"""
async with db.scoped_session(self.session_maker) as session:
if not load_relations:
result = await session.execute(self.select().where(Entity.id == entity_id))
return result.scalars().one_or_none()
if not load_relations:
result = await session.execute(self.select().where(Entity.id == entity_id))
return result.scalars().one_or_none()
return await self.select_by_id(session, entity_id)
return await self.select_by_id(session, entity_id)
async def _find_one_by_query(self, query, *, load_relations: bool) -> Optional[Entity]:
async def _find_one_by_query(
self, session: AsyncSession, query, *, load_relations: bool
) -> Optional[Entity]:
"""Return one entity row with optional eager loading."""
if load_relations:
return await self.find_one(query)
return await self.find_one(session, query)
result = await self.execute_query(query, use_query_options=False)
result = await self.execute_query(session, query, use_query_options=False)
return result.scalars().one_or_none()
async def get_by_external_id(
self, external_id: str, *, load_relations: bool = True
self, session: AsyncSession, external_id: str, *, load_relations: bool = True
) -> Optional[Entity]:
"""Get entity by external UUID.
@@ -69,10 +70,10 @@ class EntityRepository(Repository[Entity]):
Entity if found, None otherwise
"""
query = self.select().where(Entity.external_id == external_id)
return await self._find_one_by_query(query, load_relations=load_relations)
return await self._find_one_by_query(session, query, load_relations=load_relations)
async def get_by_permalink(
self, permalink: str, *, load_relations: bool = True
self, session: AsyncSession, permalink: str, *, load_relations: bool = True
) -> Optional[Entity]:
"""Get entity by permalink.
@@ -80,9 +81,11 @@ class EntityRepository(Repository[Entity]):
permalink: Unique identifier for the entity
"""
query = self.select().where(Entity.permalink == permalink)
return await self._find_one_by_query(query, load_relations=load_relations)
return await self._find_one_by_query(session, query, load_relations=load_relations)
async def get_by_title(self, title: str, *, load_relations: bool = True) -> Sequence[Entity]:
async def get_by_title(
self, session: AsyncSession, title: str, *, load_relations: bool = True
) -> Sequence[Entity]:
"""Get entities by title, ordered by shortest path first.
When multiple entities share the same title (in different folders),
@@ -97,11 +100,11 @@ class EntityRepository(Repository[Entity]):
.where(Entity.title == title)
.order_by(func.length(Entity.file_path), Entity.file_path)
)
result = await self.execute_query(query, use_query_options=load_relations)
result = await self.execute_query(session, query, use_query_options=load_relations)
return list(result.scalars().all())
async def get_by_file_path(
self, file_path: Union[Path, str], *, load_relations: bool = True
self, session: AsyncSession, file_path: Union[Path, str], *, load_relations: bool = True
) -> Optional[Entity]:
"""Get entity by file_path.
@@ -109,13 +112,13 @@ class EntityRepository(Repository[Entity]):
file_path: Path to the entity file (will be converted to string internally)
"""
query = self.select().where(Entity.file_path == Path(file_path).as_posix())
return await self._find_one_by_query(query, load_relations=load_relations)
return await self._find_one_by_query(session, query, load_relations=load_relations)
# -------------------------------------------------------------------------
# Lightweight methods for permalink resolution (no eager loading)
# -------------------------------------------------------------------------
async def permalink_exists(self, permalink: str) -> bool:
async def permalink_exists(self, session: AsyncSession, permalink: str) -> bool:
"""Check if a permalink exists without loading the full entity.
This is much faster than get_by_permalink() as it skips eager loading
@@ -129,10 +132,12 @@ class EntityRepository(Repository[Entity]):
"""
query = select(Entity.id).where(Entity.permalink == permalink).limit(1)
query = self._add_project_filter(query)
result = await self.execute_query(query, use_query_options=False)
result = await self.execute_query(session, query, use_query_options=False)
return result.scalar_one_or_none() is not None
async def get_file_path_for_permalink(self, permalink: str) -> Optional[str]:
async def get_file_path_for_permalink(
self, session: AsyncSession, permalink: str
) -> Optional[str]:
"""Get the file_path for a permalink without loading the full entity.
Use when you only need the file_path, not the full entity with relations.
@@ -145,10 +150,12 @@ class EntityRepository(Repository[Entity]):
"""
query = select(Entity.file_path).where(Entity.permalink == permalink)
query = self._add_project_filter(query)
result = await self.execute_query(query, use_query_options=False)
result = await self.execute_query(session, query, use_query_options=False)
return result.scalar_one_or_none()
async def get_permalink_for_file_path(self, file_path: Union[Path, str]) -> Optional[str]:
async def get_permalink_for_file_path(
self, session: AsyncSession, file_path: Union[Path, str]
) -> Optional[str]:
"""Get the permalink for a file_path without loading the full entity.
Use when you only need the permalink, not the full entity with relations.
@@ -161,10 +168,10 @@ class EntityRepository(Repository[Entity]):
"""
query = select(Entity.permalink).where(Entity.file_path == Path(file_path).as_posix())
query = self._add_project_filter(query)
result = await self.execute_query(query, use_query_options=False)
result = await self.execute_query(session, query, use_query_options=False)
return result.scalar_one_or_none()
async def get_all_permalinks(self) -> List[str]:
async def get_all_permalinks(self, session: AsyncSession) -> List[str]:
"""Get all permalinks for this project.
Optimized for bulk operations - returns only permalink strings
@@ -175,11 +182,11 @@ class EntityRepository(Repository[Entity]):
"""
query = select(Entity.permalink)
query = self._add_project_filter(query)
result = await self.execute_query(query, use_query_options=False)
result = await self.execute_query(session, query, use_query_options=False)
return list(result.scalars().all())
async def find_by_ids_for_hydration(
self, ids: List[int], *, include_cross_project: bool = False
self, session: AsyncSession, ids: List[int], *, include_cross_project: bool = False
) -> Sequence[Entity]:
"""Fetch minimal entity fields needed for context hydration.
@@ -203,10 +210,10 @@ class EntityRepository(Repository[Entity]):
if not include_cross_project:
query = self._add_project_filter(query)
result = await self.execute_query(query, use_query_options=False)
result = await self.execute_query(session, query, use_query_options=False)
return list(result.scalars().all())
async def get_permalink_to_file_path_map(self) -> dict[str, str]:
async def get_permalink_to_file_path_map(self, session: AsyncSession) -> dict[str, str]:
"""Get a mapping of permalink -> file_path for all entities.
Optimized for bulk permalink resolution - loads minimal data in one query.
@@ -216,10 +223,10 @@ class EntityRepository(Repository[Entity]):
"""
query = select(Entity.permalink, Entity.file_path)
query = self._add_project_filter(query)
result = await self.execute_query(query, use_query_options=False)
result = await self.execute_query(session, query, use_query_options=False)
return {row.permalink: row.file_path for row in result.all()}
async def get_file_path_to_permalink_map(self) -> dict[str, str]:
async def get_file_path_to_permalink_map(self, session: AsyncSession) -> dict[str, str]:
"""Get a mapping of file_path -> permalink for all entities.
Optimized for bulk permalink resolution - loads minimal data in one query.
@@ -229,7 +236,7 @@ class EntityRepository(Repository[Entity]):
"""
query = select(Entity.file_path, Entity.permalink)
query = self._add_project_filter(query)
result = await self.execute_query(query, use_query_options=False)
result = await self.execute_query(session, query, use_query_options=False)
return {row.file_path: row.permalink for row in result.all()}
async def get_by_file_paths(
@@ -262,7 +269,7 @@ class EntityRepository(Repository[Entity]):
result = await session.execute(query) # pragma: no cover
return list(result.all()) # pragma: no cover
async def find_by_checksum(self, checksum: str) -> Sequence[Entity]:
async def find_by_checksum(self, session: AsyncSession, checksum: str) -> Sequence[Entity]:
"""Find entities with the given checksum.
Used for move detection - finds entities that may have been moved to a new path.
@@ -276,10 +283,12 @@ class EntityRepository(Repository[Entity]):
"""
query = self.select().where(Entity.checksum == checksum)
# Don't load relationships for move detection - we only need file_path and checksum
result = await self.execute_query(query, use_query_options=False)
result = await self.execute_query(session, query, use_query_options=False)
return list(result.scalars().all())
async def find_by_checksums(self, checksums: Sequence[str]) -> Sequence[Entity]:
async def find_by_checksums(
self, session: AsyncSession, checksums: Sequence[str]
) -> Sequence[Entity]:
"""Find entities with any of the given checksums (batch query for move detection).
This is a batch-optimized version of find_by_checksum() that queries multiple checksums
@@ -304,16 +313,18 @@ class EntityRepository(Repository[Entity]):
# Query: SELECT * FROM entities WHERE checksum IN (checksum1, checksum2, ...)
query = self.select().where(Entity.checksum.in_(checksums)) # pragma: no cover
# Don't load relationships for move detection - we only need file_path and checksum
result = await self.execute_query(query, use_query_options=False) # pragma: no cover
result = await self.execute_query(
session, query, use_query_options=False
) # pragma: no cover
return list(result.scalars().all()) # pragma: no cover
async def delete_by_file_path(self, file_path: Union[Path, str]) -> bool:
async def delete_by_file_path(self, session: AsyncSession, file_path: Union[Path, str]) -> bool:
"""Delete entity with the provided file_path.
Args:
file_path: Path to the entity file (will be converted to string internally)
"""
return await self.delete_by_fields(file_path=Path(file_path).as_posix())
return await self.delete_by_fields(session, file_path=Path(file_path).as_posix())
def get_load_options(self) -> List[LoaderOption]:
"""Get SQLAlchemy loader options for eager loading relationships."""
@@ -327,7 +338,9 @@ class EntityRepository(Repository[Entity]):
selectinload(Entity.incoming_relations).selectinload(Relation.to_entity),
]
async def find_by_permalinks(self, permalinks: List[str]) -> Sequence[Entity]:
async def find_by_permalinks(
self, session: AsyncSession, permalinks: List[str]
) -> Sequence[Entity]:
"""Find multiple entities by their permalink.
Args:
@@ -342,10 +355,10 @@ class EntityRepository(Repository[Entity]):
self.select().options(*self.get_load_options()).where(Entity.permalink.in_(permalinks))
)
result = await self.execute_query(query)
result = await self.execute_query(session, query)
return list(result.scalars().all())
async def upsert_entity(self, entity: Entity) -> Entity:
async def upsert_entity(self, session: AsyncSession, entity: Entity) -> Entity:
"""Insert or update entity using simple try/catch with database-level conflict resolution.
Handles file_path race conditions by checking for existing entity on IntegrityError.
@@ -357,99 +370,84 @@ class EntityRepository(Repository[Entity]):
Returns:
The inserted or updated entity
"""
async with db.scoped_session(self.session_maker) as session:
# Set project_id if applicable and not already set
self._set_project_id_if_needed(entity)
# Set project_id if applicable and not already set
self._set_project_id_if_needed(entity)
# Try simple insert first
try:
# Try simple insert first.
try:
async with session.begin_nested():
session.add(entity)
await session.flush()
except IntegrityError as e:
# Check if this is a FOREIGN KEY constraint failure
# SQLite: "FOREIGN KEY constraint failed"
# Postgres: "violates foreign key constraint"
error_str = str(e)
if (
"FOREIGN KEY constraint failed" in error_str
or "violates foreign key constraint" in error_str
):
# Import locally to avoid circular dependency (repository -> services -> repository)
from basic_memory.services.exceptions import SyncFatalError
# Return with relationships loaded
query = (
self.select()
.where(Entity.file_path == entity.file_path)
.options(*self.get_load_options())
# Project doesn't exist in database - this is a fatal sync error
raise SyncFatalError(
f"Cannot sync file '{entity.file_path}': "
f"project_id={entity.project_id} does not exist in database. "
f"The project may have been deleted. This sync will be terminated."
) from e
# Re-query after the nested rollback to get a fresh, attached entity.
existing_result = await session.execute(
select(Entity)
.where(Entity.file_path == entity.file_path, Entity.project_id == entity.project_id)
.options(*self.get_load_options())
)
existing_entity = existing_result.scalar_one_or_none()
if existing_entity:
# File path conflict - update the existing entity
logger.debug(
f"Resolving file_path conflict for {entity.file_path}, "
f"entity_id={existing_entity.id}, observations={len(entity.observations)}"
)
result = await session.execute(query)
found = result.scalar_one_or_none()
if not found: # pragma: no cover
raise RuntimeError(
f"Failed to retrieve entity after insert: {entity.file_path}"
)
return found
# Use merge to avoid session state conflicts. Preserve stable
# external_id so external references survive re-indexing.
entity.id = existing_entity.id
entity.external_id = existing_entity.external_id
except IntegrityError as e:
# Check if this is a FOREIGN KEY constraint failure
# SQLite: "FOREIGN KEY constraint failed"
# Postgres: "violates foreign key constraint"
error_str = str(e)
if (
"FOREIGN KEY constraint failed" in error_str
or "violates foreign key constraint" in error_str
):
# Import locally to avoid circular dependency (repository -> services -> repository)
from basic_memory.services.exceptions import SyncFatalError
# Ensure observations reference the correct entity_id.
for obs in entity.observations:
obs.entity_id = existing_entity.id
obs.id = None
# Project doesn't exist in database - this is a fatal sync error
raise SyncFatalError(
f"Cannot sync file '{entity.file_path}': "
f"project_id={entity.project_id} does not exist in database. "
f"The project may have been deleted. This sync will be terminated."
) from e
merged_entity = await session.merge(entity)
await session.flush()
await session.rollback()
# Re-query after rollback to get a fresh, attached entity
existing_result = await session.execute(
# Re-query to get proper relationships loaded.
final_result = await session.execute(
select(Entity)
.where(
Entity.file_path == entity.file_path, Entity.project_id == entity.project_id
)
.where(Entity.id == merged_entity.id)
.options(*self.get_load_options())
)
existing_entity = existing_result.scalar_one_or_none()
return final_result.scalar_one()
if existing_entity:
# File path conflict - update the existing entity
logger.debug(
f"Resolving file_path conflict for {entity.file_path}, "
f"entity_id={existing_entity.id}, observations={len(entity.observations)}"
)
# Use merge to avoid session state conflicts
# Set the ID to update existing entity
entity.id = existing_entity.id
# Preserve the stable external_id so that external references
# (e.g. public share links) survive re-indexing
entity.external_id = existing_entity.external_id
# No file_path conflict - must be permalink conflict.
return await self._handle_permalink_conflict(entity, session)
# Ensure observations reference the correct entity_id
for obs in entity.observations:
obs.entity_id = existing_entity.id
# Clear any existing ID to force INSERT as new observation
obs.id = None
# Return with relationships loaded.
query = (
self.select()
.where(Entity.file_path == entity.file_path)
.options(*self.get_load_options())
)
result = await session.execute(query)
found = result.scalar_one_or_none()
if not found: # pragma: no cover
raise RuntimeError(f"Failed to retrieve entity after insert: {entity.file_path}")
return found
# Merge the entity which will update the existing one
merged_entity = await session.merge(entity)
await session.commit()
# Re-query to get proper relationships loaded
final_result = await session.execute(
select(Entity)
.where(Entity.id == merged_entity.id)
.options(*self.get_load_options())
)
return final_result.scalar_one()
else:
# No file_path conflict - must be permalink conflict
# Generate unique permalink and retry
entity = await self._handle_permalink_conflict(entity, session)
return entity
async def get_all_file_paths(self) -> List[str]:
async def get_all_file_paths(self, session: AsyncSession) -> List[str]:
"""Get all file paths for this project - optimized for deletion detection.
Returns only file_path strings without loading entities or relationships.
@@ -461,10 +459,10 @@ class EntityRepository(Repository[Entity]):
query = select(Entity.file_path)
query = self._add_project_filter(query)
result = await self.execute_query(query, use_query_options=False)
result = await self.execute_query(session, query, use_query_options=False)
return list(result.scalars().all())
async def find_without_relations(self) -> Sequence[Entity]:
async def find_without_relations(self, session: AsyncSession) -> Sequence[Entity]:
"""Find entities that have no incoming or outgoing relations."""
# Trigger: entity appears as a source in any relation.
# Why: even unresolved outgoing links mean the entity references another node.
@@ -477,10 +475,10 @@ class EntityRepository(Repository[Entity]):
has_incoming = exists().where(Relation.to_id == Entity.id)
query = self.select().where(~has_outgoing).where(~has_incoming).order_by(Entity.file_path)
result = await self.execute_query(query, use_query_options=False)
result = await self.execute_query(session, query, use_query_options=False)
return list(result.scalars().all())
async def get_distinct_directories(self) -> List[str]:
async def get_distinct_directories(self, session: AsyncSession) -> List[str]:
"""Extract unique directory paths from file_path column.
Optimized method for getting directory structure without loading full entities
@@ -494,7 +492,7 @@ class EntityRepository(Repository[Entity]):
query = self._add_project_filter(query)
# Execute with use_query_options=False to skip eager loading
result = await self.execute_query(query, use_query_options=False)
result = await self.execute_query(session, query, use_query_options=False)
file_paths = [row for row in result.scalars().all()]
# Parse file paths to extract unique directories
@@ -508,7 +506,9 @@ class EntityRepository(Repository[Entity]):
return sorted(directories)
async def find_by_directory_prefix(self, directory_prefix: str) -> Sequence[Entity]:
async def find_by_directory_prefix(
self, session: AsyncSession, directory_prefix: str
) -> Sequence[Entity]:
"""Find entities whose file_path starts with the given directory prefix.
Optimized method for listing directory contents without loading all entities.
@@ -524,7 +524,7 @@ class EntityRepository(Repository[Entity]):
# Build SQL LIKE pattern
if directory_prefix == "" or directory_prefix == "/":
# Root directory - return all entities
return await self.find_all()
return await self.find_all(session)
# Remove leading/trailing slashes for consistency
directory_prefix = directory_prefix.strip("/")
@@ -537,7 +537,7 @@ class EntityRepository(Repository[Entity]):
query = self.select().where(Entity.file_path.like(pattern))
# Skip eager loading - we only need basic entity fields for directory trees
result = await self.execute_query(query, use_query_options=False)
result = await self.execute_query(session, query, use_query_options=False)
return list(result.scalars().all())
async def _handle_permalink_conflict(self, entity: Entity, session: AsyncSession) -> Entity:
@@ -4,9 +4,8 @@ from pathlib import Path
from typing import Any, Mapping, Optional
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from sqlalchemy.ext.asyncio import AsyncSession
from basic_memory import db
from basic_memory.models import Entity, NoteContent
from basic_memory.repository.repository import Repository
@@ -30,9 +29,9 @@ NOTE_CONTENT_MUTABLE_FIELDS = frozenset(
class NoteContentRepository(Repository[NoteContent]):
"""Repository for project-scoped note materialization state."""
def __init__(self, session_maker: async_sessionmaker[AsyncSession], project_id: int):
"""Initialize with session maker and project-scoped filtering."""
super().__init__(session_maker, NoteContent, project_id=project_id)
def __init__(self, project_id: int):
"""Initialize with project-scoped filtering."""
super().__init__(NoteContent, project_id=project_id)
def _coerce_note_content(
self, data: Mapping[str, Any] | NoteContent
@@ -75,16 +74,22 @@ class NoteContentRepository(Repository[NoteContent]):
note_content.external_id = entity.external_id
note_content.file_path = Path(entity.file_path).as_posix()
async def get_by_entity_id(self, entity_id: int) -> Optional[NoteContent]:
async def get_by_entity_id(
self, session: AsyncSession, entity_id: int
) -> Optional[NoteContent]:
"""Get note content by the owning entity identifier."""
return await self.find_by_id(entity_id)
return await self.find_by_id(session, entity_id)
async def get_by_external_id(self, external_id: str) -> Optional[NoteContent]:
async def get_by_external_id(
self, session: AsyncSession, external_id: str
) -> Optional[NoteContent]:
"""Get note content by the mirrored entity external identifier."""
query = self.select().where(NoteContent.external_id == external_id)
return await self.find_one(query)
return await self.find_one(session, query)
async def get_by_file_path(self, file_path: Path | str) -> Optional[NoteContent]:
async def get_by_file_path(
self, session: AsyncSession, file_path: Path | str
) -> Optional[NoteContent]:
"""Get note content by file path, preferring rows whose entity still owns that path."""
normalized_path = Path(file_path).as_posix()
@@ -104,88 +109,88 @@ class NoteContentRepository(Repository[NoteContent]):
.options(*self.get_load_options())
)
async with db.scoped_session(self.session_maker) as session:
result = await session.execute(query)
return result.scalars().first()
result = await session.execute(query)
return result.scalars().first()
async def create(self, data: Mapping[str, Any] | NoteContent) -> NoteContent:
async def create(
self, session: AsyncSession, data: Mapping[str, Any] | NoteContent
) -> NoteContent:
"""Create a note_content row aligned to its owning entity."""
note_content, _ = self._coerce_note_content(data)
async with db.scoped_session(self.session_maker) as session:
await self._align_identity_fields(session, note_content)
session.add(note_content)
await session.flush()
await self._align_identity_fields(session, note_content)
session.add(note_content)
await session.flush()
created = await self.select_by_id(session, note_content.entity_id)
if created is None: # pragma: no cover
raise ValueError(
f"Can't find NoteContent for entity {note_content.entity_id} after add"
)
return created
created = await self.select_by_id(session, note_content.entity_id)
if created is None: # pragma: no cover
raise ValueError(
f"Can't find NoteContent for entity {note_content.entity_id} after add"
)
return created
async def upsert(self, data: Mapping[str, Any] | NoteContent) -> NoteContent:
async def upsert(
self, session: AsyncSession, data: Mapping[str, Any] | NoteContent
) -> NoteContent:
"""Insert or update note_content while keeping mirrored identity fields in sync."""
note_content, provided_fields = self._coerce_note_content(data)
async with db.scoped_session(self.session_maker) as session:
await self._align_identity_fields(session, note_content)
existing = await self.select_by_id(session, note_content.entity_id)
if existing is None:
session.add(note_content)
await session.flush()
created = await self.select_by_id(session, note_content.entity_id)
if created is None: # pragma: no cover
raise ValueError(
f"Can't find NoteContent for entity {note_content.entity_id} after upsert"
)
return created
fields_to_update = (provided_fields - {"entity_id"}) | {
"project_id",
"external_id",
"file_path",
}
for column_name in fields_to_update:
setattr(existing, column_name, getattr(note_content, column_name))
await self._align_identity_fields(session, note_content)
existing = await self.select_by_id(session, note_content.entity_id)
if existing is None:
session.add(note_content)
await session.flush()
updated = await self.select_by_id(session, existing.entity_id)
if updated is None: # pragma: no cover
created = await self.select_by_id(session, note_content.entity_id)
if created is None: # pragma: no cover
raise ValueError(
f"Can't find NoteContent for entity {existing.entity_id} after upsert"
f"Can't find NoteContent for entity {note_content.entity_id} after upsert"
)
return updated
return created
async def update_state_fields(self, entity_id: int, **updates: Any) -> Optional[NoteContent]:
fields_to_update = (provided_fields - {"entity_id"}) | {
"project_id",
"external_id",
"file_path",
}
for column_name in fields_to_update:
setattr(existing, column_name, getattr(note_content, column_name))
await session.flush()
updated = await self.select_by_id(session, existing.entity_id)
if updated is None: # pragma: no cover
raise ValueError(f"Can't find NoteContent for entity {existing.entity_id} after upsert")
return updated
async def update_state_fields(
self, session: AsyncSession, entity_id: int, **updates: Any
) -> Optional[NoteContent]:
"""Update sync fields and re-align project_id, external_id, and file_path from entity."""
invalid_fields = set(updates) - NOTE_CONTENT_MUTABLE_FIELDS
if invalid_fields:
invalid_list = ", ".join(sorted(invalid_fields))
raise ValueError(f"Unsupported note_content update fields: {invalid_list}")
async with db.scoped_session(self.session_maker) as session:
note_content = await self.select_by_id(session, entity_id)
if note_content is None:
return None
note_content = await self.select_by_id(session, entity_id)
if note_content is None:
return None
await self._align_identity_fields(session, note_content)
for field_name, value in updates.items():
setattr(note_content, field_name, value)
await self._align_identity_fields(session, note_content)
for field_name, value in updates.items():
setattr(note_content, field_name, value)
await session.flush()
updated = await self.select_by_id(session, entity_id)
if updated is None: # pragma: no cover
raise ValueError(f"Can't find NoteContent for entity {entity_id} after update")
return updated
await session.flush()
updated = await self.select_by_id(session, entity_id)
if updated is None: # pragma: no cover
raise ValueError(f"Can't find NoteContent for entity {entity_id} after update")
return updated
async def delete_by_entity_id(self, entity_id: int) -> bool:
async def delete_by_entity_id(self, session: AsyncSession, entity_id: int) -> bool:
"""Delete note_content by entity identifier."""
async with db.scoped_session(self.session_maker) as session:
note_content = await self.select_by_id(session, entity_id)
if note_content is None:
return False
note_content = await self.select_by_id(session, entity_id)
if note_content is None:
return False
await session.delete(note_content)
return True
await session.delete(note_content)
await session.flush()
return True
@@ -3,7 +3,7 @@
from typing import Dict, List, Sequence
from sqlalchemy import select
from sqlalchemy.ext.asyncio import async_sessionmaker
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import selectinload
from sqlalchemy.orm.interfaces import LoaderOption
@@ -14,44 +14,46 @@ from basic_memory.repository.repository import Repository
class ObservationRepository(Repository[Observation]):
"""Repository for Observation model with memory-specific operations."""
def __init__(self, session_maker: async_sessionmaker, project_id: int):
"""Initialize with session maker and project_id filter.
def __init__(self, project_id: int):
"""Initialize with project_id filter.
Args:
session_maker: SQLAlchemy session maker
project_id: Project ID to filter all operations by
"""
super().__init__(session_maker, Observation, project_id=project_id)
super().__init__(Observation, project_id=project_id)
def get_load_options(self) -> List[LoaderOption]:
"""Eager-load parent entity to prevent N+1 if obs.entity is accessed."""
return [selectinload(Observation.entity)]
async def find_by_entity(self, entity_id: int) -> Sequence[Observation]:
async def find_by_entity(self, session: AsyncSession, entity_id: int) -> Sequence[Observation]:
"""Find all observations for a specific entity."""
query = select(Observation).filter(Observation.entity_id == entity_id)
result = await self.execute_query(query)
query = self.select().filter(Observation.entity_id == entity_id)
result = await self.execute_query(session, query)
return result.scalars().all()
async def find_by_context(self, context: str) -> Sequence[Observation]:
async def find_by_context(self, session: AsyncSession, context: str) -> Sequence[Observation]:
"""Find observations with a specific context."""
query = select(Observation).filter(Observation.context == context)
result = await self.execute_query(query)
query = self.select().filter(Observation.context == context)
result = await self.execute_query(session, query)
return result.scalars().all()
async def find_by_category(self, category: str) -> Sequence[Observation]:
async def find_by_category(self, session: AsyncSession, category: str) -> Sequence[Observation]:
"""Find observations with a specific context."""
query = select(Observation).filter(Observation.category == category)
result = await self.execute_query(query)
query = self.select().filter(Observation.category == category)
result = await self.execute_query(session, query)
return result.scalars().all()
async def observation_categories(self) -> Sequence[str]:
async def observation_categories(self, session: AsyncSession) -> Sequence[str]:
"""Return a list of all observation categories."""
query = select(Observation.category).distinct()
result = await self.execute_query(query, use_query_options=False)
query = self._add_project_filter(query)
result = await self.execute_query(session, query, use_query_options=False)
return result.scalars().all()
async def find_by_entities(self, entity_ids: List[int]) -> Dict[int, List[Observation]]:
async def find_by_entities(
self, session: AsyncSession, entity_ids: List[int]
) -> Dict[int, List[Observation]]:
"""Find all observations for multiple entities in a single query.
Args:
@@ -64,8 +66,8 @@ class ObservationRepository(Repository[Observation]):
return {}
# Query observations for all entities in the list
query = select(Observation).filter(Observation.entity_id.in_(entity_ids))
result = await self.execute_query(query)
query = self.select().filter(Observation.entity_id.in_(entity_ids))
result = await self.execute_query(session, query)
observations = result.scalars().all()
# Group observations by entity_id
@@ -18,6 +18,7 @@ from basic_memory.repository.search_index_row import SearchIndexRow
from basic_memory.repository.search_repository_base import (
SearchRepositoryBase,
VectorChunkState,
relaxed_query_words,
)
from basic_memory.repository.metadata_filters import parse_metadata_filters
from basic_memory.repository.semantic_errors import SemanticDependenciesMissingError
@@ -176,6 +177,14 @@ class PostgresSearchRepository(SearchRepositoryBase):
# For non-Boolean queries, prepare single term
return self._prepare_single_term(term, is_prefix)
@staticmethod
def _relaxed_tsquery_text(search_text: Optional[str]) -> Optional[str]:
"""OR-relaxed tsquery expression for a failed strict query, or None."""
words = relaxed_query_words(search_text)
if not words:
return None
return " | ".join(f"{word}:*" for word in words)
def _prepare_boolean_query(self, query: str) -> str:
"""Convert Boolean query to tsquery format.
@@ -236,7 +245,12 @@ class PostgresSearchRepository(SearchRepositoryBase):
# Handle multi-word queries
if " " in cleaned_term:
words = [w for w in cleaned_term.split() if w.strip()]
# Strip sentence punctuation from word edges so question-form
# queries produce clean lexemes (parity with SQLite FTS5 prep).
# The tsquery tokenizer ignores this punctuation anyway; leaving it
# in only risks tsquery syntax errors. Interior characters are kept.
words = [w.strip("?!.,;") for w in cleaned_term.split()]
words = [w for w in words if w]
if not words:
# All characters were special chars, search won't match anything
# Return a safe search term that won't cause syntax errors
@@ -249,8 +263,11 @@ class PostgresSearchRepository(SearchRepositoryBase):
# Join with AND operator
return " & ".join(prepared_words)
# Single word
cleaned_term = cleaned_term.strip()
# Single word: strip edge punctuation; guard the now-empty case so a
# bare ":*"/"" never reaches tsquery.
cleaned_term = cleaned_term.strip().strip("?!.,;")
if not cleaned_term:
return "NOSPECIALCHARS:*"
if is_prefix:
return f"{cleaned_term}:*"
else:
@@ -908,6 +925,8 @@ class PostgresSearchRepository(SearchRepositoryBase):
min_similarity: Optional[float] = None,
limit: int = 10,
offset: int = 0,
allow_relaxed: bool = False,
session: AsyncSession | None = None,
) -> List[SearchIndexRow]:
"""Search across all indexed content using PostgreSQL tsvector."""
# --- Dispatch vector / hybrid modes (shared logic) ---
@@ -978,10 +997,32 @@ class PostgresSearchRepository(SearchRepositoryBase):
"""
logger.trace(f"Search {sql} params: {params}")
try:
async with db.scoped_session(self.session_maker) as session:
result = await session.execute(text(sql), params)
async def run_search(active_session: AsyncSession):
result = await active_session.execute(text(sql), params)
rows = result.fetchall()
# Trigger: multi-word natural-language query matched nothing
# under the default all-terms-AND tsquery semantics.
# Why: questions rarely have every word in one document;
# without relaxation the FTS half of hybrid search contributes
# zero candidates (parity with the SQLite path).
# Outcome: one retry with OR-joined prefix lexemes; ts_rank
# still ranks multi-term matches first.
relaxed = (
self._relaxed_tsquery_text(search_text) if allow_relaxed and not rows else None
)
if relaxed and params.get("text"):
params["text"] = relaxed
result = await active_session.execute(text(sql), params)
rows = result.fetchall()
return rows
try:
if session is not None:
rows = await run_search(session)
else:
async with db.scoped_session(self.session_maker) as owned_session:
rows = await run_search(owned_session)
except Exception as e:
if self._is_tsquery_syntax_error(e):
logger.warning(f"tsquery syntax error for search term: {search_text}, error: {e}")
@@ -5,6 +5,6 @@ from basic_memory.models.project import Project
class ProjectInfoRepository(Repository):
"""Repository for statistics queries."""
def __init__(self, session_maker):
def __init__(self):
# Initialize with Project model as a reference
super().__init__(session_maker, Project)
super().__init__(Project)
+101 -102
View File
@@ -7,9 +7,8 @@ from typing import Optional, Sequence, Union
from loguru import logger
from sqlalchemy import Executable, inspect as sa_inspect, select, text
from sqlalchemy.exc import NoResultFound, OperationalError
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from sqlalchemy.ext.asyncio import AsyncSession
from basic_memory import db
from basic_memory.models.project import Project
from basic_memory.repository.repository import Repository
@@ -79,20 +78,22 @@ class ProjectRepository(Repository[Project]):
Each entity, observation, and relation belongs to a specific project.
"""
def __init__(self, session_maker: async_sessionmaker[AsyncSession]):
"""Initialize with session maker."""
super().__init__(session_maker, Project)
def __init__(self):
"""Initialize the repository."""
super().__init__(Project)
async def get_by_name(self, name: str) -> Optional[Project]:
async def get_by_name(self, session: AsyncSession, name: str) -> Optional[Project]:
"""Get project by name (exact match).
Args:
name: Unique name of the project
"""
query = self.select().where(Project.name == name)
return await self.find_one(query)
return await self.find_one(session, query)
async def get_by_name_case_insensitive(self, name: str) -> Optional[Project]:
async def get_by_name_case_insensitive(
self, session: AsyncSession, name: str
) -> Optional[Project]:
"""Get project by name (case-insensitive match).
Args:
@@ -102,27 +103,27 @@ class ProjectRepository(Repository[Project]):
Project if found, None otherwise
"""
query = self.select().where(Project.name.ilike(name))
return await self.find_one(query)
return await self.find_one(session, query)
async def get_by_permalink(self, permalink: str) -> Optional[Project]:
async def get_by_permalink(self, session: AsyncSession, permalink: str) -> Optional[Project]:
"""Get project by permalink.
Args:
permalink: URL-friendly identifier for the project
"""
query = self.select().where(Project.permalink == permalink)
return await self.find_one(query)
return await self.find_one(session, query)
async def get_by_path(self, path: Union[Path, str]) -> Optional[Project]:
async def get_by_path(self, session: AsyncSession, path: Union[Path, str]) -> Optional[Project]:
"""Get project by filesystem path.
Args:
path: Path to the project directory (will be converted to string internally)
"""
query = self.select().where(Project.path == Path(path).as_posix())
return await self.find_one(query)
return await self.find_one(session, query)
async def get_by_id(self, project_id: int) -> Optional[Project]:
async def get_by_id(self, session: AsyncSession, project_id: int) -> Optional[Project]:
"""Get project by numeric ID.
Args:
@@ -131,10 +132,11 @@ class ProjectRepository(Repository[Project]):
Returns:
Project if found, None otherwise
"""
async with db.scoped_session(self.session_maker) as session:
return await self.select_by_id(session, project_id)
return await self.select_by_id(session, project_id)
async def get_by_external_id(self, external_id: str) -> Optional[Project]:
async def get_by_external_id(
self, session: AsyncSession, external_id: str
) -> Optional[Project]:
"""Get project by external UUID.
Args:
@@ -144,20 +146,20 @@ class ProjectRepository(Repository[Project]):
Project if found, None otherwise
"""
query = self.select().where(Project.external_id == external_id)
return await self.find_one(query)
return await self.find_one(session, query)
async def get_default_project(self) -> Optional[Project]:
async def get_default_project(self, session: AsyncSession) -> Optional[Project]:
"""Get the default project (the one marked as is_default=True)."""
query = self.select().where(Project.is_default.is_(True))
return await self.find_one(query)
return await self.find_one(session, query)
async def get_active_projects(self) -> Sequence[Project]:
async def get_active_projects(self, session: AsyncSession) -> Sequence[Project]:
"""Get all active projects."""
query = self.select().where(Project.is_active == True) # noqa: E712
result = await self.execute_query(query)
result = await self.execute_query(session, query)
return list(result.scalars().all())
async def set_as_default(self, project_id: int) -> Optional[Project]:
async def set_as_default(self, session: AsyncSession, project_id: int) -> Optional[Project]:
"""Set a project as the default and unset previous default.
Args:
@@ -166,22 +168,21 @@ class ProjectRepository(Repository[Project]):
Returns:
The updated project if found, None otherwise
"""
async with db.scoped_session(self.session_maker) as session:
# First, clear the default flag for all projects using direct SQL
await session.execute(
text("UPDATE project SET is_default = NULL WHERE is_default IS NOT NULL")
)
# First, clear the default flag for all projects using direct SQL
await session.execute(
text("UPDATE project SET is_default = NULL WHERE is_default IS NOT NULL")
)
await session.flush()
# Set the new default project
target_project = await self.select_by_id(session, project_id)
if target_project:
target_project.is_default = True
await session.flush()
return target_project
return None # pragma: no cover
# Set the new default project
target_project = await self.select_by_id(session, project_id)
if target_project:
target_project.is_default = True
await session.flush()
return target_project
return None # pragma: no cover
async def delete(self, entity_id: int) -> bool:
async def delete(self, session: AsyncSession, entity_id: int) -> bool:
"""Delete a project and its derived search rows in one transaction.
The cascade picture differs by backend:
@@ -204,62 +205,60 @@ class ProjectRepository(Repository[Project]):
Inspect the connection once and skip whichever is missing.
"""
logger.debug(f"Deleting Project and search rows for project_id: {entity_id}")
async with db.scoped_session(self.session_maker) as session:
try:
result = await session.execute(
select(self.Model).filter(self.primary_key == entity_id)
)
project = result.scalars().one()
except NoResultFound:
logger.debug(f"No Project found to delete: {entity_id}")
return False
try:
result = await session.execute(select(self.Model).filter(self.primary_key == entity_id))
project = result.scalars().one()
except NoResultFound:
logger.debug(f"No Project found to delete: {entity_id}")
return False
dialect_name = session.bind.dialect.name if session.bind else "sqlite"
is_sqlite = dialect_name == "sqlite"
dialect_name = session.bind.dialect.name if session.bind else "sqlite"
is_sqlite = dialect_name == "sqlite"
existing_tables = await session.run_sync(
lambda sync_session: set(sa_inspect(sync_session.connection()).get_table_names())
existing_tables = await session.run_sync(
lambda sync_session: set(sa_inspect(sync_session.connection()).get_table_names())
)
# search_index: SQLite has no FK on the FTS5 virtual table; Postgres
# cascades from the project FK, so the explicit DELETE is redundant.
if is_sqlite and "search_index" in existing_tables:
await session.execute(
text("DELETE FROM search_index WHERE project_id = :project_id"),
{"project_id": entity_id},
)
# search_index: SQLite has no FK on the FTS5 virtual table; Postgres
# cascades from the project FK, so the explicit DELETE is redundant.
if is_sqlite and "search_index" in existing_tables:
await session.execute(
text("DELETE FROM search_index WHERE project_id = :project_id"),
{"project_id": entity_id},
)
# search_vector_chunks: no FK to project on either backend, so both
# backends need this. SQLite must purge vec0 embeddings first
# (rowid pseudocolumn — Postgres uses chunk_id and would 500 here);
# Postgres' chunk_id FK CASCADE handles its embeddings cleanup when
# we delete the chunk rows below.
if "search_vector_chunks" in existing_tables:
if is_sqlite and "search_vector_embeddings" in existing_tables:
# Extension loading is per-connection. We must load vec0 on
# *this* session before the DELETE; otherwise a different
# pooled connection might have written embeddings that we'd
# silently leave behind.
if await _load_sqlite_vec_on_session(session):
await session.execute(
text(
"DELETE FROM search_vector_embeddings WHERE rowid IN ("
"SELECT id FROM search_vector_chunks "
"WHERE project_id = :project_id)"
),
{"project_id": entity_id},
)
await session.execute(
text("DELETE FROM search_vector_chunks WHERE project_id = :project_id"),
{"project_id": entity_id},
)
# search_vector_chunks: no FK to project on either backend, so both
# backends need this. SQLite must purge vec0 embeddings first
# (rowid pseudocolumn — Postgres uses chunk_id and would 500 here);
# Postgres' chunk_id FK CASCADE handles its embeddings cleanup when
# we delete the chunk rows below.
if "search_vector_chunks" in existing_tables:
if is_sqlite and "search_vector_embeddings" in existing_tables:
# Extension loading is per-connection. We must load vec0 on
# *this* session before the DELETE; otherwise a different
# pooled connection might have written embeddings that we'd
# silently leave behind.
if await _load_sqlite_vec_on_session(session):
await session.execute(
text(
"DELETE FROM search_vector_embeddings WHERE rowid IN ("
"SELECT id FROM search_vector_chunks "
"WHERE project_id = :project_id)"
),
{"project_id": entity_id},
)
await session.execute(
text("DELETE FROM search_vector_chunks WHERE project_id = :project_id"),
{"project_id": entity_id},
)
await session.delete(project)
logger.debug(f"Deleted Project and search rows for project_id: {entity_id}")
return True
await session.delete(project)
await session.flush()
logger.debug(f"Deleted Project and search rows for project_id: {entity_id}")
return True
async def scalar_vec_query(
self, query: Executable, params: Optional[dict] = None
self, session: AsyncSession, query: Executable, params: Optional[dict] = None
) -> Optional[int]:
"""Run a scalar COUNT query that reads the sqlite-vec vec0 table.
@@ -271,16 +270,17 @@ class ProjectRepository(Repository[Project]):
Returns None when sqlite-vec cannot be loaded on this Python build, so
callers can fall back to the genuinely-missing-dependency path.
"""
async with db.scoped_session(self.session_maker) as session:
# Trigger: query reads the vec0-backed search_vector_embeddings table.
# Why: vec0 modules are only visible on a connection that loaded sqlite-vec.
# Outcome: load the extension here, or signal absence so the caller degrades.
if not await _load_sqlite_vec_on_session(session):
return None
result = await session.execute(query, params)
return result.scalar()
# Trigger: query reads the vec0-backed search_vector_embeddings table.
# Why: vec0 modules are only visible on a connection that loaded sqlite-vec.
# Outcome: load the extension here, or signal absence so the caller degrades.
if not await _load_sqlite_vec_on_session(session):
return None
result = await session.execute(query, params)
return result.scalar()
async def update_path(self, project_id: int, new_path: str) -> Optional[Project]:
async def update_path(
self, session: AsyncSession, project_id: int, new_path: str
) -> Optional[Project]:
"""Update project path.
Args:
@@ -290,10 +290,9 @@ class ProjectRepository(Repository[Project]):
Returns:
The updated project if found, None otherwise
"""
async with db.scoped_session(self.session_maker) as session:
project = await self.select_by_id(session, project_id)
if project:
project.path = new_path
await session.flush()
return project
return None
project = await self.select_by_id(session, project_id)
if project:
project.path = new_path
await session.flush()
return project
return None
@@ -6,11 +6,10 @@ from sqlalchemy import and_, delete, select
from sqlalchemy.engine import CursorResult
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
from sqlalchemy.ext.asyncio import async_sessionmaker
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import selectinload, aliased
from sqlalchemy.orm.interfaces import LoaderOption
from basic_memory import db
from basic_memory.models import Relation, Entity
from basic_memory.repository.repository import Repository
@@ -18,17 +17,20 @@ from basic_memory.repository.repository import Repository
class RelationRepository(Repository[Relation]):
"""Repository for Relation model with memory-specific operations."""
def __init__(self, session_maker: async_sessionmaker, project_id: int):
"""Initialize with session maker and project_id filter.
def __init__(self, project_id: int):
"""Initialize with project_id filter.
Args:
session_maker: SQLAlchemy session maker
project_id: Project ID to filter all operations by
"""
super().__init__(session_maker, Relation, project_id=project_id)
super().__init__(Relation, project_id=project_id)
async def find_relation(
self, from_permalink: str, to_permalink: str, relation_type: str
self,
session: AsyncSession,
from_permalink: str,
to_permalink: str,
relation_type: str,
) -> Optional[Relation]:
"""Find a relation by its from and to path IDs."""
from_entity = aliased(Entity)
@@ -46,36 +48,44 @@ class RelationRepository(Repository[Relation]):
)
)
)
return await self.find_one(query)
query = self._add_project_filter(query)
return await self.find_one(session, query)
async def find_by_entities(self, from_id: int, to_id: int) -> Sequence[Relation]:
async def find_by_entities(
self, session: AsyncSession, from_id: int, to_id: int
) -> Sequence[Relation]:
"""Find all relations between two entities."""
query = select(Relation).where((Relation.from_id == from_id) & (Relation.to_id == to_id))
result = await self.execute_query(query)
query = self.select().where((Relation.from_id == from_id) & (Relation.to_id == to_id))
result = await self.execute_query(session, query)
return result.scalars().all()
async def find_by_type(self, relation_type: str) -> Sequence[Relation]:
async def find_by_type(self, session: AsyncSession, relation_type: str) -> Sequence[Relation]:
"""Find all relations of a specific type."""
query = select(Relation).filter(Relation.relation_type == relation_type)
result = await self.execute_query(query)
query = self.select().filter(Relation.relation_type == relation_type)
result = await self.execute_query(session, query)
return result.scalars().all()
async def delete_outgoing_relations_from_entity(self, entity_id: int) -> None:
async def delete_outgoing_relations_from_entity(
self, session: AsyncSession, entity_id: int
) -> None:
"""Delete outgoing relations for an entity.
Only deletes relations where this entity is the source (from_id),
as these are the ones owned by this entity's markdown file.
"""
async with db.scoped_session(self.session_maker) as session:
await session.execute(delete(Relation).where(Relation.from_id == entity_id))
query = delete(Relation).where(Relation.from_id == entity_id)
query = query.where(Relation.project_id == self.project_id)
await session.execute(query)
async def find_unresolved_relations(self) -> Sequence[Relation]:
async def find_unresolved_relations(self, session: AsyncSession) -> Sequence[Relation]:
"""Find all unresolved relations, where to_id is null."""
query = select(Relation).filter(Relation.to_id.is_(None))
result = await self.execute_query(query)
query = self.select().filter(Relation.to_id.is_(None))
result = await self.execute_query(session, query)
return result.scalars().all()
async def find_unresolved_relations_for_entity(self, entity_id: int) -> Sequence[Relation]:
async def find_unresolved_relations_for_entity(
self, session: AsyncSession, entity_id: int
) -> Sequence[Relation]:
"""Find unresolved relations for a specific entity.
Args:
@@ -84,11 +94,13 @@ class RelationRepository(Repository[Relation]):
Returns:
List of unresolved relations where this entity is the source.
"""
query = select(Relation).filter(Relation.from_id == entity_id, Relation.to_id.is_(None))
result = await self.execute_query(query)
query = self.select().filter(Relation.from_id == entity_id, Relation.to_id.is_(None))
result = await self.execute_query(session, query)
return result.scalars().all()
async def add_all_ignore_duplicates(self, relations: List[Relation]) -> int:
async def add_all_ignore_duplicates(
self, session: AsyncSession, relations: List[Relation]
) -> int:
"""Bulk insert relations, ignoring duplicates.
Uses ON CONFLICT DO NOTHING to skip relations that would violate the
@@ -120,27 +132,23 @@ class RelationRepository(Repository[Relation]):
for r in relations
]
async with db.scoped_session(self.session_maker) as session:
# Check dialect to use appropriate insert
dialect_name = session.bind.dialect.name if session.bind else "sqlite"
# Check dialect to use appropriate insert
dialect_name = session.bind.dialect.name if session.bind else "sqlite"
if dialect_name == "postgresql": # pragma: no cover
# PostgreSQL: use RETURNING to count inserted rows
# (rowcount is 0 for ON CONFLICT DO NOTHING)
stmt = ( # pragma: no cover
pg_insert(Relation)
.values(values)
.on_conflict_do_nothing()
.returning(Relation.id)
)
result = await session.execute(stmt) # pragma: no cover
return len(result.fetchall()) # pragma: no cover
else:
# SQLite: rowcount works correctly
stmt = sqlite_insert(Relation).values(values)
stmt = stmt.on_conflict_do_nothing()
result = cast(CursorResult[Any], await session.execute(stmt))
return result.rowcount if result.rowcount > 0 else 0
if dialect_name == "postgresql": # pragma: no cover
# PostgreSQL: use RETURNING to count inserted rows
# (rowcount is 0 for ON CONFLICT DO NOTHING)
stmt = ( # pragma: no cover
pg_insert(Relation).values(values).on_conflict_do_nothing().returning(Relation.id)
)
result = await session.execute(stmt) # pragma: no cover
return len(result.fetchall()) # pragma: no cover
else:
# SQLite: rowcount works correctly
stmt = sqlite_insert(Relation).values(values)
stmt = stmt.on_conflict_do_nothing()
result = cast(CursorResult[Any], await session.execute(stmt))
return result.rowcount if result.rowcount > 0 else 0
def get_load_options(self) -> List[LoaderOption]:
return [selectinload(Relation.from_entity), selectinload(Relation.to_entity)]
+181 -201
View File
@@ -1,7 +1,6 @@
"""Base repository implementation."""
from typing import Type, Optional, Any, Sequence, TypeVar, List, Dict, cast
from typing import Optional, Any, Sequence, TypeVar, List, Dict, cast
from loguru import logger
from sqlalchemy import (
@@ -16,27 +15,23 @@ from sqlalchemy import (
update as sqlalchemy_update,
)
from sqlalchemy.engine import CursorResult
from sqlalchemy.exc import NoResultFound
from sqlalchemy.ext.asyncio import async_sessionmaker, AsyncSession
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm.interfaces import LoaderOption
from sqlalchemy.sql.elements import ColumnElement
from basic_memory import db
from basic_memory.models import Base
T = TypeVar("T", bound=Base)
class Repository[T: Base]:
"""Base repository implementation with generic CRUD operations."""
"""Base repository implementation with explicit caller-owned sessions."""
def __init__(
self,
session_maker: async_sessionmaker[AsyncSession],
Model: Type[T],
Model: type[T],
project_id: Optional[int] = None,
):
self.session_maker = session_maker
self.project_id = project_id
if Model:
self.Model = Model
@@ -98,62 +93,61 @@ class Repository[T: Base]:
result = await session.execute(query)
return result.scalars().all()
async def add(self, model: T) -> T:
async def add(self, session: AsyncSession, model: T) -> T:
"""
Add a model to the repository. This will also add related objects
:param session: the caller-owned session to use
:param model: the model to add
:return: the added model instance
"""
async with db.scoped_session(self.session_maker) as session:
# Set project_id if applicable and not already set
self._set_project_id_if_needed(model)
# Set project_id if applicable and not already set
self._set_project_id_if_needed(model)
session.add(model)
await session.flush()
session.add(model)
await session.flush()
# Query within same session
found = await self.select_by_id(session, model.id) # pyright: ignore [reportAttributeAccessIssue]
if found is None: # pragma: no cover
logger.error(
"Failed to retrieve model after add",
model_type=self.Model.__name__,
model_id=model.id, # pyright: ignore
)
raise ValueError(
f"Can't find {self.Model.__name__} with ID {model.id} after session.add" # pyright: ignore
)
return found
# Query within same session
found = await self.select_by_id(session, model.id) # pyright: ignore [reportAttributeAccessIssue]
if found is None: # pragma: no cover
logger.error(
"Failed to retrieve model after add",
model_type=self.Model.__name__,
model_id=model.id, # pyright: ignore
)
raise ValueError(
f"Can't find {self.Model.__name__} with ID {model.id} after session.add" # pyright: ignore
)
return found
async def add_all(self, models: List[T]) -> Sequence[T]:
async def add_all(self, session: AsyncSession, models: List[T]) -> Sequence[T]:
"""
Add a list of models to the repository. This will also add related objects
:param session: the caller-owned session to use
:param models: the models to add
:return: the added models instances
"""
async with db.scoped_session(self.session_maker) as session:
# set the project id if not present in models
for model in models:
self._set_project_id_if_needed(model)
# set the project id if not present in models
for model in models:
self._set_project_id_if_needed(model)
session.add_all(models)
await session.flush()
session.add_all(models)
await session.flush()
# Query within same session
return await self.select_by_ids(session, [m.id for m in models]) # pyright: ignore [reportAttributeAccessIssue]
# Query within same session
return await self.select_by_ids(session, [m.id for m in models]) # pyright: ignore [reportAttributeAccessIssue]
async def add_all_no_return(self, models: List[T]) -> int:
async def add_all_no_return(self, session: AsyncSession, models: List[T]) -> int:
"""Insert models without reloading them afterward."""
if not models:
return 0
async with db.scoped_session(self.session_maker) as session:
for model in models:
self._set_project_id_if_needed(model)
for model in models:
self._set_project_id_if_needed(model)
session.add_all(models)
await session.flush()
logger.debug(f"Added {len(models)} {self.Model.__name__} records")
return len(models)
session.add_all(models)
await session.flush()
logger.debug(f"Added {len(models)} {self.Model.__name__} records")
return len(models)
def select(self, *entities: Any) -> Select:
"""Create a new SELECT statement.
@@ -170,55 +164,55 @@ class Repository[T: Base]:
return self._add_project_filter(query)
async def find_all(
self, skip: int = 0, limit: Optional[int] = None, use_load_options: bool = True
self,
session: AsyncSession,
skip: int = 0,
limit: Optional[int] = None,
use_load_options: bool = True,
) -> Sequence[T]:
"""Fetch records from the database with pagination.
Args:
session: The caller-owned session to use
skip: Number of records to skip
limit: Maximum number of records to return
use_load_options: Whether to apply eager loading options (default: True)
"""
logger.debug(f"Finding all {self.Model.__name__} (skip={skip}, limit={limit})")
async with db.scoped_session(self.session_maker) as session:
query = select(self.Model).offset(skip)
query = select(self.Model).offset(skip)
# Only apply load options if requested
if use_load_options:
query = query.options(*self.get_load_options())
# Only apply load options if requested
if use_load_options:
query = query.options(*self.get_load_options())
# Add project filter if applicable
query = self._add_project_filter(query)
# Add project filter if applicable
query = self._add_project_filter(query)
if limit:
query = query.limit(limit)
if limit:
query = query.limit(limit)
result = await session.execute(query)
result = await session.execute(query)
items = result.scalars().all()
logger.debug(f"Found {len(items)} {self.Model.__name__} records")
return items
items = result.scalars().all()
logger.debug(f"Found {len(items)} {self.Model.__name__} records")
return items
async def find_by_id(self, entity_id: int) -> Optional[T]:
async def find_by_id(self, session: AsyncSession, entity_id: Any) -> Optional[T]:
"""Fetch an entity by its unique identifier."""
logger.debug(f"Finding {self.Model.__name__} by ID: {entity_id}")
return await self.select_by_id(session, entity_id)
async with db.scoped_session(self.session_maker) as session:
return await self.select_by_id(session, entity_id)
async def find_by_ids(self, ids: List[int]) -> Sequence[T]:
async def find_by_ids(self, session: AsyncSession, ids: List[Any]) -> Sequence[T]:
"""Fetch multiple entities by their identifiers in a single query."""
logger.debug(f"Finding {self.Model.__name__} by IDs: {ids}")
return await self.select_by_ids(session, ids)
async with db.scoped_session(self.session_maker) as session:
return await self.select_by_ids(session, ids)
async def find_one(self, query: Select[tuple[T]]) -> Optional[T]:
async def find_one(self, session: AsyncSession, query: Select[tuple[T]]) -> Optional[T]:
"""Execute a query and retrieve a single record."""
# add in load options
query = query.options(*self.get_load_options())
result = await self.execute_query(query)
result = await self.execute_query(session, query)
entity = result.scalars().one_or_none()
if entity:
@@ -227,12 +221,42 @@ class Repository[T: Base]:
logger.trace(f"No {self.Model.__name__} found")
return entity
async def create(self, data: dict) -> T:
async def create(self, session: AsyncSession, data: dict[str, Any]) -> T:
"""Create a new record from a model instance."""
logger.debug(f"Creating {self.Model.__name__} from entity_data: {data}")
async with db.scoped_session(self.session_maker) as session:
# Only include valid columns that are provided in entity_data
model_data = self.get_model_data(data)
# Only include valid columns that are provided in entity_data
model_data = self.get_model_data(data)
# Add project_id if applicable and not already provided
if self.has_project_id and self.project_id is not None and "project_id" not in model_data:
model_data["project_id"] = self.project_id
model = self.Model(**model_data)
session.add(model)
await session.flush()
return_instance = await self.select_by_id(session, model.id) # pyright: ignore [reportAttributeAccessIssue]
if return_instance is None: # pragma: no cover
logger.error(
"Failed to retrieve model after create",
model_type=self.Model.__name__,
model_id=model.id, # pyright: ignore
)
raise ValueError(
f"Can't find {self.Model.__name__} with ID {model.id} after session.add" # pyright: ignore
)
return return_instance
async def create_all(
self, session: AsyncSession, data_list: List[dict[str, Any]]
) -> Sequence[T]:
"""Create multiple records in a single transaction."""
logger.debug(f"Bulk creating {len(data_list)} {self.Model.__name__} instances")
# Only include valid columns that are provided in entity_data
model_list = []
for d in data_list:
model_data = self.get_model_data(d)
# Add project_id if applicable and not already provided
if (
@@ -240,169 +264,126 @@ class Repository[T: Base]:
and self.project_id is not None
and "project_id" not in model_data
):
model_data["project_id"] = self.project_id
model_data["project_id"] = self.project_id # pragma: no cover
model = self.Model(**model_data)
session.add(model)
await session.flush()
model_list.append(self.Model(**model_data))
return_instance = await self.select_by_id(session, model.id) # pyright: ignore [reportAttributeAccessIssue]
if return_instance is None: # pragma: no cover
logger.error(
"Failed to retrieve model after create",
model_type=self.Model.__name__,
model_id=model.id, # pyright: ignore
)
raise ValueError(
f"Can't find {self.Model.__name__} with ID {model.id} after session.add" # pyright: ignore
)
return return_instance
session.add_all(model_list)
await session.flush()
async def create_all(self, data_list: List[dict]) -> Sequence[T]:
"""Create multiple records in a single transaction."""
logger.debug(f"Bulk creating {len(data_list)} {self.Model.__name__} instances")
return await self.select_by_ids(session, [model.id for model in model_list]) # pyright: ignore [reportAttributeAccessIssue]
async with db.scoped_session(self.session_maker) as session:
# Only include valid columns that are provided in entity_data
model_list = []
for d in data_list:
model_data = self.get_model_data(d)
# Add project_id if applicable and not already provided
if (
self.has_project_id
and self.project_id is not None
and "project_id" not in model_data
):
model_data["project_id"] = self.project_id # pragma: no cover
model_list.append(self.Model(**model_data))
session.add_all(model_list)
await session.flush()
return await self.select_by_ids(session, [model.id for model in model_list]) # pyright: ignore [reportAttributeAccessIssue]
async def update(self, entity_id: int, entity_data: dict[str, Any] | T) -> Optional[T]:
async def update(
self,
session: AsyncSession,
entity_id: Any,
entity_data: dict[str, Any] | T,
) -> Optional[T]:
"""Update an entity with the given data."""
logger.debug(f"Updating {self.Model.__name__} {entity_id} with data: {entity_data}")
async with db.scoped_session(self.session_maker) as session:
try:
result = await session.execute(
select(self.Model).filter(self.primary_key == entity_id)
)
entity = result.scalars().one()
entity = await self.select_by_id(session, entity_id)
if entity is None:
logger.debug(f"No {self.Model.__name__} found to update: {entity_id}")
return None
if isinstance(entity_data, dict):
update_data = cast(dict[str, Any], entity_data)
for key in self.valid_columns:
if key in update_data:
setattr(entity, key, update_data[key])
if isinstance(entity_data, dict):
update_data = cast(dict[str, Any], entity_data)
for key in self.valid_columns:
if key in update_data:
setattr(entity, key, update_data[key])
elif isinstance(entity_data, self.Model):
for column in self.valid_columns:
setattr(entity, column, getattr(entity_data, column))
elif isinstance(entity_data, self.Model):
for column in self.valid_columns:
setattr(entity, column, getattr(entity_data, column))
await session.flush() # Make sure changes are flushed
await session.refresh(entity) # Refresh
await session.flush()
await session.refresh(entity)
logger.debug(f"Updated {self.Model.__name__}: {entity_id}")
return await self.select_by_id(session, entity.id) # pyright: ignore [reportAttributeAccessIssue]
logger.debug(f"Updated {self.Model.__name__}: {entity_id}")
return await self.select_by_id(session, entity.id) # pyright: ignore [reportAttributeAccessIssue]
except NoResultFound:
logger.debug(f"No {self.Model.__name__} found to update: {entity_id}")
return None
async def update_fields(self, entity_id: Any, entity_data: dict[str, Any]) -> bool:
async def update_fields(
self, session: AsyncSession, entity_id: Any, entity_data: dict[str, Any]
) -> bool:
"""Update columns without reloading the model graph afterward."""
update_data = {k: v for k, v in entity_data.items() if k in self.valid_columns}
if not update_data:
return True
async with db.scoped_session(self.session_maker) as session:
conditions = [self.primary_key == entity_id]
if self.has_project_id and self.project_id is not None:
conditions.append(getattr(self.Model, "project_id") == self.project_id)
conditions = [self.primary_key == entity_id]
if self.has_project_id and self.project_id is not None:
conditions.append(getattr(self.Model, "project_id") == self.project_id)
result = cast(
CursorResult[Any],
await session.execute(
sqlalchemy_update(self.Model).where(and_(*conditions)).values(**update_data)
),
)
return result.rowcount > 0
result = cast(
CursorResult[Any],
await session.execute(
sqlalchemy_update(self.Model).where(and_(*conditions)).values(**update_data)
),
)
return result.rowcount > 0
async def delete(self, entity_id: int) -> bool:
async def delete(self, session: AsyncSession, entity_id: int) -> bool:
"""Delete an entity from the database."""
logger.debug(f"Deleting {self.Model.__name__}: {entity_id}")
async with db.scoped_session(self.session_maker) as session:
try:
result = await session.execute(
select(self.Model).filter(self.primary_key == entity_id)
)
entity = result.scalars().one()
await session.delete(entity)
entity = await self.select_by_id(session, entity_id)
if entity is None:
logger.debug(f"No {self.Model.__name__} found to delete: {entity_id}")
return False
logger.debug(f"Deleted {self.Model.__name__}: {entity_id}")
return True
except NoResultFound:
logger.debug(f"No {self.Model.__name__} found to delete: {entity_id}")
return False
await session.delete(entity)
await session.flush()
async def delete_by_ids(self, ids: List[int]) -> int:
logger.debug(f"Deleted {self.Model.__name__}: {entity_id}")
return True
async def delete_by_ids(self, session: AsyncSession, ids: List[int]) -> int:
"""Delete records matching given IDs."""
logger.debug(f"Deleting {self.Model.__name__} by ids: {ids}")
async with db.scoped_session(self.session_maker) as session:
conditions = [self.primary_key.in_(ids)]
conditions = [self.primary_key.in_(ids)]
# Add project_id filter if applicable
if self.has_project_id and self.project_id is not None: # pragma: no cover
conditions.append(getattr(self.Model, "project_id") == self.project_id)
# Add project_id filter if applicable
if self.has_project_id and self.project_id is not None: # pragma: no cover
conditions.append(getattr(self.Model, "project_id") == self.project_id)
query = delete(self.Model).where(and_(*conditions))
result = cast(CursorResult[Any], await session.execute(query))
logger.debug(f"Deleted {result.rowcount} records")
return result.rowcount
query = delete(self.Model).where(and_(*conditions))
result = cast(CursorResult[Any], await session.execute(query))
logger.debug(f"Deleted {result.rowcount} records")
return result.rowcount
async def delete_by_fields(self, **filters: Any) -> bool:
async def delete_by_fields(self, session: AsyncSession, **filters: Any) -> bool:
"""Delete records matching given field values."""
logger.debug(f"Deleting {self.Model.__name__} by fields: {filters}")
async with db.scoped_session(self.session_maker) as session:
conditions = [getattr(self.Model, field) == value for field, value in filters.items()]
conditions = [getattr(self.Model, field) == value for field, value in filters.items()]
# Add project_id filter if applicable
if self.has_project_id and self.project_id is not None:
conditions.append(getattr(self.Model, "project_id") == self.project_id)
# Add project_id filter if applicable
if self.has_project_id and self.project_id is not None:
conditions.append(getattr(self.Model, "project_id") == self.project_id)
query = delete(self.Model).where(and_(*conditions))
result = cast(CursorResult[Any], await session.execute(query))
deleted = result.rowcount > 0
logger.debug(f"Deleted {result.rowcount} records")
return deleted
query = delete(self.Model).where(and_(*conditions))
result = cast(CursorResult[Any], await session.execute(query))
deleted = result.rowcount > 0
logger.debug(f"Deleted {result.rowcount} records")
return deleted
async def count(self, query: Executable | None = None) -> int:
async def count(self, session: AsyncSession, query: Executable | None = None) -> int:
"""Count entities in the database table."""
async with db.scoped_session(self.session_maker) as session:
if query is None:
query = select(func.count()).select_from(self.Model)
# Add project filter if applicable
if (
isinstance(query, Select)
and self.has_project_id
and self.project_id is not None
):
query = query.where(
getattr(self.Model, "project_id") == self.project_id
) # pragma: no cover
if query is None:
query = select(func.count()).select_from(self.Model)
# Add project filter if applicable
if isinstance(query, Select) and self.has_project_id and self.project_id is not None:
query = query.where(
getattr(self.Model, "project_id") == self.project_id
) # pragma: no cover
result = await session.execute(query)
scalar = result.scalar()
count = scalar if scalar is not None else 0
logger.debug(f"Counted {count} {self.Model.__name__} records")
return count
result = await session.execute(query)
scalar = result.scalar()
count = scalar if scalar is not None else 0
logger.debug(f"Counted {count} {self.Model.__name__} records")
return count
async def execute_query(
self,
session: AsyncSession,
query: Executable,
params: Optional[Dict[str, Any]] = None,
use_query_options: bool = True,
@@ -411,9 +392,8 @@ class Repository[T: Base]:
query = query.options(*self.get_load_options()) if use_query_options else query
logger.trace(f"Executing query: {query}, params: {params}")
async with db.scoped_session(self.session_maker) as session:
result = await session.execute(query, params)
return result
result = await session.execute(query, params)
return result
def get_load_options(self) -> List[LoaderOption]:
"""Get list of loader options for eager loading relationships.
@@ -36,6 +36,7 @@ class SearchIndexRow:
category: Optional[str] = None # observations
from_id: Optional[int] = None # relations
to_id: Optional[int] = None # relations
to_name: Optional[str] = None # relations: literal target text, set even when unresolved
relation_type: Optional[str] = None # relations
# Matched chunk text from vector search (the actual content that matched the query)
@@ -94,6 +95,7 @@ class SearchIndexRow:
else self.metadata,
"from_id": self.from_id,
"to_id": self.to_id,
"to_name": self.to_name,
"relation_type": self.relation_type,
"entity_id": self.entity_id,
"category": self.category,
@@ -48,6 +48,8 @@ class SearchRepository(Protocol):
min_similarity: Optional[float] = None,
limit: int = 10,
offset: int = 0,
allow_relaxed: bool = False,
session: AsyncSession | None = None,
) -> List[SearchIndexRow]:
"""Search across indexed content."""
...
@@ -40,6 +40,55 @@ BULLET_PATTERN = re.compile(r"^[\-\*]\s+")
OVERSIZED_ENTITY_VECTOR_SHARD_SIZE = 256
_SQLITE_MAX_PREPARE_WINDOW = 8
# Interrogative/function words contribute lexical noise when a strict
# full-text query is relaxed: "when OR did OR a" matches loud wrong documents
# that displace genuine results from the ranking window.
RELAXATION_STOPWORDS = frozenset(
"a an and are as at be but by did do does for from had has have how i in is it of on "
"or that the their they this to was we were what when where which who whom whose why "
"will with you your".split()
)
def relaxed_query_words(search_text: Optional[str]) -> Optional[list[str]]:
"""Content-bearing words for OR-relaxing a strict full-text query.
Returns None when relaxation must not apply. These eligibility rules match
SearchService._is_relaxed_fts_fallback_eligible so the hybrid FTS branch
relaxes exactly the same query shapes as the service-level FTS path:
- empty / quoted / explicit-boolean queries (user intent is not
second-guessed);
- fewer than three alphanumeric tokens (short queries like "New Feature"
over-broaden under OR and in hybrid the relaxed FTS-only rows normalize
to 1.0 and can outrank the vector result the user wanted);
- any pure-digit token ("root note 1", "SPEC 16") identifier-like queries
over-broaden and create false positives under OR.
"""
if not search_text:
return None
stripped = search_text.strip()
if '"' in stripped or any(op in f" {stripped} " for op in (" AND ", " OR ", " NOT ")):
return None
# Eligibility checks run on raw alphanumeric tokens (parity with the
# service), before stopword filtering.
tokens = re.findall(r"[A-Za-z0-9]+", stripped.lower())
if len(tokens) < 3 or any(token.isdigit() for token in tokens):
return None
words = [word.strip("?!.,;:") for word in stripped.split()]
words = [
word
for word in words
if word and word.isalnum() and word.lower() not in RELAXATION_STOPWORDS
]
return words or None
# Entity, observation, and relation rows in search_index carry ids from independent
# auto-increment sequences, so a bare id is ambiguous across row types. Every map in
# the vector/hybrid retrieval path must key rows by (type, id) to avoid collisions.
type SearchIndexKey = tuple[str, int]
@dataclass
class VectorSyncBatchResult:
@@ -224,6 +273,7 @@ class SearchRepositoryBase(ABC):
min_similarity: Optional[float] = None,
limit: int = 10,
offset: int = 0,
allow_relaxed: bool = False,
) -> List[SearchIndexRow]:
"""Search across all indexed content.
@@ -1857,7 +1907,7 @@ class SearchRepositoryBase(ABC):
# ------------------------------------------------------------------
@staticmethod
def _parse_chunk_key(chunk_key: str) -> tuple[str, int]:
def _parse_chunk_key(chunk_key: str) -> SearchIndexKey:
"""Parse a chunk_key like 'observation:5:0' into (type, search_index_id)."""
parts = chunk_key.split(":")
return parts[0], int(parts[1])
@@ -1932,26 +1982,27 @@ class SearchRepositoryBase(ABC):
hydrate_start = time.perf_counter()
# Build per-search_index_row similarity scores from chunk-level results.
# Each chunk_key encodes the search_index row type and id.
# Each chunk_key encodes the search_index row type and id; keep both as the
# key because different row types can share the same numeric id (#982).
# Track the best similarity per row (for ranking) and all chunks (for context).
similarity_by_si_id: dict[int, float] = {}
chunks_by_si_id: dict[int, list[tuple[float, str]]] = {}
similarity_by_si_key: dict[SearchIndexKey, float] = {}
chunks_by_si_key: dict[SearchIndexKey, list[tuple[float, str]]] = {}
for row in vector_rows:
chunk_key = row.get("chunk_key", "")
distance = float(row["best_distance"])
similarity = self._distance_to_similarity(distance)
chunk_text = row.get("chunk_text", "")
try:
_, si_id = self._parse_chunk_key(chunk_key)
si_key = self._parse_chunk_key(chunk_key)
except (ValueError, IndexError):
# Fallback: group by entity_id for chunks without parseable keys
continue
current = similarity_by_si_id.get(si_id)
current = similarity_by_si_key.get(si_key)
if current is None or similarity > current:
similarity_by_si_id[si_id] = similarity
chunks_by_si_id.setdefault(si_id, []).append((similarity, chunk_text))
similarity_by_si_key[si_key] = similarity
chunks_by_si_key.setdefault(si_key, []).append((similarity, chunk_text))
if not similarity_by_si_id:
if not similarity_by_si_key:
hydrate_ms = (time.perf_counter() - hydrate_start) * 1000
_log_vector_summary()
return []
@@ -1962,16 +2013,17 @@ class SearchRepositoryBase(ABC):
min_similarity if min_similarity is not None else self._semantic_min_similarity
)
if effective_min_similarity > 0.0:
similarity_by_si_id = {
k: v for k, v in similarity_by_si_id.items() if v >= effective_min_similarity
similarity_by_si_key = {
k: v for k, v in similarity_by_si_key.items() if v >= effective_min_similarity
}
if not similarity_by_si_id:
if not similarity_by_si_key:
hydrate_ms = (time.perf_counter() - hydrate_start) * 1000
_log_vector_summary()
return []
# Fetch the actual search_index rows
si_ids = list(similarity_by_si_id.keys())
# Fetch the actual search_index rows. Colliding (type, id) keys share one
# bare id, so deduplicate while preserving first-seen order.
si_ids = list(dict.fromkeys(si_id for _, si_id in similarity_by_si_key))
search_index_rows = await self._fetch_search_index_rows_by_ids(si_ids)
# Apply optional filters if requested
@@ -2003,16 +2055,14 @@ class SearchRepositoryBase(ABC):
limit=VECTOR_FILTER_SCAN_LIMIT,
offset=0,
)
# Use (id, type) tuples to avoid collisions between different
# Use (type, id) tuples to avoid collisions between different
# search_index row types that share the same auto-increment id.
allowed_keys = {(row.id, row.type) for row in filtered_rows if row.id is not None}
search_index_rows = {
k: v for k, v in search_index_rows.items() if (v.id, v.type) in allowed_keys
}
allowed_keys = {(row.type, row.id) for row in filtered_rows if row.id is not None}
search_index_rows = {k: v for k, v in search_index_rows.items() if k in allowed_keys}
ranked_rows: list[SearchIndexRow] = []
for si_id, similarity in similarity_by_si_id.items():
row = search_index_rows.get(si_id)
for si_key, similarity in similarity_by_si_key.items():
row = search_index_rows.get(si_key)
if row is None:
continue
@@ -2022,7 +2072,7 @@ class SearchRepositoryBase(ABC):
if content_snippet and len(content_snippet) <= SMALL_NOTE_CONTENT_LIMIT:
matched_chunk_text = content_snippet
else:
si_chunks = chunks_by_si_id.get(si_id, [])
si_chunks = chunks_by_si_key.get(si_key, [])
si_chunks.sort(key=lambda c: c[0], reverse=True)
top_texts = [text for _, text in si_chunks[:TOP_CHUNKS_PER_RESULT]]
matched_chunk_text = "\n---\n".join(top_texts) if top_texts else None
@@ -2088,8 +2138,12 @@ class SearchRepositoryBase(ABC):
async def _fetch_search_index_rows_by_ids(
self, row_ids: list[int]
) -> dict[int, SearchIndexRow]:
"""Fetch search_index rows by their primary key (id), any type."""
) -> dict[SearchIndexKey, SearchIndexRow]:
"""Fetch search_index rows by id, keyed by (type, id) to disambiguate types.
A bare id can match one row per type (independent id sequences), so the
result must carry every matching row rather than letting one clobber another.
"""
if not row_ids:
return {}
placeholders = ",".join(f":id_{idx}" for idx in range(len(row_ids)))
@@ -2106,11 +2160,11 @@ class SearchRepositoryBase(ABC):
WHERE project_id = :project_id
AND id IN ({placeholders})
"""
result: dict[int, SearchIndexRow] = {}
result: dict[SearchIndexKey, SearchIndexRow] = {}
async with db.scoped_session(self.session_maker) as session:
row_result = await session.execute(text(sql), params)
for row in row_result.fetchall():
result[row.id] = SearchIndexRow(
result[(row.type, row.id)] = SearchIndexRow(
project_id=self.project_id,
id=row.id,
title=row.title,
@@ -2156,7 +2210,7 @@ class SearchRepositoryBase(ABC):
) -> List[SearchIndexRow]:
"""Fuse FTS and vector results using score-based fusion.
Uses search_index row id as the fusion key. The formula
Uses the search_index (type, id) pair as the fusion key. The formula
``max(vec, fts) + FUSION_BONUS * min(vec, fts)`` preserves
the dominant signal and rewards dual-source agreement.
"""
@@ -2165,6 +2219,9 @@ class SearchRepositoryBase(ABC):
query_start = time.perf_counter()
candidate_limit = max(self._semantic_vector_k, (limit + offset) * 10)
fts_start = time.perf_counter()
# allow_relaxed: question-form queries rarely AND-match, and a dead FTS
# branch silently degrades hybrid to vector-only ranking. Fusion plus
# bm25 keep relaxed lexical candidates from dominating precision.
fts_results = await self.search(
search_text=search_text,
permalink=permalink,
@@ -2178,6 +2235,7 @@ class SearchRepositoryBase(ABC):
retrieval_mode=SearchRetrievalMode.FTS,
limit=candidate_limit,
offset=0,
allow_relaxed=True,
)
fts_ms = (time.perf_counter() - fts_start) * 1000
vector_start = time.perf_counter()
@@ -2199,17 +2257,19 @@ class SearchRepositoryBase(ABC):
vector_ms = (time.perf_counter() - vector_start) * 1000
fusion_start = time.perf_counter()
# --- Score-based fusion keyed on search_index row id ---
# --- Score-based fusion keyed on (type, id) ---
# A bare row id collides across row types (independent id sequences), so
# fusion must key on (type, id) or distinct rows would merge (#982).
# FTS scores are normalized to [0, 1] (BM25 is unbounded).
# Vector scores are used raw — already calibrated [0, 1] by _distance_to_similarity().
rows_by_id: dict[int, SearchIndexRow] = {}
rows_by_key: dict[SearchIndexKey, SearchIndexRow] = {}
# Normalize FTS scores to [0, 1] — handles both SQLite (negative bm25)
# and Postgres (positive ts_rank) by using absolute values
fts_abs = [abs(row.score or 0.0) for row in fts_results]
fts_max = max(fts_abs) if fts_abs else 1.0
fts_scores: dict[int, float] = {}
fts_scores: dict[SearchIndexKey, float] = {}
for row in fts_results:
if row.id is None:
continue
@@ -2217,32 +2277,32 @@ class SearchRepositoryBase(ABC):
# Gate: FTS scores below threshold contribute zero
if norm < FTS_GATE_THRESHOLD:
norm = 0.0
fts_scores[row.id] = norm
rows_by_id[row.id] = row
fts_scores[(row.type, row.id)] = norm
rows_by_key[(row.type, row.id)] = row
vec_scores: dict[int, float] = {}
vec_scores: dict[SearchIndexKey, float] = {}
for row in vector_results:
if row.id is None:
continue
# Trigger: no re-normalization by vec_max
# Why: vector similarity is already calibrated [0, 1]; re-normalizing
# inflates weak matches when the entire result set is mediocre
vec_scores[row.id] = row.score or 0.0
rows_by_id[row.id] = row
vec_scores[(row.type, row.id)] = row.score or 0.0
rows_by_key[(row.type, row.id)] = row
# Fuse: max(v, f) + FUSION_BONUS * min(v, f)
# Preserves the dominant signal; bonus rewards dual-source agreement.
# Output range: [0, 1.3] for dual-source, [0, 1.0] for single-source.
fused_scores: dict[int, float] = {}
for row_id in fts_scores.keys() | vec_scores.keys():
v = vec_scores.get(row_id, 0.0)
f = fts_scores.get(row_id, 0.0)
fused_scores[row_id] = max(v, f) + FUSION_BONUS * min(v, f)
fused_scores: dict[SearchIndexKey, float] = {}
for row_key in fts_scores.keys() | vec_scores.keys():
v = vec_scores.get(row_key, 0.0)
f = fts_scores.get(row_key, 0.0)
fused_scores[row_key] = max(v, f) + FUSION_BONUS * min(v, f)
ranked = sorted(fused_scores.items(), key=lambda item: item[1], reverse=True)
output: list[SearchIndexRow] = []
for row_id, fused_score in ranked[offset : offset + limit]:
row = rows_by_id[row_id]
for row_key, fused_score in ranked[offset : offset + limit]:
row = rows_by_key[row_key]
# Trigger: FTS-only results have no matched_chunk_text from vector search.
# Why: without chunk text, API falls back to truncated content, losing answer text.
# Outcome: FTS-only results get full content_snippet as matched_chunk.
@@ -23,7 +23,10 @@ from basic_memory.models.search import (
from basic_memory.repository.embedding_provider import EmbeddingProvider
from basic_memory.repository.embedding_provider_factory import create_embedding_provider
from basic_memory.repository.search_index_row import SearchIndexRow
from basic_memory.repository.search_repository_base import SearchRepositoryBase
from basic_memory.repository.search_repository_base import (
SearchRepositoryBase,
relaxed_query_words,
)
from basic_memory.repository.metadata_filters import parse_metadata_filters, build_sqlite_json_path
from basic_memory.repository.semantic_errors import SemanticDependenciesMissingError
from basic_memory.schemas.search import SearchItemType, SearchRetrievalMode
@@ -255,6 +258,19 @@ class SQLiteSearchRepository(SearchRepositoryBase):
if "*" in term and all(c.isalnum() or c in "*_-" for c in term):
return term
# Natural-language queries arrive with sentence punctuation that FTS5
# treats as syntax ("When did Melanie paint a sunrise?"). The tokenizer
# ignores this punctuation in the INDEX, so stripping it from word
# edges loses nothing — but leaving it forces the whole question into
# an exact-phrase match that returns zero rows, silently disabling the
# FTS half of hybrid search. Interior characters (hyphens, slashes —
# permalinks and paths) are untouched.
if " " in term:
words = [word.strip("?!.,;:") for word in term.split()]
term = " ".join(word for word in words if word)
if not term:
return ""
# Characters that can cause FTS5 syntax errors when used as operators
# We're more conservative here - only quote when we detect problematic patterns
problematic_chars = [
@@ -351,6 +367,14 @@ class SQLiteSearchRepository(SearchRepositoryBase):
# For non-Boolean queries, use the single term preparation logic
return self._prepare_single_term(term, is_prefix)
@staticmethod
def _relaxed_fts_text(search_text: Optional[str]) -> Optional[str]:
"""OR-relaxed FTS5 expression for a failed strict query, or None."""
words = relaxed_query_words(search_text)
if not words:
return None
return " OR ".join(f"{word}*" for word in words)
# ------------------------------------------------------------------
# sqlite-vec extension loading (SQLite-specific)
# ------------------------------------------------------------------
@@ -953,8 +977,16 @@ class SQLiteSearchRepository(SearchRepositoryBase):
min_similarity: Optional[float] = None,
limit: int = 10,
offset: int = 0,
allow_relaxed: bool = False,
session: AsyncSession | None = None,
) -> List[SearchIndexRow]:
"""Search across all indexed content using SQLite FTS5."""
"""Search across all indexed content using SQLite FTS5.
``allow_relaxed=True`` retries a zero-result strict multi-word query
with OR-joined content terms. Only the hybrid path opts in: its FTS
branch otherwise contributes nothing for question-form queries.
Service-level FTS searches keep their own conservative fallback.
"""
# --- Dispatch vector / hybrid modes (shared logic) ---
dispatched = await self._dispatch_retrieval_mode(
search_text=search_text,
@@ -1017,10 +1049,31 @@ class SQLiteSearchRepository(SearchRepositoryBase):
"""
logger.trace(f"Search {sql} params: {params}")
try:
async with db.scoped_session(self.session_maker) as session:
result = await session.execute(text(sql), params)
async def run_search(active_session: AsyncSession):
result = await active_session.execute(text(sql), params)
rows = result.fetchall()
# Trigger: multi-word natural-language query matched nothing
# under the default all-terms-AND semantics.
# Why: questions ("when did X do Y") rarely have every word in
# one document; without relaxation the FTS half of hybrid
# search contributes zero candidates and ranking degrades to
# vector-only.
# Outcome: one retry with OR-joined prefix terms; bm25 still
# ranks multi-term matches first.
relaxed = self._relaxed_fts_text(search_text) if allow_relaxed and not rows else None
if relaxed and params.get("text"):
params["text"] = relaxed
result = await active_session.execute(text(sql), params)
rows = result.fetchall()
return rows
try:
if session is not None:
rows = await run_search(session)
else:
async with db.scoped_session(self.session_maker) as owned_session:
rows = await run_search(owned_session)
except Exception as e:
# Handle FTS5 syntax errors and provide user-friendly feedback
if self._is_fts5_syntax_error(e): # pragma: no cover
+4 -1
View File
@@ -19,7 +19,6 @@ from pathlib import Path
from typing import List, Optional, Annotated, Dict
from annotated_types import MinLen, MaxLen
from dateparser import parse
from pydantic import BaseModel, BeforeValidator, Field, model_validator, computed_field
@@ -92,6 +91,10 @@ def parse_timeframe(timeframe: str) -> datetime:
parse_timeframe('1d') -> 2025-06-04 14:50:00-07:00 (24 hours ago with local timezone)
parse_timeframe('1 week ago') -> 2025-05-29 14:50:00-07:00 (1 week ago with local timezone)
"""
# Deferred: dateparser costs ~0.13s to import; schemas load on every CLI
# start, but timeframe parsing only happens per request (#886).
from dateparser import parse
if timeframe.lower() == "today":
# For "today", return 1 day ago to ensure we capture recent activity across timezones
# This handles the case where client and server are in different timezones
+3
View File
@@ -155,6 +155,9 @@ class RelationSummary(BaseModel):
from_entity_id: Optional[int] = None
from_entity_external_id: Optional[str] = None
to_entity: Optional[str] = None
# Literal target text from the markdown; present even when the relation is
# an unresolved forward reference (to_entity is None until the target exists)
to_name: Optional[str] = None
to_entity_id: Optional[int] = None
to_entity_external_id: Optional[str] = None
created_at: Annotated[
+63 -9
View File
@@ -2,15 +2,18 @@
from __future__ import annotations
from contextlib import asynccontextmanager
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Any, List, Optional, Tuple, TYPE_CHECKING
from typing import Any, AsyncIterator, List, Optional, Tuple, TYPE_CHECKING
from loguru import logger
from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
import logfire
from basic_memory import db
from basic_memory.repository.entity_repository import EntityRepository
from basic_memory.repository.observation_repository import ObservationRepository
from basic_memory.repository.postgres_search_repository import PostgresSearchRepository
@@ -18,6 +21,7 @@ from basic_memory.repository.search_repository import SearchRepository, SearchIn
from basic_memory.schemas.memory import MemoryUrl, memory_url_path
from basic_memory.schemas.search import SearchItemType
from basic_memory.utils import generate_permalink
from basic_memory.workspace_context import workspace_slug_for_canonical_permalinks
if TYPE_CHECKING:
from basic_memory.services.link_resolver import LinkResolver
@@ -36,6 +40,7 @@ class ContextResultRow:
from_id: Optional[int] = None
to_id: Optional[int] = None
relation_type: Optional[str] = None
to_name: Optional[str] = None
content: Optional[str] = None
category: Optional[str] = None
entity_id: Optional[int] = None
@@ -89,11 +94,21 @@ class ContextService:
entity_repository: EntityRepository,
observation_repository: ObservationRepository,
link_resolver: Optional[LinkResolver] = None,
session_maker: async_sessionmaker[AsyncSession] | None = None,
):
self.search_repository = search_repository
self.entity_repository = entity_repository
self.observation_repository = observation_repository
self.link_resolver = link_resolver
self.session_maker = session_maker
@asynccontextmanager
async def _session_scope(self) -> AsyncIterator[AsyncSession]:
"""Open a service-owned transaction for core repository reads."""
if self.session_maker is None: # pragma: no cover
raise ValueError("session_maker is required for ContextService")
async with db.scoped_session(self.session_maker) as session:
yield session
async def build_context(
self,
@@ -143,6 +158,29 @@ class ContextService:
primary = await self.search_repository.search(
permalink_match=normalized_path, limit=fetch_limit, offset=offset
)
# Trigger: a workspace-qualified pattern matched nothing while a
# workspace permalink context is active.
# Why: rows written before workspace canonicalization (or via
# clients that didn't forward workspace headers) store
# project-qualified permalinks; a workspace-prefixed pattern
# can never match those legacy rows (#957).
# Outcome: retry once with the workspace prefix stripped so the
# pattern matches the index form the rows actually carry.
if not primary:
workspace_slug = workspace_slug_for_canonical_permalinks()
ws_prefix = f"{workspace_slug}/" if workspace_slug else None
if ws_prefix and normalized_path.startswith(ws_prefix):
fallback_path = normalized_path.removeprefix(ws_prefix)
logger.debug(
f"Pattern search fallback without workspace prefix: "
f"'{fallback_path}'"
)
primary = await self.search_repository.search(
permalink_match=fallback_path,
limit=fetch_limit,
offset=offset,
)
else:
normalized_path = generate_permalink(path, split_extension=False)
logger.debug(f"Direct lookup for '{normalized_path}'")
@@ -151,9 +189,13 @@ class ContextService:
)
if not primary and self.link_resolver:
entity = await self.link_resolver.resolve_link(
path, use_search=True, strict=False
)
async with self._session_scope() as session:
entity = await self.link_resolver.resolve_link(
path,
use_search=True,
strict=False,
session=session,
)
if entity:
logger.debug(
f"LinkResolver resolved '{path}' to permalink '{entity.permalink}'"
@@ -209,9 +251,10 @@ class ContextService:
phase="load_observations",
result_count=len(entity_ids),
):
observations_by_entity = await self.observation_repository.find_by_entities(
entity_ids
)
async with self._session_scope() as session:
observations_by_entity = await self.observation_repository.find_by_entities(
session, entity_ids
)
logger.debug(f"Found observations for {len(observations_by_entity)} entities")
metadata = ContextMetadata(
@@ -392,6 +435,7 @@ class ContextService:
from_id=row.from_id,
to_id=row.to_id,
relation_type=row.relation_type,
to_name=row.to_name,
content=row.content,
category=row.category,
entity_id=row.entity_id,
@@ -426,6 +470,7 @@ class ContextService:
CAST(NULL AS INTEGER) as from_id,
CAST(NULL AS INTEGER) as to_id,
CAST(NULL AS TEXT) as relation_type,
CAST(NULL AS TEXT) as to_name,
CAST(NULL AS TEXT) as content,
CAST(NULL AS TEXT) as category,
CAST(NULL AS INTEGER) as entity_id,
@@ -478,6 +523,10 @@ class ContextService:
WHEN step_type = 1 THEN r.relation_type
ELSE NULL
END as relation_type,
CASE
WHEN step_type = 1 THEN r.to_name
ELSE NULL
END as to_name,
CAST(NULL AS TEXT) as content,
CAST(NULL AS TEXT) as category,
CAST(NULL AS INTEGER) as entity_id,
@@ -541,6 +590,7 @@ class ContextService:
from_id,
to_id,
relation_type,
to_name,
content,
category,
entity_id,
@@ -550,7 +600,7 @@ class ContextService:
FROM entity_graph
WHERE depth > 0
GROUP BY type, id, title, permalink, file_path, from_id, to_id,
relation_type, content, category, entity_id, root_id, created_at
relation_type, to_name, content, category, entity_id, root_id, created_at
ORDER BY depth, type, id
LIMIT :max_results
""")
@@ -579,6 +629,7 @@ class ContextService:
NULL as from_id,
NULL as to_id,
NULL as relation_type,
NULL as to_name,
NULL as content,
NULL as category,
NULL as entity_id,
@@ -606,6 +657,7 @@ class ContextService:
r.from_id,
r.to_id,
r.relation_type,
r.to_name,
NULL as content,
NULL as category,
NULL as entity_id,
@@ -644,6 +696,7 @@ class ContextService:
NULL as from_id,
NULL as to_id,
NULL as relation_type,
NULL as to_name,
NULL as content,
NULL as category,
NULL as entity_id,
@@ -677,6 +730,7 @@ class ContextService:
from_id,
to_id,
relation_type,
to_name,
content,
category,
entity_id,
@@ -686,7 +740,7 @@ class ContextService:
FROM entity_graph
WHERE depth > 0
GROUP BY type, id, title, permalink, file_path, from_id, to_id,
relation_type, content, category, entity_id, root_id, created_at
relation_type, to_name, content, category, entity_id, root_id, created_at
ORDER BY depth, type, id
LIMIT :max_results
""")

Some files were not shown because too many files have changed in this diff Show More