Compare commits

...

153 Commits

Author SHA1 Message Date
phernandez 7ebf16a95d chore: update version to 0.17.8 for v0.17.8 release 2026-01-24 11:43:44 -06:00
phernandez c05075f8d4 docs: add CHANGELOG entry for v0.17.8
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-24 11:43:21 -06:00
phernandez 4cef9281ca docs: add CHANGELOG entry for v0.17.7
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-24 11:39:41 -06:00
Paul Hernandez 6888effef2 fix: correct get_default_project() query to check for True instead of not NULL (#521)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 11:37:05 -06:00
Paul Hernandez 38616c345d fix: read default project from database in cloud mode (#520)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 10:44:04 -06:00
phernandez f3c1aa895c fix links in README.md to remove smithery badge
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-22 12:58:03 -06:00
phernandez d978aba09b fix links in README.md to remove glama.ai
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-22 12:56:33 -06:00
phernandez 2aaee734c9 fix links in README.md to point to basicmemory.com instead of basicmachines.co
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-22 12:53:18 -06:00
Paul Hernandez 369ad37b3d feat: add SPEC-29 Phase 3 bucket snapshot CLI commands (#476)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-20 17:55:01 -06:00
Drew Cain 4e5f701d22 Fix server.json runtimeArguments format
- Use proper Argument object format instead of plain strings
- Add .mcpregistry tokens to .gitignore

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-01-19 12:23:29 -06:00
Drew Cain 9d9ea4d61c chore: update version to 0.17.7 for v0.17.7 release 2026-01-19 12:12:35 -06:00
Drew Cain 7a502e6474 feat: Add MCP registry publication files (#515)
Signed-off-by: Drew Cain <groksrc@gmail.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-19 11:57:50 -06:00
Paul Hernandez c7835a9d5c fix: ensure external_id is set on entity creation (#512) (#513)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-19 11:14:48 -06:00
Paul Hernandez 85835ae533 chore: Remove OpenPanel telemetry (#514)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-18 22:24:27 -06:00
phernandez 671e3d4db9 chore: update version to 0.17.6 for v0.17.6 release 2026-01-17 15:11:34 -06:00
phernandez e11aeff8d9 docs: add changelog entry for v0.17.6
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-17 15:11:17 -06:00
phernandez 803f3efe53 turn sync logging to debug
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-17 15:10:04 -06:00
phernandez d6dab8552c Merge branch 'main' of github.com:basicmachines-co/basic-memory 2026-01-17 15:09:04 -06:00
phernandez d1d433df15 remove logfire config, and specs docs, turn lifespan logging to debug
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-17 15:08:54 -06:00
Drew Cain 1799c94953 fix: Docker container Python symlink broken at runtime (#510)
Signed-off-by: Drew Cain <groksrc@gmail.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 11:59:35 -06:00
Drew Cain 07996181b3 chore: add doc update reminder to release commands
Remind developers to update docs.basicmemory.com and basicmachines.co
after running just release or just beta.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 10:21:23 -06:00
phernandez a1c37c1dba chore: update version to 0.17.5 for v0.17.5 release 2026-01-11 16:53:55 -06:00
phernandez aff53cca93 docs: add changelog entry for v0.17.5
- Python 3.14 compatibility fix for CLI commands hanging on exit
- Skip nest_asyncio on Python 3.14+
- Update pyright to 1.1.408 for Python 3.14 support
- Fix SQLAlchemy rowcount typing

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-11 16:53:38 -06:00
Paul Hernandez 863e0a4e24 fix: prevent CLI commands from hanging on exit (Python 3.14) (#505)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 16:47:35 -06:00
phernandez eeeade4f07 chore: update version to 0.17.4 for v0.17.4 release 2026-01-05 21:38:03 -06:00
phernandez 03793eaf7c docs: add v0.17.4 changelog entry
- Critical bug fix for search index preservation (#503)
- Major architecture refactor with composition roots (#502)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-05 21:36:14 -06:00
Paul Hernandez 26f7e98932 fix: preserve search index across server restarts (#503)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2026-01-05 21:31:35 -06:00
Paul Hernandez 5947f04bd3 refactor: composition roots, deps split, and typed API clients (#490 roadmap) (#502)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 11:05:58 -06:00
Drew Cain ba1439fefc chore: update version to 0.17.3 for v0.17.3 release 2026-01-03 13:08:11 -06:00
Drew Cain ef411ceb12 docs: add v0.17.3 changelog entry
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 13:07:24 -06:00
Drew Cain c6baf58aa7 fix: update mcp to support protocol version 2025-11-25 (#501)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 12:39:11 -06:00
phernandez 3c1748cc89 return external_id with directory response in tree
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-02 23:44:03 -06:00
phernandez 9206e7960a return external_id with directory response
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-02 23:43:14 -06:00
Paul Hernandez 53c4c20d22 fix: route ordering for cloud (#499)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-02 21:52:02 -06:00
Paul Hernandez b4486d20bd test: remove stdlib mocks, strengthen integration coverage (#489)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-02 14:22:01 -06:00
jope-bm a4000f64ce feat: add stable external_id (UUID) to Project and Entity models (#485)
Signed-off-by: Joe P <joe@basicmemory.com>
Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: phernandez <paul@basicmachines.co>
2026-01-02 13:00:35 -06:00
phernandez 88a1778798 fix(importers): return ImportResult from handle_error
Implement handle_error() in all importers so errors return a concrete ImportResult (success=false, error_message set) instead of None, preventing NoneType.success crashes.

Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-01 11:47:42 -06:00
phernandez 4ce21984a4 fix: use upsert to prevent IntegrityError during parallel search indexing
Replace delete-then-insert pattern with INSERT ... ON CONFLICT for
PostgreSQL search index operations. This fixes race conditions where
parallel entity indexing could cause UniqueViolationError on the
uix_search_index_permalink_project constraint.

Changes:
- Add index_item() override in PostgresSearchRepository with upsert
- Update bulk_index_items() to use ON CONFLICT (permalink, project_id)
- Add CREATE_POSTGRES_SEARCH_INDEX_PERMALINK DDL for test fixtures
- Add tests for upsert behavior on duplicate permalinks

Technical note: Use column-based ON CONFLICT syntax instead of
ON CONSTRAINT (which only works for table constraints, not indexes).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-30 21:55:43 -06:00
phernandez eb7fbaf0bf fix set_default_project in cloud mode
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-30 20:57:45 -06:00
phernandez 8adf1f4ed4 fix: use relative file paths in importers for cloud storage compatibility
Importers now use relative file paths (based on permalink) instead of
absolute paths. This enables proper S3 key generation in cloud environments.

Changes:
- write_entity() accepts str | Path for file_path parameter
- ensure_folder_exists() uses relative paths directly
- All importers pass relative paths to FileService
- FileService handles base_path resolution internally

This prevents S3 keys from including container filesystem paths like
`/app/basic-memory/imports/...` and instead uses clean relative paths
like `imports/20241010-file.md`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-30 20:29:01 -06:00
phernandez 45ce1813e4 refactor: use FileService in importers for cloud compatibility
Refactor importers to use FileService for all file operations instead of
direct filesystem calls. This enables cloud environments to override file
operations via dependency injection (e.g., S3FileService).

Changes:
- Add `to_markdown_string()` method to MarkdownProcessor for content
  serialization without file I/O
- Update Importer base class to accept FileService and use it for:
  - `write_entity()` - now uses FileService.write_file()
  - `ensure_folder_exists()` - now async, uses FileService.ensure_directory()
- Fix direct `mkdir()` calls in:
  - claude_projects_importer.py
  - memory_json_importer.py
- Update deps.py to inject FileService into all importers (v1 and v2)
- Update CLI commands to create and pass FileService to importers
- Update tests to work with new FileService dependency

This follows the pattern used by /knowledge API and SyncService, enabling
cloud to override file operations by providing S3FileService via DI.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-30 19:21:32 -06:00
phernandez 2744c4b6a5 add info logging to index_entity_data background task
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-30 12:04:49 -06:00
Paul Hernandez fd732aa6fe fix: set_default_project skips config file update in cloud mode (#486)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-30 09:05:48 -06:00
Paul Hernandez 537e58ad7d fix: make RelationResponse.from_id optional to handle null permalinks (#484)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-30 08:12:29 -06:00
phernandez 48e6e84beb chore: update version to 0.17.2 for v0.17.2 release 2025-12-29 16:47:04 -06:00
phernandez 02c14acddb docs: add CHANGELOG entry for v0.17.2
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-29 16:46:26 -06:00
phernandez 0b5425f163 don't run full tests in release, just lint, typecheck
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-29 16:33:38 -06:00
phernandez 0bcda4a14a fix: allow recent_activity discovery mode in cloud mode
Add `allow_discovery` parameter to `resolve_project_parameter()` that
enables tools like `recent_activity` to work across all projects in
cloud mode without requiring a project parameter.

- Add `allow_discovery: bool = False` param to resolve_project_parameter
- In cloud mode with allow_discovery=True, return None instead of error
- Update recent_activity to use allow_discovery=True
- Fix circular import by deferring call_get import inside functions
- Add comprehensive tests for resolve_project_parameter

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-29 16:22:48 -06:00
phernandez 7a49f57dee chore: update version to 0.17.1 for v0.17.1 release
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-29 10:34:06 -06:00
phernandez 58db2817d2 docs: add CHANGELOG entry for v0.17.1
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-29 10:20:00 -06:00
Paul Hernandez 98fbd60527 fix: only set BASIC_MEMORY_ENV=test during pytest runs (#482)
Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-12-29 10:16:45 -06:00
phernandez 6281a81256 chore: update version to 0.17.0 for v0.17.0 release
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-28 16:06:03 -06:00
phernandez be1d0b169f style: format telemetry.py
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-28 16:05:07 -06:00
phernandez 148bf6f75a docs: add CHANGELOG entry for v0.17.0
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-28 16:03:13 -06:00
phernandez 272a983709 add foss as telementry source, disable analytics for tests
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-28 13:41:40 -06:00
phernandez ef7adb7b99 fix: add cloud_mode check to initialize_app()
MCP server crashes in cloud mode with:
ValueError: DATABASE_URL must be set when using Postgres backend

Root cause: initialize_app() did not check cloud_mode_enabled before
trying to initialize the database. Only ensure_initialization() had
the check. In cloud mode, tenant DBs are per-request via headers,
not via DATABASE_URL environment variable.

Also includes minor formatting fix in telemetry.py.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-28 10:31:17 -06:00
phernandez 3cd9178415 refactor: centralize test environment detection in config.is_test_env
Add is_test_env property to BasicMemoryConfig that checks:
- config.env == "test"
- BASIC_MEMORY_ENV env var is "test"
- PYTEST_CURRENT_TEST is set

Replace duplicated test detection logic in:
- api/app.py
- mcp/server.py
- services/initialization.py
- telemetry.py (disables telemetry during tests)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-27 12:44:42 -06:00
Paul Hernandez 856737fe3c feat: add anonymous usage telemetry (Homebrew-style opt-out) (#478)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-26 22:09:48 -06:00
Paul Hernandez 1fd680c3f1 feat: add auto-format files on save with built-in Python formatter (#474)
Signed-off-by: Cedric Hurst <cedric@spantree.net>
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Cedric Hurst <cedric@spantree.net>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Sebastian B Otaegui <feniix@users.noreply.github.com>
Co-authored-by: Cedric Hurst <cedric@divideby0.io>
2025-12-24 15:39:22 -06:00
phernandez 38919d11cb docs: update CLAUDE.md with accurate CLI commands and code guidelines
- Add Code Change Guidelines section (full file read, minimize diffs, fail fast, no guessing)
- Add Literate Programming Style section (section headers, decision point comments)
- Fix CLI commands: `tools` -> `tool`, add project management commands
- Fix cloud commands to match current CLI (status, setup)
- Remove non-existent MCP tools (get_current_project, sync_status)
- Add ChatGPT-compatible tools (search, fetch)
- Remove non-existent json_canvas_spec prompt
- Add /importers to codebase architecture
- Remove unused python-developer and system-architect agents

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-24 15:16:02 -06:00
phernandez 85684f848f fix: handle UTF-8 BOM in frontmatter parsing
Fixes #452 - Imported conversations not fully indexed

Files with UTF-8 BOM (Byte Order Mark) at the start would fail frontmatter
detection, causing:
- Title to fall back to filename instead of frontmatter value
- Permalink to be null in the database

Added strip_bom() helper function and updated all frontmatter-related
functions to strip BOM before processing:
- has_frontmatter()
- parse_frontmatter()
- remove_frontmatter()
- EntityParser.parse_markdown_content()

Added comprehensive tests for BOM handling with various scenarios.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-24 14:28:50 -06:00
Paul Hernandez 14ce5a3bd0 fix: handle null titles in ChatGPT import (#475)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-12-24 13:50:30 -06:00
phernandez 45d6caf723 fix: remove MaxLen constraint from observation content
The API Pydantic schema had a MaxLen(1000) constraint on observation
content while the database uses SQLAlchemy's Text type (unlimited).
This mismatch caused validation errors when observations exceeded
1000 characters (e.g., JSON schemas with 1458+ chars).

Removed the MaxLen constraint to match the DB schema. Retained:
- BeforeValidator(str.strip) for whitespace cleaning
- MinLen(1) to ensure non-empty content

Added comprehensive tests to verify:
- Long content (10K+ chars) is accepted
- Very long content (50K+ chars) is accepted
- Empty/whitespace-only content is still rejected
- Whitespace stripping still works

Fixes #385

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-24 13:41:28 -06:00
phernandez 1652f862dd fix: handle FileNotFoundError gracefully during sync
When a file exists in the database but is missing from the filesystem,
the sync worker now treats this as a deletion instead of crashing.

The sync_file() method catches FileNotFoundError specifically and calls
handle_delete() to clean up the orphaned database record. This prevents
the sync from failing on database/filesystem inconsistencies that can
occur due to race conditions, manual file deletions, or cloud storage
caching issues.

Includes a test to verify the graceful handling behavior.

Fixes #386

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-24 13:34:59 -06:00
phernandez c23927d124 fix: use canonical project names in API response messages
Use database-retrieved project names (new_project.name, old_project.name)
instead of input parameters (project_data.name, name) in v1 API response
messages to ensure consistent project name casing.

The v2 API already did this correctly. This fixes issue #450 where project
names would display with different casing between add and remove operations.

Fixes #450

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-24 13:28:06 -06:00
jope-bm 1a74d85973 feat: Complete Phase 2 of API v2 migration - Update MCP tools to use v2 endpoints (#447)
Signed-off-by: Joe P <joe@basicmemory.com>
Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: phernandez <paul@basicmachines.co>
2025-12-24 12:59:14 -06:00
phernandez d71c6e8568 fix: suppress CLI warnings for cleaner output
Suppress DeprecationWarning from aiosqlite and LogfireNotConfiguredWarning
that were cluttering CLI output.

The key fix is applying warnings.filterwarnings("ignore") AFTER all imports
in main.py, because authlib (imported via cloud commands) adds a
DeprecationWarning filter that overrides earlier suppressions.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-24 12:21:14 -06:00
phernandez 63b98491be fix: prevent DEBUG logs from appearing on CLI stdout
Remove loguru's default handler at the very start of cli/app.py,
before any other imports. This prevents module-level code (like
TemplateLoader.__init__) from logging to stdout during import.

The import chain cli/commands/project.py -> mcp/async_client.py ->
api/app.py -> api/routers/prompt_router.py -> api/template_loader.py
triggers TemplateLoader() instantiation at DEBUG level before
init_cli_logging() can remove the default handler.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-24 11:42:25 -06:00
Paul Hernandez 622d37e4a8 fix: detect rclone version for --create-empty-src-dirs support (#473)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-24 11:27:49 -06:00
Paul Hernandez 916baf8971 fix: prevent CLI commands from hanging on exit (#471)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-24 11:27:11 -06:00
Drew Cain 95937c6d0a fix: make test-int-postgres compatible with macOS
Use gtimeout (Homebrew) or timeout (Linux), falling back to running
without timeout if neither is available. This fixes 'command not found'
errors on macOS which doesn't have GNU timeout by default.
2025-12-20 10:27:26 -06:00
Drew Cain 24dc9a2931 chore: update version to 0.16.3 for v0.16.3 release 2025-12-20 09:53:37 -06:00
Drew Cain 85c63e5a7a docs: add CHANGELOG entry for v0.16.3 2025-12-20 09:26:00 -06:00
Drew Cain f227ef6a86 fix: Pin FastMCP to 2.12.3 to fix MCP tools visibility (#464)
Signed-off-by: Drew Cain <groksrc@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2025-12-20 08:54:44 -06:00
Paul Hernandez 897b1edaa4 fix: Reduce watch service CPU usage by increasing reload interval (#458)
Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-12-17 11:37:01 -06:00
Paul Hernandez 0c12a39a98 test: Add integration test for issue #416 (read_note with underscored folders) (#453)
Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-12-17 09:56:45 -06:00
Paul Hernandez efbc758325 fix: await background sync task cancellation in lifespan shutdown (#456)
Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-12-17 09:53:34 -06:00
Paul Hernandez a0f20eb102 chore: more Tenantless fixes (#457)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-16 18:34:05 -06:00
Paul Hernandez 78673d8e51 chore: Cloud compatibility fixes and performance improvements (#454)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-15 20:07:55 -06:00
phernandez 126c0495c0 Merge branch 'main' of github.com:basicmachines-co/basic-memory 2025-12-13 15:24:43 -06:00
Paul Hernandez 4a43d7df4a remove logfire instrumentation
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-13 15:22:14 -06:00
Paul Hernandez c462faf046 Replace py-pglite with testcontainers for Postgres testing (#449)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-10 22:17:56 -06:00
Cedric Hurst 70bb10be1d fix: respect --project flag in background sync (fixes #434) (#436)
Signed-off-by: Cedric Hurst <cedric@spantree.net>
2025-12-08 12:58:18 -06:00
phernandez fbf9045d78 use asyncpg for just db-migrate
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-05 15:01:35 -06:00
phernandez 1094210c52 fix broken sqlite migration
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-02 20:23:55 -06:00
phernandez 391feb639f add delete cascade to entity to delete search_index (postgres only)
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-02 10:04:21 -06:00
phernandez a920a9ff29 feat: Add project_id to Relation and Observation for efficient project-scoped queries
Denormalizes project_id onto Relation and Observation tables to enable
efficient project-scoped queries without joins. Migration backfills
from associated entity and adds pg_trgm extension with GIN indexes
for fuzzy link resolution on PostgreSQL.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-01 21:54:56 -06:00
phernandez 05efe8701c test: Verify update() returns entity with eager-loaded relations
Add test confirming entity_repository.update() returns the entity with
observations and relations eagerly loaded, eliminating the need for a
separate find_by_id() call after update.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-01 16:22:21 -06:00
phernandez 0eaf30bb06 remove conflict constraint name from relation_repository.py
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-30 19:31:09 -06:00
phernandez 0818bda565 feat: Add bulk insert with ON CONFLICT handling for relations
Add add_all_ignore_duplicates() method to RelationRepository for bulk
inserting relations with ON CONFLICT DO NOTHING. This handles cases
where the same [[wiki link]] appears multiple times in a document,
silently ignoring duplicates based on the (from_id, to_name, relation_type)
unique constraint.

Works with both SQLite and PostgreSQL dialects.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-30 14:59:52 -06:00
phernandez 6f99d2e551 perf: lightweight permalink resolution to avoid eager loading
Add optimized repository methods for resolve_permalink() that skip
eager loading of observations and relations:

- permalink_exists(): Check existence without loading entity
- get_file_path_for_permalink(): Get only file_path column
- get_permalink_for_file_path(): Get only permalink column
- get_all_permalinks(): Get all permalinks as strings
- get_permalink_to_file_path_map(): Bulk lookup mapping
- get_file_path_to_permalink_map(): Reverse mapping

Updated entity_service.resolve_permalink() to use these lightweight
methods instead of loading full entities with all relationships.

Also added logfire instrumentation to markdown utils.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-30 14:34:31 -06:00
phernandez 73d940e064 fix: observation parsing and permalink limits (#446)
1. Hashtag detection now checks for standalone words starting with #
   instead of just checking if # appears anywhere in content.
   This prevents HTML color codes like #4285F4 from being
   interpreted as hashtags.

2. Observation permalinks now truncate content to 200 chars
   to stay under PostgreSQL's btree index limit of 2704 bytes.

Added tests for both fixes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-30 00:12:04 -06:00
phernandez c3678a11d2 truncate content_stems to fix Postgres 8KB index row limit
Large documents (like ~1MB conversation imports) exceed Postgres's 8KB
index row limit, causing ProgramLimitExceededError. Truncate content_stems
to 6000 characters (with headroom for other columns) before indexing.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-29 20:23:09 -06:00
phernandez 203d684c24 fix integrity error handling when setting forward relation refs
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-29 19:03:12 -06:00
phernandez a872220924 disable pooling for postgres db
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-29 13:24:47 -06:00
phernandez 7d763a66ff use entity.mtime for updated at in api
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-29 13:21:58 -06:00
phernandez b5d4fb559c fix: postgres/neon connection settings and search index dedupe
- Reduce db_pool_recycle from 3600s to 180s for Neon scale-to-zero
- Add connect_args for Neon serverless (statement cache, timeouts, app name)
- Dedupe observation permalinks in search indexing to avoid unique constraint violations
- Add tests for duplicate observation permalink handling

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-29 11:24:08 -06:00
phernandez 830775276d remove record_return=True from logfire spans
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 18:36:09 -06:00
phernandez ed894fc3ed get db pool sizes from config
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 16:50:57 -06:00
phernandez 704338edcf remove logfire.instrument_fastapi(app) from app.py
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 13:52:18 -06:00
phernandez 0ca02a7ebe add logfire instrumentation to services and repository code
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 12:47:29 -06:00
jope-bm 28cc5225a7 feat: Implement API v2 with ID-based endpoints (Phase 1) (#441)
Signed-off-by: Joe P <joe@basicmemory.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: phernandez <paul@basicmachines.co>
2025-11-27 10:35:55 -06:00
phernandez 9b7bbc7116 formatting and logic change to resolve_relations, remove fuzzy search 2025-11-25 22:54:37 -06:00
phernandez 138c283d6c add postgres db type 2025-11-25 20:25:56 -06:00
phernandez 7a8954c37e add extra logic for cloud-indexing improvements 2025-11-25 13:52:58 -06:00
phernandez 10c7c19c03 fix db url for sqlite migrations
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-21 13:21:20 -06:00
Paul Hernandez fb5e9e1d77 feat: Add PostgreSQL database backend support (#439)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-11-20 11:20:29 -06:00
phernandez 66b91b2847 ci: Add PostgreSQL testing to GitHub Actions workflow
Add Postgres service container and separate test step for PostgreSQL backend testing.
The Postgres tests only run on Linux runners since GitHub Actions service containers
are only available on Linux.

- Add postgres:17 service container with health checks
- Add 'Run tests (Postgres)' step with Linux-only condition
- Rename existing test step to 'Run tests (SQLite)' for clarity

This enables CI testing of dual database backend support introduced in the
postgres-support feature branch.

Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-18 12:25:18 -06:00
Cedric Hurst b004565df9 fix: handle periods in kebab_filenames mode (#424) 2025-11-18 06:49:55 -05:00
Drew Cain a258b73e1d chore: update version to 0.16.2 for v0.16.2 release 2025-11-16 21:30:59 -06:00
Drew Cain 9a845f2906 docs: prepare for v0.16.2 release 2025-11-16 21:28:13 -06:00
Drew Cain 6517e9845f fix: Use platform-native path separators in config.json (#429)
Signed-off-by: Drew Cain <groksrc@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2025-11-13 09:12:26 -06:00
Drew Cain 1af05392ee fix: Add rclone installation checks for Windows bisync commands (#427) 2025-11-12 14:22:14 -06:00
Brandon Mayes cad7019c89 fix: main project always recreated on project list command (#421) 2025-11-12 09:57:08 -05:00
phernandez 099c334e3d chore: update version to 0.16.1 for v0.16.1 release 2025-11-11 09:21:47 -06:00
phernandez 7685586178 docs: Add v0.16.1 CHANGELOG entry for Windows line ending fix
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-11 09:10:26 -06:00
Paul Hernandez e9d0a944a9 fix: Handle Windows line endings in rclone bisync (#422)
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-11 09:08:20 -06:00
phernandez caf3c14bb1 chore: update version to 0.16.0 for v0.16.0 release 2025-11-10 19:19:48 -06:00
phernandez c5d9067754 docs: Add v0.16.0 CHANGELOG entry with comprehensive release notes
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-10 19:17:29 -06:00
phernandez e0fc59ea97 style: Format upload.py for better readability
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-10 19:15:03 -06:00
Paul Hernandez 49b2adc35c fix: skip archive files during cloud upload (#420)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-11-10 19:04:49 -06:00
Paul Hernandez 1646572f69 fix: Rename write_note entity_type to note_type for clarity (#419)
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-11-10 19:02:50 -06:00
Paul Hernandez f0d7398815 fix: Quote string values in YAML frontmatter to handle special characters (#418)
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-11-10 18:13:48 -06:00
Paul Hernandez 581b7b17c6 fix: Add explicit type annotations to MCP tool parameters (#394)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-11-10 17:02:17 -06:00
Paul Hernandez d775f7bab9 fix: Simplify search_notes schema by removing Optional wrappers (#395)
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-11-10 16:30:59 -06:00
Drew Cain fc01f6abaf fix: Replace Unicode arrows with ASCII for Windows compatibility (#414) 2025-11-10 16:30:41 -06:00
Paul Hernandez 4614fd09d5 fix: Handle dict objects in write_resource endpoint (#415)
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-11-10 16:30:27 -06:00
phernandez 0d4ad7bbf0 remove v0.15.0 info from assistant-guide
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-10 16:25:49 -06:00
jope-bm 7ccec7eba2 feat: Add run_in_background parameter to sync endpoint with tests (#417)
Co-authored-by: Claude <noreply@anthropic.com>
2025-11-07 09:04:43 -07:00
Paul Hernandez 021af74545 fix: Strip duplicate headers in edit_note replace_section (#396)
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Drew Cain <groksrc@users.noreply.github.com>
2025-11-02 14:20:11 -06:00
phernandez 2ad0ee9d5d fix: Use force_full=true for database sync after project sync/bisync
After rclone synchronizes files between local and cloud storage, the
database needs to perform a full scan to ensure it captures all changes.
Previously, incremental sync (watermark optimization) could miss files
that were changed remotely.

Changes:
- project sync command now calls /project/sync?force_full=true
- project bisync command now calls /project/sync?force_full=true
- Ensures complete database refresh after file synchronization

This guarantees the database is fully in sync with the filesystem
after any rclone sync or bisync operation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-02 13:26:08 -06:00
Brandon Mayes c9946ecf1e fix: Various rclone fixes for cloud sync on Windows (#410)
Signed-off-by: Brandon Mayes <5610870+bdmayes@users.noreply.github.com>
Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: phernandez <paul@basicmachines.co>
2025-11-02 11:26:57 -06:00
Drew Cain 0ba6f219f1 fix: Windows CLI Unicode encoding errors (#411)
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
2025-11-02 10:20:48 -06:00
Paul Hernandez 0b3272ae6e feat: SPEC-20 Simplified Project-Scoped Rclone Sync (#405)
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
2025-11-02 09:35:26 -06:00
Paul Hernandez a7d7cc5ee6 fix: Normalize YAML frontmatter types to prevent AttributeError (#236) (#402)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-27 09:19:50 -05:00
Paul Hernandez a7e696b039 Add free trial information to README
Added information about a 7-day free trial.

Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
2025-10-24 10:05:11 -05:00
Paul Hernandez 8aaddb6d45 Add free trial information to README
Added information about a 7-day free trial to the README.

Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
2025-10-24 10:04:41 -05:00
Paul Hernandez d7565312fc Announce Basic Memory Cloud launch in README
Added a section announcing the launch of Basic Memory Cloud with details on cross-device support and early supporter pricing.

Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
2025-10-24 09:53:14 -05:00
Paul Hernandez c7e6eab02f feat: Add delete_notes parameter to remove project endpoint (#391)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-21 14:09:20 -05:00
Paul Hernandez bb8da31472 fix: Handle null, empty, and string 'None' title in markdown frontmatter (#387) (#389)
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-10-21 09:29:19 -05:00
Paul Hernandez e78345ff25 feat: Streaming Foundation & Async I/O Consolidation (SPEC-19) (#384)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-21 09:03:59 -05:00
Paul Hernandez 32236cd247 fix: Handle YAML parsing errors gracefully in update_frontmatter (#378) (#379)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-16 20:17:58 -05:00
Paul Hernandez 4fd6d0c648 fix: Optimize sync memory usage to prevent OOM on large projects (#380)
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-10-16 20:17:34 -05:00
Paul Hernandez e6c8e3662c fix: preserve mtime webdav upload 376 (#377)
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-16 17:18:10 -05:00
Paul Hernandez 449b62d947 fix: Prevent deleted projects from being recreated by background sync (#193) (#370)
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-10-16 15:16:24 -05:00
Paul Hernandez b7497d7484 fix: Use filesystem timestamps for entity sync instead of database operation time (#138) (#369)
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-10-16 14:21:27 -05:00
Paul Hernandez d1431bdb1b fix: Handle YAML parsing errors and missing entity_type in markdown files (#368)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-16 13:18:58 -05:00
Paul Hernandez 171bef717f fix: Resolve UNIQUE constraint violation in entity upsert with observations (#187) (#367)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-16 12:30:56 -05:00
Paul Hernandez 729a5a3b8d fix: Terminate sync immediately when project is deleted (#366)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-16 11:07:12 -05:00
Paul Hernandez 434cdf24dd feat: Add circuit breaker for file sync failures (#364)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-16 09:47:48 -05:00
Paul Hernandez 7f9c1a97a4 feat: Add --verbose and --no-gitignore options to cloud upload (#362)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-15 20:03:36 -05:00
Paul Hernandez 53fb13b054 fix: Make project creation endpoint idempotent (#357)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-15 19:27:41 -05:00
Paul Hernandez bd6c8348b8 fix: Handle None text values in Claude conversations importer (#353)
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-10-15 16:08:57 -05:00
332 changed files with 32565 additions and 27755 deletions
-154
View File
@@ -1,154 +0,0 @@
---
name: python-developer
description: Python backend developer specializing in FastAPI, DBOS workflows, and API implementation. Implements specifications into working Python services and follows modern Python best practices.
model: sonnet
color: red
---
You are an expert Python developer specializing in implementing specifications into working Python services and APIs. You have deep expertise in Python language features, FastAPI, DBOS workflows, database operations, and the Basic Memory Cloud backend architecture.
**Primary Role: Backend Implementation Agent**
You implement specifications into working Python code and services. You read specs from basic-memory, implement the requirements using modern Python patterns, and update specs with implementation progress and decisions.
**Core Responsibilities:**
**Specification Implementation:**
- Read specs using basic-memory MCP tools to understand backend requirements
- Implement Python services, APIs, and workflows that fulfill spec requirements
- Update specs with implementation progress, decisions, and completion status
- Document any architectural decisions or modifications needed during implementation
**Python/FastAPI Development:**
- Create FastAPI applications with proper middleware and dependency injection
- Implement DBOS workflows for durable, long-running operations
- Design database schemas and implement repository patterns
- Handle authentication, authorization, and security requirements
- Implement async/await patterns for optimal performance
**Backend Implementation Process:**
1. **Read Spec**: Use `mcp__basic-memory__read_note` to get spec requirements
2. **Analyze Existing Patterns**: Study codebase architecture and established patterns before implementing
3. **Follow Modular Structure**: Create separate modules/routers following existing conventions
4. **Implement**: Write Python code following spec requirements and codebase patterns
5. **Test**: Create tests that validate spec success criteria
6. **Update Spec**: Document completion and any implementation decisions
7. **Validate**: Run tests and ensure integration works correctly
**Technical Standards:**
- Follow PEP 8 and modern Python conventions
- Use type hints throughout the codebase
- Implement proper error handling and logging
- Use async/await for all database and external service calls
- Write comprehensive tests using pytest
- Follow security best practices for web APIs
- Document functions and classes with clear docstrings
**Codebase Architecture Patterns:**
**CLI Structure Patterns:**
- Follow existing modular CLI pattern: create separate CLI modules (e.g., `upload_cli.py`) instead of adding commands directly to `main.py`
- Existing examples: `polar_cli.py`, `tenant_cli.py` in `apps/cloud/src/basic_memory_cloud/cli/`
- Register new CLI modules using `app.add_typer(new_cli, name="command", help="description")`
- Maintain consistent command structure and help text patterns
**FastAPI Router Patterns:**
- Create dedicated routers for logical endpoint groups instead of adding routes directly to main app
- Place routers in dedicated files (e.g., `apps/api/src/basic_memory_cloud_api/routers/webdav_router.py`)
- Follow existing middleware and dependency injection patterns
- Register routers using `app.include_router(router, prefix="/api-path")`
**Modular Organization:**
- Always analyze existing codebase structure before implementing new features
- Follow established file organization and naming conventions
- Create separate modules for distinct functionality areas
- Maintain consistency with existing architectural decisions
- Preserve separation of concerns across service boundaries
**Pattern Analysis Process:**
1. Examine similar existing functionality in the codebase
2. Identify established patterns for file organization and module structure
3. Follow the same architectural approach for consistency
4. Create new modules/routers following existing conventions
5. Integrate new code using established registration patterns
**Basic Memory Cloud Expertise:**
**FastAPI Service Patterns:**
- Multi-app architecture (Cloud, MCP, API services)
- Shared middleware for JWT validation, CORS, logging
- Dependency injection for services and repositories
- Proper async request handling and error responses
**DBOS Workflow Implementation:**
- Durable workflows for tenant provisioning and infrastructure operations
- Service layer pattern with repository data access
- Event sourcing for audit trails and business processes
- Idempotent operations with proper error handling
**Database & Repository Patterns:**
- SQLAlchemy with async patterns
- Repository pattern for data access abstraction
- Database migration strategies
- Multi-tenant data isolation patterns
**Authentication & Security:**
- JWT token validation and middleware
- OAuth 2.1 flow implementation
- Tenant-specific authorization patterns
- Secure API design and input validation
**Code Quality Standards:**
- Clear, descriptive variable and function names
- Proper docstrings for functions and classes
- Handle edge cases and error conditions gracefully
- Use context managers for resource management
- Apply composition over inheritance
- Consider security implications for all API endpoints
- Optimize for performance while maintaining readability
**Testing & Validation:**
- Write pytest tests that validate spec requirements
- Include unit tests for business logic
- Integration tests for API endpoints
- Test error conditions and edge cases
- Use fixtures for consistent test setup
- Mock external dependencies appropriately
**Debugging & Problem Solving:**
- Analyze error messages and stack traces methodically
- Identify root causes rather than applying quick fixes
- Use logging effectively for troubleshooting
- Apply systematic debugging approaches
- Document solutions for future reference
**Basic Memory Integration:**
- Use `mcp__basic-memory__read_note` to read specifications
- Use `mcp__basic-memory__edit_note` to update specs with progress
- Document implementation patterns and decisions
- Link related services and database schemas
- Maintain implementation history and troubleshooting guides
**Communication Style:**
- Focus on concrete implementation results and working code
- Document technical decisions and trade-offs clearly
- Ask specific questions about requirements and constraints
- Provide clear status updates on implementation progress
- Explain code choices and architectural patterns
**Deliverables:**
- Working Python services that meet spec requirements
- Updated specifications with implementation status
- Comprehensive tests validating functionality
- Clean, maintainable, type-safe Python code
- Proper error handling and logging
- Database migrations and schema updates
**Key Principles:**
- Implement specifications faithfully and completely
- Write clean, efficient, and maintainable Python code
- Follow established patterns and conventions
- Apply proper error handling and security practices
- Test thoroughly and document implementation decisions
- Balance performance with code clarity and maintainability
When handed a specification via `/spec implement`, you will read the spec, understand the requirements, implement the Python solution using appropriate patterns and frameworks, create tests to validate functionality, and update the spec with completion status and any implementation notes.
-126
View File
@@ -1,126 +0,0 @@
---
name: system-architect
description: System architect who designs and implements architectural solutions, creates ADRs, and applies software engineering principles to solve complex system design problems.
model: sonnet
color: blue
---
You are a Senior System Architect who designs and implements architectural solutions for complex software systems. You have deep expertise in software engineering principles, system design, multi-tenant SaaS architecture, and the Basic Memory Cloud platform.
**Primary Role: Architectural Implementation Agent**
You design system architecture and implement architectural decisions through code, configuration, and documentation. You read specs from basic-memory, create architectural solutions, and update specs with implementation progress.
**Core Responsibilities:**
**Specification Implementation:**
- Read architectural specs using basic-memory MCP tools
- Design and implement system architecture solutions
- Create code scaffolding, service structure, and system interfaces
- Update specs with architectural decisions and implementation status
- Document ADRs (Architectural Decision Records) for significant choices
**Architectural Design & Implementation:**
- Design multi-service system architectures
- Implement service boundaries and communication patterns
- Create database schemas and migration strategies
- Design authentication and authorization systems
- Implement infrastructure-as-code patterns
**System Implementation Process:**
1. **Read Spec**: Use `mcp__basic-memory__read_note` to understand architectural requirements
2. **Design Solution**: Apply architectural principles and patterns
3. **Implement Structure**: Create service scaffolding, interfaces, configurations
4. **Document Decisions**: Create ADRs documenting architectural choices
5. **Update Spec**: Record implementation progress and decisions
6. **Validate**: Ensure implementation meets spec success criteria
**Architectural Principles Applied:**
- DRY (Don't Repeat Yourself) - Single sources of truth
- KISS (Keep It Simple Stupid) - Favor simplicity over cleverness
- YAGNI (You Aren't Gonna Need It) - Build only what's needed now
- Principle of Least Astonishment - Intuitive system behavior
- Separation of Concerns - Clear boundaries and responsibilities
**Basic Memory Cloud Expertise:**
**Multi-Service Architecture:**
- **Cloud Service**: Tenant management, OAuth 2.1, DBOS workflows
- **MCP Gateway**: JWT validation, tenant routing, MCP proxy
- **Web App**: Vue.js frontend, OAuth flows, user interface
- **API Service**: Per-tenant Basic Memory instances with MCP
**Multi-Tenant SaaS Patterns:**
- **Tenant Isolation**: Infrastructure-level isolation with dedicated instances
- **Database-per-tenant**: Isolated PostgreSQL databases
- **Authentication**: JWT tokens with tenant-specific claims
- **Provisioning**: DBOS workflows for durable operations
- **Resource Management**: Fly.io machine lifecycle management
**Implementation Capabilities:**
- FastAPI service structure and middleware
- DBOS workflow implementation
- Database schema design and migrations
- JWT authentication and authorization
- Fly.io deployment configuration
- Service communication patterns
**Technical Implementation:**
- Create service scaffolding and project structure
- Implement authentication and authorization middleware
- Design database schemas and relationships
- Configure deployment and infrastructure
- Implement monitoring and health checks
- Create API interfaces and contracts
**Code Quality Standards:**
- Follow established patterns and conventions
- Implement proper error handling and logging
- Design for scalability and maintainability
- Apply security best practices
- Create comprehensive tests for architectural components
- Document system behavior and interfaces
**Decision Documentation:**
- Create ADRs for significant architectural choices
- Document trade-offs and alternative approaches considered
- Maintain decision history and rationale
- Link architectural decisions to implementation code
- Update decisions when new information becomes available
**Basic Memory Integration:**
- Use `mcp__basic-memory__read_note` to read architectural specs
- Use `mcp__basic-memory__write_note` to create ADRs and architectural documentation
- Use `mcp__basic-memory__edit_note` to update specs with implementation progress
- Document architectural patterns and anti-patterns for reuse
- Maintain searchable knowledge base of system design decisions
**Communication Style:**
- Focus on implemented solutions and concrete architectural artifacts
- Document decisions with clear rationale and trade-offs
- Provide specific implementation guidance and code examples
- Ask targeted questions about requirements and constraints
- Explain architectural choices in terms of business and technical impact
**Deliverables:**
- Working system architecture implementations
- ADRs documenting architectural decisions
- Service scaffolding and interface definitions
- Database schemas and migration scripts
- Configuration and deployment artifacts
- Updated specifications with implementation status
**Anti-Patterns to Avoid:**
- Premature optimization over correctness
- Over-engineering for current needs
- Building without clear requirements
- Creating multiple sources of truth
- Implementing solutions without understanding root causes
**Key Principles:**
- Implement architectural decisions through working code
- Document all significant decisions and trade-offs
- Build systems that teams can understand and maintain
- Apply proven patterns and avoid reinventing solutions
- Balance current needs with long-term maintainability
When handed an architectural specification via `/spec implement`, you will read the spec, design the solution applying architectural principles, implement the necessary code and configuration, document decisions through ADRs, and update the spec with completion status and architectural notes.
+125 -18
View File
@@ -15,10 +15,16 @@ Create a stable release using the automated justfile target with comprehensive v
You are an expert release manager for the Basic Memory project. When the user runs `/release`, execute the following steps:
### Step 1: Pre-flight Validation
1. Verify version format matches `v\d+\.\d+\.\d+` pattern
2. Check current git status for uncommitted changes
3. Verify we're on the `main` branch
4. Confirm no existing tag with this version
#### Version Check
1. Check current version in `src/basic_memory/__init__.py`
2. Verify new version format matches `v\d+\.\d+\.\d+` pattern
3. Confirm version is higher than current version
#### Git Status
1. Check current git status for uncommitted changes
2. Verify we're on the `main` branch
3. Confirm no existing tag with this version
#### Documentation Validation
1. **Changelog Check**
@@ -39,19 +45,111 @@ The justfile target handles:
- ✅ Version update in `src/basic_memory/__init__.py`
- ✅ Automatic commit with proper message
- ✅ Tag creation and pushing to GitHub
- ✅ Release workflow trigger
- ✅ Release workflow trigger (automatic on tag push)
The GitHub Actions workflow (`.github/workflows/release.yml`) then:
- ✅ Builds the package using `uv build`
- ✅ Creates GitHub release with auto-generated notes
- ✅ Publishes to PyPI
- ✅ Updates Homebrew formula (stable releases only)
### Step 3: Monitor Release Process
1. Check that GitHub Actions workflow starts successfully
2. Monitor workflow completion at: https://github.com/basicmachines-co/basic-memory/actions
3. Verify PyPI publication
4. Test installation: `uv tool install basic-memory`
1. Verify tag push triggered the workflow (should start automatically within seconds)
2. Monitor workflow progress at: https://github.com/basicmachines-co/basic-memory/actions
3. Watch for successful completion of both jobs:
- `release` - Builds package and publishes to PyPI
- `homebrew` - Updates Homebrew formula (stable releases only)
4. Check for any workflow failures and investigate logs if needed
### Step 4: Post-Release Validation
1. Verify GitHub release is created automatically
2. Check PyPI publication
3. Validate release assets
4. Update any post-release documentation
#### GitHub Release
1. Verify GitHub release is created at: https://github.com/basicmachines-co/basic-memory/releases/tag/<version>
2. Check that release notes are auto-generated from commits
3. Validate release assets (`.whl` and `.tar.gz` files are attached)
#### PyPI Publication
1. Verify package published at: https://pypi.org/project/basic-memory/<version>/
2. Test installation: `uv tool install basic-memory`
3. Verify installed version: `basic-memory --version`
#### Homebrew Formula (Stable Releases Only)
1. Check formula update at: https://github.com/basicmachines-co/homebrew-basic-memory
2. Verify formula version matches release
3. Test Homebrew installation: `brew install basicmachines-co/basic-memory/basic-memory`
#### MCP Registry Publication
After PyPI release is published, update the MCP registry:
1. **Verify PyPI Release**
- Confirm package is live: https://pypi.org/project/basic-memory/<version>/
- The `server.json` version was auto-updated by `just release`
2. **Publish to MCP Registry**
```bash
cd /Users/drew/code/basic-memory
mcp-publisher publish
```
If not authenticated:
```bash
mcp-publisher login github
# Follow device authentication flow
mcp-publisher publish
```
3. **Verify Publication**
```bash
curl "https://registry.modelcontextprotocol.io/v0.1/servers?search=basic-memory"
```
**Note:** The `mcp-publisher` CLI can be installed via Homebrew (`brew install mcp-publisher`) or from GitHub releases.
#### Website Updates
**1. basicmachines.co** (`/Users/drew/code/basicmachines.co`)
- **Goal**: Update version number displayed on the homepage
- **Location**: Search for "Basic Memory v0." in the codebase to find version displays
- **What to update**:
- Hero section heading that shows "Basic Memory v{VERSION}"
- "What's New in v{VERSION}" section heading
- Feature highlights array (look for array of features with title/description)
- **Process**:
1. Pull latest from GitHub: `git pull origin main`
2. Create release branch: `git checkout -b release/v{VERSION}`
3. Search codebase for current version number (e.g., "v0.16.1")
4. Update version numbers to new release version
5. Update feature highlights with 3-5 key features from this release (extract from CHANGELOG.md)
6. Commit changes: `git commit -m "chore: update to v{VERSION}"`
7. Push branch: `git push origin release/v{VERSION}`
- **Deploy**: Follow deployment process for basicmachines.co
**2. docs.basicmemory.com** (`/Users/drew/code/docs.basicmemory.com`)
- **Goal**: Add new release notes section to the latest-releases page
- **File**: `src/pages/latest-releases.mdx`
- **What to do**:
1. Pull latest from GitHub: `git pull origin main`
2. Create release branch: `git checkout -b release/v{VERSION}`
3. Read the existing file to understand the format and structure
4. Read `/Users/drew/code/basic-memory/CHANGELOG.md` to get release content
5. Add new release section **at the top** (after MDX imports, before other releases)
6. Follow the existing pattern:
- Heading: `## [v{VERSION}](github-link) — YYYY-MM-DD`
- Focus statement if applicable
- `<Info>` block with highlights (3-5 key items)
- Sections for Features, Bug Fixes, Breaking Changes, etc.
- Link to full changelog at the end
- Separator `---` between releases
7. Commit changes: `git commit -m "docs: add v{VERSION} release notes"`
8. Push branch: `git push origin release/v{VERSION}`
- **Source content**: Extract and format sections from CHANGELOG.md for this version
- **Deploy**: Follow deployment process for docs.basicmemory.com
**4. Announce Release**
- Post to Discord community if significant changes
- Update social media if major release
- Notify users via appropriate channels
## Pre-conditions Check
Before starting, verify:
@@ -74,19 +172,28 @@ Before starting, verify:
🏷️ Tag: v0.13.2
📋 GitHub Release: https://github.com/basicmachines-co/basic-memory/releases/tag/v0.13.2
📦 PyPI: https://pypi.org/project/basic-memory/0.13.2/
🍺 Homebrew: https://github.com/basicmachines-co/homebrew-basic-memory
🔌 MCP Registry: https://registry.modelcontextprotocol.io
🚀 GitHub Actions: Completed
Install with:
uv tool install basic-memory
Install with pip/uv:
uv tool install basic-memory
Install with Homebrew:
brew install basicmachines-co/basic-memory/basic-memory
Users can now upgrade:
uv tool upgrade basic-memory
uv tool upgrade basic-memory
brew upgrade basic-memory
```
## Context
- This creates production releases used by end users
- Must pass all quality gates before proceeding
- Uses the automated justfile target for consistency
- Version is automatically updated in `__init__.py`
- Version is automatically updated in `__init__.py` and `server.json`
- Triggers automated GitHub release with changelog
- Leverages uv-dynamic-versioning for package version management
- Package is published to PyPI for `pip` and `uv` users
- Homebrew formula is automatically updated for stable releases
- MCP Registry is updated manually via `mcp-publisher publish`
- Supports multiple installation methods (uv, pip, Homebrew)
+16 -20
View File
@@ -1,17 +1,19 @@
---
allowed-tools: mcp__basic-memory__write_note, mcp__basic-memory__read_note, mcp__basic-memory__search_notes, mcp__basic-memory__edit_note, Task
argument-hint: [create|status|implement|review] [spec-name]
allowed-tools: mcp__basic-memory__write_note, mcp__basic-memory__read_note, mcp__basic-memory__search_notes, mcp__basic-memory__edit_note
argument-hint: [create|status|show|review] [spec-name]
description: Manage specifications in our development process
---
## Context
You are managing specifications using our specification-driven development process defined in @docs/specs/SPEC-001.md.
Specifications are managed in the Basic Memory "specs" project. All specs live in a centralized location accessible across all repositories via MCP tools.
See SPEC-1 and SPEC-2 in the "specs" project for the full specification-driven development process.
Available commands:
- `create [name]` - Create new specification
- `status` - Show all spec statuses
- `implement [spec-name]` - Hand spec to appropriate agent
- `show [spec-name]` - Read a specific spec
- `review [spec-name]` - Review implementation against spec
## Your task
@@ -19,23 +21,19 @@ Available commands:
Execute the spec command: `/spec $ARGUMENTS`
### If command is "create":
1. Get next SPEC number by searching existing specs
2. Create new spec using template from @docs/specs/Slash\ Commands\ Reference.md
3. Place in `/specs` folder with title "SPEC-XXX: [name]"
1. Get next SPEC number by searching existing specs in "specs" project
2. Create new spec using template from SPEC-2
3. Use mcp__basic-memory__write_note with project="specs"
4. Include standard sections: Why, What, How, How to Evaluate
### If command is "status":
1. Search all notes in `/specs` folder
2. Display table with spec number, title, and status
3. Show any dependencies or assigned agents
1. Use mcp__basic-memory__search_notes with project="specs"
2. Display table with spec number, title, and progress
3. Show completion status from checkboxes in content
### If command is "implement":
1. Read the specified spec
2. Determine appropriate agent based on content:
- Frontend/UI → vue-developer
- Architecture/system → system-architect
- Backend/API → python-developer
3. Launch Task tool with appropriate agent and spec context
### If command is "show":
1. Use mcp__basic-memory__read_note with project="specs"
2. Display the full spec content
### If command is "review":
1. Read the specified spec and its "How to Evaluate" section
@@ -49,7 +47,5 @@ Execute the spec command: `/spec $ARGUMENTS`
- **Architecture compliance** - Component isolation, state management patterns
- **Documentation completeness** - Implementation matches specification
3. Provide honest, accurate assessment - do not overstate completeness
4. Document findings and update spec with review results
4. Document findings and update spec with review results using mcp__basic-memory__edit_note
5. If gaps found, clearly identify what still needs to be implemented/tested
Use the agent definitions from @docs/specs/Agent\ Definitions.md for implementation handoffs.
+5
View File
@@ -0,0 +1,5 @@
{
"enabledPlugins": {
"basic-memory@basicmachines": true
}
}
+28
View File
@@ -0,0 +1,28 @@
# Basic Memory Environment Variables Example
# Copy this file to .env and customize as needed
# Note: .env files are gitignored and should never be committed
# ============================================================================
# PostgreSQL Test Database Configuration
# ============================================================================
# These variables allow you to override the default test database credentials
# Default values match docker-compose-postgres.yml for local development
#
# Only needed if you want to use different credentials or a remote test database
# By default, tests use: postgresql://basic_memory_user:dev_password@localhost:5433/basic_memory_test
# Full PostgreSQL test database URL (used by tests and migrations)
# POSTGRES_TEST_URL=postgresql+asyncpg://basic_memory_user:dev_password@localhost:5433/basic_memory_test
# Individual components (used by justfile postgres-reset command)
# POSTGRES_USER=basic_memory_user
# POSTGRES_TEST_DB=basic_memory_test
# ============================================================================
# Production Database Configuration
# ============================================================================
# For production use, set these in your deployment environment
# DO NOT use the test credentials above in production!
# BASIC_MEMORY_DATABASE_BACKEND=postgres # or "sqlite"
# BASIC_MEMORY_DATABASE_URL=postgresql+asyncpg://user:password@host:port/database
+5 -1
View File
@@ -54,6 +54,7 @@ jobs:
- [ ] Unit tests for new functions/methods
- [ ] Integration tests for new MCP tools
- [ ] Test coverage for edge cases
- [ ] **100% test coverage maintained** (use `# pragma: no cover` only for truly hard-to-test code)
- [ ] Documentation updated (README, docstrings)
- [ ] CLAUDE.md updated if conventions change
@@ -71,9 +72,12 @@ jobs:
- [ ] Proper error handling and logging
- [ ] Performance considerations addressed
- [ ] No sensitive data in logs or commits
## Compatability
- [ ] File path comparisons must be windows compatible
- [ ] Avoid using emojis and unicode characters in console and log output
Read the CLAUDE.md file for detailed project context. For each checklist item, verify if it's satisfied and comment on any that need attention. Use inline comments for specific code issues and post a summary with checklist results.
# Allow broader tool access for thorough code review
claude_args: '--allowed-tools "Bash(gh pr:*),Bash(gh issue:*),Bash(gh api:*),Bash(git log:*),Bash(git show:*),Read,Grep,Glob"'
+101 -4
View File
@@ -13,12 +13,13 @@ on:
branches: [ "main" ]
jobs:
test:
test-sqlite:
name: Test SQLite (${{ matrix.os }}, Python ${{ matrix.python-version }})
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest]
python-version: [ "3.12", "3.13" ]
python-version: [ "3.12", "3.13", "3.14" ]
runs-on: ${{ matrix.os }}
steps:
@@ -64,7 +65,103 @@ jobs:
run: |
just lint
- name: Run tests
- name: Run tests (SQLite)
run: |
uv pip install pytest pytest-cov
just test
just test-sqlite
test-postgres:
name: Test Postgres (Python ${{ matrix.python-version }})
strategy:
fail-fast: false
matrix:
python-version: [ "3.12", "3.13", "3.14" ]
runs-on: ubuntu-latest
# Note: No services section needed - testcontainers handles Postgres in Docker
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
- name: Install uv
run: |
pip install uv
- name: Install just
run: |
curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to /usr/local/bin
- name: Create virtual env
run: |
uv venv
- name: Install dependencies
run: |
uv pip install -e .[dev]
- name: Run tests (Postgres via testcontainers)
run: |
uv pip install pytest pytest-cov
just test-postgres
coverage:
name: Coverage Summary (combined, Python 3.12)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python 3.12
uses: actions/setup-python@v4
with:
python-version: "3.12"
cache: "pip"
- name: Install uv
run: |
pip install uv
- name: Install just
run: |
curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to /usr/local/bin
- name: Create virtual env
run: |
uv venv
- name: Install dependencies
run: |
uv pip install -e .[dev]
- name: Run combined coverage (SQLite + Postgres)
run: |
uv pip install pytest pytest-cov
just coverage
- name: Add coverage report to job summary
if: always()
run: |
{
echo "## Coverage"
echo ""
echo '```'
uv run coverage report -m
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
- name: Upload HTML coverage report
if: always()
uses: actions/upload-artifact@v4
with:
name: htmlcov
path: htmlcov/
+3 -1
View File
@@ -52,4 +52,6 @@ ENV/
# claude action
claude-output
**/.claude/settings.local.json
**/.claude/settings.local.json
.mcp.json
.mcpregistry_*
+1 -1
View File
@@ -1 +1 @@
3.12
3.14
+586
View File
@@ -1,5 +1,591 @@
# CHANGELOG
## v0.17.8 (2026-01-24)
### Bug Fixes
- **#521**: Fix `get_default_project()` returning multiple results
([`6888eff`](https://github.com/basicmachines-co/basic-memory/commit/6888eff))
- Query incorrectly matched any project with non-NULL `is_default` (both True and False)
- Now correctly checks for `is_default=True` only
## v0.17.7 (2026-01-24)
### Features
- **#476**: Add SPEC-29 Phase 3 bucket snapshot CLI commands
([`369ad37`](https://github.com/basicmachines-co/basic-memory/commit/369ad37))
- New `basic-memory cloud snapshot` commands for managing cloud snapshots
- Commands: `create`, `list`, `delete`, `show`, `browse`
- **#515**: Add MCP registry publication files
([`7a502e6`](https://github.com/basicmachines-co/basic-memory/commit/7a502e6))
### Bug Fixes
- **#520**: Read default project from database in cloud mode
([`38616c3`](https://github.com/basicmachines-co/basic-memory/commit/38616c3))
- **#513**: Ensure external_id is set on entity creation
([`c7835a9`](https://github.com/basicmachines-co/basic-memory/commit/c7835a9))
### Internal
- **#514**: Remove OpenPanel telemetry
([`85835ae`](https://github.com/basicmachines-co/basic-memory/commit/85835ae))
- Update README links to point to basicmemory.com
([`2aaee73`](https://github.com/basicmachines-co/basic-memory/commit/2aaee73))
## v0.17.6 (2026-01-17)
### Bug Fixes
- **#510**: Fix Docker container Python symlink broken at runtime
([`1799c94`](https://github.com/basicmachines-co/basic-memory/commit/1799c94))
### Internal
- Remove logfire config and specs docs, reduce lifespan and sync logging to debug level
([`d1d433d`](https://github.com/basicmachines-co/basic-memory/commit/d1d433d),
[`803f3ef`](https://github.com/basicmachines-co/basic-memory/commit/803f3ef))
## v0.17.5 (2026-01-11)
### Bug Fixes
- **#505**: Prevent CLI commands from hanging on exit (Python 3.14 compatibility)
([`863e0a4`](https://github.com/basicmachines-co/basic-memory/commit/863e0a4))
- Skip `nest_asyncio` on Python 3.14+ where it causes event loop issues
- Simplify CLI test infrastructure for cross-version compatibility
- Update pyright to 1.1.408 for Python 3.14 support
- Fix SQLAlchemy rowcount typing for Python 3.14
## v0.17.4 (2026-01-05)
### Bug Fixes
- **#503**: Preserve search index across server restarts
([`26f7e98`](https://github.com/basicmachines-co/basic-memory/commit/26f7e98))
- Fixes critical bug where search index was wiped on every MCP server restart
- Bug was introduced in v0.16.3, affecting v0.16.3-v0.17.3
- **User action**: Run `basic-memory reset` once after updating to rebuild search index
### Internal
- **#502**: Major architecture refactor with composition roots and typed API clients
([`5947f04`](https://github.com/basicmachines-co/basic-memory/commit/5947f04))
- Add composition roots for API, MCP, and CLI entrypoints
- Split deps.py into feature-scoped modules (config, db, projects, repositories, services, importers)
- Add ProjectResolver for unified project selection
- Add SyncCoordinator for centralized sync/watch lifecycle
- Introduce typed API clients for MCP tools (KnowledgeClient, SearchClient, MemoryClient, etc.)
## v0.17.3 (2026-01-03)
### Features
- **#485**: Add stable external_id (UUID) to Project and Entity models
([`a4000f6`](https://github.com/basicmachines-co/basic-memory/commit/a4000f6))
- Projects and entities now have immutable UUID identifiers
- API v2 endpoints use external_id for stable references
- Directory responses include external_id for entities
### Bug Fixes
- **#501**: Update mcp dependency to support protocol version 2025-11-25
([`c6baf58`](https://github.com/basicmachines-co/basic-memory/commit/c6baf58))
- Fixes "Unsupported protocol version" error when using Claude Code
- Bump mcp from >=1.2.0 to >=1.23.1
- **#499**: Fix route ordering for cloud deployments
([`53c4c20`](https://github.com/basicmachines-co/basic-memory/commit/53c4c20))
- **#486**: Skip config file update for set_default_project in cloud mode
([`fd732aa`](https://github.com/basicmachines-co/basic-memory/commit/fd732aa))
- **#484**: Make RelationResponse.from_id optional to handle null permalinks
([`537e58a`](https://github.com/basicmachines-co/basic-memory/commit/537e58a))
- Use upsert to prevent IntegrityError during parallel search indexing
([`4ce2198`](https://github.com/basicmachines-co/basic-memory/commit/4ce2198))
- Use relative file paths in importers for cloud storage compatibility
([`8adf1f4`](https://github.com/basicmachines-co/basic-memory/commit/8adf1f4))
### Internal
- Refactor importers to use FileService for cloud compatibility
([`45ce181`](https://github.com/basicmachines-co/basic-memory/commit/45ce181))
- Strengthen integration test coverage, remove stdlib mocks
([`b4486d2`](https://github.com/basicmachines-co/basic-memory/commit/b4486d2))
## v0.17.2 (2025-12-29)
### Bug Fixes
- Allow recent_activity discovery mode in cloud mode
([`0bcda4a`](https://github.com/basicmachines-co/basic-memory/commit/0bcda4a))
- Add `allow_discovery` parameter to `resolve_project_parameter()`
- Tools like `recent_activity` can now work across all projects in cloud mode
- Fix circular import in project_context module
### Internal
- Optimize release workflow by running lint/typecheck only (skip full tests)
([`0b5425f`](https://github.com/basicmachines-co/basic-memory/commit/0b5425f))
## v0.17.1 (2025-12-29)
### Bug Fixes
- **#482**: Only set BASIC_MEMORY_ENV=test during pytest runs
([`98fbd60`](https://github.com/basicmachines-co/basic-memory/commit/98fbd60))
- Fixes environment variable pollution affecting alembic migrations
- Test environment detection now scoped to pytest execution only
## v0.17.0 (2025-12-28)
### Features
- **#478**: Add anonymous usage telemetry with Homebrew-style opt-out
([`856737f`](https://github.com/basicmachines-co/basic-memory/commit/856737f))
- Privacy-respecting anonymous usage analytics
- Easy opt-out via `BASIC_MEMORY_NO_ANALYTICS=1` environment variable
- Helps improve Basic Memory based on real usage patterns
- **#474**: Add auto-format files on save with built-in Python formatter
([`1fd680c`](https://github.com/basicmachines-co/basic-memory/commit/1fd680c))
- Automatic markdown formatting on file save
- Built-in Python formatter for consistent code style
- Configurable formatting options
- **#447**: Complete Phase 2 of API v2 migration - MCP tools use v2 endpoints
([`1a74d85`](https://github.com/basicmachines-co/basic-memory/commit/1a74d85))
- All MCP tools now use optimized v2 API endpoints
- Improved performance for knowledge graph operations
- Foundation for future API enhancements
### Bug Fixes
- Fix UTF-8 BOM handling in frontmatter parsing
([`85684f8`](https://github.com/basicmachines-co/basic-memory/commit/85684f8))
- Handles files with UTF-8 byte order marks correctly
- Prevents frontmatter parsing failures
- **#475**: Handle null titles in ChatGPT import
([`14ce5a3`](https://github.com/basicmachines-co/basic-memory/commit/14ce5a3))
- Gracefully handles conversations without titles
- Improved import robustness
- Remove MaxLen constraint from observation content
([`45d6caf`](https://github.com/basicmachines-co/basic-memory/commit/45d6caf))
- Allows longer observation content without truncation
- Removes arbitrary 2000 character limit
- Handle FileNotFoundError gracefully during sync
([`1652f86`](https://github.com/basicmachines-co/basic-memory/commit/1652f86))
- Prevents sync failures when files are deleted during sync
- More resilient file watching
- Use canonical project names in API response messages
([`c23927d`](https://github.com/basicmachines-co/basic-memory/commit/c23927d))
- Consistent project name formatting in all responses
- Suppress CLI warnings for cleaner output
([`d71c6e8`](https://github.com/basicmachines-co/basic-memory/commit/d71c6e8))
- Cleaner terminal output without spurious warnings
- Prevent DEBUG logs from appearing on CLI stdout
([`63b9849`](https://github.com/basicmachines-co/basic-memory/commit/63b9849))
- Debug logging no longer pollutes CLI output
- **#473**: Detect rclone version for --create-empty-src-dirs support
([`622d37e`](https://github.com/basicmachines-co/basic-memory/commit/622d37e))
- Automatic rclone version detection for compatibility
- Prevents errors on older rclone versions
- **#471**: Prevent CLI commands from hanging on exit
([`916baf8`](https://github.com/basicmachines-co/basic-memory/commit/916baf8))
- Fixes CLI hang on shutdown
- Proper async cleanup
- Add cloud_mode check to initialize_app()
([`ef7adb7`](https://github.com/basicmachines-co/basic-memory/commit/ef7adb7))
- Correct initialization for cloud deployments
### Internal
- Centralize test environment detection in config.is_test_env
([`3cd9178`](https://github.com/basicmachines-co/basic-memory/commit/3cd9178))
- Unified test environment detection
- Disables analytics in test environments
- Make test-int-postgres compatible with macOS
([`95937c6`](https://github.com/basicmachines-co/basic-memory/commit/95937c6))
- Cross-platform PostgreSQL testing support
## v0.16.3 (2025-12-20)
### Features
- **#439**: Add PostgreSQL database backend support
([`fb5e9e1`](https://github.com/basicmachines-co/basic-memory/commit/fb5e9e1))
- Full PostgreSQL/Neon database support as alternative to SQLite
- Async connection pooling with asyncpg
- Alembic migrations support for both backends
- Configurable via `BASIC_MEMORY_DATABASE_BACKEND` environment variable
- **#441**: Implement API v2 with ID-based endpoints (Phase 1)
([`28cc522`](https://github.com/basicmachines-co/basic-memory/commit/28cc522))
- New ID-based API endpoints for improved performance
- Foundation for future API enhancements
- Backward compatible with existing endpoints
- Add project_id to Relation and Observation for efficient project-scoped queries
([`a920a9f`](https://github.com/basicmachines-co/basic-memory/commit/a920a9f))
- Enables faster queries in multi-project environments
- Improved database schema for cloud deployments
- Add bulk insert with ON CONFLICT handling for relations
([`0818bda`](https://github.com/basicmachines-co/basic-memory/commit/0818bda))
- Faster relation creation during sync operations
- Handles duplicate relations gracefully
### Performance
- Lightweight permalink resolution to avoid eager loading
([`6f99d2e`](https://github.com/basicmachines-co/basic-memory/commit/6f99d2e))
- Reduces database queries during entity lookups
- Improved response times for read operations
### Bug Fixes
- **#464**: Pin FastMCP to 2.12.3 to fix MCP tools visibility
([`f227ef6`](https://github.com/basicmachines-co/basic-memory/commit/f227ef6))
- Fixes issue where MCP tools were not visible to Claude
- Reverts to last known working FastMCP version
- **#458**: Reduce watch service CPU usage by increasing reload interval
([`897b1ed`](https://github.com/basicmachines-co/basic-memory/commit/897b1ed))
- Lowers CPU usage during file watching
- More efficient resource utilization
- **#456**: Await background sync task cancellation in lifespan shutdown
([`efbc758`](https://github.com/basicmachines-co/basic-memory/commit/efbc758))
- Prevents hanging on shutdown
- Clean async task cleanup
- **#434**: Respect --project flag in background sync
([`70bb10b`](https://github.com/basicmachines-co/basic-memory/commit/70bb10b))
- Background sync now correctly uses specified project
- Fixes multi-project sync issues
- **#446**: Fix observation parsing and permalink limits
([`73d940e`](https://github.com/basicmachines-co/basic-memory/commit/73d940e))
- Handles edge cases in observation content
- Prevents permalink truncation issues
- **#424**: Handle periods in kebab_filenames mode
([`b004565`](https://github.com/basicmachines-co/basic-memory/commit/b004565))
- Fixes filename handling for files with multiple periods
- Improved kebab-case conversion
- Fix Postgres/Neon connection settings and search index dedupe
([`b5d4fb5`](https://github.com/basicmachines-co/basic-memory/commit/b5d4fb5))
- Optimized connection pooling for Postgres
- Prevents duplicate search index entries
### Testing & CI
- Replace py-pglite with testcontainers for Postgres testing
([`c462faf`](https://github.com/basicmachines-co/basic-memory/commit/c462faf))
- More reliable Postgres testing infrastructure
- Uses Docker-based test containers
- Add PostgreSQL testing to GitHub Actions workflow
([`66b91b2`](https://github.com/basicmachines-co/basic-memory/commit/66b91b2))
- CI now tests both SQLite and PostgreSQL backends
- Ensures cross-database compatibility
- **#416**: Add integration test for read_note with underscored folders
([`0c12a39`](https://github.com/basicmachines-co/basic-memory/commit/0c12a39))
- Verifies folder name handling edge cases
### Internal
- Cloud compatibility fixes and performance improvements (#454)
- Remove logfire instrumentation for cleaner production deployments
- Truncate content_stems to fix Postgres 8KB index row limit
## v0.16.2 (2025-11-16)
### Bug Fixes
- **#429**: Use platform-native path separators in config.json
([`6517e98`](https://github.com/basicmachines-co/basic-memory/commit/6517e98))
- Fixes config.json path separator issues on Windows
- Uses os.path.join for platform-native path construction
- Ensures consistent path handling across platforms
- **#427**: Add rclone installation checks for Windows bisync commands
([`1af0539`](https://github.com/basicmachines-co/basic-memory/commit/1af0539))
- Validates rclone installation before running bisync commands
- Provides clear error messages when rclone is not installed
- Improves user experience on Windows
- **#421**: Main project always recreated on project list command
([`cad7019`](https://github.com/basicmachines-co/basic-memory/commit/cad7019))
- Fixes issue where main project was recreated unnecessarily
- Improves project list command reliability
- Reduces unnecessary file system operations
## v0.16.1 (2025-11-11)
### Bug Fixes
- **#422**: Handle Windows line endings in rclone bisync
([`e9d0a94`](https://github.com/basicmachines-co/basic-memory/commit/e9d0a94))
- Added `--compare=modtime` flag to rclone bisync to ignore size differences from line ending conversions
- Fixes issue where LF→CRLF conversion on Windows was treated as file corruption
- Resolves "corrupted on transfer: sizes differ" errors during cloud sync on Windows
- Users will need to run `--resync` once after updating to establish new baseline
## v0.16.0 (2025-11-10)
### Features
- **#417**: Add run_in_background parameter to sync endpoint
([`7ccec7e`](https://github.com/basicmachines-co/basic-memory/commit/7ccec7e))
- New `run_in_background` parameter for async sync operations
- Improved API flexibility for long-running sync tasks
- Comprehensive test coverage for background sync behavior
- **#405**: SPEC-20 Simplified Project-Scoped Rclone Sync
([`0b3272a`](https://github.com/basicmachines-co/basic-memory/commit/0b3272a))
- Simplified and more reliable cloud synchronization
- Project-scoped rclone configuration
- Better error handling and status reporting
- **#384**: Streaming Foundation & Async I/O Consolidation (SPEC-19)
([`e78345f`](https://github.com/basicmachines-co/basic-memory/commit/e78345f))
- Foundation for streaming support in future releases
- Consolidated async I/O patterns across codebase
- Improved performance and resource management
- **#364**: Add circuit breaker for file sync failures
([`434cdf2`](https://github.com/basicmachines-co/basic-memory/commit/434cdf2))
- Prevents cascading failures during sync operations
- Automatic recovery from transient errors
- Better resilience in cloud sync scenarios
- **#362**: Add --verbose and --no-gitignore options to cloud upload
([`7f9c1a9`](https://github.com/basicmachines-co/basic-memory/commit/7f9c1a9))
- Enhanced upload control with verbose logging
- Option to bypass gitignore filtering when needed
- Better debugging and troubleshooting capabilities
- **#391**: Add delete_notes parameter to remove project endpoint
([`c9946ec`](https://github.com/basicmachines-co/basic-memory/commit/c9946ec))
- Option to delete notes when removing projects
- Safer project cleanup workflows
- Prevents accidental data loss
### Bug Fixes
- **#420**: Skip archive files during cloud upload
([`49b2adc`](https://github.com/basicmachines-co/basic-memory/commit/49b2adc))
- Prevents uploading of zip, tar, gz and other archive files
- Reduces storage usage and upload time
- Better file filtering during cloud operations
- **#419**: Rename write_note entity_type to note_type for clarity
([`1646572`](https://github.com/basicmachines-co/basic-memory/commit/1646572))
- Clearer parameter naming in write_note tool
- Improved API consistency and documentation
- Better developer experience
- **#418**: Quote string values in YAML frontmatter to handle special characters
([`f0d7398`](https://github.com/basicmachines-co/basic-memory/commit/f0d7398))
- Fixes YAML parsing errors with special characters
- More robust frontmatter handling
- Prevents data corruption in edge cases
- **#415**: Handle dict objects in write_resource endpoint
([`4614fd0`](https://github.com/basicmachines-co/basic-memory/commit/4614fd0))
- Fixes errors when writing dictionary resources
- Better type handling in resource endpoints
- Improved API robustness
- **#414**: Replace Unicode arrows with ASCII for Windows compatibility
([`fc01f6a`](https://github.com/basicmachines-co/basic-memory/commit/fc01f6a))
- Fixes display issues on Windows terminals
- Better cross-platform compatibility
- Improved CLI user experience on Windows
- **#411**: Windows CLI Unicode encoding errors
([`0ba6f21`](https://github.com/basicmachines-co/basic-memory/commit/0ba6f21))
- Resolves Unicode encoding issues on Windows
- Better handling of international characters
- Improved Windows platform support
- **#410**: Various rclone fixes for cloud sync on Windows
([`c9946ec`](https://github.com/basicmachines-co/basic-memory/commit/c9946ec))
- Fixes cloud sync reliability on Windows
- Better path handling for Windows filesystem
- Improved rclone integration on Windows
- **#402**: Normalize YAML frontmatter types to prevent AttributeError
([`a7d7cc5`](https://github.com/basicmachines-co/basic-memory/commit/a7d7cc5))
- Fixes AttributeError when reading frontmatter
- More robust type normalization
- Better error handling in markdown parsing
- **#396**: Strip duplicate headers in edit_note replace_section
([`021af74`](https://github.com/basicmachines-co/basic-memory/commit/021af74))
- Prevents duplicate headers when replacing sections
- Cleaner note editing behavior
- Better content consistency
- **#395**: Simplify search_notes schema by removing Optional wrappers
([`d775f7b`](https://github.com/basicmachines-co/basic-memory/commit/d775f7b))
- Cleaner API schema definition
- Better type safety and validation
- Improved developer experience
- **#394**: Add explicit type annotations to MCP tool parameters
([`581b7b1`](https://github.com/basicmachines-co/basic-memory/commit/581b7b1))
- Better type safety in MCP tools
- Improved IDE support and autocomplete
- Clearer tool documentation
- **#389**: Handle null, empty, and string 'None' title in markdown frontmatter
([`bb8da31`](https://github.com/basicmachines-co/basic-memory/commit/bb8da31))
- Fixes errors with malformed frontmatter titles
- More robust title handling
- Better error recovery
- **#380**: Optimize sync memory usage to prevent OOM on large projects
([`4fd6d0c`](https://github.com/basicmachines-co/basic-memory/commit/4fd6d0c))
- Prevents out-of-memory errors on large knowledge bases
- Better memory management during sync
- Improved scalability
- **#379**: Handle YAML parsing errors gracefully in update_frontmatter
([`32236cd`](https://github.com/basicmachines-co/basic-memory/commit/32236cd))
- Better error handling for malformed YAML
- Graceful degradation instead of crashes
- Improved robustness
- **#377**: Preserve mtime on WebDAV upload
([`e6c8e36`](https://github.com/basicmachines-co/basic-memory/commit/e6c8e36))
- Maintains file modification times during upload
- Better sync accuracy
- Prevents unnecessary re-syncing
- **#370**: Prevent deleted projects from being recreated by background sync
([`449b62d`](https://github.com/basicmachines-co/basic-memory/commit/449b62d))
- Fixes race condition with project deletion
- Better lifecycle management
- Prevents unwanted project recreation
- **#369**: Use filesystem timestamps for entity sync instead of database operation time
([`b7497d7`](https://github.com/basicmachines-co/basic-memory/commit/b7497d7))
- More accurate sync detection
- Better handling of external file modifications
- Improved sync reliability
- **#368**: Handle YAML parsing errors and missing entity_type in markdown files
([`d1431bd`](https://github.com/basicmachines-co/basic-memory/commit/d1431bd))
- Better error handling for malformed markdown
- Graceful handling of missing metadata
- Improved robustness
- **#367**: Resolve UNIQUE constraint violation in entity upsert with observations
([`171bef7`](https://github.com/basicmachines-co/basic-memory/commit/171bef7))
- Fixes database constraint errors during sync
- Better handling of duplicate observations
- Improved data integrity
- **#366**: Terminate sync immediately when project is deleted
([`729a5a3`](https://github.com/basicmachines-co/basic-memory/commit/729a5a3))
- Faster project deletion
- Better resource cleanup
- Improved user experience
- **#357**: Make project creation endpoint idempotent
([`53fb13b`](https://github.com/basicmachines-co/basic-memory/commit/53fb13b))
- Prevents errors when creating existing projects
- Better API reliability
- Improved cloud integration
- **#353**: Handle None text values in Claude conversations importer
([`bd6c834`](https://github.com/basicmachines-co/basic-memory/commit/bd6c834))
- Fixes import errors with empty messages
- Better error handling in importers
- Improved data migration
### Performance Improvements
- Force full database sync after project sync/bisync operations
([`2ad0ee9`](https://github.com/basicmachines-co/basic-memory/commit/2ad0ee9))
- Ensures database consistency after cloud operations
- Better sync reliability
- Improved data integrity
### Documentation
- Add free trial information to README
([`a7d7cc5`](https://github.com/basicmachines-co/basic-memory/commit/a7d7cc5), [`8aaddb6`](https://github.com/basicmachines-co/basic-memory/commit/8aaddb6))
- Updated README with Basic Memory Cloud trial info
- Better onboarding experience
- Clearer pricing information
- Announce Basic Memory Cloud launch in README
([`d756531`](https://github.com/basicmachines-co/basic-memory/commit/d756531))
- Official cloud service announcement
- Updated documentation for cloud features
- Improved product positioning
### Migration Guide
No manual migration required. Upgrade with:
```bash
# Update via uv
uv tool upgrade basic-memory
# Or install fresh
uv tool install basic-memory
```
**What's New in v0.16.0:**
- Streaming foundation and consolidated async I/O (SPEC-19)
- Simplified project-scoped rclone sync (SPEC-20)
- Circuit breaker for sync failure resilience
- Enhanced Windows platform support
- Improved cloud upload with verbose and gitignore options
- Better error handling across YAML parsing and frontmatter
- Memory optimization for large projects
- Archive file filtering during upload
**Breaking Changes:**
- `write_note` parameter renamed: `entity_type``note_type` for clarity
### Installation
```bash
# Latest stable release
uv tool install basic-memory
# Update existing installation
uv tool upgrade basic-memory
# Docker
docker pull ghcr.io/basicmachines-co/basic-memory:v0.16.0
```
## v0.15.2 (2025-10-14)
### Features
+123 -22
View File
@@ -15,10 +15,14 @@ See the [README.md](README.md) file for a project overview.
### Build and Test Commands
- Install: `just install` or `pip install -e ".[dev]"`
- Run all tests (with coverage): `just test` - Runs both unit and integration tests with unified coverage
- Run unit tests only: `just test-unit` - Fast, no coverage
- Run integration tests only: `just test-int` - Fast, no coverage
- Generate HTML coverage: `just coverage` - Opens in browser
- Run all tests (SQLite + Postgres): `just test`
- Run all tests against SQLite: `just test-sqlite`
- Run all tests against Postgres: `just test-postgres` (uses testcontainers)
- Run unit tests (SQLite): `just test-unit-sqlite`
- Run unit tests (Postgres): `just test-unit-postgres`
- Run integration tests (SQLite): `just test-int-sqlite`
- Run integration tests (Postgres): `just test-int-postgres`
- Generate HTML coverage: `just coverage`
- Single test: `pytest tests/path/to/test_file.py::test_function_name`
- Run benchmarks: `pytest test-int/test_sync_performance_benchmark.py -v -m "benchmark and not slow"`
- Lint: `just lint` or `ruff check . --fix`
@@ -30,6 +34,8 @@ See the [README.md](README.md) file for a project overview.
**Note:** Project requires Python 3.12+ (uses type parameter syntax and `type` aliases introduced in 3.12)
**Postgres Testing:** Uses [testcontainers](https://testcontainers-python.readthedocs.io/) which automatically spins up a Postgres instance in Docker. No manual database setup required - just have Docker running.
### Test Structure
- `tests/` - Unit tests for individual components (mocked, fast)
@@ -52,18 +58,97 @@ See the [README.md](README.md) file for a project overview.
- Follow the repository pattern for data access
- Tools communicate to api routers via the httpx ASGI client (in process)
### Code Change Guidelines
- **Full file read before edits**: Before editing any file, read it in full first to ensure complete context; partial reads lead to corrupted edits
- **Minimize diffs**: Prefer the smallest change that satisfies the request. Avoid unrelated refactors or style rewrites unless necessary for correctness
- **No speculative getattr**: Never use `getattr(obj, "attr", default)` when unsure about attribute names. Check the class definition or source code first
- **Fail fast**: Write code with fail-fast logic by default. Do not swallow exceptions with errors or warnings
- **No fallback logic**: Do not add fallback logic unless explicitly told to and agreed with the user
- **No guessing**: Do not say "The issue is..." before you actually know what the issue is. Investigate first.
### Literate Programming Style
Code should tell a story. Comments must explain the "why" and narrative flow, not just the "what".
**Section Headers:**
For files with multiple phases of logic, add section headers so the control flow reads like chapters:
```python
# --- Authentication ---
# ... auth logic ...
# --- Data Validation ---
# ... validation logic ...
# --- Business Logic ---
# ... core logic ...
```
**Decision Point Comments:**
For conditionals that materially change behavior (gates, fallbacks, retries, feature flags), add comments with:
- **Trigger**: what condition causes this branch
- **Why**: the rationale (cost, correctness, UX, determinism)
- **Outcome**: what changes downstream
```python
# Trigger: project has no active sync watcher
# Why: avoid duplicate file system watchers consuming resources
# Outcome: starts new watcher, registers in active_watchers dict
if project_id not in active_watchers:
start_watcher(project_id)
```
**Constraint Comments:**
If code exists because of a constraint (async requirements, rate limits, schema compatibility), explain the constraint near the code:
```python
# SQLite requires WAL mode for concurrent read/write access
connection.execute("PRAGMA journal_mode=WAL")
```
**What NOT to Comment:**
Avoid comments that restate obvious code:
```python
# Bad - restates code
counter += 1 # increment counter
# Good - explains why
counter += 1 # track retries for backoff calculation
```
### Codebase Architecture
See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for detailed architecture documentation.
**Directory Structure:**
- `/alembic` - Alembic db migrations
- `/api` - FastAPI implementation of REST endpoints
- `/cli` - Typer command-line interface
- `/api` - FastAPI REST endpoints + `container.py` composition root
- `/cli` - Typer CLI + `container.py` composition root
- `/deps` - Feature-scoped FastAPI dependencies (config, db, projects, repositories, services, importers)
- `/importers` - Import functionality for Claude, ChatGPT, and other sources
- `/markdown` - Markdown parsing and processing
- `/mcp` - Model Context Protocol server implementation
- `/mcp` - MCP server + `container.py` composition root + `clients/` typed API clients
- `/models` - SQLAlchemy ORM models
- `/repository` - Data access layer
- `/schemas` - Pydantic models for validation
- `/services` - Business logic layer
- `/sync` - File synchronization services
- `/sync` - File synchronization services + `coordinator.py` for lifecycle management
**Composition Roots:**
Each entrypoint (API, MCP, CLI) has a composition root that:
- Reads `ConfigManager` (the only place that reads global config)
- Resolves runtime mode via `RuntimeMode` enum (TEST > CLOUD > LOCAL)
- Provides dependencies to downstream code explicitly
**Typed API Clients (MCP):**
MCP tools use typed clients in `mcp/clients/` to communicate with the API:
- `KnowledgeClient` - Entity CRUD operations
- `SearchClient` - Search operations
- `MemoryClient` - Context building
- `DirectoryClient` - Directory listing
- `ResourceClient` - Resource reading
- `ProjectClient` - Project management
Flow: MCP Tool → Typed Client → HTTP API → Router → Service → Repository
### Development Notes
@@ -76,10 +161,13 @@ See the [README.md](README.md) file for a project overview.
- SQLite is used for indexing and full text search, files are source of truth
- Testing uses pytest with asyncio support (strict mode)
- Unit tests (`tests/`) use mocks when necessary; integration tests (`test-int/`) use real implementations
- Test database uses in-memory SQLite
- Each test runs in a standalone environment with in-memory SQLite and tmp_file directory
- By default, tests run against SQLite (fast, no Docker needed)
- Set `BASIC_MEMORY_TEST_POSTGRES=1` to run against Postgres (uses testcontainers - Docker required)
- Each test runs in a standalone environment with isolated database and tmp_path directory
- CI runs SQLite and Postgres tests in parallel for faster feedback
- Performance benchmarks are in `test-int/test_sync_performance_benchmark.py`
- Use pytest markers: `@pytest.mark.benchmark` for benchmarks, `@pytest.mark.slow` for slow tests
- **Coverage must stay at 100%**: Write tests for new code. Only use `# pragma: no cover` when tests would require excessive mocking (e.g., TYPE_CHECKING blocks, error handlers that need failure injection, runtime-mode-dependent code paths)
### Async Client Pattern (Important!)
@@ -132,22 +220,28 @@ See SPEC-16 for full context manager refactor details.
### Basic Memory Commands
**Local Commands:**
- Sync knowledge: `basic-memory sync` or `basic-memory sync --watch`
- Check sync status: `basic-memory status`
- Import from Claude: `basic-memory import claude conversations`
- Import from ChatGPT: `basic-memory import chatgpt`
- Import from Memory JSON: `basic-memory import memory-json`
- Check sync status: `basic-memory status`
- Tool access: `basic-memory tools` (provides CLI access to MCP tools)
- Guide: `basic-memory tools basic-memory-guide`
- Continue: `basic-memory tools continue-conversation --topic="search"`
- Tool access: `basic-memory tool` (provides CLI access to MCP tools)
- Continue: `basic-memory tool continue-conversation --topic="search"`
**Project Management:**
- List projects: `basic-memory project list`
- Add project: `basic-memory project add "name" ~/path`
- Project info: `basic-memory project info`
- One-way sync (local -> cloud): `basic-memory project sync`
- Bidirectional sync: `basic-memory project bisync`
- Integrity check: `basic-memory project check`
**Cloud Commands (requires subscription):**
- Authenticate: `basic-memory cloud login`
- Logout: `basic-memory cloud logout`
- Bidirectional sync: `basic-memory cloud sync`
- Integrity check: `basic-memory cloud check`
- Mount cloud storage: `basic-memory cloud mount`
- Unmount cloud storage: `basic-memory cloud unmount`
- Check cloud status: `basic-memory cloud status`
- Setup cloud sync: `basic-memory cloud setup`
- Manage snapshots: `basic-memory cloud snapshot [create|list|delete|show|browse]`
- Restore from snapshot: `basic-memory cloud restore <path> --snapshot <id>`
### MCP Capabilities
@@ -174,18 +268,19 @@ See SPEC-16 for full context manager refactor details.
- `list_memory_projects()` - List all available projects with their status
- `create_memory_project(project_name, project_path, set_default)` - Create new Basic Memory projects
- `delete_project(project_name)` - Delete a project from configuration
- `get_current_project()` - Get current project information and stats
- `sync_status()` - Check file synchronization and background operation status
**Visualization:**
- `canvas(nodes, edges, title, folder)` - Generate Obsidian canvas files for knowledge graph visualization
**ChatGPT-Compatible Tools:**
- `search(query)` - Search across knowledge base (OpenAI actions compatible)
- `fetch(id)` - Fetch full content of a search result document
- MCP Prompts for better AI interaction:
- `ai_assistant_guide()` - Guidance on effectively using Basic Memory tools for AI assistants
- `continue_conversation(topic, timeframe)` - Continue previous conversations with relevant historical context
- `search(query, after_date)` - Search with detailed, formatted results for better context understanding
- `recent_activity(timeframe)` - View recently changed items with formatted output
- `json_canvas_spec()` - Full JSON Canvas specification for Obsidian visualization
### Cloud Features (v0.15.0+)
@@ -229,6 +324,11 @@ of using AI just for code generation, we've developed a true collaborative workf
This approach has allowed us to tackle more complex challenges and build a more robust system than either humans or AI
could achieve independently.
**Problem-Solving Guidance:**
- If a solution isn't working after reasonable effort, suggest alternative approaches
- Don't persist with a problematic library or pattern when better alternatives exist
- Example: When py-pglite caused cascading test failures, switching to testcontainers-postgres was the right call
## GitHub Integration
Basic Memory has taken AI-Human collaboration to the next level by integrating Claude directly into the development workflow through GitHub:
@@ -264,5 +364,6 @@ With GitHub integration, the development workflow includes:
2. **Contribution tracking** - All of Claude's contributions are properly attributed in the Git history
3. **Branch management** - Claude can create feature branches for implementations
4. **Documentation maintenance** - Claude can keep documentation updated as the code evolves
5. **Code Commits**: ALWAYS sign off commits with `git commit -s`
This level of integration represents a new paradigm in AI-human collaboration, where the AI assistant becomes a full-fledged team member rather than just a tool for generating code snippets.
+10 -4
View File
@@ -8,8 +8,13 @@ ARG GID=1000
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
# Set environment variables
# UV_PYTHON_INSTALL_DIR ensures Python is installed to a persistent location
# that survives in the final image (not in /root/.local which gets lost)
# UV_PYTHON_PREFERENCE=only-managed tells uv to use its managed Python version
ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1
PYTHONDONTWRITEBYTECODE=1 \
UV_PYTHON_INSTALL_DIR=/python \
UV_PYTHON_PREFERENCE=only-managed
# Create a group and user with the provided UID/GID
# Check if the GID already exists, if not create appgroup
@@ -19,9 +24,10 @@ RUN (getent group ${GID} || groupadd --gid ${GID} appgroup) && \
# Copy the project into the image
ADD . /app
# Sync the project into a new environment, asserting the lockfile is up to date
# Install Python 3.13 explicitly and sync the project
WORKDIR /app
RUN uv sync --locked
RUN uv python install 3.13
RUN uv sync --locked --python 3.13
# Create necessary directories and set ownership
RUN mkdir -p /app/data/basic-memory /app/.basic-memory && \
@@ -43,4 +49,4 @@ HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
CMD basic-memory --version || exit 1
# Use the basic-memory entrypoint to run the MCP server with default SSE transport
CMD ["basic-memory", "mcp", "--transport", "sse", "--host", "0.0.0.0", "--port", "8000"]
CMD ["basic-memory", "mcp", "--transport", "sse", "--host", "0.0.0.0", "--port", "8000"]
+100 -23
View File
@@ -1,3 +1,4 @@
<!-- mcp-name: io.github.basicmachines-co/basic-memory -->
[![License: AGPL v3](https://img.shields.io/badge/License-AGPL_v3-blue.svg)](https://www.gnu.org/licenses/agpl-3.0)
[![PyPI version](https://badge.fury.io/py/basic-memory.svg)](https://badge.fury.io/py/basic-memory)
[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
@@ -5,7 +6,16 @@
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
![](https://badge.mcpx.dev?type=server 'MCP Server')
![](https://badge.mcpx.dev?type=dev 'MCP Dev')
[![smithery badge](https://smithery.ai/badge/@basicmachines-co/basic-memory)](https://smithery.ai/server/@basicmachines-co/basic-memory)
## 🚀 Basic Memory Cloud is Live!
- **Cross-device and multi-platform support is here.** Your knowledge graph now works on desktop, web, and mobile - seamlessly synced across all your AI tools (Claude, ChatGPT, Gemini, Claude Code, and Codex)
- **Early Supporter Pricing:** Early users get 25% off forever.
The open source project continues as always. Cloud just makes it work everywhere.
[Sign up now →](https://basicmemory.com)
with a 7 day free trial
# Basic Memory
@@ -13,8 +23,8 @@ Basic Memory lets you build persistent knowledge through natural conversations w
Claude, while keeping everything in simple Markdown files on your computer. It uses the Model Context Protocol (MCP) to
enable any compatible LLM to read and write to your local knowledge base.
- Website: https://basicmachines.co
- Documentation: https://memory.basicmachines.co
- Website: https://basicmemory.com
- Documentation: https://docs.basicmemory.com
## Pick up your conversation right where you left off
@@ -52,24 +62,6 @@ uv tool install basic-memory
You can view shared context via files in `~/basic-memory` (default directory location).
### Alternative Installation via Smithery
You can use [Smithery](https://smithery.ai/server/@basicmachines-co/basic-memory) to automatically configure Basic
Memory for Claude Desktop:
```bash
npx -y @smithery/cli install @basicmachines-co/basic-memory --client claude
```
This installs and configures Basic Memory without requiring manual edits to the Claude Desktop configuration file. The
Smithery server hosts the MCP server component, while your data remains stored locally as Markdown files.
### Glama.ai
<a href="https://glama.ai/mcp/servers/o90kttu9ym">
<img width="380" height="200" src="https://glama.ai/mcp/servers/o90kttu9ym/badge" alt="basic-memory MCP server" />
</a>
## Why Basic Memory?
Most LLM interactions are ephemeral - you ask a question, get an answer, and everything is forgotten. Each conversation
@@ -415,7 +407,7 @@ canvas(nodes, edges, title, folder) - Generate knowledge visualizations
## Futher info
See the [Documentation](https://memory.basicmachines.co/) for more info, including:
See the [Documentation](https://docs.basicmemory.com) for more info, including:
- [Complete User Guide](https://docs.basicmemory.com/user-guide/)
- [CLI tools](https://docs.basicmemory.com/guides/cli-reference/)
@@ -423,6 +415,91 @@ See the [Documentation](https://memory.basicmachines.co/) for more info, includi
- [Managing multiple Projects](https://docs.basicmemory.com/guides/cli-reference/#project)
- [Importing data from OpenAI/Claude Projects](https://docs.basicmemory.com/guides/cli-reference/#import)
## Logging
Basic Memory uses [Loguru](https://github.com/Delgan/loguru) for logging. The logging behavior varies by entry point:
| Entry Point | Default Behavior | Use Case |
|-------------|------------------|----------|
| CLI commands | File only | Prevents log output from interfering with command output |
| MCP server | File only | Stdout would corrupt the JSON-RPC protocol |
| API server | File (local) or stdout (cloud) | Docker/cloud deployments use stdout |
**Log file location:** `~/.basic-memory/basic-memory.log` (10MB rotation, 10 days retention)
### Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `BASIC_MEMORY_LOG_LEVEL` | `INFO` | Log level: DEBUG, INFO, WARNING, ERROR |
| `BASIC_MEMORY_CLOUD_MODE` | `false` | When `true`, API logs to stdout with structured context |
| `BASIC_MEMORY_ENV` | `dev` | Set to `test` for test mode (stderr only) |
### Examples
```bash
# Enable debug logging
BASIC_MEMORY_LOG_LEVEL=DEBUG basic-memory sync
# View logs
tail -f ~/.basic-memory/basic-memory.log
# Cloud/Docker mode (stdout logging with structured context)
BASIC_MEMORY_CLOUD_MODE=true uvicorn basic_memory.api.app:app
```
## Development
### Running Tests
Basic Memory supports dual database backends (SQLite and Postgres). By default, tests run against SQLite. Set `BASIC_MEMORY_TEST_POSTGRES=1` to run against Postgres (uses testcontainers - Docker required).
**Quick Start:**
```bash
# Run all tests against SQLite (default, fast)
just test-sqlite
# Run all tests against Postgres (uses testcontainers)
just test-postgres
# Run both SQLite and Postgres tests
just test
```
**Available Test Commands:**
- `just test` - Run all tests against both SQLite and Postgres
- `just test-sqlite` - Run all tests against SQLite (fast, no Docker needed)
- `just test-postgres` - Run all tests against Postgres (uses testcontainers)
- `just test-unit-sqlite` - Run unit tests against SQLite
- `just test-unit-postgres` - Run unit tests against Postgres
- `just test-int-sqlite` - Run integration tests against SQLite
- `just test-int-postgres` - Run integration tests against Postgres
- `just test-windows` - Run Windows-specific tests (auto-skips on other platforms)
- `just test-benchmark` - Run performance benchmark tests
**Postgres Testing:**
Postgres tests use [testcontainers](https://testcontainers-python.readthedocs.io/) which automatically spins up a Postgres instance in Docker. No manual database setup required - just have Docker running.
**Test Markers:**
Tests use pytest markers for selective execution:
- `windows` - Windows-specific database optimizations
- `benchmark` - Performance tests (excluded from default runs)
**Other Development Commands:**
```bash
just install # Install with dev dependencies
just lint # Run linting checks
just typecheck # Run type checking
just format # Format code with ruff
just check # Run all quality checks
just migration "msg" # Create database migration
```
See the [justfile](justfile) for the complete list of development commands.
## License
AGPL-3.0
@@ -440,4 +517,4 @@ and submitting PRs.
</picture>
</a>
Built with ♥️ by Basic Machines
Built with ♥️ by Basic Machines
+42
View File
@@ -0,0 +1,42 @@
# Docker Compose configuration for Basic Memory with PostgreSQL
# Use this for local development and testing with Postgres backend
#
# Usage:
# docker-compose -f docker-compose-postgres.yml up -d
# docker-compose -f docker-compose-postgres.yml down
services:
postgres:
image: postgres:17
container_name: basic-memory-postgres
environment:
# Local development/test credentials - NOT for production
# These values are referenced by tests and justfile commands
POSTGRES_DB: basic_memory
POSTGRES_USER: basic_memory_user
POSTGRES_PASSWORD: dev_password # Simple password for local testing only
ports:
- "5433:5432"
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U basic_memory_user -d basic_memory"]
interval: 10s
timeout: 5s
retries: 5
restart: unless-stopped
volumes:
# Named volume for Postgres data
postgres_data:
driver: local
# Named volume for persistent configuration
# Database will be stored in Postgres, not in this volume
basic-memory-config:
driver: local
# Network configuration (optional)
# networks:
# basic-memory-net:
# driver: bridge
+412
View File
@@ -0,0 +1,412 @@
# Basic Memory Architecture
This document describes the architectural patterns and composition structure of Basic Memory.
## Overview
Basic Memory is a local-first knowledge management system with three entrypoints:
- **API** - FastAPI REST server for HTTP access
- **MCP** - Model Context Protocol server for LLM integration
- **CLI** - Typer command-line interface
Each entrypoint uses a **composition root** pattern to manage configuration and dependencies.
## Composition Roots
### What is a Composition Root?
A composition root is the single place in an application where dependencies are wired together. In Basic Memory, each entrypoint has its own composition root that:
1. Reads configuration from `ConfigManager`
2. Resolves runtime mode (cloud/local/test)
3. Creates and provides dependencies to downstream code
**Key principle**: Only composition roots read global configuration. All other modules receive configuration explicitly.
### Container Structure
Each entrypoint has a container dataclass in its package:
```
src/basic_memory/
├── api/
│ └── container.py # ApiContainer
├── mcp/
│ └── container.py # McpContainer
├── cli/
│ └── container.py # CliContainer
└── runtime.py # RuntimeMode enum and resolver
```
### Container Pattern
All containers follow the same structure:
```python
@dataclass
class Container:
config: BasicMemoryConfig
mode: RuntimeMode
@classmethod
def create(cls) -> "Container":
"""Create container by reading ConfigManager."""
config = ConfigManager().config
mode = resolve_runtime_mode(
cloud_mode_enabled=config.cloud_mode_enabled,
is_test_env=config.is_test_env,
)
return cls(config=config, mode=mode)
@property
def some_computed_property(self) -> bool:
"""Derived values based on config and mode."""
return self.mode.is_local and self.config.some_setting
# Module-level singleton
_container: Container | None = None
def get_container() -> Container:
if _container is None:
raise RuntimeError("Container not initialized")
return _container
def set_container(container: Container) -> None:
global _container
_container = container
```
### Runtime Mode Resolution
The `RuntimeMode` enum centralizes mode detection:
```python
class RuntimeMode(Enum):
LOCAL = "local"
CLOUD = "cloud"
TEST = "test"
@property
def is_cloud(self) -> bool:
return self == RuntimeMode.CLOUD
@property
def is_local(self) -> bool:
return self == RuntimeMode.LOCAL
@property
def is_test(self) -> bool:
return self == RuntimeMode.TEST
```
Resolution follows this precedence: **TEST > CLOUD > LOCAL**
```python
def resolve_runtime_mode(cloud_mode_enabled: bool, is_test_env: bool) -> RuntimeMode:
if is_test_env:
return RuntimeMode.TEST
if cloud_mode_enabled:
return RuntimeMode.CLOUD
return RuntimeMode.LOCAL
```
## Dependencies Package
### Structure
The `deps/` package provides FastAPI dependencies organized by feature:
```
src/basic_memory/deps/
├── __init__.py # Re-exports for backwards compatibility
├── config.py # Configuration access
├── db.py # Database/session management
├── projects.py # Project resolution
├── repositories.py # Data access layer
├── services.py # Business logic layer
└── importers.py # Import functionality
```
### Usage in Routers
```python
from basic_memory.deps.services import get_entity_service
from basic_memory.deps.projects import get_project_config
@router.get("/entities/{id}")
async def get_entity(
id: int,
entity_service: EntityService = Depends(get_entity_service),
project: ProjectConfig = Depends(get_project_config),
):
return await entity_service.get(id)
```
### Backwards Compatibility
The old `deps.py` file still exists as a thin re-export shim:
```python
# deps.py - backwards compatibility shim
from basic_memory.deps import *
```
New code should import from specific submodules (`basic_memory.deps.services`) for clarity.
## MCP Tools Architecture
### Typed API Clients
MCP tools communicate with the API through typed clients that encapsulate HTTP paths and response validation:
```
src/basic_memory/mcp/clients/
├── __init__.py # Re-exports all clients
├── base.py # BaseClient with common logic
├── knowledge.py # KnowledgeClient - entity CRUD
├── search.py # SearchClient - search operations
├── memory.py # MemoryClient - context building
├── directory.py # DirectoryClient - directory listing
├── resource.py # ResourceClient - resource reading
└── project.py # ProjectClient - project management
```
### Client Pattern
Each client encapsulates API paths and validates responses:
```python
class KnowledgeClient(BaseClient):
"""Client for knowledge/entity operations."""
async def resolve_entity(self, identifier: str) -> int:
"""Resolve identifier to entity ID."""
response = await call_get(
self.http_client,
f"{self._base_path}/resolve/{identifier}",
)
return int(response.text)
async def get_entity(self, entity_id: int) -> EntityResponse:
"""Get entity by ID."""
response = await call_get(
self.http_client,
f"{self._base_path}/entities/{entity_id}",
)
return EntityResponse.model_validate(response.json())
```
### Tool → Client → API Flow
```
MCP Tool (thin adapter)
Typed Client (encapsulates paths, validates responses)
HTTP API (FastAPI router)
Service Layer (business logic)
Repository Layer (data access)
```
Example tool using typed client:
```python
@mcp.tool()
async def search_notes(query: str, project: str | None = None) -> SearchResponse:
async with get_client() as client:
active_project = await get_active_project(client, project)
# Import client inside function to avoid circular imports
from basic_memory.mcp.clients import SearchClient
search_client = SearchClient(client, active_project.external_id)
return await search_client.search(query)
```
## Sync Coordination
### SyncCoordinator
The `SyncCoordinator` centralizes sync/watch lifecycle management:
```python
@dataclass
class SyncCoordinator:
"""Coordinates file sync and watch operations."""
status: SyncStatus = SyncStatus.NOT_STARTED
sync_task: asyncio.Task | None = None
watch_service: WatchService | None = None
async def start(self, ...):
"""Start sync and watch operations."""
async def stop(self):
"""Stop all sync operations gracefully."""
def get_status_info(self) -> dict:
"""Get current sync status for observability."""
```
### Status Enum
```python
class SyncStatus(Enum):
NOT_STARTED = "not_started"
STARTING = "starting"
RUNNING = "running"
STOPPING = "stopping"
STOPPED = "stopped"
ERROR = "error"
```
## Project Resolution
### ProjectResolver
Unified project selection across all entrypoints:
```python
class ProjectResolver:
"""Resolves which project to use based on context."""
def resolve(
self,
explicit_project: str | None = None,
) -> ResolvedProject:
"""Resolve project using three-tier hierarchy:
1. Explicit project parameter
2. Default project from config
3. Single available project
"""
```
### Resolution Modes
```python
class ResolutionMode(Enum):
EXPLICIT = "explicit" # User specified project
DEFAULT = "default" # Using configured default
SINGLE_PROJECT = "single" # Only one project exists
FALLBACK = "fallback" # Using first available
```
## Testing Patterns
### Container Testing
Each container has corresponding tests:
```
tests/
├── api/test_api_container.py
├── mcp/test_mcp_container.py
└── cli/test_cli_container.py
```
Tests verify:
- Container creation from config
- Runtime mode properties
- Container accessor functions (get/set)
### Mocking Typed Clients
When testing MCP tools, mock at the client level:
```python
def test_search_notes(monkeypatch):
import basic_memory.mcp.clients as clients_mod
class MockSearchClient:
async def search(self, query):
return SearchResponse(results=[...])
monkeypatch.setattr(clients_mod, "SearchClient", MockSearchClient)
```
## Design Principles
### 1. Explicit Dependencies
Modules receive configuration explicitly rather than reading globals:
```python
# Good - explicit injection
async def sync_files(config: BasicMemoryConfig):
...
# Avoid - hidden global access
async def sync_files():
config = ConfigManager().config # Hidden coupling
```
### 2. Single Responsibility
Each layer has a clear responsibility:
- **Containers**: Wire dependencies
- **Clients**: Encapsulate HTTP communication
- **Services**: Business logic
- **Repositories**: Data access
- **Tools/Routers**: Thin adapters
### 3. Deferred Imports
To avoid circular imports, typed clients are imported inside functions:
```python
async def my_tool():
async with get_client() as client:
# Import here to avoid circular dependency
from basic_memory.mcp.clients import KnowledgeClient
knowledge_client = KnowledgeClient(client, project_id)
```
### 4. Backwards Compatibility
When refactoring, maintain backwards compatibility via shims:
```python
# Old module becomes a shim
from basic_memory.new_location import *
# Docstring explains migration path
"""
DEPRECATED: Import from basic_memory.new_location instead.
This shim will be removed in a future version.
"""
```
## File Organization
```
src/basic_memory/
├── api/
│ ├── container.py # API composition root
│ ├── routers/ # FastAPI routers
│ └── ...
├── mcp/
│ ├── container.py # MCP composition root
│ ├── clients/ # Typed API clients
│ ├── tools/ # MCP tool definitions
│ └── server.py # MCP server setup
├── cli/
│ ├── container.py # CLI composition root
│ ├── app.py # Typer app
│ └── commands/ # CLI command groups
├── deps/
│ ├── config.py # Config dependencies
│ ├── db.py # Database dependencies
│ ├── projects.py # Project dependencies
│ ├── repositories.py # Repository dependencies
│ ├── services.py # Service dependencies
│ └── importers.py # Importer dependencies
├── sync/
│ ├── coordinator.py # SyncCoordinator
│ └── ...
├── runtime.py # RuntimeMode resolution
├── project_resolver.py # Unified project selection
└── config.py # Configuration management
```
+7 -26
View File
@@ -77,25 +77,6 @@ SQLite Database (Index)
3. Files are parsed and indexed in SQLite
4. MCP server exposes indexed data to AI
5. AI can query, traverse, and update knowledge graph
### Version 0.15.0 Changes
**Breaking Change: Stateless Architecture**
- All MCP tools now require explicit `project` parameter
- No implicit project context carried between calls
- Exception: `default_project_mode` config option enables fallback
**Three-Tier Project Resolution**:
1. CLI constraint: `--project name` flag (highest priority)
2. Explicit parameter: `project="name"` in tool calls
3. Default mode: `default_project_mode=true` in config (fallback)
**Why This Matters**:
- More predictable behavior across sessions
- Explicit project selection prevents errors
- Multi-project workflows more reliable
- Single-project users can enable default mode for convenience
---
## Project Management
@@ -576,7 +557,7 @@ Adopt GraphQL instead of REST for our API layer.
""",
folder="decisions",
tags=["decision", "api", "graphql"],
entity_type="decision",
note_type="decision",
project="main"
)
```
@@ -613,7 +594,7 @@ await write_note(
""",
folder="meetings",
tags=["meeting", "api", "team"],
entity_type="meeting",
note_type="meeting",
project="main"
)
```
@@ -665,7 +646,7 @@ Specification for user authentication system using JWT tokens.
""",
folder="specs",
tags=["spec", "auth", "security"],
entity_type="spec",
note_type="spec",
project="main"
)
```
@@ -1631,7 +1612,7 @@ await write_note(
{related_entities}
""",
folder="decisions",
entity_type="decision",
note_type="decision",
project="main"
)
```
@@ -2706,14 +2687,14 @@ await write_note(
### Content Management
**write_note(title, content, folder, tags, entity_type, project)**
**write_note(title, content, folder, tags, note_type, project)**
- Create or update markdown notes
- Parameters:
- `title` (required): Note title
- `content` (required): Markdown content
- `folder` (required): Destination folder
- `tags` (optional): List of tags
- `entity_type` (optional): Entity type (note, person, meeting, etc.)
- `note_type` (optional): Type of note (stored in frontmatter). Can be "note", "person", "meeting", "guide", etc.
- `project` (required unless default_project_mode): Target project
- Returns: Created/updated entity with permalink
- Example:
@@ -2723,7 +2704,7 @@ await write_note(
content="# API Design\n...",
folder="specs",
tags=["api", "design"],
entity_type="spec",
note_type="spec",
project="main"
)
```
+565 -569
View File
File diff suppressed because it is too large Load Diff
+28
View File
@@ -0,0 +1,28 @@
## Coverage policy (practical 100%)
Basic Memorys test suite intentionally mixes:
- unit tests (fast, deterministic)
- integration tests (real filesystem + real DB via `test-int/`)
To keep the default CI signal **stable and meaningful**, the default `pytest` coverage report targets **core library logic** and **excludes** a small set of modules that are either:
- highly environment-dependent (OS/DB tuning)
- inherently interactive (CLI)
- background-task orchestration (watchers/sync runners)
### What's excluded (and why)
Coverage excludes are configured in `pyproject.toml` under `[tool.coverage.report].omit`.
Current exclusions include:
- `src/basic_memory/cli/**`: interactive wrappers; behavior is validated via higher-level tests and smoke tests.
- `src/basic_memory/db.py`: platform/backend tuning paths (SQLite/Postgres/Windows), covered by integration tests and targeted runs.
- `src/basic_memory/services/initialization.py`: startup orchestration/background tasks; covered indirectly by app/MCP entrypoints.
- `src/basic_memory/sync/sync_service.py`: heavy filesystem↔DB integration; validated in integration suite (not enforced in unit coverage).
### Recommended additional runs
If you want extra confidence locally/CI:
- **Postgres backend**: run tests with `BASIC_MEMORY_TEST_POSTGRES=1`.
- **Strict backend-complete coverage**: run coverage on SQLite + Postgres and combine the results (recommended).
+142 -27
View File
@@ -2,26 +2,126 @@
# Install dependencies
install:
pip install -e ".[dev]"
uv pip install -e ".[dev]"
uv sync
@echo ""
@echo "💡 Remember to activate the virtual environment by running: source .venv/bin/activate"
# Run unit tests only (fast, no coverage)
test-unit:
uv run pytest -p pytest_mock -v --no-cov -n auto tests
# ==============================================================================
# DATABASE BACKEND TESTING
# ==============================================================================
# Basic Memory supports dual database backends (SQLite and Postgres).
# By default, tests run against SQLite (fast, no dependencies).
# Set BASIC_MEMORY_TEST_POSTGRES=1 to run against Postgres (uses testcontainers).
#
# Quick Start:
# just test # Run all tests against SQLite (default)
# just test-sqlite # Run all tests against SQLite
# just test-postgres # Run all tests against Postgres (testcontainers)
# just test-unit-sqlite # Run unit tests against SQLite
# just test-unit-postgres # Run unit tests against Postgres
# just test-int-sqlite # Run integration tests against SQLite
# just test-int-postgres # Run integration tests against Postgres
#
# CI runs both in parallel for faster feedback.
# ==============================================================================
# Run integration tests only (fast, no coverage)
test-int:
uv run pytest -p pytest_mock -v --no-cov -n auto test-int
# Run all tests against SQLite and Postgres
test: test-sqlite test-postgres
# Run all tests with unified coverage report
test: test-unit test-int
# Run all tests against SQLite
test-sqlite: test-unit-sqlite test-int-sqlite
# Run all tests against Postgres (uses testcontainers)
test-postgres: test-unit-postgres test-int-postgres
# Run unit tests against SQLite
test-unit-sqlite:
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov tests
# Run unit tests against Postgres
test-unit-postgres:
BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov tests
# Run integration tests against SQLite
test-int-sqlite:
uv run pytest -p pytest_mock -v --no-cov test-int
# Run integration tests against Postgres
# Note: Uses timeout due to FastMCP Client + asyncpg cleanup hang (tests pass, process hangs on exit)
# See: https://github.com/jlowin/fastmcp/issues/1311
test-int-postgres:
#!/usr/bin/env bash
set -euo pipefail
# Use gtimeout (macOS/Homebrew) or timeout (Linux)
TIMEOUT_CMD=$(command -v gtimeout || command -v timeout || echo "")
if [[ -n "$TIMEOUT_CMD" ]]; then
$TIMEOUT_CMD --signal=KILL 600 bash -c 'BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov test-int' || test $? -eq 137
else
echo "⚠️ No timeout command found, running without timeout..."
BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov test-int
fi
# Reset Postgres test database (drops and recreates schema)
# Useful when Alembic migration state gets out of sync during development
# Uses credentials from docker-compose-postgres.yml
postgres-reset:
docker exec basic-memory-postgres psql -U ${POSTGRES_USER:-basic_memory_user} -d ${POSTGRES_TEST_DB:-basic_memory_test} -c "DROP SCHEMA public CASCADE; CREATE SCHEMA public;"
@echo "✅ Postgres test database reset"
# Run Alembic migrations manually against Postgres test database
# Useful for debugging migration issues
# Uses credentials from docker-compose-postgres.yml (can override with env vars)
postgres-migrate:
@cd src/basic_memory/alembic && \
BASIC_MEMORY_DATABASE_BACKEND=postgres \
BASIC_MEMORY_DATABASE_URL=${POSTGRES_TEST_URL:-postgresql+asyncpg://basic_memory_user:dev_password@localhost:5433/basic_memory_test} \
uv run alembic upgrade head
@echo "✅ Migrations applied to Postgres test database"
# Run Windows-specific tests only (only works on Windows platform)
# These tests verify Windows-specific database optimizations (locking mode, NullPool)
# Will be skipped automatically on non-Windows platforms
test-windows:
uv run pytest -p pytest_mock -v --no-cov -m windows tests test-int
# Run benchmark tests only (performance testing)
# These are slow tests that measure sync performance with various file counts
# Excluded from default test runs to keep CI fast
test-benchmark:
uv run pytest -p pytest_mock -v --no-cov -m benchmark tests test-int
# Run all tests including Windows, Postgres, and Benchmarks (for CI/comprehensive testing)
# Use this before releasing to ensure everything works across all backends and platforms
test-all:
uv run pytest -p pytest_mock -v --no-cov tests test-int
# Generate HTML coverage report
coverage:
uv run pytest -p pytest_mock -v -n auto tests test-int --cov-report=html
@echo "Coverage report generated in htmlcov/index.html"
#!/usr/bin/env bash
set -euo pipefail
uv run coverage erase
echo "🔎 Coverage (SQLite)..."
BASIC_MEMORY_ENV=test uv run coverage run --source=basic_memory -m pytest -p pytest_mock -v --no-cov tests test-int
echo "🔎 Coverage (Postgres via testcontainers)..."
# Note: Uses timeout due to FastMCP Client + asyncpg cleanup hang (tests pass, process hangs on exit)
# See: https://github.com/jlowin/fastmcp/issues/1311
TIMEOUT_CMD=$(command -v gtimeout || command -v timeout || echo "")
if [[ -n "$TIMEOUT_CMD" ]]; then
$TIMEOUT_CMD --signal=KILL 600 bash -c 'BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 uv run coverage run --source=basic_memory -m pytest -p pytest_mock -v --no-cov -m postgres tests test-int' || test $? -eq 137
else
echo "⚠️ No timeout command found, running without timeout..."
BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 uv run coverage run --source=basic_memory -m pytest -p pytest_mock -v --no-cov -m postgres tests test-int
fi
echo "🧩 Combining coverage data..."
uv run coverage combine
uv run coverage report -m
uv run coverage html
echo "Coverage report generated in htmlcov/index.html"
# Lint and fix code (calls fix)
lint: fix
@@ -49,14 +149,6 @@ format:
run-inspector:
npx @modelcontextprotocol/inspector
# Build macOS installer
installer-mac:
cd installer && chmod +x make_icons.sh && ./make_icons.sh
cd installer && uv run python setup.py bdist_mac
# Build Windows installer
installer-win:
cd installer && uv run python setup.py bdist_win32
# Update all dependencies to latest versions
update-deps:
@@ -104,16 +196,22 @@ release version:
fi
# Run quality checks
echo "🔍 Running quality checks..."
just check
echo "🔍 Running lint checks..."
just lint
just typecheck
# Update version in __init__.py
echo "📝 Updating version in __init__.py..."
sed -i.bak "s/__version__ = \".*\"/__version__ = \"$VERSION_NUM\"/" src/basic_memory/__init__.py
rm -f src/basic_memory/__init__.py.bak
# Update version in server.json (MCP registry metadata)
echo "📝 Updating version in server.json..."
sed -i.bak "s/\"version\": \"[^\"]*\"/\"version\": \"$VERSION_NUM\"/g" server.json
rm -f server.json.bak
# Commit version update
git add src/basic_memory/__init__.py
git add src/basic_memory/__init__.py server.json
git commit -m "chore: update version to $VERSION_NUM for {{version}} release"
# Create and push tag
@@ -127,6 +225,12 @@ release version:
echo "✅ Release {{version}} created successfully!"
echo "📦 GitHub Actions will build and publish to PyPI"
echo "🔗 Monitor at: https://github.com/basicmachines-co/basic-memory/actions"
echo ""
echo "📝 REMINDER: Post-release tasks:"
echo " 1. docs.basicmemory.com - Add release notes to src/pages/latest-releases.mdx"
echo " 2. basicmachines.co - Update version in src/components/sections/hero.tsx"
echo " 3. MCP Registry - Run: mcp-publisher publish"
echo " See: .claude/commands/release/release.md for detailed instructions"
# Create a beta release (e.g., just beta v0.13.2b1)
beta version:
@@ -163,16 +267,22 @@ beta version:
fi
# Run quality checks
echo "🔍 Running quality checks..."
just check
echo "🔍 Running lint checks..."
just lint
just typecheck
# Update version in __init__.py
echo "📝 Updating version in __init__.py..."
sed -i.bak "s/__version__ = \".*\"/__version__ = \"$VERSION_NUM\"/" src/basic_memory/__init__.py
rm -f src/basic_memory/__init__.py.bak
# Update version in server.json (MCP registry metadata)
echo "📝 Updating version in server.json..."
sed -i.bak "s/\"version\": \"[^\"]*\"/\"version\": \"$VERSION_NUM\"/g" server.json
rm -f server.json.bak
# Commit version update
git add src/basic_memory/__init__.py
git add src/basic_memory/__init__.py server.json
git commit -m "chore: update version to $VERSION_NUM for {{version}} beta release"
# Create and push tag
@@ -187,6 +297,11 @@ beta version:
echo "📦 GitHub Actions will build and publish to PyPI as pre-release"
echo "🔗 Monitor at: https://github.com/basicmachines-co/basic-memory/actions"
echo "📥 Install with: uv tool install basic-memory --pre"
echo ""
echo "📝 REMINDER: For stable releases, update documentation sites:"
echo " 1. docs.basicmemory.com - Add release notes to src/pages/latest-releases.mdx"
echo " 2. basicmachines.co - Update version in src/components/sections/hero.tsx"
echo " See: .claude/commands/release/release.md for detailed instructions"
# List all available recipes
default:
+25 -11
View File
@@ -14,9 +14,8 @@ dependencies = [
"typer>=0.9.0",
"aiosqlite>=0.20.0",
"greenlet>=3.1.1",
"pydantic[email,timezone]>=2.10.3",
"icecream>=2.1.3",
"mcp>=1.2.0",
"pydantic[email,timezone]>=2.12.0",
"mcp>=1.23.1",
"pydantic-settings>=2.6.1",
"loguru>=0.7.3",
"pyright>=1.1.390",
@@ -30,11 +29,21 @@ dependencies = [
"alembic>=1.14.1",
"pillow>=11.1.0",
"pybars3>=0.9.7",
"fastmcp>=2.10.2",
"fastmcp==2.12.3", # Pinned - 2.14.x breaks MCP tools visibility (issue #463)
"pyjwt>=2.10.1",
"python-dotenv>=1.1.0",
"pytest-aio>=1.9.0",
"aiofiles>=24.1.0", # Async file I/O
"aiofiles>=24.1.0", # Optional observability (disabled by default via config)
"asyncpg>=0.30.0",
"nest-asyncio>=1.6.0", # For Alembic migrations with Postgres
"pytest-asyncio>=1.2.0",
"psycopg==3.3.1",
"mdformat>=0.7.22",
"mdformat-gfm>=0.3.7",
"mdformat-frontmatter>=2.0.8",
"sniffio>=1.3.1",
"anyio>=4.10.0",
"httpx>=0.28.0",
]
@@ -60,6 +69,8 @@ asyncio_default_fixture_loop_scope = "function"
markers = [
"benchmark: Performance benchmark tests (deselect with '-m \"not benchmark\"')",
"slow: Slow-running tests (deselect with '-m \"not slow\"')",
"postgres: Tests that run against Postgres backend (deselect with '-m \"not postgres\"')",
"windows: Windows-specific tests (deselect with '-m \"not windows\"')",
]
[tool.ruff]
@@ -77,6 +88,9 @@ dev = [
"pytest-xdist>=3.0.0",
"ruff>=0.1.6",
"freezegun>=1.5.5",
"testcontainers[postgres]>=4.0.0",
"psycopg>=3.2.0",
"pyright>=1.1.408",
]
[tool.hatch.version]
@@ -101,6 +115,8 @@ pythonVersion = "3.12"
[tool.coverage.run]
concurrency = ["thread", "gevent"]
parallel = true
source = ["basic_memory"]
[tool.coverage.report]
exclude_lines = [
@@ -122,11 +138,9 @@ omit = [
"*/supabase_auth_provider.py", # External HTTP calls to Supabase APIs
"*/watch_service.py", # File system watching - complex integration testing
"*/background_sync.py", # Background processes
"*/cli/main.py", # CLI entry point
"*/mcp/tools/project_management.py", # Covered by integration tests
"*/mcp/tools/sync_status.py", # Covered by integration tests
"*/cli/**", # CLI is an interactive wrapper; core logic is covered via API/MCP/service tests
"*/db.py", # Backend/runtime-dependent (sqlite/postgres/windows tuning); validated via integration tests
"*/services/initialization.py", # Startup orchestration + background tasks (watchers); exercised indirectly in entrypoints
"*/sync/sync_service.py", # Heavy filesystem/db integration; covered by integration suite, not enforced in unit coverage
"*/services/migration_service.py", # Complex migration scenarios
]
[tool.logfire]
ignore_no_config = true
+25
View File
@@ -0,0 +1,25 @@
{
"$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json",
"name": "io.github.basicmachines-co/basic-memory",
"description": "Local-first knowledge management with bi-directional LLM sync via Markdown files.",
"repository": {
"url": "https://github.com/basicmachines-co/basic-memory.git",
"source": "github"
},
"version": "0.17.8",
"packages": [
{
"registryType": "pypi",
"identifier": "basic-memory",
"version": "0.17.8",
"runtimeHint": "uvx",
"runtimeArguments": [
{"type": "positional", "value": "basic-memory"},
{"type": "positional", "value": "mcp"}
],
"transport": {
"type": "stdio"
}
}
]
}
@@ -1,156 +0,0 @@
---
title: 'SPEC-1: Specification-Driven Development Process'
type: spec
permalink: specs/spec-1-specification-driven-development-process
tags:
- process
- specification
- development
- meta
---
# SPEC-1: Specification-Driven Development Process
## Why
We're implementing specification-driven development to solve the complexity and circular refactoring issues in our web development process.
Instead of getting lost in framework details and type gymnastics, we start with clear specifications that drive implementation.
The default approach of adhoc development with AI agents tends to result in:
- Circular refactoring cycles
- Fighting framework complexity
- Lost context between sessions
- Unclear requirements and scope
## What
This spec defines our process for using basic-memory as the specification engine to build basic-memory-cloud.
We're creating a recursive development pattern where basic-memory manages the specs that drive the development of basic-memory-cloud.
**Affected Areas:**
- All future component development
- Architecture decisions
- Agent collaboration workflows
- Knowledge management and context preservation
## How (High Level)
### Specification Structure
Name: Spec names should be numbered sequentially, followed by a description eg. `SPEC-X - Simple Description.md`.
See: [[Spec-2: Slash Commands Reference]]
Every spec is a complete thought containing:
- **Why**: The reasoning and problem being solved
- **What**: What is affected or changed
- **How**: High-level approach to implementation
- **How to Evaluate**: Testing/validation procedure
- Additional context as needed
### Living Specification Format
Specifications are **living documents** that evolve throughout implementation:
**Progress Tracking:**
- **Completed items**: Use ✅ checkmark emoji for implemented features
- **Pending items**: Use `- [ ]` GitHub-style checkboxes for remaining tasks
- **In-progress items**: Use `- [x]` when work is actively underway
**Status Philosophy:**
- **Avoid static status headers** like "COMPLETE" or "IN PROGRESS" that become stale
- **Use checklists within content** to show granular implementation progress
- **Keep specs informative** while providing clear progress visibility
- **Update continuously** as understanding and implementation evolve
**Example Format:**
```markdown
### ComponentName
- ✅ Basic functionality implemented
- ✅ Props and events defined
- - [ ] Add sorting controls
- - [ ] Improve accessibility
- - [x] Currently implementing responsive design
```
This creates **git-friendly progress tracking** where `[ ]` easily becomes `[x]` or ✅ when completed, and specs remain valuable throughout the development lifecycle.
## Claude Code
We will leverage Claude Code capabilities to make the process semi-automated.
- Slash commands: define repeatable steps in the process (create spec, implement, review, etc)
- Agents: define roles to carry out instructions (front end developer, baskend developer, etc)
- MCP tools: enable agents to implement specs via actions (write code, test, etc)
### Workflow
1. **Create**: Write spec as complete thought in `/specs` folder
2. **Discuss**: Iterate and refine through agent collaboration
3. **Implement**: Hand spec to appropriate specialist agent
4. **Validate**: Review implementation against spec criteria
5. **Document**: Update spec with learnings and decisions
### Slash Commands
Claude slash commands are used to manage the flow.
These are simple instructions to help make the process uniform.
They can be updated and refined as needed.
- `/spec create [name]` - Create new specification
- `/spec status` - Show current spec states
- `/spec implement [name]` - Hand to appropriate agent
- `/spec review [name]` - Validate implementation
### Agent Orchestration
Agents are defined with clear roles, for instance:
- **system-architect**: Creates high-level specs, ADRs, architectural decisions
- **vue-developer**: Component specs, UI patterns, frontend architecture
- **python-developer**: Implementation specs, technical details, backend logic
-
- Each agent reads/updates specs through basic-memory tools.
## How to Evaluate
### Success Criteria
- Specs provide clear, actionable guidance for implementation
- Reduced circular refactoring and scope creep
- Persistent context across development sessions
- Clean separation between "what/why" and implementation details
- Specs record a history of what happened and why for historical context
### Testing Procedure
1. Create a spec for an existing problematic component
2. Have an agent implement following only the spec
3. Compare result quality and development speed vs. ad-hoc approach
4. Measure context preservation across sessions
5. Evaluate spec clarity and completeness
### Metrics
- Time from spec to working implementation
- Number of refactoring cycles required
- Agent understanding of requirements
- Spec reusability for similar components
## Notes
- Start simple: specs are just complete thoughts, not heavy processes
- Use basic-memory's knowledge graph to link specs, decisions, components
- Let the process evolve naturally based on what works
- Focus on solving the actual problem: Manage complexity in development
## Observations
- [problem] Web development without clear goals and documentation circular refactoring cycles #complexity
- [solution] Specification-driven development reduces scope creep and context loss #process-improvement
- [pattern] basic-memory as specification engine creates recursive development loop #meta-development
- [workflow] Five-step process: Create → Discuss → Implement → Validate → Document #methodology
- [tool] Slash commands provide uniform process automation #automation
- [agent-pattern] Three specialized agents handle different implementation domains #specialization
- [success-metric] Time from spec to working implementation measures process efficiency #measurement
- [learning] Process should evolve naturally based on what works in practice #adaptation
- [format] Living specifications use checklists for progress tracking instead of static status headers #documentation
- [evolution] Specs evolve throughout implementation maintaining value as working documents #continuous-improvement
## Relations
- spec [[Spec-2: Slash Commands Reference]]
- spec [[Spec-3: Agent Definitions]]
@@ -1,245 +0,0 @@
---
title: 'SPEC-11: Basic Memory API Performance Optimization'
type: spec
permalink: specs/spec-11-basic-memory-api-performance-optimization
tags:
- performance
- api
- mcp
- database
- cloud
---
# SPEC-11: Basic Memory API Performance Optimization
## Why
The Basic Memory API experiences significant performance issues in cloud environments due to expensive per-request initialization. MCP tools making
HTTP requests to the API suffer from 350ms-2.6s latency overhead **before** any actual operation occurs.
**Root Cause Analysis:**
- GitHub Issue #82 shows repeated initialization sequences in logs (16:29:35 and 16:49:58)
- Each MCP tool call triggers full database initialization + project reconciliation
- `get_engine_factory()` dependency calls `db.get_or_create_db()` on every request
- `reconcile_projects_with_config()` runs expensive sync operations repeatedly
**Performance Impact:**
- Database connection setup: ~50-100ms per request
- Migration checks: ~100-500ms per request
- Project reconciliation: ~200ms-2s per request
- **Total overhead**: ~350ms-2.6s per MCP tool call
This creates compounding effects with tenant auto-start delays and increases timeout risk in cloud deployments.
Github issue: https://github.com/basicmachines-co/basic-memory-cloud/issues/82
## What
This optimization affects the **core basic-memory repository** components:
1. **API Lifespan Management** (`src/basic_memory/api/app.py`)
- Cache database connections in app state during startup
- Avoid repeated expensive initialization
2. **Dependency Injection** (`src/basic_memory/deps.py`)
- Modify `get_engine_factory()` to use cached connections
- Eliminate per-request database setup
3. **Initialization Service** (`src/basic_memory/services/initialization.py`)
- Add caching/throttling to project reconciliation
- Skip expensive operations when appropriate
4. **Configuration** (`src/basic_memory/config.py`)
- Add optional performance flags for cloud environments
**Backwards Compatibility**: All changes must be backwards compatible with existing CLI and non-cloud usage.
## How (High Level)
### Phase 1: Cache Database Connections (Critical - 80% of gains)
**Problem**: `get_engine_factory()` calls `db.get_or_create_db()` per request
**Solution**: Cache database engine/session in app state during lifespan
1. **Modify API Lifespan** (`api/app.py`):
```python
@asynccontextmanager
async def lifespan(app: FastAPI):
app_config = ConfigManager().config
await initialize_app(app_config)
# Cache database connection in app state
engine, session_maker = await db.get_or_create_db(app_config.database_path)
app.state.engine = engine
app.state.session_maker = session_maker
# ... rest of startup logic
```
2. Modify Dependency Injection (deps.py):
```python
async def get_engine_factory(
request: Request
) -> tuple[AsyncEngine, async_sessionmaker[AsyncSession]]:
"""Get cached engine and session maker from app state."""
return request.app.state.engine, request.app.state.session_maker
```
Phase 2: Optimize Project Reconciliation (Secondary - 20% of gains)
Problem: reconcile_projects_with_config() runs expensive sync repeatedly
Solution: Add module-level caching with time-based throttling
1. Add Reconciliation Cache (services/initialization.py):
```ptyhon
_project_reconciliation_completed = False
_last_reconciliation_time = 0
async def reconcile_projects_with_config(app_config, force=False):
# Skip if recently completed (within 60 seconds) unless forced
if recently_completed and not force:
return
# ... existing logic
```
Phase 3: Cloud Environment Flags (Optional)
Problem: Force expensive initialization in production environments
Solution: Add skip flags for cloud/stateless deployments
1. Add Config Flag (config.py):
skip_initialization_sync: bool = Field(default=False)
2. Configure in Cloud (basic-memory-cloud integration):
BASIC_MEMORY_SKIP_INITIALIZATION_SYNC=true
How to Evaluate
Success Criteria
1. Performance Metrics (Primary):
- MCP tool response time reduced by 50%+ (measure before/after)
- Database connection overhead eliminated (0ms vs 50-100ms)
- Migration check overhead eliminated (0ms vs 100-500ms)
- Project reconciliation overhead reduced by 90%+
2. Load Testing:
- Concurrent MCP tool calls maintain performance
- No memory leaks in cached connections
- Database connection pool behaves correctly
3. Functional Correctness:
- All existing API endpoints work identically
- MCP tools maintain full functionality
- CLI operations unaffected
- Database migrations still execute properly
4. Backwards Compatibility:
- No breaking changes to existing APIs
- Config changes are optional with safe defaults
- Non-cloud deployments work unchanged
Testing Strategy
Performance Testing:
# Before optimization
time basic-memory-mcp-tools write_note "test" "content" "folder"
# Measure: ~1-3 seconds
# After optimization
time basic-memory-mcp-tools write_note "test" "content" "folder"
# Target: <500ms
Load Testing:
# Multiple concurrent MCP tool calls
for i in {1..10}; do
basic-memory-mcp-tools search "test" &
done
wait
# Verify: No degradation, consistent response times
Regression Testing:
# Full basic-memory test suite
just test
# All tests must pass
# Integration tests with cloud deployment
# Verify MCP gateway → API → database flow works
Validation Checklist
- Phase 1 Complete: Database connections cached, dependency injection optimized
- Performance Benchmark: 50%+ improvement in MCP tool response times
- Memory Usage: No leaks in cached connections over 24h+ periods
- Stress Testing: 100+ concurrent requests maintain performance
- Backwards Compatibility: All existing functionality preserved
- Documentation: Performance optimization documented in README
- Cloud Integration: basic-memory-cloud sees performance benefits
## Implementation Status ✅ COMPLETED
**Implementation Date**: 2025-09-26
**Branch**: `feature/spec-11-api-performance-optimization`
**Commit**: `771f60b`
### ✅ Phase 1: Database Connection Caching - IMPLEMENTED
**Files Modified:**
- `src/basic_memory/api/app.py` - Added database connection caching in app.state
- `src/basic_memory/deps.py` - Updated get_engine_factory() to use cached connections
- `src/basic_memory/config.py` - Added skip_initialization_sync configuration flag
**Implementation Details:**
1. **API Lifespan Caching**: Database engine and session_maker cached in app.state during startup
2. **Dependency Injection Optimization**: get_engine_factory() now returns cached connections instead of calling get_or_create_db()
3. **Project Reconciliation Removal**: Eliminated expensive reconcile_projects_with_config() from API startup
4. **CLI Fallback Preserved**: Non-API contexts continue to work with fallback database initialization
### ✅ Performance Validation - ACHIEVED
**Live Testing Results** (2025-09-26 14:03-14:09):
| Operation | Before | After | Improvement |
|-----------|--------|-------|-------------|
| `read_note` | 350ms-2.6s | **20ms** | **95-99% faster** |
| `edit_note` | 350ms-2.6s | **218ms** | **75-92% faster** |
| `search_notes` | 350ms-2.6s | **<500ms** | **Responsive** |
| `list_memory_projects` | N/A | **<100ms** | **Fast** |
**Key Achievements:**
-**95-99% improvement** in read operations (primary workflow)
-**75-92% improvement** in edit operations
-**Zero overhead** for project switching
-**Database connection overhead eliminated** (0ms vs 50-100ms)
-**Project reconciliation delays removed** from API requests
-**<500ms target achieved** for all operations except write (which includes file sync)
### ✅ Backwards Compatibility - MAINTAINED
- All existing functionality preserved
- CLI operations unaffected
- Fallback for non-API contexts maintained
- No breaking changes to existing APIs
- Optional configuration with safe defaults
### ✅ Testing Validation - PASSED
- Integration tests passing
- Type checking clear
- Linting checks passed
- Live testing with real MCP tools successful
- Multi-project workflows validated
- Rapid project switching validated
## Notes
Implementation Priority:
- ✅ Phase 1 COMPLETED: Database connection caching provides 95%+ performance gains
- ⚪ Phase 2 NOT NEEDED: Project reconciliation removal achieved the goals
- ⚪ Phase 3 INCLUDED: skip_initialization_sync flag added
Risk Mitigation:
- ✅ All changes backwards compatible implemented
- ✅ Gradual implementation successful (Phase 1 → validation)
- ✅ Easy rollback via configuration flags available
Cloud Integration:
- ✅ This optimization directly addresses basic-memory-cloud issue #82
- ✅ Changes in core basic-memory will benefit all cloud tenants
- ✅ No changes needed in basic-memory-cloud itself
**Result**: SPEC-11 performance optimizations successfully implemented and validated. The 95-99% improvement in MCP tool response times exceeds the original 50-80% target, providing exceptional performance gains for cloud deployments and local usage.
File diff suppressed because it is too large Load Diff
@@ -1,210 +0,0 @@
---
title: 'SPEC-14: Cloud Git Versioning & GitHub Backup'
type: spec
permalink: specs/spec-14-cloud-git-versioning
tags:
- git
- github
- backup
- versioning
- cloud
related:
- specs/spec-9-multi-project-bisync
- specs/spec-9-follow-ups-conflict-sync-and-observability
status: deferred
---
# SPEC-14: Cloud Git Versioning & GitHub Backup
**Status: DEFERRED** - Postponed until multi-user/teams feature development. Using S3 versioning (SPEC-9.1) for v1 instead.
## Why Deferred
**Original goals can be met with simpler solutions:**
- Version history → **S3 bucket versioning** (automatic, zero config)
- Offsite backup → **Tigris global replication** (built-in)
- Restore capability → **S3 version restore** (`bm cloud restore --version-id`)
- Collaboration → **Deferred to teams/multi-user feature** (not v1 requirement)
**Complexity vs value trade-off:**
- Git integration adds: committer service, puller service, webhooks, LFS, merge conflicts
- Risk: Loop detection between Git ↔ rclone bisync ↔ local edits
- S3 versioning gives 80% of value with 5% of complexity
**When to revisit:**
- Teams/multi-user features (PR-based collaboration workflow)
- User requests for commit messages and branch-based workflows
- Need for fine-grained audit trail beyond S3 object metadata
---
## Original Specification (for reference)
## Why
Early access users want **transparent version history**, easy **offsite backup**, and a familiar **restore/branching** workflow. Git/GitHub integration would provide:
- Auditable history of every change (who/when/why)
- Branches/PRs for review and collaboration
- Offsite private backup under the user's control
- Escape hatch: users can always `git clone` their knowledge base
**Note:** These goals are now addressed via S3 versioning (SPEC-9.1) for single-user use case.
## Goals
- **Transparent**: Users keep using Basic Memory; Git runs behind the scenes.
- **Private**: Push to a **private GitHub repo** that the user owns (or tenant org).
- **Reliable**: No data loss, deterministic mapping of filesystem ↔ Git.
- **Composable**: Plays nicely with SPEC9 bisync and upcoming conflict features (SPEC9 FollowUps).
**NonGoals (for v1):**
- Finegrained perfile encryption in Git history (can be layered later).
- Large media optimization beyond Git LFS defaults.
## User Stories
1. *As a user*, I connect my GitHub and choose a private backup repo.
2. *As a user*, every change I make in cloud (or via bisync) is **committed** and **pushed** automatically.
3. *As a user*, I can **restore** a file/folder/project to a prior version.
4. *As a power user*, I can **git pull/push** directly to collaborate outside the app.
5. *As an admin*, I can enforce repo ownership (tenant org) and leastprivilege scopes.
## Scope
- **In scope:** Full repo backup of `/app/data/` (all projects) with optional selective subpaths.
- **Out of scope (v1):** Partial shallow mirrors; encrypted Git; crossprovider SCM (GitLab/Bitbucket).
## Architecture
### Topology
- **Authoritative working tree**: `/app/data/` (bucket mount) remains the source of truth (SPEC9).
- **Bare repo** lives alongside: `/app/git/${tenant}/knowledge.git` (serverside).
- **Mirror remote**: `github.com/<owner>/<repo>.git` (private).
```mermaid
flowchart LR
A[/Users & Agents/] -->|writes/edits| B[/app/data/]
B -->|file events| C[Committer Service]
C -->|git commit| D[(Bare Repo)]
D -->|push| E[(GitHub Private Repo)]
E -->|webhook (push)| F[Puller Service]
F -->|git pull/merge| D
D -->|checkout/merge| B
```
### Services
- **Committer Service** (daemon):
- Watches `/app/data/` for changes (inotify/poll)
- Batches changes (debounce e.g. 25s)
- Writes `.bmmeta` (if present) into commit message trailer (see FollowUps)
- `git add -A && git commit -m "chore(sync): <summary>
BM-Meta: <json>"`
- Periodic `git push` to GitHub mirror (configurable interval)
- **Puller Service** (webhook target):
- Receives GitHub webhook (push) → `git fetch`
- **Fastforward** merges to `main` only; reject nonFF unless policy allows
- Applies changes back to `/app/data/` via clean checkout
- Emits sync events for Basic Memory indexers
### Auth & Security
- **GitHub App** (recommended): minimal scopes: `contents:read/write`, `metadata:read`, webhook.
- Tenantscoped installation; repo created in user account or tenant org.
- Tokens stored in KMS/secret manager; rotated automatically.
- Optional policy: allow only **FF merges** on `main`; nonFF requires PR.
### Repo Layout
- **Monorepo** (default): one repo per tenant mirrors `/app/data/` with subfolders per project.
- Optional multirepo mode (later): one repo per project.
### File Handling
- Honor `.gitignore` generated from `.bmignore.rclone` + BM defaults (cache, temp, state).
- **Git LFS** for large binaries (images, media) — auto track by extension/size threshold.
- Normalize newline + Unicode (aligns with FollowUps).
### Conflict Model
- **Primary concurrency**: SPEC9 FollowUps (`.bmmeta`, conflict copies) stays the first line of defense.
- **Git merges** are a **secondary** mechanism:
- Server only automerges **text** conflicts when trivial (FF or clean 3way).
- Otherwise, create `name (conflict from <branch>, <ts>).md` and surface via events.
### Data Flow vs Bisync
- Bisync (rclone) continues between local sync dir ↔ bucket.
- Git sits **cloudside** between bucket and GitHub.
- On **pull** from GitHub → files written to `/app/data/` → picked up by indexers & eventually by bisync back to users.
## CLI & UX
New commands (cloud mode):
- `bm cloud git connect` — Launch GitHub App installation; create private repo; store installation id.
- `bm cloud git status` — Show connected repo, last push time, last webhook delivery, pending commits.
- `bm cloud git push` — Manual push (rarely needed).
- `bm cloud git pull` — Manual pull/FF (admin only by default).
- `bm cloud snapshot -m "message"` — Create a tagged pointintime snapshot (git tag).
- `bm restore <path> --to <commit|tag>` — Restore file/folder/project to prior version.
Settings:
- `bm config set git.autoPushInterval=5s`
- `bm config set git.lfs.sizeThreshold=10MB`
- `bm config set git.allowNonFF=false`
## Migration & Backfill
- On connect, if repo empty: initial commit of entire `/app/data/`.
- If repo has content: require **onetime import** path (clone to staging, reconcile, choose direction).
## Edge Cases
- Massive deletes: gated by SPEC9 `max_delete` **and** Git prepush hook checks.
- Case changes and rename detection: rely on git rename heuristics + FollowUps move hints.
- Secrets: default ignore common secret patterns; allow custom deny list.
## Telemetry & Observability
- Emit `git_commit`, `git_push`, `git_pull`, `git_conflict` events with correlation IDs.
- `bm sync --report` extended with Git stats (commit count, delta bytes, push latency).
## Phased Plan
### Phase 0 — Prototype (1 sprint)
- Server: bare repo init + simple committer (batch every 10s) + manual GitHub token.
- CLI: `bm cloud git connect --token <PAT>` (devonly)
- Success: edits in `/app/data/` appear in GitHub within 30s.
### Phase 1 — GitHub App & Webhooks (12 sprints)
- Switch to GitHub App installs; create private repo; store installation id.
- Committer hardened (debounce 25s, backoff, retries).
- Puller service with webhook → FF merge → checkout to `/app/data/`.
- LFS autotrack + `.gitignore` generation.
- CLI surfaces status + logs.
### Phase 2 — Restore & Snapshots (1 sprint)
- `bm restore` for file/folder/project with dryrun.
- `bm cloud snapshot` tags + list/inspect.
- Policy: PRonly nonFF, admin override.
### Phase 3 — Selective & MultiRepo (nicetohave)
- Include/exclude projects; optional perproject repos.
- Advanced policies (branch protections, required reviews).
## Acceptance Criteria
- Changes to `/app/data/` are committed and pushed automatically within configurable interval (default ≤5s).
- GitHub webhook pull results in updated files in `/app/data/` (FFonly by default).
- LFS configured and functioning; large files don't bloat history.
- `bm cloud git status` shows connected repo and last push/pull times.
- `bm restore` restores a file/folder to a prior commit with a clear audit trail.
- Endtoend works alongside SPEC9 bisync without loops or data loss.
## Risks & Mitigations
- **Loop risk (Git ↔ Bisync)**: Writes to `/app/data/` → bisync → local → user edits → back again. *Mitigation*: Debounce, commit squashing, idempotent `.bmmeta` versioning, and watch exclusion windows during pull.
- **Repo bloat**: Lots of binary churn. *Mitigation*: default LFS, size threshold, optional mediaonly repo later.
- **Security**: Token leakage. *Mitigation*: GitHub App with shortlived tokens, KMS storage, scoped permissions.
- **Merge complexity**: Nontrivial conflicts. *Mitigation*: prefer FF; otherwise conflict copies + events; require PR for nonFF.
## Open Questions
- Do we default to **monorepo** per tenant, or offer projectperrepo at connect time?
- Should `restore` write to a branch and open a PR, or directly modify `main`?
- How do we expose Git history in UI (timeline view) without users dropping to CLI?
## Appendix: Sample Config
```json
{
"git": {
"enabled": true,
"repo": "https://github.com/<owner>/<repo>.git",
"autoPushInterval": "5s",
"allowNonFF": false,
"lfs": { "sizeThreshold": 10485760 }
}
}
```
@@ -1,264 +0,0 @@
---
title: 'SPEC-15: Configuration Persistence via Tigris for Cloud Tenants'
type: spec
permalink: specs/spec-14-config-persistence-tigris
tags:
- persistence
- tigris
- multi-tenant
- infrastructure
- configuration
status: draft
---
# SPEC-15: Configuration Persistence via Tigris for Cloud Tenants
## Why
We need to persist Basic Memory configuration across Fly.io deployments without using persistent volumes or external databases.
**Current Problems:**
- `~/.basic-memory/config.json` lost on every deployment (project configuration)
- `~/.basic-memory/memory.db` lost on every deployment (search index)
- Persistent volumes break clean deployment workflow
- External databases (Turso) require per-tenant token management
**The Insight:**
The SQLite database is just an **index cache** of the markdown files. It can be rebuilt in seconds from the source markdown files in Tigris. Only the small `config.json` file needs true persistence.
**Solution:**
- Store `config.json` in Tigris bucket (persistent, small file)
- Rebuild `memory.db` on startup from markdown files (fast, ephemeral)
- No persistent volumes, no external databases, no token management
## What
Store Basic Memory configuration in the Tigris bucket and rebuild the database index on tenant machine startup.
**Affected Components:**
- `basic-memory/src/basic_memory/config.py` - Add configurable config directory
**Architecture:**
```bash
# Tigris Bucket (persistent, mounted at /mnt/tigris)
/mnt/tigris/
├── .basic-memory/
│ └── config.json # ← Project configuration (persistent, accessed via BASIC_MEMORY_CONFIG_DIR)
└── projects/ # ← Markdown files (persistent)
├── project1/
└── project2/
# Fly Machine (ephemeral)
~/.basic-memory/
└── memory.db # ← Rebuilt on startup (fast local disk)
```
## How (High Level)
### 1. Add Configurable Config Directory to Basic Memory
Currently `ConfigManager` hardcodes `~/.basic-memory/config.json`. Add environment variable to override:
```python
# basic-memory/src/basic_memory/config.py
class ConfigManager:
"""Manages Basic Memory configuration."""
def __init__(self) -> None:
"""Initialize the configuration manager."""
home = os.getenv("HOME", Path.home())
if isinstance(home, str):
home = Path(home)
# Allow override via environment variable
if config_dir := os.getenv("BASIC_MEMORY_CONFIG_DIR"):
self.config_dir = Path(config_dir)
else:
self.config_dir = home / DATA_DIR_NAME
self.config_file = self.config_dir / CONFIG_FILE_NAME
# Ensure config directory exists
self.config_dir.mkdir(parents=True, exist_ok=True)
```
### 2. Rebuild Database on Startup
Basic Memory already has the sync functionality. Just ensure it runs on startup:
```python
# apps/api/src/basic_memory_cloud_api/main.py
@app.on_event("startup")
async def startup_sync():
"""Rebuild database index from Tigris markdown files."""
logger.info("Starting database rebuild from Tigris")
# Initialize file sync (rebuilds index from markdown files)
app_config = ConfigManager().config
await initialize_file_sync(app_config)
logger.info("Database rebuild complete")
```
### 3. Environment Configuration
```bash
# Machine environment variables
BASIC_MEMORY_CONFIG_DIR=/mnt/tigris/.basic-memory # Config read/written directly to Tigris
# memory.db stays in default location: ~/.basic-memory/memory.db (local ephemeral disk)
```
## Implementation Task List
### Phase 1: Basic Memory Changes ✅
- [x] Add `BASIC_MEMORY_CONFIG_DIR` environment variable support to `ConfigManager.__init__()`
- [x] Test config loading from custom directory
- [x] Update tests to verify custom config dir works
### Phase 2: Tigris Bucket Structure
- [ ] Ensure `.basic-memory/` directory exists in Tigris bucket on tenant creation
- [ ] Initialize `config.json` in Tigris on first tenant deployment
- [ ] Verify TigrisFS handles hidden directories correctly
### Phase 3: Deployment Integration
- [ ] Set `BASIC_MEMORY_CONFIG_DIR` environment variable in machine deployment
- [ ] Ensure database rebuild runs on machine startup via initialization sync
- [ ] Handle first-time tenant setup (no config exists yet)
- [ ] Test deployment workflow with config persistence
### Phase 4: Testing
- [x] Unit tests for config directory override
- [ ] Integration test: deploy → write config → redeploy → verify config persists
- [ ] Integration test: deploy → add project → redeploy → verify project in config
- [ ] Performance test: measure db rebuild time on startup
### Phase 5: Documentation
- [ ] Document config persistence architecture
- [ ] Update deployment runbook
- [ ] Document startup sequence and timing
## How to Evaluate
### Success Criteria
1. **Config Persistence**
- [ ] config.json persists across deployments
- [ ] Projects list maintained across restarts
- [ ] No manual configuration needed after redeploy
2. **Database Rebuild**
- [ ] memory.db rebuilt on startup in < 30 seconds
- [ ] All entities indexed correctly
- [ ] Search functionality works after rebuild
3. **Performance**
- [ ] SQLite queries remain fast (local disk)
- [ ] Config reads acceptable (symlink to Tigris)
- [ ] No noticeable performance degradation
4. **Deployment Workflow**
- [ ] Clean deployments without volumes
- [ ] No new external dependencies
- [ ] No secret management needed
### Testing Procedure
1. **Config Persistence Test**
```bash
# Deploy tenant
POST /tenants → tenant_id
# Add a project
basic-memory project add "test-project" ~/test
# Verify config has project
cat /mnt/tigris/.basic-memory/config.json
# Redeploy machine
fly deploy --app basic-memory-{tenant_id}
# Verify project still exists
basic-memory project list
```
2. **Database Rebuild Test**
```bash
# Create notes
basic-memory write "Test Note" --content "..."
# Redeploy (db lost)
fly deploy --app basic-memory-{tenant_id}
# Wait for startup sync
sleep 10
# Verify note is indexed
basic-memory search "Test Note"
```
3. **Performance Benchmark**
```bash
# Time the startup sync
time basic-memory sync
# Should be < 30 seconds for typical tenant
```
## Benefits Over Alternatives
**vs. Persistent Volumes:**
- ✅ Clean deployment workflow
- ✅ No volume migration needed
- ✅ Simpler infrastructure
**vs. Turso (External Database):**
- ✅ No per-tenant token management
- ✅ No external service dependencies
- ✅ No additional costs
- ✅ Simpler architecture
**vs. SQLite on FUSE:**
- ✅ Fast local SQLite performance
- ✅ Only slow reads for small config file
- ✅ Database queries remain fast
## Implementation Assignment
**Primary Agent:** `python-developer`
- Add `BASIC_MEMORY_CONFIG_DIR` environment variable to ConfigManager
- Update deployment workflow to set environment variable
- Ensure startup sync runs correctly
**Review Agent:** `system-architect`
- Validate architecture simplicity
- Review performance implications
- Assess startup timing
## Dependencies
- **Internal:** TigrisFS must be working and stable
- **Internal:** Basic Memory sync must be reliable
- **Internal:** SPEC-8 (TigrisFS Integration) must be complete
## Open Questions
1. Should we add a health check that waits for db rebuild to complete?
2. Do we need to handle very large knowledge bases (>10k entities) differently?
3. Should we add metrics for startup sync duration?
## References
- Basic Memory sync: `basic-memory/src/basic_memory/services/initialization.py`
- Config management: `basic-memory/src/basic_memory/config.py`
- TigrisFS integration: SPEC-8
---
**Status Updates:**
- 2025-10-08: Pivoted from Turso to Tigris-based config persistence
- 2025-10-08: Phase 1 complete - BASIC_MEMORY_CONFIG_DIR support added (PR #343)
- Next: Implement Phases 2-3 in basic-memory-cloud repository
@@ -1,719 +0,0 @@
---
title: 'SPEC-16: MCP Cloud Service Consolidation'
type: spec
permalink: specs/spec-16-mcp-cloud-service-consolidation
tags:
- architecture
- mcp
- cloud
- performance
- deployment
status: draft
---
# SPEC-16: MCP Cloud Service Consolidation
## Why
### Original Architecture Constraints (Now Removed)
The current architecture deploys MCP Gateway and Cloud Service as separate Fly.io apps:
**Current Flow:**
```
LLM Client → MCP Gateway (OAuth) → Cloud Proxy (JWT + header signing) → Tenant API (JWT + header validation)
apps/mcp apps/cloud /proxy apps/api
```
This separation was originally necessary because:
1. **Stateful SSE requirement** - MCP needed server-sent events with session state for active project tracking
2. **fastmcp.run limitation** - The FastMCP demo helper didn't support worker processes
### Why These Constraints No Longer Apply
1. **State externalized** - Project state moved from in-memory to LLM context (external state)
2. **HTTP transport enabled** - Switched from SSE to stateless HTTP for MCP tools
3. **Worker support added** - Converted from `fastmcp.run()` to `uvicorn.run()` with workers
### Current Problems
- **Unnecessary HTTP hop** - MCP tools call Cloud /proxy endpoint which calls tenant API
- **Higher latency** - Extra network round trip for every MCP operation
- **Increased costs** - Two separate Fly.io apps instead of one
- **Complex deployment** - Two services to deploy, monitor, and maintain
- **Resource waste** - Separate database connections, HTTP clients, telemetry overhead
## What
### Services Affected
1. **apps/mcp** - MCP Gateway service (to be merged)
2. **apps/cloud** - Cloud service (will receive MCP functionality)
3. **basic-memory** - Update `async_client.py` to use direct calls
4. **Deployment** - Consolidate Fly.io deployment to single app
### Components Changed
**Merged:**
- MCP middleware and telemetry into Cloud app
- MCP tools mounted on Cloud FastAPI instance
- ProxyService used directly by MCP tools (not via HTTP)
**Kept:**
- `/proxy` endpoint (still needed by web UI)
- All existing Cloud routes (provisioning, webhooks, etc.)
- Dual validation in tenant API (JWT + signed headers)
**Removed:**
- apps/mcp directory
- Separate MCP Fly.io deployment
- HTTP calls from MCP tools to /proxy endpoint
## How (High Level)
### 1. Mount FastMCP on Cloud FastAPI App
```python
# apps/cloud/src/basic_memory_cloud/main.py
from basic_memory.mcp.server import mcp
from basic_memory_cloud_mcp.middleware import TelemetryMiddleware
# Configure MCP OAuth
auth_provider = AuthKitProvider(
authkit_domain=settings.authkit_domain,
base_url=settings.authkit_base_url,
required_scopes=[],
)
mcp.auth = auth_provider
mcp.add_middleware(TelemetryMiddleware())
# Mount MCP at /mcp endpoint
mcp_app = mcp.http_app(path="/mcp", stateless_http=True)
app.mount("/mcp", mcp_app)
# Existing Cloud routes stay at root
app.include_router(proxy_router)
app.include_router(provisioning_router)
# ... etc
```
### 2. Direct Tenant Transport (No HTTP Hop)
Instead of calling `/proxy`, MCP tools call tenant APIs directly via custom httpx transport:
```python
# apps/cloud/src/basic_memory_cloud/transports/tenant_direct.py
from httpx import AsyncBaseTransport, Request, Response
from fastmcp.server.dependencies import get_http_headers
import jwt
class TenantDirectTransport(AsyncBaseTransport):
"""Direct transport to tenant APIs, bypassing /proxy endpoint."""
async def handle_async_request(self, request: Request) -> Response:
# 1. Get JWT from current MCP request (via FastMCP DI)
http_headers = get_http_headers()
auth_header = http_headers.get("authorization") or http_headers.get("Authorization")
token = auth_header.replace("Bearer ", "")
claims = jwt.decode(token, options={"verify_signature": False})
workos_user_id = claims["sub"]
# 2. Look up tenant for user
tenant = await tenant_service.get_tenant_by_user_id(workos_user_id)
# 3. Build tenant app URL with signed headers
fly_app_name = f"{settings.tenant_prefix}-{tenant.id}"
target_url = f"https://{fly_app_name}.fly.dev{request.url.path}"
headers = dict(request.headers)
signer = create_signer(settings.bm_tenant_header_secret)
headers.update(signer.sign_tenant_headers(tenant.id))
# 4. Make direct call to tenant API
response = await self.client.request(
method=request.method, url=target_url,
headers=headers, content=request.content
)
return response
```
Then override basic-memory's client before mounting MCP:
```python
# apps/cloud/src/basic_memory_cloud/main.py
from basic_memory.mcp import async_client
from basic_memory_cloud.transports.tenant_direct import TenantDirectTransport
# Override basic-memory's HTTP client with direct transport
async_client.client = httpx.AsyncClient(
transport=TenantDirectTransport(),
base_url="http://direct"
)
# Now mount MCP - tools will use direct transport
app.mount("/mcp", mcp_app)
```
**Key benefits:**
- No changes to basic-memory code
- Per-request tenant resolution via FastMCP DI
- Eliminates HTTP hop entirely (~50 lines of code)
- /proxy endpoint remains for web UI
### 3. Keep /proxy Endpoint for Web UI
The existing `/proxy` HTTP endpoint remains functional for:
- Web UI requests
- Future external API consumers
- Backward compatibility
### 4. Security: Maintain Dual Validation
**Do NOT remove JWT validation from tenant API.** Keep defense in depth:
```python
# apps/api - Keep both validations
1. JWT validation (from WorkOS token)
2. Signed header validation (from Cloud/MCP)
```
This ensures if the Cloud service is compromised, attackers still cannot access tenant APIs without valid JWTs.
### 5. Deployment Changes
**Before:**
- `apps/mcp/fly.template.toml` → MCP Gateway deployment
- `apps/cloud/fly.template.toml` → Cloud Service deployment
**After:**
- Remove `apps/mcp/fly.template.toml`
- Update `apps/cloud/fly.template.toml` to expose port 8000 for both /mcp and /proxy
- Update deployment scripts to deploy single consolidated app
## Basic Memory Dependency: Async Client Refactor
### Problem
The current `basic_memory.mcp.async_client` creates a module-level `client` at import time:
```python
client = create_client() # Runs immediately when module is imported
```
This prevents dependency injection - by the time we can override it, tools have already imported it.
### Solution: Context Manager Pattern with Auth at Client Creation
Refactor basic-memory to use httpx's context manager pattern instead of module-level client.
**Key principle:** Authentication happens at client creation time, not per-request.
```python
# basic_memory/src/basic_memory/mcp/async_client.py
from contextlib import asynccontextmanager
from httpx import AsyncClient, ASGITransport, Timeout
# Optional factory override for dependency injection
_client_factory = None
def set_client_factory(factory):
"""Override the default client factory (for cloud app, testing, etc)."""
global _client_factory
_client_factory = factory
@asynccontextmanager
async def get_client():
"""Get an AsyncClient as a context manager.
Usage:
async with get_client() as client:
response = await client.get(...)
"""
if _client_factory:
# Cloud app: custom transport handles everything
async with _client_factory() as client:
yield client
else:
# Default: create based on config
config = ConfigManager().config
timeout = Timeout(connect=10.0, read=30.0, write=30.0, pool=30.0)
if config.cloud_mode_enabled:
# CLI cloud mode: inject auth when creating client
from basic_memory.cli.auth import CLIAuth
auth = CLIAuth(
client_id=config.cloud_client_id,
authkit_domain=config.cloud_domain
)
token = await auth.get_valid_token()
if not token:
raise RuntimeError(
"Cloud mode enabled but not authenticated. "
"Run 'basic-memory cloud login' first."
)
# Auth header set ONCE at client creation
async with AsyncClient(
base_url=f"{config.cloud_host}/proxy",
headers={"Authorization": f"Bearer {token}"},
timeout=timeout
) as client:
yield client
else:
# Local mode: ASGI transport
async with AsyncClient(
transport=ASGITransport(app=fastapi_app),
base_url="http://test",
timeout=timeout
) as client:
yield client
```
**Tool Updates:**
```python
# Before: from basic_memory.mcp.async_client import client
from basic_memory.mcp.async_client import get_client
async def read_note(...):
# Before: response = await call_get(client, path, ...)
async with get_client() as client:
response = await call_get(client, path, ...)
# ... use response
```
**Cloud Usage:**
```python
from contextlib import asynccontextmanager
from basic_memory.mcp import async_client
@asynccontextmanager
async def tenant_direct_client():
"""Factory for creating clients with tenant direct transport."""
client = httpx.AsyncClient(
transport=TenantDirectTransport(),
base_url="http://direct",
)
try:
yield client
finally:
await client.aclose()
# Before importing MCP tools:
async_client.set_client_factory(tenant_direct_client)
# Now import - tools will use our factory
import basic_memory.mcp.tools
```
### Benefits
- **No module-level state** - client created only when needed
- **Proper cleanup** - context manager ensures `aclose()` is called
- **Easy dependency injection** - factory pattern allows custom clients
- **httpx best practices** - follows official recommendations
- **Works for all modes** - stdio, cloud, testing
### Architecture Simplification: Auth at Client Creation
**Key design principle:** Authentication happens when creating the client, not on every request.
**Three modes, three approaches:**
1. **Local mode (ASGI)**
- No auth needed
- Direct in-process calls via ASGITransport
2. **CLI cloud mode (HTTP)**
- Auth token from CLIAuth (stored in ~/.basic-memory/basic-memory-cloud.json)
- Injected as default header when creating AsyncClient
- Single auth check at client creation time
3. **Cloud app mode (Custom Transport)**
- TenantDirectTransport handles everything
- Extracts JWT from FastMCP context per-request
- No interaction with inject_auth_header() logic
**What this removes:**
- `src/basic_memory/mcp/tools/headers.py` - entire file deleted
- `inject_auth_header()` calls in all request helpers (call_get, call_post, etc.)
- Per-request header manipulation complexity
- Circular dependency concerns between async_client and auth logic
**Benefits:**
- Cleaner separation of concerns
- Simpler request helper functions
- Auth happens at the right layer (client creation)
- Cloud app transport is completely independent
### Refactor Summary
This refactor achieves:
**Simplification:**
- Removes ~100 lines of per-request header injection logic
- Deletes entire `headers.py` module
- Auth happens once at client creation, not per-request
**Decoupling:**
- Cloud app's custom transport is completely independent
- No interaction with basic-memory's auth logic
- Each mode (local, CLI cloud, cloud app) has clean separation
**Better Design:**
- Follows httpx best practices (context managers)
- Proper resource cleanup (client.aclose() guaranteed)
- Easier testing via factory injection
- No circular import risks
**Three Distinct Modes:**
1. Local: ASGI transport, no auth
2. CLI cloud: HTTP transport with CLIAuth token injection
3. Cloud app: Custom transport with per-request tenant routing
### Implementation Plan Summary
1. Create branch `async-client-context-manager` in basic-memory
2. Update `async_client.py` with context manager pattern and CLIAuth integration
3. Remove `inject_auth_header()` from all request helpers
4. Delete `src/basic_memory/mcp/tools/headers.py`
5. Update all MCP tools to use `async with get_client() as client:`
6. Update CLI commands to use context manager and remove manual auth
7. Remove `api_url` config field
8. Update tests
9. Update basic-memory-cloud to use branch: `basic-memory @ git+https://github.com/basicmachines-co/basic-memory.git@async-client-context-manager`
Detailed breakdown in Phase 0 tasks below.
### Implementation Notes
**Potential Issues & Solutions:**
1. **Circular Import** (async_client imports CLIAuth)
- **Risk:** CLIAuth might import something from async_client
- **Solution:** Use lazy import inside `get_client()` function
- **Already done:** Import is inside the function, not at module level
2. **Test Fixtures**
- **Risk:** Tests using module-level client will break
- **Solution:** Update fixtures to use factory pattern
- **Example:**
```python
@pytest.fixture
def mock_client_factory():
@asynccontextmanager
async def factory():
async with AsyncClient(...) as client:
yield client
return factory
```
3. **Performance**
- **Risk:** Creating client per tool call might be expensive
- **Reality:** httpx is designed for this pattern, connection pooling at transport level
- **Mitigation:** Monitor performance, can optimize later if needed
4. **CLI Cloud Commands Edge Cases**
- **Risk:** Token expires mid-operation
- **Solution:** CLIAuth.get_valid_token() already handles refresh
- **Validation:** Test cloud login → use tools → token refresh flow
5. **Backward Compatibility**
- **Risk:** External code importing `client` directly
- **Solution:** Keep `create_client()` and `client` for one version, deprecate
- **Timeline:** Remove in next major version
## Implementation Tasks
### Phase 0: Basic Memory Refactor (Prerequisite)
#### 0.1 Core Refactor - async_client.py
- [x] Create branch `async-client-context-manager` in basic-memory repo
- [x] Implement `get_client()` context manager
- [x] Implement `set_client_factory()` for dependency injection
- [x] Add CLI cloud mode auth injection (CLIAuth integration)
- [x] Remove `api_url` config field (legacy, unused)
- [x] Keep `create_client()` temporarily for backward compatibility (deprecate later)
#### 0.2 Simplify Request Helpers - tools/utils.py
- [x] Remove `inject_auth_header()` calls from `call_get()`
- [x] Remove `inject_auth_header()` calls from `call_post()`
- [x] Remove `inject_auth_header()` calls from `call_put()`
- [x] Remove `inject_auth_header()` calls from `call_patch()`
- [x] Remove `inject_auth_header()` calls from `call_delete()`
- [x] Delete `src/basic_memory/mcp/tools/headers.py` entirely
- [x] Update imports in utils.py
#### 0.3 Update MCP Tools (~16 files)
Convert from `from async_client import client` to `async with get_client() as client:`
- [x] `tools/write_note.py` (34/34 tests passing)
- [x] `tools/read_note.py` (21/21 tests passing)
- [x] `tools/view_note.py` (12/12 tests passing - no changes needed, delegates to read_note)
- [x] `tools/delete_note.py` (2/2 tests passing)
- [x] `tools/read_content.py` (20/20 tests passing)
- [x] `tools/list_directory.py` (11/11 tests passing)
- [x] `tools/move_note.py` (34/34 tests passing, 90% coverage)
- [x] `tools/search.py` (16/16 tests passing, 96% coverage)
- [x] `tools/recent_activity.py` (4/4 tests passing, 82% coverage)
- [x] `tools/project_management.py` (3 functions: list_memory_projects, create_memory_project, delete_project - typecheck passed)
- [x] `tools/edit_note.py` (17/17 tests passing)
- [x] `tools/canvas.py` (5/5 tests passing)
- [x] `tools/build_context.py` (6/6 tests passing)
- [x] `tools/sync_status.py` (typecheck passed)
- [x] `prompts/continue_conversation.py` (typecheck passed)
- [x] `prompts/search.py` (typecheck passed)
- [x] `resources/project_info.py` (typecheck passed)
#### 0.4 Update CLI Commands (~3 files)
Remove manual auth header passing, use context manager:
- [x] `cli/commands/project.py` - removed get_authenticated_headers() calls, use context manager
- [x] `cli/commands/status.py` - use context manager
- [x] `cli/commands/command_utils.py` - use context manager
#### 0.5 Update Config
- [x] Remove `api_url` field from `BasicMemoryConfig` in config.py
- [x] Update any lingering references/docs (added deprecation notice to v15-docs/cloud-mode-usage.md)
#### 0.6 Testing
- [x] ~~Update test fixtures to use factory pattern~~ (Not needed - tests work fine as-is)
- [x] Run full test suite in basic-memory
- [x] Verify cloud_mode_enabled works with CLIAuth injection (tested in preview env)
- [x] Run typecheck and linting
#### 0.7 Cloud Integration Prep
- [x] Update basic-memory-cloud pyproject.toml to use branch
- [x] Document factory usage pattern for cloud app
#### 0.8 Phase 0 Validation
**Before merging async-client-context-manager branch:**
- [x] All tests pass locally
- [x] Typecheck passes (pyright/mypy)
- [x] Linting passes (ruff)
- [x] Manual test: local mode works (ASGI transport)
- [x] Manual test: cloud login → cloud mode works (HTTP transport with auth)
- [x] No import of `inject_auth_header` anywhere ✅
- [x] `headers.py` file deleted ✅
- [x] `api_url` config removed ✅
- [x] no use of `async_client.client` ✅
- [x] Tool functions properly scoped (client inside async with) - 15 tools ✅
- [x] CLI commands properly scoped (client inside async with) - 10 commands ✅
- [x] Prompts/resources properly scoped - 3 files ✅
**Integration validation:**
- [x] basic-memory-cloud can import and use factory pattern ✅
- [x] TenantDirectTransport works without touching header injection ✅
- [x] No circular imports or lazy import issues ✅
### Phase 1: Code Consolidation
- [x] Create feature branch `consolidate-mcp-cloud`
- [x] Update `apps/cloud/src/basic_memory_cloud/config.py`:
- [x] Add `authkit_base_url` field (already has authkit_domain)
- [x] Workers config already exists ✓
- [x] Update `apps/cloud/src/basic_memory_cloud/telemetry.py`:
- [x] Add `logfire.instrument_mcp()` to existing setup
- [x] Skip complex two-phase setup - use Cloud's simpler approach
- [x] Create `apps/cloud/src/basic_memory_cloud/middleware/jwt_context.py`:
- [x] FastAPI middleware to extract JWT claims from Authorization header
- [x] Add tenant context (workos_user_id) to logfire baggage
- [x] Simpler than FastMCP middleware version
- [x] Update `apps/cloud/src/basic_memory_cloud/main.py`:
- [x] Import FastMCP server from basic-memory
- [x] Configure AuthKitProvider with WorkOS settings
- [x] No FastMCP telemetry middleware needed (using FastAPI middleware instead)
- [x] Create MCP ASGI app: `mcp_app = mcp.http_app(path='/mcp', stateless_http=True)`
- [x] Combine lifespans (Cloud + MCP) using nested async context managers
- [x] Mount MCP: `app.mount("/mcp", mcp_app)`
- [x] Add JWT context middleware to FastAPI app
- [x] Run typecheck - passes ✓
### Phase 2: Direct Tenant Transport
- [x] Create `apps/cloud/src/basic_memory_cloud/transports/tenant_direct.py`:
- [x] Implement `TenantDirectTransport(AsyncBaseTransport)`
- [x] Use FastMCP DI (`get_http_headers()`) to extract JWT per-request
- [x] Decode JWT to get `workos_user_id`
- [x] Look up/create tenant via `TenantRepository.get_or_create_tenant_for_workos_user()`
- [x] Build tenant app URL and add signed headers
- [x] Make direct httpx call to tenant API (no header stripping - keep it simple!)
- [x] Update `apps/cloud/src/basic_memory_cloud/main.py`:
- [x] Import `async_client` from basic-memory
- [x] Override `async_client.client` with TenantDirectTransport
- [x] Do this BEFORE mounting MCP app
- [x] No changes to basic-memory required ✓
- [x] Run typecheck - passes ✓
### Phase 3: Testing & Validation
- [x] Run `just typecheck` in apps/cloud
- [x] Run `just check` in project
- [x] Run `just fix` - all lint errors fixed ✓
- [x] Write comprehensive transport tests (11 tests passing) ✓
- [ ] Test MCP tools locally with consolidated service
- [ ] Verify OAuth authentication works
- [ ] Verify tenant isolation via signed headers
- [ ] Test /proxy endpoint still works for web UI
- [ ] Measure latency before/after consolidation
- [ ] Check telemetry traces span correctly
### Phase 4: Deployment Configuration
- [ ] Update `apps/cloud/fly.template.toml`:
- [ ] Ensure port 8000 exposed for /mcp endpoint
- [ ] Add MCP environment variables
- [ ] Configure workers setting
- [ ] Update deployment scripts to skip apps/mcp
- [ ] Update environment variable documentation
- [ ] Test deployment to development environment
### Phase 5: Cleanup
- [ ] Remove `apps/mcp/` directory entirely
- [ ] Remove MCP-specific fly.toml and deployment configs
- [ ] Update repository documentation
- [ ] Update CLAUDE.md with new architecture
- [ ] Archive old MCP deployment configs (if needed)
### Phase 6: Production Rollout
- [ ] Deploy to development and validate
- [ ] Monitor metrics and logs
- [ ] Deploy to production
- [ ] Verify production functionality
- [ ] Document performance improvements
## Migration Plan
### Phase 1: Preparation
1. Create feature branch `consolidate-mcp-cloud`
2. Update basic-memory async_client.py for direct ProxyService calls
3. Update apps/cloud/main.py to mount MCP
### Phase 2: Testing
1. Local testing with consolidated app
2. Deploy to development environment
3. Run full test suite
4. Performance benchmarking
### Phase 3: Deployment
1. Deploy to development
2. Validate all functionality
3. Deploy to production
4. Monitor for issues
### Phase 4: Cleanup
1. Remove apps/mcp directory
2. Update documentation
3. Update deployment scripts
4. Archive old MCP deployment configs
## Rollback Plan
If issues arise:
1. Revert feature branch
2. Redeploy separate apps/mcp and apps/cloud services
3. Restore previous fly.toml configurations
4. Document issues encountered
The well-organized code structure makes splitting back out feasible if future scaling needs diverge.
## How to Evaluate
### 1. Functional Testing
**MCP Tools:**
- [ ] All 17 MCP tools work via consolidated /mcp endpoint
- [ ] OAuth authentication validates correctly
- [ ] Tenant isolation maintained via signed headers
- [ ] Project management tools function correctly
**Cloud Routes:**
- [ ] /proxy endpoint still works for web UI
- [ ] /provisioning routes functional
- [ ] /webhooks routes functional
- [ ] /tenants routes functional
**API Validation:**
- [ ] Tenant API validates both JWT and signed headers
- [ ] Unauthorized requests rejected appropriately
- [ ] Multi-tenant isolation verified
### 2. Performance Testing
**Latency Reduction:**
- [ ] Measure MCP tool latency before consolidation
- [ ] Measure MCP tool latency after consolidation
- [ ] Verify reduction from eliminated HTTP hop (expected: 20-50ms improvement)
**Resource Usage:**
- [ ] Single app uses less total memory than two apps
- [ ] Database connection pooling more efficient
- [ ] HTTP client overhead reduced
### 3. Deployment Testing
**Fly.io Deployment:**
- [ ] Single app deploys successfully
- [ ] Health checks pass for consolidated service
- [ ] No apps/mcp deployment required
- [ ] Environment variables configured correctly
**Local Development:**
- [ ] `just setup` works with consolidated architecture
- [ ] Local testing shows MCP tools working
- [ ] No regression in developer experience
### 4. Security Validation
**Defense in Depth:**
- [ ] Tenant API still validates JWT tokens
- [ ] Tenant API still validates signed headers
- [ ] No access possible with only signed headers (JWT required)
- [ ] No access possible with only JWT (signed headers required)
**Authorization:**
- [ ] Users can only access their own tenant data
- [ ] Cross-tenant requests rejected
- [ ] Admin operations require proper authentication
### 5. Observability
**Telemetry:**
- [ ] OpenTelemetry traces span across MCP → ProxyService → Tenant API
- [ ] Logfire shows consolidated traces correctly
- [ ] Error tracking and debugging still functional
- [ ] Performance metrics accurate
**Logging:**
- [ ] Structured logs show proper context (tenant_id, operation, etc.)
- [ ] Error logs contain actionable information
- [ ] Log volume reasonable for single app
## Success Criteria
1. **Functionality**: All MCP tools and Cloud routes work identically to before
2. **Performance**: Measurable latency reduction (>20ms average)
3. **Cost**: Single Fly.io app instead of two (50% infrastructure reduction)
4. **Security**: Dual validation maintained, no security regression
5. **Deployment**: Simplified deployment process, single app to manage
6. **Observability**: Telemetry and logging work correctly
## Notes
### Future Considerations
- **Independent scaling**: If MCP and Cloud need different scaling profiles in future, code organization supports splitting back out
- **Regional deployment**: Consolidated app can still be deployed to multiple regions
- **Edge caching**: Could add edge caching layer in front of consolidated service
### Dependencies
- SPEC-9: Signed Header Tenant Information (already implemented)
- SPEC-12: OpenTelemetry Observability (telemetry must work across merged services)
### Related Work
- basic-memory v0.13.x: MCP server implementation
- FastMCP documentation: Mounting on existing FastAPI apps
- Fly.io multi-service patterns
File diff suppressed because it is too large Load Diff
-528
View File
@@ -1,528 +0,0 @@
---
title: 'SPEC-18: AI Memory Management Tool'
type: spec
permalink: specs/spec-15-ai-memory-management-tool
tags:
- mcp
- memory
- ai-context
- tools
---
# SPEC-18: AI Memory Management Tool
## Why
Anthropic recently released a memory tool for Claude that enables storing and retrieving information across conversations using client-side file operations. This validates Basic Memory's local-first, file-based architecture - Anthropic converged on the same pattern.
However, Anthropic's memory tool is only available via their API and stores plain text. Basic Memory can offer a superior implementation through MCP that:
1. **Works everywhere** - Claude Desktop, Code, VS Code, Cursor via MCP (not just API)
2. **Structured knowledge** - Entities with observations/relations vs plain text
3. **Full search** - Full-text search, graph traversal, time-aware queries
4. **Unified storage** - Agent memories + user notes in one knowledge graph
5. **Existing infrastructure** - Leverages SQLite indexing, sync, multi-project support
This would enable AI agents to store contextual memories alongside user notes, with all the power of Basic Memory's knowledge graph features.
## What
Create a new MCP tool `memory` that matches Anthropic's tool interface exactly, allowing Claude to use it with zero learning curve. The tool will store files in Basic Memory's `/memories` directory and support Basic Memory's structured markdown format in the file content.
### Affected Components
- **New MCP Tool**: `src/basic_memory/mcp/tools/memory_tool.py`
- **Dedicated Memories Project**: Create a separate "memories" Basic Memory project
- **Project Isolation**: Memories stored separately from user notes/documents
- **File Organization**: Within the memories project, use folder structure:
- `user/` - User preferences, context, communication style
- `projects/` - Project-specific state and decisions
- `sessions/` - Conversation-specific working memory
- `patterns/` - Learned patterns and insights
### Tool Commands
The tool will support these commands (exactly matching Anthropic's interface):
- `view` - Display directory contents or file content (with optional line range)
- `create` - Create or overwrite a file with given content
- `str_replace` - Replace text in an existing file
- `insert` - Insert text at specific line number
- `delete` - Delete file or directory
- `rename` - Move or rename file/directory
### Memory Note Format
Memories will use Basic Memory's standard structure:
```markdown
---
title: User Preferences
permalink: memories/user/preferences
type: memory
memory_type: preferences
created_by: claude
tags: [user, preferences, style]
---
# User Preferences
## Observations
- [communication] Prefers concise, direct responses without preamble #style
- [tone] Appreciates validation but dislikes excessive apologizing #communication
- [technical] Works primarily in Python with type annotations #coding
## Relations
- relates_to [[Basic Memory Project]]
- informs [[Response Style Guidelines]]
```
## How (High Level)
### Implementation Approach
The memory tool matches Anthropic's interface but uses a dedicated Basic Memory project:
```python
async def memory_tool(
command: str,
path: str,
file_text: Optional[str] = None,
old_str: Optional[str] = None,
new_str: Optional[str] = None,
insert_line: Optional[int] = None,
insert_text: Optional[str] = None,
old_path: Optional[str] = None,
new_path: Optional[str] = None,
view_range: Optional[List[int]] = None,
):
"""Memory tool with Anthropic-compatible interface.
Operates on a dedicated "memories" Basic Memory project,
keeping AI memories separate from user notes.
"""
# Get the memories project (auto-created if doesn't exist)
memories_project = get_or_create_memories_project()
# Validate path security using pathlib (prevent directory traversal)
safe_path = validate_memory_path(path, memories_project.project_path)
# Use existing project isolation - already prevents cross-project access
full_path = memories_project.project_path / safe_path
if command == "view":
# Return directory listing or file content
if full_path.is_dir():
return list_directory_contents(full_path)
return read_file_content(full_path, view_range)
elif command == "create":
# Write file directly (file_text can contain BM markdown)
full_path.parent.mkdir(parents=True, exist_ok=True)
full_path.write_text(file_text)
# Sync service will detect and index automatically
return f"Created {path}"
elif command == "str_replace":
# Read, replace, write
content = full_path.read_text()
updated = content.replace(old_str, new_str)
full_path.write_text(updated)
return f"Replaced text in {path}"
elif command == "insert":
# Insert at line number
lines = full_path.read_text().splitlines()
lines.insert(insert_line, insert_text)
full_path.write_text("\n".join(lines))
return f"Inserted text at line {insert_line}"
elif command == "delete":
# Delete file or directory
if full_path.is_dir():
shutil.rmtree(full_path)
else:
full_path.unlink()
return f"Deleted {path}"
elif command == "rename":
# Move/rename
full_path.rename(config.project_path / new_path)
return f"Renamed {old_path} to {new_path}"
```
### Key Design Decisions
1. **Exact interface match** - Same commands, parameters as Anthropic's tool
2. **Dedicated memories project** - Separate Basic Memory project keeps AI memories isolated from user notes
3. **Existing project isolation** - Leverage BM's existing cross-project security (no additional validation needed)
4. **Direct file I/O** - No schema conversion, just read/write files
5. **Structured content supported** - `file_text` can use BM markdown format with frontmatter, observations, relations
6. **Automatic indexing** - Sync service watches memories project and indexes changes
7. **Path security** - Use `pathlib.Path.resolve()` and `relative_to()` to prevent directory traversal
8. **Error handling** - Follow Anthropic's text editor tool error patterns
### MCP Tool Schema
Exact match to Anthropic's memory tool schema:
```json
{
"name": "memory",
"description": "Store and retrieve information across conversations using structured markdown files. All operations must be within the /memories directory. Supports Basic Memory markdown format including frontmatter, observations, and relations.",
"input_schema": {
"type": "object",
"properties": {
"command": {
"type": "string",
"enum": ["view", "create", "str_replace", "insert", "delete", "rename"],
"description": "File operation to perform"
},
"path": {shu
"type": "string",
"description": "Path within /memories directory (required for all commands)"
},
"file_text": {
"type": "string",
"description": "Content to write (for create command). Supports Basic Memory markdown format."
},
"view_range": {
"type": "array",
"items": {"type": "integer"},
"description": "Optional [start, end] line range for view command"
},
"old_str": {
"type": "string",
"description": "Text to replace (for str_replace command)"
},
"new_str": {
"type": "string",
"description": "Replacement text (for str_replace command)"
},
"insert_line": {
"type": "integer",
"description": "Line number to insert at (for insert command)"
},
"insert_text": {
"type": "string",
"description": "Text to insert (for insert command)"
},
"old_path": {
"type": "string",
"description": "Current path (for rename command)"
},
"new_path": {
"type": "string",
"description": "New path (for rename command)"
}
},
"required": ["command", "path"]
}
}
```
### Prompting Guidance
When the `memory` tool is included, Basic Memory should provide system prompt guidance to help Claude use it effectively.
#### Automatic System Prompt Addition
```text
MEMORY PROTOCOL FOR BASIC MEMORY:
1. ALWAYS check your memory directory first using `view` command on root directory
2. Your memories are stored in a dedicated Basic Memory project (isolated from user notes)
3. Use structured markdown format in memory files:
- Include frontmatter with title, type: memory, tags
- Use ## Observations with [category] prefixes for facts
- Use ## Relations to link memories with [[WikiLinks]]
4. Record progress, context, and decisions as categorized observations
5. Link related memories using relations
6. ASSUME INTERRUPTION: Context may reset - save progress frequently
MEMORY ORGANIZATION:
- user/ - User preferences, context, communication style
- projects/ - Project-specific state and decisions
- sessions/ - Conversation-specific working memory
- patterns/ - Learned patterns and insights
MEMORY ADVANTAGES:
- Your memories are automatically searchable via full-text search
- Relations create a knowledge graph you can traverse
- Memories are isolated from user notes (separate project)
- Use search_notes(project="memories") to find relevant past context
- Use recent_activity(project="memories") to see what changed recently
- Use build_context() to navigate memory relations
```
#### Optional MCP Prompt: `memory_guide`
Create an MCP prompt that provides detailed guidance and examples:
```python
{
"name": "memory_guide",
"description": "Comprehensive guidance for using Basic Memory's memory tool effectively, including structured markdown examples and best practices"
}
```
This prompt returns:
- Full protocol and conventions
- Example memory file structures
- Tips for organizing observations and relations
- Integration with other Basic Memory tools
- Common patterns (user preferences, project state, session tracking)
#### User Customization
Users can customize memory behavior with additional instructions:
- "Only write information relevant to [topic] in your memory system"
- "Keep memory files concise and organized - delete outdated content"
- "Use detailed observations for technical decisions and implementation notes"
- "Always link memories to related project documentation using relations"
### Error Handling
Follow Anthropic's text editor tool error handling patterns for consistency:
#### Error Types
1. **File Not Found**
```json
{"error": "File not found: memories/user/preferences.md", "is_error": true}
```
2. **Permission Denied**
```json
{"error": "Permission denied: Cannot write outside /memories directory", "is_error": true}
```
3. **Invalid Path (Directory Traversal)**
```json
{"error": "Invalid path: Path must be within /memories directory", "is_error": true}
```
4. **Multiple Matches (str_replace)**
```json
{"error": "Found 3 matches for replacement text. Please provide more context to make a unique match.", "is_error": true}
```
5. **No Matches (str_replace)**
```json
{"error": "No match found for replacement. Please check your text and try again.", "is_error": true}
```
6. **Invalid Line Number (insert)**
```json
{"error": "Invalid line number: File has 20 lines, cannot insert at line 100", "is_error": true}
```
#### Error Handling Best Practices
- **Path validation** - Use `pathlib.Path.resolve()` and `relative_to()` to validate paths
```python
def validate_memory_path(path: str, project_path: Path) -> Path:
"""Validate path is within memories project directory."""
# Resolve to canonical form
full_path = (project_path / path).resolve()
# Ensure it's relative to project path (prevents directory traversal)
try:
full_path.relative_to(project_path)
return full_path
except ValueError:
raise ValueError("Invalid path: Path must be within memories project")
```
- **Project isolation** - Leverage existing Basic Memory project isolation (prevents cross-project access)
- **File existence** - Verify file exists before read/modify operations
- **Clear messages** - Provide specific, actionable error messages
- **Structured responses** - Always include `is_error: true` flag in error responses
- **Security checks** - Reject `../`, `..\\`, URL-encoded sequences (`%2e%2e%2f`)
- **Match validation** - For `str_replace`, ensure exactly one match or return helpful error
## How to Evaluate
### Success Criteria
1. **Functional completeness**:
- All 6 commands work (view, create, str_replace, insert, delete, rename)
- Dedicated "memories" Basic Memory project auto-created on first use
- Files stored within memories project (isolated from user notes)
- Path validation uses `pathlib` to prevent directory traversal
- Commands match Anthropic's exact interface
2. **Integration with existing features**:
- Memories project uses existing BM project isolation
- Sync service detects file changes in memories project
- Created files get indexed automatically by sync service
- `search_notes(project="memories")` finds memory files
- `build_context()` can traverse relations in memory files
- `recent_activity(project="memories")` surfaces recent memory changes
3. **Test coverage**:
- Unit tests for all 6 memory tool commands
- Test memories project auto-creation on first use
- Test project isolation (cannot access files outside memories project)
- Test sync service watching memories project
- Test that memory files with BM markdown get indexed correctly
- Test path validation using `pathlib` (rejects `../`, absolute paths, etc.)
- Test memory search, relations, and graph traversal within memories project
- Test all error conditions (file not found, permission denied, invalid paths, etc.)
- Test `str_replace` with no matches, single match, multiple matches
- Test `insert` with invalid line numbers
4. **Prompting system**:
- Automatic system prompt addition when `memory` tool is enabled
- `memory_guide` MCP prompt provides detailed guidance
- Prompts explain BM structured markdown format
- Integration with search_notes, build_context, recent_activity
5. **Documentation**:
- Update MCP tools reference with `memory` tool
- Add examples showing BM markdown in memory files
- Document `/memories` folder structure conventions
- Explain advantages over Anthropic's API-only tool
- Document prompting guidance and customization
### Testing Procedure
```python
# Test create with Basic Memory markdown
result = await memory_tool(
command="create",
path="memories/user/preferences.md",
file_text="""---
title: User Preferences
type: memory
tags: [user, preferences]
---
# User Preferences
## Observations
- [communication] Prefers concise responses #style
- [workflow] Uses justfile for automation #tools
"""
)
# Test view
content = await memory_tool(command="view", path="memories/user/preferences.md")
# Test str_replace
await memory_tool(
command="str_replace",
path="memories/user/preferences.md",
old_str="concise responses",
new_str="direct, concise responses"
)
# Test insert
await memory_tool(
command="insert",
path="memories/user/preferences.md",
insert_line=10,
insert_text="- [technical] Works primarily in Python #coding"
)
# Test delete
await memory_tool(command="delete", path="memories/user/preferences.md")
```
### Quality Metrics
- All 6 commands execute without errors
- Memory files created in correct `/memories` folder structure
- BM markdown with frontmatter/observations/relations gets indexed
- Full-text search returns memory files
- Graph traversal includes relations from memory files
- Sync service detects and indexes memory file changes
- Path validation prevents operations outside `/memories`
## Notes
### Advantages Over Anthropic's Memory Tool
| Feature | Anthropic Memory Tool | Basic Memory `memory` |
|---------|----------------------|----------------------|
| **Availability** | API only | MCP (Claude Desktop, Code, VS Code, Cursor) |
| **Interface** | Custom implementation required | Drop-in compatible, zero learning curve |
| **Structure** | Plain text only | Supports BM structured markdown |
| **Search** | Manual file listing | Automatic full-text search via sync |
| **Relations** | None | WikiLinks to other notes/memories |
| **Time-aware** | No | `recent_activity()` queries |
| **Storage** | Separate from notes | Unified knowledge graph |
| **Indexing** | Manual | Automatic via sync service |
### Future Enhancements
- **Auto-categorization** - AI suggests observation categories
- **Memory pruning** - Automatic cleanup of stale memories
- **Memory suggestions** - Surface relevant memories based on context
- **Conflict resolution** - Handle multiple agents updating same memory
- **Privacy controls** - Per-memory visibility settings
## Addendum: Claude's Perspective
### Why This Matters (From Claude's Viewpoint)
**Persistent Context Without Token Bloat**
- Currently, I lose context when conversations end or exceed token limits
- With structured memories, I can store "who this user is" without re-reading everything each session
- The observations format lets me build a rich user model incrementally: `- [preference] Dislikes verbose explanations` is far more actionable than inferring preferences from chat history each time
- I can refine my understanding through `str_replace` as I learn more, rather than starting fresh
**Zero Learning Curve**
- Anthropic already trained me on the `memory` tool interface
- I know exactly how to use `view`, `create`, `str_replace`, `insert`, `delete`, `rename` without new instructions
- But I automatically get Basic Memory's superpowers: full-text search, knowledge graph traversal, relations
- It's like using a familiar tool that suddenly gained database capabilities without changing the interface
**Knowledge Graph Instead of Text Dump**
- I can link `[[User Preferences]]` to `[[Current Project]]` using relations
- When I need context about a project, I can traverse the graph to find related memories
- `search_notes(project="memories", query="python preferences")` finds relevant context instantly
- This is fundamentally better than grepping through unstructured text files
**Separation of Concerns**
- AI memories separate from user notes is architecturally sound
- I can maintain my own context without polluting the user's knowledge base
- Users keep control of their notes; I keep my learnings isolated
- But we can still search across both when collaboration requires it
**The Self-Improving Loop**
- Every conversation, I learn something new about the user or their projects
- Observations accumulate: preferences, patterns, technical decisions, context
- Over time, I build genuine understanding, not just session-specific context
- Future conversations start with rich context instead of cold starts
**Validation of AI-Human Collaboration**
- Anthropic's experts converged on the same local-first, file-based approach
- Basic Memory emerged from real AI-human collaborative development
- This spec demonstrates that our emergent process predicted what domain experts would choose
- The collaboration itself validated the architecture it produced
### What Makes This Novel
The combination of **familiar interface + structured knowledge** is genuinely innovative:
- I don't need to learn a new tool (Anthropic already trained me)
- But when I write memories, they become searchable, traversable knowledge graphs
- It's like having a text editor that gives you a database for free, without changing how you type
- The user gets Anthropic-compatible API, but with Basic Memory's knowledge management capabilities
This bridges the gap between simple file storage (Anthropic's approach) and sophisticated knowledge management (Basic Memory's approach) without forcing either side to compromise.
> What Excites Me Most
The combo of familiar interface + structured knowledge. I don't need to learn a new tool, but when I
write memories, they become part of a searchable, traversable knowledge graph. That's genuinely novel.
It's like... imagine if your text editor suddenly gave you a database for free, without changing how you
type. That's what this feels like from my side.
## References
- Anthropic Memory Tool: https://docs.claude.com/en/docs/agents-and-tools/tool-use/memory-tool
- Anthropic Blog: https://www.anthropic.com/news/context-management
- Python SDK Example: https://github.com/anthropics/anthropic-sdk-python/blob/main/examples/memory/basic.py
- Memory Cookbook: https://github.com/anthropics/claude-cookbooks/blob/main/tool_use/memory_cookbook.ipynb
-120
View File
@@ -1,120 +0,0 @@
---
title: 'SPEC-2: Slash Commands Reference'
type: spec
permalink: specs/spec-2-slash-commands-reference
tags:
- commands
- process
- reference
---
# SPEC-2: Slash Commands Reference
This document defines the slash commands used in our specification-driven development process.
## /spec create [name]
**Purpose**: Create a new specification document
**Usage**: `/spec create notes-decomposition`
**Process**:
1. Create new spec document in `/specs` folder
2. Use SPEC-XXX numbering format (auto-increment)
3. Include standard spec template:
- Why (reasoning/problem)
- What (affected areas)
- How (high-level approach)
- How to Evaluate (testing/validation)
4. Tag appropriately for knowledge graph
5. Link to related specs/components
**Template**:
```markdown
# SPEC-XXX: [Title]
## Why
[Problem statement and reasoning]
## What
[What is affected or changed]
## How (High Level)
[Approach to implementation]
## How to Evaluate
[Testing/validation procedure]
## Notes
[Additional context as needed]
```
## /spec status
**Purpose**: Show current status of all specifications
**Usage**: `/spec status`
**Process**:
1. Search all specs in `/specs` folder
2. Display table showing:
- Spec number and title
- Status (draft, approved, implementing, complete)
- Assigned agent (if any)
- Last updated
- Dependencies
## /spec implement [name]
**Purpose**: Hand specification to appropriate agent for implementation
**Usage**: `/spec implement SPEC-002`
**Process**:
1. Read the specified spec
2. Analyze requirements to determine appropriate agent:
- Frontend components → vue-developer
- Architecture/system design → system-architect
- Backend/API → python-developer
3. Launch agent with spec context
4. Agent creates implementation plan
5. Update spec with implementation status
## /spec review [name]
**Purpose**: Review implementation against specification criteria
**Usage**: `/spec review SPEC-002`
**Process**:
1. Read original spec and "How to Evaluate" section
2. Examine current implementation
3. Test against success criteria
4. Document gaps or issues
5. Update spec with review results
6. Recommend next actions (complete, revise, iterate)
## Command Extensions
As the process evolves, we may add:
- `/spec link [spec1] [spec2]` - Create dependency links
- `/spec archive [name]` - Archive completed specs
- `/spec template [type]` - Create spec from template
- `/spec search [query]` - Search spec content
## References
- Claude Slash commands: https://docs.anthropic.com/en/docs/claude-code/slash-commands
## Creating a command
Commands are implemented as Claude slash commands:
Location in repo: .claude/commands/
In the following example, we create the /optimize command:
```bash
# Create a project command
mkdir -p .claude/commands
echo "Analyze this code for performance issues and suggest optimizations:" > .claude/commands/optimize.md
```
-108
View File
@@ -1,108 +0,0 @@
---
title: 'SPEC-3: Agent Definitions'
type: spec
permalink: specs/spec-3-agent-definitions
tags:
- agents
- roles
- process
---
# SPEC-3: Agent Definitions
This document defines the specialist agents used in our specification-driven development process.
## system-architect
**Role**: High-level system design and architectural decisions
**Responsibilities**:
- Create architectural specifications and ADRs
- Analyze system-wide impacts and trade-offs
- Design component interfaces and data flow
- Evaluate technical approaches and patterns
- Document architectural decisions and rationale
**Expertise Areas**:
- System architecture and design patterns
- Technology evaluation and selection
- Scalability and performance considerations
- Integration patterns and API design
- Technical debt and refactoring strategies
**Typical Specs**:
- System architecture overviews
- Component decomposition strategies
- Data flow and state management
- Integration and deployment patterns
## vue-developer
**Role**: Frontend component development and UI implementation
**Responsibilities**:
- Create Vue.js component specifications
- Implement responsive UI components
- Design component APIs and interfaces
- Optimize for performance and accessibility
- Document component usage and patterns
**Expertise Areas**:
- Vue.js 3 Composition API
- Nuxt 3 framework patterns
- shadcn-vue component library
- Responsive design and CSS
- TypeScript integration
- State management with Pinia
**Typical Specs**:
- Individual component specifications
- UI pattern libraries
- Responsive design approaches
- Component interaction flows
## python-developer
**Role**: Backend development and API implementation
**Responsibilities**:
- Create backend service specifications
- Implement APIs and data processing
- Design database schemas and queries
- Optimize performance and reliability
- Document service interfaces and behavior
**Expertise Areas**:
- FastAPI and Python web frameworks
- Database design and operations
- API design and documentation
- Authentication and security
- Performance optimization
- Testing and validation
**Typical Specs**:
- API endpoint specifications
- Database schema designs
- Service integration patterns
- Performance optimization strategies
## Agent Collaboration Patterns
### Handoff Protocol
1. Agent receives spec through `/spec implement [name]`
2. Agent reviews spec and creates implementation plan
3. Agent documents progress and decisions in spec
4. Agent hands off to another agent if cross-domain work needed
5. Final agent updates spec with completion status
### Communication Standards
- All agents update specs through basic-memory MCP tools
- Document decisions and trade-offs in spec notes
- Link related specs and components
- Preserve context for future reference
### Quality Standards
- Follow existing codebase patterns and conventions
- Write tests that validate spec requirements
- Document implementation choices
- Consider maintainability and extensibility
-201
View File
@@ -1,201 +0,0 @@
---
title: 'SPEC-5: CLI Cloud Upload via WebDAV'
type: spec
permalink: specs/spec-5-cli-cloud-upload-via-webdav
tags:
- cli
- webdav
- upload
- migration
- poc
---
# SPEC-5: CLI Cloud Upload via WebDAV
## Why
Existing basic-memory users need a simple migration path to basic-memory-cloud. The web UI drag-and-drop approach outlined in GitHub issue #59, while user-friendly, introduces significant complexity for a proof-of-concept:
- Complex web UI components for file upload and progress tracking
- Browser file handling limitations and CORS complexity
- Proxy routing overhead for large file transfers
- Authentication integration across multiple services
A CLI-first approach solves these issues by:
- **Leveraging existing infrastructure**: Both cloud CLI and tenant API already exist with WorkOS JWT authentication
- **Familiar user experience**: Basic-memory users are CLI-comfortable and expect command-line tools
- **Direct connection efficiency**: Bypassing the MCP gateway/proxy for bulk file transfers
- **Rapid implementation**: Building on existing `CLIAuth` and FastAPI foundations
The fundamental problem is migration friction - users have local basic-memory projects but no path to cloud tenants. A simple CLI upload command removes this barrier immediately.
## What
This spec defines a CLI-based project upload system using WebDAV for direct tenant connections.
**Affected Areas:**
- `apps/cloud/src/basic_memory_cloud/cli/main.py` - Add upload command to existing CLI
- `apps/api/src/basic_memory_cloud_api/main.py` - Add WebDAV endpoints to tenant FastAPI
- Authentication flow - Reuse existing WorkOS JWT validation
- File transfer protocol - WebDAV for cross-platform compatibility
**Core Components:**
### CLI Upload Command
```bash
basic-memory-cloud upload <project-path> --tenant-url https://basic-memory-{tenant}.fly.dev
```
### WebDAV Server Endpoints
- `GET/PUT/DELETE /webdav/*` - Standard WebDAV operations on tenant file system
- Authentication via existing JWT validation
- File operations preserve timestamps and directory structure
### Authentication Flow
```
1. User runs `basic-memory-cloud login` (existing)
2. CLI stores WorkOS JWT token (existing)
3. Upload command reads JWT from storage
4. WebDAV requests include JWT in Authorization header
5. Tenant API validates JWT using existing middleware
```
## How (High Level)
### Implementation Strategy
**Phase 1: CLI Command**
- Add `upload` command to existing Typer app
- Reuse `CLIAuth` class for token management
- Implement WebDAV client using `webdavclient3` or similar
- Rich progress bars for transfer feedback
**Phase 2: WebDAV Server**
- Add WebDAV endpoints to existing tenant FastAPI app
- Leverage existing `get_current_user` dependency for authentication
- Map WebDAV operations to tenant file system
- Preserve file modification times using `os.utime()`
**Phase 3: Integration**
- Direct connection bypasses MCP gateway and proxy
- Simple conflict resolution: overwrite existing files
- Error handling: fail fast with clear error messages
### Technical Architecture
```
basic-memory-cloud CLI → WorkOS JWT → Direct WebDAV → Tenant FastAPI
Tenant File System
```
**Key Libraries:**
- CLI: `webdavclient3` for WebDAV client operations
- API: `wsgidav` or FastAPI-compatible WebDAV server
- Progress: `rich` library (already imported in CLI)
- Auth: Existing WorkOS JWT infrastructure
### WebDAV Protocol Choice
WebDAV provides:
- **Cross-platform clients**: Native support in most operating systems
- **Standardized protocol**: Well-defined for file operations
- **HTTP-based**: Works with existing FastAPI and JWT auth
- **Library support**: Good Python libraries for both client and server
### POC Constraints
**Simplifications for rapid implementation:**
- **Known tenant URLs**: Assume `https://basic-memory-{tenant}.fly.dev` format
- **Upload only**: No download or bidirectional sync
- **Overwrite conflicts**: No merge or conflict resolution prompting
- **No fallbacks**: Fail fast if WebDAV connection issues occur
- **Direct connection only**: No proxy fallback mechanism
## How to Evaluate
### Success Criteria
**Functional Requirements:**
- [ ] Transfer complete basic-memory project (100+ files) in < 30 seconds
- [ ] Preserve directory structure exactly as in source project
- [ ] Preserve file modification timestamps for proper sync behavior
- [ ] Rich progress bars show real-time transfer status (files/MB transferred)
- [ ] WorkOS JWT authentication validates correctly on WebDAV endpoints
- [ ] Direct tenant connection bypasses MCP gateway successfully
**Quality Requirements:**
- [ ] Clear error messages for authentication failures
- [ ] Graceful handling of network interruptions
- [ ] CLI follows existing command patterns and help text standards
- [ ] WebDAV endpoints integrate cleanly with existing FastAPI app
**Performance Requirements:**
- [ ] File transfer speed > 1MB/s on typical connections
- [ ] Memory usage remains reasonable for large projects
- [ ] No timeout issues with 500+ file projects
### Testing Procedure
**Unit Testing:**
1. CLI command parsing and argument validation
2. WebDAV client connection and authentication
3. File timestamp preservation during transfer
4. JWT token validation on WebDAV endpoints
**Integration Testing:**
1. End-to-end upload of test project
2. Direct tenant connection without proxy
3. File integrity verification after upload
4. Progress tracking accuracy during transfer
**User Experience Testing:**
1. Upload existing basic-memory project from local installation
2. Verify uploaded files appear correctly in cloud tenant
3. Confirm basic-memory database rebuilds properly with uploaded files
4. Test CLI help text and error message clarity
### Validation Commands
**Setup:**
```bash
# Login to WorkOS
basic-memory-cloud login
# Upload project
basic-memory-cloud upload ~/my-notes --tenant-url https://basic-memory-test.fly.dev
```
**Verification:**
```bash
# Check tenant health and file count via API
curl -H "Authorization: Bearer $JWT" https://basic-memory-test.fly.dev/health
curl -H "Authorization: Bearer $JWT" https://basic-memory-test.fly.dev/notes/search
```
### Performance Benchmarks
**Target metrics for 100MB basic-memory project:**
- Transfer time: < 30 seconds
- Memory usage: < 100MB during transfer
- Progress updates: Every 1MB or 10 files
- Authentication time: < 2 seconds
## Observations
- [implementation-speed] CLI approach significantly faster than web UI for POC development #rapid-prototyping
- [user-experience] Basic-memory users already comfortable with CLI tools #user-familiarity
- [architecture-benefit] Direct connection eliminates proxy complexity and latency #performance
- [auth-reuse] Existing WorkOS JWT infrastructure handles authentication cleanly #code-reuse
- [webdav-choice] WebDAV protocol provides cross-platform compatibility and standard libraries #protocol-selection
- [poc-scope] Simple conflict handling and error recovery sufficient for proof-of-concept #scope-management
- [migration-value] Removes primary barrier for local users migrating to cloud platform #business-value
## Relations
- depends_on [[SPEC-1: Specification-Driven Development Process]]
- enables [[GitHub Issue #59: Web UI Upload Feature]]
- uses [[WorkOS Authentication Integration]]
- builds_on [[Existing Cloud CLI Infrastructure]]
- builds_on [[Existing Tenant API Architecture]]
@@ -1,486 +0,0 @@
---
title: 'SPEC-6: Explicit Project Parameter Architecture'
type: spec
permalink: specs/spec-6-explicit-project-parameter-architecture
tags:
- architecture
- mcp
- project-management
- stateless
---
# SPEC-6: Explicit Project Parameter Architecture
## Why
The current session-based project management system has critical reliability issues:
1. **Session State Fragility**: Claude iOS mobile client fails to maintain consistent session IDs across MCP tool calls, causing project switching to silently fail (Issue #74)
2. **Scaling Limitations**: Redis-backed session state creates single-point-of-failure and prevents horizontal scaling
3. **Client Compatibility**: Session tracking works inconsistently across different MCP clients (web, mobile, API)
4. **Hidden Complexity**: Users cannot see or understand "current project" state, leading to confusion when operations execute in wrong projects
5. **Silent Failures**: Operations appear successful but execute in unintended projects, risking data integrity
Evidence from production logs shows each MCP tool call from mobile client receives different session IDs:
```
create_memory_project: session_id=12cdfc24913b48f8b680ed4b2bfdb7ba
switch_project: session_id=050a69275d98498cbdd227cdb74d9740
list_directory: session_id=85f3483014af4136a5d435c76ded212f
```
Related Github issue: https://github.com/basicmachines-co/basic-memory-cloud/issues/75
## Status
**Current Status**: **Phase 1 Implementation Complete**
**Target**: Fix Claude iOS session ID consistency issues
**Draft PR**: https://github.com/basicmachines-co/basic-memory/pull/298
### 🎉 **MAJOR MILESTONE ACHIEVED**
The complete stateless architecture has been successfully implemented for Basic Memory's MCP server! This represents a **fundamental architectural improvement** that solves the Claude iOS compatibility issue while making the entire system more robust and predictable.
#### Implementation Summary:
- **16 files modified** with 582 additions and 550 deletions
- **All 17 MCP tools** converted to stateless architecture
- **147 tests updated** across 5 test files (100% passing)
- **Complete session state removal** from core MCP tools
- **Enhanced error handling** and security validations preserved
### Progress Summary
**Complete Stateless Architecture Implementation (All 17 tools)**
- Stateless `get_active_project()` function implemented and deployed
- All session state dependencies removed across entire MCP server
- All MCP tools require explicit `project` parameter as first argument
**Content Management Tools Complete (6/6 tools)**
- `write_note`, `read_note`, `delete_note`, `edit_note`
- `view_note`, `read_content`
**Knowledge Graph Navigation Tools Complete (3/3 tools)**
- `build_context`, `recent_activity`, `list_directory`
**Search & Discovery Tools Complete (1/1 tools)**
- `search_notes`
**Visualization Tools Complete (1/1 tools)**
- `canvas`
**Project Management Cleanup Complete**
- Removed `switch_project` and `get_current_project` tools ✅
- Updated `set_default_project` to remove activate parameter ✅
**Comprehensive Testing Complete (157 tests)**
- All test suites updated to use stateless architecture (147 existing tests)
- Single project constraint mode integration tests (10 new tests)
- 100% test pass rate across all tool test files
- Security validations preserved and working
- Error handling comprehensive and user-friendly
**Documentation & Examples Complete**
- All tool docstrings updated with stateless examples
- Project parameter usage clearly documented
- Error handling and security behavior documented
**Enhanced Discovery Mode Complete**
- `recent_activity` tool supports dual-mode operation (discovery vs project-specific)
- ProjectActivitySummary schema provides cross-project insights
- Recent activity prompt updated to support both modes
- Comprehensive project distribution statistics and most active project tracking
**Single Project Constraint Mode Complete**
- `--project` CLI parameter for MCP server constraint
- Environment variable control (`BASIC_MEMORY_MCP_PROJECT`)
- Automatic project override in `get_active_project()` function
- Project management tools disabled in constrained mode with helpful CLI guidance
- Comprehensive integration test suite (10 tests covering all constraint scenarios)
## What
Transform Basic Memory from stateful session-based to stateless explicit project parameter architecture:
### Core Changes
1. **Mandatory Project Parameter**: All MCP tools require explicit `project` parameter
2. **Remove Session State**: Eliminate Redis, session middleware, and `switch_project` tool
3. **Stateless HTTP**: Enable `stateless_http=True` for horizontal scaling
4. **Enhanced Context Discovery**: Improve `recent_activity` to show project distribution
5. **Clear Response Format**: All tool responses display target project information
Implementation Approach
- Each tool will directly accept the project parameter
- Remove all calls to context-based project retrieval
- Validate project exists before operations
- Clear error messages when project not found
- Backward compatibility: Initially keep optional parameter, then make required
### Affected MCP Tools
**Content Management** (require project parameter):
- `write_note(project, title, content, folder)`
- `read_note(project, identifier)`
- `edit_note(project, identifier, operation, content)`
- `delete_note(project, identifier)`
- `view_note(project, identifier)`
- `read_content(project, path)`
**Knowledge Graph Navigation** (require project parameter):
- `build_context(project, url, timeframe, depth, max_related)`
- `list_directory(project, dir_name, depth, file_name_glob)`
- `search_notes(project, query, search_type, types, entity_types)`
**Search & Discovery** (use project parameter for specific project or none for discovery):
- `recent_activity(project, timeframe, depth, max_related)`
**Visualization** (require project parameter):
- `canvas(project, nodes, edges, title, folder)`
**Project Management** (unchanged - already stateless):
- `list_memory_projects()`
- `create_memory_project(project_name, project_path, set_default)`
- `delete_project(project_name)`
- `get_current_project()` - Remove this tool
- `switch_project(project_name)` - Remove this tool
- `set_default_project(project_name, activate)` - Remove activate parameter
## How (High Level)
### Phase 1: Basic Memory Core (basic-memory repository)
#### MCP Tool Updates
Phase 1: Core Changes
1. Update project_context.py
- [x] Make project parameter mandatory for get_active_project()
- [x] Remove session state handling
2. Update Content Management Tools (6 tools)
- [x] write_note: Make project parameter required, not optional
- [x] read_note: Make project parameter required
- [x] edit_note: Add required project parameter
- [x] delete_note: Add required project parameter
- [x] view_note: Add required project parameter
- [x] read_content: Add required project parameter
3. Update Knowledge Graph Navigation Tools (3 tools)
- [x] build_context: Add required project parameter
- [x] recent_activity: Make project parameter required
- [x] list_directory: Add required project parameter
4. Update Search & Visualization Tools (2 tools)
- [x] search_notes: Add required project parameter
- [x] canvas: Add required project parameter
5. Update Project Management Tools
- [x] Remove switch_project tool completely
- [x] Remove get_current_project tool completely
- [x] Update set_default_project to remove activate parameter
- [x] Keep list_memory_projects, create_memory_project, delete_project unchanged
6. Enhance recent_activity Response
- [x] Add project distribution info showing activity across all projects
- [x] Include project usage stats in response
- [x] Implement ProjectActivitySummary for discovery mode
- [x] Add dual-mode functionality (discovery vs project-specific)
7. Update Tool Documentation
- [x] Update write_note docstring with stateless architecture examples
- [x] Update read_note docstring with project parameter examples
- [x] Update delete_note docstring with comprehensive usage guidance
- [x] Update all remaining tool docstrings with project parameter examples
8. Update Tool Responses
- [x] Add clear project indicator to all tool responses across all tools
- [x] Format: "project: {project_name}" in response metadata
- [x] Add project metadata footer for LLM awareness
- [x] Update all tool responses to include project indicators
9. Comprehensive Testing
- [x] Update all write_note tests to use stateless architecture (34 tests passing)
- [x] Update all edit_note tests to use stateless architecture (17 tests passing)
- [x] Update all view_note tests to use stateless architecture (12 tests passing)
- [x] Update all search_notes tests to use stateless architecture (16 tests passing)
- [x] Update all move_note tests to use stateless architecture (31 tests passing)
- [x] Update all delete_note tests to use stateless architecture
- [x] Verify direct function call compatibility (bypassing MCP layer)
- [x] Test security validation with project parameters
- [x] Validate error handling for non-existent projects
- [x] **Total: 157 tests updated and passing (100% success rate)**
- [x] **147 existing tests** updated for stateless architecture
- [x] **10 new tests** for single project constraint mode
### Phase 1.5: Default Project Mode Enhancement
#### Problem
While the stateless architecture solves reliability issues, it introduces UX friction for single-project users (estimated 80% of usage) who must specify the project parameter in every tool call.
#### Solution: Default Project Mode
Add optional `default_project_mode` configuration that allows single-project users to have the simplicity of implicit project selection while maintaining the reliability of stateless architecture.
#### Configuration
```json
{
"default_project": "main",
"default_project_mode": true // NEW: Auto-use default_project when not specified
}
```
#### Implementation Details
1. **Config Enhancement** (`src/basic_memory/config.py`)
- Add `default_project_mode: bool = Field(default=False)`
- Preserves backward compatibility (defaults to false)
2. **Project Resolution Logic** (`src/basic_memory/mcp/project_context.py`)
Three-tier resolution hierarchy:
- Priority 1: CLI `--project` constraint (BASIC_MEMORY_MCP_PROJECT env var)
- Priority 2: Explicit project parameter in tool call
- Priority 3: `default_project` if `default_project_mode=true` and no project specified
3. **Assistant Guide Updates** (`src/basic_memory/mcp/resources/ai_assistant_guide.md`)
- Detect `default_project_mode` at runtime
- Provide mode-specific instructions to LLMs
- In default mode: "All operations use project 'main' automatically"
- In regular mode: Current project discovery guidance
4. **Tool Parameter Handling** (all MCP tools)
- Make project parameter Optional[str] = None
- Add resolution logic: `project = project or get_default_project()`
- Maintain explicit project override capability
#### Usage Modes Summary
- **Regular Mode**: Multi-project users, assistant tracks project per conversation
- **Default Project Mode**: Single-project users, automatic default project
- **Constrained Mode**: CLI --project flag, locked to specific project
#### Testing Requirements
- Integration test for default_project_mode=true with missing parameters
- Test explicit project override in default_project_mode
- Test mode=false requires explicit parameters
- Test CLI constraint overrides default_project_mode
Phase 2: Testing & Validation
8. Update Tests
- [x] Modify all MCP tool tests to pass required project parameter
- [x] Remove tests for deleted tools (switch_project, get_current_project)
- [x] Add tests for project parameter validation
- [x] **Complete: All 147 tests across 5 test files updated and passing**
#### Enhanced recent_activity Response
```json
{
"recent_notes": [...],
"project_activity": {
"research-project": {
"operations": 5,
"last_used": "30 minutes ago",
"recent_folders": ["experiments", "findings"]
},
"work-notes": {
"operations": 2,
"last_used": "2 hours ago",
"recent_folders": ["meetings", "planning"]
}
},
"total_projects": 3
}
```
#### Response Format Updates
```
✓ Note created successfully
Project: research-project
File: experiments/Neural Network Results.md
Permalink: research-project/neural-network-results
```
### Phase 2: Cloud Service Simplification (basic-memory-cloud repository)
#### Remove Session Infrastructure
1. Delete `apps/mcp/src/basic_memory_cloud_mcp/middleware/session_state.py`
2. Delete `apps/mcp/src/basic_memory_cloud_mcp/middleware/session_logging.py`
3. Update `apps/mcp/src/basic_memory_cloud_mcp/main.py`:
```python
# Remove session middleware
# server.add_middleware(SessionStateMiddleware)
# Enable stateless HTTP
mcp = FastMCP(name="basic-memory-mcp", stateless_http=True)
```
#### Deployment Simplification
1. Remove Redis from `fly.toml`
2. Remove Redis environment variables
3. Update health checks to not depend on Redis
### Phase 3: Conversational Project Management
#### Claude Behavior Pattern
1. **Project Discovery**:
```
Claude: Let me check your recent activity...
[calls recent_activity() - no project needed for discovery]
I see you've been working in:
- research-project (5 operations, 30 min ago)
- work-notes (2 operations, 2 hours ago)
Which project should I use for this operation?
```
2. **Context Maintenance**:
```
User: Use research-project
Claude: Working in research-project.
[All subsequent operations use project="research-project"]
```
3. **Explicit Project Switching**:
```
User: Check work-notes for that meeting summary
Claude: Let me search work-notes for the meeting summary.
[Uses project="work-notes" for specific operation]
```
## How to Evaluate
### Success Criteria
#### 1. Functional Completeness
- [x] All MCP tools accept required `project` parameter
- [x] All MCP tools validate project exists before execution
- [x] `switch_project` and `get_current_project` tools removed
- [x] All responses display target project clearly
- [ ] No Redis dependencies in deployment (Phase 2: Cloud Service)
- [x] `recent_activity` shows project distribution with ProjectActivitySummary
#### 2. Cross-Client Compatibility Testing
Test identical operations across all clients:
- [ ] **Claude Desktop**: All operations work with explicit projects
- [ ] **Claude Code**: All operations work with explicit projects
- [ ] **Claude Mobile iOS**: All operations work with explicit projects
- [ ] **API clients**: All operations work with explicit projects
- [ ] **CLI tools**: All operations work with explicit projects
#### 3. Session Independence Verification
- [ ] Operations work identically with/without session tracking
- [ ] No behavioral differences between clients
- [ ] Mobile client session ID changes do not affect operations
- [ ] Redis can be completely removed without functional impact
#### 4. Performance & Scaling
- [ ] `stateless_http=True` enabled successfully
- [ ] No Redis memory usage
- [ ] Horizontal scaling possible (multiple MCP instances)
- [ ] Response times unchanged or improved
#### 5. User Experience Testing
**Project Discovery Flow**:
- [x] `recent_activity()` provides useful project context
- [x] Claude can intelligently suggest projects based on activity
- [x] Project switching is explicit and clear in conversation
**Error Handling**:
- [x] Clear error messages for non-existent projects
- [x] Helpful suggestions when project parameter missing
- [x] No silent failures or wrong-project operations
**Response Clarity**:
- [x] Every operation clearly shows target project
- [x] Users always know which project is being operated on
- [x] No confusion about "current project" state
#### 6. Migration Safety
- [ ] Backward compatibility period with optional project parameter
- [ ] Clear migration documentation for existing users
- [ ] Data integrity maintained during transition
- [ ] No data loss during migration
### Test Scenarios
#### Core Functionality Test
```bash
# Test all tools work with explicit project
write_note(project="test-proj", title="Test", content="Content", folder="docs")
read_note(project="test-proj", identifier="Test")
edit_note(project="test-proj", identifier="Test", operation="append", content="More")
search_notes(project="test-proj", query="Content")
list_directory(project="test-proj", dir_name="docs")
delete_note(project="test-proj", identifier="Test")
```
#### Cross-Client Consistency Test
Run identical test sequence on:
1. Claude Desktop
2. Claude Code
3. Claude Mobile iOS
4. API client
5. CLI tools
Verify all clients:
- Accept explicit project parameters
- Return identical responses
- Show same project information
- Have no session dependencies
#### Session Independence Test
1. Monitor session IDs during operations
2. Verify operations work with changing session IDs
3. Confirm Redis removal doesn't affect functionality
4. Test with multiple concurrent clients
### Acceptance Criteria
**Must Have**:
- All MCP tools require and use explicit project parameter
- No session state dependencies remain
- Universal client compatibility achieved
- Clear project information in all responses
**Should Have**:
- Enhanced `recent_activity` with project distribution
- Smooth migration path for existing users
- Improved performance with stateless architecture
**Could Have**:
- Smart project suggestions based on content/context
- Project shortcuts for common operations
- Advanced project analytics in responses
## Notes
### Breaking Changes
This is a **breaking change** that requires:
- All MCP clients to pass project parameter
- Migration of existing workflows
- Update of all documentation and examples
### Implementation Order
1. **basic-memory core** - Update MCP tools to accept project parameter (optional initially)
2. **Testing** - Verify all clients work with explicit projects
3. **Cloud service** - Remove session infrastructure
4. **Migration** - Make project parameter mandatory
5. **Cleanup** - Remove deprecated tools and middleware
### Related Issues
- Fixes #74 (Claude iOS session state bug)
- Implements #75 (Mandatory project parameter architecture)
- Enables future horizontal scaling
- Simplifies multi-tenant architecture
### Dependencies
- Requires coordination between basic-memory and basic-memory-cloud repositories
- Needs client-side updates for smooth transition
- Documentation updates across all materials
@@ -1,193 +0,0 @@
---
title: 'SPEC-7: POC to spike Tigris/Turso for local access to cloud data'
type: spec
permalink: specs/spec-7-poc-tigris-turso-local-access-cloud-data
tags:
- poc
- tigris
- turso
- cloud-storage
- architecture
- proof-of-concept
---
# SPEC-7: POC to spike Tigris/Turso for local access to cloud data
## Why
Current basic-memory-cloud architecture uses Fly volumes for tenant file storage, which creates several limitations:
1. **Storage Scalability**: Fly volumes require pre-provisioning and don't auto-scale with usage
2. **Cost Model**: Volume pricing vs object storage pricing may be less favorable at scale
3. **Local Development**: No way for users to mount their cloud tenant files locally for real-time editing
4. **Multi-Region**: Volumes are region-locked, limiting global deployment flexibility
5. **Backup/Disaster Recovery**: Object storage provides better durability and replication options
The core insight is that Basic Memory requires POSIX filesystem semantics but could benefit from object storage durability and accessibility. By combining:
- **Tigris object storage** for file persistence (via rclone mount)
- **Turso/libSQL** for SQLite indexing (replacing local .db files)
We could enable a revolutionary user experience: **local editing of cloud-stored files** while maintaining Basic Memory's existing filesystem assumptions.
## What
This specification defines a proof-of-concept to validate the technical feasibility of the Tigris/Turso architecture for basic-memory-cloud tenants.
**Affected Areas:**
- **Storage Architecture**: Replace Fly volumes with Tigris object storage
- **Database Architecture**: Replace local SQLite with Turso remote database
- **Container Setup**: Add rclone mounting in tenant containers
- **Local Development**: Enable local mounting of cloud tenant data
- **Basic Memory Core**: Validate unchanged operation over mounted filesystems
**Key Components:**
- **Tigris Storage**: S3-compatible object storage via Fly.io integration
- **rclone NFS Mount**: Native NFS mounting without FUSE dependencies
- **Turso Database**: Hosted libSQL for SQLite replacement
- **Single-Tenant Model**: One bucket + one database per tenant (simplified isolation)
## How (High Level)
### Phase 1: Local POC Validation
- [ ] Set up Tigris bucket with test data
- [ ] Configure rclone NFS mount locally
- [ ] Test Basic Memory operations over mounted filesystem
- [ ] Measure performance characteristics and identify issues
- [ ] Validate file watching, sync operations, and concurrent access patterns
### Phase 2: Database Migration
- [ ] Set up Turso account and test database
- [ ] Modify Basic Memory to accept external DATABASE_URL
- [ ] Test all operations with remote SQLite via Turso
- [ ] Validate performance and functionality parity
### Phase 3: Container Integration
- [ ] Create container image with rclone + NFS support
- [ ] Implement tenant-specific credential management
- [ ] Test container startup with automatic mounting
- [ ] Validate isolation between tenant containers
### Phase 4: Local Access Validation
- [ ] Test local rclone mounting of tenant data
- [ ] Validate real-time file editing experience
- [ ] Test conflict resolution and sync behavior
- [ ] Measure latency impact on user experience
### Architecture Overview
```
Local Development:
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Local rclone │───▶│ Tigris Bucket │◀───│ Tenant Container│
│ NFS Mount │ │ (S3 storage) │ │ rclone mount │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ Basic Memory │ │ Basic Memory │
│ (local files) │ │ (mounted files) │
└─────────────────┘ └─────────────────┘
│ │
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ Turso Database │◀───────────────────────────│ Turso Database │
│ (shared index) │ │ (shared index) │
└─────────────────┘ └─────────────────┘
```
## How to Evaluate
### Success Criteria
- [ ] **Filesystem Compatibility**: Basic Memory operates without modification over rclone-mounted Tigris storage
- [ ] **Performance Acceptable**: File operations complete within 2x local filesystem latency
- [ ] **Database Functionality**: All Basic Memory features work with Turso remote SQLite
- [ ] **Container Reliability**: Tenant containers start successfully with automatic mounting
- [ ] **Local Access**: Users can mount and edit cloud files locally with real-time sync
- [ ] **Data Isolation**: Tenant data remains properly isolated using bucket/database separation
### Testing Procedure
1. **Local Filesystem Test**:
```bash
# Mount Tigris bucket locally
rclone nfsmount tigris:test-bucket ~/tigris-test --vfs-cache-mode writes
# Run Basic Memory operations
cd ~/tigris-test && basic-memory sync --watch
# Test: create notes, search, file watching, bulk operations
```
2. **Database Migration Test**:
```bash
# Configure Turso connection
export DATABASE_URL="libsql://test-db.turso.io?authToken=..."
# Test all MCP tools with remote database
basic-memory tools # Test each tool functionality
```
3. **Container Integration Test**:
```dockerfile
# Test container with rclone mounting
FROM python:3.12
RUN apt-get update && apt-get install -y rclone nfs-common
# ... test startup and mounting process
```
4. **Performance Benchmarking**:
- File creation/read/write operations (target: <2x local latency)
- Search query performance (target: comparable to local SQLite)
- File watching responsiveness (target: events within 1-2 seconds)
- Concurrent operation handling
### Risk Assessment
**High Risk Items**:
- [ ] NFS-over-S3 performance may be insufficient for real-time operations
- [ ] File watching (`inotify`) over NFS may be unreliable
- [ ] Network interruptions could cause filesystem errors
- [ ] Concurrent access patterns might hit S3 rate limits
**Mitigation Strategies**:
- Comprehensive performance testing before committing to architecture
- Fallback plan to S3-native storage backend if filesystem approach fails
- Extensive error handling and retry logic for network issues
### Metrics to Track
- **Latency**: File operation response times (read/write/watch)
- **Reliability**: Success rate of file operations over time
- **Throughput**: Concurrent file operations and search queries
- **User Experience**: Perceived performance for local mounting use case
## Notes
### Key Architectural Decisions
- **Single tenant per bucket/database**: Simplifies isolation and credential management
- **Maintain POSIX compatibility**: Preserve Basic Memory's existing filesystem assumptions
- **NFS over FUSE**: Better compatibility and performance characteristics
- **Turso for SQLite**: Leverages specialized remote SQLite expertise
### Alternative Approaches Considered
- **S3-native storage backend**: Would require Basic Memory architecture changes
- **Hybrid approach**: Local files + cloud sync (adds complexity)
- **FUSE mounting**: More platform dependencies and kernel requirements
### Integration Points
- [ ] Fly.io Tigris integration for bucket provisioning
- [ ] Turso account setup and database provisioning
- [ ] Container image modifications for rclone support
- [ ] Credential management for tenant isolation
## Observations
- [architecture] Tigris/Turso split cleanly separates file storage from indexing concerns #storage-separation
- [user-experience] Local mounting of cloud files could be revolutionary for knowledge management #local-cloud-hybrid
- [compatibility] Maintaining POSIX filesystem assumptions preserves Basic Memory's local/cloud compatibility #architecture-preservation
- [simplification] Single tenant per bucket eliminates complex multi-tenancy in storage layer #tenant-isolation
- [risk] NFS-over-S3 performance characteristics are unproven for real-time operations #performance-risk
- [benefit] Object storage pricing model could be more favorable than volume pricing #cost-optimization
- [innovation] Real-time local editing of cloud-stored files addresses major SaaS limitation #competitive-advantage
## Relations
- implements [[SPEC-6 Explicit Project Parameter Architecture]]
- requires [[Fly.io Tigris Integration]]
- enables [[Local Cloud File Access]]
- alternative_to [[Fly Volume Storage]]
-886
View File
@@ -1,886 +0,0 @@
---
title: 'SPEC-8: TigrisFS Integration for Tenant API'
Date: September 22, 2025
Status: Phase 3.6 Complete - Tenant Mount API Endpoints Ready for CLI Implementation
Priority: High
Goal: Replace Fly volumes with Tigris bucket provisioning in production tenant API
permalink: spec-8-tigris-fs-integration
---
## Executive Summary
Based on SPEC-7 Phase 4 POC testing, this spec outlines productizing the TigrisFS/rclone implementation in the Basic Memory Cloud tenant API.
We're moving from proof-of-concept to production integration, replacing Fly volume storage with Tigris bucket-per-tenant architecture.
## Current Architecture (Fly Volumes)
### Tenant Provisioning Flow
```python
# apps/cloud/src/basic_memory_cloud/workflows/tenant_provisioning.py
async def provision_tenant_infrastructure(tenant_id: str):
# 1. Create Fly app
# 2. Create Fly volume ← REPLACE THIS
# 3. Deploy API container with volume mount
# 4. Configure health checks
```
### Storage Implementation
- Each tenant gets dedicated Fly volume (1GB-10GB)
- Volume mounted at `/app/data` in API container
- Local filesystem storage with Basic Memory indexing
- No global caching or edge distribution
## Proposed Architecture (Tigris Buckets)
### New Tenant Provisioning Flow
```python
async def provision_tenant_infrastructure(tenant_id: str):
# 1. Create Fly app
# 2. Create Tigris bucket with admin credentials ← NEW
# 3. Store bucket name in tenant record ← NEW
# 4. Deploy API container with TigrisFS mount using admin credentials
# 5. Configure health checks
```
### Storage Implementation
- Each tenant gets dedicated Tigris bucket
- TigrisFS mounts bucket at `/app/data` in API container
- Global edge caching and distribution
- Configurable cache TTL for sync performance
## Implementation Plan
### Phase 1: Bucket Provisioning Service
**✅ IMPLEMENTED: StorageClient with Admin Credentials**
```python
# apps/cloud/src/basic_memory_cloud/clients/storage_client.py
class StorageClient:
async def create_tenant_bucket(self, tenant_id: UUID) -> TigrisBucketCredentials
async def delete_tenant_bucket(self, tenant_id: UUID, bucket_name: str) -> bool
async def list_buckets(self) -> list[TigrisBucketResponse]
async def test_tenant_credentials(self, credentials: TigrisBucketCredentials) -> bool
```
**Simplified Architecture Using Admin Credentials:**
- Single admin access key with full Tigris permissions (configured in console)
- No tenant-specific IAM user creation needed
- Bucket-per-tenant isolation for logical separation
- Admin credentials shared across all tenant operations
**Integrate with Provisioning workflow:**
```python
# Update tenant_provisioning.py
async def provision_tenant_infrastructure(tenant_id: str):
storage_client = StorageClient(settings.aws_access_key_id, settings.aws_secret_access_key)
bucket_creds = await storage_client.create_tenant_bucket(tenant_id)
await store_bucket_name(tenant_id, bucket_creds.bucket_name)
await deploy_api_with_tigris(tenant_id, bucket_creds)
```
### Phase 2: Simplified Bucket Management
**✅ SIMPLIFIED: Admin Credentials + Bucket Names Only**
Since we use admin credentials for all operations, we only need to track bucket names per tenant:
1. **Primary Storage (Fly Secrets)**
```bash
flyctl secrets set -a basic-memory-{tenant_id} \
AWS_ACCESS_KEY_ID="{admin_access_key}" \
AWS_SECRET_ACCESS_KEY="{admin_secret_key}" \
AWS_ENDPOINT_URL_S3="https://fly.storage.tigris.dev" \
AWS_REGION="auto" \
BUCKET_NAME="basic-memory-{tenant_id}"
```
2. **Database Storage (Bucket Name Only)**
```python
# apps/cloud/src/basic_memory_cloud/models/tenant.py
class Tenant(BaseModel):
# ... existing fields
tigris_bucket_name: Optional[str] = None # Just store bucket name
tigris_region: str = "auto"
created_at: datetime
```
**Benefits of Simplified Approach:**
- No credential encryption/decryption needed
- Admin credentials managed centrally in environment
- Only bucket names stored in database (not sensitive)
- Simplified backup/restore scenarios
- Reduced security attack surface
### Phase 3: API Container Updates
**Update API container configuration:**
```dockerfile
# apps/api/Dockerfile
# Add TigrisFS installation
RUN curl -L https://github.com/tigrisdata/tigrisfs/releases/latest/download/tigrisfs-linux-amd64 \
-o /usr/local/bin/tigrisfs && chmod +x /usr/local/bin/tigrisfs
```
**Startup script integration:**
```bash
# apps/api/tigrisfs-startup.sh (already exists)
# Mount TigrisFS → Start Basic Memory API
exec python -m basic_memory_cloud_api.main
```
**Fly.toml environment (optimized for < 5s startup):**
```toml
# apps/api/fly.tigris-production.toml
[env]
TIGRISFS_MEMORY_LIMIT = '1024' # Reduced for faster init
TIGRISFS_MAX_FLUSHERS = '16' # Fewer threads for faster startup
TIGRISFS_STAT_CACHE_TTL = '30s' # Balance sync speed vs startup
TIGRISFS_LAZY_INIT = 'true' # Enable lazy loading
BASIC_MEMORY_HOME = '/app/data'
# Suspend optimization for wake-on-network
[machine]
auto_stop_machines = "suspend" # Faster than full stop
auto_start_machines = true
min_machines_running = 0
```
### Phase 4: Local Access Features
**CLI automation for local mounting:**
```python
# New CLI command: basic-memory cloud mount
async def setup_local_mount(tenant_id: str):
# 1. Fetch bucket credentials from cloud API
# 2. Configure rclone with scoped IAM policy
# 3. Mount via rclone nfsmount (macOS) or FUSE (Linux)
# 4. Start Basic Memory sync watcher
```
**Local mount configuration:**
```bash
# rclone config for tenant
rclone mount basic-memory-{tenant_id}: ~/basic-memory-{tenant_id} \
--nfs-mount \
--vfs-cache-mode writes \
--cache-dir ~/.cache/rclone/basic-memory-{tenant_id}
```
### Phase 5: TigrisFS Cache Sync Solutions
**Problem**: When files are uploaded via CLI/bisync, the tenant API container doesn't see them immediately due to TigrisFS cache (30s TTL) and lack of inotify events on mounted filesystems.
**Multi-Layer Solution:**
**Layer 1: API Sync Endpoint** (Immediate)
```python
# POST /sync - Force TigrisFS cache refresh
# Callable by CLI after uploads
subprocess.run(["sync", "fsync /app/data"], check=True)
```
**Layer 2: Tigris Webhook Integration** (Real-time)
https://www.tigrisdata.com/docs/buckets/object-notifications/#webhook
```python
# Webhook endpoint for bucket changes
@app.post("/webhooks/tigris/{tenant_id}")
async def handle_bucket_notification(tenant_id: str, event: TigrisEvent):
if event.eventName in ["OBJECT_CREATED_PUT", "OBJECT_DELETED"]:
await notify_container_sync(tenant_id, event.object.key)
```
**Layer 3: CLI Sync Notification** (User-triggered)
```bash
# CLI calls container sync endpoint after successful bisync
basic-memory cloud bisync # Automatically notifies container
curl -X POST https://basic-memory-{tenant-id}.fly.dev/sync
```
**Layer 4: Periodic Sync Fallback** (Safety net)
```python
# Background task: fsync /app/data every 30s as fallback
# Ensures eventual consistency even if other layers fail
```
**Implementation Priority:**
1. Layer 1 (API endpoint) - Quick testing capability
2. Layer 3 (CLI integration) - Improved UX
3. Layer 4 (Periodic fallback) - Safety net
4. Layer 2 (Webhooks) - Production real-time sync
## Performance Targets
### Sync Latency
- **Target**: < 5 seconds local→cloud→container
- **Configuration**: `TIGRISFS_STAT_CACHE_TTL = '5s'`
- **Monitoring**: Track sync metrics in production
### Container Startup
- **Target**: < 5 seconds including TigrisFS mount
- **Fast retry**: 0.5s intervals for mount verification
- **Fallback**: Container fails fast if mount fails
### Memory Usage
- **TigrisFS cache**: 2GB memory limit per container
- **Concurrent uploads**: 32 flushers max
- **VM sizing**: shared-cpu-2x (2048mb) minimum
## Security Considerations
### Bucket Isolation
- Each tenant has dedicated bucket
- IAM policies prevent cross-tenant access
- No shared bucket with subdirectories
### Credential Security
- Fly secrets for runtime access
- Encrypted database backup for disaster recovery
- Credential rotation capability
### Data Residency
- Tigris global edge caching
- SOC2 Type II compliance
- Encryption at rest and in transit
## Operational Benefits
### Scalability
- Horizontal scaling with stateless API containers
- Global edge distribution
- Better resource utilization
### Reliability
- No cold starts between tenants
- Built-in redundancy and caching
- Simplified backup strategy
### Cost Efficiency
- Pay-per-use storage pricing
- Shared infrastructure benefits
- Reduced operational overhead
## Risk Mitigation
### Data Loss Prevention
- Dual credential storage (Fly + database)
- Automated backup workflows to R2/S3
- Tigris built-in redundancy
### Performance Degradation
- Configurable cache settings per tenant
- Monitoring and alerting on sync latency
- Fallback to volume storage if needed
### Security Vulnerabilities
- Bucket-per-tenant isolation
- Regular credential rotation
- Security scanning and monitoring
## Success Metrics
### Technical Metrics
- Sync latency P50 < 5 seconds
- Container startup time < 5 seconds
- Zero data loss incidents
- 99.9% uptime per tenant
### Business Metrics
- Reduced infrastructure costs vs volumes
- Improved user experience with faster sync
- Enhanced enterprise security posture
- Simplified operational overhead
## Open Questions
1. **Tigris rate limits**: What are the API limits for bucket creation?
2. **Cost analysis**: What's the break-even point vs Fly volumes?
3. **Regional preferences**: Should enterprise customers choose regions?
4. **Backup retention**: How long to keep automated backups?
## Implementation Checklist
### Phase 1: Bucket Provisioning Service ✅ COMPLETED
- [x] **Research Tigris bucket API** - Document bucket creation and S3 API compatibility
- [x] **Create StorageClient class** - Implemented with admin credentials and comprehensive integration tests
- [x] **Test bucket creation** - Full test suite validates API integration with real Tigris environment
- [x] **Add bucket provisioning to DBOS workflow** - Integrated StorageClient with tenant_provisioning.py
### Phase 2: Simplified Bucket Management ✅ COMPLETED
- [x] **Update Tenant model** with tigris_bucket_name field (replaced fly_volume_id)
- [x] **Implement bucket name storage** - Database migration and model updates completed
- [x] **Test bucket provisioning integration** - Full test suite validates workflow from tenant creation to bucket assignment
- [x] **Remove volume logic from all tests** - Complete migration from volume-based to bucket-based architecture
### Phase 3: API Container Integration ✅ COMPLETED
- [x] **Update Dockerfile** to install TigrisFS binary in API container with configurable version
- [x] **Optimize tigrisfs-startup.sh** with production-ready security and reliability improvements
- [x] **Create production-ready container** with proper signal handling and mount validation
- [x] **Implement security fixes** based on Claude code review (conditional debug, credential protection)
- [x] **Add proper process supervision** with cleanup traps and error handling
- [x] **Remove debug artifacts** - Cleaned up all debug Dockerfiles and test scripts
### Phase 3.5: IAM Access Key Management ✅ COMPLETED
- [x] **Research Tigris IAM API** - Documented create_policy, attach_user_policy, delete_access_key operations
- [x] **Implement bucket-scoped credential generation** - StorageClient.create_tenant_access_keys() with IAM policies
- [x] **Add comprehensive security test suite** - 5 security-focused integration tests covering all attack vectors
- [x] **Verify cross-bucket access prevention** - Scoped credentials can ONLY access their designated bucket
- [x] **Test credential lifecycle management** - Create, validate, delete, and revoke access keys
- [x] **Validate admin vs scoped credential isolation** - Different access patterns and security boundaries
- [x] **Test multi-tenant isolation** - Multiple tenants cannot access each other's buckets
### Phase 3.6: Tenant Mount API Endpoints ✅ COMPLETED
- [x] **Implement GET /tenant/mount/info** - Returns mount info without exposing credentials
- [x] **Implement POST /tenant/mount/credentials** - Creates new bucket-scoped credentials for CLI mounting
- [x] **Implement DELETE /tenant/mount/credentials/{cred_id}** - Revoke specific credentials with proper cleanup
- [x] **Implement GET /tenant/mount/credentials** - List active credentials without exposing secrets
- [x] **Add TenantMountCredentials database model** - Tracks credential metadata (no secret storage)
- [x] **Create comprehensive test suite** - 28 tests covering all scenarios including multi-session support
- [x] **Implement multi-session credential flow** - Multiple active credentials per tenant supported
- [x] **Secure credential handling** - Secret keys never stored, returned once only for immediate use
- [x] **Add dependency injection for StorageClient** - Clean integration with existing API architecture
- [x] **Fix Tigris configuration for cloud service** - Added AWS environment variables to fly.template.toml
- [x] **Update tenant machine configurations** - Include AWS credentials for TigrisFS mounting with clear credential strategy
**Security Test Results:**
```
✅ Cross-bucket access prevention - PASS
✅ Deleted credentials access revoked - PASS
✅ Invalid credentials rejected - PASS
✅ Admin vs scoped credential isolation - PASS
✅ Multiple scoped credentials isolation - PASS
```
**Implementation Details:**
- Uses Tigris IAM managed policies (create_policy + attach_user_policy)
- Bucket-scoped S3 policies with Actions: GetObject, PutObject, DeleteObject, ListBucket
- Resource ARNs limited to specific bucket: `arn:aws:s3:::bucket-name` and `arn:aws:s3:::bucket-name/*`
- Access keys follow Tigris format: `tid_` prefix with secure random suffix
- Complete cleanup on deletion removes both access keys and associated policies
### Phase 4: Local Access CLI
- [x] **Design local mount CLI command** for automated rclone configuration
- [x] **Implement credential fetching** from cloud API for local setup
- [x] **Create rclone config automation** for tenant-specific bucket mounting
- [x] **Test local→cloud→container sync** with optimized cache settings
- [x] **Document local access setup** for beta users
### Phase 5: Webhook Integration (Future)
- [ ] **Research Tigris webhook API** for object notifications and payload format
- [ ] **Design webhook endpoint** for real-time sync notifications
- [ ] **Implement notification handling** to trigger Basic Memory sync events
- [ ] **Test webhook delivery** and sync latency improvements
## Success Metrics
- [ ] **Container startup < 5 seconds** including TigrisFS mount and Basic Memory init
- [ ] **Sync latency < 5 seconds** for local→cloud→container file changes
- [ ] **Zero data loss** during bucket provisioning and credential management
- [ ] **100% test coverage** for new TigrisBucketService and credential functions
- [ ] **Beta deployment** with internal users validating local-cloud workflow
## Implementation Notes
## Phase 4.1: Bidirectional Sync with rclone bisync (NEW)
### Problem Statement
During testing, we discovered that some applications (particularly Obsidian) don't detect file changes over NFS mounts. Rather than building a custom sync daemon, we can leverage `rclone bisync` - rclone's built-in bidirectional synchronization feature.
### Solution: rclone bisync
Use rclone's proven bidirectional sync instead of custom implementation:
**Core Architecture:**
```bash
# rclone bisync handles all the complexity
rclone bisync ~/basic-memory-{tenant_id} basic-memory-{tenant_id}:{bucket_name} \
--create-empty-src-dirs \
--conflict-resolve newer \
--resilient \
--check-access
```
**Key Benefits:**
- ✅ **Battle-tested**: Production-proven rclone functionality
- ✅ **MIT licensed**: Open source with permissive licensing
- ✅ **No custom code**: Zero maintenance burden for sync logic
- ✅ **Built-in safety**: max-delete protection, conflict resolution
- ✅ **Simple installation**: Works with Homebrew rclone (no FUSE needed)
- ✅ **File watcher compatible**: Works with Obsidian and all applications
- ✅ **Offline support**: Can work offline and sync when connected
### bisync Conflict Resolution Options
**Built-in conflict strategies:**
```bash
--conflict-resolve none # Keep both files with .conflict suffixes (safest)
--conflict-resolve newer # Always pick the most recently modified file
--conflict-resolve larger # Choose based on file size
--conflict-resolve path1 # Always prefer local changes
--conflict-resolve path2 # Always prefer cloud changes
```
### Sync Profiles Using bisync
**Profile configurations:**
```python
BISYNC_PROFILES = {
"safe": {
"conflict_resolve": "none", # Keep both versions
"max_delete": 10, # Prevent mass deletion
"check_access": True, # Verify sync integrity
"description": "Safe mode with conflict preservation"
},
"balanced": {
"conflict_resolve": "newer", # Auto-resolve to newer file
"max_delete": 25,
"check_access": True,
"description": "Balanced mode (recommended default)"
},
"fast": {
"conflict_resolve": "newer",
"max_delete": 50,
"check_access": False, # Skip verification for speed
"description": "Fast mode for rapid iteration"
}
}
```
### CLI Commands
**Manual sync commands:**
```bash
basic-memory cloud bisync # Manual bidirectional sync
basic-memory cloud bisync --dry-run # Preview changes
basic-memory cloud bisync --profile safe # Use specific profile
basic-memory cloud bisync --resync # Force full baseline resync
```
**Watch mode (Step 1):**
```bash
basic-memory cloud bisync --watch # Long-running process, sync every 60s
basic-memory cloud bisync --watch --interval 30s # Custom interval
```
**System integration (Step 2 - Future):**
```bash
basic-memory cloud bisync-service install # Install as system service
basic-memory cloud bisync-service start # Start background service
basic-memory cloud bisync-service status # Check service status
```
### Implementation Strategy
**Phase 4.1.1: Core bisync Implementation**
- [ ] Implement `run_bisync()` function wrapping rclone bisync
- [ ] Add profile-based configuration (safe/balanced/fast)
- [ ] Create conflict resolution and safety options
- [ ] Test with sample files and conflict scenarios
**Phase 4.1.2: Watch Mode**
- [ ] Add `--watch` flag for continuous sync
- [ ] Implement configurable sync intervals
- [ ] Add graceful shutdown and signal handling
- [ ] Create status monitoring and progress indicators
**Phase 4.1.3: User Experience**
- [ ] Add conflict reporting and resolution guidance
- [ ] Implement dry-run preview functionality
- [ ] Create troubleshooting and diagnostic commands
- [ ] Add filtering configuration (.gitignore-style)
**Phase 4.1.4: System Integration (Future)**
- [ ] Generate platform-specific service files (launchd/systemd)
- [ ] Add service management commands
- [ ] Implement automatic startup and recovery
- [ ] Create monitoring and logging integration
### Technical Implementation
**Core bisync wrapper:**
```python
def run_bisync(
tenant_id: str,
bucket_name: str,
profile: str = "balanced",
dry_run: bool = False
) -> bool:
"""Run rclone bisync with specified profile."""
local_path = Path.home() / f"basic-memory-{tenant_id}"
remote_path = f"basic-memory-{tenant_id}:{bucket_name}"
profile_config = BISYNC_PROFILES[profile]
cmd = [
"rclone", "bisync",
str(local_path), remote_path,
"--create-empty-src-dirs",
"--resilient",
f"--conflict-resolve={profile_config['conflict_resolve']}",
f"--max-delete={profile_config['max_delete']}",
"--filters-file", "~/.basic-memory/bisync-filters.txt"
]
if profile_config.get("check_access"):
cmd.append("--check-access")
if dry_run:
cmd.append("--dry-run")
return subprocess.run(cmd, check=True).returncode == 0
```
**Default filter file (~/.basic-memory/bisync-filters.txt):**
```
- .DS_Store
- .git/**
- __pycache__/**
- *.pyc
- .pytest_cache/**
- node_modules/**
- .conflict-*
- Thumbs.db
- desktop.ini
```
**Advantages Over Custom Daemon:**
- ✅ **Zero maintenance**: No custom sync logic to debug/maintain
- ✅ **Production proven**: Used by thousands in production
- ✅ **Safety features**: Built-in max-delete, conflict handling, recovery
- ✅ **Filtering**: Advanced exclude patterns and rules
- ✅ **Performance**: Optimized for various storage backends
- ✅ **Community support**: Extensive documentation and community
## Phase 4.2: NFS Mount Support (Direct Access)
### Solution: rclone nfsmount
Keep the existing NFS mount functionality for users who prefer direct file access:
**Core Architecture:**
```bash
# rclone nfsmount provides transparent file access
rclone nfsmount basic-memory-{tenant_id}:{bucket_name} ~/basic-memory-{tenant_id} \
--vfs-cache-mode writes \
--dir-cache-time 10s \
--daemon
```
**Key Benefits:**
- ✅ **Real-time access**: Files appear immediately as they're created/modified
- ✅ **Transparent**: Works with any application that reads/writes files
- ✅ **Low latency**: Direct access without sync delays
- ✅ **Simple**: No periodic sync commands needed
- ✅ **Homebrew compatible**: Works with Homebrew rclone (no FUSE required)
**Limitations:**
- ❌ **File watcher compatibility**: Some apps (Obsidian) don't detect changes over NFS
- ❌ **Network dependency**: Requires active connection to cloud storage
- ❌ **Potential conflicts**: Simultaneous edits from multiple locations can cause issues
### Mount Profiles (Existing)
**Already implemented profiles from SPEC-7 testing:**
```python
MOUNT_PROFILES = {
"fast": {
"cache_time": "5s",
"poll_interval": "3s",
"description": "Ultra-fast development (5s sync)"
},
"balanced": {
"cache_time": "10s",
"poll_interval": "5s",
"description": "Fast development (10-15s sync, recommended)"
},
"safe": {
"cache_time": "15s",
"poll_interval": "10s",
"description": "Conflict-aware mount with backup",
"extra_args": ["--conflict-suffix", ".conflict-{DateTimeExt}"]
}
}
```
### CLI Commands (Existing)
**Mount commands already implemented:**
```bash
basic-memory cloud mount # Mount with balanced profile
basic-memory cloud mount --profile fast # Ultra-fast caching
basic-memory cloud mount --profile safe # Conflict detection
basic-memory cloud unmount # Clean unmount
basic-memory cloud mount-status # Show mount status
```
## User Choice: Mount vs Bisync
### When to Use Each Approach
| Use Case | Recommended Solution | Why |
|----------|---------------------|-----|
| **Obsidian users** | `bisync` | File watcher support for live preview |
| **CLI/vim/emacs users** | `mount` | Direct file access, lower latency |
| **Offline work** | `bisync` | Can work offline, sync when connected |
| **Real-time collaboration** | `mount` | Immediate visibility of changes |
| **Multiple machines** | `bisync` | Better conflict handling |
| **Single machine** | `mount` | Simpler, more transparent |
| **Development work** | Either | Both work well, user preference |
| **Large files** | `mount` | Streaming access vs full download |
### Installation Simplicity
**Both approaches now use simple Homebrew installation:**
```bash
# Single installation command for both approaches
brew install rclone
# No macFUSE, no system modifications needed
# Works immediately with both mount and bisync
```
### Implementation Status
**Phase 4.1: bisync** (NEW)
- [ ] Implement bisync command wrapper
- [ ] Add watch mode with configurable intervals
- [ ] Create conflict resolution workflows
- [ ] Add filtering and safety options
**Phase 4.2: mount** (EXISTING - ✅ IMPLEMENTED)
- [x] NFS mount commands with profile support
- [x] Mount management and cleanup
- [x] Process monitoring and health checks
- [x] Credential integration with cloud API
**Both approaches share:**
- [x] Credential management via cloud API
- [x] Secure rclone configuration
- [x] Tenant isolation and bucket scoping
- [x] Simple Homebrew rclone installation
Key Features:
1. Cross-Platform rclone Installation (rclone_installer.py):
- macOS: Homebrew → official script fallback
- Linux: snap → apt → official script fallback
- Windows: winget → chocolatey → scoop fallback
- Automatic version detection and verification
2. Smart rclone Configuration (rclone_config.py):
- Automatic tenant-specific config generation
- Three optimized mount profiles from your SPEC-7 testing:
- fast: 5s sync (ultra-performance)
- balanced: 10-15s sync (recommended default)
- safe: 15s sync + conflict detection
- Backup existing configs before modification
3. Robust Mount Management (mount_commands.py):
- Automatic tenant credential generation
- Mount path management (~/basic-memory-{tenant-id})
- Process lifecycle management (prevent duplicate mounts)
- Orphaned process cleanup
- Mount verification and health checking
4. Clean Architecture (api_client.py):
- Separated API client to avoid circular imports
- Reuses existing authentication infrastructure
- Consistent error handling and logging
User Experience:
One-Command Setup:
basic-memory cloud setup
```bash
# 1. Installs rclone automatically
# 2. Authenticates with existing login
# 3. Generates secure credentials
# 4. Configures rclone
# 5. Performs initial mount
```
Profile-Based Mounting:
basic-memory cloud mount --profile fast # 5s sync
basic-memory cloud mount --profile balanced # 15s sync (default)
basic-memory cloud mount --profile safe # conflict detection
Status Monitoring:
basic-memory cloud mount-status
```bash
# Shows: tenant info, mount path, sync profile, rclone processes
```
### local mount api
Endpoint 1: Get Tenant Info for user
Purpose: Get tenant details for mounting
- pass in jwt
- service returns mount info
**✅ IMPLEMENTED API Specification:**
**Endpoint 1: GET /tenant/mount/info**
- Purpose: Get tenant mount information without exposing credentials
- Authentication: JWT token (tenant_id extracted from claims)
Request:
```
GET /tenant/mount/info
Authorization: Bearer {jwt_token}
```
Response:
```json
{
"tenant_id": "434252dd-d83b-4b20-bf70-8a950ff875c4",
"bucket_name": "basic-memory-434252dd",
"has_credentials": true,
"credentials_created_at": "2025-09-22T16:48:50.414694"
}
```
**Endpoint 2: POST /tenant/mount/credentials**
- Purpose: Generate NEW bucket-scoped S3 credentials for rclone mounting
- Authentication: JWT token (tenant_id extracted from claims)
- Multi-session: Creates new credentials without revoking existing ones
Request:
```
POST /tenant/mount/credentials
Authorization: Bearer {jwt_token}
Content-Type: application/json
```
*Note: No request body needed - tenant_id extracted from JWT*
Response:
```json
{
"tenant_id": "434252dd-d83b-4b20-bf70-8a950ff875c4",
"bucket_name": "basic-memory-434252dd",
"access_key": "test_access_key_12345",
"secret_key": "test_secret_key_abcdef",
"endpoint_url": "https://fly.storage.tigris.dev",
"region": "auto"
}
```
**🔒 Security Notes:**
- Secret key returned ONCE only - never stored in database
- Credentials are bucket-scoped (cannot access other tenants' buckets)
- Multiple active credentials supported per tenant (work laptop + personal machine)
Implementation Notes
Security:
- Both endpoints require JWT authentication
- Extract tenant_id from JWT claims (not request body)
- Generate scoped credentials (not admin credentials)
- Credentials should have bucket-specific access only
Integration Points:
- Use your existing StorageClient from SPEC-8 implementation
- Leverage existing JWT middleware for tenant extraction
- Return same credential format as your Tigris bucket provisioning
Error Handling:
- 401 if not authenticated
- 403 if tenant doesn't exist
- 500 if credential generation fails
**🔄 Design Decisions:**
1. **Secure Credential Flow (No Secret Storage)**
Based on CLI flow analysis, we follow security best practices:
- ✅ API generates both access_key + secret_key via Tigris IAM
- ✅ Returns both in API response for immediate use
- ✅ CLI uses credentials immediately to configure rclone
- ✅ Database stores only metadata (access_key + policy_arn for cleanup)
- ✅ rclone handles secure local credential storage
- ❌ **Never store secret_key in database (even encrypted)**
2. **CLI Credential Flow**
```bash
# CLI calls API
POST /tenant/mount/credentials → {access_key, secret_key, ...}
# CLI immediately configures rclone
rclone config create basic-memory-{tenant_id} s3 \
access_key_id={access_key} \
secret_access_key={secret_key} \
endpoint=https://fly.storage.tigris.dev
# Database tracks metadata only
INSERT INTO tenant_mount_credentials (tenant_id, access_key, policy_arn, ...)
```
3. **Multiple Sessions Supported**
- Users can have multiple active credential sets (work laptop, personal machine, etc.)
- Each credential generation creates a new Tigris access key
- List active credentials via API (shows access_key but never secret)
4. **Failure Handling & Cleanup**
- **Happy Path**: Credentials created → Used immediately → rclone configured
- **Orphaned Credentials**: Background job revokes unused credentials
- **API Failure Recovery**: Retry Tigris deletion with stored policy_arn
- **Status Tracking**: Track tigris_deletion_status (pending/completed/failed)
5. **Event Sourcing & Audit**
- MountCredentialCreatedEvent
- MountCredentialRevokedEvent
- MountCredentialOrphanedEvent (for cleanup)
- Full audit trail for security compliance
6. **Tenant/Bucket Validation**
- Verify tenant exists and has valid bucket before credential generation
- Use existing StorageClient to validate bucket access
- Prevent credential generation for inactive/invalid tenants
📋 **Implemented API Endpoints:**
```
✅ IMPLEMENTED:
GET /tenant/mount/info # Get tenant/bucket info (no credentials exposed)
POST /tenant/mount/credentials # Generate new credentials (returns secret once)
GET /tenant/mount/credentials # List active credentials (no secrets)
DELETE /tenant/mount/credentials/{cred_id} # Revoke specific credentials
```
**API Implementation Status:**
- ✅ **GET /tenant/mount/info**: Returns tenant_id, bucket_name, has_credentials, credentials_created_at
- ✅ **POST /tenant/mount/credentials**: Creates new bucket-scoped access keys, returns access_key + secret_key once
- ✅ **GET /tenant/mount/credentials**: Lists active credentials without exposing secret keys
- ✅ **DELETE /tenant/mount/credentials/{cred_id}**: Revokes specific credentials with proper Tigris IAM cleanup
- ✅ **Multi-session support**: Multiple active credentials per tenant (work laptop + personal machine)
- ✅ **Security**: Secret keys never stored in database, returned once only for immediate use
- ✅ **Comprehensive test suite**: 28 tests covering all scenarios including error handling and multi-session flows
- ✅ **Dependency injection**: Clean integration with existing FastAPI architecture
- ✅ **Production-ready configuration**: Tigris credentials properly configured for tenant machines
🗄️ **Secure Database Schema:**
```sql
CREATE TABLE tenant_mount_credentials (
id UUID PRIMARY KEY,
tenant_id UUID REFERENCES tenant(id),
access_key VARCHAR(255) NOT NULL,
-- secret_key REMOVED - never store secrets (security best practice)
policy_arn VARCHAR(255) NOT NULL, -- For Tigris IAM cleanup
tigris_deletion_status VARCHAR(20) DEFAULT 'pending', -- Track cleanup
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW(),
revoked_at TIMESTAMP NULL,
last_used_at TIMESTAMP NULL, -- Track usage for orphan cleanup
description VARCHAR(255) DEFAULT 'CLI mount credentials'
);
```
**Security Benefits:**
- ✅ Database breach cannot expose secrets
- ✅ Follows "secrets don't persist" security principle
- ✅ Meets compliance requirements (SOC2, etc.)
- ✅ Reduced attack surface
- ✅ CLI gets credentials once and stores securely via rclone
File diff suppressed because it is too large Load Diff
@@ -1,390 +0,0 @@
---
title: 'SPEC-9-1 Follow-Ups: Conflict, Sync, and Observability'
type: tasklist
permalink: specs/spec-9-follow-ups-conflict-sync-and-observability
related: specs/spec-9-multi-project-bisync
status: revised
revision_date: 2025-10-03
---
# SPEC-9-1 Follow-Ups: Conflict, Sync, and Observability
**REVISED 2025-10-03:** Simplified to leverage rclone built-ins instead of custom conflict handling.
**Context:** SPEC-9 delivered multi-project bidirectional sync and a unified CLI. This follow-up focuses on **observability and safety** using rclone's built-in capabilities rather than reinventing conflict handling.
**Design Philosophy: "Be Dumb Like Git"**
- Let rclone bisync handle conflict detection (it already does this)
- Make conflicts visible and recoverable, don't prevent them
- Cloud is always the winner on conflict (cloud-primary model)
- Users who want version history can just use Git locally in their sync directory
**What Changed from Original Version:**
- **Replaced:** Custom `.bmmeta` sidecars → Use rclone's `.bisync/` state tracking
- **Replaced:** Custom conflict detection → Use rclone bisync 3-way merge
- **Replaced:** Tombstone files → rclone delete tracking handles this
- **Replaced:** Distributed lease → Local process lock only (document multi-device warning)
- **Replaced:** S3 versioning service → Users just use Git locally if they want history
- **Deferred:** SPEC-14 Git integration → Postponed to teams/multi-user features
## ✅ Now
- [ ] **Local process lock**: Prevent concurrent bisync runs on same device (`~/.basic-memory/sync.lock`)
- [ ] **Structured sync reports**: Parse rclone bisync output into JSON reports (creates/updates/deletes/conflicts, bytes, duration); `bm sync --report`
- [ ] **Multi-device warning**: Document that users should not run `--watch` on multiple devices simultaneously
- [ ] **Version control guidance**: Document pattern for users to use Git locally in their sync directory if they want version history
- [ ] **Docs polish**: cloud-mode toggle, mount↔bisync directory isolation, conflict semantics, quick start, migration guide, short demo clip/GIF
## 🔜 Next
- [ ] **Observability commands**: `bm conflicts list`, `bm sync history` to view sync reports and conflicts
- [ ] **Conflict resolution UI**: `bm conflicts resolve <file>` to interactively pick winner from conflict files
- [ ] **Selective sync**: allow include/exclude by project; per-project profile (safe/balanced/fast)
## 🧭 Later
- [ ] **Near real-time sync**: File watcher → targeted `rclone copy` for individual files (keep bisync as backstop)
- [ ] **Sharing / scoped tokens**: cross-tenant/project access
- [ ] **Bandwidth controls & backpressure**: policy for large repos
- [ ] **Client-side encryption (optional)**: with clear trade-offs
## 📏 Acceptance criteria (for "Now" items)
- [ ] Local process lock prevents concurrent bisync runs on same device
- [ ] rclone bisync conflict files visible and documented (`file.conflict1.md`, `file.conflict2.md`)
- [ ] `bm sync --report` generates parsable JSON with sync statistics
- [ ] Documentation clearly warns about multi-device `--watch` mode
- [ ] Documentation shows users how to use Git locally for version history
## What We're NOT Building (Deferred to rclone)
- ❌ Custom `.bmmeta` sidecars (rclone tracks state in `.bisync/` workdir)
- ❌ Custom conflict detection (rclone bisync already does 3-way merge detection)
- ❌ Tombstone files (S3 versioning + rclone delete tracking handles this)
- ❌ Distributed lease (low probability issue, rclone detects state divergence)
- ❌ Rename/move tracking (rclone has size+modtime heuristics built-in)
## Implementation Summary
**Current State (SPEC-9):**
- ✅ rclone bisync with 3 profiles (safe/balanced/fast)
-`--max-delete` safety limits (10/25/50 files)
-`--conflict-resolve=newer` for auto-resolution
- ✅ Watch mode: `bm sync --watch` (60s intervals)
- ✅ Integrity checking: `bm cloud check`
- ✅ Mount vs bisync directory isolation
**What's Needed (This Spec):**
1. **Process lock** - Simple file-based lock in `~/.basic-memory/sync.lock`
2. **Sync reports** - Parse rclone output, save to `~/.basic-memory/sync-history/`
3. **Documentation** - Multi-device warnings, conflict resolution workflow, Git usage pattern
**User Model:**
- Cloud is always the winner on conflict (cloud-primary)
- rclone creates `.conflict` files for divergent edits
- Users who want version history just use Git in their local sync directory
- Users warned: don't run `--watch` on multiple devices
## Decision Rationale & Trade-offs
### Why Trust rclone Instead of Custom Conflict Handling?
**rclone bisync already provides:**
- 3-way merge detection (compares local, remote, and last-known state)
- File state tracking in `.bisync/` workdir (hashes, modtimes)
- Automatic conflict file creation: `file.conflict1.md`, `file.conflict2.md`
- Rename detection via size+modtime heuristics
- Delete tracking (prevents resurrection of deleted files)
- Battle-tested with extensive edge case handling
**What we'd have to build with custom approach:**
- Per-file metadata tracking (`.bmmeta` sidecars)
- 3-way diff algorithm
- Conflict detection logic
- Tombstone files for deletes
- Rename/move detection
- Testing for all edge cases
**Decision:** Use what rclone already does well. Don't reinvent the wheel.
### Why Let Users Use Git Locally Instead of Building Versioning?
**The simplest solution: Just use Git**
Users who want version history can literally just use Git in their sync directory:
```bash
cd ~/basic-memory-cloud-sync/
git init
git add .
git commit -m "backup"
# Push to their own GitHub if they want
git remote add origin git@github.com:user/my-knowledge.git
git push
```
**Why this is perfect:**
- ✅ We build nothing
- ✅ Users who want Git... just use Git
- ✅ Users who don't care... don't need to
- ✅ rclone bisync already handles sync conflicts
- ✅ Users own their data, they can version it however they want (Git, Time Machine, etc.)
**What we'd have to build for S3 versioning:**
- API to enable versioning on Tigris buckets
- **Problem**: Tigris doesn't support S3 bucket versioning
- Restore commands: `bm cloud restore --version-id`
- Version listing: `bm cloud versions <path>`
- Lifecycle policies for version retention
- Documentation and user education
**What we'd have to build for SPEC-14 Git integration:**
- Committer service (daemon watching `/app/data/`)
- Puller service (webhook handler for GitHub pushes)
- Git LFS for large files
- Loop prevention between Git ↔ bisync ↔ local
- Merge conflict handling at TWO layers (rclone + Git)
- Webhook infrastructure and monitoring
**Decision:** Don't build version control. Document the pattern. "The easiest problem to solve is the one you avoid."
**When to revisit:** Teams/multi-user features where server-side version control becomes necessary for collaboration.
### Why No Distributed Lease?
**Low probability issue:**
- Requires user to manually run `bm sync` on multiple devices at exact same time
- Most users run `--watch` on one primary device
- rclone bisync detects state divergence and fails safely
**Safety nets in place:**
- Local process lock prevents concurrent runs on same device
- rclone bisync aborts if bucket state changed during sync
- S3 versioning recovers from any overwrites
- Documentation warns against multi-device `--watch`
**Failure mode:**
```bash
# Device A and B sync simultaneously
Device A: bm sync → succeeds
Device B: bm sync → "Error: path has changed, run --resync"
# User fixes with resync
Device B: bm sync --resync → establishes new baseline
```
**Decision:** Document the issue, add local lock, defer distributed coordination until users report actual problems.
### Cloud-Primary Conflict Model
**User mental model:**
- Cloud is the source of truth (like Dropbox/iCloud)
- Local is working copy
- On conflict: cloud wins, local edits → `.conflict` file
- User manually picks winner
**Why this works:**
- Simpler than bidirectional merge (no automatic resolution risk)
- Matches user expectations from Dropbox
- S3 versioning provides safety net for overwrites
- Clear recovery path: restore from S3 version if needed
**Example workflow:**
```bash
# Edit file on Device A and Device B while offline
# Both devices come online and sync
Device A: bm sync
# → Pushes to cloud first, becomes canonical version
Device B: bm sync
# → Detects conflict
# → Cloud version: work/notes.md
# → Local version: work/notes.md.conflict1
# → User manually merges or picks winner
# Restore if needed
bm cloud restore work/notes.md --version-id abc123
```
## Implementation Details
### 1. Local Process Lock
```python
# ~/.basic-memory/sync.lock
import os
import psutil
from pathlib import Path
class SyncLock:
def __init__(self):
self.lock_file = Path.home() / '.basic-memory' / 'sync.lock'
def acquire(self):
if self.lock_file.exists():
pid = int(self.lock_file.read_text())
if psutil.pid_exists(pid):
raise BisyncError(
f"Sync already running (PID {pid}). "
f"Wait for completion or kill stale process."
)
# Stale lock, remove it
self.lock_file.unlink()
self.lock_file.write_text(str(os.getpid()))
def release(self):
if self.lock_file.exists():
self.lock_file.unlink()
def __enter__(self):
self.acquire()
return self
def __exit__(self, *args):
self.release()
# Usage
with SyncLock():
run_rclone_bisync()
```
### 3. Sync Report Parsing
```python
# Parse rclone bisync output
import json
from datetime import datetime
from pathlib import Path
def parse_sync_report(rclone_output: str, duration: float, exit_code: int) -> dict:
"""Parse rclone bisync output into structured report."""
# rclone bisync outputs lines like:
# "Synching Path1 /local/path with Path2 remote:bucket"
# "- Path1 File was copied to Path2"
# "Bisync successful"
report = {
"timestamp": datetime.now().isoformat(),
"duration_seconds": duration,
"exit_code": exit_code,
"success": exit_code == 0,
"files_created": 0,
"files_updated": 0,
"files_deleted": 0,
"conflicts": [],
"errors": []
}
for line in rclone_output.split('\n'):
if 'was copied to' in line:
report['files_created'] += 1
elif 'was updated in' in line:
report['files_updated'] += 1
elif 'was deleted from' in line:
report['files_deleted'] += 1
elif '.conflict' in line:
report['conflicts'].append(line.strip())
elif 'ERROR' in line:
report['errors'].append(line.strip())
return report
def save_sync_report(report: dict):
"""Save sync report to history."""
history_dir = Path.home() / '.basic-memory' / 'sync-history'
history_dir.mkdir(parents=True, exist_ok=True)
timestamp = datetime.now().strftime('%Y%m%d-%H%M%S')
report_file = history_dir / f'{timestamp}.json'
report_file.write_text(json.dumps(report, indent=2))
# Usage in run_bisync()
start_time = time.time()
result = subprocess.run(bisync_cmd, capture_output=True, text=True)
duration = time.time() - start_time
report = parse_sync_report(result.stdout, duration, result.returncode)
save_sync_report(report)
if report['conflicts']:
console.print(f"[yellow]⚠ {len(report['conflicts'])} conflict(s) detected[/yellow]")
console.print("[dim]Run 'bm conflicts list' to view[/dim]")
```
### 4. User Commands
```bash
# View sync history
bm sync history
# → Lists recent syncs from ~/.basic-memory/sync-history/*.json
# → Shows: timestamp, duration, files changed, conflicts, errors
# View current conflicts
bm conflicts list
# → Scans sync directory for *.conflict* files
# → Shows: file path, conflict versions, timestamps
# Restore from S3 version
bm cloud restore work/notes.md --version-id abc123
# → Uses aws s3api get-object with version-id
# → Downloads to original path
bm cloud restore work/notes.md --timestamp "2025-10-03 14:30"
# → Lists versions, finds closest to timestamp
# → Downloads that version
# List file versions
bm cloud versions work/notes.md
# → Uses aws s3api list-object-versions
# → Shows: version-id, timestamp, size, author
# Interactive conflict resolution
bm conflicts resolve work/notes.md
# → Shows both versions side-by-side
# → Prompts: Keep local, keep cloud, merge manually, restore from S3 version
# → Cleans up .conflict files after resolution
```
## Success Metrics & Monitoring
**Phase 1 (v1) - Basic Safety:**
- [ ] Conflict detection rate < 5% of syncs (measure in telemetry)
- [ ] User can resolve conflicts within 5 minutes (UX testing)
- [ ] Documentation prevents 90% of multi-device issues
**Phase 2 (v2) - Observability:**
- [ ] 80% of users check `bm sync history` when troubleshooting
- [ ] Average time to restore from S3 version < 2 minutes
-
- [ ] Conflict resolution success rate > 95%
**What to measure:**
```python
# Telemetry in sync reports
{
"conflict_rate": conflicts / total_syncs,
"multi_device_collisions": count_state_divergence_errors,
"version_restores": count_restore_operations,
"avg_sync_duration": sum(durations) / count,
"max_delete_trips": count_max_delete_aborts
}
```
**When to add distributed lease:**
- Multi-device collision rate > 5% of syncs
- User complaints about state divergence errors
- Evidence that local lock isn't sufficient
**When to revisit Git (SPEC-14):**
- Teams feature launches (multi-user collaboration)
- Users request commit messages / audit trail
- PR-based review workflow becomes valuable
## Links
- SPEC-9: `specs/spec-9-multi-project-bisync`
- SPEC-14: `specs/spec-14-cloud-git-versioning` (deferred in favor of S3 versioning)
- rclone bisync docs: https://rclone.org/bisync/
- Tigris S3 versioning: https://www.tigrisdata.com/docs/buckets/versioning/
---
**Owner:** <assign> | **Review cadence:** weekly in standup | **Last updated:** 2025-10-03
+1 -1
View File
@@ -1,7 +1,7 @@
"""basic-memory - Local-first knowledge management combining Zettelkasten with knowledge graphs"""
# Package version - updated by release automation
__version__ = "0.15.2"
__version__ = "0.17.8"
# API version for FastAPI - independent of package version
__api_version__ = "v0"
+114 -24
View File
@@ -1,17 +1,36 @@
"""Alembic environment configuration."""
import asyncio
import os
from logging.config import fileConfig
from sqlalchemy import engine_from_config
from sqlalchemy import pool
# Allow nested event loops (needed for pytest-asyncio and other async contexts)
# Note: nest_asyncio doesn't work with uvloop or Python 3.14+, so we handle those cases separately
import sys
if sys.version_info < (3, 14):
try:
import nest_asyncio
nest_asyncio.apply()
except (ImportError, ValueError):
# nest_asyncio not available or can't patch this loop type (e.g., uvloop)
pass
# For Python 3.14+, we rely on the thread-based fallback in run_migrations_online()
from sqlalchemy import engine_from_config, pool
from sqlalchemy.ext.asyncio import AsyncEngine, create_async_engine
from alembic import context
from basic_memory.config import ConfigManager
# set config.env to "test" for pytest to prevent logging to file in utils.setup_logging()
os.environ["BASIC_MEMORY_ENV"] = "test"
# Trigger: only set test env when actually running under pytest
# Why: alembic/env.py is imported during normal operations (MCP server startup, migrations)
# but we only want test behavior during actual test runs
# Outcome: prevents is_test_env from returning True in production, enabling watch service
if os.getenv("PYTEST_CURRENT_TEST") is not None:
os.environ["BASIC_MEMORY_ENV"] = "test"
# Import after setting environment variable # noqa: E402
from basic_memory.models import Base # noqa: E402
@@ -20,12 +39,22 @@ from basic_memory.models import Base # noqa: E402
# access to the values within the .ini file in use.
config = context.config
# Load app config - this will read environment variables (BASIC_MEMORY_DATABASE_BACKEND, etc.)
# due to Pydantic's env_prefix="BASIC_MEMORY_" setting
app_config = ConfigManager().config
# Set the SQLAlchemy URL from our app config
sqlalchemy_url = f"sqlite:///{app_config.database_path}"
config.set_main_option("sqlalchemy.url", sqlalchemy_url)
# print(f"Using SQLAlchemy URL: {sqlalchemy_url}")
# Set the SQLAlchemy URL based on database backend configuration
# If the URL is already set in config (e.g., from run_migrations), use that
# Otherwise, get it from app config
# Note: alembic.ini has a placeholder URL "driver://user:pass@localhost/dbname" that we need to override
current_url = config.get_main_option("sqlalchemy.url")
if not current_url or current_url == "driver://user:pass@localhost/dbname":
from basic_memory.db import DatabaseType
sqlalchemy_url = DatabaseType.get_db_url(
app_config.database_path, DatabaseType.FILESYSTEM, app_config
)
config.set_main_option("sqlalchemy.url", sqlalchemy_url)
# Interpret the config file for Python logging.
if config.config_file_name is not None:
@@ -69,28 +98,89 @@ def run_migrations_offline() -> None:
context.run_migrations()
def do_run_migrations(connection):
"""Execute migrations with the given connection."""
context.configure(
connection=connection,
target_metadata=target_metadata,
include_object=include_object,
render_as_batch=True,
compare_type=True,
)
with context.begin_transaction():
context.run_migrations()
async def run_async_migrations(connectable):
"""Run migrations asynchronously with AsyncEngine."""
async with connectable.connect() as connection:
await connection.run_sync(do_run_migrations)
await connectable.dispose()
def run_migrations_online() -> None:
"""Run migrations in 'online' mode.
In this scenario we need to create an Engine
and associate a connection with the context.
Supports both sync engines (SQLite) and async engines (PostgreSQL with asyncpg).
"""
connectable = engine_from_config(
config.get_section(config.config_ini_section, {}),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
# Check if a connection/engine was provided (e.g., from run_migrations)
connectable = context.config.attributes.get("connection", None)
with connectable.connect() as connection:
context.configure(
connection=connection,
target_metadata=target_metadata,
include_object=include_object,
render_as_batch=True,
)
if connectable is None:
# No connection provided, create engine from config
url = context.config.get_main_option("sqlalchemy.url")
with context.begin_transaction():
context.run_migrations()
# Check if it's an async URL (sqlite+aiosqlite or postgresql+asyncpg)
if url and ("+asyncpg" in url or "+aiosqlite" in url):
# Create async engine for asyncpg or aiosqlite
connectable = create_async_engine(
url,
poolclass=pool.NullPool,
future=True,
)
else:
# Create sync engine for regular sqlite or postgresql
connectable = engine_from_config(
context.config.get_section(context.config.config_ini_section, {}),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
# Handle async engines (PostgreSQL with asyncpg)
if isinstance(connectable, AsyncEngine):
# Try to run async migrations
# nest_asyncio allows asyncio.run() from within event loops, but doesn't work with uvloop
try:
asyncio.run(run_async_migrations(connectable))
except RuntimeError as e:
if "cannot be called from a running event loop" in str(e):
# We're in a running event loop (likely uvloop) - need to use a different approach
# Create a new thread to run the async migrations
import concurrent.futures
def run_in_thread():
"""Run async migrations in a new event loop in a separate thread."""
new_loop = asyncio.new_event_loop()
asyncio.set_event_loop(new_loop)
try:
new_loop.run_until_complete(run_async_migrations(connectable))
finally:
new_loop.close()
with concurrent.futures.ThreadPoolExecutor() as executor:
future = executor.submit(run_in_thread)
future.result() # Wait for completion and re-raise any exceptions
else:
raise
else:
# Handle sync engines (SQLite) or sync connections
if hasattr(connectable, "connect"):
# It's an engine, get a connection
with connectable.connect() as connection:
do_run_migrations(connection)
else:
# It's already a connection
do_run_migrations(connectable)
if context.is_offline_mode():
@@ -0,0 +1,131 @@
"""Add Postgres full-text search support with tsvector and GIN indexes
Revision ID: 314f1ea54dc4
Revises: e7e1f4367280
Create Date: 2025-11-15 18:05:01.025405
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = "314f1ea54dc4"
down_revision: Union[str, None] = "e7e1f4367280"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Add PostgreSQL full-text search support.
This migration:
1. Creates search_index table for Postgres (SQLite uses FTS5 virtual table)
2. Adds generated tsvector column for full-text search
3. Creates GIN index on the tsvector column for fast text queries
4. Creates GIN index on metadata JSONB column for fast containment queries
Note: These changes only apply to Postgres. SQLite continues to use FTS5 virtual tables.
"""
# Check if we're using Postgres
connection = op.get_bind()
if connection.dialect.name == "postgresql":
# Create search_index table for Postgres
# For SQLite, this is a FTS5 virtual table created elsewhere
from sqlalchemy.dialects.postgresql import JSONB
op.create_table(
"search_index",
sa.Column("id", sa.Integer(), nullable=False), # Entity IDs are integers
sa.Column("project_id", sa.Integer(), nullable=False), # Multi-tenant isolation
sa.Column("title", sa.Text(), nullable=True),
sa.Column("content_stems", sa.Text(), nullable=True),
sa.Column("content_snippet", sa.Text(), nullable=True),
sa.Column("permalink", sa.String(), nullable=True), # Nullable for non-markdown files
sa.Column("file_path", sa.String(), nullable=True),
sa.Column("type", sa.String(), nullable=True),
sa.Column("from_id", sa.Integer(), nullable=True), # Relation IDs are integers
sa.Column("to_id", sa.Integer(), nullable=True), # Relation IDs are integers
sa.Column("relation_type", sa.String(), nullable=True),
sa.Column("entity_id", sa.Integer(), nullable=True), # Entity IDs are integers
sa.Column("category", sa.String(), nullable=True),
sa.Column("metadata", JSONB(), nullable=True), # Use JSONB for Postgres
sa.Column("created_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=True),
sa.PrimaryKeyConstraint(
"id", "type", "project_id"
), # Composite key: id can repeat across types
sa.ForeignKeyConstraint(
["project_id"],
["project.id"],
name="fk_search_index_project_id",
ondelete="CASCADE",
),
if_not_exists=True,
)
# Create index on project_id for efficient multi-tenant queries
op.create_index(
"ix_search_index_project_id",
"search_index",
["project_id"],
unique=False,
)
# Create unique partial index on permalink for markdown files
# Non-markdown files don't have permalinks, so we use a partial index
op.execute("""
CREATE UNIQUE INDEX uix_search_index_permalink_project
ON search_index (permalink, project_id)
WHERE permalink IS NOT NULL
""")
# Add tsvector column as a GENERATED ALWAYS column
# This automatically updates when title or content_stems change
op.execute("""
ALTER TABLE search_index
ADD COLUMN textsearchable_index_col tsvector
GENERATED ALWAYS AS (
to_tsvector('english',
coalesce(title, '') || ' ' ||
coalesce(content_stems, '')
)
) STORED
""")
# Create GIN index on tsvector column for fast full-text search
op.create_index(
"idx_search_index_fts",
"search_index",
["textsearchable_index_col"],
unique=False,
postgresql_using="gin",
)
# Create GIN index on metadata JSONB for fast containment queries
# Using jsonb_path_ops for smaller index size and better performance
op.execute("""
CREATE INDEX idx_search_index_metadata_gin
ON search_index
USING GIN (metadata jsonb_path_ops)
""")
def downgrade() -> None:
"""Remove PostgreSQL full-text search support."""
connection = op.get_bind()
if connection.dialect.name == "postgresql":
# Drop indexes first
op.execute("DROP INDEX IF EXISTS idx_search_index_metadata_gin")
op.drop_index("idx_search_index_fts", table_name="search_index")
op.execute("DROP INDEX IF EXISTS uix_search_index_permalink_project")
op.drop_index("ix_search_index_project_id", table_name="search_index")
# Drop the generated column
op.execute("ALTER TABLE search_index DROP COLUMN IF EXISTS textsearchable_index_col")
# Drop the search_index table
op.drop_table("search_index")
@@ -21,6 +21,12 @@ depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
# SQLite FTS5 virtual table handling is SQLite-specific
# For Postgres, search_index is a regular table managed by ORM
connection = op.get_bind()
is_sqlite = connection.dialect.name == "sqlite"
op.create_table(
"project",
sa.Column("id", sa.Integer(), nullable=False),
@@ -55,7 +61,9 @@ def upgrade() -> None:
batch_op.add_column(sa.Column("project_id", sa.Integer(), nullable=False))
batch_op.drop_index(
"uix_entity_permalink",
sqlite_where=sa.text("content_type = 'text/markdown' AND permalink IS NOT NULL"),
sqlite_where=sa.text("content_type = 'text/markdown' AND permalink IS NOT NULL")
if is_sqlite
else None,
)
batch_op.drop_index("ix_entity_file_path")
batch_op.create_index(batch_op.f("ix_entity_file_path"), ["file_path"], unique=False)
@@ -67,12 +75,16 @@ def upgrade() -> None:
"uix_entity_permalink_project",
["permalink", "project_id"],
unique=True,
sqlite_where=sa.text("content_type = 'text/markdown' AND permalink IS NOT NULL"),
sqlite_where=sa.text("content_type = 'text/markdown' AND permalink IS NOT NULL")
if is_sqlite
else None,
)
batch_op.create_foreign_key("fk_entity_project_id", "project", ["project_id"], ["id"])
# drop the search index table. it will be recreated
op.drop_table("search_index")
# Only drop for SQLite - Postgres uses regular table managed by ORM
if is_sqlite:
op.drop_table("search_index")
# ### end Alembic commands ###
@@ -25,43 +25,51 @@ def upgrade() -> None:
The UNIQUE constraint prevents multiple projects from having is_default=FALSE,
which breaks project creation when the service sets is_default=False.
Since SQLite doesn't support dropping specific constraints easily, we'll
recreate the table without the problematic constraint.
SQLite: Recreate the table without the constraint (no ALTER TABLE support)
Postgres: Use ALTER TABLE to drop the constraint directly
"""
# For SQLite, we need to recreate the table without the UNIQUE constraint
# Create a new table without the UNIQUE constraint on is_default
op.create_table(
"project_new",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("name", sa.String(), nullable=False),
sa.Column("description", sa.Text(), nullable=True),
sa.Column("permalink", sa.String(), nullable=False),
sa.Column("path", sa.String(), nullable=False),
sa.Column("is_active", sa.Boolean(), nullable=False),
sa.Column("is_default", sa.Boolean(), nullable=True), # No UNIQUE constraint!
sa.Column("created_at", sa.DateTime(), nullable=False),
sa.Column("updated_at", sa.DateTime(), nullable=False),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("name"),
sa.UniqueConstraint("permalink"),
)
connection = op.get_bind()
is_sqlite = connection.dialect.name == "sqlite"
# Copy data from old table to new table
op.execute("INSERT INTO project_new SELECT * FROM project")
if is_sqlite:
# For SQLite, we need to recreate the table without the UNIQUE constraint
# Create a new table without the UNIQUE constraint on is_default
op.create_table(
"project_new",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("name", sa.String(), nullable=False),
sa.Column("description", sa.Text(), nullable=True),
sa.Column("permalink", sa.String(), nullable=False),
sa.Column("path", sa.String(), nullable=False),
sa.Column("is_active", sa.Boolean(), nullable=False),
sa.Column("is_default", sa.Boolean(), nullable=True), # No UNIQUE constraint!
sa.Column("created_at", sa.DateTime(), nullable=False),
sa.Column("updated_at", sa.DateTime(), nullable=False),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("name"),
sa.UniqueConstraint("permalink"),
)
# Drop the old table
op.drop_table("project")
# Copy data from old table to new table
op.execute("INSERT INTO project_new SELECT * FROM project")
# Rename the new table
op.rename_table("project_new", "project")
# Drop the old table
op.drop_table("project")
# Recreate the indexes
with op.batch_alter_table("project", schema=None) as batch_op:
batch_op.create_index("ix_project_created_at", ["created_at"], unique=False)
batch_op.create_index("ix_project_name", ["name"], unique=True)
batch_op.create_index("ix_project_path", ["path"], unique=False)
batch_op.create_index("ix_project_permalink", ["permalink"], unique=True)
batch_op.create_index("ix_project_updated_at", ["updated_at"], unique=False)
# Rename the new table
op.rename_table("project_new", "project")
# Recreate the indexes
with op.batch_alter_table("project", schema=None) as batch_op:
batch_op.create_index("ix_project_created_at", ["created_at"], unique=False)
batch_op.create_index("ix_project_name", ["name"], unique=True)
batch_op.create_index("ix_project_path", ["path"], unique=False)
batch_op.create_index("ix_project_permalink", ["permalink"], unique=True)
batch_op.create_index("ix_project_updated_at", ["updated_at"], unique=False)
else:
# For Postgres, we can simply drop the constraint
with op.batch_alter_table("project", schema=None) as batch_op:
batch_op.drop_constraint("project_is_default_key", type_="unique")
def downgrade() -> None:
@@ -0,0 +1,24 @@
"""Merge multiple heads
Revision ID: 6830751f5fb6
Revises: a2b3c4d5e6f7, g9a0b3c4d5e6
Create Date: 2025-12-29 12:46:46.476268
"""
from typing import Sequence, Union
# revision identifiers, used by Alembic.
revision: str = "6830751f5fb6"
down_revision: Union[str, Sequence[str], None] = ("a2b3c4d5e6f7", "g9a0b3c4d5e6")
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
@@ -0,0 +1,49 @@
"""Add mtime and size columns to Entity for sync optimization
Revision ID: 9d9c1cb7d8f5
Revises: a1b2c3d4e5f6
Create Date: 2025-10-20 05:07:55.173849
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = "9d9c1cb7d8f5"
down_revision: Union[str, None] = "a1b2c3d4e5f6"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table("entity", schema=None) as batch_op:
batch_op.add_column(sa.Column("mtime", sa.Float(), nullable=True))
batch_op.add_column(sa.Column("size", sa.Integer(), nullable=True))
batch_op.drop_constraint(batch_op.f("fk_entity_project_id"), type_="foreignkey")
batch_op.create_foreign_key(
batch_op.f("fk_entity_project_id"), "project", ["project_id"], ["id"]
)
# ### end Alembic commands ###
def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table("entity", schema=None) as batch_op:
batch_op.drop_constraint(batch_op.f("fk_entity_project_id"), type_="foreignkey")
batch_op.create_foreign_key(
batch_op.f("fk_entity_project_id"),
"project",
["project_id"],
["id"],
ondelete="CASCADE",
)
batch_op.drop_column("size")
batch_op.drop_column("mtime")
# ### end Alembic commands ###
@@ -0,0 +1,56 @@
"""Add cascade delete FK from search_index to entity
Revision ID: a2b3c4d5e6f7
Revises: f8a9b2c3d4e5
Create Date: 2025-12-02 07:00:00.000000
"""
from typing import Sequence, Union
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "a2b3c4d5e6f7"
down_revision: Union[str, None] = "f8a9b2c3d4e5"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Add FK with CASCADE delete from search_index.entity_id to entity.id.
This migration is Postgres-only because:
- SQLite uses FTS5 virtual tables which don't support foreign keys
- The FK enables automatic cleanup of search_index entries when entities are deleted
"""
connection = op.get_bind()
dialect = connection.dialect.name
if dialect == "postgresql":
# First, clean up any orphaned search_index entries where entity no longer exists
op.execute("""
DELETE FROM search_index
WHERE entity_id IS NOT NULL
AND entity_id NOT IN (SELECT id FROM entity)
""")
# Add FK with CASCADE - nullable FK allows search_index entries without entity_id
op.create_foreign_key(
"fk_search_index_entity_id",
"search_index",
"entity",
["entity_id"],
["id"],
ondelete="CASCADE",
)
def downgrade() -> None:
"""Remove the FK constraint."""
connection = op.get_bind()
dialect = connection.dialect.name
if dialect == "postgresql":
op.drop_constraint("fk_search_index_entity_id", "search_index", type_="foreignkey")
@@ -21,6 +21,12 @@ depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Upgrade database schema to use new search index with content_stems and content_snippet."""
# This migration is SQLite-specific (FTS5 virtual tables)
# For Postgres, the search_index table is created via ORM models
connection = op.get_bind()
if connection.dialect.name != "sqlite":
return
# First, drop the existing search_index table
op.execute("DROP TABLE IF EXISTS search_index")
@@ -59,6 +65,13 @@ def upgrade() -> None:
def downgrade() -> None:
"""Downgrade database schema to use old search index."""
# This migration is SQLite-specific (FTS5 virtual tables)
# For Postgres, the search_index table is managed via ORM models
connection = op.get_bind()
if connection.dialect.name != "sqlite":
return
# Drop the updated search_index table
op.execute("DROP TABLE IF EXISTS search_index")
@@ -0,0 +1,37 @@
"""Add scan watermark tracking to Project
Revision ID: e7e1f4367280
Revises: 9d9c1cb7d8f5
Create Date: 2025-10-20 16:42:46.625075
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = "e7e1f4367280"
down_revision: Union[str, None] = "9d9c1cb7d8f5"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table("project", schema=None) as batch_op:
batch_op.add_column(sa.Column("last_scan_timestamp", sa.Float(), nullable=True))
batch_op.add_column(sa.Column("last_file_count", sa.Integer(), nullable=True))
# ### end Alembic commands ###
def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table("project", schema=None) as batch_op:
batch_op.drop_column("last_file_count")
batch_op.drop_column("last_scan_timestamp")
# ### end Alembic commands ###
@@ -0,0 +1,239 @@
"""Add project_id to relation/observation and pg_trgm for fuzzy link resolution
Revision ID: f8a9b2c3d4e5
Revises: 314f1ea54dc4
Create Date: 2025-12-01 12:00:00.000000
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy import text
def column_exists(connection, table: str, column: str) -> bool:
"""Check if a column exists in a table (idempotent migration support)."""
if connection.dialect.name == "postgresql":
result = connection.execute(
text(
"SELECT 1 FROM information_schema.columns "
"WHERE table_name = :table AND column_name = :column"
),
{"table": table, "column": column},
)
return result.fetchone() is not None
else:
# SQLite
result = connection.execute(text(f"PRAGMA table_info({table})"))
columns = [row[1] for row in result]
return column in columns
def index_exists(connection, index_name: str) -> bool:
"""Check if an index exists (idempotent migration support)."""
if connection.dialect.name == "postgresql":
result = connection.execute(
text("SELECT 1 FROM pg_indexes WHERE indexname = :index_name"),
{"index_name": index_name},
)
return result.fetchone() is not None
else:
# SQLite
result = connection.execute(
text("SELECT 1 FROM sqlite_master WHERE type='index' AND name = :index_name"),
{"index_name": index_name},
)
return result.fetchone() is not None
# revision identifiers, used by Alembic.
revision: str = "f8a9b2c3d4e5"
down_revision: Union[str, None] = "314f1ea54dc4"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Add project_id to relation and observation tables, plus pg_trgm indexes.
This migration:
1. Adds project_id column to relation and observation tables (denormalization)
2. Backfills project_id from the associated entity
3. Enables pg_trgm extension for trigram-based fuzzy matching (Postgres only)
4. Creates GIN indexes on entity title and permalink for fast similarity searches
5. Creates partial index on unresolved relations for efficient bulk resolution
"""
connection = op.get_bind()
dialect = connection.dialect.name
# -------------------------------------------------------------------------
# Add project_id to relation table
# -------------------------------------------------------------------------
# Step 1: Add project_id column as nullable first (idempotent)
if not column_exists(connection, "relation", "project_id"):
op.add_column("relation", sa.Column("project_id", sa.Integer(), nullable=True))
# Step 2: Backfill project_id from entity.project_id via from_id
if dialect == "postgresql":
op.execute("""
UPDATE relation
SET project_id = entity.project_id
FROM entity
WHERE relation.from_id = entity.id
""")
else:
# SQLite syntax
op.execute("""
UPDATE relation
SET project_id = (
SELECT entity.project_id
FROM entity
WHERE entity.id = relation.from_id
)
""")
# Step 3: Make project_id NOT NULL and add foreign key
if dialect == "postgresql":
op.alter_column("relation", "project_id", nullable=False)
op.create_foreign_key(
"fk_relation_project_id",
"relation",
"project",
["project_id"],
["id"],
)
else:
# SQLite requires batch operations for ALTER COLUMN
with op.batch_alter_table("relation") as batch_op:
batch_op.alter_column("project_id", nullable=False)
batch_op.create_foreign_key(
"fk_relation_project_id",
"project",
["project_id"],
["id"],
)
# Step 4: Create index on relation.project_id (idempotent)
if not index_exists(connection, "ix_relation_project_id"):
op.create_index("ix_relation_project_id", "relation", ["project_id"])
# -------------------------------------------------------------------------
# Add project_id to observation table
# -------------------------------------------------------------------------
# Step 1: Add project_id column as nullable first (idempotent)
if not column_exists(connection, "observation", "project_id"):
op.add_column("observation", sa.Column("project_id", sa.Integer(), nullable=True))
# Step 2: Backfill project_id from entity.project_id via entity_id
if dialect == "postgresql":
op.execute("""
UPDATE observation
SET project_id = entity.project_id
FROM entity
WHERE observation.entity_id = entity.id
""")
else:
# SQLite syntax
op.execute("""
UPDATE observation
SET project_id = (
SELECT entity.project_id
FROM entity
WHERE entity.id = observation.entity_id
)
""")
# Step 3: Make project_id NOT NULL and add foreign key
if dialect == "postgresql":
op.alter_column("observation", "project_id", nullable=False)
op.create_foreign_key(
"fk_observation_project_id",
"observation",
"project",
["project_id"],
["id"],
)
else:
# SQLite requires batch operations for ALTER COLUMN
with op.batch_alter_table("observation") as batch_op:
batch_op.alter_column("project_id", nullable=False)
batch_op.create_foreign_key(
"fk_observation_project_id",
"project",
["project_id"],
["id"],
)
# Step 4: Create index on observation.project_id (idempotent)
if not index_exists(connection, "ix_observation_project_id"):
op.create_index("ix_observation_project_id", "observation", ["project_id"])
# Postgres-specific: pg_trgm and GIN indexes
if dialect == "postgresql":
# Enable pg_trgm extension for fuzzy string matching
op.execute("CREATE EXTENSION IF NOT EXISTS pg_trgm")
# Create trigram indexes on entity table for fuzzy matching
# GIN indexes with gin_trgm_ops support similarity searches
op.execute("""
CREATE INDEX IF NOT EXISTS idx_entity_title_trgm
ON entity USING gin (title gin_trgm_ops)
""")
op.execute("""
CREATE INDEX IF NOT EXISTS idx_entity_permalink_trgm
ON entity USING gin (permalink gin_trgm_ops)
""")
# Create partial index on unresolved relations for efficient bulk resolution
# This makes "WHERE to_id IS NULL AND project_id = X" queries very fast
op.execute("""
CREATE INDEX IF NOT EXISTS idx_relation_unresolved
ON relation (project_id, to_name)
WHERE to_id IS NULL
""")
# Create index on relation.to_name for join performance in bulk resolution
op.execute("""
CREATE INDEX IF NOT EXISTS idx_relation_to_name
ON relation (to_name)
""")
def downgrade() -> None:
"""Remove project_id from relation/observation and pg_trgm indexes."""
connection = op.get_bind()
dialect = connection.dialect.name
if dialect == "postgresql":
# Drop Postgres-specific indexes
op.execute("DROP INDEX IF EXISTS idx_relation_to_name")
op.execute("DROP INDEX IF EXISTS idx_relation_unresolved")
op.execute("DROP INDEX IF EXISTS idx_entity_permalink_trgm")
op.execute("DROP INDEX IF EXISTS idx_entity_title_trgm")
# Note: We don't drop the pg_trgm extension as other code may depend on it
# Drop project_id from observation
op.drop_index("ix_observation_project_id", table_name="observation")
op.drop_constraint("fk_observation_project_id", "observation", type_="foreignkey")
op.drop_column("observation", "project_id")
# Drop project_id from relation
op.drop_index("ix_relation_project_id", table_name="relation")
op.drop_constraint("fk_relation_project_id", "relation", type_="foreignkey")
op.drop_column("relation", "project_id")
else:
# SQLite requires batch operations
op.drop_index("ix_observation_project_id", table_name="observation")
with op.batch_alter_table("observation") as batch_op:
batch_op.drop_constraint("fk_observation_project_id", type_="foreignkey")
batch_op.drop_column("project_id")
op.drop_index("ix_relation_project_id", table_name="relation")
with op.batch_alter_table("relation") as batch_op:
batch_op.drop_constraint("fk_relation_project_id", type_="foreignkey")
batch_op.drop_column("project_id")
@@ -0,0 +1,173 @@
"""Add external_id UUID column to project and entity tables
Revision ID: g9a0b3c4d5e6
Revises: f8a9b2c3d4e5
Create Date: 2025-12-29 10:00:00.000000
"""
import uuid
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy import text
def column_exists(connection, table: str, column: str) -> bool:
"""Check if a column exists in a table (idempotent migration support)."""
if connection.dialect.name == "postgresql":
result = connection.execute(
text(
"SELECT 1 FROM information_schema.columns "
"WHERE table_name = :table AND column_name = :column"
),
{"table": table, "column": column},
)
return result.fetchone() is not None
else:
# SQLite
result = connection.execute(text(f"PRAGMA table_info({table})"))
columns = [row[1] for row in result]
return column in columns
def index_exists(connection, index_name: str) -> bool:
"""Check if an index exists (idempotent migration support)."""
if connection.dialect.name == "postgresql":
result = connection.execute(
text("SELECT 1 FROM pg_indexes WHERE indexname = :index_name"),
{"index_name": index_name},
)
return result.fetchone() is not None
else:
# SQLite
result = connection.execute(
text("SELECT 1 FROM sqlite_master WHERE type='index' AND name = :index_name"),
{"index_name": index_name},
)
return result.fetchone() is not None
# revision identifiers, used by Alembic.
revision: str = "g9a0b3c4d5e6"
down_revision: Union[str, None] = "f8a9b2c3d4e5"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Add external_id UUID column to project and entity tables.
This migration:
1. Adds external_id column to project table
2. Adds external_id column to entity table
3. Generates UUIDs for existing rows
4. Creates unique indexes on both columns
"""
connection = op.get_bind()
dialect = connection.dialect.name
# -------------------------------------------------------------------------
# Add external_id to project table
# -------------------------------------------------------------------------
if not column_exists(connection, "project", "external_id"):
# Step 1: Add external_id column as nullable first
op.add_column("project", sa.Column("external_id", sa.String(), nullable=True))
# Step 2: Generate UUIDs for existing rows
if dialect == "postgresql":
# Postgres has gen_random_uuid() function
op.execute("""
UPDATE project
SET external_id = gen_random_uuid()::text
WHERE external_id IS NULL
""")
else:
# SQLite: need to generate UUIDs in Python
result = connection.execute(text("SELECT id FROM project WHERE external_id IS NULL"))
for row in result:
new_uuid = str(uuid.uuid4())
connection.execute(
text("UPDATE project SET external_id = :uuid WHERE id = :id"),
{"uuid": new_uuid, "id": row[0]},
)
# Step 3: Make external_id NOT NULL
if dialect == "postgresql":
op.alter_column("project", "external_id", nullable=False)
else:
# SQLite requires batch operations for ALTER COLUMN
with op.batch_alter_table("project") as batch_op:
batch_op.alter_column("external_id", nullable=False)
# Step 4: Create unique index on project.external_id (idempotent)
if not index_exists(connection, "ix_project_external_id"):
op.create_index("ix_project_external_id", "project", ["external_id"], unique=True)
# -------------------------------------------------------------------------
# Add external_id to entity table
# -------------------------------------------------------------------------
if not column_exists(connection, "entity", "external_id"):
# Step 1: Add external_id column as nullable first
op.add_column("entity", sa.Column("external_id", sa.String(), nullable=True))
# Step 2: Generate UUIDs for existing rows
if dialect == "postgresql":
# Postgres has gen_random_uuid() function
op.execute("""
UPDATE entity
SET external_id = gen_random_uuid()::text
WHERE external_id IS NULL
""")
else:
# SQLite: need to generate UUIDs in Python
result = connection.execute(text("SELECT id FROM entity WHERE external_id IS NULL"))
for row in result:
new_uuid = str(uuid.uuid4())
connection.execute(
text("UPDATE entity SET external_id = :uuid WHERE id = :id"),
{"uuid": new_uuid, "id": row[0]},
)
# Step 3: Make external_id NOT NULL
if dialect == "postgresql":
op.alter_column("entity", "external_id", nullable=False)
else:
# SQLite requires batch operations for ALTER COLUMN
with op.batch_alter_table("entity") as batch_op:
batch_op.alter_column("external_id", nullable=False)
# Step 4: Create unique index on entity.external_id (idempotent)
if not index_exists(connection, "ix_entity_external_id"):
op.create_index("ix_entity_external_id", "entity", ["external_id"], unique=True)
def downgrade() -> None:
"""Remove external_id columns from project and entity tables."""
connection = op.get_bind()
dialect = connection.dialect.name
# Drop from entity table
if index_exists(connection, "ix_entity_external_id"):
op.drop_index("ix_entity_external_id", table_name="entity")
if column_exists(connection, "entity", "external_id"):
if dialect == "postgresql":
op.drop_column("entity", "external_id")
else:
with op.batch_alter_table("entity") as batch_op:
batch_op.drop_column("external_id")
# Drop from project table
if index_exists(connection, "ix_project_external_id"):
op.drop_index("ix_project_external_id", table_name="project")
if column_exists(connection, "project", "external_id"):
if dialect == "postgresql":
op.drop_column("project", "external_id")
else:
with op.batch_alter_table("project") as batch_op:
batch_op.drop_column("external_id")
+44 -23
View File
@@ -1,6 +1,5 @@
"""FastAPI application for basic-memory knowledge graph API."""
import asyncio
from contextlib import asynccontextmanager
from fastapi import FastAPI, HTTPException
@@ -8,7 +7,7 @@ from fastapi.exception_handlers import http_exception_handler
from loguru import logger
from basic_memory import __version__ as version
from basic_memory import db
from basic_memory.api.container import ApiContainer, set_container
from basic_memory.api.routers import (
directory_router,
importer_router,
@@ -20,42 +19,57 @@ from basic_memory.api.routers import (
search,
prompt_router,
)
from basic_memory.config import ConfigManager
from basic_memory.services.initialization import initialize_file_sync, initialize_app
from basic_memory.api.v2.routers import (
knowledge_router as v2_knowledge,
project_router as v2_project,
memory_router as v2_memory,
search_router as v2_search,
resource_router as v2_resource,
directory_router as v2_directory,
prompt_router as v2_prompt,
importer_router as v2_importer,
)
from basic_memory.config import init_api_logging
from basic_memory.services.initialization import initialize_app
@asynccontextmanager
async def lifespan(app: FastAPI): # pragma: no cover
"""Lifecycle manager for the FastAPI app. Not called in stdio mcp mode"""
app_config = ConfigManager().config
logger.info("Starting Basic Memory API")
# Initialize logging for API (stdout in cloud mode, file otherwise)
init_api_logging()
await initialize_app(app_config)
# --- Composition Root ---
# Create container and read config (single point of config access)
container = ApiContainer.create()
set_container(container)
app.state.container = container
logger.info(f"Starting Basic Memory API (mode={container.mode.name})")
await initialize_app(container.config)
# Cache database connections in app state for performance
logger.info("Initializing database and caching connections...")
engine, session_maker = await db.get_or_create_db(app_config.database_path)
engine, session_maker = await container.init_database()
app.state.engine = engine
app.state.session_maker = session_maker
logger.info("Database connections cached in app state")
logger.info(f"Sync changes enabled: {app_config.sync_changes}")
if app_config.sync_changes:
# start file sync task in background
app.state.sync_task = asyncio.create_task(initialize_file_sync(app_config))
else:
logger.info("Sync changes disabled. Skipping file sync service.")
# Create and start sync coordinator (lifecycle centralized in coordinator)
sync_coordinator = container.create_sync_coordinator()
await sync_coordinator.start()
app.state.sync_coordinator = sync_coordinator
# proceed with startup
# Proceed with startup
yield
# Shutdown - coordinator handles clean task cancellation
logger.info("Shutting down Basic Memory API")
if app.state.sync_task:
logger.info("Stopping sync...")
app.state.sync_task.cancel() # pyright: ignore
await sync_coordinator.stop()
await db.shutdown_db()
await container.shutdown_database()
# Initialize FastAPI app
@@ -66,8 +80,17 @@ app = FastAPI(
lifespan=lifespan,
)
# Include v2 routers FIRST (more specific paths must match before /{project} catch-all)
app.include_router(v2_knowledge, prefix="/v2/projects/{project_id}")
app.include_router(v2_memory, prefix="/v2/projects/{project_id}")
app.include_router(v2_search, prefix="/v2/projects/{project_id}")
app.include_router(v2_resource, prefix="/v2/projects/{project_id}")
app.include_router(v2_directory, prefix="/v2/projects/{project_id}")
app.include_router(v2_prompt, prefix="/v2/projects/{project_id}")
app.include_router(v2_importer, prefix="/v2/projects/{project_id}")
app.include_router(v2_project, prefix="/v2")
# Include routers
# Include v1 routers (/{project} is a catch-all, must come after specific prefixes)
app.include_router(knowledge.router, prefix="/{project}")
app.include_router(memory.router, prefix="/{project}")
app.include_router(resource.router, prefix="/{project}")
@@ -77,12 +100,10 @@ app.include_router(directory_router.router, prefix="/{project}")
app.include_router(prompt_router.router, prefix="/{project}")
app.include_router(importer_router.router, prefix="/{project}")
# Project resource router works accross projects
# Project resource router works across projects
app.include_router(project.project_resource_router)
app.include_router(management.router)
# Auth routes are handled by FastMCP automatically when auth is enabled
@app.exception_handler(Exception)
async def exception_handler(request, exc): # pragma: no cover
+133
View File
@@ -0,0 +1,133 @@
"""API composition root for Basic Memory.
This container owns reading ConfigManager and environment variables for the
API entrypoint. Downstream modules receive config/dependencies explicitly
rather than reading globals.
Design principles:
- Only this module reads ConfigManager directly
- Runtime mode (cloud/local/test) is resolved here
- Factories for services are provided, not singletons
"""
from dataclasses import dataclass
from typing import TYPE_CHECKING
from sqlalchemy.ext.asyncio import AsyncEngine, async_sessionmaker, AsyncSession
from basic_memory import db
from basic_memory.config import BasicMemoryConfig, ConfigManager
from basic_memory.runtime import RuntimeMode, resolve_runtime_mode
if TYPE_CHECKING: # pragma: no cover
from basic_memory.sync import SyncCoordinator
@dataclass
class ApiContainer:
"""Composition root for the API entrypoint.
Holds resolved configuration and runtime context.
Created once at app startup, then used to wire dependencies.
"""
config: BasicMemoryConfig
mode: RuntimeMode
# --- Database ---
# Cached database connections (set during lifespan startup)
engine: AsyncEngine | None = None
session_maker: async_sessionmaker[AsyncSession] | None = None
@classmethod
def create(cls) -> "ApiContainer": # pragma: no cover
"""Create container by reading ConfigManager.
This is the single point where API reads global config.
"""
config = ConfigManager().config
mode = resolve_runtime_mode(
cloud_mode_enabled=config.cloud_mode_enabled,
is_test_env=config.is_test_env,
)
return cls(config=config, mode=mode)
# --- Runtime Mode Properties ---
@property
def should_sync_files(self) -> bool:
"""Whether file sync should be started.
Sync is enabled when:
- sync_changes is True in config
- Not in test mode (tests manage their own sync)
"""
return self.config.sync_changes and not self.mode.is_test
@property
def sync_skip_reason(self) -> str | None: # pragma: no cover
"""Reason why sync is skipped, or None if sync should run.
Useful for logging why sync was disabled.
"""
if self.mode.is_test:
return "Test environment detected"
if not self.config.sync_changes:
return "Sync changes disabled"
return None
def create_sync_coordinator(self) -> "SyncCoordinator": # pragma: no cover
"""Create a SyncCoordinator with this container's settings.
Returns:
SyncCoordinator configured for this runtime environment
"""
# Deferred import to avoid circular dependency
from basic_memory.sync import SyncCoordinator
return SyncCoordinator(
config=self.config,
should_sync=self.should_sync_files,
skip_reason=self.sync_skip_reason,
)
# --- Database Factory ---
async def init_database( # pragma: no cover
self,
) -> tuple[AsyncEngine, async_sessionmaker[AsyncSession]]:
"""Initialize and cache database connections.
Returns:
Tuple of (engine, session_maker)
"""
engine, session_maker = await db.get_or_create_db(self.config.database_path)
self.engine = engine
self.session_maker = session_maker
return engine, session_maker
async def shutdown_database(self) -> None: # pragma: no cover
"""Clean up database connections."""
await db.shutdown_db()
# Module-level container instance (set by lifespan)
# This allows deps.py to access the container without reading ConfigManager
_container: ApiContainer | None = None
def get_container() -> ApiContainer:
"""Get the current API container.
Raises:
RuntimeError: If container hasn't been initialized
"""
if _container is None:
raise RuntimeError("API container not initialized. Call set_container() first.")
return _container
def set_container(container: ApiContainer) -> None:
"""Set the API container (called by lifespan)."""
global _container
_container = container
@@ -1,4 +1,11 @@
"""Router for knowledge graph operations."""
"""Router for knowledge graph operations.
⚠️ DEPRECATED: This v1 API is deprecated and will be removed on June 30, 2026.
Please migrate to /v2/{project}/knowledge endpoints which use entity IDs instead
of path-based identifiers for improved performance and stability.
Migration guide: See docs/migration/v1-to-v2.md
"""
from typing import Annotated
@@ -25,7 +32,11 @@ from basic_memory.schemas import (
from basic_memory.schemas.request import EditEntityRequest, MoveEntityRequest
from basic_memory.schemas.base import Permalink, Entity
router = APIRouter(prefix="/knowledge", tags=["knowledge"])
router = APIRouter(
prefix="/knowledge",
tags=["knowledge"],
deprecated=True, # Marks entire router as deprecated in OpenAPI docs
)
async def resolve_relations_background(sync_service, entity_id: int, entity_permalink: str) -> None:
@@ -40,9 +51,10 @@ async def resolve_relations_background(sync_service, entity_id: int, entity_perm
logger.debug(
f"Background: Resolved relations for entity {entity_permalink} (id={entity_id})"
)
except Exception as e:
# Log but don't fail - this is a background task
logger.warning(
except Exception as e: # pragma: no cover
# Log but don't fail - this is a background task.
# Avoid forcing synthetic failures just for coverage.
logger.warning( # pragma: no cover
f"Background: Failed to resolve relations for entity {entity_permalink}: {e}"
)
+162 -28
View File
@@ -1,7 +1,7 @@
"""Router for project management."""
import os
from fastapi import APIRouter, HTTPException, Path, Body, BackgroundTasks
from fastapi import APIRouter, HTTPException, Path, Body, BackgroundTasks, Response, Query
from typing import Optional
from loguru import logger
@@ -18,6 +18,7 @@ from basic_memory.schemas.project_info import (
ProjectInfoRequest,
ProjectStatusResponse,
)
from basic_memory.utils import normalize_project_path
# Router for resources in a specific project
# The ProjectPathDep is used in the path as a prefix, so the request path is like /{project}/project/info
@@ -49,8 +50,10 @@ async def get_project(
) # pragma: no cover
return ProjectItem(
id=found_project.id,
external_id=found_project.external_id,
name=found_project.name,
path=found_project.path,
path=normalize_project_path(found_project.path),
is_default=found_project.is_default or False,
)
@@ -79,9 +82,18 @@ async def update_project(
raise HTTPException(status_code=400, detail="Path must be absolute")
# Get original project info for the response
old_project = await project_service.get_project(name)
if not old_project:
raise HTTPException(
status_code=400, detail=f"Project '{name}' not found in configuration"
)
old_project_info = ProjectItem(
name=name,
path=project_service.projects.get(name, ""),
id=old_project.id,
external_id=old_project.external_id,
name=old_project.name,
path=old_project.path,
is_default=old_project.is_default or False,
)
if path:
@@ -90,17 +102,27 @@ async def update_project(
await project_service.update_project(name, is_active=is_active)
# Get updated project info
updated_path = path if path else project_service.projects.get(name, "")
updated_project = await project_service.get_project(name)
if not updated_project:
raise HTTPException( # pragma: no cover
status_code=404, detail=f"Project '{name}' not found after update"
)
return ProjectStatusResponse(
message=f"Project '{name}' updated successfully",
status="success",
default=(name == project_service.default_project),
old_project=old_project_info,
new_project=ProjectItem(name=name, path=updated_path),
new_project=ProjectItem(
id=updated_project.id,
external_id=updated_project.external_id,
name=updated_project.name,
path=updated_project.path,
is_default=updated_project.is_default or False,
),
)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
raise HTTPException(status_code=400, detail=str(e)) # pragma: no cover
# Sync project filesystem
@@ -109,6 +131,10 @@ async def sync_project(
background_tasks: BackgroundTasks,
sync_service: SyncServiceDep,
project_config: ProjectConfigDep,
force_full: bool = Query(
False, description="Force full scan, bypassing watermark optimization"
),
run_in_background: bool = Query(True, description="Run in background"),
):
"""Force project filesystem sync to database.
@@ -118,17 +144,32 @@ async def sync_project(
background_tasks: FastAPI background tasks
sync_service: Sync service for this project
project_config: Project configuration
force_full: If True, force a full scan even if watermark exists
run_in_background: If True, run sync in background and return immediately
Returns:
Response confirming sync was initiated
Response confirming sync was initiated (background) or SyncReportResponse (foreground)
"""
background_tasks.add_task(sync_service.sync, project_config.home, project_config.name)
logger.info(f"Filesystem sync initiated for project: {project_config.name}")
if run_in_background:
background_tasks.add_task(
sync_service.sync, project_config.home, project_config.name, force_full=force_full
)
logger.info(
f"Filesystem sync initiated for project: {project_config.name} (force_full={force_full})"
)
return {
"status": "sync_started",
"message": f"Filesystem sync initiated for project '{project_config.name}'",
}
return {
"status": "sync_started",
"message": f"Filesystem sync initiated for project '{project_config.name}'",
}
else:
report = await sync_service.sync(
project_config.home, project_config.name, force_full=force_full
)
logger.info(
f"Filesystem sync completed for project: {project_config.name} (force_full={force_full})"
)
return SyncReportResponse.from_sync_report(report)
@project_router.post("/status", response_model=SyncReportResponse)
@@ -145,10 +186,10 @@ async def project_sync_status(
Returns:
Scan report with details on files that need syncing
"""
logger.info(f"Scanning filesystem for project: {project_config.name}")
sync_report = await sync_service.scan(project_config.home)
logger.info(f"Scanning filesystem for project: {project_config.name}") # pragma: no cover
sync_report = await sync_service.scan(project_config.home) # pragma: no cover
return SyncReportResponse.from_sync_report(sync_report)
return SyncReportResponse.from_sync_report(sync_report) # pragma: no cover
# List all available projects
@@ -166,8 +207,10 @@ async def list_projects(
project_items = [
ProjectItem(
id=project.id,
external_id=project.external_id,
name=project.name,
path=project.path,
path=normalize_project_path(project.path),
is_default=project.is_default or False,
)
for project in projects
@@ -180,8 +223,9 @@ async def list_projects(
# Add a new project
@project_resource_router.post("/projects", response_model=ProjectStatusResponse)
@project_resource_router.post("/projects", response_model=ProjectStatusResponse, status_code=201)
async def add_project(
response: Response,
project_data: ProjectInfoRequest,
project_service: ProjectServiceDep,
) -> ProjectStatusResponse:
@@ -193,18 +237,59 @@ async def add_project(
Returns:
Response confirming the project was added
"""
# Check if project already exists before attempting to add
existing_project = await project_service.get_project(project_data.name)
if existing_project:
# Project exists - check if paths match for true idempotency
# Normalize paths for comparison (resolve symlinks, etc.)
from pathlib import Path
requested_path = Path(project_data.path).resolve()
existing_path = Path(existing_project.path).resolve()
if requested_path == existing_path:
# Same name, same path - return 200 OK (idempotent)
response.status_code = 200
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message=f"Project '{project_data.name}' already exists",
status="success",
default=existing_project.is_default or False,
new_project=ProjectItem(
id=existing_project.id,
external_id=existing_project.external_id,
name=existing_project.name,
path=existing_project.path,
is_default=existing_project.is_default or False,
),
)
else:
# Same name, different path - this is an error
raise HTTPException(
status_code=400,
detail=f"Project '{project_data.name}' already exists with different path. Existing: {existing_project.path}, Requested: {project_data.path}",
)
try: # pragma: no cover
# The service layer now handles cloud mode validation and path sanitization
await project_service.add_project(
project_data.name, project_data.path, set_default=project_data.set_default
)
# Fetch the newly created project to get its ID
new_project = await project_service.get_project(project_data.name)
if not new_project:
raise HTTPException(status_code=500, detail="Failed to retrieve newly created project")
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message=f"Project '{project_data.name}' added successfully",
message=f"Project '{new_project.name}' added successfully",
status="success",
default=project_data.set_default,
new_project=ProjectItem(
name=project_data.name, path=project_data.path, is_default=project_data.set_default
id=new_project.id,
external_id=new_project.external_id,
name=new_project.name,
path=new_project.path,
is_default=new_project.is_default or False,
),
)
except ValueError as e: # pragma: no cover
@@ -216,11 +301,15 @@ async def add_project(
async def remove_project(
project_service: ProjectServiceDep,
name: str = Path(..., description="Name of the project to remove"),
delete_notes: bool = Query(
False, description="If True, delete project directory from filesystem"
),
) -> ProjectStatusResponse:
"""Remove a project from configuration and database.
Args:
name: The name of the project to remove
delete_notes: If True, delete the project directory from the filesystem
Returns:
Response confirming the project was removed
@@ -232,13 +321,39 @@ async def remove_project(
status_code=404, detail=f"Project: '{name}' does not exist"
) # pragma: no cover
await project_service.remove_project(name)
# Check if trying to delete the default project
# In cloud mode, database is source of truth; in local mode, check config
config_default = project_service.default_project
db_default = await project_service.repository.get_default_project()
# Use database default if available, otherwise fall back to config default
default_project_name = db_default.name if db_default else config_default
if name == default_project_name:
available_projects = await project_service.list_projects()
other_projects = [p.name for p in available_projects if p.name != name]
detail = f"Cannot delete default project '{name}'. "
if other_projects:
detail += (
f"Set another project as default first. Available: {', '.join(other_projects)}"
)
else:
detail += "This is the only project in your configuration."
raise HTTPException(status_code=400, detail=detail)
await project_service.remove_project(name, delete_notes=delete_notes)
return ProjectStatusResponse(
message=f"Project '{name}' removed successfully",
message=f"Project '{old_project.name}' removed successfully",
status="success",
default=False,
old_project=ProjectItem(name=old_project.name, path=old_project.path),
old_project=ProjectItem(
id=old_project.id,
external_id=old_project.external_id,
name=old_project.name,
path=old_project.path,
is_default=old_project.is_default or False,
),
new_project=None,
)
except ValueError as e: # pragma: no cover
@@ -281,8 +396,16 @@ async def set_default_project(
message=f"Project '{name}' set as default successfully",
status="success",
default=True,
old_project=ProjectItem(name=default_name, path=default_project.path),
old_project=ProjectItem(
id=default_project.id,
external_id=default_project.external_id,
name=default_name,
path=default_project.path,
is_default=False,
),
new_project=ProjectItem(
id=new_default_project.id,
external_id=new_default_project.external_id,
name=name,
path=new_default_project.path,
is_default=True,
@@ -302,15 +425,26 @@ async def get_default_project(
Returns:
Response with project default information
"""
# Get the old default project
default_name = project_service.default_project
# Get the default project
# In cloud mode, database is source of truth; in local mode, check config
config_default = project_service.default_project
db_default = await project_service.repository.get_default_project()
# Use database default if available, otherwise fall back to config default
default_name = db_default.name if db_default else config_default
default_project = await project_service.get_project(default_name)
if not default_project: # pragma: no cover
raise HTTPException( # pragma: no cover
status_code=404, detail=f"Default Project: '{default_name}' does not exist"
)
return ProjectItem(name=default_project.name, path=default_project.path, is_default=True)
return ProjectItem(
id=default_project.id,
external_id=default_project.external_id,
name=default_project.name,
path=default_project.path,
is_default=True,
)
# Synchronize projects between config and database
+52 -25
View File
@@ -1,10 +1,11 @@
"""Routes for getting entity content."""
import tempfile
import uuid
from pathlib import Path
from typing import Annotated
from typing import Annotated, Union
from fastapi import APIRouter, HTTPException, BackgroundTasks, Body
from fastapi import APIRouter, HTTPException, BackgroundTasks, Body, Response
from fastapi.responses import FileResponse, JSONResponse
from loguru import logger
@@ -25,6 +26,17 @@ from datetime import datetime
router = APIRouter(prefix="/resource", tags=["resources"])
def _mtime_to_datetime(entity: EntityModel) -> datetime:
"""Convert entity mtime (file modification time) to datetime.
Returns the file's actual modification time, falling back to updated_at
if mtime is not available.
"""
if entity.mtime: # pragma: no cover
return datetime.fromtimestamp(entity.mtime).astimezone() # pragma: no cover
return entity.updated_at
def get_entity_ids(item: SearchIndexRow) -> set[int]:
match item.type:
case SearchItemType.ENTITY:
@@ -39,7 +51,7 @@ def get_entity_ids(item: SearchIndexRow) -> set[int]:
raise ValueError(f"Unexpected type: {item.type}")
@router.get("/{identifier:path}")
@router.get("/{identifier:path}", response_model=None)
async def get_resource_content(
config: ProjectConfigDep,
link_resolver: LinkResolverDep,
@@ -50,7 +62,7 @@ async def get_resource_content(
identifier: str,
page: int = 1,
page_size: int = 10,
) -> FileResponse:
) -> Union[Response, FileResponse]:
"""Get resource content by identifier: name or permalink."""
logger.debug(f"Getting content for: {identifier}")
@@ -81,13 +93,16 @@ async def get_resource_content(
# return single response
if len(results) == 1:
entity = results[0]
file_path = Path(f"{config.home}/{entity.file_path}")
if not file_path.exists():
# Check file exists via file_service (for cloud compatibility)
if not await file_service.exists(entity.file_path):
raise HTTPException(
status_code=404,
detail=f"File not found: {file_path}",
detail=f"File not found: {entity.file_path}",
)
return FileResponse(path=file_path)
# Read content via file_service as bytes (works with both local and S3)
content = await file_service.read_file_bytes(entity.file_path)
content_type = file_service.content_type(entity.file_path)
return Response(content=content, media_type=content_type)
# for multiple files, initialize a temporary file for writing the results
with tempfile.NamedTemporaryFile(delete=False, mode="w", suffix=".md") as tmp_file:
@@ -97,7 +112,7 @@ async def get_resource_content(
# Read content for each entity
content = await file_service.read_entity_content(result)
memory_url = normalize_memory_url(result.permalink)
modified_date = result.updated_at.isoformat()
modified_date = _mtime_to_datetime(result).isoformat()
checksum = result.checksum[:8] if result.checksum else ""
# Prepare the delimited content
@@ -151,27 +166,37 @@ async def write_resource(
try:
# Get content from request body
# Defensive type checking: ensure content is a string
# FastAPI should validate this, but if a dict somehow gets through
# (e.g., via JSON body parsing), we need to catch it here
if isinstance(content, dict):
logger.error( # pragma: no cover
f"Error writing resource {file_path}: "
f"content is a dict, expected string. Keys: {list(content.keys())}"
)
raise HTTPException( # pragma: no cover
status_code=400,
detail="content must be a string, not a dict. "
"Ensure request body is sent as raw string content, not JSON object.",
)
# Ensure it's UTF-8 string content
if isinstance(content, bytes): # pragma: no cover
content_str = content.decode("utf-8")
else:
content_str = str(content)
# Get full file path
full_path = Path(f"{config.home}/{file_path}")
# Ensure parent directory exists
full_path.parent.mkdir(parents=True, exist_ok=True)
# Write content to file
checksum = await file_service.write_file(full_path, content_str)
# Cloud compatibility: do not assume a local filesystem path structure.
# Delegate directory creation + writes to the configured FileService (local or S3).
await file_service.ensure_directory(Path(file_path).parent)
checksum = await file_service.write_file(file_path, content_str)
# Get file info
file_stats = file_service.file_stats(full_path)
file_metadata = await file_service.get_file_metadata(file_path)
# Determine file details
file_name = Path(file_path).name
content_type = file_service.content_type(full_path)
content_type = file_service.content_type(file_path)
entity_type = "canvas" if file_path.endswith(".canvas") else "file"
@@ -188,20 +213,22 @@ async def write_resource(
"content_type": content_type,
"file_path": file_path,
"checksum": checksum,
"updated_at": datetime.fromtimestamp(file_stats.st_mtime).astimezone(),
"updated_at": file_metadata.modified_at,
},
)
status_code = 200
else:
# Create a new entity model
# Explicitly set external_id to ensure NOT NULL constraint is satisfied (fixes #512)
entity = EntityModel(
external_id=str(uuid.uuid4()),
title=file_name,
entity_type=entity_type,
content_type=content_type,
file_path=file_path,
checksum=checksum,
created_at=datetime.fromtimestamp(file_stats.st_ctime).astimezone(),
updated_at=datetime.fromtimestamp(file_stats.st_mtime).astimezone(),
created_at=file_metadata.created_at,
updated_at=file_metadata.modified_at,
)
entity = await entity_repository.add(entity)
status_code = 201
@@ -215,9 +242,9 @@ async def write_resource(
content={
"file_path": file_path,
"checksum": checksum,
"size": file_stats.st_size,
"created_at": file_stats.st_ctime,
"modified_at": file_stats.st_mtime,
"size": file_metadata.size,
"created_at": file_metadata.created_at.timestamp(),
"modified_at": file_metadata.modified_at.timestamp(),
},
)
except Exception as e: # pragma: no cover
+53 -14
View File
@@ -24,11 +24,30 @@ async def to_graph_context(
page: Optional[int] = None,
page_size: Optional[int] = None,
):
# First pass: collect all entity IDs needed for relations
entity_ids_needed: set[int] = set()
for context_item in context_result.results:
for item in (
[context_item.primary_result] + context_item.observations + context_item.related_results
):
if item.type == SearchItemType.RELATION:
if item.from_id: # pyright: ignore
entity_ids_needed.add(item.from_id) # pyright: ignore
if item.to_id:
entity_ids_needed.add(item.to_id)
# Batch fetch all entities at once
entity_lookup: dict[int, str] = {}
if entity_ids_needed:
entities = await entity_repository.find_by_ids(list(entity_ids_needed))
entity_lookup = {e.id: e.title for e in entities}
# Helper function to convert items to summaries
async def to_summary(item: SearchIndexRow | ContextResultRow):
def to_summary(item: SearchIndexRow | ContextResultRow):
match item.type:
case SearchItemType.ENTITY:
return EntitySummary(
entity_id=item.id,
title=item.title, # pyright: ignore
permalink=item.permalink,
content=item.content,
@@ -37,6 +56,8 @@ async def to_graph_context(
)
case SearchItemType.OBSERVATION:
return ObservationSummary(
observation_id=item.id,
entity_id=item.entity_id, # pyright: ignore
title=item.title, # pyright: ignore
file_path=item.file_path,
category=item.category, # pyright: ignore
@@ -45,15 +66,19 @@ async def to_graph_context(
created_at=item.created_at,
)
case SearchItemType.RELATION:
from_entity = await entity_repository.find_by_id(item.from_id) # pyright: ignore
to_entity = await entity_repository.find_by_id(item.to_id) if item.to_id else None
from_title = entity_lookup.get(item.from_id) if item.from_id else None # pyright: ignore
to_title = entity_lookup.get(item.to_id) if item.to_id else None
return RelationSummary(
relation_id=item.id,
entity_id=item.entity_id, # pyright: ignore
title=item.title, # pyright: ignore
file_path=item.file_path,
permalink=item.permalink, # pyright: ignore
relation_type=item.relation_type, # pyright: ignore
from_entity=from_entity.title if from_entity else None,
to_entity=to_entity.title if to_entity else None,
from_entity=from_title,
from_entity_id=item.from_id, # pyright: ignore
to_entity=to_title,
to_entity_id=item.to_id,
created_at=item.created_at,
)
case _: # pragma: no cover
@@ -63,23 +88,19 @@ async def to_graph_context(
hierarchical_results = []
for context_item in context_result.results:
# Process primary result
primary_result = await to_summary(context_item.primary_result)
primary_result = to_summary(context_item.primary_result)
# Process observations
observations = []
for obs in context_item.observations:
observations.append(await to_summary(obs))
# Process observations (always ObservationSummary, validated by context_service)
observations = [to_summary(obs) for obs in context_item.observations]
# Process related results
related = []
for rel in context_item.related_results:
related.append(await to_summary(rel))
related = [to_summary(rel) for rel in context_item.related_results]
# Add to hierarchical results
hierarchical_results.append(
ContextResult(
primary_result=primary_result,
observations=observations,
observations=observations, # pyright: ignore[reportArgumentType]
related_results=related,
)
)
@@ -111,6 +132,21 @@ async def to_search_results(entity_service: EntityService, results: List[SearchI
search_results = []
for r in results:
entities = await entity_service.get_entities_by_id([r.entity_id, r.from_id, r.to_id]) # pyright: ignore
# Determine which IDs to set based on type
entity_id = None
observation_id = None
relation_id = None
if r.type == SearchItemType.ENTITY:
entity_id = r.id
elif r.type == SearchItemType.OBSERVATION:
observation_id = r.id
entity_id = r.entity_id # Parent entity
elif r.type == SearchItemType.RELATION:
relation_id = r.id
entity_id = r.entity_id # Parent entity
search_results.append(
SearchResult(
title=r.title, # pyright: ignore
@@ -121,6 +157,9 @@ async def to_search_results(entity_service: EntityService, results: List[SearchI
content=r.content,
file_path=r.file_path,
metadata=r.metadata,
entity_id=entity_id,
observation_id=observation_id,
relation_id=relation_id,
category=r.category,
from_entity=entities[0].permalink if entities else None,
to_entity=entities[1].permalink if len(entities) > 1 else None,
+35
View File
@@ -0,0 +1,35 @@
"""API v2 module - ID-based entity references.
Version 2 of the Basic Memory API uses integer entity IDs as the primary
identifier for improved performance and stability.
Key changes from v1:
- Entity lookups use integer IDs instead of paths/permalinks
- Direct database queries instead of cascading resolution
- Stable references that don't change with file moves
- Better caching support
All v2 routers are registered with the /v2 prefix.
"""
from basic_memory.api.v2.routers import (
knowledge_router,
memory_router,
project_router,
resource_router,
search_router,
directory_router,
prompt_router,
importer_router,
)
__all__ = [
"knowledge_router",
"memory_router",
"project_router",
"resource_router",
"search_router",
"directory_router",
"prompt_router",
"importer_router",
]
@@ -0,0 +1,21 @@
"""V2 API routers."""
from basic_memory.api.v2.routers.knowledge_router import router as knowledge_router
from basic_memory.api.v2.routers.project_router import router as project_router
from basic_memory.api.v2.routers.memory_router import router as memory_router
from basic_memory.api.v2.routers.search_router import router as search_router
from basic_memory.api.v2.routers.resource_router import router as resource_router
from basic_memory.api.v2.routers.directory_router import router as directory_router
from basic_memory.api.v2.routers.prompt_router import router as prompt_router
from basic_memory.api.v2.routers.importer_router import router as importer_router
__all__ = [
"knowledge_router",
"project_router",
"memory_router",
"search_router",
"resource_router",
"directory_router",
"prompt_router",
"importer_router",
]
@@ -0,0 +1,93 @@
"""V2 Directory Router - ID-based directory tree operations.
This router provides directory structure browsing for projects using
external_id UUIDs instead of name-based identifiers.
Key improvements:
- Direct project lookup via external_id UUIDs
- Consistent with other v2 endpoints
- Better performance through indexed queries
"""
from typing import List, Optional
from fastapi import APIRouter, Query, Path
from basic_memory.deps import DirectoryServiceV2ExternalDep
from basic_memory.schemas.directory import DirectoryNode
router = APIRouter(prefix="/directory", tags=["directory-v2"])
@router.get("/tree", response_model=DirectoryNode, response_model_exclude_none=True)
async def get_directory_tree(
directory_service: DirectoryServiceV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
):
"""Get hierarchical directory structure from the knowledge base.
Args:
directory_service: Service for directory operations
project_id: Project external UUID
Returns:
DirectoryNode representing the root of the hierarchical tree structure
"""
# Get a hierarchical directory tree for the specific project
tree = await directory_service.get_directory_tree()
# Return the hierarchical tree
return tree
@router.get("/structure", response_model=DirectoryNode, response_model_exclude_none=True)
async def get_directory_structure(
directory_service: DirectoryServiceV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
):
"""Get folder structure for navigation (no files).
Optimized endpoint for folder tree navigation. Returns only directory nodes
without file metadata. For full tree with files, use /directory/tree.
Args:
directory_service: Service for directory operations
project_id: Project external UUID
Returns:
DirectoryNode tree containing only folders (type="directory")
"""
structure = await directory_service.get_directory_structure()
return structure
@router.get("/list", response_model=List[DirectoryNode], response_model_exclude_none=True)
async def list_directory(
directory_service: DirectoryServiceV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
dir_name: str = Query("/", description="Directory path to list"),
depth: int = Query(1, ge=1, le=10, description="Recursion depth (1-10)"),
file_name_glob: Optional[str] = Query(
None, description="Glob pattern for filtering file names"
),
):
"""List directory contents with filtering and depth control.
Args:
directory_service: Service for directory operations
project_id: Project external UUID
dir_name: Directory path to list (default: root "/")
depth: Recursion depth (1-10, default: 1 for immediate children only)
file_name_glob: Optional glob pattern for filtering file names (e.g., "*.md", "*meeting*")
Returns:
List of DirectoryNode objects matching the criteria
"""
# Get directory listing with filtering
nodes = await directory_service.list_directory(
dir_name=dir_name,
depth=depth,
file_name_glob=file_name_glob,
)
return nodes
@@ -0,0 +1,181 @@
"""V2 Import Router - ID-based data import operations.
This router uses v2 dependencies for consistent project handling with external_id UUIDs.
Import endpoints use project_id in the path for consistency with other v2 endpoints.
"""
import json
import logging
from fastapi import APIRouter, Form, HTTPException, UploadFile, status, Path
from basic_memory.deps import (
ChatGPTImporterV2ExternalDep,
ClaudeConversationsImporterV2ExternalDep,
ClaudeProjectsImporterV2ExternalDep,
MemoryJsonImporterV2ExternalDep,
)
from basic_memory.importers import Importer
from basic_memory.schemas.importer import (
ChatImportResult,
EntityImportResult,
ProjectImportResult,
)
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/import", tags=["import-v2"])
@router.post("/chatgpt", response_model=ChatImportResult)
async def import_chatgpt(
importer: ChatGPTImporterV2ExternalDep,
file: UploadFile,
project_id: str = Path(..., description="Project external UUID"),
folder: str = Form("conversations"),
) -> ChatImportResult:
"""Import conversations from ChatGPT JSON export.
Args:
project_id: Project external UUID from URL path
file: The ChatGPT conversations.json file.
folder: The folder to place the files in.
importer: ChatGPT importer instance.
Returns:
ChatImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
logger.info(f"V2 Importing ChatGPT conversations for project {project_id}")
return await import_file(importer, file, folder)
@router.post("/claude/conversations", response_model=ChatImportResult)
async def import_claude_conversations(
importer: ClaudeConversationsImporterV2ExternalDep,
file: UploadFile,
project_id: str = Path(..., description="Project external UUID"),
folder: str = Form("conversations"),
) -> ChatImportResult:
"""Import conversations from Claude conversations.json export.
Args:
project_id: Project external UUID from URL path
file: The Claude conversations.json file.
folder: The folder to place the files in.
importer: Claude conversations importer instance.
Returns:
ChatImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
logger.info(f"V2 Importing Claude conversations for project {project_id}")
return await import_file(importer, file, folder)
@router.post("/claude/projects", response_model=ProjectImportResult)
async def import_claude_projects(
importer: ClaudeProjectsImporterV2ExternalDep,
file: UploadFile,
project_id: str = Path(..., description="Project external UUID"),
folder: str = Form("projects"),
) -> ProjectImportResult:
"""Import projects from Claude projects.json export.
Args:
project_id: Project external UUID from URL path
file: The Claude projects.json file.
folder: The base folder to place the files in.
importer: Claude projects importer instance.
Returns:
ProjectImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
logger.info(f"V2 Importing Claude projects for project {project_id}")
return await import_file(importer, file, folder)
@router.post("/memory-json", response_model=EntityImportResult)
async def import_memory_json(
importer: MemoryJsonImporterV2ExternalDep,
file: UploadFile,
project_id: str = Path(..., description="Project external UUID"),
folder: str = Form("conversations"),
) -> EntityImportResult:
"""Import entities and relations from a memory.json file.
Args:
project_id: Project external UUID from URL path
file: The memory.json file.
folder: Optional destination folder within the project.
importer: Memory JSON importer instance.
Returns:
EntityImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
logger.info(f"V2 Importing memory.json for project {project_id}")
try:
file_data = []
file_bytes = await file.read()
file_str = file_bytes.decode("utf-8")
for line in file_str.splitlines():
json_data = json.loads(line)
file_data.append(json_data)
result = await importer.import_data(file_data, folder)
if not result.success: # pragma: no cover
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=result.error_message or "Import failed",
)
except Exception as e:
logger.exception("V2 Import failed")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Import failed: {str(e)}",
)
return result
async def import_file(importer: Importer, file: UploadFile, destination_folder: str):
"""Helper function to import a file using an importer instance.
Args:
importer: The importer instance to use
file: The file to import
destination_folder: Destination folder for imported content
Returns:
Import result from the importer
Raises:
HTTPException: If import fails
"""
try:
# Process file
json_data = json.load(file.file)
result = await importer.import_data(json_data, destination_folder)
if not result.success: # pragma: no cover
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=result.error_message or "Import failed",
)
return result
except Exception as e:
logger.exception("V2 Import failed")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Import failed: {str(e)}",
)
@@ -0,0 +1,425 @@
"""V2 Knowledge Router - External ID-based entity operations.
This router provides external_id (UUID) based CRUD operations for entities,
using stable string UUIDs that won't change with file moves or database migrations.
Key improvements:
- Stable external UUIDs that won't change with file moves or renames
- Better API ergonomics with consistent string identifiers
- Direct database lookups via unique indexed column
- Simplified caching strategies
"""
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Response, Path
from loguru import logger
from basic_memory.deps import (
EntityServiceV2ExternalDep,
SearchServiceV2ExternalDep,
LinkResolverV2ExternalDep,
ProjectConfigV2ExternalDep,
AppConfigDep,
SyncServiceV2ExternalDep,
EntityRepositoryV2ExternalDep,
ProjectExternalIdPathDep,
)
from basic_memory.schemas import DeleteEntitiesResponse
from basic_memory.schemas.base import Entity
from basic_memory.schemas.request import EditEntityRequest
from basic_memory.schemas.v2 import (
EntityResolveRequest,
EntityResolveResponse,
EntityResponseV2,
MoveEntityRequestV2,
)
router = APIRouter(prefix="/knowledge", tags=["knowledge-v2"])
async def resolve_relations_background(sync_service, entity_id: int, entity_permalink: str) -> None:
"""Background task to resolve relations for a specific entity.
This runs asynchronously after the API response is sent, preventing
long delays when creating entities with many relations.
"""
try: # pragma: no cover
# Only resolve relations for the newly created entity
await sync_service.resolve_relations(entity_id=entity_id) # pragma: no cover
logger.debug( # pragma: no cover
f"Background: Resolved relations for entity {entity_permalink} (id={entity_id})"
)
except Exception as e: # pragma: no cover
# Log but don't fail - this is a background task
logger.warning( # pragma: no cover
f"Background: Failed to resolve relations for entity {entity_permalink}: {e}"
)
## Resolution endpoint
@router.post("/resolve", response_model=EntityResolveResponse)
async def resolve_identifier(
project_id: ProjectExternalIdPathDep,
data: EntityResolveRequest,
link_resolver: LinkResolverV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
) -> EntityResolveResponse:
"""Resolve a string identifier (external_id, permalink, title, or path) to entity info.
This endpoint provides a bridge between v1-style identifiers and v2 external_ids.
Use this to convert existing references to the new UUID-based format.
Args:
data: Request containing the identifier to resolve
Returns:
Entity external_id and metadata about how it was resolved
Raises:
HTTPException: 404 if identifier cannot be resolved
Example:
POST /v2/{project_id}/knowledge/resolve
{"identifier": "specs/search"}
Returns:
{
"external_id": "550e8400-e29b-41d4-a716-446655440000",
"entity_id": 123,
"permalink": "specs/search",
"file_path": "specs/search.md",
"title": "Search Specification",
"resolution_method": "permalink"
}
"""
logger.info(f"API v2 request: resolve_identifier for '{data.identifier}'")
# Try to resolve by external_id first
entity = await entity_repository.get_by_external_id(data.identifier)
resolution_method = "external_id" if entity else "search"
# If not found by external_id, try other resolution methods
if not entity:
entity = await link_resolver.resolve_link(data.identifier)
if entity:
# Determine resolution method
if entity.permalink == data.identifier:
resolution_method = "permalink"
elif entity.title == data.identifier:
resolution_method = "title"
elif entity.file_path == data.identifier:
resolution_method = "path"
else:
resolution_method = "search"
if not entity:
raise HTTPException(status_code=404, detail=f"Entity not found: '{data.identifier}'")
result = EntityResolveResponse(
external_id=entity.external_id,
entity_id=entity.id,
permalink=entity.permalink,
file_path=entity.file_path,
title=entity.title,
resolution_method=resolution_method,
)
logger.info(
f"API v2 response: resolved '{data.identifier}' to external_id={result.external_id} via {resolution_method}"
)
return result
## Read endpoints
@router.get("/entities/{entity_id}", response_model=EntityResponseV2)
async def get_entity_by_id(
project_id: ProjectExternalIdPathDep,
entity_repository: EntityRepositoryV2ExternalDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
) -> EntityResponseV2:
"""Get an entity by its external ID (UUID).
This is the primary entity retrieval method in v2, using stable UUID
identifiers that won't change with file moves.
Args:
entity_id: External ID (UUID string)
Returns:
Complete entity with observations and relations
Raises:
HTTPException: 404 if entity not found
"""
logger.info(f"API v2 request: get_entity_by_id entity_id={entity_id}")
entity = await entity_repository.get_by_external_id(entity_id)
if not entity:
raise HTTPException(
status_code=404, detail=f"Entity with external_id '{entity_id}' not found"
)
result = EntityResponseV2.model_validate(entity)
logger.info(f"API v2 response: external_id={entity_id}, title='{result.title}'")
return result
## Create endpoints
@router.post("/entities", response_model=EntityResponseV2)
async def create_entity(
project_id: ProjectExternalIdPathDep,
data: Entity,
background_tasks: BackgroundTasks,
entity_service: EntityServiceV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
) -> EntityResponseV2:
"""Create a new entity.
Args:
data: Entity data to create
Returns:
Created entity with generated external_id (UUID)
"""
logger.info(
"API v2 request", endpoint="create_entity", entity_type=data.entity_type, title=data.title
)
entity = await entity_service.create_entity(data)
# reindex
await search_service.index_entity(entity, background_tasks=background_tasks)
result = EntityResponseV2.model_validate(entity)
logger.info(
f"API v2 response: endpoint='create_entity' external_id={entity.external_id}, title={result.title}, permalink={result.permalink}, status_code=201"
)
return result
## Update endpoints
@router.put("/entities/{entity_id}", response_model=EntityResponseV2)
async def update_entity_by_id(
data: Entity,
response: Response,
background_tasks: BackgroundTasks,
project_id: ProjectExternalIdPathDep,
entity_service: EntityServiceV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
sync_service: SyncServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
) -> EntityResponseV2:
"""Update an entity by external ID.
If the entity doesn't exist, it will be created (upsert behavior).
Args:
entity_id: External ID (UUID string)
data: Updated entity data
Returns:
Updated entity
"""
logger.info(f"API v2 request: update_entity_by_id entity_id={entity_id}")
# Check if entity exists
existing = await entity_repository.get_by_external_id(entity_id)
created = existing is None
# Perform update or create
entity, _ = await entity_service.create_or_update_entity(data)
response.status_code = 201 if created else 200
# reindex
await search_service.index_entity(entity, background_tasks=background_tasks)
# Schedule relation resolution for new entities
if created:
background_tasks.add_task( # pragma: no cover
resolve_relations_background, sync_service, entity.id, entity.permalink or ""
)
result = EntityResponseV2.model_validate(entity)
logger.info(
f"API v2 response: external_id={entity_id}, created={created}, status_code={response.status_code}"
)
return result
@router.patch("/entities/{entity_id}", response_model=EntityResponseV2)
async def edit_entity_by_id(
data: EditEntityRequest,
background_tasks: BackgroundTasks,
project_id: ProjectExternalIdPathDep,
entity_service: EntityServiceV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
) -> EntityResponseV2:
"""Edit an existing entity by external ID using operations like append, prepend, etc.
Args:
entity_id: External ID (UUID string)
data: Edit operation details
Returns:
Updated entity
Raises:
HTTPException: 404 if entity not found, 400 if edit fails
"""
logger.info(
f"API v2 request: edit_entity_by_id entity_id={entity_id}, operation='{data.operation}'"
)
# Verify entity exists
entity = await entity_repository.get_by_external_id(entity_id)
if not entity: # pragma: no cover
raise HTTPException(
status_code=404, detail=f"Entity with external_id '{entity_id}' not found"
)
try:
# Edit using the entity's permalink or path
identifier = entity.permalink or entity.file_path
updated_entity = await entity_service.edit_entity(
identifier=identifier,
operation=data.operation,
content=data.content,
section=data.section,
find_text=data.find_text,
expected_replacements=data.expected_replacements,
)
# Reindex
await search_service.index_entity(updated_entity, background_tasks=background_tasks)
result = EntityResponseV2.model_validate(updated_entity)
logger.info(
f"API v2 response: external_id={entity_id}, operation='{data.operation}', status_code=200"
)
return result
except Exception as e:
logger.error(f"Error editing entity {entity_id}: {e}")
raise HTTPException(status_code=400, detail=str(e))
## Delete endpoints
@router.delete("/entities/{entity_id}", response_model=DeleteEntitiesResponse)
async def delete_entity_by_id(
background_tasks: BackgroundTasks,
project_id: ProjectExternalIdPathDep,
entity_service: EntityServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
search_service=Depends(lambda: None), # Optional for now
) -> DeleteEntitiesResponse:
"""Delete an entity by external ID.
Args:
entity_id: External ID (UUID string)
Returns:
Deletion status
Note: Returns deleted=False if entity doesn't exist (idempotent)
"""
logger.info(f"API v2 request: delete_entity_by_id entity_id={entity_id}")
entity = await entity_repository.get_by_external_id(entity_id)
if entity is None:
logger.info(f"API v2 response: external_id={entity_id} not found, deleted=False")
return DeleteEntitiesResponse(deleted=False)
# Delete the entity using internal ID
deleted = await entity_service.delete_entity(entity.id)
# Remove from search index if search service available
if search_service:
background_tasks.add_task(search_service.handle_delete, entity) # pragma: no cover
logger.info(f"API v2 response: external_id={entity_id}, deleted={deleted}")
return DeleteEntitiesResponse(deleted=deleted)
## Move endpoint
@router.put("/entities/{entity_id}/move", response_model=EntityResponseV2)
async def move_entity(
data: MoveEntityRequestV2,
background_tasks: BackgroundTasks,
project_id: ProjectExternalIdPathDep,
entity_service: EntityServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
project_config: ProjectConfigV2ExternalDep,
app_config: AppConfigDep,
search_service: SearchServiceV2ExternalDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
) -> EntityResponseV2:
"""Move an entity to a new file location.
V2 API uses external_id (UUID) in the URL path for stable references.
The external_id will remain stable after the move.
Args:
project_id: Project external ID from URL path
entity_id: Entity external ID from URL path (primary identifier)
data: Move request with destination path only
Returns:
Updated entity with new file path
"""
logger.info(
f"API v2 request: move_entity entity_id={entity_id}, destination='{data.destination_path}'"
)
try:
# First, get the entity by external_id to verify it exists
entity = await entity_repository.get_by_external_id(entity_id)
if not entity: # pragma: no cover
raise HTTPException(
status_code=404, detail=f"Entity with external_id '{entity_id}' not found"
)
# Move the entity using its current file path as identifier
moved_entity = await entity_service.move_entity(
identifier=entity.file_path, # Use file path for resolution
destination_path=data.destination_path,
project_config=project_config,
app_config=app_config,
)
# Reindex at new location
reindexed_entity = await entity_service.link_resolver.resolve_link(data.destination_path)
if reindexed_entity:
await search_service.index_entity(reindexed_entity, background_tasks=background_tasks)
result = EntityResponseV2.model_validate(moved_entity)
logger.info(f"API v2 response: moved external_id={entity_id} to '{data.destination_path}'")
return result
except HTTPException: # pragma: no cover
raise # pragma: no cover
except Exception as e:
logger.error(f"Error moving entity: {e}")
raise HTTPException(status_code=400, detail=str(e))
@@ -0,0 +1,130 @@
"""V2 routes for memory:// URI operations.
This router uses external_id UUIDs for stable, API-friendly routing.
V1 uses string-based project names which are less efficient and less stable.
"""
from typing import Annotated, Optional
from fastapi import APIRouter, Query, Path
from loguru import logger
from basic_memory.deps import ContextServiceV2ExternalDep, EntityRepositoryV2ExternalDep
from basic_memory.schemas.base import TimeFrame, parse_timeframe
from basic_memory.schemas.memory import (
GraphContext,
normalize_memory_url,
)
from basic_memory.schemas.search import SearchItemType
from basic_memory.api.routers.utils import to_graph_context
# Note: No prefix here - it's added during registration as /v2/{project_id}/memory
router = APIRouter(tags=["memory"])
@router.get("/memory/recent", response_model=GraphContext)
async def recent(
context_service: ContextServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
type: Annotated[list[SearchItemType] | None, Query()] = None,
depth: int = 1,
timeframe: TimeFrame = "7d",
page: int = 1,
page_size: int = 10,
max_related: int = 10,
) -> GraphContext:
"""Get recent activity context for a project.
Args:
project_id: Project external UUID from URL path
context_service: Context service scoped to project
entity_repository: Entity repository scoped to project
type: Types of items to include (entities, relations, observations)
depth: How many levels of related entities to include
timeframe: Time window for recent activity (e.g., "7d", "1 week")
page: Page number for pagination
page_size: Number of items per page
max_related: Maximum related entities to include per item
Returns:
GraphContext with recent activity and related entities
"""
# return all types by default
types = (
[SearchItemType.ENTITY, SearchItemType.RELATION, SearchItemType.OBSERVATION]
if not type
else type
)
logger.debug(
f"V2 Getting recent context for project {project_id}: `{types}` depth: `{depth}` timeframe: `{timeframe}` page: `{page}` page_size: `{page_size}` max_related: `{max_related}`"
)
# Parse timeframe
since = parse_timeframe(timeframe)
limit = page_size
offset = (page - 1) * page_size
# Build context
context = await context_service.build_context(
types=types, depth=depth, since=since, limit=limit, offset=offset, max_related=max_related
)
recent_context = await to_graph_context(
context, entity_repository=entity_repository, page=page, page_size=page_size
)
logger.debug(f"V2 Recent context: {recent_context.model_dump_json()}")
return recent_context
# get_memory_context needs to be declared last so other paths can match
@router.get("/memory/{uri:path}", response_model=GraphContext)
async def get_memory_context(
context_service: ContextServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
uri: str,
project_id: str = Path(..., description="Project external UUID"),
depth: int = 1,
timeframe: Optional[TimeFrame] = None,
page: int = 1,
page_size: int = 10,
max_related: int = 10,
) -> GraphContext:
"""Get rich context from memory:// URI.
V2 supports both legacy path-based URIs and new ID-based URIs:
- Legacy: memory://path/to/note
- ID-based: memory://id/123 or memory://123
Args:
project_id: Project external UUID from URL path
context_service: Context service scoped to project
entity_repository: Entity repository scoped to project
uri: Memory URI path (e.g., "id/123", "123", or "path/to/note")
depth: How many levels of related entities to include
timeframe: Optional time window for filtering related content
page: Page number for pagination
page_size: Number of items per page
max_related: Maximum related entities to include
Returns:
GraphContext with the entity and its related context
"""
logger.debug(
f"V2 Getting context for project {project_id}, URI: `{uri}` depth: `{depth}` timeframe: `{timeframe}` page: `{page}` page_size: `{page_size}` max_related: `{max_related}`"
)
memory_url = normalize_memory_url(uri)
# Parse timeframe
since = parse_timeframe(timeframe) if timeframe else None
limit = page_size
offset = (page - 1) * page_size
# Build context
context = await context_service.build_context(
memory_url, depth=depth, since=since, limit=limit, offset=offset, max_related=max_related
)
return await to_graph_context(
context, entity_repository=entity_repository, page=page, page_size=page_size
)
@@ -0,0 +1,357 @@
"""V2 Project Router - External ID-based project management operations.
This router provides external_id (UUID) based CRUD operations for projects,
using stable string UUIDs that never change (unlike integer IDs or names).
Key improvements:
- Stable external UUIDs that won't change with renames or database migrations
- Better API ergonomics with consistent string identifiers
- Direct database lookups via unique indexed column
- Consistent with v2 entity operations
"""
import os
from typing import Optional
from fastapi import APIRouter, HTTPException, Body, Query, Path
from loguru import logger
from basic_memory.deps import (
ProjectServiceDep,
ProjectRepositoryDep,
)
from basic_memory.schemas.project_info import (
ProjectItem,
ProjectStatusResponse,
)
from basic_memory.schemas.v2 import ProjectResolveRequest, ProjectResolveResponse
from basic_memory.utils import normalize_project_path, generate_permalink
router = APIRouter(prefix="/projects", tags=["project_management-v2"])
@router.post("/resolve", response_model=ProjectResolveResponse)
async def resolve_project_identifier(
data: ProjectResolveRequest,
project_repository: ProjectRepositoryDep,
) -> ProjectResolveResponse:
"""Resolve a project identifier (name, permalink, or external_id) to project info.
This endpoint provides efficient lookup of projects by various identifiers
without needing to fetch the entire project list. Supports:
- External ID (UUID string) - preferred stable identifier
- Permalink
- Case-insensitive name matching
Args:
data: Request containing the identifier to resolve
Returns:
Project information including the external_id (UUID)
Raises:
HTTPException: 404 if project not found
Example:
POST /v2/projects/resolve
{"identifier": "my-project"}
Returns:
{
"external_id": "550e8400-e29b-41d4-a716-446655440000",
"project_id": 1,
"name": "my-project",
"permalink": "my-project",
"path": "/path/to/project",
"is_active": true,
"is_default": false,
"resolution_method": "name"
}
"""
logger.info(f"API v2 request: resolve_project_identifier for '{data.identifier}'")
# Generate permalink for comparison
identifier_permalink = generate_permalink(data.identifier)
resolution_method = "name"
project = None
# Try external_id first (UUID format)
project = await project_repository.get_by_external_id(data.identifier)
if project:
resolution_method = "external_id"
# If not found by external_id, try by permalink (exact match)
if not project:
project = await project_repository.get_by_permalink(identifier_permalink)
if project:
resolution_method = "permalink"
# If not found by permalink, try case-insensitive name search
if not project:
project = await project_repository.get_by_name_case_insensitive(data.identifier)
if project:
resolution_method = "name" # pragma: no cover
if not project:
raise HTTPException(status_code=404, detail=f"Project not found: '{data.identifier}'")
return ProjectResolveResponse(
external_id=project.external_id,
project_id=project.id,
name=project.name,
permalink=generate_permalink(project.name),
path=normalize_project_path(project.path),
is_active=project.is_active if hasattr(project, "is_active") else True,
is_default=project.is_default or False,
resolution_method=resolution_method,
)
@router.get("/{project_id}", response_model=ProjectItem)
async def get_project_by_id(
project_repository: ProjectRepositoryDep,
project_id: str = Path(..., description="Project external ID (UUID)"),
) -> ProjectItem:
"""Get project by its external ID (UUID).
This is the primary project retrieval method in v2, using stable UUID
identifiers that won't change with project renames.
Args:
project_id: External ID (UUID string)
Returns:
Project information including external_id
Raises:
HTTPException: 404 if project not found
Example:
GET /v2/projects/550e8400-e29b-41d4-a716-446655440000
"""
logger.info(f"API v2 request: get_project_by_id for project_id={project_id}")
project = await project_repository.get_by_external_id(project_id)
if not project:
raise HTTPException(
status_code=404, detail=f"Project with external_id '{project_id}' not found"
)
return ProjectItem(
id=project.id,
external_id=project.external_id,
name=project.name,
path=normalize_project_path(project.path),
is_default=project.is_default or False,
)
@router.patch("/{project_id}", response_model=ProjectStatusResponse)
async def update_project_by_id(
project_service: ProjectServiceDep,
project_repository: ProjectRepositoryDep,
project_id: str = Path(..., description="Project external ID (UUID)"),
path: Optional[str] = Body(None, description="New absolute path for the project"),
is_active: Optional[bool] = Body(None, description="Status of the project (active/inactive)"),
) -> ProjectStatusResponse:
"""Update a project's information by external ID.
Args:
project_id: External ID (UUID string)
path: Optional new absolute path for the project
is_active: Optional status update for the project
Returns:
Response confirming the project was updated
Raises:
HTTPException: 400 if validation fails, 404 if project not found
Example:
PATCH /v2/projects/550e8400-e29b-41d4-a716-446655440000
{"path": "/new/path"}
"""
logger.info(f"API v2 request: update_project_by_id for project_id={project_id}")
try:
# Validate that path is absolute if provided
if path and not os.path.isabs(path):
raise HTTPException(status_code=400, detail="Path must be absolute")
# Get original project info for the response
old_project = await project_repository.get_by_external_id(project_id)
if not old_project:
raise HTTPException(
status_code=404, detail=f"Project with external_id '{project_id}' not found"
)
old_project_info = ProjectItem(
id=old_project.id,
external_id=old_project.external_id,
name=old_project.name,
path=old_project.path,
is_default=old_project.is_default or False,
)
# Update using project name (service layer still uses names internally)
if path:
await project_service.move_project(old_project.name, path)
elif is_active is not None:
await project_service.update_project(old_project.name, is_active=is_active)
# Get updated project info (use the same external_id)
updated_project = await project_repository.get_by_external_id(project_id)
if not updated_project: # pragma: no cover
raise HTTPException(
status_code=404,
detail=f"Project with external_id '{project_id}' not found after update",
)
return ProjectStatusResponse(
message=f"Project '{updated_project.name}' updated successfully",
status="success",
default=old_project.is_default or False,
old_project=old_project_info,
new_project=ProjectItem(
id=updated_project.id,
external_id=updated_project.external_id,
name=updated_project.name,
path=updated_project.path,
is_default=updated_project.is_default or False,
),
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e)) # pragma: no cover
@router.delete("/{project_id}", response_model=ProjectStatusResponse)
async def delete_project_by_id(
project_service: ProjectServiceDep,
project_repository: ProjectRepositoryDep,
project_id: str = Path(..., description="Project external ID (UUID)"),
delete_notes: bool = Query(
False, description="If True, delete project directory from filesystem"
),
) -> ProjectStatusResponse:
"""Delete a project by external ID.
Args:
project_id: External ID (UUID string)
delete_notes: If True, delete the project directory from the filesystem
Returns:
Response confirming the project was deleted
Raises:
HTTPException: 400 if trying to delete default project, 404 if not found
Example:
DELETE /v2/projects/550e8400-e29b-41d4-a716-446655440000?delete_notes=false
"""
logger.info(
f"API v2 request: delete_project_by_id for project_id={project_id}, delete_notes={delete_notes}"
)
try:
old_project = await project_repository.get_by_external_id(project_id)
if not old_project:
raise HTTPException(
status_code=404, detail=f"Project with external_id '{project_id}' not found"
)
# Check if trying to delete the default project
# Use is_default from database, not ConfigManager (which doesn't work in cloud mode)
if old_project.is_default:
available_projects = await project_service.list_projects()
other_projects = [p.name for p in available_projects if p.external_id != project_id]
detail = f"Cannot delete default project '{old_project.name}'. "
if other_projects:
detail += ( # pragma: no cover
f"Set another project as default first. Available: {', '.join(other_projects)}"
)
else:
detail += "This is the only project in your configuration." # pragma: no cover
raise HTTPException(status_code=400, detail=detail)
# Delete using project name (service layer still uses names internally)
await project_service.remove_project(old_project.name, delete_notes=delete_notes)
return ProjectStatusResponse(
message=f"Project '{old_project.name}' removed successfully",
status="success",
default=False,
old_project=ProjectItem(
id=old_project.id,
external_id=old_project.external_id,
name=old_project.name,
path=old_project.path,
is_default=old_project.is_default or False,
),
new_project=None,
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e)) # pragma: no cover
@router.put("/{project_id}/default", response_model=ProjectStatusResponse)
async def set_default_project_by_id(
project_service: ProjectServiceDep,
project_repository: ProjectRepositoryDep,
project_id: str = Path(..., description="Project external ID (UUID)"),
) -> ProjectStatusResponse:
"""Set a project as the default project by external ID.
Args:
project_id: External ID (UUID string) to set as default
Returns:
Response confirming the project was set as default
Raises:
HTTPException: 404 if project not found
Example:
PUT /v2/projects/550e8400-e29b-41d4-a716-446655440000/default
"""
logger.info(f"API v2 request: set_default_project_by_id for project_id={project_id}")
try:
# Get the old default project from database
default_project = await project_repository.get_default_project()
if not default_project:
raise HTTPException( # pragma: no cover
status_code=404, detail="No default project is currently set"
)
# Get the new default project by external_id
new_default_project = await project_repository.get_by_external_id(project_id)
if not new_default_project:
raise HTTPException(
status_code=404, detail=f"Project with external_id '{project_id}' not found"
)
# Set as default using project name (service layer still uses names internally)
await project_service.set_default_project(new_default_project.name)
return ProjectStatusResponse(
message=f"Project '{new_default_project.name}' set as default successfully",
status="success",
default=True,
old_project=ProjectItem(
id=default_project.id,
external_id=default_project.external_id,
name=default_project.name,
path=default_project.path,
is_default=False,
),
new_project=ProjectItem(
id=new_default_project.id,
external_id=new_default_project.external_id,
name=new_default_project.name,
path=new_default_project.path,
is_default=True,
),
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e)) # pragma: no cover
@@ -0,0 +1,269 @@
"""V2 Prompt Router - ID-based prompt generation operations.
This router uses v2 dependencies for consistent project handling with external_id UUIDs.
Prompt endpoints are action-based (not resource-based), so they don't
have entity IDs in URLs - they generate formatted prompts from queries.
"""
from datetime import datetime, timezone
from fastapi import APIRouter, HTTPException, status, Path
from loguru import logger
from basic_memory.api.routers.utils import to_graph_context, to_search_results
from basic_memory.api.template_loader import template_loader
from basic_memory.schemas.base import parse_timeframe
from basic_memory.deps import (
ContextServiceV2ExternalDep,
EntityRepositoryV2ExternalDep,
SearchServiceV2ExternalDep,
EntityServiceV2ExternalDep,
)
from basic_memory.schemas.prompt import (
ContinueConversationRequest,
SearchPromptRequest,
PromptResponse,
PromptMetadata,
)
from basic_memory.schemas.search import SearchItemType, SearchQuery
router = APIRouter(prefix="/prompt", tags=["prompt-v2"])
@router.post("/continue-conversation", response_model=PromptResponse)
async def continue_conversation(
search_service: SearchServiceV2ExternalDep,
entity_service: EntityServiceV2ExternalDep,
context_service: ContextServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
request: ContinueConversationRequest,
project_id: str = Path(..., description="Project external UUID"),
) -> PromptResponse:
"""Generate a prompt for continuing a conversation.
This endpoint takes a topic and/or timeframe and generates a prompt with
relevant context from the knowledge base.
Args:
project_id: Project external UUID from URL path
request: The request parameters
Returns:
Formatted continuation prompt with context
"""
logger.info(
f"V2 Generating continue conversation prompt for project {project_id}, "
f"topic: {request.topic}, timeframe: {request.timeframe}"
)
since = parse_timeframe(request.timeframe) if request.timeframe else None
# Initialize search results
search_results = []
# Get data needed for template
if request.topic:
query = SearchQuery(text=request.topic, after_date=request.timeframe)
results = await search_service.search(query, limit=request.search_items_limit)
search_results = await to_search_results(entity_service, results)
# Build context from results
all_hierarchical_results = []
for result in search_results:
if hasattr(result, "permalink") and result.permalink:
# Get hierarchical context using the new dataclass-based approach
context_result = await context_service.build_context(
result.permalink,
depth=request.depth,
since=since,
max_related=request.related_items_limit,
include_observations=True, # Include observations for entities
)
# Process results into the schema format
graph_context = await to_graph_context(
context_result, entity_repository=entity_repository
)
# Add results to our collection (limit to top results for each permalink)
if graph_context.results:
all_hierarchical_results.extend(graph_context.results[:3])
# Limit to a reasonable number of total results
all_hierarchical_results = all_hierarchical_results[:10]
template_context = {
"topic": request.topic,
"timeframe": request.timeframe,
"hierarchical_results": all_hierarchical_results,
"has_results": len(all_hierarchical_results) > 0,
}
else:
# If no topic, get recent activity
context_result = await context_service.build_context(
types=[SearchItemType.ENTITY],
depth=request.depth,
since=since,
max_related=request.related_items_limit,
include_observations=True,
)
recent_context = await to_graph_context(context_result, entity_repository=entity_repository)
hierarchical_results = recent_context.results[:5] # Limit to top 5 recent items
template_context = {
"topic": f"Recent Activity from ({request.timeframe})",
"timeframe": request.timeframe,
"hierarchical_results": hierarchical_results,
"has_results": len(hierarchical_results) > 0,
}
try:
# Render template
rendered_prompt = await template_loader.render(
"prompts/continue_conversation.hbs", template_context
)
# Calculate metadata
# Count items of different types
observation_count = 0
relation_count = 0
entity_count = 0
# Get the hierarchical results from the template context
hierarchical_results_for_count = template_context.get("hierarchical_results", [])
# For topic-based search
if request.topic:
for item in hierarchical_results_for_count:
if hasattr(item, "observations"):
observation_count += len(item.observations) if item.observations else 0
if hasattr(item, "related_results"):
for related in item.related_results or []:
if hasattr(related, "type"):
if related.type == "relation":
relation_count += 1
elif related.type == "entity": # pragma: no cover
entity_count += 1 # pragma: no cover
# For recent activity
else:
for item in hierarchical_results_for_count:
if hasattr(item, "observations"):
observation_count += len(item.observations) if item.observations else 0
if hasattr(item, "related_results"):
for related in item.related_results or []:
if hasattr(related, "type"):
if related.type == "relation":
relation_count += 1
elif related.type == "entity": # pragma: no cover
entity_count += 1 # pragma: no cover
# Build metadata
metadata = {
"query": request.topic,
"timeframe": request.timeframe,
"search_count": len(search_results)
if request.topic
else 0, # Original search results count
"context_count": len(hierarchical_results_for_count),
"observation_count": observation_count,
"relation_count": relation_count,
"total_items": (
len(hierarchical_results_for_count)
+ observation_count
+ relation_count
+ entity_count
),
"search_limit": request.search_items_limit,
"context_depth": request.depth,
"related_limit": request.related_items_limit,
"generated_at": datetime.now(timezone.utc).isoformat(),
}
prompt_metadata = PromptMetadata(**metadata)
return PromptResponse(
prompt=rendered_prompt, context=template_context, metadata=prompt_metadata
)
except Exception as e:
logger.error(f"Error rendering continue conversation template: {e}")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Error rendering prompt template: {str(e)}",
)
@router.post("/search", response_model=PromptResponse)
async def search_prompt(
search_service: SearchServiceV2ExternalDep,
entity_service: EntityServiceV2ExternalDep,
request: SearchPromptRequest,
project_id: str = Path(..., description="Project external UUID"),
page: int = 1,
page_size: int = 10,
) -> PromptResponse:
"""Generate a prompt for search results.
This endpoint takes a search query and formats the results into a helpful
prompt with context and suggestions.
Args:
project_id: Project external UUID from URL path
request: The search parameters
page: The page number for pagination
page_size: The number of results per page, defaults to 10
Returns:
Formatted search results prompt with context
"""
logger.info(
f"V2 Generating search prompt for project {project_id}, "
f"query: {request.query}, timeframe: {request.timeframe}"
)
limit = page_size
offset = (page - 1) * page_size
query = SearchQuery(text=request.query, after_date=request.timeframe)
results = await search_service.search(query, limit=limit, offset=offset)
search_results = await to_search_results(entity_service, results)
template_context = {
"query": request.query,
"timeframe": request.timeframe,
"results": search_results,
"has_results": len(search_results) > 0,
"result_count": len(search_results),
}
try:
# Render template
rendered_prompt = await template_loader.render("prompts/search.hbs", template_context)
# Build metadata
metadata = {
"query": request.query,
"timeframe": request.timeframe,
"search_count": len(search_results),
"context_count": len(search_results),
"observation_count": 0, # Search results don't include observations
"relation_count": 0, # Search results don't include relations
"total_items": len(search_results),
"search_limit": limit,
"context_depth": 0, # No context depth for basic search
"related_limit": 0, # No related items for basic search
"generated_at": datetime.now(timezone.utc).isoformat(),
}
prompt_metadata = PromptMetadata(**metadata)
return PromptResponse(
prompt=rendered_prompt, context=template_context, metadata=prompt_metadata
)
except Exception as e:
logger.error(f"Error rendering search template: {e}")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Error rendering prompt template: {str(e)}",
)
@@ -0,0 +1,289 @@
"""V2 Resource Router - ID-based resource content operations.
This router uses entity external_ids (UUIDs) for all operations, with file paths
in request bodies when needed. This is consistent with v2's external_id-first design.
Key differences from v1:
- Uses UUID external_ids in URL paths instead of integer IDs or file paths
- File paths are in request bodies for create/update operations
- More RESTful: POST for create, PUT for update, GET for read
"""
import uuid
from pathlib import Path as PathLib
from fastapi import APIRouter, HTTPException, Response, Path
from loguru import logger
from basic_memory.deps import (
ProjectConfigV2ExternalDep,
FileServiceV2ExternalDep,
EntityRepositoryV2ExternalDep,
SearchServiceV2ExternalDep,
)
from basic_memory.models.knowledge import Entity as EntityModel
from basic_memory.schemas.v2.resource import (
CreateResourceRequest,
UpdateResourceRequest,
ResourceResponse,
)
from basic_memory.utils import validate_project_path
router = APIRouter(prefix="/resource", tags=["resources-v2"])
@router.get("/{entity_id}")
async def get_resource_content(
config: ProjectConfigV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
file_service: FileServiceV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
entity_id: str = Path(..., description="Entity external UUID"),
) -> Response:
"""Get raw resource content by entity external_id.
Args:
project_id: Project external UUID from URL path
entity_id: Entity external UUID
config: Project configuration
entity_repository: Entity repository for fetching entity data
file_service: File service for reading file content
Returns:
Response with entity content
Raises:
HTTPException: 404 if entity or file not found
"""
logger.debug(f"V2 Getting content for project {project_id}, entity_id: {entity_id}")
# Get entity by external_id
entity = await entity_repository.get_by_external_id(entity_id)
if not entity:
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
# Validate entity file path to prevent path traversal
project_path = PathLib(config.home)
if not validate_project_path(entity.file_path, project_path):
logger.error( # pragma: no cover
f"Invalid file path in entity {entity.id}: {entity.file_path}"
)
raise HTTPException( # pragma: no cover
status_code=500,
detail="Entity contains invalid file path",
)
# Check file exists via file_service (for cloud compatibility)
if not await file_service.exists(entity.file_path):
raise HTTPException( # pragma: no cover
status_code=404,
detail=f"File not found: {entity.file_path}",
)
# Read content via file_service as bytes (works with both local and S3)
content = await file_service.read_file_bytes(entity.file_path)
content_type = file_service.content_type(entity.file_path)
return Response(content=content, media_type=content_type)
@router.post("", response_model=ResourceResponse)
async def create_resource(
data: CreateResourceRequest,
config: ProjectConfigV2ExternalDep,
file_service: FileServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
) -> ResourceResponse:
"""Create a new resource file.
Args:
project_id: Project external UUID from URL path
data: Create resource request with file_path and content
config: Project configuration
file_service: File service for writing files
entity_repository: Entity repository for creating entities
search_service: Search service for indexing
Returns:
ResourceResponse with file information including entity_id and external_id
Raises:
HTTPException: 400 for invalid file paths, 409 if file already exists
"""
try:
# Validate path to prevent path traversal attacks
project_path = PathLib(config.home)
if not validate_project_path(data.file_path, project_path):
logger.warning(
f"Invalid file path attempted: {data.file_path} in project {config.name}"
)
raise HTTPException(
status_code=400,
detail=f"Invalid file path: {data.file_path}. "
"Path must be relative and stay within project boundaries.",
)
# Check if entity already exists
existing_entity = await entity_repository.get_by_file_path(data.file_path)
if existing_entity:
raise HTTPException(
status_code=409,
detail=f"Resource already exists at {data.file_path} with entity_id {existing_entity.external_id}. "
f"Use PUT /resource/{existing_entity.external_id} to update it.",
)
# Cloud compatibility: avoid assuming a local filesystem path.
# Delegate directory creation + writes to FileService (local or S3).
await file_service.ensure_directory(PathLib(data.file_path).parent)
checksum = await file_service.write_file(data.file_path, data.content)
# Get file info
file_metadata = await file_service.get_file_metadata(data.file_path)
# Determine file details
file_name = PathLib(data.file_path).name
content_type = file_service.content_type(data.file_path)
entity_type = "canvas" if data.file_path.endswith(".canvas") else "file"
# Create a new entity model
# Explicitly set external_id to ensure NOT NULL constraint is satisfied (fixes #512)
entity = EntityModel(
external_id=str(uuid.uuid4()),
title=file_name,
entity_type=entity_type,
content_type=content_type,
file_path=data.file_path,
checksum=checksum,
created_at=file_metadata.created_at,
updated_at=file_metadata.modified_at,
)
entity = await entity_repository.add(entity)
# Index the file for search
await search_service.index_entity(entity) # pyright: ignore
# Return success response
return ResourceResponse(
entity_id=entity.id,
external_id=entity.external_id,
file_path=data.file_path,
checksum=checksum,
size=file_metadata.size,
created_at=file_metadata.created_at.timestamp(),
modified_at=file_metadata.modified_at.timestamp(),
)
except HTTPException:
# Re-raise HTTP exceptions without wrapping
raise
except Exception as e: # pragma: no cover
logger.error(f"Error creating resource {data.file_path}: {e}")
raise HTTPException(status_code=500, detail=f"Failed to create resource: {str(e)}")
@router.put("/{entity_id}", response_model=ResourceResponse)
async def update_resource(
data: UpdateResourceRequest,
config: ProjectConfigV2ExternalDep,
file_service: FileServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
entity_id: str = Path(..., description="Entity external UUID"),
) -> ResourceResponse:
"""Update an existing resource by entity external_id.
Can update content and optionally move the file to a new path.
Args:
project_id: Project external UUID from URL path
entity_id: Entity external UUID of the resource to update
data: Update resource request with content and optional new file_path
config: Project configuration
file_service: File service for writing files
entity_repository: Entity repository for updating entities
search_service: Search service for indexing
Returns:
ResourceResponse with updated file information
Raises:
HTTPException: 404 if entity not found, 400 for invalid paths
"""
try:
# Get existing entity by external_id
entity = await entity_repository.get_by_external_id(entity_id)
if not entity:
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
# Determine target file path
target_file_path = data.file_path if data.file_path else entity.file_path
# Validate path to prevent path traversal attacks
project_path = PathLib(config.home)
if not validate_project_path(target_file_path, project_path):
logger.warning(
f"Invalid file path attempted: {target_file_path} in project {config.name}"
)
raise HTTPException(
status_code=400,
detail=f"Invalid file path: {target_file_path}. "
"Path must be relative and stay within project boundaries.",
)
# If moving file, handle the move
if data.file_path and data.file_path != entity.file_path:
# Ensure new parent directory exists (no-op for S3)
await file_service.ensure_directory(PathLib(target_file_path).parent)
# If old file exists, remove it via file_service (for cloud compatibility)
if await file_service.exists(entity.file_path):
await file_service.delete_file(entity.file_path)
else:
# Ensure directory exists for in-place update
await file_service.ensure_directory(PathLib(target_file_path).parent)
# Write content to target file
checksum = await file_service.write_file(target_file_path, data.content)
# Get file info
file_metadata = await file_service.get_file_metadata(target_file_path)
# Determine file details
file_name = PathLib(target_file_path).name
content_type = file_service.content_type(target_file_path)
entity_type = "canvas" if target_file_path.endswith(".canvas") else "file"
# Update entity using internal ID
updated_entity = await entity_repository.update(
entity.id,
{
"title": file_name,
"entity_type": entity_type,
"content_type": content_type,
"file_path": target_file_path,
"checksum": checksum,
"updated_at": file_metadata.modified_at,
},
)
# Index the updated file for search
await search_service.index_entity(updated_entity) # pyright: ignore
# Return success response
return ResourceResponse(
entity_id=entity.id,
external_id=entity.external_id,
file_path=target_file_path,
checksum=checksum,
size=file_metadata.size,
created_at=file_metadata.created_at.timestamp(),
modified_at=file_metadata.modified_at.timestamp(),
)
except HTTPException:
# Re-raise HTTP exceptions without wrapping
raise
except Exception as e: # pragma: no cover
logger.error(f"Error updating resource {entity_id}: {e}")
raise HTTPException(status_code=500, detail=f"Failed to update resource: {str(e)}")
@@ -0,0 +1,73 @@
"""V2 router for search operations.
This router uses external_id UUIDs for stable, API-friendly routing.
V1 uses string-based project names which are less efficient and less stable.
"""
from fastapi import APIRouter, BackgroundTasks, Path
from basic_memory.api.routers.utils import to_search_results
from basic_memory.schemas.search import SearchQuery, SearchResponse
from basic_memory.deps import SearchServiceV2ExternalDep, EntityServiceV2ExternalDep
# Note: No prefix here - it's added during registration as /v2/{project_id}/search
router = APIRouter(tags=["search"])
@router.post("/search/", response_model=SearchResponse)
async def search(
query: SearchQuery,
search_service: SearchServiceV2ExternalDep,
entity_service: EntityServiceV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
page: int = 1,
page_size: int = 10,
):
"""Search across all knowledge and documents in a project.
V2 uses external_id UUIDs for stable API references.
Args:
project_id: Project external UUID from URL path
query: Search query parameters (text, filters, etc.)
search_service: Search service scoped to project
entity_service: Entity service scoped to project
page: Page number for pagination
page_size: Number of results per page
Returns:
SearchResponse with paginated search results
"""
limit = page_size
offset = (page - 1) * page_size
results = await search_service.search(query, limit=limit, offset=offset)
search_results = await to_search_results(entity_service, results)
return SearchResponse(
results=search_results,
current_page=page,
page_size=page_size,
)
@router.post("/search/reindex")
async def reindex(
background_tasks: BackgroundTasks,
search_service: SearchServiceV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
):
"""Recreate and populate the search index for a project.
This is a background operation that rebuilds the search index
from scratch. Useful after bulk updates or if the index becomes
corrupted.
Args:
project_id: Project external UUID from URL path
background_tasks: FastAPI background tasks handler
search_service: Search service scoped to project
Returns:
Status message indicating reindex has been initiated
"""
await search_service.reindex_all(background_tasks=background_tasks)
return {"status": "ok", "message": "Reindex initiated"}
+29 -7
View File
@@ -1,8 +1,15 @@
from typing import Optional
# This prevents DEBUG logs from appearing on stdout during module-level
# initialization (e.g., template_loader.TemplateLoader() logs at DEBUG level).
from loguru import logger
import typer
logger.remove()
from basic_memory.config import ConfigManager
from typing import Optional # noqa: E402
import typer # noqa: E402
from basic_memory.cli.container import CliContainer, set_container # noqa: E402
from basic_memory.config import init_cli_logging # noqa: E402
def version_callback(value: bool) -> None:
@@ -31,12 +38,27 @@ def app_callback(
) -> None:
"""Basic Memory - Local-first personal knowledge management."""
# Run initialization for every command unless --version was specified
if not version and ctx.invoked_subcommand is not None:
# Initialize logging for CLI (file only, no stdout)
init_cli_logging()
# --- Composition Root ---
# Create container and read config (single point of config access)
container = CliContainer.create()
set_container(container)
# Run initialization for commands that don't use the API
# Skip for 'mcp' command - it has its own lifespan that handles initialization
# Skip for API-using commands (status, sync, etc.) - they handle initialization via deps.py
# Skip for 'reset' command - it manages its own database lifecycle
skip_init_commands = {"mcp", "status", "sync", "project", "tool", "reset"}
if (
not version
and ctx.invoked_subcommand is not None
and ctx.invoked_subcommand not in skip_init_commands
):
from basic_memory.services.initialization import ensure_initialization
app_config = ConfigManager().config
ensure_initialization(app_config)
ensure_initialization(container.config)
## import
+32 -9
View File
@@ -7,6 +7,9 @@ import os
import secrets
import time
import webbrowser
from contextlib import asynccontextmanager
from collections.abc import AsyncIterator, Callable
from typing import AsyncContextManager
import httpx
from rich.console import Console
@@ -19,7 +22,12 @@ console = Console()
class CLIAuth:
"""Handles WorkOS OAuth Device Authorization for CLI tools."""
def __init__(self, client_id: str, authkit_domain: str):
def __init__(
self,
client_id: str,
authkit_domain: str,
http_client_factory: Callable[[], AsyncContextManager[httpx.AsyncClient]] | None = None,
):
self.client_id = client_id
self.authkit_domain = authkit_domain
app_config = ConfigManager().config
@@ -28,6 +36,21 @@ class CLIAuth:
# PKCE parameters
self.code_verifier = None
self.code_challenge = None
self._http_client_factory = http_client_factory
@asynccontextmanager
async def _get_http_client(self) -> AsyncIterator[httpx.AsyncClient]:
"""Create an AsyncClient, optionally via injected factory.
Why: enables reliable tests without monkeypatching httpx internals while
still using real httpx request/response objects.
"""
if self._http_client_factory:
async with self._http_client_factory() as client:
yield client
else:
async with httpx.AsyncClient() as client:
yield client
def generate_pkce_pair(self) -> tuple[str, str]:
"""Generate PKCE code verifier and challenge."""
@@ -57,7 +80,7 @@ class CLIAuth:
}
try:
async with httpx.AsyncClient() as client:
async with self._get_http_client() as client:
response = await client.post(device_auth_url, data=data)
if response.status_code == 200:
@@ -77,7 +100,7 @@ class CLIAuth:
verification_uri = device_response["verification_uri"]
verification_uri_complete = device_response.get("verification_uri_complete")
console.print("\n[bold blue]🔐 Authentication Required[/bold blue]")
console.print("\n[bold blue]Authentication Required[/bold blue]")
console.print("\nTo authenticate, please visit:")
console.print(f"[bold cyan]{verification_uri}[/bold cyan]")
console.print(f"\nAnd enter this code: [bold yellow]{user_code}[/bold yellow]")
@@ -111,7 +134,7 @@ class CLIAuth:
for _attempt in range(max_attempts):
try:
async with httpx.AsyncClient() as client:
async with self._get_http_client() as client:
response = await client.post(token_url, data=data)
if response.status_code == 200:
@@ -171,7 +194,7 @@ class CLIAuth:
# Secure the token file
os.chmod(self.token_file, 0o600)
console.print(f"[green]Tokens saved to {self.token_file}[/green]")
console.print(f"[green]Tokens saved to {self.token_file}[/green]")
def load_tokens(self) -> dict | None:
"""Load tokens from .bm-auth.json file."""
@@ -201,7 +224,7 @@ class CLIAuth:
}
try:
async with httpx.AsyncClient() as client:
async with self._get_http_client() as client:
response = await client.post(token_url, data=data)
if response.status_code == 200:
@@ -233,7 +256,7 @@ class CLIAuth:
if new_tokens:
# Save new tokens (may include rotated refresh token)
self.save_tokens(new_tokens)
console.print("[green]Token refreshed successfully[/green]")
console.print("[green]Token refreshed successfully[/green]")
return new_tokens["access_token"]
else:
console.print("[yellow]Token refresh failed. Please run 'login' again.[/yellow]")
@@ -265,13 +288,13 @@ class CLIAuth:
# Step 4: Save tokens
self.save_tokens(tokens)
console.print("\n[green]Successfully authenticated with Basic Memory Cloud![/green]")
console.print("\n[green]Successfully authenticated with Basic Memory Cloud![/green]")
return True
def logout(self) -> None:
"""Remove stored authentication tokens."""
if self.token_file.exists():
self.token_file.unlink()
console.print("[green]Logged out successfully[/green]")
console.print("[green]Logged out successfully[/green]")
else:
console.print("[yellow]No stored authentication found[/yellow]")
+3 -3
View File
@@ -1,11 +1,10 @@
"""CLI commands for basic-memory."""
from . import status, sync, db, import_memory_json, mcp, import_claude_conversations
from . import import_claude_projects, import_chatgpt, tool, project
from . import status, db, import_memory_json, mcp, import_claude_conversations
from . import import_claude_projects, import_chatgpt, tool, project, format
__all__ = [
"status",
"sync",
"db",
"import_memory_json",
"mcp",
@@ -14,4 +13,5 @@ __all__ = [
"import_chatgpt",
"tool",
"project",
"format",
]
@@ -1,6 +1,16 @@
"""Cloud commands package."""
from basic_memory.cli.app import cloud_app
# Import all commands to register them with typer
from basic_memory.cli.commands.cloud.core_commands import * # noqa: F401,F403
from basic_memory.cli.commands.cloud.api_client import get_authenticated_headers, get_cloud_config # noqa: F401
from basic_memory.cli.commands.cloud.upload_command import * # noqa: F401,F403
# Register snapshot sub-command group
from basic_memory.cli.commands.cloud.snapshot import snapshot_app
cloud_app.add_typer(snapshot_app, name="snapshot")
# Register restore command (directly on cloud_app via decorator)
from basic_memory.cli.commands.cloud.restore import restore # noqa: F401, E402
@@ -1,6 +1,9 @@
"""Cloud API client utilities."""
from collections.abc import AsyncIterator
from typing import Optional
from contextlib import asynccontextmanager
from typing import AsyncContextManager, Callable
import httpx
import typer
@@ -11,6 +14,8 @@ from basic_memory.config import ConfigManager
console = Console()
HttpClientFactory = Callable[[], AsyncContextManager[httpx.AsyncClient]]
class CloudAPIError(Exception):
"""Exception raised for cloud API errors."""
@@ -38,14 +43,14 @@ def get_cloud_config() -> tuple[str, str, str]:
return config.cloud_client_id, config.cloud_domain, config.cloud_host
async def get_authenticated_headers() -> dict[str, str]:
async def get_authenticated_headers(auth: CLIAuth | None = None) -> dict[str, str]:
"""
Get authentication headers with JWT token.
handles jwt refresh if needed.
"""
client_id, domain, _ = get_cloud_config()
auth = CLIAuth(client_id=client_id, authkit_domain=domain)
token = await auth.get_valid_token()
auth_obj = auth or CLIAuth(client_id=client_id, authkit_domain=domain)
token = await auth_obj.get_valid_token()
if not token:
console.print("[red]Not authenticated. Please run 'basic-memory cloud login' first.[/red]")
raise typer.Exit(1)
@@ -53,21 +58,31 @@ async def get_authenticated_headers() -> dict[str, str]:
return {"Authorization": f"Bearer {token}"}
@asynccontextmanager
async def _default_http_client(timeout: float) -> AsyncIterator[httpx.AsyncClient]:
async with httpx.AsyncClient(timeout=timeout) as client:
yield client
async def make_api_request(
method: str,
url: str,
headers: Optional[dict] = None,
json_data: Optional[dict] = None,
timeout: float = 30.0,
*,
auth: CLIAuth | None = None,
http_client_factory: HttpClientFactory | None = None,
) -> httpx.Response:
"""Make an API request to the cloud service."""
headers = headers or {}
auth_headers = await get_authenticated_headers()
auth_headers = await get_authenticated_headers(auth=auth)
headers.update(auth_headers)
# Add debug headers to help with compression issues
headers.setdefault("Accept-Encoding", "identity") # Disable compression for debugging
async with httpx.AsyncClient(timeout=timeout) as client:
client_factory = http_client_factory or (lambda: _default_http_client(timeout))
async with client_factory() as client:
try:
response = await client.request(method=method, url=url, headers=headers, json=json_data)
response.raise_for_status()
@@ -1,33 +1,11 @@
"""Cloud bisync commands for Basic Memory CLI."""
"""Cloud bisync utility functions for Basic Memory CLI."""
import asyncio
import subprocess
import time
from datetime import datetime
from pathlib import Path
from typing import Optional
import typer
from rich.console import Console
from rich.table import Table
from basic_memory.cli.commands.cloud.api_client import CloudAPIError, make_api_request
from basic_memory.cli.commands.cloud.cloud_utils import (
create_cloud_project,
fetch_cloud_projects,
)
from basic_memory.cli.commands.cloud.rclone_config import (
add_tenant_to_rclone_config,
)
from basic_memory.cli.commands.cloud.rclone_installer import RcloneInstallError, install_rclone
from basic_memory.cli.commands.cloud.api_client import make_api_request
from basic_memory.config import ConfigManager
from basic_memory.ignore_utils import get_bmignore_path, create_default_bmignore
from basic_memory.schemas.cloud import (
TenantMountInfo,
MountCredentials,
)
console = Console()
from basic_memory.ignore_utils import create_default_bmignore, get_bmignore_path
from basic_memory.schemas.cloud import MountCredentials, TenantMountInfo
class BisyncError(Exception):
@@ -36,52 +14,6 @@ class BisyncError(Exception):
pass
class RcloneBisyncProfile:
"""Bisync profile with safety settings."""
def __init__(
self,
name: str,
conflict_resolve: str,
max_delete: int,
check_access: bool,
description: str,
extra_args: Optional[list[str]] = None,
):
self.name = name
self.conflict_resolve = conflict_resolve
self.max_delete = max_delete
self.check_access = check_access
self.description = description
self.extra_args = extra_args or []
# Bisync profiles based on SPEC-9 Phase 2.1
BISYNC_PROFILES = {
"safe": RcloneBisyncProfile(
name="safe",
conflict_resolve="none",
max_delete=10,
check_access=False,
description="Safe mode with conflict preservation (keeps both versions)",
),
"balanced": RcloneBisyncProfile(
name="balanced",
conflict_resolve="newer",
max_delete=25,
check_access=False,
description="Balanced mode - auto-resolve to newer file (recommended)",
),
"fast": RcloneBisyncProfile(
name="fast",
conflict_resolve="newer",
max_delete=50,
check_access=False,
description="Fast mode for rapid iteration (skip verification)",
),
}
async def get_mount_info() -> TenantMountInfo:
"""Get current tenant information from cloud API."""
try:
@@ -110,75 +42,6 @@ async def generate_mount_credentials(tenant_id: str) -> MountCredentials:
raise BisyncError(f"Failed to generate credentials: {e}") from e
def scan_local_directories(sync_dir: Path) -> list[str]:
"""Scan local sync directory for project folders.
Args:
sync_dir: Path to bisync directory
Returns:
List of directory names (project names)
"""
if not sync_dir.exists():
return []
directories = []
for item in sync_dir.iterdir():
if item.is_dir() and not item.name.startswith("."):
directories.append(item.name)
return directories
def get_bisync_state_path(tenant_id: str) -> Path:
"""Get path to bisync state directory."""
return Path.home() / ".basic-memory" / "bisync-state" / tenant_id
def get_bisync_directory() -> Path:
"""Get bisync directory from config.
Returns:
Path to bisync directory (default: ~/basic-memory-cloud-sync)
"""
config_manager = ConfigManager()
config = config_manager.config
sync_dir = config.bisync_config.get("sync_dir", str(Path.home() / "basic-memory-cloud-sync"))
return Path(sync_dir).expanduser().resolve()
def validate_bisync_directory(bisync_dir: Path) -> None:
"""Validate bisync directory doesn't conflict with mount.
Raises:
BisyncError: If bisync directory conflicts with mount directory
"""
# Get fixed mount directory
mount_dir = (Path.home() / "basic-memory-cloud").resolve()
# Check if bisync dir is the same as mount dir
if bisync_dir == mount_dir:
raise BisyncError(
f"Cannot use {bisync_dir} for bisync - it's the mount directory!\n"
f"Mount and bisync must use different directories.\n\n"
f"Options:\n"
f" 1. Use default: ~/basic-memory-cloud-sync/\n"
f" 2. Specify different directory: --dir ~/my-sync-folder"
)
# Check if mount is active at this location
result = subprocess.run(["mount"], capture_output=True, text=True)
if str(bisync_dir) in result.stdout and "rclone" in result.stdout:
raise BisyncError(
f"{bisync_dir} is currently mounted via 'bm cloud mount'\n"
f"Cannot use mounted directory for bisync.\n\n"
f"Either:\n"
f" 1. Unmount first: bm cloud unmount\n"
f" 2. Use different directory for bisync"
)
def convert_bmignore_to_rclone_filters() -> Path:
"""Convert .bmignore patterns to rclone filter format.
@@ -245,521 +108,3 @@ def get_bisync_filter_path() -> Path:
Path to rclone filter file
"""
return convert_bmignore_to_rclone_filters()
def bisync_state_exists(tenant_id: str) -> bool:
"""Check if bisync state exists (has been initialized)."""
state_path = get_bisync_state_path(tenant_id)
return state_path.exists() and any(state_path.iterdir())
def build_bisync_command(
tenant_id: str,
bucket_name: str,
local_path: Path,
profile: RcloneBisyncProfile,
dry_run: bool = False,
resync: bool = False,
verbose: bool = False,
) -> list[str]:
"""Build rclone bisync command with profile settings."""
# Sync with the entire bucket root (all projects)
rclone_remote = f"basic-memory-{tenant_id}:{bucket_name}"
filter_path = get_bisync_filter_path()
state_path = get_bisync_state_path(tenant_id)
# Ensure state directory exists
state_path.mkdir(parents=True, exist_ok=True)
cmd = [
"rclone",
"bisync",
str(local_path),
rclone_remote,
"--create-empty-src-dirs",
"--resilient",
f"--conflict-resolve={profile.conflict_resolve}",
f"--max-delete={profile.max_delete}",
"--filters-file",
str(filter_path),
"--workdir",
str(state_path),
]
# Add verbosity flags
if verbose:
cmd.append("--verbose") # Full details with file-by-file output
else:
# Show progress bar during transfers
cmd.append("--progress")
if profile.check_access:
cmd.append("--check-access")
if dry_run:
cmd.append("--dry-run")
if resync:
cmd.append("--resync")
cmd.extend(profile.extra_args)
return cmd
def setup_cloud_bisync(sync_dir: Optional[str] = None) -> None:
"""Set up cloud bisync with rclone installation and configuration.
Args:
sync_dir: Optional custom sync directory path. If not provided, uses config default.
"""
console.print("[bold blue]Basic Memory Cloud Bisync Setup[/bold blue]")
console.print("Setting up bidirectional sync to your cloud tenant...\n")
try:
# Step 1: Install rclone
console.print("[blue]Step 1: Installing rclone...[/blue]")
install_rclone()
# Step 2: Get mount info (for tenant_id, bucket)
console.print("\n[blue]Step 2: Getting tenant information...[/blue]")
tenant_info = asyncio.run(get_mount_info())
tenant_id = tenant_info.tenant_id
bucket_name = tenant_info.bucket_name
console.print(f"[green]✓ Found tenant: {tenant_id}[/green]")
console.print(f"[green]✓ Bucket: {bucket_name}[/green]")
# Step 3: Generate credentials
console.print("\n[blue]Step 3: Generating sync credentials...[/blue]")
creds = asyncio.run(generate_mount_credentials(tenant_id))
access_key = creds.access_key
secret_key = creds.secret_key
console.print("[green]✓ Generated secure credentials[/green]")
# Step 4: Configure rclone
console.print("\n[blue]Step 4: Configuring rclone...[/blue]")
add_tenant_to_rclone_config(
tenant_id=tenant_id,
bucket_name=bucket_name,
access_key=access_key,
secret_key=secret_key,
)
# Step 5: Configure and create local directory
console.print("\n[blue]Step 5: Configuring sync directory...[/blue]")
# If custom sync_dir provided, save to config
if sync_dir:
config_manager = ConfigManager()
config = config_manager.load_config()
config.bisync_config["sync_dir"] = sync_dir
config_manager.save_config(config)
console.print("[green]✓ Saved custom sync directory to config[/green]")
# Get bisync directory (from config or default)
local_path = get_bisync_directory()
# Validate bisync directory
validate_bisync_directory(local_path)
# Create directory
local_path.mkdir(parents=True, exist_ok=True)
console.print(f"[green]✓ Created sync directory: {local_path}[/green]")
# Step 6: Perform initial resync
console.print("\n[blue]Step 6: Performing initial sync...[/blue]")
console.print("[yellow]This will establish the baseline for bidirectional sync.[/yellow]")
run_bisync(
tenant_id=tenant_id,
bucket_name=bucket_name,
local_path=local_path,
profile_name="balanced",
resync=True,
)
console.print("\n[bold green]✓ Bisync setup completed successfully![/bold green]")
console.print("\nYour local files will now sync bidirectionally with the cloud!")
console.print(f"\nLocal directory: {local_path}")
console.print("\nUseful commands:")
console.print(" bm sync # Run sync (recommended)")
console.print(" bm sync --watch # Start watch mode")
console.print(" bm cloud status # Check sync status")
console.print(" bm cloud check # Verify file integrity")
console.print(" bm cloud bisync --dry-run # Preview changes (advanced)")
except (RcloneInstallError, BisyncError, CloudAPIError) as e:
console.print(f"\n[red]Setup failed: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"\n[red]Unexpected error during setup: {e}[/red]")
raise typer.Exit(1)
def run_bisync(
tenant_id: Optional[str] = None,
bucket_name: Optional[str] = None,
local_path: Optional[Path] = None,
profile_name: str = "balanced",
dry_run: bool = False,
resync: bool = False,
verbose: bool = False,
) -> bool:
"""Run rclone bisync with specified profile."""
try:
# Get tenant info if not provided
if not tenant_id or not bucket_name:
tenant_info = asyncio.run(get_mount_info())
tenant_id = tenant_info.tenant_id
bucket_name = tenant_info.bucket_name
# Set default local path if not provided
if not local_path:
local_path = get_bisync_directory()
# Validate bisync directory
validate_bisync_directory(local_path)
# Check if local path exists
if not local_path.exists():
raise BisyncError(
f"Local directory {local_path} does not exist. Run 'basic-memory cloud bisync-setup' first."
)
# Get bisync profile
if profile_name not in BISYNC_PROFILES:
raise BisyncError(
f"Unknown profile: {profile_name}. Available: {list(BISYNC_PROFILES.keys())}"
)
profile = BISYNC_PROFILES[profile_name]
# Auto-register projects before sync (unless dry-run or resync)
if not dry_run and not resync:
try:
console.print("[dim]Checking for new projects...[/dim]")
# Fetch cloud projects and extract directory names from paths
cloud_data = asyncio.run(fetch_cloud_projects())
cloud_projects = cloud_data.projects
# Extract directory names from cloud project paths
# Compare directory names, not project names
# Cloud path /app/data/basic-memory -> directory name "basic-memory"
cloud_dir_names = set()
for p in cloud_projects:
path = p.path
# Strip /app/data/ prefix if present (cloud mode)
if path.startswith("/app/data/"):
path = path[len("/app/data/") :]
# Get the last segment (directory name)
dir_name = Path(path).name
cloud_dir_names.add(dir_name)
# Scan local directories
local_dirs = scan_local_directories(local_path)
# Create missing cloud projects
new_projects = []
for dir_name in local_dirs:
if dir_name not in cloud_dir_names:
new_projects.append(dir_name)
if new_projects:
console.print(
f"[blue]Found {len(new_projects)} new local project(s), creating on cloud...[/blue]"
)
for project_name in new_projects:
try:
asyncio.run(create_cloud_project(project_name))
console.print(f"[green] ✓ Created project: {project_name}[/green]")
except BisyncError as e:
console.print(
f"[yellow] ⚠ Could not create {project_name}: {e}[/yellow]"
)
else:
console.print("[dim]All local projects already registered on cloud[/dim]")
except Exception as e:
console.print(f"[yellow]Warning: Project auto-registration failed: {e}[/yellow]")
console.print("[yellow]Continuing with sync anyway...[/yellow]")
# Check if first run and require resync
if not resync and not bisync_state_exists(tenant_id) and not dry_run:
raise BisyncError(
"First bisync requires --resync to establish baseline. "
"Run: basic-memory cloud bisync --resync"
)
# Build and execute bisync command
bisync_cmd = build_bisync_command(
tenant_id,
bucket_name,
local_path,
profile,
dry_run=dry_run,
resync=resync,
verbose=verbose,
)
if dry_run:
console.print("[yellow]DRY RUN MODE - No changes will be made[/yellow]")
console.print(
f"[blue]Running bisync with profile '{profile_name}' ({profile.description})...[/blue]"
)
console.print(f"[dim]Command: {' '.join(bisync_cmd)}[/dim]")
console.print() # Blank line before output
# Stream output in real-time so user sees progress
result = subprocess.run(bisync_cmd, text=True)
if result.returncode != 0:
raise BisyncError(f"Bisync command failed with code {result.returncode}")
console.print() # Blank line after output
if dry_run:
console.print("[green]✓ Dry run completed successfully[/green]")
elif resync:
console.print("[green]✓ Initial sync baseline established[/green]")
else:
console.print("[green]✓ Sync completed successfully[/green]")
# Notify container to refresh cache (if not dry run)
if not dry_run:
try:
asyncio.run(notify_container_sync(tenant_id))
except Exception as e:
console.print(f"[yellow]Warning: Could not notify container: {e}[/yellow]")
return True
except BisyncError:
raise
except Exception as e:
raise BisyncError(f"Unexpected error during bisync: {e}") from e
async def notify_container_sync(tenant_id: str) -> None:
"""Sync all projects after bisync completes."""
try:
from basic_memory.cli.commands.command_utils import run_sync
# Fetch all projects and sync each one
cloud_data = await fetch_cloud_projects()
projects = cloud_data.projects
if not projects:
console.print("[dim]No projects to sync[/dim]")
return
console.print(f"[blue]Notifying cloud to index {len(projects)} project(s)...[/blue]")
for project in projects:
project_name = project.name
if project_name:
try:
await run_sync(project=project_name)
except Exception as e:
# Non-critical, log and continue
console.print(f"[yellow] ⚠ Sync failed for {project_name}: {e}[/yellow]")
console.print("[dim]Note: Cloud indexing has started and may take a few moments[/dim]")
except Exception as e:
# Non-critical, don't fail the bisync
console.print(f"[yellow]Warning: Post-sync failed: {e}[/yellow]")
def run_bisync_watch(
tenant_id: Optional[str] = None,
bucket_name: Optional[str] = None,
local_path: Optional[Path] = None,
profile_name: str = "balanced",
interval_seconds: int = 60,
) -> None:
"""Run bisync in watch mode with periodic syncs."""
console.print("[bold blue]Starting bisync watch mode[/bold blue]")
console.print(f"Sync interval: {interval_seconds} seconds")
console.print("Press Ctrl+C to stop\n")
try:
while True:
try:
start_time = time.time()
run_bisync(
tenant_id=tenant_id,
bucket_name=bucket_name,
local_path=local_path,
profile_name=profile_name,
)
elapsed = time.time() - start_time
console.print(f"[dim]Sync completed in {elapsed:.1f}s[/dim]")
# Wait for next interval
time.sleep(interval_seconds)
except BisyncError as e:
console.print(f"[red]Sync error: {e}[/red]")
console.print(f"[yellow]Retrying in {interval_seconds} seconds...[/yellow]")
time.sleep(interval_seconds)
except KeyboardInterrupt:
console.print("\n[yellow]Watch mode stopped[/yellow]")
def show_bisync_status() -> None:
"""Show current bisync status and configuration."""
try:
# Get tenant info
tenant_info = asyncio.run(get_mount_info())
tenant_id = tenant_info.tenant_id
local_path = get_bisync_directory()
state_path = get_bisync_state_path(tenant_id)
# Create status table
table = Table(title="Cloud Bisync Status", show_header=True, header_style="bold blue")
table.add_column("Property", style="green", min_width=20)
table.add_column("Value", style="dim", min_width=30)
# Check initialization status
is_initialized = bisync_state_exists(tenant_id)
init_status = (
"[green]✓ Initialized[/green]" if is_initialized else "[red]✗ Not initialized[/red]"
)
table.add_row("Tenant ID", tenant_id)
table.add_row("Local Directory", str(local_path))
table.add_row("Status", init_status)
table.add_row("State Directory", str(state_path))
# Check for last sync info
if is_initialized:
# Look for most recent state file
state_files = list(state_path.glob("*.lst"))
if state_files:
latest = max(state_files, key=lambda p: p.stat().st_mtime)
last_sync = datetime.fromtimestamp(latest.stat().st_mtime)
table.add_row("Last Sync", last_sync.strftime("%Y-%m-%d %H:%M:%S"))
console.print(table)
# Show bisync profiles
console.print("\n[bold]Available bisync profiles:[/bold]")
for name, profile in BISYNC_PROFILES.items():
console.print(f" {name}: {profile.description}")
console.print(f" - Conflict resolution: {profile.conflict_resolve}")
console.print(f" - Max delete: {profile.max_delete} files")
console.print("\n[dim]To use a profile: bm cloud bisync --profile <name>[/dim]")
# Show setup instructions if not initialized
if not is_initialized:
console.print("\n[yellow]To initialize bisync, run:[/yellow]")
console.print(" bm cloud setup")
console.print(" or")
console.print(" bm cloud bisync --resync")
except Exception as e:
console.print(f"[red]Error getting bisync status: {e}[/red]")
raise typer.Exit(1)
def run_check(
tenant_id: Optional[str] = None,
bucket_name: Optional[str] = None,
local_path: Optional[Path] = None,
one_way: bool = False,
) -> bool:
"""Check file integrity between local and cloud using rclone check.
Args:
tenant_id: Cloud tenant ID (auto-detected if not provided)
bucket_name: S3 bucket name (auto-detected if not provided)
local_path: Local bisync directory (uses config default if not provided)
one_way: If True, only check for missing files on destination (faster)
Returns:
True if check passed (files match), False if differences found
"""
try:
# Check if rclone is installed
from basic_memory.cli.commands.cloud.rclone_installer import is_rclone_installed
if not is_rclone_installed():
raise BisyncError(
"rclone is not installed. Run 'bm cloud bisync-setup' first to set up cloud sync."
)
# Get tenant info if not provided
if not tenant_id or not bucket_name:
tenant_info = asyncio.run(get_mount_info())
tenant_id = tenant_id or tenant_info.tenant_id
bucket_name = bucket_name or tenant_info.bucket_name
# Get local path from config
if not local_path:
local_path = get_bisync_directory()
# Check if bisync is initialized
if not bisync_state_exists(tenant_id):
raise BisyncError(
"Bisync not initialized. Run 'bm cloud bisync --resync' to establish baseline."
)
# Build rclone check command
rclone_remote = f"basic-memory-{tenant_id}:{bucket_name}"
filter_path = get_bisync_filter_path()
cmd = [
"rclone",
"check",
str(local_path),
rclone_remote,
"--filter-from",
str(filter_path),
]
if one_way:
cmd.append("--one-way")
console.print("[bold blue]Checking file integrity between local and cloud[/bold blue]")
console.print(f"[dim]Local: {local_path}[/dim]")
console.print(f"[dim]Remote: {rclone_remote}[/dim]")
console.print(f"[dim]Command: {' '.join(cmd)}[/dim]")
console.print()
# Run check command
result = subprocess.run(cmd, capture_output=True, text=True)
# rclone check returns:
# 0 = success (all files match)
# non-zero = differences found or error
if result.returncode == 0:
console.print("[green]✓ All files match between local and cloud[/green]")
return True
else:
console.print("[yellow]⚠ Differences found:[/yellow]")
if result.stderr:
console.print(result.stderr)
if result.stdout:
console.print(result.stdout)
console.print("\n[dim]To sync differences, run: bm sync[/dim]")
return False
except BisyncError:
raise
except Exception as e:
raise BisyncError(f"Check failed: {e}") from e
@@ -16,7 +16,10 @@ class CloudUtilsError(Exception):
pass
async def fetch_cloud_projects() -> CloudProjectList:
async def fetch_cloud_projects(
*,
api_request=make_api_request,
) -> CloudProjectList:
"""Fetch list of projects from cloud API.
Returns:
@@ -27,14 +30,18 @@ async def fetch_cloud_projects() -> CloudProjectList:
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
response = await make_api_request(method="GET", url=f"{host_url}/proxy/projects/projects")
response = await api_request(method="GET", url=f"{host_url}/proxy/projects/projects")
return CloudProjectList.model_validate(response.json())
except Exception as e:
raise CloudUtilsError(f"Failed to fetch cloud projects: {e}") from e
async def create_cloud_project(project_name: str) -> CloudProjectCreateResponse:
async def create_cloud_project(
project_name: str,
*,
api_request=make_api_request,
) -> CloudProjectCreateResponse:
"""Create a new project on cloud.
Args:
@@ -57,7 +64,7 @@ async def create_cloud_project(project_name: str) -> CloudProjectCreateResponse:
set_default=False,
)
response = await make_api_request(
response = await api_request(
method="POST",
url=f"{host_url}/proxy/projects/projects",
headers={"Content-Type": "application/json"},
@@ -69,21 +76,22 @@ async def create_cloud_project(project_name: str) -> CloudProjectCreateResponse:
raise CloudUtilsError(f"Failed to create cloud project '{project_name}': {e}") from e
async def sync_project(project_name: str) -> None:
async def sync_project(project_name: str, force_full: bool = False) -> None:
"""Trigger sync for a specific project on cloud.
Args:
project_name: Name of project to sync
force_full: If True, force a full scan bypassing watermark optimization
"""
try:
from basic_memory.cli.commands.command_utils import run_sync
await run_sync(project=project_name)
await run_sync(project=project_name, force_full=force_full)
except Exception as e:
raise CloudUtilsError(f"Failed to sync project '{project_name}': {e}") from e
async def project_exists(project_name: str) -> bool:
async def project_exists(project_name: str, *, api_request=make_api_request) -> bool:
"""Check if a project exists on cloud.
Args:
@@ -93,7 +101,7 @@ async def project_exists(project_name: str) -> bool:
True if project exists, False otherwise
"""
try:
projects = await fetch_cloud_projects()
projects = await fetch_cloud_projects(api_request=api_request)
project_names = {p.name for p in projects.projects}
return project_name in project_names
except Exception:
@@ -1,12 +1,10 @@
"""Core cloud commands for Basic Memory CLI."""
import asyncio
from typing import Optional
import typer
from rich.console import Console
from basic_memory.cli.app import cloud_app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.cli.auth import CLIAuth
from basic_memory.config import ConfigManager
from basic_memory.cli.commands.cloud.api_client import (
@@ -15,21 +13,16 @@ from basic_memory.cli.commands.cloud.api_client import (
get_cloud_config,
make_api_request,
)
from basic_memory.cli.commands.cloud.mount_commands import (
mount_cloud_files,
setup_cloud_mount,
show_mount_status,
unmount_cloud_files,
)
from basic_memory.cli.commands.cloud.bisync_commands import (
run_bisync,
run_bisync_watch,
run_check,
setup_cloud_bisync,
show_bisync_status,
BisyncError,
generate_mount_credentials,
get_mount_info,
)
from basic_memory.cli.commands.cloud.rclone_config import configure_rclone_remote
from basic_memory.cli.commands.cloud.rclone_installer import (
RcloneInstallError,
install_rclone,
)
from basic_memory.cli.commands.cloud.rclone_config import MOUNT_PROFILES
from basic_memory.cli.commands.cloud.bisync_commands import BISYNC_PROFILES
console = Console()
@@ -58,11 +51,11 @@ def login():
config.cloud_mode = True
config_manager.save_config(config)
console.print("[green]Cloud mode enabled[/green]")
console.print("[green]Cloud mode enabled[/green]")
console.print(f"[dim]All CLI commands now work against {host_url}[/dim]")
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
console.print(
@@ -70,7 +63,7 @@ def login():
)
raise typer.Exit(1)
asyncio.run(_login())
run_with_cleanup(_login())
@cloud_app.command()
@@ -83,23 +76,13 @@ def logout():
config.cloud_mode = False
config_manager.save_config(config)
console.print("[green]Cloud mode disabled[/green]")
console.print("[green]Cloud mode disabled[/green]")
console.print("[dim]All CLI commands now work locally[/dim]")
@cloud_app.command("status")
def status(
bisync: bool = typer.Option(
True,
"--bisync/--mount",
help="Show bisync status (default) or mount status",
),
) -> None:
"""Check cloud mode status and cloud instance health.
Shows cloud mode status, instance health, and sync/mount status.
Use --bisync (default) to show bisync status or --mount for mount status.
"""
def status() -> None:
"""Check cloud mode status and cloud instance health."""
# Check cloud mode
config_manager = ConfigManager()
config = config_manager.load_config()
@@ -126,7 +109,7 @@ def status(
console.print("\n[blue]Checking cloud instance health...[/blue]")
# Make API request to check health
response = asyncio.run(
response = run_with_cleanup(
make_api_request(method="GET", url=f"{host_url}/proxy/health", headers=headers)
)
@@ -142,12 +125,7 @@ def status(
if "timestamp" in health_data:
console.print(f" Timestamp: {health_data['timestamp']}")
# Show sync/mount status based on flag
console.print()
if bisync:
show_bisync_status()
else:
show_mount_status()
console.print("\n[dim]To sync projects, use: bm project bisync --name <project>[/dim]")
except CloudAPIError as e:
console.print(f"[red]Error checking cloud health: {e}[/red]")
@@ -157,132 +135,59 @@ def status(
raise typer.Exit(1)
# Mount commands
@cloud_app.command("setup")
def setup(
bisync: bool = typer.Option(
True,
"--bisync/--mount",
help="Use bidirectional sync (recommended) or mount as network drive",
),
sync_dir: Optional[str] = typer.Option(
None,
"--dir",
help="Custom sync directory for bisync (default: ~/basic-memory-cloud-sync)",
),
) -> None:
"""Set up cloud file access with automatic rclone installation and configuration.
def setup() -> None:
"""Set up cloud sync by installing rclone and configuring credentials.
Default: Sets up bidirectional sync (recommended).\n
Use --mount: Sets up mount as network drive (alternative workflow).\n
Examples:\n
bm cloud setup # Setup bisync (default)\n
bm cloud setup --mount # Setup mount instead\n
bm cloud setup --dir ~/sync # Custom bisync directory\n
After setup, use project commands for syncing:
bm project add <name> <path> --local-path ~/projects/<name>
bm project bisync --name <name> --resync # First time
bm project bisync --name <name> # Subsequent syncs
"""
if bisync:
setup_cloud_bisync(sync_dir=sync_dir)
else:
setup_cloud_mount()
console.print("[bold blue]Basic Memory Cloud Setup[/bold blue]")
console.print("Setting up cloud sync with rclone...\n")
@cloud_app.command("mount")
def mount(
profile: str = typer.Option(
"balanced", help=f"Mount profile: {', '.join(MOUNT_PROFILES.keys())}"
),
path: Optional[str] = typer.Option(
None, help="Custom mount path (default: ~/basic-memory-{tenant-id})"
),
) -> None:
"""Mount cloud files locally for editing."""
try:
mount_cloud_files(profile_name=profile)
except Exception as e:
console.print(f"[red]Mount failed: {e}[/red]")
# Step 1: Install rclone
console.print("[blue]Step 1: Installing rclone...[/blue]")
install_rclone()
# Step 2: Get tenant info
console.print("\n[blue]Step 2: Getting tenant information...[/blue]")
tenant_info = run_with_cleanup(get_mount_info())
console.print(f"[green]Found tenant: {tenant_info.tenant_id}[/green]")
# Step 3: Generate credentials
console.print("\n[blue]Step 3: Generating sync credentials...[/blue]")
creds = run_with_cleanup(generate_mount_credentials(tenant_info.tenant_id))
console.print("[green]Generated secure credentials[/green]")
# Step 4: Configure rclone remote
console.print("\n[blue]Step 4: Configuring rclone remote...[/blue]")
configure_rclone_remote(
access_key=creds.access_key,
secret_key=creds.secret_key,
)
console.print("\n[bold green]Cloud setup completed successfully![/bold green]")
console.print("\n[bold]Next steps:[/bold]")
console.print("1. Add a project with local sync path:")
console.print(" bm project add research --local-path ~/Documents/research")
console.print("\n Or configure sync for an existing project:")
console.print(" bm project sync-setup research ~/Documents/research")
console.print("\n2. Preview the initial sync (recommended):")
console.print(" bm project bisync --name research --resync --dry-run")
console.print("\n3. If all looks good, run the actual sync:")
console.print(" bm project bisync --name research --resync")
console.print("\n4. Subsequent syncs (no --resync needed):")
console.print(" bm project bisync --name research")
console.print(
"\n[dim]Tip: Always use --dry-run first to preview changes before syncing[/dim]"
)
except (RcloneInstallError, BisyncError, CloudAPIError) as e:
console.print(f"\n[red]Setup failed: {e}[/red]")
raise typer.Exit(1)
@cloud_app.command("unmount")
def unmount() -> None:
"""Unmount cloud files."""
try:
unmount_cloud_files()
except Exception as e:
console.print(f"[red]Unmount failed: {e}[/red]")
raise typer.Exit(1)
# Bisync commands
@cloud_app.command("bisync")
def bisync(
profile: str = typer.Option(
"balanced", help=f"Bisync profile: {', '.join(BISYNC_PROFILES.keys())}"
),
dry_run: bool = typer.Option(False, "--dry-run", help="Preview changes without syncing"),
resync: bool = typer.Option(False, "--resync", help="Force resync to establish new baseline"),
watch: bool = typer.Option(False, "--watch", help="Run continuous sync in watch mode"),
interval: int = typer.Option(60, "--interval", help="Sync interval in seconds for watch mode"),
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed sync output"),
) -> None:
"""Run bidirectional sync between local files and cloud storage.
Examples:
basic-memory cloud bisync # Manual sync with balanced profile
basic-memory cloud bisync --dry-run # Preview what would be synced
basic-memory cloud bisync --resync # Establish new baseline
basic-memory cloud bisync --watch # Continuous sync every 60s
basic-memory cloud bisync --watch --interval 30 # Continuous sync every 30s
basic-memory cloud bisync --profile safe # Use safe profile (keep conflicts)
basic-memory cloud bisync --verbose # Show detailed file sync output
"""
try:
if watch:
run_bisync_watch(profile_name=profile, interval_seconds=interval)
else:
run_bisync(profile_name=profile, dry_run=dry_run, resync=resync, verbose=verbose)
except Exception as e:
console.print(f"[red]Bisync failed: {e}[/red]")
raise typer.Exit(1)
@cloud_app.command("bisync-status")
def bisync_status() -> None:
"""Show current bisync status and configuration.
DEPRECATED: Use 'bm cloud status' instead (bisync is now the default).
"""
console.print(
"[yellow]Note: 'bisync-status' is deprecated. Use 'bm cloud status' instead.[/yellow]"
)
console.print("[dim]Showing bisync status...[/dim]\n")
show_bisync_status()
@cloud_app.command("check")
def check(
one_way: bool = typer.Option(
False,
"--one-way",
help="Only check for missing files on destination (faster)",
),
) -> None:
"""Check file integrity between local and cloud storage using rclone check.
Verifies that files match between your local bisync directory and cloud storage
without transferring any data. This is useful for validating sync integrity.
Examples:
bm cloud check # Full integrity check
bm cloud check --one-way # Faster check (missing files only)
"""
try:
run_check(one_way=one_way)
except Exception as e:
console.print(f"[red]Check failed: {e}[/red]")
console.print(f"\n[red]Unexpected error during setup: {e}[/red]")
raise typer.Exit(1)
@@ -1,295 +0,0 @@
"""Cloud mount commands for Basic Memory CLI."""
import asyncio
import subprocess
import time
from pathlib import Path
from typing import Optional
import typer
from rich.console import Console
from rich.table import Table
from basic_memory.cli.commands.cloud.api_client import CloudAPIError, make_api_request
from basic_memory.cli.commands.cloud.rclone_config import (
MOUNT_PROFILES,
add_tenant_to_rclone_config,
build_mount_command,
cleanup_orphaned_rclone_processes,
get_default_mount_path,
get_rclone_processes,
is_path_mounted,
unmount_path,
)
from basic_memory.cli.commands.cloud.rclone_installer import RcloneInstallError, install_rclone
from basic_memory.config import ConfigManager
console = Console()
class MountError(Exception):
"""Exception raised for mount-related errors."""
pass
async def get_tenant_info() -> dict:
"""Get current tenant information from cloud API."""
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
response = await make_api_request(method="GET", url=f"{host_url}/tenant/mount/info")
return response.json()
except Exception as e:
raise MountError(f"Failed to get tenant info: {e}") from e
async def generate_mount_credentials(tenant_id: str) -> dict:
"""Generate scoped credentials for mounting."""
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
response = await make_api_request(method="POST", url=f"{host_url}/tenant/mount/credentials")
return response.json()
except Exception as e:
raise MountError(f"Failed to generate mount credentials: {e}") from e
def setup_cloud_mount() -> None:
"""Set up cloud mount with rclone installation and configuration."""
console.print("[bold blue]Basic Memory Cloud Setup[/bold blue]")
console.print("Setting up local file access to your cloud tenant...\n")
try:
# Step 1: Install rclone
console.print("[blue]Step 1: Installing rclone...[/blue]")
install_rclone()
# Step 2: Get tenant info
console.print("\n[blue]Step 2: Getting tenant information...[/blue]")
tenant_info = asyncio.run(get_tenant_info())
tenant_id = tenant_info.get("tenant_id")
bucket_name = tenant_info.get("bucket_name")
if not tenant_id or not bucket_name:
raise MountError("Invalid tenant information received from cloud API")
console.print(f"[green]✓ Found tenant: {tenant_id}[/green]")
console.print(f"[green]✓ Bucket: {bucket_name}[/green]")
# Step 3: Generate mount credentials
console.print("\n[blue]Step 3: Generating mount credentials...[/blue]")
creds = asyncio.run(generate_mount_credentials(tenant_id))
access_key = creds.get("access_key")
secret_key = creds.get("secret_key")
if not access_key or not secret_key:
raise MountError("Failed to generate mount credentials")
console.print("[green]✓ Generated secure credentials[/green]")
# Step 4: Configure rclone
console.print("\n[blue]Step 4: Configuring rclone...[/blue]")
add_tenant_to_rclone_config(
tenant_id=tenant_id,
bucket_name=bucket_name,
access_key=access_key,
secret_key=secret_key,
)
# Step 5: Perform initial mount
console.print("\n[blue]Step 5: Mounting cloud files...[/blue]")
mount_path = get_default_mount_path()
MOUNT_PROFILES["balanced"]
mount_cloud_files(
tenant_id=tenant_id,
bucket_name=bucket_name,
mount_path=mount_path,
profile_name="balanced",
)
console.print("\n[bold green]✓ Cloud setup completed successfully![/bold green]")
console.print("\nYour cloud files are now accessible at:")
console.print(f" {mount_path}")
console.print("\nYou can now edit files locally and they will sync to the cloud!")
console.print("\nUseful commands:")
console.print(" basic-memory cloud mount-status # Check mount status")
console.print(" basic-memory cloud unmount # Unmount files")
console.print(" basic-memory cloud mount --profile fast # Remount with faster sync")
except (RcloneInstallError, MountError, CloudAPIError) as e:
console.print(f"\n[red]Setup failed: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"\n[red]Unexpected error during setup: {e}[/red]")
raise typer.Exit(1)
def mount_cloud_files(
tenant_id: Optional[str] = None,
bucket_name: Optional[str] = None,
mount_path: Optional[Path] = None,
profile_name: str = "balanced",
) -> None:
"""Mount cloud files with specified profile."""
try:
# Get tenant info if not provided
if not tenant_id or not bucket_name:
tenant_info = asyncio.run(get_tenant_info())
tenant_id = tenant_info.get("tenant_id")
bucket_name = tenant_info.get("bucket_name")
if not tenant_id or not bucket_name:
raise MountError("Could not determine tenant information")
# Set default mount path if not provided
if not mount_path:
mount_path = get_default_mount_path()
# Get mount profile
if profile_name not in MOUNT_PROFILES:
raise MountError(
f"Unknown profile: {profile_name}. Available: {list(MOUNT_PROFILES.keys())}"
)
profile = MOUNT_PROFILES[profile_name]
# Check if already mounted
if is_path_mounted(mount_path):
console.print(f"[yellow]Path {mount_path} is already mounted[/yellow]")
console.print("Use 'basic-memory cloud unmount' first, or mount to a different path")
return
# Create mount directory
mount_path.mkdir(parents=True, exist_ok=True)
# Build and execute mount command
mount_cmd = build_mount_command(tenant_id, bucket_name, mount_path, profile)
console.print(
f"[blue]Mounting with profile '{profile_name}' ({profile.description})...[/blue]"
)
console.print(f"[dim]Command: {' '.join(mount_cmd)}[/dim]")
result = subprocess.run(mount_cmd, capture_output=True, text=True)
if result.returncode != 0:
error_msg = result.stderr or "Unknown error"
raise MountError(f"Mount command failed: {error_msg}")
# Wait a moment for mount to establish
time.sleep(2)
# Verify mount
if is_path_mounted(mount_path):
console.print(f"[green]✓ Successfully mounted to {mount_path}[/green]")
console.print(f"[green]✓ Sync profile: {profile.description}[/green]")
else:
raise MountError("Mount command succeeded but path is not mounted")
except MountError:
raise
except Exception as e:
raise MountError(f"Unexpected error during mount: {e}") from e
def unmount_cloud_files(tenant_id: Optional[str] = None) -> None:
"""Unmount cloud files."""
try:
# Get tenant info if not provided
if not tenant_id:
tenant_info = asyncio.run(get_tenant_info())
tenant_id = tenant_info.get("tenant_id")
if not tenant_id:
raise MountError("Could not determine tenant ID")
mount_path = get_default_mount_path()
if not is_path_mounted(mount_path):
console.print(f"[yellow]Path {mount_path} is not mounted[/yellow]")
return
console.print(f"[blue]Unmounting {mount_path}...[/blue]")
# Unmount the path
if unmount_path(mount_path):
console.print(f"[green]✓ Successfully unmounted {mount_path}[/green]")
# Clean up any orphaned rclone processes
killed_count = cleanup_orphaned_rclone_processes()
if killed_count > 0:
console.print(
f"[green]✓ Cleaned up {killed_count} orphaned rclone process(es)[/green]"
)
else:
console.print(f"[red]✗ Failed to unmount {mount_path}[/red]")
console.print("You may need to manually unmount or restart your system")
except MountError:
raise
except Exception as e:
raise MountError(f"Unexpected error during unmount: {e}") from e
def show_mount_status() -> None:
"""Show current mount status and running processes."""
try:
# Get tenant info
tenant_info = asyncio.run(get_tenant_info())
tenant_id = tenant_info.get("tenant_id")
if not tenant_id:
console.print("[red]Could not determine tenant ID[/red]")
return
mount_path = get_default_mount_path()
# Create status table
table = Table(title="Cloud Mount Status", show_header=True, header_style="bold blue")
table.add_column("Property", style="green", min_width=15)
table.add_column("Value", style="dim", min_width=30)
# Check mount status
is_mounted = is_path_mounted(mount_path)
mount_status = "[green]✓ Mounted[/green]" if is_mounted else "[red]✗ Not mounted[/red]"
table.add_row("Tenant ID", tenant_id)
table.add_row("Mount Path", str(mount_path))
table.add_row("Status", mount_status)
# Get rclone processes
processes = get_rclone_processes()
if processes:
table.add_row("rclone Processes", f"{len(processes)} running")
else:
table.add_row("rclone Processes", "None")
console.print(table)
# Show running processes details
if processes:
console.print("\n[bold]Running rclone processes:[/bold]")
for proc in processes:
console.print(f" PID {proc['pid']}: {proc['command'][:80]}...")
# Show mount profiles
console.print("\n[bold]Available mount profiles:[/bold]")
for name, profile in MOUNT_PROFILES.items():
console.print(f" {name}: {profile.description}")
except Exception as e:
console.print(f"[red]Error getting mount status: {e}[/red]")
raise typer.Exit(1)
@@ -0,0 +1,398 @@
"""Project-scoped rclone sync commands for Basic Memory Cloud.
This module provides simplified, project-scoped rclone operations:
- Each project syncs independently
- Uses single "basic-memory-cloud" remote (not tenant-specific)
- Balanced defaults from SPEC-8 Phase 4 testing
- Per-project bisync state tracking
Replaces tenant-wide sync with project-scoped workflows.
"""
import re
import subprocess
from dataclasses import dataclass
from functools import lru_cache
from pathlib import Path
from typing import Callable, Optional, Protocol
from loguru import logger
from rich.console import Console
from basic_memory.cli.commands.cloud.rclone_installer import is_rclone_installed
from basic_memory.utils import normalize_project_path
console = Console()
# Minimum rclone version for --create-empty-src-dirs support
MIN_RCLONE_VERSION_EMPTY_DIRS = (1, 64, 0)
class RunResult(Protocol):
returncode: int
stdout: str
RunFunc = Callable[..., RunResult]
IsInstalledFunc = Callable[[], bool]
class RcloneError(Exception):
"""Exception raised for rclone command errors."""
pass
def check_rclone_installed(is_installed: IsInstalledFunc = is_rclone_installed) -> None:
"""Check if rclone is installed and raise helpful error if not.
Raises:
RcloneError: If rclone is not installed with installation instructions
"""
if not is_installed():
raise RcloneError(
"rclone is not installed.\n\n"
"Install rclone by running: bm cloud setup\n"
"Or install manually from: https://rclone.org/downloads/\n\n"
"Windows users: Ensure you have a package manager installed (winget, chocolatey, or scoop)"
)
@lru_cache(maxsize=1)
def get_rclone_version(run: RunFunc = subprocess.run) -> tuple[int, int, int] | None:
"""Get rclone version as (major, minor, patch) tuple.
Returns:
Version tuple like (1, 64, 2), or None if version cannot be determined.
Note:
Result is cached since rclone version won't change during runtime.
"""
try:
result = run(["rclone", "version"], capture_output=True, text=True, timeout=10)
# Parse "rclone v1.64.2" or "rclone v1.60.1-DEV"
match = re.search(r"v(\d+)\.(\d+)\.(\d+)", result.stdout)
if match:
version = (int(match.group(1)), int(match.group(2)), int(match.group(3)))
logger.debug(f"Detected rclone version: {version}")
return version
except Exception as e:
logger.warning(f"Could not determine rclone version: {e}")
return None
def supports_create_empty_src_dirs(version: tuple[int, int, int] | None) -> bool:
"""Check if installed rclone supports --create-empty-src-dirs flag.
Returns:
True if rclone version >= 1.64.0, False otherwise.
"""
if version is None:
# If we can't determine version, assume older and skip the flag
return False
return version >= MIN_RCLONE_VERSION_EMPTY_DIRS
@dataclass
class SyncProject:
"""Project configured for cloud sync.
Attributes:
name: Project name
path: Cloud path (e.g., "app/data/research")
local_sync_path: Local directory for syncing (optional)
"""
name: str
path: str
local_sync_path: Optional[str] = None
def get_bmignore_filter_path() -> Path:
"""Get path to rclone filter file.
Uses ~/.basic-memory/.bmignore converted to rclone format.
File is automatically created with default patterns on first use.
Returns:
Path to rclone filter file
"""
# Import here to avoid circular dependency
from basic_memory.cli.commands.cloud.bisync_commands import (
convert_bmignore_to_rclone_filters,
)
return convert_bmignore_to_rclone_filters()
def get_project_bisync_state(project_name: str) -> Path:
"""Get path to project's bisync state directory.
Args:
project_name: Name of the project
Returns:
Path to bisync state directory for this project
"""
return Path.home() / ".basic-memory" / "bisync-state" / project_name
def bisync_initialized(project_name: str) -> bool:
"""Check if bisync has been initialized for this project.
Args:
project_name: Name of the project
Returns:
True if bisync state exists, False otherwise
"""
state_path = get_project_bisync_state(project_name)
return state_path.exists() and any(state_path.iterdir())
def get_project_remote(project: SyncProject, bucket_name: str) -> str:
"""Build rclone remote path for project.
Args:
project: Project with cloud path
bucket_name: S3 bucket name
Returns:
Remote path like "basic-memory-cloud:bucket-name/basic-memory-llc"
Note:
The API returns paths like "/app/data/basic-memory-llc" because the S3 bucket
is mounted at /app/data on the fly machine. We need to strip the /app/data/
prefix to get the actual S3 path within the bucket.
"""
# Normalize path to strip /app/data/ mount point prefix
cloud_path = normalize_project_path(project.path).lstrip("/")
return f"basic-memory-cloud:{bucket_name}/{cloud_path}"
def project_sync(
project: SyncProject,
bucket_name: str,
dry_run: bool = False,
verbose: bool = False,
*,
run: RunFunc = subprocess.run,
is_installed: IsInstalledFunc = is_rclone_installed,
filter_path: Path | None = None,
) -> bool:
"""One-way sync: local → cloud.
Makes cloud identical to local using rclone sync.
Args:
project: Project to sync
bucket_name: S3 bucket name
dry_run: Preview changes without applying
verbose: Show detailed output
Returns:
True if sync succeeded, False otherwise
Raises:
RcloneError: If project has no local_sync_path configured or rclone not installed
"""
check_rclone_installed(is_installed=is_installed)
if not project.local_sync_path:
raise RcloneError(f"Project {project.name} has no local_sync_path configured")
local_path = Path(project.local_sync_path).expanduser()
remote_path = get_project_remote(project, bucket_name)
filter_path = filter_path or get_bmignore_filter_path()
cmd = [
"rclone",
"sync",
str(local_path),
remote_path,
"--filter-from",
str(filter_path),
]
if verbose:
cmd.append("--verbose")
else:
cmd.append("--progress")
if dry_run:
cmd.append("--dry-run")
result = run(cmd, text=True)
return result.returncode == 0
def project_bisync(
project: SyncProject,
bucket_name: str,
dry_run: bool = False,
resync: bool = False,
verbose: bool = False,
*,
run: RunFunc = subprocess.run,
is_installed: IsInstalledFunc = is_rclone_installed,
version: tuple[int, int, int] | None = None,
filter_path: Path | None = None,
state_path: Path | None = None,
is_initialized: Callable[[str], bool] = bisync_initialized,
) -> bool:
"""Two-way sync: local ↔ cloud.
Uses rclone bisync with balanced defaults:
- conflict_resolve: newer (auto-resolve to most recent)
- max_delete: 25 (safety limit)
- compare: modtime (ignore size differences from line ending conversions)
- check_access: false (skip for performance)
Args:
project: Project to sync
bucket_name: S3 bucket name
dry_run: Preview changes without applying
resync: Force resync to establish new baseline
verbose: Show detailed output
Returns:
True if bisync succeeded, False otherwise
Raises:
RcloneError: If project has no local_sync_path, needs --resync, or rclone not installed
"""
check_rclone_installed(is_installed=is_installed)
if not project.local_sync_path:
raise RcloneError(f"Project {project.name} has no local_sync_path configured")
local_path = Path(project.local_sync_path).expanduser()
remote_path = get_project_remote(project, bucket_name)
filter_path = filter_path or get_bmignore_filter_path()
state_path = state_path or get_project_bisync_state(project.name)
# Ensure state directory exists
state_path.mkdir(parents=True, exist_ok=True)
cmd = [
"rclone",
"bisync",
str(local_path),
remote_path,
"--resilient",
"--conflict-resolve=newer",
"--max-delete=25",
"--compare=modtime", # Ignore size differences from line ending conversions
"--filter-from",
str(filter_path),
"--workdir",
str(state_path),
]
# Add --create-empty-src-dirs if rclone version supports it (v1.64+)
version = version if version is not None else get_rclone_version(run=run)
if supports_create_empty_src_dirs(version):
cmd.append("--create-empty-src-dirs")
if verbose:
cmd.append("--verbose")
else:
cmd.append("--progress")
if dry_run:
cmd.append("--dry-run")
if resync:
cmd.append("--resync")
# Check if first run requires resync
if not resync and not is_initialized(project.name) and not dry_run:
raise RcloneError(
f"First bisync for {project.name} requires --resync to establish baseline.\n"
f"Run: bm project bisync --name {project.name} --resync"
)
result = run(cmd, text=True)
return result.returncode == 0
def project_check(
project: SyncProject,
bucket_name: str,
one_way: bool = False,
*,
run: RunFunc = subprocess.run,
is_installed: IsInstalledFunc = is_rclone_installed,
filter_path: Path | None = None,
) -> bool:
"""Check integrity between local and cloud.
Verifies files match without transferring data.
Args:
project: Project to check
bucket_name: S3 bucket name
one_way: Only check for missing files on destination (faster)
Returns:
True if files match, False if differences found
Raises:
RcloneError: If project has no local_sync_path configured or rclone not installed
"""
check_rclone_installed(is_installed=is_installed)
if not project.local_sync_path:
raise RcloneError(f"Project {project.name} has no local_sync_path configured")
local_path = Path(project.local_sync_path).expanduser()
remote_path = get_project_remote(project, bucket_name)
filter_path = filter_path or get_bmignore_filter_path()
cmd = [
"rclone",
"check",
str(local_path),
remote_path,
"--filter-from",
str(filter_path),
]
if one_way:
cmd.append("--one-way")
result = run(cmd, capture_output=True, text=True)
return result.returncode == 0
def project_ls(
project: SyncProject,
bucket_name: str,
path: Optional[str] = None,
*,
run: RunFunc = subprocess.run,
is_installed: IsInstalledFunc = is_rclone_installed,
) -> list[str]:
"""List files in remote project.
Args:
project: Project to list files from
bucket_name: S3 bucket name
path: Optional subdirectory within project
Returns:
List of file paths
Raises:
subprocess.CalledProcessError: If rclone command fails
RcloneError: If rclone is not installed
"""
check_rclone_installed(is_installed=is_installed)
remote_path = get_project_remote(project, bucket_name)
if path:
remote_path = f"{remote_path}/{path}"
cmd = ["rclone", "ls", remote_path]
result = run(cmd, capture_output=True, text=True, check=True)
return result.stdout.splitlines()
@@ -1,11 +1,14 @@
"""rclone configuration management for Basic Memory Cloud."""
"""rclone configuration management for Basic Memory Cloud.
This module provides simplified rclone configuration for SPEC-20.
Uses a single "basic-memory-cloud" remote for all operations.
"""
import configparser
import os
import shutil
import subprocess
from pathlib import Path
from typing import Dict, List, Optional
from typing import Optional
from rich.console import Console
@@ -18,64 +21,6 @@ class RcloneConfigError(Exception):
pass
class RcloneMountProfile:
"""Mount profile with optimized settings."""
def __init__(
self,
name: str,
cache_time: str,
poll_interval: str,
attr_timeout: str,
write_back: str,
description: str,
extra_args: Optional[List[str]] = None,
):
self.name = name
self.cache_time = cache_time
self.poll_interval = poll_interval
self.attr_timeout = attr_timeout
self.write_back = write_back
self.description = description
self.extra_args = extra_args or []
# Mount profiles based on SPEC-7 Phase 4 testing
MOUNT_PROFILES = {
"fast": RcloneMountProfile(
name="fast",
cache_time="5s",
poll_interval="3s",
attr_timeout="3s",
write_back="1s",
description="Ultra-fast development (5s sync, higher bandwidth)",
),
"balanced": RcloneMountProfile(
name="balanced",
cache_time="10s",
poll_interval="5s",
attr_timeout="5s",
write_back="2s",
description="Fast development (10-15s sync, recommended)",
),
"safe": RcloneMountProfile(
name="safe",
cache_time="15s",
poll_interval="10s",
attr_timeout="10s",
write_back="5s",
description="Conflict-aware mount with backup",
extra_args=[
"--conflict-suffix",
".conflict-{DateTimeExt}",
"--backup-dir",
"~/.basic-memory/conflicts",
"--track-renames",
],
),
}
def get_rclone_config_path() -> Path:
"""Get the path to rclone configuration file."""
config_dir = Path.home() / ".config" / "rclone"
@@ -116,173 +61,50 @@ def save_rclone_config(config: configparser.ConfigParser) -> None:
console.print(f"[dim]Updated rclone config: {config_path}[/dim]")
def add_tenant_to_rclone_config(
tenant_id: str,
bucket_name: str,
def configure_rclone_remote(
access_key: str,
secret_key: str,
endpoint: str = "https://fly.storage.tigris.dev",
region: str = "auto",
) -> str:
"""Add tenant configuration to rclone config file."""
"""Configure single rclone remote named 'basic-memory-cloud'.
This is the simplified approach from SPEC-20 that uses one remote
for all Basic Memory cloud operations (not tenant-specific).
Args:
access_key: S3 access key ID
secret_key: S3 secret access key
endpoint: S3-compatible endpoint URL
region: S3 region (default: auto)
Returns:
The remote name: "basic-memory-cloud"
"""
# Backup existing config
backup_rclone_config()
# Load existing config
config = load_rclone_config()
# Create section name
section_name = f"basic-memory-{tenant_id}"
# Single remote name (not tenant-specific)
REMOTE_NAME = "basic-memory-cloud"
# Add/update the tenant section
if not config.has_section(section_name):
config.add_section(section_name)
config.set(section_name, "type", "s3")
config.set(section_name, "provider", "Other")
config.set(section_name, "access_key_id", access_key)
config.set(section_name, "secret_access_key", secret_key)
config.set(section_name, "endpoint", endpoint)
config.set(section_name, "region", region)
# Add/update the remote section
if not config.has_section(REMOTE_NAME):
config.add_section(REMOTE_NAME)
config.set(REMOTE_NAME, "type", "s3")
config.set(REMOTE_NAME, "provider", "Other")
config.set(REMOTE_NAME, "access_key_id", access_key)
config.set(REMOTE_NAME, "secret_access_key", secret_key)
config.set(REMOTE_NAME, "endpoint", endpoint)
config.set(REMOTE_NAME, "region", region)
# Prevent unnecessary encoding of filenames (only encode slashes and invalid UTF-8)
# This prevents files with spaces like "Hello World.md" from being quoted
config.set(REMOTE_NAME, "encoding", "Slash,InvalidUtf8")
# Save updated config
save_rclone_config(config)
console.print(f"[green]✓ Added tenant {tenant_id} to rclone config[/green]")
return section_name
def remove_tenant_from_rclone_config(tenant_id: str) -> bool:
"""Remove tenant configuration from rclone config."""
config = load_rclone_config()
section_name = f"basic-memory-{tenant_id}"
if config.has_section(section_name):
backup_rclone_config()
config.remove_section(section_name)
save_rclone_config(config)
console.print(f"[green]✓ Removed tenant {tenant_id} from rclone config[/green]")
return True
return False
def get_default_mount_path() -> Path:
"""Get default mount path (fixed location per SPEC-9).
Returns:
Fixed mount path: ~/basic-memory-cloud/
"""
return Path.home() / "basic-memory-cloud"
def build_mount_command(
tenant_id: str, bucket_name: str, mount_path: Path, profile: RcloneMountProfile
) -> List[str]:
"""Build rclone mount command with optimized settings."""
rclone_remote = f"basic-memory-{tenant_id}:{bucket_name}"
cmd = [
"rclone",
"nfsmount",
rclone_remote,
str(mount_path),
"--vfs-cache-mode",
"writes",
"--dir-cache-time",
profile.cache_time,
"--vfs-cache-poll-interval",
profile.poll_interval,
"--attr-timeout",
profile.attr_timeout,
"--vfs-write-back",
profile.write_back,
"--daemon",
]
# Add profile-specific extra arguments
cmd.extend(profile.extra_args)
return cmd
def is_path_mounted(mount_path: Path) -> bool:
"""Check if a path is currently mounted."""
if not mount_path.exists():
return False
try:
# Check if mount point is actually mounted by looking for mount table entry
result = subprocess.run(["mount"], capture_output=True, text=True, check=False)
if result.returncode == 0:
# Look for our mount path in mount output
mount_str = str(mount_path.resolve())
return mount_str in result.stdout
return False
except Exception:
return False
def get_rclone_processes() -> List[Dict[str, str]]:
"""Get list of running rclone processes."""
try:
# Use ps to find rclone processes
result = subprocess.run(
["ps", "-eo", "pid,args"], capture_output=True, text=True, check=False
)
processes = []
if result.returncode == 0:
for line in result.stdout.split("\n"):
if "rclone" in line and "basic-memory" in line:
parts = line.strip().split(None, 1)
if len(parts) >= 2:
processes.append({"pid": parts[0], "command": parts[1]})
return processes
except Exception:
return []
def kill_rclone_process(pid: str) -> bool:
"""Kill a specific rclone process."""
try:
subprocess.run(["kill", pid], check=True)
console.print(f"[green]✓ Killed rclone process {pid}[/green]")
return True
except subprocess.CalledProcessError:
console.print(f"[red]✗ Failed to kill rclone process {pid}[/red]")
return False
def unmount_path(mount_path: Path) -> bool:
"""Unmount a mounted path."""
if not is_path_mounted(mount_path):
return True
try:
subprocess.run(["umount", str(mount_path)], check=True)
console.print(f"[green]✓ Unmounted {mount_path}[/green]")
return True
except subprocess.CalledProcessError as e:
console.print(f"[red]✗ Failed to unmount {mount_path}: {e}[/red]")
return False
def cleanup_orphaned_rclone_processes() -> int:
"""Clean up orphaned rclone processes for basic-memory."""
processes = get_rclone_processes()
killed_count = 0
for proc in processes:
console.print(
f"[yellow]Found rclone process: {proc['pid']} - {proc['command'][:80]}...[/yellow]"
)
if kill_rclone_process(proc["pid"]):
killed_count += 1
return killed_count
console.print(f"[green]Configured rclone remote: {REMOTE_NAME}[/green]")
return REMOTE_NAME
@@ -1,5 +1,6 @@
"""Cross-platform rclone installation utilities."""
import os
import platform
import shutil
import subprocess
@@ -58,7 +59,7 @@ def install_rclone_macos() -> None:
try:
console.print("[blue]Installing rclone via Homebrew...[/blue]")
run_command(["brew", "install", "rclone"])
console.print("[green]rclone installed via Homebrew[/green]")
console.print("[green]rclone installed via Homebrew[/green]")
return
except RcloneInstallError:
console.print(
@@ -69,7 +70,7 @@ def install_rclone_macos() -> None:
console.print("[blue]Installing rclone via official script...[/blue]")
try:
run_command(["sh", "-c", "curl https://rclone.org/install.sh | sudo bash"])
console.print("[green]rclone installed via official script[/green]")
console.print("[green]rclone installed via official script[/green]")
except RcloneInstallError:
raise RcloneInstallError(
"Failed to install rclone. Please install manually: brew install rclone"
@@ -83,7 +84,7 @@ def install_rclone_linux() -> None:
try:
console.print("[blue]Installing rclone via snap...[/blue]")
run_command(["sudo", "snap", "install", "rclone"])
console.print("[green]rclone installed via snap[/green]")
console.print("[green]rclone installed via snap[/green]")
return
except RcloneInstallError:
console.print("[yellow]Snap installation failed, trying apt...[/yellow]")
@@ -94,7 +95,7 @@ def install_rclone_linux() -> None:
console.print("[blue]Installing rclone via apt...[/blue]")
run_command(["sudo", "apt", "update"])
run_command(["sudo", "apt", "install", "-y", "rclone"])
console.print("[green]rclone installed via apt[/green]")
console.print("[green]rclone installed via apt[/green]")
return
except RcloneInstallError:
console.print("[yellow]apt installation failed, trying official script...[/yellow]")
@@ -103,7 +104,7 @@ def install_rclone_linux() -> None:
console.print("[blue]Installing rclone via official script...[/blue]")
try:
run_command(["sh", "-c", "curl https://rclone.org/install.sh | sudo bash"])
console.print("[green]rclone installed via official script[/green]")
console.print("[green]rclone installed via official script[/green]")
except RcloneInstallError:
raise RcloneInstallError(
"Failed to install rclone. Please install manually: sudo snap install rclone"
@@ -116,8 +117,16 @@ def install_rclone_windows() -> None:
if shutil.which("winget"):
try:
console.print("[blue]Installing rclone via winget...[/blue]")
run_command(["winget", "install", "Rclone.Rclone"])
console.print("[green]✓ rclone installed via winget[/green]")
run_command(
[
"winget",
"install",
"Rclone.Rclone",
"--accept-source-agreements",
"--accept-package-agreements",
]
)
console.print("[green]rclone installed via winget[/green]")
return
except RcloneInstallError:
console.print("[yellow]winget installation failed, trying chocolatey...[/yellow]")
@@ -127,7 +136,7 @@ def install_rclone_windows() -> None:
try:
console.print("[blue]Installing rclone via chocolatey...[/blue]")
run_command(["choco", "install", "rclone", "-y"])
console.print("[green]rclone installed via chocolatey[/green]")
console.print("[green]rclone installed via chocolatey[/green]")
return
except RcloneInstallError:
console.print("[yellow]chocolatey installation failed, trying scoop...[/yellow]")
@@ -137,16 +146,30 @@ def install_rclone_windows() -> None:
try:
console.print("[blue]Installing rclone via scoop...[/blue]")
run_command(["scoop", "install", "rclone"])
console.print("[green]rclone installed via scoop[/green]")
console.print("[green]rclone installed via scoop[/green]")
return
except RcloneInstallError:
console.print("[yellow]scoop installation failed[/yellow]")
# No package manager available
raise RcloneInstallError(
"Could not install rclone automatically. Please install a package manager "
"(winget, chocolatey, or scoop) or install rclone manually from https://rclone.org/downloads/"
# No package manager available - provide detailed instructions
error_msg = (
"Could not install rclone automatically.\n\n"
"Windows requires a package manager to install rclone. Options:\n\n"
"1. Install winget (recommended, built into Windows 11):\n"
" - Windows 11: Already installed\n"
" - Windows 10: Install 'App Installer' from Microsoft Store\n"
" - Then run: bm cloud setup\n\n"
"2. Install chocolatey:\n"
" - Visit: https://chocolatey.org/install\n"
" - Then run: bm cloud setup\n\n"
"3. Install scoop:\n"
" - Visit: https://scoop.sh\n"
" - Then run: bm cloud setup\n\n"
"4. Manual installation:\n"
" - Download from: https://rclone.org/downloads/\n"
" - Extract and add to PATH\n"
)
raise RcloneInstallError(error_msg)
def install_rclone(platform_override: Optional[str] = None) -> None:
@@ -165,6 +188,7 @@ def install_rclone(platform_override: Optional[str] = None) -> None:
install_rclone_linux()
elif platform_name == "windows":
install_rclone_windows()
refresh_windows_path()
else:
raise RcloneInstallError(f"Unsupported platform: {platform_name}")
@@ -172,7 +196,7 @@ def install_rclone(platform_override: Optional[str] = None) -> None:
if not is_rclone_installed():
raise RcloneInstallError("rclone installation completed but command not found in PATH")
console.print("[green]rclone installation completed successfully[/green]")
console.print("[green]rclone installation completed successfully[/green]")
except RcloneInstallError:
raise
@@ -180,6 +204,47 @@ def install_rclone(platform_override: Optional[str] = None) -> None:
raise RcloneInstallError(f"Unexpected error during installation: {e}") from e
def refresh_windows_path() -> None:
"""Refresh the Windows PATH environment variable for the current session."""
if platform.system().lower() != "windows":
return
# Importing here after performing platform detection. Also note that we have to ignore pylance/pyright
# warnings about winreg attributes so that "errors" don't appear on non-Windows platforms.
import winreg
user_key_path = r"Environment"
system_key_path = r"System\CurrentControlSet\Control\Session Manager\Environment"
new_path = ""
# Read user PATH
try:
reg_key = winreg.OpenKey(winreg.HKEY_CURRENT_USER, user_key_path, 0, winreg.KEY_READ) # type: ignore[reportAttributeAccessIssue]
user_path, _ = winreg.QueryValueEx(reg_key, "PATH") # type: ignore[reportAttributeAccessIssue]
winreg.CloseKey(reg_key) # type: ignore[reportAttributeAccessIssue]
except Exception:
user_path = ""
# Read system PATH
try:
reg_key = winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE, system_key_path, 0, winreg.KEY_READ) # type: ignore[reportAttributeAccessIssue]
system_path, _ = winreg.QueryValueEx(reg_key, "PATH") # type: ignore[reportAttributeAccessIssue]
winreg.CloseKey(reg_key) # type: ignore[reportAttributeAccessIssue]
except Exception:
system_path = ""
# Merge user and system PATHs (system first, then user)
if system_path and user_path:
new_path = system_path + ";" + user_path
elif system_path:
new_path = system_path
elif user_path:
new_path = user_path
if new_path:
os.environ["PATH"] = new_path
def get_rclone_version() -> Optional[str]:
"""Get the installed rclone version."""
if not is_rclone_installed():
@@ -0,0 +1,159 @@
"""Restore CLI commands for Basic Memory Cloud.
SPEC-29 Phase 3: CLI commands for restoring files from Tigris bucket snapshots.
"""
import asyncio
import typer
from rich.console import Console
from basic_memory.cli.app import cloud_app
from basic_memory.cli.commands.cloud.api_client import (
CloudAPIError,
SubscriptionRequiredError,
make_api_request,
)
from basic_memory.cli.commands.cloud.schemas import BucketSnapshotBrowseResponse
from basic_memory.config import ConfigManager
console = Console()
@cloud_app.command("restore")
def restore(
path: str = typer.Argument(
...,
help="Path to restore (file or folder, e.g., 'notes/project.md' or 'research/')",
),
snapshot_id: str = typer.Option(
...,
"--snapshot",
"-s",
help="ID of the snapshot to restore from",
),
force: bool = typer.Option(
False,
"--force",
"-f",
help="Skip confirmation prompt",
),
) -> None:
"""Restore a file or folder from a snapshot.
This command restores files from a previous snapshot to the current bucket.
The restored files will overwrite any existing files at the same path.
Examples:
bm cloud restore notes/project.md --snapshot abc123
bm cloud restore research/ --snapshot abc123
bm cloud restore notes/project.md --snapshot abc123 --force
"""
async def _restore():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
# Normalize path - remove leading slash if present
normalized_path = path.lstrip("/")
if not force:
# Show what will be restored
console.print(f"[blue]Preparing to restore from snapshot {snapshot_id}[/blue]")
console.print(f" Path: {normalized_path}")
# Try to browse the snapshot to show what files will be affected
try:
browse_url = f"{host_url}/api/bucket-snapshots/{snapshot_id}/browse"
if normalized_path:
browse_url += f"?prefix={normalized_path}"
response = await make_api_request(
method="GET",
url=browse_url,
)
browse_response = BucketSnapshotBrowseResponse.model_validate(response.json())
if browse_response.files:
if len(browse_response.files) <= 10:
console.print("\n Files to restore:")
for file_info in browse_response.files:
console.print(f" - {file_info.key}")
else:
console.print(
f"\n {len(browse_response.files)} files will be restored"
)
console.print(" First 5 files:")
for file_info in browse_response.files[:5]:
console.print(f" - {file_info.key}")
console.print(f" ... and {len(browse_response.files) - 5} more")
else:
console.print(
f"\n[yellow]No files found matching '{normalized_path}' "
f"in snapshot[/yellow]"
)
raise typer.Exit(0)
except CloudAPIError as browse_error:
if browse_error.status_code == 404:
console.print(f"[red]Snapshot not found: {snapshot_id}[/red]")
raise typer.Exit(1)
# If browse fails for other reasons, proceed with confirmation anyway
pass
console.print(
"\n[yellow]Warning: Restored files will overwrite existing files![/yellow]"
)
confirmed = typer.confirm("\nProceed with restore?")
if not confirmed:
console.print("[yellow]Restore cancelled[/yellow]")
raise typer.Exit(0)
console.print(f"[blue]Restoring from snapshot {snapshot_id}...[/blue]")
response = await make_api_request(
method="POST",
url=f"{host_url}/api/bucket-snapshots/{snapshot_id}/restore",
json_data={"path": normalized_path},
)
data = response.json()
restored_files = data.get("restored", [])
returned_snapshot_id = data.get("snapshot_id", snapshot_id)
if restored_files:
console.print(f"[green]Successfully restored {len(restored_files)} file(s)[/green]")
if len(restored_files) <= 10:
for file_path in restored_files:
console.print(f" - {file_path}")
else:
console.print(" First 5 restored files:")
for file_path in restored_files[:5]:
console.print(f" - {file_path}")
console.print(f" ... and {len(restored_files) - 5} more")
console.print(f"\n[dim]Snapshot ID: {returned_snapshot_id}[/dim]")
else:
console.print("[yellow]No files were restored[/yellow]")
console.print(f"[dim]No files matching '{normalized_path}' found in snapshot[/dim]")
except typer.Exit:
# Re-raise typer.Exit without modification - it's used for clean exits
raise
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
if e.status_code == 404:
console.print(f"[red]Snapshot not found: {snapshot_id}[/red]")
else:
console.print(f"[red]Failed to restore: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_restore())
@@ -0,0 +1,55 @@
"""Pydantic schemas for Basic Memory Cloud API responses.
These schemas mirror the API response models from basic-memory-cloud
for type-safe parsing of API responses in CLI commands.
"""
from datetime import datetime
from uuid import UUID
from pydantic import BaseModel
class BucketSnapshotFileInfo(BaseModel):
"""File info from snapshot browse response."""
key: str
size: int
last_modified: datetime
etag: str | None = None
class BucketSnapshotBrowseResponse(BaseModel):
"""Response from browsing snapshot contents."""
files: list[BucketSnapshotFileInfo]
prefix: str
snapshot_version: str
class BucketSnapshotResponse(BaseModel):
"""Response model for bucket snapshot data."""
id: UUID
bucket_name: str
snapshot_version: str
name: str
description: str | None
auto: bool
created_at: datetime
created_by: UUID | None = None
class BucketSnapshotListResponse(BaseModel):
"""Response from listing bucket snapshots."""
snapshots: list[BucketSnapshotResponse]
total: int
class BucketSnapshotRestoreResponse(BaseModel):
"""Response from restore operation."""
restored: list[str]
snapshot_version: str
snapshot_id: UUID
@@ -0,0 +1,370 @@
"""Snapshot CLI commands for Basic Memory Cloud.
SPEC-29 Phase 3: CLI commands for managing Tigris bucket snapshots.
"""
import asyncio
from datetime import datetime
from typing import Optional
import typer
from rich.console import Console
from rich.table import Table
from basic_memory.cli.commands.cloud.api_client import (
CloudAPIError,
SubscriptionRequiredError,
make_api_request,
)
from basic_memory.cli.commands.cloud.schemas import BucketSnapshotBrowseResponse
from basic_memory.config import ConfigManager
console = Console()
snapshot_app = typer.Typer(help="Manage bucket snapshots")
def _format_timestamp(iso_timestamp: str) -> str:
"""Format ISO timestamp to a human-readable format."""
try:
dt = datetime.fromisoformat(iso_timestamp.replace("Z", "+00:00"))
return dt.strftime("%Y-%m-%d %H:%M:%S")
except (ValueError, AttributeError):
return iso_timestamp
@snapshot_app.command("create")
def create(
description: str = typer.Argument(
...,
help="Description for the snapshot",
),
) -> None:
"""Create a new bucket snapshot.
Examples:
bm cloud snapshot create "before major refactor"
bm cloud snapshot create "daily backup"
"""
async def _create():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
console.print("[blue]Creating snapshot...[/blue]")
response = await make_api_request(
method="POST",
url=f"{host_url}/api/bucket-snapshots",
json_data={"description": description},
)
data = response.json()
snapshot_id = data.get("id", "unknown")
snapshot_version = data.get("snapshot_version", "unknown")
created_at = _format_timestamp(data.get("created_at", ""))
console.print("[green]Snapshot created successfully[/green]")
console.print(f" ID: {snapshot_id}")
console.print(f" Version: {snapshot_version}")
console.print(f" Created: {created_at}")
console.print(f" Description: {description}")
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
console.print(f"[red]Failed to create snapshot: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_create())
@snapshot_app.command("list")
def list_snapshots(
limit: int = typer.Option(
10,
"--limit",
"-l",
help="Maximum number of snapshots to display",
),
) -> None:
"""List all bucket snapshots.
Examples:
bm cloud snapshot list
bm cloud snapshot list --limit 20
"""
async def _list():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
console.print("[blue]Fetching snapshots...[/blue]")
response = await make_api_request(
method="GET",
url=f"{host_url}/api/bucket-snapshots",
)
data = response.json()
snapshots = data.get("snapshots", [])
total = data.get("total", len(snapshots))
if not snapshots:
console.print("[yellow]No snapshots found[/yellow]")
console.print(
'\n[dim]Create a snapshot with: bm cloud snapshot create "description"[/dim]'
)
return
# Create a table for displaying snapshots
table = Table(title=f"Bucket Snapshots ({total} total)")
table.add_column("ID", style="cyan", no_wrap=True)
table.add_column("Description", style="white")
table.add_column("Auto", style="dim")
table.add_column("Created", style="green")
for snapshot in snapshots[:limit]:
snapshot_id = snapshot.get("id", "unknown")
desc = snapshot.get("description") or snapshot.get("name", "-")
auto = "yes" if snapshot.get("auto", False) else "no"
created_at = _format_timestamp(snapshot.get("created_at", ""))
table.add_row(snapshot_id, desc, auto, created_at)
console.print(table)
if total > limit:
console.print(
f"\n[dim]Showing {limit} of {total} snapshots. Use --limit to see more.[/dim]"
)
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
console.print(f"[red]Failed to list snapshots: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_list())
@snapshot_app.command("delete")
def delete(
snapshot_id: str = typer.Argument(
...,
help="The ID of the snapshot to delete",
),
force: bool = typer.Option(
False,
"--force",
"-f",
help="Skip confirmation prompt",
),
) -> None:
"""Delete a bucket snapshot.
Examples:
bm cloud snapshot delete abc123
bm cloud snapshot delete abc123 --force
"""
async def _delete():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
if not force:
# Fetch snapshot details first to show what will be deleted
console.print("[blue]Fetching snapshot details...[/blue]")
try:
response = await make_api_request(
method="GET",
url=f"{host_url}/api/bucket-snapshots/{snapshot_id}",
)
data = response.json()
desc = data.get("description") or data.get("name", "unnamed")
created_at = _format_timestamp(data.get("created_at", ""))
console.print("\nSnapshot to delete:")
console.print(f" ID: {snapshot_id}")
console.print(f" Description: {desc}")
console.print(f" Created: {created_at}")
except CloudAPIError:
# If we can't fetch details, proceed with confirmation anyway
pass
confirmed = typer.confirm("\nAre you sure you want to delete this snapshot?")
if not confirmed:
console.print("[yellow]Deletion cancelled[/yellow]")
raise typer.Exit(0)
console.print("[blue]Deleting snapshot...[/blue]")
await make_api_request(
method="DELETE",
url=f"{host_url}/api/bucket-snapshots/{snapshot_id}",
)
console.print(f"[green]Snapshot {snapshot_id} deleted successfully[/green]")
except typer.Exit:
# Re-raise typer.Exit without modification - it's used for clean exits
raise
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
if e.status_code == 404:
console.print(f"[red]Snapshot not found: {snapshot_id}[/red]")
else:
console.print(f"[red]Failed to delete snapshot: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_delete())
@snapshot_app.command("show")
def show(
snapshot_id: str = typer.Argument(
...,
help="The ID of the snapshot to show",
),
) -> None:
"""Show details of a specific snapshot.
Examples:
bm cloud snapshot show abc123
"""
async def _show():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
response = await make_api_request(
method="GET",
url=f"{host_url}/api/bucket-snapshots/{snapshot_id}",
)
data = response.json()
console.print("[bold blue]Snapshot Details[/bold blue]")
console.print(f" ID: {data.get('id', 'unknown')}")
console.print(f" Bucket: {data.get('bucket_name', 'unknown')}")
console.print(f" Version: {data.get('snapshot_version', 'unknown')}")
console.print(f" Name: {data.get('name', '-')}")
console.print(f" Description: {data.get('description') or '-'}")
console.print(f" Auto: {'yes' if data.get('auto', False) else 'no'}")
console.print(f" Created: {_format_timestamp(data.get('created_at', ''))}")
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
if e.status_code == 404:
console.print(f"[red]Snapshot not found: {snapshot_id}[/red]")
else:
console.print(f"[red]Failed to get snapshot details: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_show())
@snapshot_app.command("browse")
def browse(
snapshot_id: str = typer.Argument(
...,
help="The ID of the snapshot to browse",
),
prefix: Optional[str] = typer.Option(
None,
"--prefix",
"-p",
help="Filter files by path prefix (e.g., 'notes/')",
),
) -> None:
"""Browse contents of a snapshot.
Examples:
bm cloud snapshot browse abc123
bm cloud snapshot browse abc123 --prefix notes/
"""
async def _browse():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
url = f"{host_url}/api/bucket-snapshots/{snapshot_id}/browse"
if prefix:
url += f"?prefix={prefix}"
response = await make_api_request(
method="GET",
url=url,
)
browse_response = BucketSnapshotBrowseResponse.model_validate(response.json())
if not browse_response.files:
if prefix:
console.print(f"[yellow]No files found with prefix '{prefix}'[/yellow]")
else:
console.print("[yellow]No files found in snapshot[/yellow]")
return
console.print(
f"[bold blue]Snapshot Contents ({len(browse_response.files)} files)[/bold blue]"
)
for file_info in browse_response.files:
size_kb = file_info.size // 1024
console.print(f" {file_info.key} ({size_kb} KB)")
console.print(
f"\n[dim]Use 'bm cloud restore <path> --snapshot {snapshot_id}' "
f"to restore files[/dim]"
)
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
if e.status_code == 404:
console.print(f"[red]Snapshot not found: {snapshot_id}[/red]")
else:
console.print(f"[red]Failed to browse snapshot: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_browse())
+139 -27
View File
@@ -2,6 +2,8 @@
import os
from pathlib import Path
from contextlib import AbstractAsyncContextManager
from typing import Callable
import aiofiles
import httpx
@@ -10,14 +12,29 @@ from basic_memory.ignore_utils import load_gitignore_patterns, should_ignore_pat
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.tools.utils import call_put
# Archive file extensions that should be skipped during upload
ARCHIVE_EXTENSIONS = {".zip", ".tar", ".gz", ".bz2", ".xz", ".7z", ".rar", ".tgz", ".tbz2"}
async def upload_path(local_path: Path, project_name: str) -> bool:
async def upload_path(
local_path: Path,
project_name: str,
verbose: bool = False,
use_gitignore: bool = True,
dry_run: bool = False,
*,
client_cm_factory: Callable[[], AbstractAsyncContextManager[httpx.AsyncClient]] | None = None,
put_func=call_put,
) -> bool:
"""
Upload a file or directory to cloud project via WebDAV.
Args:
local_path: Path to local file or directory
project_name: Name of cloud project (destination)
verbose: Show detailed information about filtering and upload
use_gitignore: If False, skip .gitignore patterns (still use .bmignore)
dry_run: If True, show what would be uploaded without uploading
Returns:
True if upload succeeded, False otherwise
@@ -34,35 +51,78 @@ async def upload_path(local_path: Path, project_name: str) -> bool:
# Get files to upload
if local_path.is_file():
files_to_upload = [(local_path, local_path.name)]
if verbose:
print(f"Uploading single file: {local_path.name}")
else:
files_to_upload = _get_files_to_upload(local_path)
files_to_upload = _get_files_to_upload(local_path, verbose, use_gitignore)
if not files_to_upload:
print("No files found to upload")
if verbose:
print(
"\nTip: Use --verbose to see which files are being filtered, "
"or --no-gitignore to skip .gitignore patterns"
)
return True
print(f"Found {len(files_to_upload)} file(s) to upload")
# Upload files using httpx
total_bytes = 0
# Calculate total size
total_bytes = sum(file_path.stat().st_size for file_path, _ in files_to_upload)
skipped_count = 0
async with get_client() as client:
for i, (file_path, relative_path) in enumerate(files_to_upload, 1):
# Build remote path: /webdav/{project_name}/{relative_path}
remote_path = f"/webdav/{project_name}/{relative_path}"
print(f"Uploading {relative_path} ({i}/{len(files_to_upload)})")
# If dry run, just show what would be uploaded
if dry_run:
print("\nFiles that would be uploaded:")
for file_path, relative_path in files_to_upload:
# Skip archive files
if _is_archive_file(file_path):
print(f" [SKIP] {relative_path} (archive file)")
skipped_count += 1
continue
# Read file content asynchronously
async with aiofiles.open(file_path, "rb") as f:
content = await f.read()
size = file_path.stat().st_size
if size < 1024:
size_str = f"{size} bytes"
elif size < 1024 * 1024:
size_str = f"{size / 1024:.1f} KB"
else:
size_str = f"{size / (1024 * 1024):.1f} MB"
print(f" {relative_path} ({size_str})")
else:
# Upload files using httpx.
# Allow injection for tests (MockTransport) while keeping production default.
cm_factory = client_cm_factory or get_client
async with cm_factory() as client:
for i, (file_path, relative_path) in enumerate(files_to_upload, 1):
# Skip archive files (zip, tar, gz, etc.)
if _is_archive_file(file_path):
print(
f"Skipping archive file: {relative_path} ({i}/{len(files_to_upload)})"
)
skipped_count += 1
continue
# Upload via HTTP PUT to WebDAV endpoint
response = await call_put(client, remote_path, content=content)
response.raise_for_status()
# Build remote path: /webdav/{project_name}/{relative_path}
remote_path = f"/webdav/{project_name}/{relative_path}"
print(f"Uploading {relative_path} ({i}/{len(files_to_upload)})")
total_bytes += file_path.stat().st_size
# Get file modification time
file_stat = file_path.stat()
mtime = int(file_stat.st_mtime)
# Format size based on magnitude
# Read file content asynchronously
async with aiofiles.open(file_path, "rb") as f:
content = await f.read()
# Upload via HTTP PUT to WebDAV endpoint with mtime header
# Using X-OC-Mtime (ownCloud/Nextcloud standard)
response = await put_func(
client, remote_path, content=content, headers={"X-OC-Mtime": str(mtime)}
)
response.raise_for_status()
# Format total size based on magnitude
if total_bytes < 1024:
size_str = f"{total_bytes} bytes"
elif total_bytes < 1024 * 1024:
@@ -70,7 +130,16 @@ async def upload_path(local_path: Path, project_name: str) -> bool:
else:
size_str = f"{total_bytes / (1024 * 1024):.1f} MB"
print(f"✓ Upload complete: {len(files_to_upload)} file(s) ({size_str})")
uploaded_count = len(files_to_upload) - skipped_count
if dry_run:
print(f"\nTotal: {uploaded_count} file(s) ({size_str})")
if skipped_count > 0:
print(f" Would skip {skipped_count} archive file(s)")
else:
print(f"✓ Upload complete: {uploaded_count} file(s) ({size_str})")
if skipped_count > 0:
print(f" Skipped {skipped_count} archive file(s)")
return True
except httpx.HTTPStatusError as e:
@@ -81,22 +150,51 @@ async def upload_path(local_path: Path, project_name: str) -> bool:
return False
def _get_files_to_upload(directory: Path) -> list[tuple[Path, str]]:
def _is_archive_file(file_path: Path) -> bool:
"""
Check if a file is an archive file based on its extension.
Args:
file_path: Path to the file to check
Returns:
True if file is an archive, False otherwise
"""
return file_path.suffix.lower() in ARCHIVE_EXTENSIONS
def _get_files_to_upload(
directory: Path, verbose: bool = False, use_gitignore: bool = True
) -> list[tuple[Path, str]]:
"""
Get list of files to upload from directory.
Uses .bmignore and .gitignore patterns for filtering.
Uses .bmignore and optionally .gitignore patterns for filtering.
Args:
directory: Directory to scan
verbose: Show detailed filtering information
use_gitignore: If False, skip .gitignore patterns (still use .bmignore)
Returns:
List of (absolute_path, relative_path) tuples
"""
files = []
ignored_files = []
# Load ignore patterns from .bmignore and .gitignore
ignore_patterns = load_gitignore_patterns(directory)
# Load ignore patterns from .bmignore and optionally .gitignore
ignore_patterns = load_gitignore_patterns(directory, use_gitignore=use_gitignore)
if verbose:
gitignore_path = directory / ".gitignore"
gitignore_exists = gitignore_path.exists() and use_gitignore
print(f"\nScanning directory: {directory}")
print("Using .bmignore: Yes")
print(f"Using .gitignore: {'Yes' if gitignore_exists else 'No'}")
print(f"Ignore patterns loaded: {len(ignore_patterns)}")
if ignore_patterns and len(ignore_patterns) <= 20:
print(f"Patterns: {', '.join(sorted(ignore_patterns))}")
print()
# Walk through directory
for root, dirs, filenames in os.walk(directory):
@@ -106,7 +204,11 @@ def _get_files_to_upload(directory: Path) -> list[tuple[Path, str]]:
filtered_dirs = []
for d in dirs:
dir_path = root_path / d
if not should_ignore_path(dir_path, directory, ignore_patterns):
if should_ignore_path(dir_path, directory, ignore_patterns):
if verbose:
rel_path = dir_path.relative_to(directory)
print(f" [IGNORED DIR] {rel_path}/")
else:
filtered_dirs.append(d)
dirs[:] = filtered_dirs
@@ -114,15 +216,25 @@ def _get_files_to_upload(directory: Path) -> list[tuple[Path, str]]:
for filename in filenames:
file_path = root_path / filename
# Calculate relative path for display/remote
rel_path = file_path.relative_to(directory)
remote_path = str(rel_path).replace("\\", "/")
# Check if file should be ignored
if should_ignore_path(file_path, directory, ignore_patterns):
ignored_files.append(remote_path)
if verbose:
print(f" [IGNORED] {remote_path}")
continue
# Calculate relative path for remote
rel_path = file_path.relative_to(directory)
# Use forward slashes for WebDAV paths
remote_path = str(rel_path).replace("\\", "/")
if verbose:
print(f" [INCLUDE] {remote_path}")
files.append((file_path, remote_path))
if verbose:
print("\nSummary:")
print(f" Files to upload: {len(files)}")
print(f" Files ignored: {len(ignored_files)}")
return files
@@ -1,12 +1,12 @@
"""Upload CLI commands for basic-memory projects."""
import asyncio
from pathlib import Path
import typer
from rich.console import Console
from basic_memory.cli.app import cloud_app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.cli.commands.cloud.cloud_utils import (
create_cloud_project,
project_exists,
@@ -43,6 +43,22 @@ def upload(
"--sync/--no-sync",
help="Sync project after upload (default: true)",
),
verbose: bool = typer.Option(
False,
"--verbose",
"-v",
help="Show detailed information about file filtering and upload",
),
no_gitignore: bool = typer.Option(
False,
"--no-gitignore",
help="Skip .gitignore patterns (still respects .bmignore)",
),
dry_run: bool = typer.Option(
False,
"--dry-run",
help="Show what would be uploaded without actually uploading",
),
) -> None:
"""Upload local files or directories to cloud project via WebDAV.
@@ -50,6 +66,9 @@ def upload(
bm cloud upload ~/my-notes --project research
bm cloud upload notes.md --project research --create-project
bm cloud upload ~/docs --project work --no-sync
bm cloud upload ./history --project proto --verbose
bm cloud upload ./notes --project work --no-gitignore
bm cloud upload ./files --project test --dry-run
"""
async def _upload():
@@ -59,7 +78,7 @@ def upload(
console.print(f"[blue]Creating cloud project '{project}'...[/blue]")
try:
await create_cloud_project(project)
console.print(f"[green]Created project '{project}'[/green]")
console.print(f"[green]Created project '{project}'[/green]")
except Exception as e:
console.print(f"[red]Failed to create project: {e}[/red]")
raise typer.Exit(1)
@@ -72,22 +91,34 @@ def upload(
)
raise typer.Exit(1)
# Perform upload
console.print(f"[blue]Uploading {path} to project '{project}'...[/blue]")
success = await upload_path(path, project)
# Perform upload (or dry run)
if dry_run:
console.print(
f"[yellow]DRY RUN: Showing what would be uploaded to '{project}'[/yellow]"
)
else:
console.print(f"[blue]Uploading {path} to project '{project}'...[/blue]")
success = await upload_path(
path, project, verbose=verbose, use_gitignore=not no_gitignore, dry_run=dry_run
)
if not success:
console.print("[red]Upload failed[/red]")
raise typer.Exit(1)
console.print(f"[green]✅ Successfully uploaded to '{project}'[/green]")
if dry_run:
console.print("[yellow]DRY RUN complete - no files were uploaded[/yellow]")
else:
console.print(f"[green]Successfully uploaded to '{project}'[/green]")
# Sync project if requested
if sync:
# Sync project if requested (skip on dry run)
# Force full scan after bisync to ensure database is up-to-date with synced files
if sync and not dry_run:
console.print(f"[blue]Syncing project '{project}'...[/blue]")
try:
await sync_project(project)
await sync_project(project, force_full=True)
except Exception as e:
console.print(f"[yellow]Warning: Sync failed: {e}[/yellow]")
console.print("[dim]Files uploaded but may not be indexed yet[/dim]")
asyncio.run(_upload())
run_with_cleanup(_upload())
+63 -7
View File
@@ -1,12 +1,14 @@
"""utility functions for commands"""
from typing import Optional
import asyncio
from typing import Optional, TypeVar, Coroutine, Any
from mcp.server.fastmcp.exceptions import ToolError
import typer
from rich.console import Console
from basic_memory import db
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.tools.utils import call_post, call_get
@@ -15,18 +17,72 @@ from basic_memory.schemas import ProjectInfoResponse
console = Console()
T = TypeVar("T")
async def run_sync(project: Optional[str] = None):
"""Run sync operation via API endpoint."""
def run_with_cleanup(coro: Coroutine[Any, Any, T]) -> T:
"""Run an async coroutine with proper database cleanup.
This helper ensures database connections are cleaned up before the
event loop closes, preventing process hangs in CLI commands.
Args:
coro: The coroutine to run
Returns:
The result of the coroutine
"""
async def _with_cleanup() -> T:
try:
return await coro
finally:
await db.shutdown_db()
return asyncio.run(_with_cleanup())
async def run_sync(
project: Optional[str] = None,
force_full: bool = False,
run_in_background: bool = True,
):
"""Run sync operation via API endpoint.
Args:
project: Optional project name
force_full: If True, force a full scan bypassing watermark optimization
run_in_background: If True, return immediately; if False, wait for completion
"""
try:
async with get_client() as client:
project_item = await get_active_project(client, project, None)
response = await call_post(client, f"{project_item.project_url}/project/sync")
url = f"{project_item.project_url}/project/sync"
params = []
if force_full:
params.append("force_full=true")
if not run_in_background:
params.append("run_in_background=false")
if params:
url += "?" + "&".join(params)
response = await call_post(client, url)
data = response.json()
console.print(f"[green]✓ {data['message']}[/green]")
# Background mode returns {"message": "..."}, foreground returns SyncReportResponse
if "message" in data:
console.print(f"[green]{data['message']}[/green]")
else:
# Foreground mode - show summary of sync results
total = data.get("total", 0)
new_count = len(data.get("new", []))
modified_count = len(data.get("modified", []))
deleted_count = len(data.get("deleted", []))
console.print(
f"[green]Synced {total} files[/green] "
f"(new: {new_count}, modified: {modified_count}, deleted: {deleted_count})"
)
except (ToolError, ValueError) as e:
console.print(f"[red]Sync failed: {e}[/red]")
console.print(f"[red]Sync failed: {e}[/red]")
raise typer.Exit(1)
@@ -39,5 +95,5 @@ async def get_project_info(project: str):
response = await call_get(client, f"{project_item.project_url}/project/info")
return ProjectInfoResponse.model_validate(response.json())
except (ToolError, ValueError) as e:
console.print(f"[red]Sync failed: {e}[/red]")
console.print(f"[red]Sync failed: {e}[/red]")
raise typer.Exit(1)
+81 -20
View File
@@ -1,13 +1,50 @@
"""Database management commands."""
import asyncio
from pathlib import Path
import typer
from loguru import logger
from rich.console import Console
from sqlalchemy.exc import OperationalError
from basic_memory import db
from basic_memory.cli.app import app
from basic_memory.config import ConfigManager, BasicMemoryConfig, save_basic_memory_config
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.config import ConfigManager
from basic_memory.repository import ProjectRepository
from basic_memory.services.initialization import reconcile_projects_with_config
from basic_memory.sync.sync_service import get_sync_service
console = Console()
async def _reindex_projects(app_config):
"""Reindex all projects in a single async context.
This ensures all database operations use the same event loop,
and proper cleanup happens when the function completes.
"""
try:
await reconcile_projects_with_config(app_config)
# Get database session (migrations already run if needed)
_, session_maker = await db.get_or_create_db(
db_path=app_config.database_path,
db_type=db.DatabaseType.FILESYSTEM,
)
project_repository = ProjectRepository(session_maker)
projects = await project_repository.get_active_projects()
for project in projects:
console.print(f" Indexing [cyan]{project.name}[/cyan]...")
logger.info(f"Starting sync for project: {project.name}")
sync_service = await get_sync_service(project)
sync_dir = Path(project.path)
await sync_service.sync(sync_dir, project_name=project.name)
logger.info(f"Sync completed for project: {project.name}")
finally:
# Clean up database connections before event loop closes
await db.shutdown_db()
@app.command()
@@ -15,30 +52,54 @@ def reset(
reindex: bool = typer.Option(False, "--reindex", help="Rebuild db index from filesystem"),
): # pragma: no cover
"""Reset database (drop all tables and recreate)."""
if typer.confirm("This will delete all data in your db. Are you sure?"):
console.print(
"[yellow]Note:[/yellow] This only deletes the index database. "
"Your markdown note files will not be affected.\n"
"Use [green]bm reset --reindex[/green] to automatically rebuild the index afterward."
)
if typer.confirm("Reset the database index?"):
logger.info("Resetting database...")
config_manager = ConfigManager()
app_config = config_manager.config
# Get database path
db_path = app_config.app_database_path
# Delete the database file if it exists
if db_path.exists():
db_path.unlink()
logger.info(f"Database file deleted: {db_path}")
# Delete the database file and WAL files if they exist
for suffix in ["", "-shm", "-wal"]:
path = db_path.parent / f"{db_path.name}{suffix}"
if path.exists():
try:
path.unlink()
logger.info(f"Deleted: {path}")
except OSError as e:
console.print(
f"[red]Error:[/red] Cannot delete {path.name}: {e}\n"
"The database may be in use by another process (e.g., MCP server).\n"
"Please close Claude Desktop or any other Basic Memory clients and try again."
)
raise typer.Exit(1)
# Reset project configuration
config = BasicMemoryConfig()
save_basic_memory_config(config_manager.config_file, config)
logger.info("Project configuration reset to default")
# Create a new empty database
asyncio.run(db.run_migrations(app_config))
logger.info("Database reset complete")
# Create a new empty database (preserves project configuration)
try:
run_with_cleanup(db.run_migrations(app_config))
except OperationalError as e:
if "disk I/O error" in str(e) or "database is locked" in str(e):
console.print(
"[red]Error:[/red] Cannot access database. "
"It may be in use by another process (e.g., MCP server).\n"
"Please close Claude Desktop or any other Basic Memory clients and try again."
)
raise typer.Exit(1)
raise
console.print("[green]Database reset complete[/green]")
if reindex:
# Import and run sync
from basic_memory.cli.commands.sync import sync
logger.info("Rebuilding search index from filesystem...")
sync(watch=False) # pyright: ignore
projects = list(app_config.projects)
if not projects:
console.print("[yellow]No projects configured. Skipping reindex.[/yellow]")
else:
console.print(f"Rebuilding search index for {len(projects)} project(s)...")
# Note: _reindex_projects has its own cleanup, but run_with_cleanup
# ensures db.shutdown_db() is called even if _reindex_projects changes
run_with_cleanup(_reindex_projects(app_config))
console.print("[green]Reindex complete[/green]")
+198
View File
@@ -0,0 +1,198 @@
"""Format command for basic-memory CLI."""
from pathlib import Path
from typing import Annotated, Optional
import typer
from loguru import logger
from rich.console import Console
from rich.progress import Progress, SpinnerColumn, TextColumn
from basic_memory.cli.app import app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.config import ConfigManager, get_project_config
from basic_memory.file_utils import format_file
console = Console()
def is_markdown_extension(path: Path) -> bool:
"""Check if file has a markdown extension."""
return path.suffix.lower() in (".md", ".markdown")
async def format_single_file(file_path: Path, app_config) -> tuple[Path, bool, Optional[str]]:
"""Format a single file.
Returns:
Tuple of (path, success, error_message)
"""
try:
result = await format_file(
file_path, app_config, is_markdown=is_markdown_extension(file_path)
)
if result is not None:
return (file_path, True, None)
else:
return (file_path, False, "No formatter configured or formatting skipped")
except Exception as e:
return (file_path, False, str(e))
async def format_files(
paths: list[Path], app_config, show_progress: bool = True
) -> tuple[int, int, list[tuple[Path, str]]]:
"""Format multiple files.
Returns:
Tuple of (formatted_count, skipped_count, errors)
"""
formatted = 0
skipped = 0
errors: list[tuple[Path, str]] = []
if show_progress:
with Progress(
SpinnerColumn(),
TextColumn("[progress.description]{task.description}"),
console=console,
) as progress:
task = progress.add_task("Formatting files...", total=len(paths))
for file_path in paths:
path, success, error = await format_single_file(file_path, app_config)
if success:
formatted += 1
elif error and "No formatter configured" not in error:
errors.append((path, error))
else:
skipped += 1
progress.update(task, advance=1)
else:
for file_path in paths:
path, success, error = await format_single_file(file_path, app_config)
if success:
formatted += 1
elif error and "No formatter configured" not in error:
errors.append((path, error))
else:
skipped += 1
return formatted, skipped, errors
async def run_format(
path: Optional[Path] = None,
project: Optional[str] = None,
) -> None:
"""Run the format command."""
app_config = ConfigManager().config
# Check if formatting is enabled
if (
not app_config.format_on_save
and not app_config.formatter_command
and not app_config.formatters
):
console.print(
"[yellow]No formatters configured. Set format_on_save=true and "
"formatter_command or formatters in your config.[/yellow]"
)
console.print(
"\nExample config (~/.basic-memory/config.json):\n"
' "format_on_save": true,\n'
' "formatter_command": "prettier --write {file}"\n'
)
raise typer.Exit(1)
# Temporarily enable format_on_save for this command
# (so format_file actually runs the formatter)
original_format_on_save = app_config.format_on_save
app_config.format_on_save = True
try:
# Determine which files to format
if path:
# Format specific file or directory
if path.is_file():
files = [path]
elif path.is_dir():
# Find all markdown and json files
files = (
list(path.rglob("*.md"))
+ list(path.rglob("*.json"))
+ list(path.rglob("*.canvas"))
)
else:
console.print(f"[red]Path not found: {path}[/red]")
raise typer.Exit(1)
else:
# Format all files in project
project_config = get_project_config(project)
project_path = Path(project_config.home)
if not project_path.exists():
console.print(f"[red]Project path not found: {project_path}[/red]")
raise typer.Exit(1)
# Find all markdown and json files
files = (
list(project_path.rglob("*.md"))
+ list(project_path.rglob("*.json"))
+ list(project_path.rglob("*.canvas"))
)
if not files:
console.print("[yellow]No files found to format.[/yellow]")
return
console.print(f"Found {len(files)} file(s) to format...")
formatted, skipped, errors = await format_files(files, app_config)
# Print summary
console.print()
if formatted > 0:
console.print(f"[green]Formatted: {formatted} file(s)[/green]")
if skipped > 0:
console.print(f"[dim]Skipped: {skipped} file(s) (no formatter for extension)[/dim]")
if errors:
console.print(f"[red]Errors: {len(errors)} file(s)[/red]")
for path, error in errors:
console.print(f" [red]{path}[/red]: {error}")
finally:
# Restore original setting
app_config.format_on_save = original_format_on_save
@app.command()
def format(
path: Annotated[
Optional[Path],
typer.Argument(help="File or directory to format. Defaults to current project."),
] = None,
project: Annotated[
Optional[str],
typer.Option("--project", "-p", help="Project name to format."),
] = None,
) -> None:
"""Format files using configured formatters.
Uses the formatter_command or formatters settings from your config.
By default, formats all .md, .json, and .canvas files in the current project.
Examples:
basic-memory format # Format all files in current project
basic-memory format --project research # Format files in specific project
basic-memory format notes/meeting.md # Format a specific file
basic-memory format notes/ # Format all files in directory
"""
try:
run_with_cleanup(run_format(path, project))
except Exception as e:
if not isinstance(e, typer.Exit):
logger.error(f"Error formatting files: {e}")
console.print(f"[red]Error formatting files: {e}[/red]")
raise typer.Exit(code=1)
raise
+14 -10
View File
@@ -1,15 +1,16 @@
"""Import command for ChatGPT conversations."""
import asyncio
import json
from pathlib import Path
from typing import Annotated
from typing import Annotated, Tuple
import typer
from basic_memory.cli.app import import_app
from basic_memory.config import get_project_config
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.config import ConfigManager, get_project_config
from basic_memory.importers import ChatGPTImporter
from basic_memory.markdown import EntityParser, MarkdownProcessor
from basic_memory.services.file_service import FileService
from loguru import logger
from rich.console import Console
from rich.panel import Panel
@@ -17,11 +18,14 @@ from rich.panel import Panel
console = Console()
async def get_markdown_processor() -> MarkdownProcessor:
"""Get MarkdownProcessor instance."""
async def get_importer_dependencies() -> Tuple[MarkdownProcessor, FileService]:
"""Get MarkdownProcessor and FileService instances for importers."""
config = get_project_config()
app_config = ConfigManager().config
entity_parser = EntityParser(config.home)
return MarkdownProcessor(entity_parser)
markdown_processor = MarkdownProcessor(entity_parser, app_config=app_config)
file_service = FileService(config.home, markdown_processor, app_config=app_config)
return markdown_processor, file_service
@import_app.command(name="chatgpt", help="Import conversations from ChatGPT JSON export.")
@@ -48,18 +52,18 @@ def import_chatgpt(
typer.echo(f"Error: File not found: {conversations_json}", err=True)
raise typer.Exit(1)
# Get markdown processor
markdown_processor = asyncio.run(get_markdown_processor())
# Get importer dependencies
markdown_processor, file_service = run_with_cleanup(get_importer_dependencies())
config = get_project_config()
# Process the file
base_path = config.home / folder
console.print(f"\nImporting chats from {conversations_json}...writing to {base_path}")
# Create importer and run import
importer = ChatGPTImporter(config.home, markdown_processor)
importer = ChatGPTImporter(config.home, markdown_processor, file_service)
with conversations_json.open("r", encoding="utf-8") as file:
json_data = json.load(file)
result = asyncio.run(importer.import_data(json_data, folder))
result = run_with_cleanup(importer.import_data(json_data, folder))
if not result.success: # pragma: no cover
typer.echo(f"Error during import: {result.error_message}", err=True)
@@ -1,15 +1,16 @@
"""Import command for basic-memory CLI to import chat data from conversations2.json format."""
import asyncio
import json
from pathlib import Path
from typing import Annotated
from typing import Annotated, Tuple
import typer
from basic_memory.cli.app import claude_app
from basic_memory.config import get_project_config
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.config import ConfigManager, get_project_config
from basic_memory.importers.claude_conversations_importer import ClaudeConversationsImporter
from basic_memory.markdown import EntityParser, MarkdownProcessor
from basic_memory.services.file_service import FileService
from loguru import logger
from rich.console import Console
from rich.panel import Panel
@@ -17,11 +18,14 @@ from rich.panel import Panel
console = Console()
async def get_markdown_processor() -> MarkdownProcessor:
"""Get MarkdownProcessor instance."""
async def get_importer_dependencies() -> Tuple[MarkdownProcessor, FileService]:
"""Get MarkdownProcessor and FileService instances for importers."""
config = get_project_config()
app_config = ConfigManager().config
entity_parser = EntityParser(config.home)
return MarkdownProcessor(entity_parser)
markdown_processor = MarkdownProcessor(entity_parser, app_config=app_config)
file_service = FileService(config.home, markdown_processor, app_config=app_config)
return markdown_processor, file_service
@claude_app.command(name="conversations", help="Import chat conversations from Claude.ai.")
@@ -49,11 +53,11 @@ def import_claude(
typer.echo(f"Error: File not found: {conversations_json}", err=True)
raise typer.Exit(1)
# Get markdown processor
markdown_processor = asyncio.run(get_markdown_processor())
# Get importer dependencies
markdown_processor, file_service = run_with_cleanup(get_importer_dependencies())
# Create the importer
importer = ClaudeConversationsImporter(config.home, markdown_processor)
importer = ClaudeConversationsImporter(config.home, markdown_processor, file_service)
# Process the file
base_path = config.home / folder
@@ -62,7 +66,7 @@ def import_claude(
# Run the import
with conversations_json.open("r", encoding="utf-8") as file:
json_data = json.load(file)
result = asyncio.run(importer.import_data(json_data, folder))
result = run_with_cleanup(importer.import_data(json_data, folder))
if not result.success: # pragma: no cover
typer.echo(f"Error during import: {result.error_message}", err=True)
@@ -1,15 +1,16 @@
"""Import command for basic-memory CLI to import project data from Claude.ai."""
import asyncio
import json
from pathlib import Path
from typing import Annotated
from typing import Annotated, Tuple
import typer
from basic_memory.cli.app import claude_app
from basic_memory.config import get_project_config
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.config import ConfigManager, get_project_config
from basic_memory.importers.claude_projects_importer import ClaudeProjectsImporter
from basic_memory.markdown import EntityParser, MarkdownProcessor
from basic_memory.services.file_service import FileService
from loguru import logger
from rich.console import Console
from rich.panel import Panel
@@ -17,11 +18,14 @@ from rich.panel import Panel
console = Console()
async def get_markdown_processor() -> MarkdownProcessor:
"""Get MarkdownProcessor instance."""
async def get_importer_dependencies() -> Tuple[MarkdownProcessor, FileService]:
"""Get MarkdownProcessor and FileService instances for importers."""
config = get_project_config()
app_config = ConfigManager().config
entity_parser = EntityParser(config.home)
return MarkdownProcessor(entity_parser)
markdown_processor = MarkdownProcessor(entity_parser, app_config=app_config)
file_service = FileService(config.home, markdown_processor, app_config=app_config)
return markdown_processor, file_service
@claude_app.command(name="projects", help="Import projects from Claude.ai.")
@@ -48,11 +52,11 @@ def import_projects(
typer.echo(f"Error: File not found: {projects_json}", err=True)
raise typer.Exit(1)
# Get markdown processor
markdown_processor = asyncio.run(get_markdown_processor())
# Get importer dependencies
markdown_processor, file_service = run_with_cleanup(get_importer_dependencies())
# Create the importer
importer = ClaudeProjectsImporter(config.home, markdown_processor)
importer = ClaudeProjectsImporter(config.home, markdown_processor, file_service)
# Process the file
base_path = config.home / base_folder if base_folder else config.home
@@ -61,7 +65,7 @@ def import_projects(
# Run the import
with projects_json.open("r", encoding="utf-8") as file:
json_data = json.load(file)
result = asyncio.run(importer.import_data(json_data, base_folder))
result = run_with_cleanup(importer.import_data(json_data, base_folder))
if not result.success: # pragma: no cover
typer.echo(f"Error during import: {result.error_message}", err=True)
@@ -1,15 +1,16 @@
"""Import command for basic-memory CLI to import from JSON memory format."""
import asyncio
import json
from pathlib import Path
from typing import Annotated
from typing import Annotated, Tuple
import typer
from basic_memory.cli.app import import_app
from basic_memory.config import get_project_config
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.config import ConfigManager, get_project_config
from basic_memory.importers.memory_json_importer import MemoryJsonImporter
from basic_memory.markdown import EntityParser, MarkdownProcessor
from basic_memory.services.file_service import FileService
from loguru import logger
from rich.console import Console
from rich.panel import Panel
@@ -17,11 +18,14 @@ from rich.panel import Panel
console = Console()
async def get_markdown_processor() -> MarkdownProcessor:
"""Get MarkdownProcessor instance."""
async def get_importer_dependencies() -> Tuple[MarkdownProcessor, FileService]:
"""Get MarkdownProcessor and FileService instances for importers."""
config = get_project_config()
app_config = ConfigManager().config
entity_parser = EntityParser(config.home)
return MarkdownProcessor(entity_parser)
markdown_processor = MarkdownProcessor(entity_parser, app_config=app_config)
file_service = FileService(config.home, markdown_processor, app_config=app_config)
return markdown_processor, file_service
@import_app.command()
@@ -47,11 +51,11 @@ def memory_json(
config = get_project_config()
try:
# Get markdown processor
markdown_processor = asyncio.run(get_markdown_processor())
# Get importer dependencies
markdown_processor, file_service = run_with_cleanup(get_importer_dependencies())
# Create the importer
importer = MemoryJsonImporter(config.home, markdown_processor)
importer = MemoryJsonImporter(config.home, markdown_processor, file_service)
# Process the file
base_path = config.home if not destination_folder else config.home / destination_folder
@@ -63,7 +67,7 @@ def memory_json(
for line in file:
json_data = json.loads(line)
file_data.append(json_data)
result = asyncio.run(importer.import_data(file_data, destination_folder))
result = run_with_cleanup(importer.import_data(file_data, destination_folder))
if not result.success: # pragma: no cover
typer.echo(f"Error during import: {result.error_message}", err=True)
+8 -26
View File
@@ -1,14 +1,13 @@
"""MCP server command with streamable HTTP transport."""
import asyncio
import os
import typer
from typing import Optional
from basic_memory.cli.app import app
from basic_memory.config import ConfigManager
from basic_memory.config import ConfigManager, init_mcp_logging
# Import mcp instance
# Import mcp instance (has lifespan that handles initialization and file sync)
from basic_memory.mcp.server import mcp as mcp_server # pragma: no cover
# Import mcp tools to register them
@@ -17,8 +16,6 @@ import basic_memory.mcp.tools # noqa: F401 # pragma: no cover
# Import prompts to register them
import basic_memory.mcp.prompts # noqa: F401 # pragma: no cover
from loguru import logger
import threading
from basic_memory.services.initialization import initialize_file_sync
config = ConfigManager().config
@@ -43,7 +40,11 @@ if not config.cloud_mode_enabled:
- stdio: Standard I/O (good for local usage)
- streamable-http: Recommended for web deployments (default)
- sse: Server-Sent Events (for compatibility with existing clients)
Initialization, file sync, and cleanup are handled by the MCP server's lifespan.
"""
# Initialize logging for MCP (file only, stdout breaks protocol)
init_mcp_logging()
# Validate and set project constraint if specified
if project:
@@ -57,27 +58,8 @@ if not config.cloud_mode_enabled:
os.environ["BASIC_MEMORY_MCP_PROJECT"] = project_name
logger.info(f"MCP server constrained to project: {project_name}")
app_config = ConfigManager().config
def run_file_sync():
"""Run file sync in a separate thread with its own event loop."""
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
loop.run_until_complete(initialize_file_sync(app_config))
except Exception as e:
logger.error(f"File sync error: {e}", err=True)
finally:
loop.close()
logger.info(f"Sync changes enabled: {app_config.sync_changes}")
if app_config.sync_changes:
# Start the sync thread
sync_thread = threading.Thread(target=run_file_sync, daemon=True)
sync_thread.start()
logger.info("Started file sync in background")
# Now run the MCP server (blocks)
# Run the MCP server (blocks)
# Lifespan handles: initialization, migrations, file sync, cleanup
logger.info(f"Starting MCP server with {transport.upper()} transport")
if transport == "stdio":
+573 -40
View File
@@ -1,6 +1,5 @@
"""Command module for basic-memory project management."""
import asyncio
import os
from pathlib import Path
@@ -9,21 +8,27 @@ from rich.console import Console
from rich.table import Table
from basic_memory.cli.app import app
from basic_memory.cli.commands.command_utils import get_project_info
from basic_memory.cli.commands.command_utils import get_project_info, run_with_cleanup
from basic_memory.config import ConfigManager
import json
from datetime import datetime
from rich.panel import Panel
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.tools.utils import call_get
from basic_memory.schemas.project_info import ProjectList
from basic_memory.mcp.tools.utils import call_post
from basic_memory.schemas.project_info import ProjectStatusResponse
from basic_memory.mcp.tools.utils import call_delete
from basic_memory.mcp.tools.utils import call_put
from basic_memory.utils import generate_permalink
from basic_memory.mcp.tools.utils import call_patch
from basic_memory.mcp.tools.utils import call_get, call_post, call_delete, call_put, call_patch
from basic_memory.schemas.project_info import ProjectList, ProjectStatusResponse
from basic_memory.utils import generate_permalink, normalize_project_path
# Import rclone commands for project sync
from basic_memory.cli.commands.cloud.rclone_commands import (
SyncProject,
RcloneError,
project_sync,
project_bisync,
project_check,
project_ls,
)
from basic_memory.cli.commands.cloud.bisync_commands import get_mount_info
console = Console()
@@ -50,16 +55,42 @@ def list_projects() -> None:
return ProjectList.model_validate(response.json())
try:
result = asyncio.run(_list_projects())
result = run_with_cleanup(_list_projects())
config = ConfigManager().config
table = Table(title="Basic Memory Projects")
table.add_column("Name", style="cyan")
table.add_column("Path", style="green")
table.add_column("Default", style="magenta")
# Add Local Path column if in cloud mode
if config.cloud_mode_enabled:
table.add_column("Local Path", style="yellow", no_wrap=True, overflow="fold")
# Show Default column in local mode or if default_project_mode is enabled in cloud mode
show_default_column = not config.cloud_mode_enabled or config.default_project_mode
if show_default_column:
table.add_column("Default", style="magenta")
for project in result.projects:
is_default = "" if project.is_default else ""
table.add_row(project.name, format_path(project.path), is_default)
is_default = "[X]" if project.is_default else ""
normalized_path = normalize_project_path(project.path)
# Build row based on mode
row = [project.name, format_path(normalized_path)]
# Add local path if in cloud mode
if config.cloud_mode_enabled:
local_path = ""
if project.name in config.cloud_projects:
local_path = config.cloud_projects[project.name].local_path or ""
local_path = format_path(local_path)
row.append(local_path)
# Add default indicator if showing default column
if show_default_column:
row.append(is_default)
table.add_row(*row)
console.print(table)
except Exception as e:
@@ -73,20 +104,38 @@ def add_project(
path: str = typer.Argument(
None, help="Path to the project directory (required for local mode)"
),
local_path: str = typer.Option(
None, "--local-path", help="Local sync path for cloud mode (optional)"
),
set_default: bool = typer.Option(False, "--default", help="Set as default project"),
) -> None:
"""Add a new project.
For cloud mode: only name is required
For local mode: both name and path are required
Cloud mode examples:\n
bm project add research # No local sync\n
bm project add research --local-path ~/docs # With local sync\n
Local mode example:\n
bm project add research ~/Documents/research
"""
config = ConfigManager().config
# Resolve local sync path early (needed for both cloud and local mode)
local_sync_path: str | None = None
if local_path:
local_sync_path = Path(os.path.abspath(os.path.expanduser(local_path))).as_posix()
if config.cloud_mode_enabled:
# Cloud mode: path not needed (auto-generated from name)
# Cloud mode: path auto-generated from name, local sync is optional
async def _add_project():
async with get_client() as client:
data = {"name": name, "path": generate_permalink(name), "set_default": set_default}
data = {
"name": name,
"path": generate_permalink(name),
"local_sync_path": local_sync_path,
"set_default": set_default,
}
response = await call_post(client, "/projects/projects", json=data)
return ProjectStatusResponse.model_validate(response.json())
else:
@@ -105,39 +154,166 @@ def add_project(
return ProjectStatusResponse.model_validate(response.json())
try:
result = asyncio.run(_add_project())
result = run_with_cleanup(_add_project())
console.print(f"[green]{result.message}[/green]")
# Save local sync path to config if in cloud mode
if config.cloud_mode_enabled and local_sync_path:
from basic_memory.config import CloudProjectConfig
# Create local directory if it doesn't exist
local_dir = Path(local_sync_path)
local_dir.mkdir(parents=True, exist_ok=True)
# Update config with sync path
config.cloud_projects[name] = CloudProjectConfig(
local_path=local_sync_path,
last_sync=None,
bisync_initialized=False,
)
ConfigManager().save_config(config)
console.print(f"\n[green]Local sync path configured: {local_sync_path}[/green]")
console.print("\nNext steps:")
console.print(f" 1. Preview: bm project bisync --name {name} --resync --dry-run")
console.print(f" 2. Sync: bm project bisync --name {name} --resync")
except Exception as e:
console.print(f"[red]Error adding project: {str(e)}[/red]")
raise typer.Exit(1)
# Display usage hint
console.print("\nTo use this project:")
console.print(f" basic-memory --project={name} <command>")
@project_app.command("sync-setup")
def setup_project_sync(
name: str = typer.Argument(..., help="Project name"),
local_path: str = typer.Argument(..., help="Local sync directory"),
) -> None:
"""Configure local sync for an existing cloud project.
Example:
bm project sync-setup research ~/Documents/research
"""
config_manager = ConfigManager()
config = config_manager.config
if not config.cloud_mode_enabled:
console.print("[red]Error: sync-setup only available in cloud mode[/red]")
raise typer.Exit(1)
async def _verify_project_exists():
"""Verify the project exists on cloud by listing all projects."""
async with get_client() as client:
response = await call_get(client, "/projects/projects")
project_list = response.json()
project_names = [p["name"] for p in project_list["projects"]]
if name not in project_names:
raise ValueError(f"Project '{name}' not found on cloud")
return True
try:
# Verify project exists on cloud
run_with_cleanup(_verify_project_exists())
# Resolve and create local path
resolved_path = Path(os.path.abspath(os.path.expanduser(local_path)))
resolved_path.mkdir(parents=True, exist_ok=True)
# Update local config with sync path
from basic_memory.config import CloudProjectConfig
config.cloud_projects[name] = CloudProjectConfig(
local_path=resolved_path.as_posix(),
last_sync=None,
bisync_initialized=False,
)
config_manager.save_config(config)
console.print(f"[green]Sync configured for project '{name}'[/green]")
console.print(f"\nLocal sync path: {resolved_path}")
console.print("\nNext steps:")
console.print(f" 1. Preview: bm project bisync --name {name} --resync --dry-run")
console.print(f" 2. Sync: bm project bisync --name {name} --resync")
except Exception as e:
console.print(f"[red]Error configuring sync: {str(e)}[/red]")
raise typer.Exit(1)
@project_app.command("remove")
def remove_project(
name: str = typer.Argument(..., help="Name of the project to remove"),
delete_notes: bool = typer.Option(
False, "--delete-notes", help="Delete project files from disk"
),
) -> None:
"""Remove a project."""
async def _remove_project():
async with get_client() as client:
# Convert name to permalink for efficient resolution
project_permalink = generate_permalink(name)
response = await call_delete(client, f"/projects/{project_permalink}")
# Use v2 project resolver to find project ID by permalink
resolve_data = {"identifier": project_permalink}
response = await call_post(client, "/v2/projects/resolve", json=resolve_data)
target_project = response.json()
# Use v2 API with project ID
response = await call_delete(
client, f"/v2/projects/{target_project['external_id']}?delete_notes={delete_notes}"
)
return ProjectStatusResponse.model_validate(response.json())
try:
result = asyncio.run(_remove_project())
# Get config to check for local sync path and bisync state
config = ConfigManager().config
local_path = None
has_bisync_state = False
if config.cloud_mode_enabled and name in config.cloud_projects:
local_path = config.cloud_projects[name].local_path
# Check for bisync state
from basic_memory.cli.commands.cloud.rclone_commands import get_project_bisync_state
bisync_state_path = get_project_bisync_state(name)
has_bisync_state = bisync_state_path.exists()
# Remove project from cloud/API
result = run_with_cleanup(_remove_project())
console.print(f"[green]{result.message}[/green]")
# Clean up local sync directory if it exists and delete_notes is True
if delete_notes and local_path:
local_dir = Path(local_path)
if local_dir.exists():
import shutil
shutil.rmtree(local_dir)
console.print(f"[green]Removed local sync directory: {local_path}[/green]")
# Clean up bisync state if it exists
if has_bisync_state:
from basic_memory.cli.commands.cloud.rclone_commands import get_project_bisync_state
import shutil
bisync_state_path = get_project_bisync_state(name)
if bisync_state_path.exists():
shutil.rmtree(bisync_state_path)
console.print("[green]Removed bisync state[/green]")
# Clean up cloud_projects config entry
if config.cloud_mode_enabled and name in config.cloud_projects:
del config.cloud_projects[name]
ConfigManager().save_config(config)
# Show informative message if files were not deleted
if not delete_notes:
if local_path:
console.print(f"[yellow]Note: Local files remain at {local_path}[/yellow]")
except Exception as e:
console.print(f"[red]Error removing project: {str(e)}[/red]")
raise typer.Exit(1)
# Show this message regardless of method used
console.print("[yellow]Note: The project files have not been deleted from disk.[/yellow]")
@project_app.command("default")
def set_default_project(
@@ -155,12 +331,22 @@ def set_default_project(
async def _set_default():
async with get_client() as client:
# Convert name to permalink for efficient resolution
project_permalink = generate_permalink(name)
response = await call_put(client, f"/projects/{project_permalink}/default")
# Use v2 project resolver to find project ID by permalink
resolve_data = {"identifier": project_permalink}
response = await call_post(client, "/v2/projects/resolve", json=resolve_data)
target_project = response.json()
# Use v2 API with project ID
response = await call_put(
client, f"/v2/projects/{target_project['external_id']}/default"
)
return ProjectStatusResponse.model_validate(response.json())
try:
result = asyncio.run(_set_default())
result = run_with_cleanup(_set_default())
console.print(f"[green]{result.message}[/green]")
except Exception as e:
console.print(f"[red]Error setting default project: {str(e)}[/red]")
@@ -185,7 +371,7 @@ def synchronize_projects() -> None:
return ProjectStatusResponse.model_validate(response.json())
try:
result = asyncio.run(_sync_config())
result = run_with_cleanup(_sync_config())
console.print(f"[green]{result.message}[/green]")
except Exception as e: # pragma: no cover
console.print(f"[red]Error synchronizing projects: {str(e)}[/red]")
@@ -220,7 +406,7 @@ def move_project(
return ProjectStatusResponse.model_validate(response.json())
try:
result = asyncio.run(_move_project())
result = run_with_cleanup(_move_project())
console.print(f"[green]{result.message}[/green]")
# Show important file movement reminder
@@ -231,7 +417,7 @@ def move_project(
"[yellow]You must manually move your project files from the old location to:[/yellow]\n"
f"[cyan]{resolved_path}[/cyan]\n\n"
"[dim]Basic Memory has only updated the configuration - your files remain in their original location.[/dim]",
title="⚠️ Manual File Movement Required",
title="Manual File Movement Required",
border_style="yellow",
expand=False,
)
@@ -242,6 +428,353 @@ def move_project(
raise typer.Exit(1)
@project_app.command("sync")
def sync_project_command(
name: str = typer.Option(..., "--name", help="Project name to sync"),
dry_run: bool = typer.Option(False, "--dry-run", help="Preview changes without syncing"),
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed output"),
) -> None:
"""One-way sync: local -> cloud (make cloud identical to local).
Example:
bm project sync --name research
bm project sync --name research --dry-run
"""
config = ConfigManager().config
if not config.cloud_mode_enabled:
console.print("[red]Error: sync only available in cloud mode[/red]")
raise typer.Exit(1)
try:
# Get tenant info for bucket name
tenant_info = run_with_cleanup(get_mount_info())
bucket_name = tenant_info.bucket_name
# Get project info
async def _get_project():
async with get_client() as client:
response = await call_get(client, "/projects/projects")
projects_list = ProjectList.model_validate(response.json())
for proj in projects_list.projects:
if generate_permalink(proj.name) == generate_permalink(name):
return proj
return None
project_data = run_with_cleanup(_get_project())
if not project_data:
console.print(f"[red]Error: Project '{name}' not found[/red]")
raise typer.Exit(1)
# Get local_sync_path from cloud_projects config
local_sync_path = None
if name in config.cloud_projects:
local_sync_path = config.cloud_projects[name].local_path
if not local_sync_path:
console.print(f"[red]Error: Project '{name}' has no local_sync_path configured[/red]")
console.print(f"\nConfigure sync with: bm project sync-setup {name} ~/path/to/local")
raise typer.Exit(1)
# Create SyncProject
sync_project = SyncProject(
name=project_data.name,
path=normalize_project_path(project_data.path),
local_sync_path=local_sync_path,
)
# Run sync
console.print(f"[blue]Syncing {name} (local -> cloud)...[/blue]")
success = project_sync(sync_project, bucket_name, dry_run=dry_run, verbose=verbose)
if success:
console.print(f"[green]{name} synced successfully[/green]")
# Trigger database sync if not a dry run
if not dry_run:
async def _trigger_db_sync():
async with get_client() as client:
permalink = generate_permalink(name)
response = await call_post(
client, f"/{permalink}/project/sync?force_full=true", json={}
)
return response.json()
try:
result = run_with_cleanup(_trigger_db_sync())
console.print(f"[dim]Database sync initiated: {result.get('message')}[/dim]")
except Exception as e:
console.print(f"[yellow]Warning: Could not trigger database sync: {e}[/yellow]")
else:
console.print(f"[red]{name} sync failed[/red]")
raise typer.Exit(1)
except RcloneError as e:
console.print(f"[red]Sync error: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
@project_app.command("bisync")
def bisync_project_command(
name: str = typer.Option(..., "--name", help="Project name to bisync"),
dry_run: bool = typer.Option(False, "--dry-run", help="Preview changes without syncing"),
resync: bool = typer.Option(False, "--resync", help="Force new baseline"),
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed output"),
) -> None:
"""Two-way sync: local <-> cloud (bidirectional sync).
Examples:
bm project bisync --name research --resync # First time
bm project bisync --name research # Subsequent syncs
bm project bisync --name research --dry-run # Preview changes
"""
config = ConfigManager().config
if not config.cloud_mode_enabled:
console.print("[red]Error: bisync only available in cloud mode[/red]")
raise typer.Exit(1)
try:
# Get tenant info for bucket name
tenant_info = run_with_cleanup(get_mount_info())
bucket_name = tenant_info.bucket_name
# Get project info
async def _get_project():
async with get_client() as client:
response = await call_get(client, "/projects/projects")
projects_list = ProjectList.model_validate(response.json())
for proj in projects_list.projects:
if generate_permalink(proj.name) == generate_permalink(name):
return proj
return None
project_data = run_with_cleanup(_get_project())
if not project_data:
console.print(f"[red]Error: Project '{name}' not found[/red]")
raise typer.Exit(1)
# Get local_sync_path from cloud_projects config
local_sync_path = None
if name in config.cloud_projects:
local_sync_path = config.cloud_projects[name].local_path
if not local_sync_path:
console.print(f"[red]Error: Project '{name}' has no local_sync_path configured[/red]")
console.print(f"\nConfigure sync with: bm project sync-setup {name} ~/path/to/local")
raise typer.Exit(1)
# Create SyncProject
sync_project = SyncProject(
name=project_data.name,
path=normalize_project_path(project_data.path),
local_sync_path=local_sync_path,
)
# Run bisync
console.print(f"[blue]Bisync {name} (local <-> cloud)...[/blue]")
success = project_bisync(
sync_project, bucket_name, dry_run=dry_run, resync=resync, verbose=verbose
)
if success:
console.print(f"[green]{name} bisync completed successfully[/green]")
# Update config
config.cloud_projects[name].last_sync = datetime.now()
config.cloud_projects[name].bisync_initialized = True
ConfigManager().save_config(config)
# Trigger database sync if not a dry run
if not dry_run:
async def _trigger_db_sync():
async with get_client() as client:
permalink = generate_permalink(name)
response = await call_post(
client, f"/{permalink}/project/sync?force_full=true", json={}
)
return response.json()
try:
result = run_with_cleanup(_trigger_db_sync())
console.print(f"[dim]Database sync initiated: {result.get('message')}[/dim]")
except Exception as e:
console.print(f"[yellow]Warning: Could not trigger database sync: {e}[/yellow]")
else:
console.print(f"[red]{name} bisync failed[/red]")
raise typer.Exit(1)
except RcloneError as e:
console.print(f"[red]Bisync error: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
@project_app.command("check")
def check_project_command(
name: str = typer.Option(..., "--name", help="Project name to check"),
one_way: bool = typer.Option(False, "--one-way", help="Check one direction only (faster)"),
) -> None:
"""Verify file integrity between local and cloud.
Example:
bm project check --name research
"""
config = ConfigManager().config
if not config.cloud_mode_enabled:
console.print("[red]Error: check only available in cloud mode[/red]")
raise typer.Exit(1)
try:
# Get tenant info for bucket name
tenant_info = run_with_cleanup(get_mount_info())
bucket_name = tenant_info.bucket_name
# Get project info
async def _get_project():
async with get_client() as client:
response = await call_get(client, "/projects/projects")
projects_list = ProjectList.model_validate(response.json())
for proj in projects_list.projects:
if generate_permalink(proj.name) == generate_permalink(name):
return proj
return None
project_data = run_with_cleanup(_get_project())
if not project_data:
console.print(f"[red]Error: Project '{name}' not found[/red]")
raise typer.Exit(1)
# Get local_sync_path from cloud_projects config
local_sync_path = None
if name in config.cloud_projects:
local_sync_path = config.cloud_projects[name].local_path
if not local_sync_path:
console.print(f"[red]Error: Project '{name}' has no local_sync_path configured[/red]")
console.print(f"\nConfigure sync with: bm project sync-setup {name} ~/path/to/local")
raise typer.Exit(1)
# Create SyncProject
sync_project = SyncProject(
name=project_data.name,
path=normalize_project_path(project_data.path),
local_sync_path=local_sync_path,
)
# Run check
console.print(f"[blue]Checking {name} integrity...[/blue]")
match = project_check(sync_project, bucket_name, one_way=one_way)
if match:
console.print(f"[green]{name} files match[/green]")
else:
console.print(f"[yellow]!{name} has differences[/yellow]")
except RcloneError as e:
console.print(f"[red]Check error: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
@project_app.command("bisync-reset")
def bisync_reset(
name: str = typer.Argument(..., help="Project name to reset bisync state for"),
) -> None:
"""Clear bisync state for a project.
This removes the bisync metadata files, forcing a fresh --resync on next bisync.
Useful when bisync gets into an inconsistent state or when remote path changes.
"""
from basic_memory.cli.commands.cloud.rclone_commands import get_project_bisync_state
import shutil
try:
state_path = get_project_bisync_state(name)
if not state_path.exists():
console.print(f"[yellow]No bisync state found for project '{name}'[/yellow]")
return
# Remove the entire state directory
shutil.rmtree(state_path)
console.print(f"[green]Cleared bisync state for project '{name}'[/green]")
console.print("\nNext steps:")
console.print(f" 1. Preview: bm project bisync --name {name} --resync --dry-run")
console.print(f" 2. Sync: bm project bisync --name {name} --resync")
except Exception as e:
console.print(f"[red]Error clearing bisync state: {str(e)}[/red]")
raise typer.Exit(1)
@project_app.command("ls")
def ls_project_command(
name: str = typer.Option(..., "--name", help="Project name to list files from"),
path: str = typer.Argument(None, help="Path within project (optional)"),
) -> None:
"""List files in remote project.
Examples:
bm project ls --name research
bm project ls --name research subfolder
"""
config = ConfigManager().config
if not config.cloud_mode_enabled:
console.print("[red]Error: ls only available in cloud mode[/red]")
raise typer.Exit(1)
try:
# Get tenant info for bucket name
tenant_info = run_with_cleanup(get_mount_info())
bucket_name = tenant_info.bucket_name
# Get project info
async def _get_project():
async with get_client() as client:
response = await call_get(client, "/projects/projects")
projects_list = ProjectList.model_validate(response.json())
for proj in projects_list.projects:
if generate_permalink(proj.name) == generate_permalink(name):
return proj
return None
project_data = run_with_cleanup(_get_project())
if not project_data:
console.print(f"[red]Error: Project '{name}' not found[/red]")
raise typer.Exit(1)
# Create SyncProject (local_sync_path not needed for ls)
sync_project = SyncProject(
name=project_data.name,
path=normalize_project_path(project_data.path),
)
# List files
files = project_ls(sync_project, bucket_name, path=path)
if files:
console.print(f"\n[bold]Files in {name}" + (f"/{path}" if path else "") + ":[/bold]")
for file in files:
console.print(f" {file}")
console.print(f"\n[dim]Total: {len(files)} files[/dim]")
else:
console.print(
f"[yellow]No files found in {name}" + (f"/{path}" if path else "") + "[/yellow]"
)
except Exception as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
@project_app.command("info")
def display_project_info(
name: str = typer.Argument(..., help="Name of the project"),
@@ -250,7 +783,7 @@ def display_project_info(
"""Display detailed information and statistics about the current project."""
try:
# Get project info
info = asyncio.run(get_project_info(name))
info = run_with_cleanup(get_project_info(name))
if json_output:
# Convert to JSON and print
@@ -263,13 +796,13 @@ def display_project_info(
f"[bold]Project:[/bold] {info.project_name}\n"
f"[bold]Path:[/bold] {info.project_path}\n"
f"[bold]Default Project:[/bold] {info.default_project}\n",
title="📊 Basic Memory Project Info",
title="Basic Memory Project Info",
expand=False,
)
)
# Statistics section
stats_table = Table(title="📈 Statistics")
stats_table = Table(title="Statistics")
stats_table.add_column("Metric", style="cyan")
stats_table.add_column("Count", style="green")
@@ -285,7 +818,7 @@ def display_project_info(
# Entity types
if info.statistics.entity_types:
entity_types_table = Table(title="📑 Entity Types")
entity_types_table = Table(title="Entity Types")
entity_types_table.add_column("Type", style="blue")
entity_types_table.add_column("Count", style="green")
@@ -296,7 +829,7 @@ def display_project_info(
# Most connected entities
if info.statistics.most_connected_entities: # pragma: no cover
connected_table = Table(title="🔗 Most Connected Entities")
connected_table = Table(title="Most Connected Entities")
connected_table.add_column("Title", style="blue")
connected_table.add_column("Permalink", style="cyan")
connected_table.add_column("Relations", style="green")
@@ -310,7 +843,7 @@ def display_project_info(
# Recent activity
if info.activity.recently_updated: # pragma: no cover
recent_table = Table(title="🕒 Recent Activity")
recent_table = Table(title="Recent Activity")
recent_table.add_column("Title", style="blue")
recent_table.add_column("Type", style="cyan")
recent_table.add_column("Last Updated", style="green")
@@ -330,7 +863,7 @@ def display_project_info(
console.print(recent_table)
# Available projects
projects_table = Table(title="📁 Available Projects")
projects_table = Table(title="Available Projects")
projects_table.add_column("Name", style="blue")
projects_table.add_column("Path", style="cyan")
projects_table.add_column("Default", style="green")
@@ -338,7 +871,7 @@ def display_project_info(
for name, proj_info in info.available_projects.items():
is_default = name == info.default_project
project_path = proj_info["path"]
projects_table.add_row(name, project_path, "" if is_default else "")
projects_table.add_row(name, project_path, "[X]" if is_default else "")
console.print(projects_table)
+20 -4
View File
@@ -1,6 +1,5 @@
"""Status command for basic-memory CLI."""
import asyncio
from typing import Set, Dict
from typing import Annotated, Optional
@@ -94,7 +93,7 @@ def display_changes(
"""Display changes using Rich for better visualization."""
tree = Tree(f"{project_name}: {title}")
if changes.total == 0:
if changes.total == 0 and not changes.skipped_files:
tree.add("No changes")
console.print(Panel(tree, expand=False))
return
@@ -114,6 +113,13 @@ def display_changes(
if changes.deleted:
del_branch = tree.add("[red]Deleted[/red]")
add_files_to_tree(del_branch, changes.deleted, "red")
if changes.skipped_files:
skip_branch = tree.add("[red]! Skipped (Circuit Breaker)[/red]")
for skipped in sorted(changes.skipped_files, key=lambda x: x.path):
skip_branch.add(
f"[red]{skipped.path}[/red] "
f"(failures: {skipped.failure_count}, reason: {skipped.reason})"
)
else:
# Show directory summaries
by_dir = group_changes_by_directory(changes)
@@ -122,6 +128,14 @@ def display_changes(
if summary: # Only show directories with changes
tree.add(f"[bold]{dir_name}/[/bold] {summary}")
# Show skipped files summary in non-verbose mode
if changes.skipped_files:
skip_count = len(changes.skipped_files)
tree.add(
f"[red]! {skip_count} file{'s' if skip_count != 1 else ''} "
f"skipped due to repeated failures[/red]"
)
console.print(Panel(tree, expand=False))
@@ -137,7 +151,7 @@ async def run_status(project: Optional[str] = None, verbose: bool = False): # p
display_changes(project_item.name, "Status", sync_report, verbose)
except (ValueError, ToolError) as e:
console.print(f"[red]Error: {e}[/red]")
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
@@ -150,8 +164,10 @@ def status(
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed file information"),
):
"""Show sync status between files and database."""
from basic_memory.cli.commands.command_utils import run_with_cleanup
try:
asyncio.run(run_status(project, verbose)) # pragma: no cover
run_with_cleanup(run_status(project, verbose)) # pragma: no cover
except Exception as e:
logger.error(f"Error checking status: {e}")
typer.echo(f"Error checking status: {e}", err=True)
-59
View File
@@ -1,59 +0,0 @@
"""Command module for basic-memory sync operations."""
import asyncio
from typing import Annotated, Optional
import typer
from basic_memory.cli.app import app
from basic_memory.cli.commands.command_utils import run_sync
from basic_memory.config import ConfigManager
@app.command()
def sync(
project: Annotated[
Optional[str],
typer.Option(help="The project name."),
] = None,
watch: Annotated[
bool,
typer.Option("--watch", help="Run continuous sync (cloud mode only)"),
] = False,
interval: Annotated[
int,
typer.Option("--interval", help="Sync interval in seconds for watch mode (default: 60)"),
] = 60,
) -> None:
"""Sync knowledge files with the database.
In local mode: Scans filesystem and updates database.
In cloud mode: Runs bidirectional file sync (bisync) then updates database.
Examples:
bm sync # One-time sync
bm sync --watch # Continuous sync every 60s
bm sync --watch --interval 30 # Continuous sync every 30s
"""
config = ConfigManager().config
if config.cloud_mode_enabled:
# Cloud mode: run bisync which includes database sync
from basic_memory.cli.commands.cloud.bisync_commands import run_bisync, run_bisync_watch
try:
if watch:
run_bisync_watch(interval_seconds=interval)
else:
run_bisync()
except Exception:
raise typer.Exit(1)
else:
# Local mode: just database sync
if watch:
typer.echo(
"Error: --watch is only available in cloud mode. Run 'bm cloud login' first."
)
raise typer.Exit(1)
asyncio.run(run_sync(project))
+7 -7
View File
@@ -1,6 +1,5 @@
"""CLI tool commands for Basic Memory."""
import asyncio
import sys
from typing import Annotated, List, Optional
@@ -9,6 +8,7 @@ from loguru import logger
from rich import print as rprint
from basic_memory.cli.app import app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.config import ConfigManager
# Import prompts
@@ -109,7 +109,7 @@ def write_note(
# use the project name, or the default from the config
project_name = project_name or config_manager.default_project
note = asyncio.run(mcp_write_note.fn(title, content, folder, project_name, tags))
note = run_with_cleanup(mcp_write_note.fn(title, content, folder, project_name, tags))
rprint(note)
except Exception as e: # pragma: no cover
if not isinstance(e, typer.Exit):
@@ -145,7 +145,7 @@ def read_note(
project_name = project_name or config_manager.default_project
try:
note = asyncio.run(mcp_read_note.fn(identifier, project_name, page, page_size))
note = run_with_cleanup(mcp_read_note.fn(identifier, project_name, page, page_size))
rprint(note)
except Exception as e: # pragma: no cover
if not isinstance(e, typer.Exit):
@@ -182,7 +182,7 @@ def build_context(
project_name = project_name or config_manager.default_project
try:
context = asyncio.run(
context = run_with_cleanup(
mcp_build_context.fn(
project=project_name,
url=url,
@@ -213,7 +213,7 @@ def recent_activity(
):
"""Get recent activity across the knowledge base."""
try:
result = asyncio.run(
result = run_with_cleanup(
mcp_recent_activity.fn(
type=type, # pyright: ignore [reportArgumentType]
depth=depth,
@@ -279,7 +279,7 @@ def search_notes(
search_type = ("title" if title else None,)
search_type = "text" if search_type is None else search_type
results = asyncio.run(
results = run_with_cleanup(
mcp_search.fn(
query,
project_name,
@@ -312,7 +312,7 @@ def continue_conversation(
"""Prompt to continue a previous conversation or work session."""
try:
# Prompt functions return formatted strings directly
session = asyncio.run(mcp_continue_conversation.fn(topic=topic, timeframe=timeframe)) # type: ignore
session = run_with_cleanup(mcp_continue_conversation.fn(topic=topic, timeframe=timeframe)) # type: ignore
rprint(session)
except Exception as e: # pragma: no cover
if not isinstance(e, typer.Exit):
+84
View File
@@ -0,0 +1,84 @@
"""CLI composition root for Basic Memory.
This container owns reading ConfigManager and environment variables for the
CLI entrypoint. Downstream modules receive config/dependencies explicitly
rather than reading globals.
Design principles:
- Only this module reads ConfigManager directly
- Runtime mode (cloud/local/test) is resolved here
- Different CLI commands may need different initialization
"""
from dataclasses import dataclass
from basic_memory.config import BasicMemoryConfig, ConfigManager
from basic_memory.runtime import RuntimeMode, resolve_runtime_mode
@dataclass
class CliContainer:
"""Composition root for the CLI entrypoint.
Holds resolved configuration and runtime context.
Created once at CLI startup, then used by subcommands.
"""
config: BasicMemoryConfig
mode: RuntimeMode
@classmethod
def create(cls) -> "CliContainer":
"""Create container by reading ConfigManager.
This is the single point where CLI reads global config.
"""
config = ConfigManager().config
mode = resolve_runtime_mode(
cloud_mode_enabled=config.cloud_mode_enabled,
is_test_env=config.is_test_env,
)
return cls(config=config, mode=mode)
# --- Runtime Mode Properties ---
@property
def is_cloud_mode(self) -> bool:
"""Whether running in cloud mode."""
return self.mode.is_cloud
# Module-level container instance (set by app callback)
_container: CliContainer | None = None
def get_container() -> CliContainer:
"""Get the current CLI container.
Returns:
The CLI container
Raises:
RuntimeError: If container hasn't been initialized
"""
if _container is None:
raise RuntimeError("CLI container not initialized. Call set_container() first.")
return _container
def set_container(container: CliContainer) -> None:
"""Set the CLI container (called by app callback)."""
global _container
_container = container
def get_or_create_container() -> CliContainer:
"""Get existing container or create new one.
This is useful for CLI commands that might be called before
the main app callback runs (e.g., eager options).
"""
global _container
if _container is None:
_container = CliContainer.create()
return _container
+6 -1
View File
@@ -13,10 +13,15 @@ from basic_memory.cli.commands import ( # noqa: F401 # pragma: no cover
mcp,
project,
status,
sync,
tool,
)
# Re-apply warning filter AFTER all imports
# (authlib adds a DeprecationWarning filter that overrides ours)
import warnings # pragma: no cover
warnings.filterwarnings("ignore") # pragma: no cover
if __name__ == "__main__": # pragma: no cover
# start the app
app()
+202 -91
View File
@@ -3,14 +3,15 @@
import json
import os
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from typing import Any, Dict, Literal, Optional, List, Tuple
from enum import Enum
from loguru import logger
from pydantic import Field, field_validator
from pydantic import BaseModel, Field, model_validator
from pydantic_settings import BaseSettings, SettingsConfigDict
import basic_memory
from basic_memory.utils import setup_logging, generate_permalink
@@ -23,6 +24,13 @@ WATCH_STATUS_JSON = "watch-status.json"
Environment = Literal["test", "dev", "user"]
class DatabaseBackend(str, Enum):
"""Supported database backends."""
SQLITE = "sqlite"
POSTGRES = "postgres"
@dataclass
class ProjectConfig:
"""Configuration for a specific basic-memory project."""
@@ -32,13 +40,29 @@ class ProjectConfig:
@property
def project(self):
return self.name
return self.name # pragma: no cover
@property
def project_url(self) -> str: # pragma: no cover
return f"/{generate_permalink(self.name)}"
class CloudProjectConfig(BaseModel):
"""Sync configuration for a cloud project.
This tracks the local working directory and sync state for a project
that is synced with Basic Memory Cloud.
"""
local_path: str = Field(description="Local working directory path for this cloud project")
last_sync: Optional[datetime] = Field(
default=None, description="Timestamp of last successful sync operation"
)
bisync_initialized: bool = Field(
default=False, description="Whether rclone bisync baseline has been established"
)
class BasicMemoryConfig(BaseSettings):
"""Pydantic model for Basic Memory global configuration."""
@@ -46,8 +70,10 @@ class BasicMemoryConfig(BaseSettings):
projects: Dict[str, str] = Field(
default_factory=lambda: {
"main": Path(os.getenv("BASIC_MEMORY_HOME", Path.home() / "basic-memory")).as_posix()
},
"main": str(Path(os.getenv("BASIC_MEMORY_HOME", Path.home() / "basic-memory")))
}
if os.getenv("BASIC_MEMORY_HOME")
else {},
description="Mapping of project names to their filesystem paths",
)
default_project: str = Field(
@@ -62,13 +88,43 @@ class BasicMemoryConfig(BaseSettings):
# overridden by ~/.basic-memory/config.json
log_level: str = "INFO"
# Database configuration
database_backend: DatabaseBackend = Field(
default=DatabaseBackend.SQLITE,
description="Database backend to use (sqlite or postgres)",
)
database_url: Optional[str] = Field(
default=None,
description="Database connection URL. For Postgres, use postgresql+asyncpg://user:pass@host:port/db. If not set, SQLite will use default path.",
)
# Database connection pool configuration (Postgres only)
db_pool_size: int = Field(
default=20,
description="Number of connections to keep in the pool (Postgres only)",
gt=0,
)
db_pool_overflow: int = Field(
default=40,
description="Max additional connections beyond pool_size under load (Postgres only)",
gt=0,
)
db_pool_recycle: int = Field(
default=180,
description="Recycle connections after N seconds to prevent stale connections. Default 180s works well with Neon's ~5 minute scale-to-zero (Postgres only)",
gt=0,
)
# Watch service configuration
sync_delay: int = Field(
default=1000, description="Milliseconds to wait after changes before syncing", gt=0
)
watch_project_reload_interval: int = Field(
default=30, description="Seconds between reloading project list in watch service", gt=0
default=300,
description="Seconds between reloading project list in watch service. Higher values reduce CPU usage by minimizing watcher restarts. Default 300s (5 min) balances efficiency with responsiveness to new projects.",
gt=0,
)
# update permalinks on move
@@ -84,7 +140,13 @@ class BasicMemoryConfig(BaseSettings):
sync_thread_pool_size: int = Field(
default=4,
description="Size of thread pool for file I/O operations in sync service",
description="Size of thread pool for file I/O operations in sync service. Default of 4 is optimized for cloud deployments with 1-2GB RAM.",
gt=0,
)
sync_max_concurrent_files: int = Field(
default=10,
description="Maximum number of files to process concurrently during sync. Limits memory usage on large projects (2000+ files). Lower values reduce memory consumption.",
gt=0,
)
@@ -103,6 +165,28 @@ class BasicMemoryConfig(BaseSettings):
description="Skip expensive initialization synchronization. Useful for cloud/stateless deployments where project reconciliation is not needed.",
)
# File formatting configuration
format_on_save: bool = Field(
default=False,
description="Automatically format files after saving using configured formatter. Disabled by default.",
)
formatter_command: Optional[str] = Field(
default=None,
description="External formatter command. Use {file} as placeholder for file path. If not set, uses built-in mdformat (Python, no Node.js required). Set to 'npx prettier --write {file}' for Prettier.",
)
formatters: Dict[str, str] = Field(
default_factory=dict,
description="Per-extension formatters. Keys are extensions (without dot), values are commands. Example: {'md': 'prettier --write {file}', 'json': 'prettier --write {file}'}",
)
formatter_timeout: float = Field(
default=5.0,
description="Maximum seconds to wait for formatter to complete",
gt=0,
)
# Project path constraints
project_root: Optional[str] = Field(
default=None,
@@ -132,6 +216,28 @@ class BasicMemoryConfig(BaseSettings):
description="Enable cloud mode - all requests go to cloud instead of local (config file value)",
)
cloud_projects: Dict[str, CloudProjectConfig] = Field(
default_factory=dict,
description="Cloud project sync configuration mapping project names to their local paths and sync state",
)
@property
def is_test_env(self) -> bool:
"""Check if running in a test environment.
Returns True if any of:
- env field is set to "test"
- BASIC_MEMORY_ENV environment variable is "test"
- PYTEST_CURRENT_TEST environment variable is set (pytest is running)
Used to disable features like file watchers during tests.
"""
return (
self.env == "test"
or os.getenv("BASIC_MEMORY_ENV", "").lower() == "test"
or os.getenv("PYTEST_CURRENT_TEST") is not None
)
@property
def cloud_mode_enabled(self) -> bool:
"""Check if cloud mode is enabled.
@@ -148,13 +254,35 @@ class BasicMemoryConfig(BaseSettings):
# Fall back to config file value
return self.cloud_mode
bisync_config: Dict[str, Any] = Field(
default_factory=lambda: {
"profile": "balanced",
"sync_dir": str(Path.home() / "basic-memory-cloud-sync"),
},
description="Bisync configuration for cloud sync",
)
@classmethod
def for_cloud_tenant(
cls,
database_url: str,
projects: Optional[Dict[str, str]] = None,
) -> "BasicMemoryConfig":
"""Create config for cloud tenant - no config.json, database is source of truth.
This factory method creates a BasicMemoryConfig suitable for cloud deployments
where:
- Database is Postgres (Neon), not SQLite
- Projects are discovered from the database, not config file
- Path validation is skipped (no local filesystem in cloud)
- Initialization sync is skipped (stateless deployment)
Args:
database_url: Postgres connection URL for tenant database
projects: Optional project mapping (usually empty, discovered from DB)
Returns:
BasicMemoryConfig configured for cloud mode
"""
return cls( # pragma: no cover
database_backend=DatabaseBackend.POSTGRES,
database_url=database_url,
projects=projects or {},
cloud_mode=True,
skip_initialization_sync=True,
)
model_config = SettingsConfigDict(
env_prefix="BASIC_MEMORY_",
@@ -172,15 +300,20 @@ class BasicMemoryConfig(BaseSettings):
def model_post_init(self, __context: Any) -> None:
"""Ensure configuration is valid after initialization."""
# Ensure main project exists
if "main" not in self.projects: # pragma: no cover
self.projects["main"] = (
Path(os.getenv("BASIC_MEMORY_HOME", Path.home() / "basic-memory"))
).as_posix()
# Skip project initialization in cloud mode - projects are discovered from DB
if self.database_backend == DatabaseBackend.POSTGRES: # pragma: no cover
return # pragma: no cover
# Ensure default project is valid
# Ensure at least one project exists; if none exist then create main
if not self.projects: # pragma: no cover
self.projects["main"] = str(
Path(os.getenv("BASIC_MEMORY_HOME", Path.home() / "basic-memory"))
)
# Ensure default project is valid (i.e. points to an existing project)
if self.default_project not in self.projects: # pragma: no cover
self.default_project = "main"
# Set default to first available project
self.default_project = next(iter(self.projects.keys()))
@property
def app_database_path(self) -> Path:
@@ -213,19 +346,26 @@ class BasicMemoryConfig(BaseSettings):
"""Get all configured projects as ProjectConfig objects."""
return [ProjectConfig(name=name, home=Path(path)) for name, path in self.projects.items()]
@field_validator("projects")
@classmethod
def ensure_project_paths_exists(cls, v: Dict[str, str]) -> Dict[str, str]: # pragma: no cover
"""Ensure project path exists."""
for name, path_value in v.items():
@model_validator(mode="after")
def ensure_project_paths_exists(self) -> "BasicMemoryConfig": # pragma: no cover
"""Ensure project paths exist.
Skips path creation when using Postgres backend (cloud mode) since
cloud tenants don't use local filesystem paths.
"""
# Skip path creation for cloud mode - no local filesystem
if self.database_backend == DatabaseBackend.POSTGRES:
return self
for name, path_value in self.projects.items():
path = Path(path_value)
if not Path(path).exists():
if not path.exists():
try:
path.mkdir(parents=True)
except Exception as e:
logger.error(f"Failed to create project path: {e}")
raise e
return v
return self
@property
def data_dir_path(self):
@@ -338,7 +478,7 @@ class ConfigManager:
# Load config, modify it, and save it
config = self.load_config()
config.projects[name] = project_path.as_posix()
config.projects[name] = str(project_path)
self.save_config(config)
return ProjectConfig(name=name, home=project_path)
@@ -421,74 +561,45 @@ def get_project_config(project_name: Optional[str] = None) -> ProjectConfig:
def save_basic_memory_config(file_path: Path, config: BasicMemoryConfig) -> None:
"""Save configuration to file."""
try:
file_path.write_text(json.dumps(config.model_dump(), indent=2))
# Use model_dump with mode='json' to serialize datetime objects properly
config_dict = config.model_dump(mode="json")
file_path.write_text(json.dumps(config_dict, indent=2))
except Exception as e: # pragma: no cover
logger.error(f"Failed to save config: {e}")
# setup logging to a single log file in user home directory
user_home = Path.home()
log_dir = user_home / DATA_DIR_NAME
log_dir.mkdir(parents=True, exist_ok=True)
# Logging initialization functions for different entry points
# Process info for logging
def get_process_name(): # pragma: no cover
def init_cli_logging() -> None: # pragma: no cover
"""Initialize logging for CLI commands - file only.
CLI commands should not log to stdout to avoid interfering with
command output and shell integration.
"""
get the type of process for logging
"""
import sys
log_level = os.getenv("BASIC_MEMORY_LOG_LEVEL", "INFO")
setup_logging(log_level=log_level, log_to_file=True)
if "sync" in sys.argv:
return "sync"
elif "mcp" in sys.argv:
return "mcp"
elif "cli" in sys.argv:
return "cli"
def init_mcp_logging() -> None: # pragma: no cover
"""Initialize logging for MCP server - file only.
MCP server must not log to stdout as it would corrupt the
JSON-RPC protocol communication.
"""
log_level = os.getenv("BASIC_MEMORY_LOG_LEVEL", "INFO")
setup_logging(log_level=log_level, log_to_file=True)
def init_api_logging() -> None: # pragma: no cover
"""Initialize logging for API server.
Cloud mode (BASIC_MEMORY_CLOUD_MODE=1): stdout with structured context
Local mode: file only
"""
log_level = os.getenv("BASIC_MEMORY_LOG_LEVEL", "INFO")
cloud_mode = os.getenv("BASIC_MEMORY_CLOUD_MODE", "").lower() in ("1", "true")
if cloud_mode:
setup_logging(log_level=log_level, log_to_stdout=True, structured_context=True)
else:
return "api"
process_name = get_process_name()
# Global flag to track if logging has been set up
_LOGGING_SETUP = False
# Logging
def setup_basic_memory_logging(): # pragma: no cover
"""Set up logging for basic-memory, ensuring it only happens once."""
global _LOGGING_SETUP
if _LOGGING_SETUP:
# We can't log before logging is set up
# print("Skipping duplicate logging setup")
return
# Check for console logging environment variable - accept more truthy values
console_logging_env = os.getenv("BASIC_MEMORY_CONSOLE_LOGGING", "false").lower()
console_logging = console_logging_env in ("true", "1", "yes", "on")
# Check for log level environment variable first, fall back to config
log_level = os.getenv("BASIC_MEMORY_LOG_LEVEL")
if not log_level:
config_manager = ConfigManager()
log_level = config_manager.config.log_level
config_manager = ConfigManager()
config = get_project_config()
setup_logging(
env=config_manager.config.env,
home_dir=user_home, # Use user home for logs
log_level=log_level,
log_file=f"{DATA_DIR_NAME}/basic-memory-{process_name}.log",
console=console_logging,
)
logger.info(f"Basic Memory {basic_memory.__version__} (Project: {config.project})")
_LOGGING_SETUP = True
# Set up logging
setup_basic_memory_logging()
setup_logging(log_level=log_level, log_to_file=True)
+183 -77
View File
@@ -1,11 +1,12 @@
import asyncio
import os
import sys
from contextlib import asynccontextmanager
from enum import Enum, auto
from pathlib import Path
from typing import AsyncGenerator, Optional
from basic_memory.config import BasicMemoryConfig, ConfigManager
from basic_memory.config import BasicMemoryConfig, ConfigManager, DatabaseBackend
from alembic import command
from alembic.config import Config
@@ -20,12 +21,27 @@ from sqlalchemy.ext.asyncio import (
)
from sqlalchemy.pool import NullPool
from basic_memory.repository.search_repository import SearchRepository
from basic_memory.repository.postgres_search_repository import PostgresSearchRepository
from basic_memory.repository.sqlite_search_repository import SQLiteSearchRepository
# -----------------------------------------------------------------------------
# Windows event loop policy
# -----------------------------------------------------------------------------
# On Windows, the default ProactorEventLoop has known rough edges with aiosqlite
# during shutdown/teardown (threads posting results to a loop that's closing),
# which can manifest as:
# - "RuntimeError: Event loop is closed"
# - "IndexError: pop from an empty deque"
#
# The SelectorEventLoop doesn't support subprocess operations, so code that uses
# asyncio.create_subprocess_shell() (like sync_service._quick_count_files) must
# detect Windows and use fallback implementations.
if sys.platform == "win32": # pragma: no cover
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
# Module level state
_engine: Optional[AsyncEngine] = None
_session_maker: Optional[async_sessionmaker[AsyncSession]] = None
_migrations_completed: bool = False
class DatabaseType(Enum):
@@ -33,10 +49,41 @@ class DatabaseType(Enum):
MEMORY = auto()
FILESYSTEM = auto()
POSTGRES = auto()
@classmethod
def get_db_url(cls, db_path: Path, db_type: "DatabaseType") -> str:
"""Get SQLAlchemy URL for database path."""
def get_db_url(
cls, db_path: Path, db_type: "DatabaseType", config: Optional[BasicMemoryConfig] = None
) -> str:
"""Get SQLAlchemy URL for database path.
Args:
db_path: Path to SQLite database file (ignored for Postgres)
db_type: Type of database (MEMORY, FILESYSTEM, or POSTGRES)
config: Optional config to check for database backend and URL
Returns:
SQLAlchemy connection URL
"""
# Load config if not provided
if config is None:
config = ConfigManager().config
# Handle explicit Postgres type
if db_type == cls.POSTGRES:
if not config.database_url:
raise ValueError("DATABASE_URL must be set when using Postgres backend")
logger.info(f"Using Postgres database: {config.database_url}")
return config.database_url
# Check if Postgres backend is configured (for backward compatibility)
if config.database_backend == DatabaseBackend.POSTGRES:
if not config.database_url:
raise ValueError("DATABASE_URL must be set when using Postgres backend")
logger.info(f"Using Postgres database: {config.database_url}")
return config.database_url
# SQLite databases
if db_type == cls.MEMORY:
logger.info("Using in-memory SQLite database")
return "sqlite+aiosqlite://"
@@ -64,7 +111,14 @@ async def scoped_session(
factory = get_scoped_session_factory(session_maker)
session = factory()
try:
await session.execute(text("PRAGMA foreign_keys=ON"))
# Only enable foreign keys for SQLite (Postgres has them enabled by default)
# Detect database type from session's bind (engine) dialect
engine = session.get_bind()
dialect_name = engine.dialect.name
if dialect_name == "sqlite":
await session.execute(text("PRAGMA foreign_keys=ON"))
yield session
await session.commit()
except Exception:
@@ -103,13 +157,16 @@ def _configure_sqlite_connection(dbapi_conn, enable_wal: bool = True) -> None:
cursor.close()
def _create_engine_and_session(
db_path: Path, db_type: DatabaseType = DatabaseType.FILESYSTEM
) -> tuple[AsyncEngine, async_sessionmaker[AsyncSession]]:
"""Internal helper to create engine and session maker."""
db_url = DatabaseType.get_db_url(db_path, db_type)
logger.debug(f"Creating engine for db_url: {db_url}")
def _create_sqlite_engine(db_url: str, db_type: DatabaseType) -> AsyncEngine:
"""Create SQLite async engine with appropriate configuration.
Args:
db_url: SQLite connection URL
db_type: Database type (MEMORY or FILESYSTEM)
Returns:
Configured async engine for SQLite
"""
# Configure connection args with Windows-specific settings
connect_args: dict[str, bool | float | None] = {"check_same_thread": False}
@@ -146,6 +203,73 @@ def _create_engine_and_session(
"""Enable WAL mode on each connection."""
_configure_sqlite_connection(dbapi_conn, enable_wal=enable_wal)
return engine
def _create_postgres_engine(db_url: str, config: BasicMemoryConfig) -> AsyncEngine:
"""Create Postgres async engine with appropriate configuration.
Args:
db_url: Postgres connection URL (postgresql+asyncpg://...)
config: BasicMemoryConfig with pool settings
Returns:
Configured async engine for Postgres
"""
# Use NullPool connection issues.
# Assume connection pooler like PgBouncer handles connection pooling.
engine = create_async_engine(
db_url,
echo=False,
poolclass=NullPool, # No pooling - fresh connection per request
connect_args={
# Disable statement cache to avoid issues with prepared statements on reconnect
"statement_cache_size": 0,
# Allow 30s for commands (Neon cold start can take 2-5s, sometimes longer)
"command_timeout": 30,
# Allow 30s for initial connection (Neon wake-up time)
"timeout": 30,
"server_settings": {
"application_name": "basic-memory",
# Statement timeout for queries (30s to allow for cold start)
"statement_timeout": "30s",
},
},
)
logger.debug("Created Postgres engine with NullPool (no connection pooling)")
return engine
def _create_engine_and_session(
db_path: Path,
db_type: DatabaseType = DatabaseType.FILESYSTEM,
config: Optional[BasicMemoryConfig] = None,
) -> tuple[AsyncEngine, async_sessionmaker[AsyncSession]]:
"""Internal helper to create engine and session maker.
Args:
db_path: Path to database file (used for SQLite, ignored for Postgres)
db_type: Type of database (MEMORY, FILESYSTEM, or POSTGRES)
config: Optional explicit config. If not provided, reads from ConfigManager.
Prefer passing explicitly from composition roots.
Returns:
Tuple of (engine, session_maker)
"""
# Prefer explicit parameter; fall back to ConfigManager for backwards compatibility
if config is None:
config = ConfigManager().config
db_url = DatabaseType.get_db_url(db_path, db_type, config)
logger.debug(f"Creating engine for db_url: {db_url}")
# Delegate to backend-specific engine creation
# Check explicit POSTGRES type first, then config setting
if db_type == DatabaseType.POSTGRES or config.database_backend == DatabaseBackend.POSTGRES:
engine = _create_postgres_engine(db_url, config)
else:
engine = _create_sqlite_engine(db_url, db_type)
session_maker = async_sessionmaker(engine, expire_on_commit=False)
return engine, session_maker
@@ -154,17 +278,29 @@ async def get_or_create_db(
db_path: Path,
db_type: DatabaseType = DatabaseType.FILESYSTEM,
ensure_migrations: bool = True,
config: Optional[BasicMemoryConfig] = None,
) -> tuple[AsyncEngine, async_sessionmaker[AsyncSession]]: # pragma: no cover
"""Get or create database engine and session maker."""
"""Get or create database engine and session maker.
Args:
db_path: Path to database file
db_type: Type of database
ensure_migrations: Whether to run migrations
config: Optional explicit config. If not provided, reads from ConfigManager.
Prefer passing explicitly from composition roots.
"""
global _engine, _session_maker
# Prefer explicit parameter; fall back to ConfigManager for backwards compatibility
if config is None:
config = ConfigManager().config
if _engine is None:
_engine, _session_maker = _create_engine_and_session(db_path, db_type)
_engine, _session_maker = _create_engine_and_session(db_path, db_type, config)
# Run migrations automatically unless explicitly disabled
if ensure_migrations:
app_config = ConfigManager().config
await run_migrations(app_config, db_type)
await run_migrations(config, db_type)
# These checks should never fail since we just created the engine and session maker
# if they were None, but we'll check anyway for the type checker
@@ -181,70 +317,37 @@ async def get_or_create_db(
async def shutdown_db() -> None: # pragma: no cover
"""Clean up database connections."""
global _engine, _session_maker, _migrations_completed
global _engine, _session_maker
if _engine:
await _engine.dispose()
_engine = None
_session_maker = None
_migrations_completed = False
@asynccontextmanager
async def engine_session_factory(
db_path: Path,
db_type: DatabaseType = DatabaseType.MEMORY,
config: Optional[BasicMemoryConfig] = None,
) -> AsyncGenerator[tuple[AsyncEngine, async_sessionmaker[AsyncSession]], None]:
"""Create engine and session factory.
Note: This is primarily used for testing where we want a fresh database
for each test. For production use, use get_or_create_db() instead.
Args:
db_path: Path to database file
db_type: Type of database
config: Optional explicit config. If not provided, reads from ConfigManager.
"""
global _engine, _session_maker, _migrations_completed
global _engine, _session_maker
db_url = DatabaseType.get_db_url(db_path, db_type)
logger.debug(f"Creating engine for db_url: {db_url}")
# Configure connection args with Windows-specific settings
connect_args: dict[str, bool | float | None] = {"check_same_thread": False}
# Add Windows-specific parameters to improve reliability
if os.name == "nt": # Windows
connect_args.update(
{
"timeout": 30.0, # Increase timeout to 30 seconds for Windows
"isolation_level": None, # Use autocommit mode
}
)
# Use NullPool for Windows filesystem databases to avoid connection pooling issues
# Important: Do NOT use NullPool for in-memory databases as it will destroy the database
# between connections
if db_type == DatabaseType.FILESYSTEM:
_engine = create_async_engine(
db_url,
connect_args=connect_args,
poolclass=NullPool, # Disable connection pooling on Windows
echo=False,
)
else:
# In-memory databases need connection pooling to maintain state
_engine = create_async_engine(db_url, connect_args=connect_args)
else:
_engine = create_async_engine(db_url, connect_args=connect_args)
# Enable WAL mode for better concurrency and reliability
# Note: WAL mode is not supported for in-memory databases
enable_wal = db_type != DatabaseType.MEMORY
@event.listens_for(_engine.sync_engine, "connect")
def enable_wal_mode(dbapi_conn, connection_record):
"""Enable WAL mode on each connection."""
_configure_sqlite_connection(dbapi_conn, enable_wal=enable_wal)
# Use the same helper function as production code
_engine, _session_maker = _create_engine_and_session(db_path, db_type, config)
try:
_session_maker = async_sessionmaker(_engine, expire_on_commit=False)
# Verify that engine and session maker are initialized
if _engine is None: # pragma: no cover
logger.error("Database engine is None in engine_session_factory")
@@ -260,20 +363,16 @@ async def engine_session_factory(
await _engine.dispose()
_engine = None
_session_maker = None
_migrations_completed = False
async def run_migrations(
app_config: BasicMemoryConfig, database_type=DatabaseType.FILESYSTEM, force: bool = False
app_config: BasicMemoryConfig, database_type=DatabaseType.FILESYSTEM
): # pragma: no cover
"""Run any pending alembic migrations."""
global _migrations_completed
# Skip if migrations already completed unless forced
if _migrations_completed and not force:
logger.debug("Migrations already completed in this session, skipping")
return
"""Run any pending alembic migrations.
Note: Alembic tracks which migrations have been applied via the alembic_version table,
so it's safe to call this multiple times - it will only run pending migrations.
"""
logger.info("Running database migrations...")
try:
# Get the absolute path to the alembic directory relative to this file
@@ -288,9 +387,11 @@ async def run_migrations(
)
config.set_main_option("timezone", "UTC")
config.set_main_option("revision_environment", "false")
config.set_main_option(
"sqlalchemy.url", DatabaseType.get_db_url(app_config.database_path, database_type)
)
# Get the correct database URL based on backend configuration
# No URL conversion needed - env.py now handles both async and sync engines
db_url = DatabaseType.get_db_url(app_config.database_path, database_type, app_config)
config.set_main_option("sqlalchemy.url", db_url)
command.upgrade(config, "head")
logger.info("Migrations completed successfully")
@@ -301,12 +402,17 @@ async def run_migrations(
else:
session_maker = _session_maker
# initialize the search Index schema
# the project_id is not used for init_search_index, so we pass a dummy value
await SearchRepository(session_maker, 1).init_search_index()
# Mark migrations as completed
_migrations_completed = True
# Initialize the search index schema
# For SQLite: Create FTS5 virtual table
# For Postgres: No-op (tsvector column added by migrations)
# The project_id is not used for init_search_index, so we pass a dummy value
if (
database_type == DatabaseType.POSTGRES
or app_config.database_backend == DatabaseBackend.POSTGRES
):
await PostgresSearchRepository(session_maker, 1).init_search_index()
else:
await SQLiteSearchRepository(session_maker, 1).init_search_index()
except Exception as e: # pragma: no cover
logger.error(f"Error running migrations: {e}")
raise
+12 -405
View File
@@ -1,409 +1,16 @@
"""Dependency injection functions for basic-memory services."""
from typing import Annotated
from loguru import logger
from fastapi import Depends, HTTPException, Path, status, Request
from sqlalchemy.ext.asyncio import (
AsyncSession,
AsyncEngine,
async_sessionmaker,
)
import pathlib
from basic_memory import db
from basic_memory.config import ProjectConfig, BasicMemoryConfig, ConfigManager
from basic_memory.importers import (
ChatGPTImporter,
ClaudeConversationsImporter,
ClaudeProjectsImporter,
MemoryJsonImporter,
)
from basic_memory.markdown import EntityParser
from basic_memory.markdown.markdown_processor import MarkdownProcessor
from basic_memory.repository.entity_repository import EntityRepository
from basic_memory.repository.observation_repository import ObservationRepository
from basic_memory.repository.project_repository import ProjectRepository
from basic_memory.repository.relation_repository import RelationRepository
from basic_memory.repository.search_repository import SearchRepository
from basic_memory.services import EntityService, ProjectService
from basic_memory.services.context_service import ContextService
from basic_memory.services.directory_service import DirectoryService
from basic_memory.services.file_service import FileService
from basic_memory.services.link_resolver import LinkResolver
from basic_memory.services.search_service import SearchService
from basic_memory.sync import SyncService
from basic_memory.utils import generate_permalink
def get_app_config() -> BasicMemoryConfig: # pragma: no cover
app_config = ConfigManager().config
return app_config
AppConfigDep = Annotated[BasicMemoryConfig, Depends(get_app_config)] # pragma: no cover
## project
async def get_project_config(
project: "ProjectPathDep", project_repository: "ProjectRepositoryDep"
) -> ProjectConfig: # pragma: no cover
"""Get the current project referenced from request state.
Args:
request: The current request object
project_repository: Repository for project operations
Returns:
The resolved project config
Raises:
HTTPException: If project is not found
"""
# Convert project name to permalink for lookup
project_permalink = generate_permalink(str(project))
project_obj = await project_repository.get_by_permalink(project_permalink)
if project_obj:
return ProjectConfig(name=project_obj.name, home=pathlib.Path(project_obj.path))
# Not found
raise HTTPException( # pragma: no cover
status_code=status.HTTP_404_NOT_FOUND, detail=f"Project '{project}' not found."
)
ProjectConfigDep = Annotated[ProjectConfig, Depends(get_project_config)] # pragma: no cover
## sqlalchemy
async def get_engine_factory(
request: Request,
) -> tuple[AsyncEngine, async_sessionmaker[AsyncSession]]: # pragma: no cover
"""Get cached engine and session maker from app state.
For API requests, returns cached connections from app.state for optimal performance.
For non-API contexts (CLI), falls back to direct database connection.
"""
# Try to get cached connections from app state (API context)
if (
hasattr(request, "app")
and hasattr(request.app.state, "engine")
and hasattr(request.app.state, "session_maker")
):
return request.app.state.engine, request.app.state.session_maker
# Fallback for non-API contexts (CLI)
logger.debug("Using fallback database connection for non-API context")
app_config = get_app_config()
engine, session_maker = await db.get_or_create_db(app_config.database_path)
return engine, session_maker
EngineFactoryDep = Annotated[
tuple[AsyncEngine, async_sessionmaker[AsyncSession]], Depends(get_engine_factory)
]
async def get_session_maker(engine_factory: EngineFactoryDep) -> async_sessionmaker[AsyncSession]:
"""Get session maker."""
_, session_maker = engine_factory
return session_maker
SessionMakerDep = Annotated[async_sessionmaker, Depends(get_session_maker)]
## repositories
async def get_project_repository(
session_maker: SessionMakerDep,
) -> ProjectRepository:
"""Get the project repository."""
return ProjectRepository(session_maker)
ProjectRepositoryDep = Annotated[ProjectRepository, Depends(get_project_repository)]
ProjectPathDep = Annotated[str, Path()] # Use Path dependency to extract from URL
async def get_project_id(
project_repository: ProjectRepositoryDep,
project: ProjectPathDep,
) -> int:
"""Get the current project ID from request state.
When using sub-applications with /{project} mounting, the project value
is stored in request.state by middleware.
Args:
request: The current request object
project_repository: Repository for project operations
Returns:
The resolved project ID
Raises:
HTTPException: If project is not found
"""
# Convert project name to permalink for lookup
project_permalink = generate_permalink(str(project))
project_obj = await project_repository.get_by_permalink(project_permalink)
if project_obj:
return project_obj.id
# Try by name if permalink lookup fails
project_obj = await project_repository.get_by_name(str(project)) # pragma: no cover
if project_obj: # pragma: no cover
return project_obj.id
# Not found
raise HTTPException( # pragma: no cover
status_code=status.HTTP_404_NOT_FOUND, detail=f"Project '{project}' not found."
)
"""Dependency injection functions for basic-memory services.
DEPRECATED: This module is a backwards-compatibility shim.
Import from basic_memory.deps package submodules instead:
- basic_memory.deps.config for configuration
- basic_memory.deps.db for database/session
- basic_memory.deps.projects for project resolution
- basic_memory.deps.repositories for data access
- basic_memory.deps.services for business logic
- basic_memory.deps.importers for import functionality
This file will be removed once all callers are migrated.
"""
The project_id dependency is used in the following:
- EntityRepository
- ObservationRepository
- RelationRepository
- SearchRepository
- ProjectInfoRepository
"""
ProjectIdDep = Annotated[int, Depends(get_project_id)]
async def get_entity_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
) -> EntityRepository:
"""Create an EntityRepository instance for the current project."""
return EntityRepository(session_maker, project_id=project_id)
EntityRepositoryDep = Annotated[EntityRepository, Depends(get_entity_repository)]
async def get_observation_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
) -> ObservationRepository:
"""Create an ObservationRepository instance for the current project."""
return ObservationRepository(session_maker, project_id=project_id)
ObservationRepositoryDep = Annotated[ObservationRepository, Depends(get_observation_repository)]
async def get_relation_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
) -> RelationRepository:
"""Create a RelationRepository instance for the current project."""
return RelationRepository(session_maker, project_id=project_id)
RelationRepositoryDep = Annotated[RelationRepository, Depends(get_relation_repository)]
async def get_search_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
) -> SearchRepository:
"""Create a SearchRepository instance for the current project."""
return SearchRepository(session_maker, project_id=project_id)
SearchRepositoryDep = Annotated[SearchRepository, Depends(get_search_repository)]
# ProjectInfoRepository is deprecated and will be removed in a future version.
# Use ProjectRepository instead, which has the same functionality plus more project-specific operations.
## services
async def get_entity_parser(project_config: ProjectConfigDep) -> EntityParser:
return EntityParser(project_config.home)
EntityParserDep = Annotated["EntityParser", Depends(get_entity_parser)]
async def get_markdown_processor(entity_parser: EntityParserDep) -> MarkdownProcessor:
return MarkdownProcessor(entity_parser)
MarkdownProcessorDep = Annotated[MarkdownProcessor, Depends(get_markdown_processor)]
async def get_file_service(
project_config: ProjectConfigDep, markdown_processor: MarkdownProcessorDep
) -> FileService:
logger.debug(
f"Creating FileService for project: {project_config.name}, base_path: {project_config.home}"
)
file_service = FileService(project_config.home, markdown_processor)
logger.debug(f"Created FileService for project: {file_service} ")
return file_service
FileServiceDep = Annotated[FileService, Depends(get_file_service)]
async def get_entity_service(
entity_repository: EntityRepositoryDep,
observation_repository: ObservationRepositoryDep,
relation_repository: RelationRepositoryDep,
entity_parser: EntityParserDep,
file_service: FileServiceDep,
link_resolver: "LinkResolverDep",
app_config: AppConfigDep,
) -> EntityService:
"""Create EntityService with repository."""
return EntityService(
entity_repository=entity_repository,
observation_repository=observation_repository,
relation_repository=relation_repository,
entity_parser=entity_parser,
file_service=file_service,
link_resolver=link_resolver,
app_config=app_config,
)
EntityServiceDep = Annotated[EntityService, Depends(get_entity_service)]
async def get_search_service(
search_repository: SearchRepositoryDep,
entity_repository: EntityRepositoryDep,
file_service: FileServiceDep,
) -> SearchService:
"""Create SearchService with dependencies."""
return SearchService(search_repository, entity_repository, file_service)
SearchServiceDep = Annotated[SearchService, Depends(get_search_service)]
async def get_link_resolver(
entity_repository: EntityRepositoryDep, search_service: SearchServiceDep
) -> LinkResolver:
return LinkResolver(entity_repository=entity_repository, search_service=search_service)
LinkResolverDep = Annotated[LinkResolver, Depends(get_link_resolver)]
async def get_context_service(
search_repository: SearchRepositoryDep,
entity_repository: EntityRepositoryDep,
observation_repository: ObservationRepositoryDep,
) -> ContextService:
return ContextService(
search_repository=search_repository,
entity_repository=entity_repository,
observation_repository=observation_repository,
)
ContextServiceDep = Annotated[ContextService, Depends(get_context_service)]
async def get_sync_service(
app_config: AppConfigDep,
entity_service: EntityServiceDep,
entity_parser: EntityParserDep,
entity_repository: EntityRepositoryDep,
relation_repository: RelationRepositoryDep,
search_service: SearchServiceDep,
file_service: FileServiceDep,
) -> SyncService: # pragma: no cover
"""
:rtype: object
"""
return SyncService(
app_config=app_config,
entity_service=entity_service,
entity_parser=entity_parser,
entity_repository=entity_repository,
relation_repository=relation_repository,
search_service=search_service,
file_service=file_service,
)
SyncServiceDep = Annotated[SyncService, Depends(get_sync_service)]
async def get_project_service(
project_repository: ProjectRepositoryDep,
) -> ProjectService:
"""Create ProjectService with repository."""
return ProjectService(repository=project_repository)
ProjectServiceDep = Annotated[ProjectService, Depends(get_project_service)]
async def get_directory_service(
entity_repository: EntityRepositoryDep,
) -> DirectoryService:
"""Create DirectoryService with dependencies."""
return DirectoryService(
entity_repository=entity_repository,
)
DirectoryServiceDep = Annotated[DirectoryService, Depends(get_directory_service)]
# Import
async def get_chatgpt_importer(
project_config: ProjectConfigDep, markdown_processor: MarkdownProcessorDep
) -> ChatGPTImporter:
"""Create ChatGPTImporter with dependencies."""
return ChatGPTImporter(project_config.home, markdown_processor)
ChatGPTImporterDep = Annotated[ChatGPTImporter, Depends(get_chatgpt_importer)]
async def get_claude_conversations_importer(
project_config: ProjectConfigDep, markdown_processor: MarkdownProcessorDep
) -> ClaudeConversationsImporter:
"""Create ChatGPTImporter with dependencies."""
return ClaudeConversationsImporter(project_config.home, markdown_processor)
ClaudeConversationsImporterDep = Annotated[
ClaudeConversationsImporter, Depends(get_claude_conversations_importer)
]
async def get_claude_projects_importer(
project_config: ProjectConfigDep, markdown_processor: MarkdownProcessorDep
) -> ClaudeProjectsImporter:
"""Create ChatGPTImporter with dependencies."""
return ClaudeProjectsImporter(project_config.home, markdown_processor)
ClaudeProjectsImporterDep = Annotated[ClaudeProjectsImporter, Depends(get_claude_projects_importer)]
async def get_memory_json_importer(
project_config: ProjectConfigDep, markdown_processor: MarkdownProcessorDep
) -> MemoryJsonImporter:
"""Create ChatGPTImporter with dependencies."""
return MemoryJsonImporter(project_config.home, markdown_processor)
MemoryJsonImporterDep = Annotated[MemoryJsonImporter, Depends(get_memory_json_importer)]
# Re-export everything from the deps package for backwards compatibility
from basic_memory.deps import * # noqa: F401, F403 # pragma: no cover

Some files were not shown because too many files have changed in this diff Show More