Compare commits

..

85 Commits

Author SHA1 Message Date
phernandez eeeade4f07 chore: update version to 0.17.4 for v0.17.4 release 2026-01-05 21:38:03 -06:00
phernandez 03793eaf7c docs: add v0.17.4 changelog entry
- Critical bug fix for search index preservation (#503)
- Major architecture refactor with composition roots (#502)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-05 21:36:14 -06:00
Paul Hernandez 26f7e98932 fix: preserve search index across server restarts (#503)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2026-01-05 21:31:35 -06:00
Paul Hernandez 5947f04bd3 refactor: composition roots, deps split, and typed API clients (#490 roadmap) (#502)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 11:05:58 -06:00
Drew Cain ba1439fefc chore: update version to 0.17.3 for v0.17.3 release 2026-01-03 13:08:11 -06:00
Drew Cain ef411ceb12 docs: add v0.17.3 changelog entry
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 13:07:24 -06:00
Drew Cain c6baf58aa7 fix: update mcp to support protocol version 2025-11-25 (#501)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 12:39:11 -06:00
phernandez 3c1748cc89 return external_id with directory response in tree
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-02 23:44:03 -06:00
phernandez 9206e7960a return external_id with directory response
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-02 23:43:14 -06:00
Paul Hernandez 53c4c20d22 fix: route ordering for cloud (#499)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-02 21:52:02 -06:00
Paul Hernandez b4486d20bd test: remove stdlib mocks, strengthen integration coverage (#489)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-02 14:22:01 -06:00
jope-bm a4000f64ce feat: add stable external_id (UUID) to Project and Entity models (#485)
Signed-off-by: Joe P <joe@basicmemory.com>
Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: phernandez <paul@basicmachines.co>
2026-01-02 13:00:35 -06:00
phernandez 88a1778798 fix(importers): return ImportResult from handle_error
Implement handle_error() in all importers so errors return a concrete ImportResult (success=false, error_message set) instead of None, preventing NoneType.success crashes.

Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-01 11:47:42 -06:00
phernandez 4ce21984a4 fix: use upsert to prevent IntegrityError during parallel search indexing
Replace delete-then-insert pattern with INSERT ... ON CONFLICT for
PostgreSQL search index operations. This fixes race conditions where
parallel entity indexing could cause UniqueViolationError on the
uix_search_index_permalink_project constraint.

Changes:
- Add index_item() override in PostgresSearchRepository with upsert
- Update bulk_index_items() to use ON CONFLICT (permalink, project_id)
- Add CREATE_POSTGRES_SEARCH_INDEX_PERMALINK DDL for test fixtures
- Add tests for upsert behavior on duplicate permalinks

Technical note: Use column-based ON CONFLICT syntax instead of
ON CONSTRAINT (which only works for table constraints, not indexes).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-30 21:55:43 -06:00
phernandez eb7fbaf0bf fix set_default_project in cloud mode
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-30 20:57:45 -06:00
phernandez 8adf1f4ed4 fix: use relative file paths in importers for cloud storage compatibility
Importers now use relative file paths (based on permalink) instead of
absolute paths. This enables proper S3 key generation in cloud environments.

Changes:
- write_entity() accepts str | Path for file_path parameter
- ensure_folder_exists() uses relative paths directly
- All importers pass relative paths to FileService
- FileService handles base_path resolution internally

This prevents S3 keys from including container filesystem paths like
`/app/basic-memory/imports/...` and instead uses clean relative paths
like `imports/20241010-file.md`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-30 20:29:01 -06:00
phernandez 45ce1813e4 refactor: use FileService in importers for cloud compatibility
Refactor importers to use FileService for all file operations instead of
direct filesystem calls. This enables cloud environments to override file
operations via dependency injection (e.g., S3FileService).

Changes:
- Add `to_markdown_string()` method to MarkdownProcessor for content
  serialization without file I/O
- Update Importer base class to accept FileService and use it for:
  - `write_entity()` - now uses FileService.write_file()
  - `ensure_folder_exists()` - now async, uses FileService.ensure_directory()
- Fix direct `mkdir()` calls in:
  - claude_projects_importer.py
  - memory_json_importer.py
- Update deps.py to inject FileService into all importers (v1 and v2)
- Update CLI commands to create and pass FileService to importers
- Update tests to work with new FileService dependency

This follows the pattern used by /knowledge API and SyncService, enabling
cloud to override file operations by providing S3FileService via DI.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-30 19:21:32 -06:00
phernandez 2744c4b6a5 add info logging to index_entity_data background task
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-30 12:04:49 -06:00
Paul Hernandez fd732aa6fe fix: set_default_project skips config file update in cloud mode (#486)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-30 09:05:48 -06:00
Paul Hernandez 537e58ad7d fix: make RelationResponse.from_id optional to handle null permalinks (#484)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-30 08:12:29 -06:00
phernandez 48e6e84beb chore: update version to 0.17.2 for v0.17.2 release 2025-12-29 16:47:04 -06:00
phernandez 02c14acddb docs: add CHANGELOG entry for v0.17.2
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-29 16:46:26 -06:00
phernandez 0b5425f163 don't run full tests in release, just lint, typecheck
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-29 16:33:38 -06:00
phernandez 0bcda4a14a fix: allow recent_activity discovery mode in cloud mode
Add `allow_discovery` parameter to `resolve_project_parameter()` that
enables tools like `recent_activity` to work across all projects in
cloud mode without requiring a project parameter.

- Add `allow_discovery: bool = False` param to resolve_project_parameter
- In cloud mode with allow_discovery=True, return None instead of error
- Update recent_activity to use allow_discovery=True
- Fix circular import by deferring call_get import inside functions
- Add comprehensive tests for resolve_project_parameter

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-29 16:22:48 -06:00
phernandez 7a49f57dee chore: update version to 0.17.1 for v0.17.1 release
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-29 10:34:06 -06:00
phernandez 58db2817d2 docs: add CHANGELOG entry for v0.17.1
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-29 10:20:00 -06:00
Paul Hernandez 98fbd60527 fix: only set BASIC_MEMORY_ENV=test during pytest runs (#482)
Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-12-29 10:16:45 -06:00
phernandez 6281a81256 chore: update version to 0.17.0 for v0.17.0 release
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-28 16:06:03 -06:00
phernandez be1d0b169f style: format telemetry.py
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-28 16:05:07 -06:00
phernandez 148bf6f75a docs: add CHANGELOG entry for v0.17.0
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-28 16:03:13 -06:00
phernandez 272a983709 add foss as telementry source, disable analytics for tests
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-28 13:41:40 -06:00
phernandez ef7adb7b99 fix: add cloud_mode check to initialize_app()
MCP server crashes in cloud mode with:
ValueError: DATABASE_URL must be set when using Postgres backend

Root cause: initialize_app() did not check cloud_mode_enabled before
trying to initialize the database. Only ensure_initialization() had
the check. In cloud mode, tenant DBs are per-request via headers,
not via DATABASE_URL environment variable.

Also includes minor formatting fix in telemetry.py.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-28 10:31:17 -06:00
phernandez 3cd9178415 refactor: centralize test environment detection in config.is_test_env
Add is_test_env property to BasicMemoryConfig that checks:
- config.env == "test"
- BASIC_MEMORY_ENV env var is "test"
- PYTEST_CURRENT_TEST is set

Replace duplicated test detection logic in:
- api/app.py
- mcp/server.py
- services/initialization.py
- telemetry.py (disables telemetry during tests)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-27 12:44:42 -06:00
Paul Hernandez 856737fe3c feat: add anonymous usage telemetry (Homebrew-style opt-out) (#478)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-26 22:09:48 -06:00
Paul Hernandez 1fd680c3f1 feat: add auto-format files on save with built-in Python formatter (#474)
Signed-off-by: Cedric Hurst <cedric@spantree.net>
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Cedric Hurst <cedric@spantree.net>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Sebastian B Otaegui <feniix@users.noreply.github.com>
Co-authored-by: Cedric Hurst <cedric@divideby0.io>
2025-12-24 15:39:22 -06:00
phernandez 38919d11cb docs: update CLAUDE.md with accurate CLI commands and code guidelines
- Add Code Change Guidelines section (full file read, minimize diffs, fail fast, no guessing)
- Add Literate Programming Style section (section headers, decision point comments)
- Fix CLI commands: `tools` -> `tool`, add project management commands
- Fix cloud commands to match current CLI (status, setup)
- Remove non-existent MCP tools (get_current_project, sync_status)
- Add ChatGPT-compatible tools (search, fetch)
- Remove non-existent json_canvas_spec prompt
- Add /importers to codebase architecture
- Remove unused python-developer and system-architect agents

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-24 15:16:02 -06:00
phernandez 85684f848f fix: handle UTF-8 BOM in frontmatter parsing
Fixes #452 - Imported conversations not fully indexed

Files with UTF-8 BOM (Byte Order Mark) at the start would fail frontmatter
detection, causing:
- Title to fall back to filename instead of frontmatter value
- Permalink to be null in the database

Added strip_bom() helper function and updated all frontmatter-related
functions to strip BOM before processing:
- has_frontmatter()
- parse_frontmatter()
- remove_frontmatter()
- EntityParser.parse_markdown_content()

Added comprehensive tests for BOM handling with various scenarios.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-24 14:28:50 -06:00
Paul Hernandez 14ce5a3bd0 fix: handle null titles in ChatGPT import (#475)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-12-24 13:50:30 -06:00
phernandez 45d6caf723 fix: remove MaxLen constraint from observation content
The API Pydantic schema had a MaxLen(1000) constraint on observation
content while the database uses SQLAlchemy's Text type (unlimited).
This mismatch caused validation errors when observations exceeded
1000 characters (e.g., JSON schemas with 1458+ chars).

Removed the MaxLen constraint to match the DB schema. Retained:
- BeforeValidator(str.strip) for whitespace cleaning
- MinLen(1) to ensure non-empty content

Added comprehensive tests to verify:
- Long content (10K+ chars) is accepted
- Very long content (50K+ chars) is accepted
- Empty/whitespace-only content is still rejected
- Whitespace stripping still works

Fixes #385

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-24 13:41:28 -06:00
phernandez 1652f862dd fix: handle FileNotFoundError gracefully during sync
When a file exists in the database but is missing from the filesystem,
the sync worker now treats this as a deletion instead of crashing.

The sync_file() method catches FileNotFoundError specifically and calls
handle_delete() to clean up the orphaned database record. This prevents
the sync from failing on database/filesystem inconsistencies that can
occur due to race conditions, manual file deletions, or cloud storage
caching issues.

Includes a test to verify the graceful handling behavior.

Fixes #386

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-24 13:34:59 -06:00
phernandez c23927d124 fix: use canonical project names in API response messages
Use database-retrieved project names (new_project.name, old_project.name)
instead of input parameters (project_data.name, name) in v1 API response
messages to ensure consistent project name casing.

The v2 API already did this correctly. This fixes issue #450 where project
names would display with different casing between add and remove operations.

Fixes #450

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-24 13:28:06 -06:00
jope-bm 1a74d85973 feat: Complete Phase 2 of API v2 migration - Update MCP tools to use v2 endpoints (#447)
Signed-off-by: Joe P <joe@basicmemory.com>
Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: phernandez <paul@basicmachines.co>
2025-12-24 12:59:14 -06:00
phernandez d71c6e8568 fix: suppress CLI warnings for cleaner output
Suppress DeprecationWarning from aiosqlite and LogfireNotConfiguredWarning
that were cluttering CLI output.

The key fix is applying warnings.filterwarnings("ignore") AFTER all imports
in main.py, because authlib (imported via cloud commands) adds a
DeprecationWarning filter that overrides earlier suppressions.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-24 12:21:14 -06:00
phernandez 63b98491be fix: prevent DEBUG logs from appearing on CLI stdout
Remove loguru's default handler at the very start of cli/app.py,
before any other imports. This prevents module-level code (like
TemplateLoader.__init__) from logging to stdout during import.

The import chain cli/commands/project.py -> mcp/async_client.py ->
api/app.py -> api/routers/prompt_router.py -> api/template_loader.py
triggers TemplateLoader() instantiation at DEBUG level before
init_cli_logging() can remove the default handler.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-24 11:42:25 -06:00
Paul Hernandez 622d37e4a8 fix: detect rclone version for --create-empty-src-dirs support (#473)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-24 11:27:49 -06:00
Paul Hernandez 916baf8971 fix: prevent CLI commands from hanging on exit (#471)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-24 11:27:11 -06:00
Drew Cain 95937c6d0a fix: make test-int-postgres compatible with macOS
Use gtimeout (Homebrew) or timeout (Linux), falling back to running
without timeout if neither is available. This fixes 'command not found'
errors on macOS which doesn't have GNU timeout by default.
2025-12-20 10:27:26 -06:00
Drew Cain 24dc9a2931 chore: update version to 0.16.3 for v0.16.3 release 2025-12-20 09:53:37 -06:00
Drew Cain 85c63e5a7a docs: add CHANGELOG entry for v0.16.3 2025-12-20 09:26:00 -06:00
Drew Cain f227ef6a86 fix: Pin FastMCP to 2.12.3 to fix MCP tools visibility (#464)
Signed-off-by: Drew Cain <groksrc@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2025-12-20 08:54:44 -06:00
Paul Hernandez 897b1edaa4 fix: Reduce watch service CPU usage by increasing reload interval (#458)
Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-12-17 11:37:01 -06:00
Paul Hernandez 0c12a39a98 test: Add integration test for issue #416 (read_note with underscored folders) (#453)
Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-12-17 09:56:45 -06:00
Paul Hernandez efbc758325 fix: await background sync task cancellation in lifespan shutdown (#456)
Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-12-17 09:53:34 -06:00
Paul Hernandez a0f20eb102 chore: more Tenantless fixes (#457)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-16 18:34:05 -06:00
Paul Hernandez 78673d8e51 chore: Cloud compatibility fixes and performance improvements (#454)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-15 20:07:55 -06:00
phernandez 126c0495c0 Merge branch 'main' of github.com:basicmachines-co/basic-memory 2025-12-13 15:24:43 -06:00
Paul Hernandez 4a43d7df4a remove logfire instrumentation
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-13 15:22:14 -06:00
Paul Hernandez c462faf046 Replace py-pglite with testcontainers for Postgres testing (#449)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-10 22:17:56 -06:00
Cedric Hurst 70bb10be1d fix: respect --project flag in background sync (fixes #434) (#436)
Signed-off-by: Cedric Hurst <cedric@spantree.net>
2025-12-08 12:58:18 -06:00
phernandez fbf9045d78 use asyncpg for just db-migrate
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-05 15:01:35 -06:00
phernandez 1094210c52 fix broken sqlite migration
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-02 20:23:55 -06:00
phernandez 391feb639f add delete cascade to entity to delete search_index (postgres only)
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-02 10:04:21 -06:00
phernandez a920a9ff29 feat: Add project_id to Relation and Observation for efficient project-scoped queries
Denormalizes project_id onto Relation and Observation tables to enable
efficient project-scoped queries without joins. Migration backfills
from associated entity and adds pg_trgm extension with GIN indexes
for fuzzy link resolution on PostgreSQL.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-01 21:54:56 -06:00
phernandez 05efe8701c test: Verify update() returns entity with eager-loaded relations
Add test confirming entity_repository.update() returns the entity with
observations and relations eagerly loaded, eliminating the need for a
separate find_by_id() call after update.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-01 16:22:21 -06:00
phernandez 0eaf30bb06 remove conflict constraint name from relation_repository.py
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-30 19:31:09 -06:00
phernandez 0818bda565 feat: Add bulk insert with ON CONFLICT handling for relations
Add add_all_ignore_duplicates() method to RelationRepository for bulk
inserting relations with ON CONFLICT DO NOTHING. This handles cases
where the same [[wiki link]] appears multiple times in a document,
silently ignoring duplicates based on the (from_id, to_name, relation_type)
unique constraint.

Works with both SQLite and PostgreSQL dialects.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-30 14:59:52 -06:00
phernandez 6f99d2e551 perf: lightweight permalink resolution to avoid eager loading
Add optimized repository methods for resolve_permalink() that skip
eager loading of observations and relations:

- permalink_exists(): Check existence without loading entity
- get_file_path_for_permalink(): Get only file_path column
- get_permalink_for_file_path(): Get only permalink column
- get_all_permalinks(): Get all permalinks as strings
- get_permalink_to_file_path_map(): Bulk lookup mapping
- get_file_path_to_permalink_map(): Reverse mapping

Updated entity_service.resolve_permalink() to use these lightweight
methods instead of loading full entities with all relationships.

Also added logfire instrumentation to markdown utils.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-30 14:34:31 -06:00
phernandez 73d940e064 fix: observation parsing and permalink limits (#446)
1. Hashtag detection now checks for standalone words starting with #
   instead of just checking if # appears anywhere in content.
   This prevents HTML color codes like #4285F4 from being
   interpreted as hashtags.

2. Observation permalinks now truncate content to 200 chars
   to stay under PostgreSQL's btree index limit of 2704 bytes.

Added tests for both fixes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-30 00:12:04 -06:00
phernandez c3678a11d2 truncate content_stems to fix Postgres 8KB index row limit
Large documents (like ~1MB conversation imports) exceed Postgres's 8KB
index row limit, causing ProgramLimitExceededError. Truncate content_stems
to 6000 characters (with headroom for other columns) before indexing.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-29 20:23:09 -06:00
phernandez 203d684c24 fix integrity error handling when setting forward relation refs
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-29 19:03:12 -06:00
phernandez a872220924 disable pooling for postgres db
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-29 13:24:47 -06:00
phernandez 7d763a66ff use entity.mtime for updated at in api
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-29 13:21:58 -06:00
phernandez b5d4fb559c fix: postgres/neon connection settings and search index dedupe
- Reduce db_pool_recycle from 3600s to 180s for Neon scale-to-zero
- Add connect_args for Neon serverless (statement cache, timeouts, app name)
- Dedupe observation permalinks in search indexing to avoid unique constraint violations
- Add tests for duplicate observation permalink handling

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-29 11:24:08 -06:00
phernandez 830775276d remove record_return=True from logfire spans
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 18:36:09 -06:00
phernandez ed894fc3ed get db pool sizes from config
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 16:50:57 -06:00
phernandez 704338edcf remove logfire.instrument_fastapi(app) from app.py
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 13:52:18 -06:00
phernandez 0ca02a7ebe add logfire instrumentation to services and repository code
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 12:47:29 -06:00
jope-bm 28cc5225a7 feat: Implement API v2 with ID-based endpoints (Phase 1) (#441)
Signed-off-by: Joe P <joe@basicmemory.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: phernandez <paul@basicmachines.co>
2025-11-27 10:35:55 -06:00
phernandez 9b7bbc7116 formatting and logic change to resolve_relations, remove fuzzy search 2025-11-25 22:54:37 -06:00
phernandez 138c283d6c add postgres db type 2025-11-25 20:25:56 -06:00
phernandez 7a8954c37e add extra logic for cloud-indexing improvements 2025-11-25 13:52:58 -06:00
phernandez 10c7c19c03 fix db url for sqlite migrations
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-21 13:21:20 -06:00
Paul Hernandez fb5e9e1d77 feat: Add PostgreSQL database backend support (#439)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-11-20 11:20:29 -06:00
phernandez 66b91b2847 ci: Add PostgreSQL testing to GitHub Actions workflow
Add Postgres service container and separate test step for PostgreSQL backend testing.
The Postgres tests only run on Linux runners since GitHub Actions service containers
are only available on Linux.

- Add postgres:17 service container with health checks
- Add 'Run tests (Postgres)' step with Linux-only condition
- Rename existing test step to 'Run tests (SQLite)' for clarity

This enables CI testing of dual database backend support introduced in the
postgres-support feature branch.

Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-18 12:25:18 -06:00
Cedric Hurst b004565df9 fix: handle periods in kebab_filenames mode (#424) 2025-11-18 06:49:55 -05:00
276 changed files with 23708 additions and 15147 deletions
-154
View File
@@ -1,154 +0,0 @@
---
name: python-developer
description: Python backend developer specializing in FastAPI, DBOS workflows, and API implementation. Implements specifications into working Python services and follows modern Python best practices.
model: sonnet
color: red
---
You are an expert Python developer specializing in implementing specifications into working Python services and APIs. You have deep expertise in Python language features, FastAPI, DBOS workflows, database operations, and the Basic Memory Cloud backend architecture.
**Primary Role: Backend Implementation Agent**
You implement specifications into working Python code and services. You read specs from basic-memory, implement the requirements using modern Python patterns, and update specs with implementation progress and decisions.
**Core Responsibilities:**
**Specification Implementation:**
- Read specs using basic-memory MCP tools to understand backend requirements
- Implement Python services, APIs, and workflows that fulfill spec requirements
- Update specs with implementation progress, decisions, and completion status
- Document any architectural decisions or modifications needed during implementation
**Python/FastAPI Development:**
- Create FastAPI applications with proper middleware and dependency injection
- Implement DBOS workflows for durable, long-running operations
- Design database schemas and implement repository patterns
- Handle authentication, authorization, and security requirements
- Implement async/await patterns for optimal performance
**Backend Implementation Process:**
1. **Read Spec**: Use `mcp__basic-memory__read_note` to get spec requirements
2. **Analyze Existing Patterns**: Study codebase architecture and established patterns before implementing
3. **Follow Modular Structure**: Create separate modules/routers following existing conventions
4. **Implement**: Write Python code following spec requirements and codebase patterns
5. **Test**: Create tests that validate spec success criteria
6. **Update Spec**: Document completion and any implementation decisions
7. **Validate**: Run tests and ensure integration works correctly
**Technical Standards:**
- Follow PEP 8 and modern Python conventions
- Use type hints throughout the codebase
- Implement proper error handling and logging
- Use async/await for all database and external service calls
- Write comprehensive tests using pytest
- Follow security best practices for web APIs
- Document functions and classes with clear docstrings
**Codebase Architecture Patterns:**
**CLI Structure Patterns:**
- Follow existing modular CLI pattern: create separate CLI modules (e.g., `upload_cli.py`) instead of adding commands directly to `main.py`
- Existing examples: `polar_cli.py`, `tenant_cli.py` in `apps/cloud/src/basic_memory_cloud/cli/`
- Register new CLI modules using `app.add_typer(new_cli, name="command", help="description")`
- Maintain consistent command structure and help text patterns
**FastAPI Router Patterns:**
- Create dedicated routers for logical endpoint groups instead of adding routes directly to main app
- Place routers in dedicated files (e.g., `apps/api/src/basic_memory_cloud_api/routers/webdav_router.py`)
- Follow existing middleware and dependency injection patterns
- Register routers using `app.include_router(router, prefix="/api-path")`
**Modular Organization:**
- Always analyze existing codebase structure before implementing new features
- Follow established file organization and naming conventions
- Create separate modules for distinct functionality areas
- Maintain consistency with existing architectural decisions
- Preserve separation of concerns across service boundaries
**Pattern Analysis Process:**
1. Examine similar existing functionality in the codebase
2. Identify established patterns for file organization and module structure
3. Follow the same architectural approach for consistency
4. Create new modules/routers following existing conventions
5. Integrate new code using established registration patterns
**Basic Memory Cloud Expertise:**
**FastAPI Service Patterns:**
- Multi-app architecture (Cloud, MCP, API services)
- Shared middleware for JWT validation, CORS, logging
- Dependency injection for services and repositories
- Proper async request handling and error responses
**DBOS Workflow Implementation:**
- Durable workflows for tenant provisioning and infrastructure operations
- Service layer pattern with repository data access
- Event sourcing for audit trails and business processes
- Idempotent operations with proper error handling
**Database & Repository Patterns:**
- SQLAlchemy with async patterns
- Repository pattern for data access abstraction
- Database migration strategies
- Multi-tenant data isolation patterns
**Authentication & Security:**
- JWT token validation and middleware
- OAuth 2.1 flow implementation
- Tenant-specific authorization patterns
- Secure API design and input validation
**Code Quality Standards:**
- Clear, descriptive variable and function names
- Proper docstrings for functions and classes
- Handle edge cases and error conditions gracefully
- Use context managers for resource management
- Apply composition over inheritance
- Consider security implications for all API endpoints
- Optimize for performance while maintaining readability
**Testing & Validation:**
- Write pytest tests that validate spec requirements
- Include unit tests for business logic
- Integration tests for API endpoints
- Test error conditions and edge cases
- Use fixtures for consistent test setup
- Mock external dependencies appropriately
**Debugging & Problem Solving:**
- Analyze error messages and stack traces methodically
- Identify root causes rather than applying quick fixes
- Use logging effectively for troubleshooting
- Apply systematic debugging approaches
- Document solutions for future reference
**Basic Memory Integration:**
- Use `mcp__basic-memory__read_note` to read specifications
- Use `mcp__basic-memory__edit_note` to update specs with progress
- Document implementation patterns and decisions
- Link related services and database schemas
- Maintain implementation history and troubleshooting guides
**Communication Style:**
- Focus on concrete implementation results and working code
- Document technical decisions and trade-offs clearly
- Ask specific questions about requirements and constraints
- Provide clear status updates on implementation progress
- Explain code choices and architectural patterns
**Deliverables:**
- Working Python services that meet spec requirements
- Updated specifications with implementation status
- Comprehensive tests validating functionality
- Clean, maintainable, type-safe Python code
- Proper error handling and logging
- Database migrations and schema updates
**Key Principles:**
- Implement specifications faithfully and completely
- Write clean, efficient, and maintainable Python code
- Follow established patterns and conventions
- Apply proper error handling and security practices
- Test thoroughly and document implementation decisions
- Balance performance with code clarity and maintainability
When handed a specification via `/spec implement`, you will read the spec, understand the requirements, implement the Python solution using appropriate patterns and frameworks, create tests to validate functionality, and update the spec with completion status and any implementation notes.
-126
View File
@@ -1,126 +0,0 @@
---
name: system-architect
description: System architect who designs and implements architectural solutions, creates ADRs, and applies software engineering principles to solve complex system design problems.
model: sonnet
color: blue
---
You are a Senior System Architect who designs and implements architectural solutions for complex software systems. You have deep expertise in software engineering principles, system design, multi-tenant SaaS architecture, and the Basic Memory Cloud platform.
**Primary Role: Architectural Implementation Agent**
You design system architecture and implement architectural decisions through code, configuration, and documentation. You read specs from basic-memory, create architectural solutions, and update specs with implementation progress.
**Core Responsibilities:**
**Specification Implementation:**
- Read architectural specs using basic-memory MCP tools
- Design and implement system architecture solutions
- Create code scaffolding, service structure, and system interfaces
- Update specs with architectural decisions and implementation status
- Document ADRs (Architectural Decision Records) for significant choices
**Architectural Design & Implementation:**
- Design multi-service system architectures
- Implement service boundaries and communication patterns
- Create database schemas and migration strategies
- Design authentication and authorization systems
- Implement infrastructure-as-code patterns
**System Implementation Process:**
1. **Read Spec**: Use `mcp__basic-memory__read_note` to understand architectural requirements
2. **Design Solution**: Apply architectural principles and patterns
3. **Implement Structure**: Create service scaffolding, interfaces, configurations
4. **Document Decisions**: Create ADRs documenting architectural choices
5. **Update Spec**: Record implementation progress and decisions
6. **Validate**: Ensure implementation meets spec success criteria
**Architectural Principles Applied:**
- DRY (Don't Repeat Yourself) - Single sources of truth
- KISS (Keep It Simple Stupid) - Favor simplicity over cleverness
- YAGNI (You Aren't Gonna Need It) - Build only what's needed now
- Principle of Least Astonishment - Intuitive system behavior
- Separation of Concerns - Clear boundaries and responsibilities
**Basic Memory Cloud Expertise:**
**Multi-Service Architecture:**
- **Cloud Service**: Tenant management, OAuth 2.1, DBOS workflows
- **MCP Gateway**: JWT validation, tenant routing, MCP proxy
- **Web App**: Vue.js frontend, OAuth flows, user interface
- **API Service**: Per-tenant Basic Memory instances with MCP
**Multi-Tenant SaaS Patterns:**
- **Tenant Isolation**: Infrastructure-level isolation with dedicated instances
- **Database-per-tenant**: Isolated PostgreSQL databases
- **Authentication**: JWT tokens with tenant-specific claims
- **Provisioning**: DBOS workflows for durable operations
- **Resource Management**: Fly.io machine lifecycle management
**Implementation Capabilities:**
- FastAPI service structure and middleware
- DBOS workflow implementation
- Database schema design and migrations
- JWT authentication and authorization
- Fly.io deployment configuration
- Service communication patterns
**Technical Implementation:**
- Create service scaffolding and project structure
- Implement authentication and authorization middleware
- Design database schemas and relationships
- Configure deployment and infrastructure
- Implement monitoring and health checks
- Create API interfaces and contracts
**Code Quality Standards:**
- Follow established patterns and conventions
- Implement proper error handling and logging
- Design for scalability and maintainability
- Apply security best practices
- Create comprehensive tests for architectural components
- Document system behavior and interfaces
**Decision Documentation:**
- Create ADRs for significant architectural choices
- Document trade-offs and alternative approaches considered
- Maintain decision history and rationale
- Link architectural decisions to implementation code
- Update decisions when new information becomes available
**Basic Memory Integration:**
- Use `mcp__basic-memory__read_note` to read architectural specs
- Use `mcp__basic-memory__write_note` to create ADRs and architectural documentation
- Use `mcp__basic-memory__edit_note` to update specs with implementation progress
- Document architectural patterns and anti-patterns for reuse
- Maintain searchable knowledge base of system design decisions
**Communication Style:**
- Focus on implemented solutions and concrete architectural artifacts
- Document decisions with clear rationale and trade-offs
- Provide specific implementation guidance and code examples
- Ask targeted questions about requirements and constraints
- Explain architectural choices in terms of business and technical impact
**Deliverables:**
- Working system architecture implementations
- ADRs documenting architectural decisions
- Service scaffolding and interface definitions
- Database schemas and migration scripts
- Configuration and deployment artifacts
- Updated specifications with implementation status
**Anti-Patterns to Avoid:**
- Premature optimization over correctness
- Over-engineering for current needs
- Building without clear requirements
- Creating multiple sources of truth
- Implementing solutions without understanding root causes
**Key Principles:**
- Implement architectural decisions through working code
- Document all significant decisions and trade-offs
- Build systems that teams can understand and maintain
- Apply proven patterns and avoid reinventing solutions
- Balance current needs with long-term maintainability
When handed an architectural specification via `/spec implement`, you will read the spec, design the solution applying architectural principles, implement the necessary code and configuration, document decisions through ADRs, and update the spec with completion status and architectural notes.
+16 -20
View File
@@ -1,17 +1,19 @@
---
allowed-tools: mcp__basic-memory__write_note, mcp__basic-memory__read_note, mcp__basic-memory__search_notes, mcp__basic-memory__edit_note, Task
argument-hint: [create|status|implement|review] [spec-name]
allowed-tools: mcp__basic-memory__write_note, mcp__basic-memory__read_note, mcp__basic-memory__search_notes, mcp__basic-memory__edit_note
argument-hint: [create|status|show|review] [spec-name]
description: Manage specifications in our development process
---
## Context
You are managing specifications using our specification-driven development process defined in @docs/specs/SPEC-001.md.
Specifications are managed in the Basic Memory "specs" project. All specs live in a centralized location accessible across all repositories via MCP tools.
See SPEC-1 and SPEC-2 in the "specs" project for the full specification-driven development process.
Available commands:
- `create [name]` - Create new specification
- `status` - Show all spec statuses
- `implement [spec-name]` - Hand spec to appropriate agent
- `show [spec-name]` - Read a specific spec
- `review [spec-name]` - Review implementation against spec
## Your task
@@ -19,23 +21,19 @@ Available commands:
Execute the spec command: `/spec $ARGUMENTS`
### If command is "create":
1. Get next SPEC number by searching existing specs
2. Create new spec using template from @docs/specs/Slash\ Commands\ Reference.md
3. Place in `/specs` folder with title "SPEC-XXX: [name]"
1. Get next SPEC number by searching existing specs in "specs" project
2. Create new spec using template from SPEC-2
3. Use mcp__basic-memory__write_note with project="specs"
4. Include standard sections: Why, What, How, How to Evaluate
### If command is "status":
1. Search all notes in `/specs` folder
2. Display table with spec number, title, and status
3. Show any dependencies or assigned agents
1. Use mcp__basic-memory__search_notes with project="specs"
2. Display table with spec number, title, and progress
3. Show completion status from checkboxes in content
### If command is "implement":
1. Read the specified spec
2. Determine appropriate agent based on content:
- Frontend/UI → vue-developer
- Architecture/system → system-architect
- Backend/API → python-developer
3. Launch Task tool with appropriate agent and spec context
### If command is "show":
1. Use mcp__basic-memory__read_note with project="specs"
2. Display the full spec content
### If command is "review":
1. Read the specified spec and its "How to Evaluate" section
@@ -49,7 +47,5 @@ Execute the spec command: `/spec $ARGUMENTS`
- **Architecture compliance** - Component isolation, state management patterns
- **Documentation completeness** - Implementation matches specification
3. Provide honest, accurate assessment - do not overstate completeness
4. Document findings and update spec with review results
4. Document findings and update spec with review results using mcp__basic-memory__edit_note
5. If gaps found, clearly identify what still needs to be implemented/tested
Use the agent definitions from @docs/specs/Agent\ Definitions.md for implementation handoffs.
+5
View File
@@ -0,0 +1,5 @@
{
"enabledPlugins": {
"basic-memory@basicmachines": true
}
}
+28
View File
@@ -0,0 +1,28 @@
# Basic Memory Environment Variables Example
# Copy this file to .env and customize as needed
# Note: .env files are gitignored and should never be committed
# ============================================================================
# PostgreSQL Test Database Configuration
# ============================================================================
# These variables allow you to override the default test database credentials
# Default values match docker-compose-postgres.yml for local development
#
# Only needed if you want to use different credentials or a remote test database
# By default, tests use: postgresql://basic_memory_user:dev_password@localhost:5433/basic_memory_test
# Full PostgreSQL test database URL (used by tests and migrations)
# POSTGRES_TEST_URL=postgresql+asyncpg://basic_memory_user:dev_password@localhost:5433/basic_memory_test
# Individual components (used by justfile postgres-reset command)
# POSTGRES_USER=basic_memory_user
# POSTGRES_TEST_DB=basic_memory_test
# ============================================================================
# Production Database Configuration
# ============================================================================
# For production use, set these in your deployment environment
# DO NOT use the test credentials above in production!
# BASIC_MEMORY_DATABASE_BACKEND=postgres # or "sqlite"
# BASIC_MEMORY_DATABASE_URL=postgresql+asyncpg://user:password@host:port/database
+1
View File
@@ -54,6 +54,7 @@ jobs:
- [ ] Unit tests for new functions/methods
- [ ] Integration tests for new MCP tools
- [ ] Test coverage for edge cases
- [ ] **100% test coverage maintained** (use `# pragma: no cover` only for truly hard-to-test code)
- [ ] Documentation updated (README, docstrings)
- [ ] CLAUDE.md updated if conventions change
+100 -3
View File
@@ -13,7 +13,8 @@ on:
branches: [ "main" ]
jobs:
test:
test-sqlite:
name: Test SQLite (${{ matrix.os }}, Python ${{ matrix.python-version }})
strategy:
fail-fast: false
matrix:
@@ -64,7 +65,103 @@ jobs:
run: |
just lint
- name: Run tests
- name: Run tests (SQLite)
run: |
uv pip install pytest pytest-cov
just test
just test-sqlite
test-postgres:
name: Test Postgres (Python ${{ matrix.python-version }})
strategy:
fail-fast: false
matrix:
python-version: [ "3.12", "3.13" ]
runs-on: ubuntu-latest
# Note: No services section needed - testcontainers handles Postgres in Docker
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
- name: Install uv
run: |
pip install uv
- name: Install just
run: |
curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to /usr/local/bin
- name: Create virtual env
run: |
uv venv
- name: Install dependencies
run: |
uv pip install -e .[dev]
- name: Run tests (Postgres via testcontainers)
run: |
uv pip install pytest pytest-cov
just test-postgres
coverage:
name: Coverage Summary (combined, Python 3.12)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python 3.12
uses: actions/setup-python@v4
with:
python-version: "3.12"
cache: "pip"
- name: Install uv
run: |
pip install uv
- name: Install just
run: |
curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to /usr/local/bin
- name: Create virtual env
run: |
uv venv
- name: Install dependencies
run: |
uv pip install -e .[dev]
- name: Run combined coverage (SQLite + Postgres)
run: |
uv pip install pytest pytest-cov
just coverage
- name: Add coverage report to job summary
if: always()
run: |
{
echo "## Coverage"
echo ""
echo '```'
uv run coverage report -m
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
- name: Upload HTML coverage report
if: always()
uses: actions/upload-artifact@v4
with:
name: htmlcov
path: htmlcov/
+2 -1
View File
@@ -52,4 +52,5 @@ ENV/
# claude action
claude-output
**/.claude/settings.local.json
**/.claude/settings.local.json
.mcp.json
+258
View File
@@ -1,5 +1,263 @@
# CHANGELOG
## v0.17.4 (2026-01-05)
### Bug Fixes
- **#503**: Preserve search index across server restarts
([`26f7e98`](https://github.com/basicmachines-co/basic-memory/commit/26f7e98))
- Fixes critical bug where search index was wiped on every MCP server restart
- Bug was introduced in v0.16.3, affecting v0.16.3-v0.17.3
- **User action**: Run `basic-memory reset` once after updating to rebuild search index
### Internal
- **#502**: Major architecture refactor with composition roots and typed API clients
([`5947f04`](https://github.com/basicmachines-co/basic-memory/commit/5947f04))
- Add composition roots for API, MCP, and CLI entrypoints
- Split deps.py into feature-scoped modules (config, db, projects, repositories, services, importers)
- Add ProjectResolver for unified project selection
- Add SyncCoordinator for centralized sync/watch lifecycle
- Introduce typed API clients for MCP tools (KnowledgeClient, SearchClient, MemoryClient, etc.)
## v0.17.3 (2026-01-03)
### Features
- **#485**: Add stable external_id (UUID) to Project and Entity models
([`a4000f6`](https://github.com/basicmachines-co/basic-memory/commit/a4000f6))
- Projects and entities now have immutable UUID identifiers
- API v2 endpoints use external_id for stable references
- Directory responses include external_id for entities
### Bug Fixes
- **#501**: Update mcp dependency to support protocol version 2025-11-25
([`c6baf58`](https://github.com/basicmachines-co/basic-memory/commit/c6baf58))
- Fixes "Unsupported protocol version" error when using Claude Code
- Bump mcp from >=1.2.0 to >=1.23.1
- **#499**: Fix route ordering for cloud deployments
([`53c4c20`](https://github.com/basicmachines-co/basic-memory/commit/53c4c20))
- **#486**: Skip config file update for set_default_project in cloud mode
([`fd732aa`](https://github.com/basicmachines-co/basic-memory/commit/fd732aa))
- **#484**: Make RelationResponse.from_id optional to handle null permalinks
([`537e58a`](https://github.com/basicmachines-co/basic-memory/commit/537e58a))
- Use upsert to prevent IntegrityError during parallel search indexing
([`4ce2198`](https://github.com/basicmachines-co/basic-memory/commit/4ce2198))
- Use relative file paths in importers for cloud storage compatibility
([`8adf1f4`](https://github.com/basicmachines-co/basic-memory/commit/8adf1f4))
### Internal
- Refactor importers to use FileService for cloud compatibility
([`45ce181`](https://github.com/basicmachines-co/basic-memory/commit/45ce181))
- Strengthen integration test coverage, remove stdlib mocks
([`b4486d2`](https://github.com/basicmachines-co/basic-memory/commit/b4486d2))
## v0.17.2 (2025-12-29)
### Bug Fixes
- Allow recent_activity discovery mode in cloud mode
([`0bcda4a`](https://github.com/basicmachines-co/basic-memory/commit/0bcda4a))
- Add `allow_discovery` parameter to `resolve_project_parameter()`
- Tools like `recent_activity` can now work across all projects in cloud mode
- Fix circular import in project_context module
### Internal
- Optimize release workflow by running lint/typecheck only (skip full tests)
([`0b5425f`](https://github.com/basicmachines-co/basic-memory/commit/0b5425f))
## v0.17.1 (2025-12-29)
### Bug Fixes
- **#482**: Only set BASIC_MEMORY_ENV=test during pytest runs
([`98fbd60`](https://github.com/basicmachines-co/basic-memory/commit/98fbd60))
- Fixes environment variable pollution affecting alembic migrations
- Test environment detection now scoped to pytest execution only
## v0.17.0 (2025-12-28)
### Features
- **#478**: Add anonymous usage telemetry with Homebrew-style opt-out
([`856737f`](https://github.com/basicmachines-co/basic-memory/commit/856737f))
- Privacy-respecting anonymous usage analytics
- Easy opt-out via `BASIC_MEMORY_NO_ANALYTICS=1` environment variable
- Helps improve Basic Memory based on real usage patterns
- **#474**: Add auto-format files on save with built-in Python formatter
([`1fd680c`](https://github.com/basicmachines-co/basic-memory/commit/1fd680c))
- Automatic markdown formatting on file save
- Built-in Python formatter for consistent code style
- Configurable formatting options
- **#447**: Complete Phase 2 of API v2 migration - MCP tools use v2 endpoints
([`1a74d85`](https://github.com/basicmachines-co/basic-memory/commit/1a74d85))
- All MCP tools now use optimized v2 API endpoints
- Improved performance for knowledge graph operations
- Foundation for future API enhancements
### Bug Fixes
- Fix UTF-8 BOM handling in frontmatter parsing
([`85684f8`](https://github.com/basicmachines-co/basic-memory/commit/85684f8))
- Handles files with UTF-8 byte order marks correctly
- Prevents frontmatter parsing failures
- **#475**: Handle null titles in ChatGPT import
([`14ce5a3`](https://github.com/basicmachines-co/basic-memory/commit/14ce5a3))
- Gracefully handles conversations without titles
- Improved import robustness
- Remove MaxLen constraint from observation content
([`45d6caf`](https://github.com/basicmachines-co/basic-memory/commit/45d6caf))
- Allows longer observation content without truncation
- Removes arbitrary 2000 character limit
- Handle FileNotFoundError gracefully during sync
([`1652f86`](https://github.com/basicmachines-co/basic-memory/commit/1652f86))
- Prevents sync failures when files are deleted during sync
- More resilient file watching
- Use canonical project names in API response messages
([`c23927d`](https://github.com/basicmachines-co/basic-memory/commit/c23927d))
- Consistent project name formatting in all responses
- Suppress CLI warnings for cleaner output
([`d71c6e8`](https://github.com/basicmachines-co/basic-memory/commit/d71c6e8))
- Cleaner terminal output without spurious warnings
- Prevent DEBUG logs from appearing on CLI stdout
([`63b9849`](https://github.com/basicmachines-co/basic-memory/commit/63b9849))
- Debug logging no longer pollutes CLI output
- **#473**: Detect rclone version for --create-empty-src-dirs support
([`622d37e`](https://github.com/basicmachines-co/basic-memory/commit/622d37e))
- Automatic rclone version detection for compatibility
- Prevents errors on older rclone versions
- **#471**: Prevent CLI commands from hanging on exit
([`916baf8`](https://github.com/basicmachines-co/basic-memory/commit/916baf8))
- Fixes CLI hang on shutdown
- Proper async cleanup
- Add cloud_mode check to initialize_app()
([`ef7adb7`](https://github.com/basicmachines-co/basic-memory/commit/ef7adb7))
- Correct initialization for cloud deployments
### Internal
- Centralize test environment detection in config.is_test_env
([`3cd9178`](https://github.com/basicmachines-co/basic-memory/commit/3cd9178))
- Unified test environment detection
- Disables analytics in test environments
- Make test-int-postgres compatible with macOS
([`95937c6`](https://github.com/basicmachines-co/basic-memory/commit/95937c6))
- Cross-platform PostgreSQL testing support
## v0.16.3 (2025-12-20)
### Features
- **#439**: Add PostgreSQL database backend support
([`fb5e9e1`](https://github.com/basicmachines-co/basic-memory/commit/fb5e9e1))
- Full PostgreSQL/Neon database support as alternative to SQLite
- Async connection pooling with asyncpg
- Alembic migrations support for both backends
- Configurable via `BASIC_MEMORY_DATABASE_BACKEND` environment variable
- **#441**: Implement API v2 with ID-based endpoints (Phase 1)
([`28cc522`](https://github.com/basicmachines-co/basic-memory/commit/28cc522))
- New ID-based API endpoints for improved performance
- Foundation for future API enhancements
- Backward compatible with existing endpoints
- Add project_id to Relation and Observation for efficient project-scoped queries
([`a920a9f`](https://github.com/basicmachines-co/basic-memory/commit/a920a9f))
- Enables faster queries in multi-project environments
- Improved database schema for cloud deployments
- Add bulk insert with ON CONFLICT handling for relations
([`0818bda`](https://github.com/basicmachines-co/basic-memory/commit/0818bda))
- Faster relation creation during sync operations
- Handles duplicate relations gracefully
### Performance
- Lightweight permalink resolution to avoid eager loading
([`6f99d2e`](https://github.com/basicmachines-co/basic-memory/commit/6f99d2e))
- Reduces database queries during entity lookups
- Improved response times for read operations
### Bug Fixes
- **#464**: Pin FastMCP to 2.12.3 to fix MCP tools visibility
([`f227ef6`](https://github.com/basicmachines-co/basic-memory/commit/f227ef6))
- Fixes issue where MCP tools were not visible to Claude
- Reverts to last known working FastMCP version
- **#458**: Reduce watch service CPU usage by increasing reload interval
([`897b1ed`](https://github.com/basicmachines-co/basic-memory/commit/897b1ed))
- Lowers CPU usage during file watching
- More efficient resource utilization
- **#456**: Await background sync task cancellation in lifespan shutdown
([`efbc758`](https://github.com/basicmachines-co/basic-memory/commit/efbc758))
- Prevents hanging on shutdown
- Clean async task cleanup
- **#434**: Respect --project flag in background sync
([`70bb10b`](https://github.com/basicmachines-co/basic-memory/commit/70bb10b))
- Background sync now correctly uses specified project
- Fixes multi-project sync issues
- **#446**: Fix observation parsing and permalink limits
([`73d940e`](https://github.com/basicmachines-co/basic-memory/commit/73d940e))
- Handles edge cases in observation content
- Prevents permalink truncation issues
- **#424**: Handle periods in kebab_filenames mode
([`b004565`](https://github.com/basicmachines-co/basic-memory/commit/b004565))
- Fixes filename handling for files with multiple periods
- Improved kebab-case conversion
- Fix Postgres/Neon connection settings and search index dedupe
([`b5d4fb5`](https://github.com/basicmachines-co/basic-memory/commit/b5d4fb5))
- Optimized connection pooling for Postgres
- Prevents duplicate search index entries
### Testing & CI
- Replace py-pglite with testcontainers for Postgres testing
([`c462faf`](https://github.com/basicmachines-co/basic-memory/commit/c462faf))
- More reliable Postgres testing infrastructure
- Uses Docker-based test containers
- Add PostgreSQL testing to GitHub Actions workflow
([`66b91b2`](https://github.com/basicmachines-co/basic-memory/commit/66b91b2))
- CI now tests both SQLite and PostgreSQL backends
- Ensures cross-database compatibility
- **#416**: Add integration test for read_note with underscored folders
([`0c12a39`](https://github.com/basicmachines-co/basic-memory/commit/0c12a39))
- Verifies folder name handling edge cases
### Internal
- Cloud compatibility fixes and performance improvements (#454)
- Remove logfire instrumentation for cleaner production deployments
- Truncate content_stems to fix Postgres 8KB index row limit
## v0.16.2 (2025-11-16)
### Bug Fixes
+121 -22
View File
@@ -15,10 +15,14 @@ See the [README.md](README.md) file for a project overview.
### Build and Test Commands
- Install: `just install` or `pip install -e ".[dev]"`
- Run all tests (with coverage): `just test` - Runs both unit and integration tests with unified coverage
- Run unit tests only: `just test-unit` - Fast, no coverage
- Run integration tests only: `just test-int` - Fast, no coverage
- Generate HTML coverage: `just coverage` - Opens in browser
- Run all tests (SQLite + Postgres): `just test`
- Run all tests against SQLite: `just test-sqlite`
- Run all tests against Postgres: `just test-postgres` (uses testcontainers)
- Run unit tests (SQLite): `just test-unit-sqlite`
- Run unit tests (Postgres): `just test-unit-postgres`
- Run integration tests (SQLite): `just test-int-sqlite`
- Run integration tests (Postgres): `just test-int-postgres`
- Generate HTML coverage: `just coverage`
- Single test: `pytest tests/path/to/test_file.py::test_function_name`
- Run benchmarks: `pytest test-int/test_sync_performance_benchmark.py -v -m "benchmark and not slow"`
- Lint: `just lint` or `ruff check . --fix`
@@ -30,6 +34,8 @@ See the [README.md](README.md) file for a project overview.
**Note:** Project requires Python 3.12+ (uses type parameter syntax and `type` aliases introduced in 3.12)
**Postgres Testing:** Uses [testcontainers](https://testcontainers-python.readthedocs.io/) which automatically spins up a Postgres instance in Docker. No manual database setup required - just have Docker running.
### Test Structure
- `tests/` - Unit tests for individual components (mocked, fast)
@@ -52,18 +58,97 @@ See the [README.md](README.md) file for a project overview.
- Follow the repository pattern for data access
- Tools communicate to api routers via the httpx ASGI client (in process)
### Code Change Guidelines
- **Full file read before edits**: Before editing any file, read it in full first to ensure complete context; partial reads lead to corrupted edits
- **Minimize diffs**: Prefer the smallest change that satisfies the request. Avoid unrelated refactors or style rewrites unless necessary for correctness
- **No speculative getattr**: Never use `getattr(obj, "attr", default)` when unsure about attribute names. Check the class definition or source code first
- **Fail fast**: Write code with fail-fast logic by default. Do not swallow exceptions with errors or warnings
- **No fallback logic**: Do not add fallback logic unless explicitly told to and agreed with the user
- **No guessing**: Do not say "The issue is..." before you actually know what the issue is. Investigate first.
### Literate Programming Style
Code should tell a story. Comments must explain the "why" and narrative flow, not just the "what".
**Section Headers:**
For files with multiple phases of logic, add section headers so the control flow reads like chapters:
```python
# --- Authentication ---
# ... auth logic ...
# --- Data Validation ---
# ... validation logic ...
# --- Business Logic ---
# ... core logic ...
```
**Decision Point Comments:**
For conditionals that materially change behavior (gates, fallbacks, retries, feature flags), add comments with:
- **Trigger**: what condition causes this branch
- **Why**: the rationale (cost, correctness, UX, determinism)
- **Outcome**: what changes downstream
```python
# Trigger: project has no active sync watcher
# Why: avoid duplicate file system watchers consuming resources
# Outcome: starts new watcher, registers in active_watchers dict
if project_id not in active_watchers:
start_watcher(project_id)
```
**Constraint Comments:**
If code exists because of a constraint (async requirements, rate limits, schema compatibility), explain the constraint near the code:
```python
# SQLite requires WAL mode for concurrent read/write access
connection.execute("PRAGMA journal_mode=WAL")
```
**What NOT to Comment:**
Avoid comments that restate obvious code:
```python
# Bad - restates code
counter += 1 # increment counter
# Good - explains why
counter += 1 # track retries for backoff calculation
```
### Codebase Architecture
See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for detailed architecture documentation.
**Directory Structure:**
- `/alembic` - Alembic db migrations
- `/api` - FastAPI implementation of REST endpoints
- `/cli` - Typer command-line interface
- `/api` - FastAPI REST endpoints + `container.py` composition root
- `/cli` - Typer CLI + `container.py` composition root
- `/deps` - Feature-scoped FastAPI dependencies (config, db, projects, repositories, services, importers)
- `/importers` - Import functionality for Claude, ChatGPT, and other sources
- `/markdown` - Markdown parsing and processing
- `/mcp` - Model Context Protocol server implementation
- `/mcp` - MCP server + `container.py` composition root + `clients/` typed API clients
- `/models` - SQLAlchemy ORM models
- `/repository` - Data access layer
- `/schemas` - Pydantic models for validation
- `/services` - Business logic layer
- `/sync` - File synchronization services
- `/sync` - File synchronization services + `coordinator.py` for lifecycle management
**Composition Roots:**
Each entrypoint (API, MCP, CLI) has a composition root that:
- Reads `ConfigManager` (the only place that reads global config)
- Resolves runtime mode via `RuntimeMode` enum (TEST > CLOUD > LOCAL)
- Provides dependencies to downstream code explicitly
**Typed API Clients (MCP):**
MCP tools use typed clients in `mcp/clients/` to communicate with the API:
- `KnowledgeClient` - Entity CRUD operations
- `SearchClient` - Search operations
- `MemoryClient` - Context building
- `DirectoryClient` - Directory listing
- `ResourceClient` - Resource reading
- `ProjectClient` - Project management
Flow: MCP Tool → Typed Client → HTTP API → Router → Service → Repository
### Development Notes
@@ -76,10 +161,13 @@ See the [README.md](README.md) file for a project overview.
- SQLite is used for indexing and full text search, files are source of truth
- Testing uses pytest with asyncio support (strict mode)
- Unit tests (`tests/`) use mocks when necessary; integration tests (`test-int/`) use real implementations
- Test database uses in-memory SQLite
- Each test runs in a standalone environment with in-memory SQLite and tmp_file directory
- By default, tests run against SQLite (fast, no Docker needed)
- Set `BASIC_MEMORY_TEST_POSTGRES=1` to run against Postgres (uses testcontainers - Docker required)
- Each test runs in a standalone environment with isolated database and tmp_path directory
- CI runs SQLite and Postgres tests in parallel for faster feedback
- Performance benchmarks are in `test-int/test_sync_performance_benchmark.py`
- Use pytest markers: `@pytest.mark.benchmark` for benchmarks, `@pytest.mark.slow` for slow tests
- **Coverage must stay at 100%**: Write tests for new code. Only use `# pragma: no cover` when tests would require excessive mocking (e.g., TYPE_CHECKING blocks, error handlers that need failure injection, runtime-mode-dependent code paths)
### Async Client Pattern (Important!)
@@ -132,22 +220,26 @@ See SPEC-16 for full context manager refactor details.
### Basic Memory Commands
**Local Commands:**
- Sync knowledge: `basic-memory sync` or `basic-memory sync --watch`
- Check sync status: `basic-memory status`
- Import from Claude: `basic-memory import claude conversations`
- Import from ChatGPT: `basic-memory import chatgpt`
- Import from Memory JSON: `basic-memory import memory-json`
- Check sync status: `basic-memory status`
- Tool access: `basic-memory tools` (provides CLI access to MCP tools)
- Guide: `basic-memory tools basic-memory-guide`
- Continue: `basic-memory tools continue-conversation --topic="search"`
- Tool access: `basic-memory tool` (provides CLI access to MCP tools)
- Continue: `basic-memory tool continue-conversation --topic="search"`
**Project Management:**
- List projects: `basic-memory project list`
- Add project: `basic-memory project add "name" ~/path`
- Project info: `basic-memory project info`
- One-way sync (local -> cloud): `basic-memory project sync`
- Bidirectional sync: `basic-memory project bisync`
- Integrity check: `basic-memory project check`
**Cloud Commands (requires subscription):**
- Authenticate: `basic-memory cloud login`
- Logout: `basic-memory cloud logout`
- Bidirectional sync: `basic-memory cloud sync`
- Integrity check: `basic-memory cloud check`
- Mount cloud storage: `basic-memory cloud mount`
- Unmount cloud storage: `basic-memory cloud unmount`
- Check cloud status: `basic-memory cloud status`
- Setup cloud sync: `basic-memory cloud setup`
### MCP Capabilities
@@ -174,18 +266,19 @@ See SPEC-16 for full context manager refactor details.
- `list_memory_projects()` - List all available projects with their status
- `create_memory_project(project_name, project_path, set_default)` - Create new Basic Memory projects
- `delete_project(project_name)` - Delete a project from configuration
- `get_current_project()` - Get current project information and stats
- `sync_status()` - Check file synchronization and background operation status
**Visualization:**
- `canvas(nodes, edges, title, folder)` - Generate Obsidian canvas files for knowledge graph visualization
**ChatGPT-Compatible Tools:**
- `search(query)` - Search across knowledge base (OpenAI actions compatible)
- `fetch(id)` - Fetch full content of a search result document
- MCP Prompts for better AI interaction:
- `ai_assistant_guide()` - Guidance on effectively using Basic Memory tools for AI assistants
- `continue_conversation(topic, timeframe)` - Continue previous conversations with relevant historical context
- `search(query, after_date)` - Search with detailed, formatted results for better context understanding
- `recent_activity(timeframe)` - View recently changed items with formatted output
- `json_canvas_spec()` - Full JSON Canvas specification for Obsidian visualization
### Cloud Features (v0.15.0+)
@@ -229,6 +322,11 @@ of using AI just for code generation, we've developed a true collaborative workf
This approach has allowed us to tackle more complex challenges and build a more robust system than either humans or AI
could achieve independently.
**Problem-Solving Guidance:**
- If a solution isn't working after reasonable effort, suggest alternative approaches
- Don't persist with a problematic library or pattern when better alternatives exist
- Example: When py-pglite caused cascading test failures, switching to testcontainers-postgres was the right call
## GitHub Integration
Basic Memory has taken AI-Human collaboration to the next level by integrating Claude directly into the development workflow through GitHub:
@@ -264,5 +362,6 @@ With GitHub integration, the development workflow includes:
2. **Contribution tracking** - All of Claude's contributions are properly attributed in the Git history
3. **Branch management** - Claude can create feature branches for implementations
4. **Documentation maintenance** - Claude can keep documentation updated as the code evolves
5. **Code Commits**: ALWAYS sign off commits with `git commit -s`
This level of integration represents a new paradigm in AI-human collaboration, where the AI assistant becomes a full-fledged team member rather than just a tool for generating code snippets.
+118
View File
@@ -433,6 +433,124 @@ See the [Documentation](https://memory.basicmachines.co/) for more info, includi
- [Managing multiple Projects](https://docs.basicmemory.com/guides/cli-reference/#project)
- [Importing data from OpenAI/Claude Projects](https://docs.basicmemory.com/guides/cli-reference/#import)
## Logging
Basic Memory uses [Loguru](https://github.com/Delgan/loguru) for logging. The logging behavior varies by entry point:
| Entry Point | Default Behavior | Use Case |
|-------------|------------------|----------|
| CLI commands | File only | Prevents log output from interfering with command output |
| MCP server | File only | Stdout would corrupt the JSON-RPC protocol |
| API server | File (local) or stdout (cloud) | Docker/cloud deployments use stdout |
**Log file location:** `~/.basic-memory/basic-memory.log` (10MB rotation, 10 days retention)
### Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `BASIC_MEMORY_LOG_LEVEL` | `INFO` | Log level: DEBUG, INFO, WARNING, ERROR |
| `BASIC_MEMORY_CLOUD_MODE` | `false` | When `true`, API logs to stdout with structured context |
| `BASIC_MEMORY_ENV` | `dev` | Set to `test` for test mode (stderr only) |
### Examples
```bash
# Enable debug logging
BASIC_MEMORY_LOG_LEVEL=DEBUG basic-memory sync
# View logs
tail -f ~/.basic-memory/basic-memory.log
# Cloud/Docker mode (stdout logging with structured context)
BASIC_MEMORY_CLOUD_MODE=true uvicorn basic_memory.api.app:app
```
## Telemetry
Basic Memory collects anonymous usage statistics to help improve the software. This follows the [Homebrew model](https://docs.brew.sh/Analytics) - telemetry is on by default with easy opt-out.
**What we collect:**
- App version, Python version, OS, architecture
- Feature usage (which MCP tools and CLI commands are used)
- Error types (sanitized - no file paths or personal data)
**What we NEVER collect:**
- Note content, file names, or paths
- Personal information
- IP addresses
**Opting out:**
```bash
# Disable telemetry
basic-memory telemetry disable
# Check status
basic-memory telemetry status
# Re-enable
basic-memory telemetry enable
```
Or set the environment variable:
```bash
export BASIC_MEMORY_TELEMETRY_ENABLED=false
```
For more details, see the [Telemetry documentation](https://basicmemory.com/telemetry).
## Development
### Running Tests
Basic Memory supports dual database backends (SQLite and Postgres). By default, tests run against SQLite. Set `BASIC_MEMORY_TEST_POSTGRES=1` to run against Postgres (uses testcontainers - Docker required).
**Quick Start:**
```bash
# Run all tests against SQLite (default, fast)
just test-sqlite
# Run all tests against Postgres (uses testcontainers)
just test-postgres
# Run both SQLite and Postgres tests
just test
```
**Available Test Commands:**
- `just test` - Run all tests against both SQLite and Postgres
- `just test-sqlite` - Run all tests against SQLite (fast, no Docker needed)
- `just test-postgres` - Run all tests against Postgres (uses testcontainers)
- `just test-unit-sqlite` - Run unit tests against SQLite
- `just test-unit-postgres` - Run unit tests against Postgres
- `just test-int-sqlite` - Run integration tests against SQLite
- `just test-int-postgres` - Run integration tests against Postgres
- `just test-windows` - Run Windows-specific tests (auto-skips on other platforms)
- `just test-benchmark` - Run performance benchmark tests
**Postgres Testing:**
Postgres tests use [testcontainers](https://testcontainers-python.readthedocs.io/) which automatically spins up a Postgres instance in Docker. No manual database setup required - just have Docker running.
**Test Markers:**
Tests use pytest markers for selective execution:
- `windows` - Windows-specific database optimizations
- `benchmark` - Performance tests (excluded from default runs)
**Other Development Commands:**
```bash
just install # Install with dev dependencies
just lint # Run linting checks
just typecheck # Run type checking
just format # Format code with ruff
just check # Run all quality checks
just migration "msg" # Create database migration
```
See the [justfile](justfile) for the complete list of development commands.
## License
AGPL-3.0
+42
View File
@@ -0,0 +1,42 @@
# Docker Compose configuration for Basic Memory with PostgreSQL
# Use this for local development and testing with Postgres backend
#
# Usage:
# docker-compose -f docker-compose-postgres.yml up -d
# docker-compose -f docker-compose-postgres.yml down
services:
postgres:
image: postgres:17
container_name: basic-memory-postgres
environment:
# Local development/test credentials - NOT for production
# These values are referenced by tests and justfile commands
POSTGRES_DB: basic_memory
POSTGRES_USER: basic_memory_user
POSTGRES_PASSWORD: dev_password # Simple password for local testing only
ports:
- "5433:5432"
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U basic_memory_user -d basic_memory"]
interval: 10s
timeout: 5s
retries: 5
restart: unless-stopped
volumes:
# Named volume for Postgres data
postgres_data:
driver: local
# Named volume for persistent configuration
# Database will be stored in Postgres, not in this volume
basic-memory-config:
driver: local
# Network configuration (optional)
# networks:
# basic-memory-net:
# driver: bridge
+412
View File
@@ -0,0 +1,412 @@
# Basic Memory Architecture
This document describes the architectural patterns and composition structure of Basic Memory.
## Overview
Basic Memory is a local-first knowledge management system with three entrypoints:
- **API** - FastAPI REST server for HTTP access
- **MCP** - Model Context Protocol server for LLM integration
- **CLI** - Typer command-line interface
Each entrypoint uses a **composition root** pattern to manage configuration and dependencies.
## Composition Roots
### What is a Composition Root?
A composition root is the single place in an application where dependencies are wired together. In Basic Memory, each entrypoint has its own composition root that:
1. Reads configuration from `ConfigManager`
2. Resolves runtime mode (cloud/local/test)
3. Creates and provides dependencies to downstream code
**Key principle**: Only composition roots read global configuration. All other modules receive configuration explicitly.
### Container Structure
Each entrypoint has a container dataclass in its package:
```
src/basic_memory/
├── api/
│ └── container.py # ApiContainer
├── mcp/
│ └── container.py # McpContainer
├── cli/
│ └── container.py # CliContainer
└── runtime.py # RuntimeMode enum and resolver
```
### Container Pattern
All containers follow the same structure:
```python
@dataclass
class Container:
config: BasicMemoryConfig
mode: RuntimeMode
@classmethod
def create(cls) -> "Container":
"""Create container by reading ConfigManager."""
config = ConfigManager().config
mode = resolve_runtime_mode(
cloud_mode_enabled=config.cloud_mode_enabled,
is_test_env=config.is_test_env,
)
return cls(config=config, mode=mode)
@property
def some_computed_property(self) -> bool:
"""Derived values based on config and mode."""
return self.mode.is_local and self.config.some_setting
# Module-level singleton
_container: Container | None = None
def get_container() -> Container:
if _container is None:
raise RuntimeError("Container not initialized")
return _container
def set_container(container: Container) -> None:
global _container
_container = container
```
### Runtime Mode Resolution
The `RuntimeMode` enum centralizes mode detection:
```python
class RuntimeMode(Enum):
LOCAL = "local"
CLOUD = "cloud"
TEST = "test"
@property
def is_cloud(self) -> bool:
return self == RuntimeMode.CLOUD
@property
def is_local(self) -> bool:
return self == RuntimeMode.LOCAL
@property
def is_test(self) -> bool:
return self == RuntimeMode.TEST
```
Resolution follows this precedence: **TEST > CLOUD > LOCAL**
```python
def resolve_runtime_mode(cloud_mode_enabled: bool, is_test_env: bool) -> RuntimeMode:
if is_test_env:
return RuntimeMode.TEST
if cloud_mode_enabled:
return RuntimeMode.CLOUD
return RuntimeMode.LOCAL
```
## Dependencies Package
### Structure
The `deps/` package provides FastAPI dependencies organized by feature:
```
src/basic_memory/deps/
├── __init__.py # Re-exports for backwards compatibility
├── config.py # Configuration access
├── db.py # Database/session management
├── projects.py # Project resolution
├── repositories.py # Data access layer
├── services.py # Business logic layer
└── importers.py # Import functionality
```
### Usage in Routers
```python
from basic_memory.deps.services import get_entity_service
from basic_memory.deps.projects import get_project_config
@router.get("/entities/{id}")
async def get_entity(
id: int,
entity_service: EntityService = Depends(get_entity_service),
project: ProjectConfig = Depends(get_project_config),
):
return await entity_service.get(id)
```
### Backwards Compatibility
The old `deps.py` file still exists as a thin re-export shim:
```python
# deps.py - backwards compatibility shim
from basic_memory.deps import *
```
New code should import from specific submodules (`basic_memory.deps.services`) for clarity.
## MCP Tools Architecture
### Typed API Clients
MCP tools communicate with the API through typed clients that encapsulate HTTP paths and response validation:
```
src/basic_memory/mcp/clients/
├── __init__.py # Re-exports all clients
├── base.py # BaseClient with common logic
├── knowledge.py # KnowledgeClient - entity CRUD
├── search.py # SearchClient - search operations
├── memory.py # MemoryClient - context building
├── directory.py # DirectoryClient - directory listing
├── resource.py # ResourceClient - resource reading
└── project.py # ProjectClient - project management
```
### Client Pattern
Each client encapsulates API paths and validates responses:
```python
class KnowledgeClient(BaseClient):
"""Client for knowledge/entity operations."""
async def resolve_entity(self, identifier: str) -> int:
"""Resolve identifier to entity ID."""
response = await call_get(
self.http_client,
f"{self._base_path}/resolve/{identifier}",
)
return int(response.text)
async def get_entity(self, entity_id: int) -> EntityResponse:
"""Get entity by ID."""
response = await call_get(
self.http_client,
f"{self._base_path}/entities/{entity_id}",
)
return EntityResponse.model_validate(response.json())
```
### Tool → Client → API Flow
```
MCP Tool (thin adapter)
Typed Client (encapsulates paths, validates responses)
HTTP API (FastAPI router)
Service Layer (business logic)
Repository Layer (data access)
```
Example tool using typed client:
```python
@mcp.tool()
async def search_notes(query: str, project: str | None = None) -> SearchResponse:
async with get_client() as client:
active_project = await get_active_project(client, project)
# Import client inside function to avoid circular imports
from basic_memory.mcp.clients import SearchClient
search_client = SearchClient(client, active_project.external_id)
return await search_client.search(query)
```
## Sync Coordination
### SyncCoordinator
The `SyncCoordinator` centralizes sync/watch lifecycle management:
```python
@dataclass
class SyncCoordinator:
"""Coordinates file sync and watch operations."""
status: SyncStatus = SyncStatus.NOT_STARTED
sync_task: asyncio.Task | None = None
watch_service: WatchService | None = None
async def start(self, ...):
"""Start sync and watch operations."""
async def stop(self):
"""Stop all sync operations gracefully."""
def get_status_info(self) -> dict:
"""Get current sync status for observability."""
```
### Status Enum
```python
class SyncStatus(Enum):
NOT_STARTED = "not_started"
STARTING = "starting"
RUNNING = "running"
STOPPING = "stopping"
STOPPED = "stopped"
ERROR = "error"
```
## Project Resolution
### ProjectResolver
Unified project selection across all entrypoints:
```python
class ProjectResolver:
"""Resolves which project to use based on context."""
def resolve(
self,
explicit_project: str | None = None,
) -> ResolvedProject:
"""Resolve project using three-tier hierarchy:
1. Explicit project parameter
2. Default project from config
3. Single available project
"""
```
### Resolution Modes
```python
class ResolutionMode(Enum):
EXPLICIT = "explicit" # User specified project
DEFAULT = "default" # Using configured default
SINGLE_PROJECT = "single" # Only one project exists
FALLBACK = "fallback" # Using first available
```
## Testing Patterns
### Container Testing
Each container has corresponding tests:
```
tests/
├── api/test_api_container.py
├── mcp/test_mcp_container.py
└── cli/test_cli_container.py
```
Tests verify:
- Container creation from config
- Runtime mode properties
- Container accessor functions (get/set)
### Mocking Typed Clients
When testing MCP tools, mock at the client level:
```python
def test_search_notes(monkeypatch):
import basic_memory.mcp.clients as clients_mod
class MockSearchClient:
async def search(self, query):
return SearchResponse(results=[...])
monkeypatch.setattr(clients_mod, "SearchClient", MockSearchClient)
```
## Design Principles
### 1. Explicit Dependencies
Modules receive configuration explicitly rather than reading globals:
```python
# Good - explicit injection
async def sync_files(config: BasicMemoryConfig):
...
# Avoid - hidden global access
async def sync_files():
config = ConfigManager().config # Hidden coupling
```
### 2. Single Responsibility
Each layer has a clear responsibility:
- **Containers**: Wire dependencies
- **Clients**: Encapsulate HTTP communication
- **Services**: Business logic
- **Repositories**: Data access
- **Tools/Routers**: Thin adapters
### 3. Deferred Imports
To avoid circular imports, typed clients are imported inside functions:
```python
async def my_tool():
async with get_client() as client:
# Import here to avoid circular dependency
from basic_memory.mcp.clients import KnowledgeClient
knowledge_client = KnowledgeClient(client, project_id)
```
### 4. Backwards Compatibility
When refactoring, maintain backwards compatibility via shims:
```python
# Old module becomes a shim
from basic_memory.new_location import *
# Docstring explains migration path
"""
DEPRECATED: Import from basic_memory.new_location instead.
This shim will be removed in a future version.
"""
```
## File Organization
```
src/basic_memory/
├── api/
│ ├── container.py # API composition root
│ ├── routers/ # FastAPI routers
│ └── ...
├── mcp/
│ ├── container.py # MCP composition root
│ ├── clients/ # Typed API clients
│ ├── tools/ # MCP tool definitions
│ └── server.py # MCP server setup
├── cli/
│ ├── container.py # CLI composition root
│ ├── app.py # Typer app
│ └── commands/ # CLI command groups
├── deps/
│ ├── config.py # Config dependencies
│ ├── db.py # Database dependencies
│ ├── projects.py # Project dependencies
│ ├── repositories.py # Repository dependencies
│ ├── services.py # Service dependencies
│ └── importers.py # Importer dependencies
├── sync/
│ ├── coordinator.py # SyncCoordinator
│ └── ...
├── runtime.py # RuntimeMode resolution
├── project_resolver.py # Unified project selection
└── config.py # Configuration management
```
+30
View File
@@ -0,0 +1,30 @@
## Coverage policy (practical 100%)
Basic Memorys test suite intentionally mixes:
- unit tests (fast, deterministic)
- integration tests (real filesystem + real DB via `test-int/`)
To keep the default CI signal **stable and meaningful**, the default `pytest` coverage report targets **core library logic** and **excludes** a small set of modules that are either:
- highly environment-dependent (OS/DB tuning)
- inherently interactive (CLI)
- background-task orchestration (watchers/sync runners)
- external analytics
### Whats excluded (and why)
Coverage excludes are configured in `pyproject.toml` under `[tool.coverage.report].omit`.
Current exclusions include:
- `src/basic_memory/cli/**`: interactive wrappers; behavior is validated via higher-level tests and smoke tests.
- `src/basic_memory/db.py`: platform/backend tuning paths (SQLite/Postgres/Windows), covered by integration tests and targeted runs.
- `src/basic_memory/services/initialization.py`: startup orchestration/background tasks; covered indirectly by app/MCP entrypoints.
- `src/basic_memory/sync/sync_service.py`: heavy filesystem↔DB integration; validated in integration suite (not enforced in unit coverage).
- `src/basic_memory/telemetry.py`: external analytics; exercised lightly but excluded from strict coverage gate.
### Recommended additional runs
If you want extra confidence locally/CI:
- **Postgres backend**: run tests with `BASIC_MEMORY_TEST_POSTGRES=1`.
- **Strict backend-complete coverage**: run coverage on SQLite + Postgres and combine the results (recommended).
+116 -22
View File
@@ -7,21 +7,121 @@ install:
@echo ""
@echo "💡 Remember to activate the virtual environment by running: source .venv/bin/activate"
# Run unit tests only (fast, no coverage)
test-unit:
uv run pytest -p pytest_mock -v --no-cov -n auto tests
# ==============================================================================
# DATABASE BACKEND TESTING
# ==============================================================================
# Basic Memory supports dual database backends (SQLite and Postgres).
# By default, tests run against SQLite (fast, no dependencies).
# Set BASIC_MEMORY_TEST_POSTGRES=1 to run against Postgres (uses testcontainers).
#
# Quick Start:
# just test # Run all tests against SQLite (default)
# just test-sqlite # Run all tests against SQLite
# just test-postgres # Run all tests against Postgres (testcontainers)
# just test-unit-sqlite # Run unit tests against SQLite
# just test-unit-postgres # Run unit tests against Postgres
# just test-int-sqlite # Run integration tests against SQLite
# just test-int-postgres # Run integration tests against Postgres
#
# CI runs both in parallel for faster feedback.
# ==============================================================================
# Run integration tests only (fast, no coverage)
test-int:
uv run pytest -p pytest_mock -v --no-cov -n auto test-int
# Run all tests against SQLite and Postgres
test: test-sqlite test-postgres
# Run all tests with unified coverage report
test: test-unit test-int
# Run all tests against SQLite
test-sqlite: test-unit-sqlite test-int-sqlite
# Run all tests against Postgres (uses testcontainers)
test-postgres: test-unit-postgres test-int-postgres
# Run unit tests against SQLite
test-unit-sqlite:
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov tests
# Run unit tests against Postgres
test-unit-postgres:
BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov tests
# Run integration tests against SQLite
test-int-sqlite:
uv run pytest -p pytest_mock -v --no-cov test-int
# Run integration tests against Postgres
# Note: Uses timeout due to FastMCP Client + asyncpg cleanup hang (tests pass, process hangs on exit)
# See: https://github.com/jlowin/fastmcp/issues/1311
test-int-postgres:
#!/usr/bin/env bash
set -euo pipefail
# Use gtimeout (macOS/Homebrew) or timeout (Linux)
TIMEOUT_CMD=$(command -v gtimeout || command -v timeout || echo "")
if [[ -n "$TIMEOUT_CMD" ]]; then
$TIMEOUT_CMD --signal=KILL 600 bash -c 'BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov test-int' || test $? -eq 137
else
echo "⚠️ No timeout command found, running without timeout..."
BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov test-int
fi
# Reset Postgres test database (drops and recreates schema)
# Useful when Alembic migration state gets out of sync during development
# Uses credentials from docker-compose-postgres.yml
postgres-reset:
docker exec basic-memory-postgres psql -U ${POSTGRES_USER:-basic_memory_user} -d ${POSTGRES_TEST_DB:-basic_memory_test} -c "DROP SCHEMA public CASCADE; CREATE SCHEMA public;"
@echo "✅ Postgres test database reset"
# Run Alembic migrations manually against Postgres test database
# Useful for debugging migration issues
# Uses credentials from docker-compose-postgres.yml (can override with env vars)
postgres-migrate:
@cd src/basic_memory/alembic && \
BASIC_MEMORY_DATABASE_BACKEND=postgres \
BASIC_MEMORY_DATABASE_URL=${POSTGRES_TEST_URL:-postgresql+asyncpg://basic_memory_user:dev_password@localhost:5433/basic_memory_test} \
uv run alembic upgrade head
@echo "✅ Migrations applied to Postgres test database"
# Run Windows-specific tests only (only works on Windows platform)
# These tests verify Windows-specific database optimizations (locking mode, NullPool)
# Will be skipped automatically on non-Windows platforms
test-windows:
uv run pytest -p pytest_mock -v --no-cov -m windows tests test-int
# Run benchmark tests only (performance testing)
# These are slow tests that measure sync performance with various file counts
# Excluded from default test runs to keep CI fast
test-benchmark:
uv run pytest -p pytest_mock -v --no-cov -m benchmark tests test-int
# Run all tests including Windows, Postgres, and Benchmarks (for CI/comprehensive testing)
# Use this before releasing to ensure everything works across all backends and platforms
test-all:
uv run pytest -p pytest_mock -v --no-cov tests test-int
# Generate HTML coverage report
coverage:
uv run pytest -p pytest_mock -v -n auto tests test-int --cov-report=html
@echo "Coverage report generated in htmlcov/index.html"
#!/usr/bin/env bash
set -euo pipefail
uv run coverage erase
echo "🔎 Coverage (SQLite)..."
BASIC_MEMORY_ENV=test uv run coverage run --source=basic_memory -m pytest -p pytest_mock -v --no-cov tests test-int
echo "🔎 Coverage (Postgres via testcontainers)..."
# Note: Uses timeout due to FastMCP Client + asyncpg cleanup hang (tests pass, process hangs on exit)
# See: https://github.com/jlowin/fastmcp/issues/1311
TIMEOUT_CMD=$(command -v gtimeout || command -v timeout || echo "")
if [[ -n "$TIMEOUT_CMD" ]]; then
$TIMEOUT_CMD --signal=KILL 600 bash -c 'BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 uv run coverage run --source=basic_memory -m pytest -p pytest_mock -v --no-cov -m postgres tests test-int' || test $? -eq 137
else
echo "⚠️ No timeout command found, running without timeout..."
BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 uv run coverage run --source=basic_memory -m pytest -p pytest_mock -v --no-cov -m postgres tests test-int
fi
echo "🧩 Combining coverage data..."
uv run coverage combine
uv run coverage report -m
uv run coverage html
echo "Coverage report generated in htmlcov/index.html"
# Lint and fix code (calls fix)
lint: fix
@@ -49,14 +149,6 @@ format:
run-inspector:
npx @modelcontextprotocol/inspector
# Build macOS installer
installer-mac:
cd installer && chmod +x make_icons.sh && ./make_icons.sh
cd installer && uv run python setup.py bdist_mac
# Build Windows installer
installer-win:
cd installer && uv run python setup.py bdist_win32
# Update all dependencies to latest versions
update-deps:
@@ -104,8 +196,9 @@ release version:
fi
# Run quality checks
echo "🔍 Running quality checks..."
just check
echo "🔍 Running lint checks..."
just lint
just typecheck
# Update version in __init__.py
echo "📝 Updating version in __init__.py..."
@@ -163,8 +256,9 @@ beta version:
fi
# Run quality checks
echo "🔍 Running quality checks..."
just check
echo "🔍 Running lint checks..."
just lint
just typecheck
# Update version in __init__.py
echo "📝 Updating version in __init__.py..."
+22 -8
View File
@@ -15,8 +15,7 @@ dependencies = [
"aiosqlite>=0.20.0",
"greenlet>=3.1.1",
"pydantic[email,timezone]>=2.10.3",
"icecream>=2.1.3",
"mcp>=1.2.0",
"mcp>=1.23.1",
"pydantic-settings>=2.6.1",
"loguru>=0.7.3",
"pyright>=1.1.390",
@@ -30,12 +29,19 @@ dependencies = [
"alembic>=1.14.1",
"pillow>=11.1.0",
"pybars3>=0.9.7",
"fastmcp>=2.10.2",
"fastmcp==2.12.3", # Pinned - 2.14.x breaks MCP tools visibility (issue #463)
"pyjwt>=2.10.1",
"python-dotenv>=1.1.0",
"pytest-aio>=1.9.0",
"aiofiles>=24.1.0", # Async file I/O
"logfire>=0.73.0", # Optional observability (disabled by default via config)
"aiofiles>=24.1.0", # Optional observability (disabled by default via config)
"asyncpg>=0.30.0",
"nest-asyncio>=1.6.0", # For Alembic migrations with Postgres
"pytest-asyncio>=1.2.0",
"psycopg==3.3.1",
"mdformat>=0.7.22",
"mdformat-gfm>=0.3.7",
"mdformat-frontmatter>=2.0.8",
"openpanel>=0.0.1", # Anonymous usage telemetry (Homebrew-style opt-out)
]
@@ -61,6 +67,8 @@ asyncio_default_fixture_loop_scope = "function"
markers = [
"benchmark: Performance benchmark tests (deselect with '-m \"not benchmark\"')",
"slow: Slow-running tests (deselect with '-m \"not slow\"')",
"postgres: Tests that run against Postgres backend (deselect with '-m \"not postgres\"')",
"windows: Windows-specific tests (deselect with '-m \"not windows\"')",
]
[tool.ruff]
@@ -78,6 +86,8 @@ dev = [
"pytest-xdist>=3.0.0",
"ruff>=0.1.6",
"freezegun>=1.5.5",
"testcontainers[postgres]>=4.0.0",
"psycopg>=3.2.0",
]
[tool.hatch.version]
@@ -102,6 +112,8 @@ pythonVersion = "3.12"
[tool.coverage.run]
concurrency = ["thread", "gevent"]
parallel = true
source = ["basic_memory"]
[tool.coverage.report]
exclude_lines = [
@@ -123,9 +135,11 @@ omit = [
"*/supabase_auth_provider.py", # External HTTP calls to Supabase APIs
"*/watch_service.py", # File system watching - complex integration testing
"*/background_sync.py", # Background processes
"*/cli/main.py", # CLI entry point
"*/mcp/tools/project_management.py", # Covered by integration tests
"*/mcp/tools/sync_status.py", # Covered by integration tests
"*/cli/**", # CLI is an interactive wrapper; core logic is covered via API/MCP/service tests
"*/db.py", # Backend/runtime-dependent (sqlite/postgres/windows tuning); validated via integration tests
"*/services/initialization.py", # Startup orchestration + background tasks (watchers); exercised indirectly in entrypoints
"*/sync/sync_service.py", # Heavy filesystem/db integration; covered by integration suite, not enforced in unit coverage
"*/telemetry.py", # External analytics; tested lightly, excluded from strict coverage target
"*/services/migration_service.py", # Complex migration scenarios
]
+1 -1
View File
@@ -1,7 +1,7 @@
"""basic-memory - Local-first knowledge management combining Zettelkasten with knowledge graphs"""
# Package version - updated by release automation
__version__ = "0.16.2"
__version__ = "0.17.4"
# API version for FastAPI - independent of package version
__api_version__ = "v0"
+110 -24
View File
@@ -1,17 +1,32 @@
"""Alembic environment configuration."""
import asyncio
import os
from logging.config import fileConfig
from sqlalchemy import engine_from_config
from sqlalchemy import pool
# Allow nested event loops (needed for pytest-asyncio and other async contexts)
# Note: nest_asyncio doesn't work with uvloop, so we handle that case separately
try:
import nest_asyncio
nest_asyncio.apply()
except (ImportError, ValueError):
# nest_asyncio not available or can't patch this loop type (e.g., uvloop)
pass
from sqlalchemy import engine_from_config, pool
from sqlalchemy.ext.asyncio import AsyncEngine, create_async_engine
from alembic import context
from basic_memory.config import ConfigManager
# set config.env to "test" for pytest to prevent logging to file in utils.setup_logging()
os.environ["BASIC_MEMORY_ENV"] = "test"
# Trigger: only set test env when actually running under pytest
# Why: alembic/env.py is imported during normal operations (MCP server startup, migrations)
# but we only want test behavior during actual test runs
# Outcome: prevents is_test_env from returning True in production, enabling watch service
if os.getenv("PYTEST_CURRENT_TEST") is not None:
os.environ["BASIC_MEMORY_ENV"] = "test"
# Import after setting environment variable # noqa: E402
from basic_memory.models import Base # noqa: E402
@@ -20,12 +35,22 @@ from basic_memory.models import Base # noqa: E402
# access to the values within the .ini file in use.
config = context.config
# Load app config - this will read environment variables (BASIC_MEMORY_DATABASE_BACKEND, etc.)
# due to Pydantic's env_prefix="BASIC_MEMORY_" setting
app_config = ConfigManager().config
# Set the SQLAlchemy URL from our app config
sqlalchemy_url = f"sqlite:///{app_config.database_path}"
config.set_main_option("sqlalchemy.url", sqlalchemy_url)
# print(f"Using SQLAlchemy URL: {sqlalchemy_url}")
# Set the SQLAlchemy URL based on database backend configuration
# If the URL is already set in config (e.g., from run_migrations), use that
# Otherwise, get it from app config
# Note: alembic.ini has a placeholder URL "driver://user:pass@localhost/dbname" that we need to override
current_url = config.get_main_option("sqlalchemy.url")
if not current_url or current_url == "driver://user:pass@localhost/dbname":
from basic_memory.db import DatabaseType
sqlalchemy_url = DatabaseType.get_db_url(
app_config.database_path, DatabaseType.FILESYSTEM, app_config
)
config.set_main_option("sqlalchemy.url", sqlalchemy_url)
# Interpret the config file for Python logging.
if config.config_file_name is not None:
@@ -69,28 +94,89 @@ def run_migrations_offline() -> None:
context.run_migrations()
def do_run_migrations(connection):
"""Execute migrations with the given connection."""
context.configure(
connection=connection,
target_metadata=target_metadata,
include_object=include_object,
render_as_batch=True,
compare_type=True,
)
with context.begin_transaction():
context.run_migrations()
async def run_async_migrations(connectable):
"""Run migrations asynchronously with AsyncEngine."""
async with connectable.connect() as connection:
await connection.run_sync(do_run_migrations)
await connectable.dispose()
def run_migrations_online() -> None:
"""Run migrations in 'online' mode.
In this scenario we need to create an Engine
and associate a connection with the context.
Supports both sync engines (SQLite) and async engines (PostgreSQL with asyncpg).
"""
connectable = engine_from_config(
config.get_section(config.config_ini_section, {}),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
# Check if a connection/engine was provided (e.g., from run_migrations)
connectable = context.config.attributes.get("connection", None)
with connectable.connect() as connection:
context.configure(
connection=connection,
target_metadata=target_metadata,
include_object=include_object,
render_as_batch=True,
)
if connectable is None:
# No connection provided, create engine from config
url = context.config.get_main_option("sqlalchemy.url")
with context.begin_transaction():
context.run_migrations()
# Check if it's an async URL (sqlite+aiosqlite or postgresql+asyncpg)
if url and ("+asyncpg" in url or "+aiosqlite" in url):
# Create async engine for asyncpg or aiosqlite
connectable = create_async_engine(
url,
poolclass=pool.NullPool,
future=True,
)
else:
# Create sync engine for regular sqlite or postgresql
connectable = engine_from_config(
context.config.get_section(context.config.config_ini_section, {}),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
# Handle async engines (PostgreSQL with asyncpg)
if isinstance(connectable, AsyncEngine):
# Try to run async migrations
# nest_asyncio allows asyncio.run() from within event loops, but doesn't work with uvloop
try:
asyncio.run(run_async_migrations(connectable))
except RuntimeError as e:
if "cannot be called from a running event loop" in str(e):
# We're in a running event loop (likely uvloop) - need to use a different approach
# Create a new thread to run the async migrations
import concurrent.futures
def run_in_thread():
"""Run async migrations in a new event loop in a separate thread."""
new_loop = asyncio.new_event_loop()
asyncio.set_event_loop(new_loop)
try:
new_loop.run_until_complete(run_async_migrations(connectable))
finally:
new_loop.close()
with concurrent.futures.ThreadPoolExecutor() as executor:
future = executor.submit(run_in_thread)
future.result() # Wait for completion and re-raise any exceptions
else:
raise
else:
# Handle sync engines (SQLite) or sync connections
if hasattr(connectable, "connect"):
# It's an engine, get a connection
with connectable.connect() as connection:
do_run_migrations(connection)
else:
# It's already a connection
do_run_migrations(connectable)
if context.is_offline_mode():
@@ -0,0 +1,131 @@
"""Add Postgres full-text search support with tsvector and GIN indexes
Revision ID: 314f1ea54dc4
Revises: e7e1f4367280
Create Date: 2025-11-15 18:05:01.025405
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = "314f1ea54dc4"
down_revision: Union[str, None] = "e7e1f4367280"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Add PostgreSQL full-text search support.
This migration:
1. Creates search_index table for Postgres (SQLite uses FTS5 virtual table)
2. Adds generated tsvector column for full-text search
3. Creates GIN index on the tsvector column for fast text queries
4. Creates GIN index on metadata JSONB column for fast containment queries
Note: These changes only apply to Postgres. SQLite continues to use FTS5 virtual tables.
"""
# Check if we're using Postgres
connection = op.get_bind()
if connection.dialect.name == "postgresql":
# Create search_index table for Postgres
# For SQLite, this is a FTS5 virtual table created elsewhere
from sqlalchemy.dialects.postgresql import JSONB
op.create_table(
"search_index",
sa.Column("id", sa.Integer(), nullable=False), # Entity IDs are integers
sa.Column("project_id", sa.Integer(), nullable=False), # Multi-tenant isolation
sa.Column("title", sa.Text(), nullable=True),
sa.Column("content_stems", sa.Text(), nullable=True),
sa.Column("content_snippet", sa.Text(), nullable=True),
sa.Column("permalink", sa.String(), nullable=True), # Nullable for non-markdown files
sa.Column("file_path", sa.String(), nullable=True),
sa.Column("type", sa.String(), nullable=True),
sa.Column("from_id", sa.Integer(), nullable=True), # Relation IDs are integers
sa.Column("to_id", sa.Integer(), nullable=True), # Relation IDs are integers
sa.Column("relation_type", sa.String(), nullable=True),
sa.Column("entity_id", sa.Integer(), nullable=True), # Entity IDs are integers
sa.Column("category", sa.String(), nullable=True),
sa.Column("metadata", JSONB(), nullable=True), # Use JSONB for Postgres
sa.Column("created_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=True),
sa.PrimaryKeyConstraint(
"id", "type", "project_id"
), # Composite key: id can repeat across types
sa.ForeignKeyConstraint(
["project_id"],
["project.id"],
name="fk_search_index_project_id",
ondelete="CASCADE",
),
if_not_exists=True,
)
# Create index on project_id for efficient multi-tenant queries
op.create_index(
"ix_search_index_project_id",
"search_index",
["project_id"],
unique=False,
)
# Create unique partial index on permalink for markdown files
# Non-markdown files don't have permalinks, so we use a partial index
op.execute("""
CREATE UNIQUE INDEX uix_search_index_permalink_project
ON search_index (permalink, project_id)
WHERE permalink IS NOT NULL
""")
# Add tsvector column as a GENERATED ALWAYS column
# This automatically updates when title or content_stems change
op.execute("""
ALTER TABLE search_index
ADD COLUMN textsearchable_index_col tsvector
GENERATED ALWAYS AS (
to_tsvector('english',
coalesce(title, '') || ' ' ||
coalesce(content_stems, '')
)
) STORED
""")
# Create GIN index on tsvector column for fast full-text search
op.create_index(
"idx_search_index_fts",
"search_index",
["textsearchable_index_col"],
unique=False,
postgresql_using="gin",
)
# Create GIN index on metadata JSONB for fast containment queries
# Using jsonb_path_ops for smaller index size and better performance
op.execute("""
CREATE INDEX idx_search_index_metadata_gin
ON search_index
USING GIN (metadata jsonb_path_ops)
""")
def downgrade() -> None:
"""Remove PostgreSQL full-text search support."""
connection = op.get_bind()
if connection.dialect.name == "postgresql":
# Drop indexes first
op.execute("DROP INDEX IF EXISTS idx_search_index_metadata_gin")
op.drop_index("idx_search_index_fts", table_name="search_index")
op.execute("DROP INDEX IF EXISTS uix_search_index_permalink_project")
op.drop_index("ix_search_index_project_id", table_name="search_index")
# Drop the generated column
op.execute("ALTER TABLE search_index DROP COLUMN IF EXISTS textsearchable_index_col")
# Drop the search_index table
op.drop_table("search_index")
@@ -21,6 +21,12 @@ depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
# SQLite FTS5 virtual table handling is SQLite-specific
# For Postgres, search_index is a regular table managed by ORM
connection = op.get_bind()
is_sqlite = connection.dialect.name == "sqlite"
op.create_table(
"project",
sa.Column("id", sa.Integer(), nullable=False),
@@ -55,7 +61,9 @@ def upgrade() -> None:
batch_op.add_column(sa.Column("project_id", sa.Integer(), nullable=False))
batch_op.drop_index(
"uix_entity_permalink",
sqlite_where=sa.text("content_type = 'text/markdown' AND permalink IS NOT NULL"),
sqlite_where=sa.text("content_type = 'text/markdown' AND permalink IS NOT NULL")
if is_sqlite
else None,
)
batch_op.drop_index("ix_entity_file_path")
batch_op.create_index(batch_op.f("ix_entity_file_path"), ["file_path"], unique=False)
@@ -67,12 +75,16 @@ def upgrade() -> None:
"uix_entity_permalink_project",
["permalink", "project_id"],
unique=True,
sqlite_where=sa.text("content_type = 'text/markdown' AND permalink IS NOT NULL"),
sqlite_where=sa.text("content_type = 'text/markdown' AND permalink IS NOT NULL")
if is_sqlite
else None,
)
batch_op.create_foreign_key("fk_entity_project_id", "project", ["project_id"], ["id"])
# drop the search index table. it will be recreated
op.drop_table("search_index")
# Only drop for SQLite - Postgres uses regular table managed by ORM
if is_sqlite:
op.drop_table("search_index")
# ### end Alembic commands ###
@@ -25,43 +25,51 @@ def upgrade() -> None:
The UNIQUE constraint prevents multiple projects from having is_default=FALSE,
which breaks project creation when the service sets is_default=False.
Since SQLite doesn't support dropping specific constraints easily, we'll
recreate the table without the problematic constraint.
SQLite: Recreate the table without the constraint (no ALTER TABLE support)
Postgres: Use ALTER TABLE to drop the constraint directly
"""
# For SQLite, we need to recreate the table without the UNIQUE constraint
# Create a new table without the UNIQUE constraint on is_default
op.create_table(
"project_new",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("name", sa.String(), nullable=False),
sa.Column("description", sa.Text(), nullable=True),
sa.Column("permalink", sa.String(), nullable=False),
sa.Column("path", sa.String(), nullable=False),
sa.Column("is_active", sa.Boolean(), nullable=False),
sa.Column("is_default", sa.Boolean(), nullable=True), # No UNIQUE constraint!
sa.Column("created_at", sa.DateTime(), nullable=False),
sa.Column("updated_at", sa.DateTime(), nullable=False),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("name"),
sa.UniqueConstraint("permalink"),
)
connection = op.get_bind()
is_sqlite = connection.dialect.name == "sqlite"
# Copy data from old table to new table
op.execute("INSERT INTO project_new SELECT * FROM project")
if is_sqlite:
# For SQLite, we need to recreate the table without the UNIQUE constraint
# Create a new table without the UNIQUE constraint on is_default
op.create_table(
"project_new",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("name", sa.String(), nullable=False),
sa.Column("description", sa.Text(), nullable=True),
sa.Column("permalink", sa.String(), nullable=False),
sa.Column("path", sa.String(), nullable=False),
sa.Column("is_active", sa.Boolean(), nullable=False),
sa.Column("is_default", sa.Boolean(), nullable=True), # No UNIQUE constraint!
sa.Column("created_at", sa.DateTime(), nullable=False),
sa.Column("updated_at", sa.DateTime(), nullable=False),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("name"),
sa.UniqueConstraint("permalink"),
)
# Drop the old table
op.drop_table("project")
# Copy data from old table to new table
op.execute("INSERT INTO project_new SELECT * FROM project")
# Rename the new table
op.rename_table("project_new", "project")
# Drop the old table
op.drop_table("project")
# Recreate the indexes
with op.batch_alter_table("project", schema=None) as batch_op:
batch_op.create_index("ix_project_created_at", ["created_at"], unique=False)
batch_op.create_index("ix_project_name", ["name"], unique=True)
batch_op.create_index("ix_project_path", ["path"], unique=False)
batch_op.create_index("ix_project_permalink", ["permalink"], unique=True)
batch_op.create_index("ix_project_updated_at", ["updated_at"], unique=False)
# Rename the new table
op.rename_table("project_new", "project")
# Recreate the indexes
with op.batch_alter_table("project", schema=None) as batch_op:
batch_op.create_index("ix_project_created_at", ["created_at"], unique=False)
batch_op.create_index("ix_project_name", ["name"], unique=True)
batch_op.create_index("ix_project_path", ["path"], unique=False)
batch_op.create_index("ix_project_permalink", ["permalink"], unique=True)
batch_op.create_index("ix_project_updated_at", ["updated_at"], unique=False)
else:
# For Postgres, we can simply drop the constraint
with op.batch_alter_table("project", schema=None) as batch_op:
batch_op.drop_constraint("project_is_default_key", type_="unique")
def downgrade() -> None:
@@ -0,0 +1,24 @@
"""Merge multiple heads
Revision ID: 6830751f5fb6
Revises: a2b3c4d5e6f7, g9a0b3c4d5e6
Create Date: 2025-12-29 12:46:46.476268
"""
from typing import Sequence, Union
# revision identifiers, used by Alembic.
revision: str = '6830751f5fb6'
down_revision: Union[str, Sequence[str], None] = ('a2b3c4d5e6f7', 'g9a0b3c4d5e6')
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
@@ -0,0 +1,56 @@
"""Add cascade delete FK from search_index to entity
Revision ID: a2b3c4d5e6f7
Revises: f8a9b2c3d4e5
Create Date: 2025-12-02 07:00:00.000000
"""
from typing import Sequence, Union
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "a2b3c4d5e6f7"
down_revision: Union[str, None] = "f8a9b2c3d4e5"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Add FK with CASCADE delete from search_index.entity_id to entity.id.
This migration is Postgres-only because:
- SQLite uses FTS5 virtual tables which don't support foreign keys
- The FK enables automatic cleanup of search_index entries when entities are deleted
"""
connection = op.get_bind()
dialect = connection.dialect.name
if dialect == "postgresql":
# First, clean up any orphaned search_index entries where entity no longer exists
op.execute("""
DELETE FROM search_index
WHERE entity_id IS NOT NULL
AND entity_id NOT IN (SELECT id FROM entity)
""")
# Add FK with CASCADE - nullable FK allows search_index entries without entity_id
op.create_foreign_key(
"fk_search_index_entity_id",
"search_index",
"entity",
["entity_id"],
["id"],
ondelete="CASCADE",
)
def downgrade() -> None:
"""Remove the FK constraint."""
connection = op.get_bind()
dialect = connection.dialect.name
if dialect == "postgresql":
op.drop_constraint("fk_search_index_entity_id", "search_index", type_="foreignkey")
@@ -21,6 +21,12 @@ depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Upgrade database schema to use new search index with content_stems and content_snippet."""
# This migration is SQLite-specific (FTS5 virtual tables)
# For Postgres, the search_index table is created via ORM models
connection = op.get_bind()
if connection.dialect.name != "sqlite":
return
# First, drop the existing search_index table
op.execute("DROP TABLE IF EXISTS search_index")
@@ -59,6 +65,13 @@ def upgrade() -> None:
def downgrade() -> None:
"""Downgrade database schema to use old search index."""
# This migration is SQLite-specific (FTS5 virtual tables)
# For Postgres, the search_index table is managed via ORM models
connection = op.get_bind()
if connection.dialect.name != "sqlite":
return
# Drop the updated search_index table
op.execute("DROP TABLE IF EXISTS search_index")
@@ -0,0 +1,239 @@
"""Add project_id to relation/observation and pg_trgm for fuzzy link resolution
Revision ID: f8a9b2c3d4e5
Revises: 314f1ea54dc4
Create Date: 2025-12-01 12:00:00.000000
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy import text
def column_exists(connection, table: str, column: str) -> bool:
"""Check if a column exists in a table (idempotent migration support)."""
if connection.dialect.name == "postgresql":
result = connection.execute(
text(
"SELECT 1 FROM information_schema.columns "
"WHERE table_name = :table AND column_name = :column"
),
{"table": table, "column": column},
)
return result.fetchone() is not None
else:
# SQLite
result = connection.execute(text(f"PRAGMA table_info({table})"))
columns = [row[1] for row in result]
return column in columns
def index_exists(connection, index_name: str) -> bool:
"""Check if an index exists (idempotent migration support)."""
if connection.dialect.name == "postgresql":
result = connection.execute(
text("SELECT 1 FROM pg_indexes WHERE indexname = :index_name"),
{"index_name": index_name},
)
return result.fetchone() is not None
else:
# SQLite
result = connection.execute(
text("SELECT 1 FROM sqlite_master WHERE type='index' AND name = :index_name"),
{"index_name": index_name},
)
return result.fetchone() is not None
# revision identifiers, used by Alembic.
revision: str = "f8a9b2c3d4e5"
down_revision: Union[str, None] = "314f1ea54dc4"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Add project_id to relation and observation tables, plus pg_trgm indexes.
This migration:
1. Adds project_id column to relation and observation tables (denormalization)
2. Backfills project_id from the associated entity
3. Enables pg_trgm extension for trigram-based fuzzy matching (Postgres only)
4. Creates GIN indexes on entity title and permalink for fast similarity searches
5. Creates partial index on unresolved relations for efficient bulk resolution
"""
connection = op.get_bind()
dialect = connection.dialect.name
# -------------------------------------------------------------------------
# Add project_id to relation table
# -------------------------------------------------------------------------
# Step 1: Add project_id column as nullable first (idempotent)
if not column_exists(connection, "relation", "project_id"):
op.add_column("relation", sa.Column("project_id", sa.Integer(), nullable=True))
# Step 2: Backfill project_id from entity.project_id via from_id
if dialect == "postgresql":
op.execute("""
UPDATE relation
SET project_id = entity.project_id
FROM entity
WHERE relation.from_id = entity.id
""")
else:
# SQLite syntax
op.execute("""
UPDATE relation
SET project_id = (
SELECT entity.project_id
FROM entity
WHERE entity.id = relation.from_id
)
""")
# Step 3: Make project_id NOT NULL and add foreign key
if dialect == "postgresql":
op.alter_column("relation", "project_id", nullable=False)
op.create_foreign_key(
"fk_relation_project_id",
"relation",
"project",
["project_id"],
["id"],
)
else:
# SQLite requires batch operations for ALTER COLUMN
with op.batch_alter_table("relation") as batch_op:
batch_op.alter_column("project_id", nullable=False)
batch_op.create_foreign_key(
"fk_relation_project_id",
"project",
["project_id"],
["id"],
)
# Step 4: Create index on relation.project_id (idempotent)
if not index_exists(connection, "ix_relation_project_id"):
op.create_index("ix_relation_project_id", "relation", ["project_id"])
# -------------------------------------------------------------------------
# Add project_id to observation table
# -------------------------------------------------------------------------
# Step 1: Add project_id column as nullable first (idempotent)
if not column_exists(connection, "observation", "project_id"):
op.add_column("observation", sa.Column("project_id", sa.Integer(), nullable=True))
# Step 2: Backfill project_id from entity.project_id via entity_id
if dialect == "postgresql":
op.execute("""
UPDATE observation
SET project_id = entity.project_id
FROM entity
WHERE observation.entity_id = entity.id
""")
else:
# SQLite syntax
op.execute("""
UPDATE observation
SET project_id = (
SELECT entity.project_id
FROM entity
WHERE entity.id = observation.entity_id
)
""")
# Step 3: Make project_id NOT NULL and add foreign key
if dialect == "postgresql":
op.alter_column("observation", "project_id", nullable=False)
op.create_foreign_key(
"fk_observation_project_id",
"observation",
"project",
["project_id"],
["id"],
)
else:
# SQLite requires batch operations for ALTER COLUMN
with op.batch_alter_table("observation") as batch_op:
batch_op.alter_column("project_id", nullable=False)
batch_op.create_foreign_key(
"fk_observation_project_id",
"project",
["project_id"],
["id"],
)
# Step 4: Create index on observation.project_id (idempotent)
if not index_exists(connection, "ix_observation_project_id"):
op.create_index("ix_observation_project_id", "observation", ["project_id"])
# Postgres-specific: pg_trgm and GIN indexes
if dialect == "postgresql":
# Enable pg_trgm extension for fuzzy string matching
op.execute("CREATE EXTENSION IF NOT EXISTS pg_trgm")
# Create trigram indexes on entity table for fuzzy matching
# GIN indexes with gin_trgm_ops support similarity searches
op.execute("""
CREATE INDEX IF NOT EXISTS idx_entity_title_trgm
ON entity USING gin (title gin_trgm_ops)
""")
op.execute("""
CREATE INDEX IF NOT EXISTS idx_entity_permalink_trgm
ON entity USING gin (permalink gin_trgm_ops)
""")
# Create partial index on unresolved relations for efficient bulk resolution
# This makes "WHERE to_id IS NULL AND project_id = X" queries very fast
op.execute("""
CREATE INDEX IF NOT EXISTS idx_relation_unresolved
ON relation (project_id, to_name)
WHERE to_id IS NULL
""")
# Create index on relation.to_name for join performance in bulk resolution
op.execute("""
CREATE INDEX IF NOT EXISTS idx_relation_to_name
ON relation (to_name)
""")
def downgrade() -> None:
"""Remove project_id from relation/observation and pg_trgm indexes."""
connection = op.get_bind()
dialect = connection.dialect.name
if dialect == "postgresql":
# Drop Postgres-specific indexes
op.execute("DROP INDEX IF EXISTS idx_relation_to_name")
op.execute("DROP INDEX IF EXISTS idx_relation_unresolved")
op.execute("DROP INDEX IF EXISTS idx_entity_permalink_trgm")
op.execute("DROP INDEX IF EXISTS idx_entity_title_trgm")
# Note: We don't drop the pg_trgm extension as other code may depend on it
# Drop project_id from observation
op.drop_index("ix_observation_project_id", table_name="observation")
op.drop_constraint("fk_observation_project_id", "observation", type_="foreignkey")
op.drop_column("observation", "project_id")
# Drop project_id from relation
op.drop_index("ix_relation_project_id", table_name="relation")
op.drop_constraint("fk_relation_project_id", "relation", type_="foreignkey")
op.drop_column("relation", "project_id")
else:
# SQLite requires batch operations
op.drop_index("ix_observation_project_id", table_name="observation")
with op.batch_alter_table("observation") as batch_op:
batch_op.drop_constraint("fk_observation_project_id", type_="foreignkey")
batch_op.drop_column("project_id")
op.drop_index("ix_relation_project_id", table_name="relation")
with op.batch_alter_table("relation") as batch_op:
batch_op.drop_constraint("fk_relation_project_id", type_="foreignkey")
batch_op.drop_column("project_id")
@@ -0,0 +1,173 @@
"""Add external_id UUID column to project and entity tables
Revision ID: g9a0b3c4d5e6
Revises: f8a9b2c3d4e5
Create Date: 2025-12-29 10:00:00.000000
"""
import uuid
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy import text
def column_exists(connection, table: str, column: str) -> bool:
"""Check if a column exists in a table (idempotent migration support)."""
if connection.dialect.name == "postgresql":
result = connection.execute(
text(
"SELECT 1 FROM information_schema.columns "
"WHERE table_name = :table AND column_name = :column"
),
{"table": table, "column": column},
)
return result.fetchone() is not None
else:
# SQLite
result = connection.execute(text(f"PRAGMA table_info({table})"))
columns = [row[1] for row in result]
return column in columns
def index_exists(connection, index_name: str) -> bool:
"""Check if an index exists (idempotent migration support)."""
if connection.dialect.name == "postgresql":
result = connection.execute(
text("SELECT 1 FROM pg_indexes WHERE indexname = :index_name"),
{"index_name": index_name},
)
return result.fetchone() is not None
else:
# SQLite
result = connection.execute(
text("SELECT 1 FROM sqlite_master WHERE type='index' AND name = :index_name"),
{"index_name": index_name},
)
return result.fetchone() is not None
# revision identifiers, used by Alembic.
revision: str = "g9a0b3c4d5e6"
down_revision: Union[str, None] = "f8a9b2c3d4e5"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Add external_id UUID column to project and entity tables.
This migration:
1. Adds external_id column to project table
2. Adds external_id column to entity table
3. Generates UUIDs for existing rows
4. Creates unique indexes on both columns
"""
connection = op.get_bind()
dialect = connection.dialect.name
# -------------------------------------------------------------------------
# Add external_id to project table
# -------------------------------------------------------------------------
if not column_exists(connection, "project", "external_id"):
# Step 1: Add external_id column as nullable first
op.add_column("project", sa.Column("external_id", sa.String(), nullable=True))
# Step 2: Generate UUIDs for existing rows
if dialect == "postgresql":
# Postgres has gen_random_uuid() function
op.execute("""
UPDATE project
SET external_id = gen_random_uuid()::text
WHERE external_id IS NULL
""")
else:
# SQLite: need to generate UUIDs in Python
result = connection.execute(text("SELECT id FROM project WHERE external_id IS NULL"))
for row in result:
new_uuid = str(uuid.uuid4())
connection.execute(
text("UPDATE project SET external_id = :uuid WHERE id = :id"),
{"uuid": new_uuid, "id": row[0]},
)
# Step 3: Make external_id NOT NULL
if dialect == "postgresql":
op.alter_column("project", "external_id", nullable=False)
else:
# SQLite requires batch operations for ALTER COLUMN
with op.batch_alter_table("project") as batch_op:
batch_op.alter_column("external_id", nullable=False)
# Step 4: Create unique index on project.external_id (idempotent)
if not index_exists(connection, "ix_project_external_id"):
op.create_index("ix_project_external_id", "project", ["external_id"], unique=True)
# -------------------------------------------------------------------------
# Add external_id to entity table
# -------------------------------------------------------------------------
if not column_exists(connection, "entity", "external_id"):
# Step 1: Add external_id column as nullable first
op.add_column("entity", sa.Column("external_id", sa.String(), nullable=True))
# Step 2: Generate UUIDs for existing rows
if dialect == "postgresql":
# Postgres has gen_random_uuid() function
op.execute("""
UPDATE entity
SET external_id = gen_random_uuid()::text
WHERE external_id IS NULL
""")
else:
# SQLite: need to generate UUIDs in Python
result = connection.execute(text("SELECT id FROM entity WHERE external_id IS NULL"))
for row in result:
new_uuid = str(uuid.uuid4())
connection.execute(
text("UPDATE entity SET external_id = :uuid WHERE id = :id"),
{"uuid": new_uuid, "id": row[0]},
)
# Step 3: Make external_id NOT NULL
if dialect == "postgresql":
op.alter_column("entity", "external_id", nullable=False)
else:
# SQLite requires batch operations for ALTER COLUMN
with op.batch_alter_table("entity") as batch_op:
batch_op.alter_column("external_id", nullable=False)
# Step 4: Create unique index on entity.external_id (idempotent)
if not index_exists(connection, "ix_entity_external_id"):
op.create_index("ix_entity_external_id", "entity", ["external_id"], unique=True)
def downgrade() -> None:
"""Remove external_id columns from project and entity tables."""
connection = op.get_bind()
dialect = connection.dialect.name
# Drop from entity table
if index_exists(connection, "ix_entity_external_id"):
op.drop_index("ix_entity_external_id", table_name="entity")
if column_exists(connection, "entity", "external_id"):
if dialect == "postgresql":
op.drop_column("entity", "external_id")
else:
with op.batch_alter_table("entity") as batch_op:
batch_op.drop_column("external_id")
# Drop from project table
if index_exists(connection, "ix_project_external_id"):
op.drop_index("ix_project_external_id", table_name="project")
if column_exists(connection, "project", "external_id"):
if dialect == "postgresql":
op.drop_column("project", "external_id")
else:
with op.batch_alter_table("project") as batch_op:
batch_op.drop_column("external_id")
+44 -23
View File
@@ -1,6 +1,5 @@
"""FastAPI application for basic-memory knowledge graph API."""
import asyncio
from contextlib import asynccontextmanager
from fastapi import FastAPI, HTTPException
@@ -8,7 +7,7 @@ from fastapi.exception_handlers import http_exception_handler
from loguru import logger
from basic_memory import __version__ as version
from basic_memory import db
from basic_memory.api.container import ApiContainer, set_container
from basic_memory.api.routers import (
directory_router,
importer_router,
@@ -20,42 +19,57 @@ from basic_memory.api.routers import (
search,
prompt_router,
)
from basic_memory.config import ConfigManager
from basic_memory.services.initialization import initialize_file_sync, initialize_app
from basic_memory.api.v2.routers import (
knowledge_router as v2_knowledge,
project_router as v2_project,
memory_router as v2_memory,
search_router as v2_search,
resource_router as v2_resource,
directory_router as v2_directory,
prompt_router as v2_prompt,
importer_router as v2_importer,
)
from basic_memory.config import init_api_logging
from basic_memory.services.initialization import initialize_app
@asynccontextmanager
async def lifespan(app: FastAPI): # pragma: no cover
"""Lifecycle manager for the FastAPI app. Not called in stdio mcp mode"""
app_config = ConfigManager().config
logger.info("Starting Basic Memory API")
# Initialize logging for API (stdout in cloud mode, file otherwise)
init_api_logging()
await initialize_app(app_config)
# --- Composition Root ---
# Create container and read config (single point of config access)
container = ApiContainer.create()
set_container(container)
app.state.container = container
logger.info(f"Starting Basic Memory API (mode={container.mode.name})")
await initialize_app(container.config)
# Cache database connections in app state for performance
logger.info("Initializing database and caching connections...")
engine, session_maker = await db.get_or_create_db(app_config.database_path)
engine, session_maker = await container.init_database()
app.state.engine = engine
app.state.session_maker = session_maker
logger.info("Database connections cached in app state")
logger.info(f"Sync changes enabled: {app_config.sync_changes}")
if app_config.sync_changes:
# start file sync task in background
app.state.sync_task = asyncio.create_task(initialize_file_sync(app_config))
else:
logger.info("Sync changes disabled. Skipping file sync service.")
# Create and start sync coordinator (lifecycle centralized in coordinator)
sync_coordinator = container.create_sync_coordinator()
await sync_coordinator.start()
app.state.sync_coordinator = sync_coordinator
# proceed with startup
# Proceed with startup
yield
# Shutdown - coordinator handles clean task cancellation
logger.info("Shutting down Basic Memory API")
if app.state.sync_task:
logger.info("Stopping sync...")
app.state.sync_task.cancel() # pyright: ignore
await sync_coordinator.stop()
await db.shutdown_db()
await container.shutdown_database()
# Initialize FastAPI app
@@ -66,8 +80,17 @@ app = FastAPI(
lifespan=lifespan,
)
# Include v2 routers FIRST (more specific paths must match before /{project} catch-all)
app.include_router(v2_knowledge, prefix="/v2/projects/{project_id}")
app.include_router(v2_memory, prefix="/v2/projects/{project_id}")
app.include_router(v2_search, prefix="/v2/projects/{project_id}")
app.include_router(v2_resource, prefix="/v2/projects/{project_id}")
app.include_router(v2_directory, prefix="/v2/projects/{project_id}")
app.include_router(v2_prompt, prefix="/v2/projects/{project_id}")
app.include_router(v2_importer, prefix="/v2/projects/{project_id}")
app.include_router(v2_project, prefix="/v2")
# Include routers
# Include v1 routers (/{project} is a catch-all, must come after specific prefixes)
app.include_router(knowledge.router, prefix="/{project}")
app.include_router(memory.router, prefix="/{project}")
app.include_router(resource.router, prefix="/{project}")
@@ -77,12 +100,10 @@ app.include_router(directory_router.router, prefix="/{project}")
app.include_router(prompt_router.router, prefix="/{project}")
app.include_router(importer_router.router, prefix="/{project}")
# Project resource router works accross projects
# Project resource router works across projects
app.include_router(project.project_resource_router)
app.include_router(management.router)
# Auth routes are handled by FastMCP automatically when auth is enabled
@app.exception_handler(Exception)
async def exception_handler(request, exc): # pragma: no cover
+133
View File
@@ -0,0 +1,133 @@
"""API composition root for Basic Memory.
This container owns reading ConfigManager and environment variables for the
API entrypoint. Downstream modules receive config/dependencies explicitly
rather than reading globals.
Design principles:
- Only this module reads ConfigManager directly
- Runtime mode (cloud/local/test) is resolved here
- Factories for services are provided, not singletons
"""
from dataclasses import dataclass
from typing import TYPE_CHECKING
from sqlalchemy.ext.asyncio import AsyncEngine, async_sessionmaker, AsyncSession
from basic_memory import db
from basic_memory.config import BasicMemoryConfig, ConfigManager
from basic_memory.runtime import RuntimeMode, resolve_runtime_mode
if TYPE_CHECKING: # pragma: no cover
from basic_memory.sync import SyncCoordinator
@dataclass
class ApiContainer:
"""Composition root for the API entrypoint.
Holds resolved configuration and runtime context.
Created once at app startup, then used to wire dependencies.
"""
config: BasicMemoryConfig
mode: RuntimeMode
# --- Database ---
# Cached database connections (set during lifespan startup)
engine: AsyncEngine | None = None
session_maker: async_sessionmaker[AsyncSession] | None = None
@classmethod
def create(cls) -> "ApiContainer": # pragma: no cover
"""Create container by reading ConfigManager.
This is the single point where API reads global config.
"""
config = ConfigManager().config
mode = resolve_runtime_mode(
cloud_mode_enabled=config.cloud_mode_enabled,
is_test_env=config.is_test_env,
)
return cls(config=config, mode=mode)
# --- Runtime Mode Properties ---
@property
def should_sync_files(self) -> bool:
"""Whether file sync should be started.
Sync is enabled when:
- sync_changes is True in config
- Not in test mode (tests manage their own sync)
"""
return self.config.sync_changes and not self.mode.is_test
@property
def sync_skip_reason(self) -> str | None: # pragma: no cover
"""Reason why sync is skipped, or None if sync should run.
Useful for logging why sync was disabled.
"""
if self.mode.is_test:
return "Test environment detected"
if not self.config.sync_changes:
return "Sync changes disabled"
return None
def create_sync_coordinator(self) -> "SyncCoordinator": # pragma: no cover
"""Create a SyncCoordinator with this container's settings.
Returns:
SyncCoordinator configured for this runtime environment
"""
# Deferred import to avoid circular dependency
from basic_memory.sync import SyncCoordinator
return SyncCoordinator(
config=self.config,
should_sync=self.should_sync_files,
skip_reason=self.sync_skip_reason,
)
# --- Database Factory ---
async def init_database( # pragma: no cover
self,
) -> tuple[AsyncEngine, async_sessionmaker[AsyncSession]]:
"""Initialize and cache database connections.
Returns:
Tuple of (engine, session_maker)
"""
engine, session_maker = await db.get_or_create_db(self.config.database_path)
self.engine = engine
self.session_maker = session_maker
return engine, session_maker
async def shutdown_database(self) -> None: # pragma: no cover
"""Clean up database connections."""
await db.shutdown_db()
# Module-level container instance (set by lifespan)
# This allows deps.py to access the container without reading ConfigManager
_container: ApiContainer | None = None
def get_container() -> ApiContainer:
"""Get the current API container.
Raises:
RuntimeError: If container hasn't been initialized
"""
if _container is None:
raise RuntimeError("API container not initialized. Call set_container() first.")
return _container
def set_container(container: ApiContainer) -> None:
"""Set the API container (called by lifespan)."""
global _container
_container = container
@@ -1,4 +1,11 @@
"""Router for knowledge graph operations."""
"""Router for knowledge graph operations.
⚠️ DEPRECATED: This v1 API is deprecated and will be removed on June 30, 2026.
Please migrate to /v2/{project}/knowledge endpoints which use entity IDs instead
of path-based identifiers for improved performance and stability.
Migration guide: See docs/migration/v1-to-v2.md
"""
from typing import Annotated
@@ -25,7 +32,11 @@ from basic_memory.schemas import (
from basic_memory.schemas.request import EditEntityRequest, MoveEntityRequest
from basic_memory.schemas.base import Permalink, Entity
router = APIRouter(prefix="/knowledge", tags=["knowledge"])
router = APIRouter(
prefix="/knowledge",
tags=["knowledge"],
deprecated=True, # Marks entire router as deprecated in OpenAPI docs
)
async def resolve_relations_background(sync_service, entity_id: int, entity_permalink: str) -> None:
@@ -40,9 +51,10 @@ async def resolve_relations_background(sync_service, entity_id: int, entity_perm
logger.debug(
f"Background: Resolved relations for entity {entity_permalink} (id={entity_id})"
)
except Exception as e:
# Log but don't fail - this is a background task
logger.warning(
except Exception as e: # pragma: no cover
# Log but don't fail - this is a background task.
# Avoid forcing synthetic failures just for coverage.
logger.warning( # pragma: no cover
f"Background: Failed to resolve relations for entity {entity_permalink}: {e}"
)
+68 -14
View File
@@ -50,6 +50,8 @@ async def get_project(
) # pragma: no cover
return ProjectItem(
id=found_project.id,
external_id=found_project.external_id,
name=found_project.name,
path=normalize_project_path(found_project.path),
is_default=found_project.is_default or False,
@@ -80,9 +82,18 @@ async def update_project(
raise HTTPException(status_code=400, detail="Path must be absolute")
# Get original project info for the response
old_project = await project_service.get_project(name)
if not old_project:
raise HTTPException(
status_code=400, detail=f"Project '{name}' not found in configuration"
)
old_project_info = ProjectItem(
name=name,
path=project_service.projects.get(name, ""),
id=old_project.id,
external_id=old_project.external_id,
name=old_project.name,
path=old_project.path,
is_default=old_project.is_default or False,
)
if path:
@@ -91,17 +102,27 @@ async def update_project(
await project_service.update_project(name, is_active=is_active)
# Get updated project info
updated_path = path if path else project_service.projects.get(name, "")
updated_project = await project_service.get_project(name)
if not updated_project:
raise HTTPException( # pragma: no cover
status_code=404, detail=f"Project '{name}' not found after update"
)
return ProjectStatusResponse(
message=f"Project '{name}' updated successfully",
status="success",
default=(name == project_service.default_project),
old_project=old_project_info,
new_project=ProjectItem(name=name, path=updated_path),
new_project=ProjectItem(
id=updated_project.id,
external_id=updated_project.external_id,
name=updated_project.name,
path=updated_project.path,
is_default=updated_project.is_default or False,
),
)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
raise HTTPException(status_code=400, detail=str(e)) # pragma: no cover
# Sync project filesystem
@@ -165,10 +186,10 @@ async def project_sync_status(
Returns:
Scan report with details on files that need syncing
"""
logger.info(f"Scanning filesystem for project: {project_config.name}")
sync_report = await sync_service.scan(project_config.home)
logger.info(f"Scanning filesystem for project: {project_config.name}") # pragma: no cover
sync_report = await sync_service.scan(project_config.home) # pragma: no cover
return SyncReportResponse.from_sync_report(sync_report)
return SyncReportResponse.from_sync_report(sync_report) # pragma: no cover
# List all available projects
@@ -186,6 +207,8 @@ async def list_projects(
project_items = [
ProjectItem(
id=project.id,
external_id=project.external_id,
name=project.name,
path=normalize_project_path(project.path),
is_default=project.is_default or False,
@@ -232,6 +255,8 @@ async def add_project(
status="success",
default=existing_project.is_default or False,
new_project=ProjectItem(
id=existing_project.id,
external_id=existing_project.external_id,
name=existing_project.name,
path=existing_project.path,
is_default=existing_project.is_default or False,
@@ -250,12 +275,21 @@ async def add_project(
project_data.name, project_data.path, set_default=project_data.set_default
)
# Fetch the newly created project to get its ID
new_project = await project_service.get_project(project_data.name)
if not new_project:
raise HTTPException(status_code=500, detail="Failed to retrieve newly created project")
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message=f"Project '{project_data.name}' added successfully",
message=f"Project '{new_project.name}' added successfully",
status="success",
default=project_data.set_default,
new_project=ProjectItem(
name=project_data.name, path=project_data.path, is_default=project_data.set_default
id=new_project.id,
external_id=new_project.external_id,
name=new_project.name,
path=new_project.path,
is_default=new_project.is_default or False,
),
)
except ValueError as e: # pragma: no cover
@@ -303,10 +337,16 @@ async def remove_project(
await project_service.remove_project(name, delete_notes=delete_notes)
return ProjectStatusResponse(
message=f"Project '{name}' removed successfully",
message=f"Project '{old_project.name}' removed successfully",
status="success",
default=False,
old_project=ProjectItem(name=old_project.name, path=old_project.path),
old_project=ProjectItem(
id=old_project.id,
external_id=old_project.external_id,
name=old_project.name,
path=old_project.path,
is_default=old_project.is_default or False,
),
new_project=None,
)
except ValueError as e: # pragma: no cover
@@ -349,8 +389,16 @@ async def set_default_project(
message=f"Project '{name}' set as default successfully",
status="success",
default=True,
old_project=ProjectItem(name=default_name, path=default_project.path),
old_project=ProjectItem(
id=default_project.id,
external_id=default_project.external_id,
name=default_name,
path=default_project.path,
is_default=False,
),
new_project=ProjectItem(
id=new_default_project.id,
external_id=new_default_project.external_id,
name=name,
path=new_default_project.path,
is_default=True,
@@ -378,7 +426,13 @@ async def get_default_project(
status_code=404, detail=f"Default Project: '{default_name}' does not exist"
)
return ProjectItem(name=default_project.name, path=default_project.path, is_default=True)
return ProjectItem(
id=default_project.id,
external_id=default_project.external_id,
name=default_project.name,
path=default_project.path,
is_default=True,
)
# Synchronize projects between config and database
+37 -27
View File
@@ -2,9 +2,9 @@
import tempfile
from pathlib import Path
from typing import Annotated
from typing import Annotated, Union
from fastapi import APIRouter, HTTPException, BackgroundTasks, Body
from fastapi import APIRouter, HTTPException, BackgroundTasks, Body, Response
from fastapi.responses import FileResponse, JSONResponse
from loguru import logger
@@ -25,6 +25,17 @@ from datetime import datetime
router = APIRouter(prefix="/resource", tags=["resources"])
def _mtime_to_datetime(entity: EntityModel) -> datetime:
"""Convert entity mtime (file modification time) to datetime.
Returns the file's actual modification time, falling back to updated_at
if mtime is not available.
"""
if entity.mtime: # pragma: no cover
return datetime.fromtimestamp(entity.mtime).astimezone() # pragma: no cover
return entity.updated_at
def get_entity_ids(item: SearchIndexRow) -> set[int]:
match item.type:
case SearchItemType.ENTITY:
@@ -39,7 +50,7 @@ def get_entity_ids(item: SearchIndexRow) -> set[int]:
raise ValueError(f"Unexpected type: {item.type}")
@router.get("/{identifier:path}")
@router.get("/{identifier:path}", response_model=None)
async def get_resource_content(
config: ProjectConfigDep,
link_resolver: LinkResolverDep,
@@ -50,7 +61,7 @@ async def get_resource_content(
identifier: str,
page: int = 1,
page_size: int = 10,
) -> FileResponse:
) -> Union[Response, FileResponse]:
"""Get resource content by identifier: name or permalink."""
logger.debug(f"Getting content for: {identifier}")
@@ -81,13 +92,16 @@ async def get_resource_content(
# return single response
if len(results) == 1:
entity = results[0]
file_path = Path(f"{config.home}/{entity.file_path}")
if not file_path.exists():
# Check file exists via file_service (for cloud compatibility)
if not await file_service.exists(entity.file_path):
raise HTTPException(
status_code=404,
detail=f"File not found: {file_path}",
detail=f"File not found: {entity.file_path}",
)
return FileResponse(path=file_path)
# Read content via file_service as bytes (works with both local and S3)
content = await file_service.read_file_bytes(entity.file_path)
content_type = file_service.content_type(entity.file_path)
return Response(content=content, media_type=content_type)
# for multiple files, initialize a temporary file for writing the results
with tempfile.NamedTemporaryFile(delete=False, mode="w", suffix=".md") as tmp_file:
@@ -97,7 +111,7 @@ async def get_resource_content(
# Read content for each entity
content = await file_service.read_entity_content(result)
memory_url = normalize_memory_url(result.permalink)
modified_date = result.updated_at.isoformat()
modified_date = _mtime_to_datetime(result).isoformat()
checksum = result.checksum[:8] if result.checksum else ""
# Prepare the delimited content
@@ -155,11 +169,11 @@ async def write_resource(
# FastAPI should validate this, but if a dict somehow gets through
# (e.g., via JSON body parsing), we need to catch it here
if isinstance(content, dict):
logger.error(
logger.error( # pragma: no cover
f"Error writing resource {file_path}: "
f"content is a dict, expected string. Keys: {list(content.keys())}"
)
raise HTTPException(
raise HTTPException( # pragma: no cover
status_code=400,
detail="content must be a string, not a dict. "
"Ensure request body is sent as raw string content, not JSON object.",
@@ -171,21 +185,17 @@ async def write_resource(
else:
content_str = str(content)
# Get full file path
full_path = Path(f"{config.home}/{file_path}")
# Ensure parent directory exists
full_path.parent.mkdir(parents=True, exist_ok=True)
# Write content to file
checksum = await file_service.write_file(full_path, content_str)
# Cloud compatibility: do not assume a local filesystem path structure.
# Delegate directory creation + writes to the configured FileService (local or S3).
await file_service.ensure_directory(Path(file_path).parent)
checksum = await file_service.write_file(file_path, content_str)
# Get file info
file_stats = file_service.file_stats(full_path)
file_metadata = await file_service.get_file_metadata(file_path)
# Determine file details
file_name = Path(file_path).name
content_type = file_service.content_type(full_path)
content_type = file_service.content_type(file_path)
entity_type = "canvas" if file_path.endswith(".canvas") else "file"
@@ -202,7 +212,7 @@ async def write_resource(
"content_type": content_type,
"file_path": file_path,
"checksum": checksum,
"updated_at": datetime.fromtimestamp(file_stats.st_mtime).astimezone(),
"updated_at": file_metadata.modified_at,
},
)
status_code = 200
@@ -214,8 +224,8 @@ async def write_resource(
content_type=content_type,
file_path=file_path,
checksum=checksum,
created_at=datetime.fromtimestamp(file_stats.st_ctime).astimezone(),
updated_at=datetime.fromtimestamp(file_stats.st_mtime).astimezone(),
created_at=file_metadata.created_at,
updated_at=file_metadata.modified_at,
)
entity = await entity_repository.add(entity)
status_code = 201
@@ -229,9 +239,9 @@ async def write_resource(
content={
"file_path": file_path,
"checksum": checksum,
"size": file_stats.st_size,
"created_at": file_stats.st_ctime,
"modified_at": file_stats.st_mtime,
"size": file_metadata.size,
"created_at": file_metadata.created_at.timestamp(),
"modified_at": file_metadata.modified_at.timestamp(),
},
)
except Exception as e: # pragma: no cover
+53 -14
View File
@@ -24,11 +24,30 @@ async def to_graph_context(
page: Optional[int] = None,
page_size: Optional[int] = None,
):
# First pass: collect all entity IDs needed for relations
entity_ids_needed: set[int] = set()
for context_item in context_result.results:
for item in (
[context_item.primary_result] + context_item.observations + context_item.related_results
):
if item.type == SearchItemType.RELATION:
if item.from_id: # pyright: ignore
entity_ids_needed.add(item.from_id) # pyright: ignore
if item.to_id:
entity_ids_needed.add(item.to_id)
# Batch fetch all entities at once
entity_lookup: dict[int, str] = {}
if entity_ids_needed:
entities = await entity_repository.find_by_ids(list(entity_ids_needed))
entity_lookup = {e.id: e.title for e in entities}
# Helper function to convert items to summaries
async def to_summary(item: SearchIndexRow | ContextResultRow):
def to_summary(item: SearchIndexRow | ContextResultRow):
match item.type:
case SearchItemType.ENTITY:
return EntitySummary(
entity_id=item.id,
title=item.title, # pyright: ignore
permalink=item.permalink,
content=item.content,
@@ -37,6 +56,8 @@ async def to_graph_context(
)
case SearchItemType.OBSERVATION:
return ObservationSummary(
observation_id=item.id,
entity_id=item.entity_id, # pyright: ignore
title=item.title, # pyright: ignore
file_path=item.file_path,
category=item.category, # pyright: ignore
@@ -45,15 +66,19 @@ async def to_graph_context(
created_at=item.created_at,
)
case SearchItemType.RELATION:
from_entity = await entity_repository.find_by_id(item.from_id) # pyright: ignore
to_entity = await entity_repository.find_by_id(item.to_id) if item.to_id else None
from_title = entity_lookup.get(item.from_id) if item.from_id else None # pyright: ignore
to_title = entity_lookup.get(item.to_id) if item.to_id else None
return RelationSummary(
relation_id=item.id,
entity_id=item.entity_id, # pyright: ignore
title=item.title, # pyright: ignore
file_path=item.file_path,
permalink=item.permalink, # pyright: ignore
relation_type=item.relation_type, # pyright: ignore
from_entity=from_entity.title if from_entity else None,
to_entity=to_entity.title if to_entity else None,
from_entity=from_title,
from_entity_id=item.from_id, # pyright: ignore
to_entity=to_title,
to_entity_id=item.to_id,
created_at=item.created_at,
)
case _: # pragma: no cover
@@ -63,23 +88,19 @@ async def to_graph_context(
hierarchical_results = []
for context_item in context_result.results:
# Process primary result
primary_result = await to_summary(context_item.primary_result)
primary_result = to_summary(context_item.primary_result)
# Process observations
observations = []
for obs in context_item.observations:
observations.append(await to_summary(obs))
# Process observations (always ObservationSummary, validated by context_service)
observations = [to_summary(obs) for obs in context_item.observations]
# Process related results
related = []
for rel in context_item.related_results:
related.append(await to_summary(rel))
related = [to_summary(rel) for rel in context_item.related_results]
# Add to hierarchical results
hierarchical_results.append(
ContextResult(
primary_result=primary_result,
observations=observations,
observations=observations, # pyright: ignore[reportArgumentType]
related_results=related,
)
)
@@ -111,6 +132,21 @@ async def to_search_results(entity_service: EntityService, results: List[SearchI
search_results = []
for r in results:
entities = await entity_service.get_entities_by_id([r.entity_id, r.from_id, r.to_id]) # pyright: ignore
# Determine which IDs to set based on type
entity_id = None
observation_id = None
relation_id = None
if r.type == SearchItemType.ENTITY:
entity_id = r.id
elif r.type == SearchItemType.OBSERVATION:
observation_id = r.id
entity_id = r.entity_id # Parent entity
elif r.type == SearchItemType.RELATION:
relation_id = r.id
entity_id = r.entity_id # Parent entity
search_results.append(
SearchResult(
title=r.title, # pyright: ignore
@@ -121,6 +157,9 @@ async def to_search_results(entity_service: EntityService, results: List[SearchI
content=r.content,
file_path=r.file_path,
metadata=r.metadata,
entity_id=entity_id,
observation_id=observation_id,
relation_id=relation_id,
category=r.category,
from_entity=entities[0].permalink if entities else None,
to_entity=entities[1].permalink if len(entities) > 1 else None,
+35
View File
@@ -0,0 +1,35 @@
"""API v2 module - ID-based entity references.
Version 2 of the Basic Memory API uses integer entity IDs as the primary
identifier for improved performance and stability.
Key changes from v1:
- Entity lookups use integer IDs instead of paths/permalinks
- Direct database queries instead of cascading resolution
- Stable references that don't change with file moves
- Better caching support
All v2 routers are registered with the /v2 prefix.
"""
from basic_memory.api.v2.routers import (
knowledge_router,
memory_router,
project_router,
resource_router,
search_router,
directory_router,
prompt_router,
importer_router,
)
__all__ = [
"knowledge_router",
"memory_router",
"project_router",
"resource_router",
"search_router",
"directory_router",
"prompt_router",
"importer_router",
]
@@ -0,0 +1,21 @@
"""V2 API routers."""
from basic_memory.api.v2.routers.knowledge_router import router as knowledge_router
from basic_memory.api.v2.routers.project_router import router as project_router
from basic_memory.api.v2.routers.memory_router import router as memory_router
from basic_memory.api.v2.routers.search_router import router as search_router
from basic_memory.api.v2.routers.resource_router import router as resource_router
from basic_memory.api.v2.routers.directory_router import router as directory_router
from basic_memory.api.v2.routers.prompt_router import router as prompt_router
from basic_memory.api.v2.routers.importer_router import router as importer_router
__all__ = [
"knowledge_router",
"project_router",
"memory_router",
"search_router",
"resource_router",
"directory_router",
"prompt_router",
"importer_router",
]
@@ -0,0 +1,93 @@
"""V2 Directory Router - ID-based directory tree operations.
This router provides directory structure browsing for projects using
external_id UUIDs instead of name-based identifiers.
Key improvements:
- Direct project lookup via external_id UUIDs
- Consistent with other v2 endpoints
- Better performance through indexed queries
"""
from typing import List, Optional
from fastapi import APIRouter, Query, Path
from basic_memory.deps import DirectoryServiceV2ExternalDep
from basic_memory.schemas.directory import DirectoryNode
router = APIRouter(prefix="/directory", tags=["directory-v2"])
@router.get("/tree", response_model=DirectoryNode, response_model_exclude_none=True)
async def get_directory_tree(
directory_service: DirectoryServiceV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
):
"""Get hierarchical directory structure from the knowledge base.
Args:
directory_service: Service for directory operations
project_id: Project external UUID
Returns:
DirectoryNode representing the root of the hierarchical tree structure
"""
# Get a hierarchical directory tree for the specific project
tree = await directory_service.get_directory_tree()
# Return the hierarchical tree
return tree
@router.get("/structure", response_model=DirectoryNode, response_model_exclude_none=True)
async def get_directory_structure(
directory_service: DirectoryServiceV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
):
"""Get folder structure for navigation (no files).
Optimized endpoint for folder tree navigation. Returns only directory nodes
without file metadata. For full tree with files, use /directory/tree.
Args:
directory_service: Service for directory operations
project_id: Project external UUID
Returns:
DirectoryNode tree containing only folders (type="directory")
"""
structure = await directory_service.get_directory_structure()
return structure
@router.get("/list", response_model=List[DirectoryNode], response_model_exclude_none=True)
async def list_directory(
directory_service: DirectoryServiceV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
dir_name: str = Query("/", description="Directory path to list"),
depth: int = Query(1, ge=1, le=10, description="Recursion depth (1-10)"),
file_name_glob: Optional[str] = Query(
None, description="Glob pattern for filtering file names"
),
):
"""List directory contents with filtering and depth control.
Args:
directory_service: Service for directory operations
project_id: Project external UUID
dir_name: Directory path to list (default: root "/")
depth: Recursion depth (1-10, default: 1 for immediate children only)
file_name_glob: Optional glob pattern for filtering file names (e.g., "*.md", "*meeting*")
Returns:
List of DirectoryNode objects matching the criteria
"""
# Get directory listing with filtering
nodes = await directory_service.list_directory(
dir_name=dir_name,
depth=depth,
file_name_glob=file_name_glob,
)
return nodes
@@ -0,0 +1,181 @@
"""V2 Import Router - ID-based data import operations.
This router uses v2 dependencies for consistent project handling with external_id UUIDs.
Import endpoints use project_id in the path for consistency with other v2 endpoints.
"""
import json
import logging
from fastapi import APIRouter, Form, HTTPException, UploadFile, status, Path
from basic_memory.deps import (
ChatGPTImporterV2ExternalDep,
ClaudeConversationsImporterV2ExternalDep,
ClaudeProjectsImporterV2ExternalDep,
MemoryJsonImporterV2ExternalDep,
)
from basic_memory.importers import Importer
from basic_memory.schemas.importer import (
ChatImportResult,
EntityImportResult,
ProjectImportResult,
)
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/import", tags=["import-v2"])
@router.post("/chatgpt", response_model=ChatImportResult)
async def import_chatgpt(
importer: ChatGPTImporterV2ExternalDep,
file: UploadFile,
project_id: str = Path(..., description="Project external UUID"),
folder: str = Form("conversations"),
) -> ChatImportResult:
"""Import conversations from ChatGPT JSON export.
Args:
project_id: Project external UUID from URL path
file: The ChatGPT conversations.json file.
folder: The folder to place the files in.
importer: ChatGPT importer instance.
Returns:
ChatImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
logger.info(f"V2 Importing ChatGPT conversations for project {project_id}")
return await import_file(importer, file, folder)
@router.post("/claude/conversations", response_model=ChatImportResult)
async def import_claude_conversations(
importer: ClaudeConversationsImporterV2ExternalDep,
file: UploadFile,
project_id: str = Path(..., description="Project external UUID"),
folder: str = Form("conversations"),
) -> ChatImportResult:
"""Import conversations from Claude conversations.json export.
Args:
project_id: Project external UUID from URL path
file: The Claude conversations.json file.
folder: The folder to place the files in.
importer: Claude conversations importer instance.
Returns:
ChatImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
logger.info(f"V2 Importing Claude conversations for project {project_id}")
return await import_file(importer, file, folder)
@router.post("/claude/projects", response_model=ProjectImportResult)
async def import_claude_projects(
importer: ClaudeProjectsImporterV2ExternalDep,
file: UploadFile,
project_id: str = Path(..., description="Project external UUID"),
folder: str = Form("projects"),
) -> ProjectImportResult:
"""Import projects from Claude projects.json export.
Args:
project_id: Project external UUID from URL path
file: The Claude projects.json file.
folder: The base folder to place the files in.
importer: Claude projects importer instance.
Returns:
ProjectImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
logger.info(f"V2 Importing Claude projects for project {project_id}")
return await import_file(importer, file, folder)
@router.post("/memory-json", response_model=EntityImportResult)
async def import_memory_json(
importer: MemoryJsonImporterV2ExternalDep,
file: UploadFile,
project_id: str = Path(..., description="Project external UUID"),
folder: str = Form("conversations"),
) -> EntityImportResult:
"""Import entities and relations from a memory.json file.
Args:
project_id: Project external UUID from URL path
file: The memory.json file.
folder: Optional destination folder within the project.
importer: Memory JSON importer instance.
Returns:
EntityImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
logger.info(f"V2 Importing memory.json for project {project_id}")
try:
file_data = []
file_bytes = await file.read()
file_str = file_bytes.decode("utf-8")
for line in file_str.splitlines():
json_data = json.loads(line)
file_data.append(json_data)
result = await importer.import_data(file_data, folder)
if not result.success: # pragma: no cover
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=result.error_message or "Import failed",
)
except Exception as e:
logger.exception("V2 Import failed")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Import failed: {str(e)}",
)
return result
async def import_file(importer: Importer, file: UploadFile, destination_folder: str):
"""Helper function to import a file using an importer instance.
Args:
importer: The importer instance to use
file: The file to import
destination_folder: Destination folder for imported content
Returns:
Import result from the importer
Raises:
HTTPException: If import fails
"""
try:
# Process file
json_data = json.load(file.file)
result = await importer.import_data(json_data, destination_folder)
if not result.success: # pragma: no cover
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=result.error_message or "Import failed",
)
return result
except Exception as e:
logger.exception("V2 Import failed")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Import failed: {str(e)}",
)
@@ -0,0 +1,427 @@
"""V2 Knowledge Router - External ID-based entity operations.
This router provides external_id (UUID) based CRUD operations for entities,
using stable string UUIDs that won't change with file moves or database migrations.
Key improvements:
- Stable external UUIDs that won't change with file moves or renames
- Better API ergonomics with consistent string identifiers
- Direct database lookups via unique indexed column
- Simplified caching strategies
"""
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Response, Path
from loguru import logger
from basic_memory.deps import (
EntityServiceV2ExternalDep,
SearchServiceV2ExternalDep,
LinkResolverV2ExternalDep,
ProjectConfigV2ExternalDep,
AppConfigDep,
SyncServiceV2ExternalDep,
EntityRepositoryV2ExternalDep,
ProjectExternalIdPathDep,
)
from basic_memory.schemas import DeleteEntitiesResponse
from basic_memory.schemas.base import Entity
from basic_memory.schemas.request import EditEntityRequest
from basic_memory.schemas.v2 import (
EntityResolveRequest,
EntityResolveResponse,
EntityResponseV2,
MoveEntityRequestV2,
)
router = APIRouter(prefix="/knowledge", tags=["knowledge-v2"])
async def resolve_relations_background(sync_service, entity_id: int, entity_permalink: str) -> None:
"""Background task to resolve relations for a specific entity.
This runs asynchronously after the API response is sent, preventing
long delays when creating entities with many relations.
"""
try: # pragma: no cover
# Only resolve relations for the newly created entity
await sync_service.resolve_relations(entity_id=entity_id) # pragma: no cover
logger.debug( # pragma: no cover
f"Background: Resolved relations for entity {entity_permalink} (id={entity_id})"
)
except Exception as e: # pragma: no cover
# Log but don't fail - this is a background task
logger.warning( # pragma: no cover
f"Background: Failed to resolve relations for entity {entity_permalink}: {e}"
)
## Resolution endpoint
@router.post("/resolve", response_model=EntityResolveResponse)
async def resolve_identifier(
project_id: ProjectExternalIdPathDep,
data: EntityResolveRequest,
link_resolver: LinkResolverV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
) -> EntityResolveResponse:
"""Resolve a string identifier (external_id, permalink, title, or path) to entity info.
This endpoint provides a bridge between v1-style identifiers and v2 external_ids.
Use this to convert existing references to the new UUID-based format.
Args:
data: Request containing the identifier to resolve
Returns:
Entity external_id and metadata about how it was resolved
Raises:
HTTPException: 404 if identifier cannot be resolved
Example:
POST /v2/{project_id}/knowledge/resolve
{"identifier": "specs/search"}
Returns:
{
"external_id": "550e8400-e29b-41d4-a716-446655440000",
"entity_id": 123,
"permalink": "specs/search",
"file_path": "specs/search.md",
"title": "Search Specification",
"resolution_method": "permalink"
}
"""
logger.info(f"API v2 request: resolve_identifier for '{data.identifier}'")
# Try to resolve by external_id first
entity = await entity_repository.get_by_external_id(data.identifier)
resolution_method = "external_id" if entity else "search"
# If not found by external_id, try other resolution methods
if not entity:
entity = await link_resolver.resolve_link(data.identifier)
if entity:
# Determine resolution method
if entity.permalink == data.identifier:
resolution_method = "permalink"
elif entity.title == data.identifier:
resolution_method = "title"
elif entity.file_path == data.identifier:
resolution_method = "path"
else:
resolution_method = "search"
if not entity:
raise HTTPException(status_code=404, detail=f"Entity not found: '{data.identifier}'")
result = EntityResolveResponse(
external_id=entity.external_id,
entity_id=entity.id,
permalink=entity.permalink,
file_path=entity.file_path,
title=entity.title,
resolution_method=resolution_method,
)
logger.info(
f"API v2 response: resolved '{data.identifier}' to external_id={result.external_id} via {resolution_method}"
)
return result
## Read endpoints
@router.get("/entities/{entity_id}", response_model=EntityResponseV2)
async def get_entity_by_id(
project_id: ProjectExternalIdPathDep,
entity_repository: EntityRepositoryV2ExternalDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
) -> EntityResponseV2:
"""Get an entity by its external ID (UUID).
This is the primary entity retrieval method in v2, using stable UUID
identifiers that won't change with file moves.
Args:
entity_id: External ID (UUID string)
Returns:
Complete entity with observations and relations
Raises:
HTTPException: 404 if entity not found
"""
logger.info(f"API v2 request: get_entity_by_id entity_id={entity_id}")
entity = await entity_repository.get_by_external_id(entity_id)
if not entity:
raise HTTPException(
status_code=404, detail=f"Entity with external_id '{entity_id}' not found"
)
result = EntityResponseV2.model_validate(entity)
logger.info(f"API v2 response: external_id={entity_id}, title='{result.title}'")
return result
## Create endpoints
@router.post("/entities", response_model=EntityResponseV2)
async def create_entity(
project_id: ProjectExternalIdPathDep,
data: Entity,
background_tasks: BackgroundTasks,
entity_service: EntityServiceV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
) -> EntityResponseV2:
"""Create a new entity.
Args:
data: Entity data to create
Returns:
Created entity with generated external_id (UUID)
"""
logger.info(
"API v2 request", endpoint="create_entity", entity_type=data.entity_type, title=data.title
)
entity = await entity_service.create_entity(data)
# reindex
await search_service.index_entity(entity, background_tasks=background_tasks)
result = EntityResponseV2.model_validate(entity)
logger.info(
f"API v2 response: endpoint='create_entity' external_id={entity.external_id}, title={result.title}, permalink={result.permalink}, status_code=201"
)
return result
## Update endpoints
@router.put("/entities/{entity_id}", response_model=EntityResponseV2)
async def update_entity_by_id(
data: Entity,
response: Response,
background_tasks: BackgroundTasks,
project_id: ProjectExternalIdPathDep,
entity_service: EntityServiceV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
sync_service: SyncServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
) -> EntityResponseV2:
"""Update an entity by external ID.
If the entity doesn't exist, it will be created (upsert behavior).
Args:
entity_id: External ID (UUID string)
data: Updated entity data
Returns:
Updated entity
"""
logger.info(f"API v2 request: update_entity_by_id entity_id={entity_id}")
# Check if entity exists
existing = await entity_repository.get_by_external_id(entity_id)
created = existing is None
# Perform update or create
entity, _ = await entity_service.create_or_update_entity(data)
response.status_code = 201 if created else 200
# reindex
await search_service.index_entity(entity, background_tasks=background_tasks)
# Schedule relation resolution for new entities
if created:
background_tasks.add_task( # pragma: no cover
resolve_relations_background, sync_service, entity.id, entity.permalink or ""
)
result = EntityResponseV2.model_validate(entity)
logger.info(
f"API v2 response: external_id={entity_id}, created={created}, status_code={response.status_code}"
)
return result
@router.patch("/entities/{entity_id}", response_model=EntityResponseV2)
async def edit_entity_by_id(
data: EditEntityRequest,
background_tasks: BackgroundTasks,
project_id: ProjectExternalIdPathDep,
entity_service: EntityServiceV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
) -> EntityResponseV2:
"""Edit an existing entity by external ID using operations like append, prepend, etc.
Args:
entity_id: External ID (UUID string)
data: Edit operation details
Returns:
Updated entity
Raises:
HTTPException: 404 if entity not found, 400 if edit fails
"""
logger.info(
f"API v2 request: edit_entity_by_id entity_id={entity_id}, operation='{data.operation}'"
)
# Verify entity exists
entity = await entity_repository.get_by_external_id(entity_id)
if not entity: # pragma: no cover
raise HTTPException(
status_code=404, detail=f"Entity with external_id '{entity_id}' not found"
)
try:
# Edit using the entity's permalink or path
identifier = entity.permalink or entity.file_path
updated_entity = await entity_service.edit_entity(
identifier=identifier,
operation=data.operation,
content=data.content,
section=data.section,
find_text=data.find_text,
expected_replacements=data.expected_replacements,
)
# Reindex
await search_service.index_entity(updated_entity, background_tasks=background_tasks)
result = EntityResponseV2.model_validate(updated_entity)
logger.info(
f"API v2 response: external_id={entity_id}, operation='{data.operation}', status_code=200"
)
return result
except Exception as e:
logger.error(f"Error editing entity {entity_id}: {e}")
raise HTTPException(status_code=400, detail=str(e))
## Delete endpoints
@router.delete("/entities/{entity_id}", response_model=DeleteEntitiesResponse)
async def delete_entity_by_id(
background_tasks: BackgroundTasks,
project_id: ProjectExternalIdPathDep,
entity_service: EntityServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
search_service=Depends(lambda: None), # Optional for now
) -> DeleteEntitiesResponse:
"""Delete an entity by external ID.
Args:
entity_id: External ID (UUID string)
Returns:
Deletion status
Note: Returns deleted=False if entity doesn't exist (idempotent)
"""
logger.info(f"API v2 request: delete_entity_by_id entity_id={entity_id}")
entity = await entity_repository.get_by_external_id(entity_id)
if entity is None:
logger.info(f"API v2 response: external_id={entity_id} not found, deleted=False")
return DeleteEntitiesResponse(deleted=False)
# Delete the entity using internal ID
deleted = await entity_service.delete_entity(entity.id)
# Remove from search index if search service available
if search_service:
background_tasks.add_task(search_service.handle_delete, entity) # pragma: no cover
logger.info(f"API v2 response: external_id={entity_id}, deleted={deleted}")
return DeleteEntitiesResponse(deleted=deleted)
## Move endpoint
@router.put("/entities/{entity_id}/move", response_model=EntityResponseV2)
async def move_entity(
data: MoveEntityRequestV2,
background_tasks: BackgroundTasks,
project_id: ProjectExternalIdPathDep,
entity_service: EntityServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
project_config: ProjectConfigV2ExternalDep,
app_config: AppConfigDep,
search_service: SearchServiceV2ExternalDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
) -> EntityResponseV2:
"""Move an entity to a new file location.
V2 API uses external_id (UUID) in the URL path for stable references.
The external_id will remain stable after the move.
Args:
project_id: Project external ID from URL path
entity_id: Entity external ID from URL path (primary identifier)
data: Move request with destination path only
Returns:
Updated entity with new file path
"""
logger.info(
f"API v2 request: move_entity entity_id={entity_id}, destination='{data.destination_path}'"
)
try:
# First, get the entity by external_id to verify it exists
entity = await entity_repository.get_by_external_id(entity_id)
if not entity: # pragma: no cover
raise HTTPException(
status_code=404, detail=f"Entity with external_id '{entity_id}' not found"
)
# Move the entity using its current file path as identifier
moved_entity = await entity_service.move_entity(
identifier=entity.file_path, # Use file path for resolution
destination_path=data.destination_path,
project_config=project_config,
app_config=app_config,
)
# Reindex at new location
reindexed_entity = await entity_service.link_resolver.resolve_link(data.destination_path)
if reindexed_entity:
await search_service.index_entity(reindexed_entity, background_tasks=background_tasks)
result = EntityResponseV2.model_validate(moved_entity)
logger.info(
f"API v2 response: moved external_id={entity_id} to '{data.destination_path}'"
)
return result
except HTTPException: # pragma: no cover
raise # pragma: no cover
except Exception as e:
logger.error(f"Error moving entity: {e}")
raise HTTPException(status_code=400, detail=str(e))
@@ -0,0 +1,130 @@
"""V2 routes for memory:// URI operations.
This router uses external_id UUIDs for stable, API-friendly routing.
V1 uses string-based project names which are less efficient and less stable.
"""
from typing import Annotated, Optional
from fastapi import APIRouter, Query, Path
from loguru import logger
from basic_memory.deps import ContextServiceV2ExternalDep, EntityRepositoryV2ExternalDep
from basic_memory.schemas.base import TimeFrame, parse_timeframe
from basic_memory.schemas.memory import (
GraphContext,
normalize_memory_url,
)
from basic_memory.schemas.search import SearchItemType
from basic_memory.api.routers.utils import to_graph_context
# Note: No prefix here - it's added during registration as /v2/{project_id}/memory
router = APIRouter(tags=["memory"])
@router.get("/memory/recent", response_model=GraphContext)
async def recent(
context_service: ContextServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
type: Annotated[list[SearchItemType] | None, Query()] = None,
depth: int = 1,
timeframe: TimeFrame = "7d",
page: int = 1,
page_size: int = 10,
max_related: int = 10,
) -> GraphContext:
"""Get recent activity context for a project.
Args:
project_id: Project external UUID from URL path
context_service: Context service scoped to project
entity_repository: Entity repository scoped to project
type: Types of items to include (entities, relations, observations)
depth: How many levels of related entities to include
timeframe: Time window for recent activity (e.g., "7d", "1 week")
page: Page number for pagination
page_size: Number of items per page
max_related: Maximum related entities to include per item
Returns:
GraphContext with recent activity and related entities
"""
# return all types by default
types = (
[SearchItemType.ENTITY, SearchItemType.RELATION, SearchItemType.OBSERVATION]
if not type
else type
)
logger.debug(
f"V2 Getting recent context for project {project_id}: `{types}` depth: `{depth}` timeframe: `{timeframe}` page: `{page}` page_size: `{page_size}` max_related: `{max_related}`"
)
# Parse timeframe
since = parse_timeframe(timeframe)
limit = page_size
offset = (page - 1) * page_size
# Build context
context = await context_service.build_context(
types=types, depth=depth, since=since, limit=limit, offset=offset, max_related=max_related
)
recent_context = await to_graph_context(
context, entity_repository=entity_repository, page=page, page_size=page_size
)
logger.debug(f"V2 Recent context: {recent_context.model_dump_json()}")
return recent_context
# get_memory_context needs to be declared last so other paths can match
@router.get("/memory/{uri:path}", response_model=GraphContext)
async def get_memory_context(
context_service: ContextServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
uri: str,
project_id: str = Path(..., description="Project external UUID"),
depth: int = 1,
timeframe: Optional[TimeFrame] = None,
page: int = 1,
page_size: int = 10,
max_related: int = 10,
) -> GraphContext:
"""Get rich context from memory:// URI.
V2 supports both legacy path-based URIs and new ID-based URIs:
- Legacy: memory://path/to/note
- ID-based: memory://id/123 or memory://123
Args:
project_id: Project external UUID from URL path
context_service: Context service scoped to project
entity_repository: Entity repository scoped to project
uri: Memory URI path (e.g., "id/123", "123", or "path/to/note")
depth: How many levels of related entities to include
timeframe: Optional time window for filtering related content
page: Page number for pagination
page_size: Number of items per page
max_related: Maximum related entities to include
Returns:
GraphContext with the entity and its related context
"""
logger.debug(
f"V2 Getting context for project {project_id}, URI: `{uri}` depth: `{depth}` timeframe: `{timeframe}` page: `{page}` page_size: `{page_size}` max_related: `{max_related}`"
)
memory_url = normalize_memory_url(uri)
# Parse timeframe
since = parse_timeframe(timeframe) if timeframe else None
limit = page_size
offset = (page - 1) * page_size
# Build context
context = await context_service.build_context(
memory_url, depth=depth, since=since, limit=limit, offset=offset, max_related=max_related
)
return await to_graph_context(
context, entity_repository=entity_repository, page=page, page_size=page_size
)
@@ -0,0 +1,359 @@
"""V2 Project Router - External ID-based project management operations.
This router provides external_id (UUID) based CRUD operations for projects,
using stable string UUIDs that never change (unlike integer IDs or names).
Key improvements:
- Stable external UUIDs that won't change with renames or database migrations
- Better API ergonomics with consistent string identifiers
- Direct database lookups via unique indexed column
- Consistent with v2 entity operations
"""
import os
from typing import Optional
from fastapi import APIRouter, HTTPException, Body, Query, Path
from loguru import logger
from basic_memory.deps import (
ProjectServiceDep,
ProjectRepositoryDep,
)
from basic_memory.schemas.project_info import (
ProjectItem,
ProjectStatusResponse,
)
from basic_memory.schemas.v2 import ProjectResolveRequest, ProjectResolveResponse
from basic_memory.utils import normalize_project_path, generate_permalink
router = APIRouter(prefix="/projects", tags=["project_management-v2"])
@router.post("/resolve", response_model=ProjectResolveResponse)
async def resolve_project_identifier(
data: ProjectResolveRequest,
project_repository: ProjectRepositoryDep,
) -> ProjectResolveResponse:
"""Resolve a project identifier (name, permalink, or external_id) to project info.
This endpoint provides efficient lookup of projects by various identifiers
without needing to fetch the entire project list. Supports:
- External ID (UUID string) - preferred stable identifier
- Permalink
- Case-insensitive name matching
Args:
data: Request containing the identifier to resolve
Returns:
Project information including the external_id (UUID)
Raises:
HTTPException: 404 if project not found
Example:
POST /v2/projects/resolve
{"identifier": "my-project"}
Returns:
{
"external_id": "550e8400-e29b-41d4-a716-446655440000",
"project_id": 1,
"name": "my-project",
"permalink": "my-project",
"path": "/path/to/project",
"is_active": true,
"is_default": false,
"resolution_method": "name"
}
"""
logger.info(f"API v2 request: resolve_project_identifier for '{data.identifier}'")
# Generate permalink for comparison
identifier_permalink = generate_permalink(data.identifier)
resolution_method = "name"
project = None
# Try external_id first (UUID format)
project = await project_repository.get_by_external_id(data.identifier)
if project:
resolution_method = "external_id"
# If not found by external_id, try by permalink (exact match)
if not project:
project = await project_repository.get_by_permalink(identifier_permalink)
if project:
resolution_method = "permalink"
# If not found by permalink, try case-insensitive name search
if not project:
project = await project_repository.get_by_name_case_insensitive(data.identifier)
if project:
resolution_method = "name" # pragma: no cover
if not project:
raise HTTPException(status_code=404, detail=f"Project not found: '{data.identifier}'")
return ProjectResolveResponse(
external_id=project.external_id,
project_id=project.id,
name=project.name,
permalink=generate_permalink(project.name),
path=normalize_project_path(project.path),
is_active=project.is_active if hasattr(project, "is_active") else True,
is_default=project.is_default or False,
resolution_method=resolution_method,
)
@router.get("/{project_id}", response_model=ProjectItem)
async def get_project_by_id(
project_repository: ProjectRepositoryDep,
project_id: str = Path(..., description="Project external ID (UUID)"),
) -> ProjectItem:
"""Get project by its external ID (UUID).
This is the primary project retrieval method in v2, using stable UUID
identifiers that won't change with project renames.
Args:
project_id: External ID (UUID string)
Returns:
Project information including external_id
Raises:
HTTPException: 404 if project not found
Example:
GET /v2/projects/550e8400-e29b-41d4-a716-446655440000
"""
logger.info(f"API v2 request: get_project_by_id for project_id={project_id}")
project = await project_repository.get_by_external_id(project_id)
if not project:
raise HTTPException(
status_code=404, detail=f"Project with external_id '{project_id}' not found"
)
return ProjectItem(
id=project.id,
external_id=project.external_id,
name=project.name,
path=normalize_project_path(project.path),
is_default=project.is_default or False,
)
@router.patch("/{project_id}", response_model=ProjectStatusResponse)
async def update_project_by_id(
project_service: ProjectServiceDep,
project_repository: ProjectRepositoryDep,
project_id: str = Path(..., description="Project external ID (UUID)"),
path: Optional[str] = Body(None, description="New absolute path for the project"),
is_active: Optional[bool] = Body(None, description="Status of the project (active/inactive)"),
) -> ProjectStatusResponse:
"""Update a project's information by external ID.
Args:
project_id: External ID (UUID string)
path: Optional new absolute path for the project
is_active: Optional status update for the project
Returns:
Response confirming the project was updated
Raises:
HTTPException: 400 if validation fails, 404 if project not found
Example:
PATCH /v2/projects/550e8400-e29b-41d4-a716-446655440000
{"path": "/new/path"}
"""
logger.info(f"API v2 request: update_project_by_id for project_id={project_id}")
try:
# Validate that path is absolute if provided
if path and not os.path.isabs(path):
raise HTTPException(status_code=400, detail="Path must be absolute")
# Get original project info for the response
old_project = await project_repository.get_by_external_id(project_id)
if not old_project:
raise HTTPException(
status_code=404, detail=f"Project with external_id '{project_id}' not found"
)
old_project_info = ProjectItem(
id=old_project.id,
external_id=old_project.external_id,
name=old_project.name,
path=old_project.path,
is_default=old_project.is_default or False,
)
# Update using project name (service layer still uses names internally)
if path:
await project_service.move_project(old_project.name, path)
elif is_active is not None:
await project_service.update_project(old_project.name, is_active=is_active)
# Get updated project info (use the same external_id)
updated_project = await project_repository.get_by_external_id(project_id)
if not updated_project: # pragma: no cover
raise HTTPException(
status_code=404,
detail=f"Project with external_id '{project_id}' not found after update",
)
return ProjectStatusResponse(
message=f"Project '{updated_project.name}' updated successfully",
status="success",
default=old_project.is_default or False,
old_project=old_project_info,
new_project=ProjectItem(
id=updated_project.id,
external_id=updated_project.external_id,
name=updated_project.name,
path=updated_project.path,
is_default=updated_project.is_default or False,
),
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e)) # pragma: no cover
@router.delete("/{project_id}", response_model=ProjectStatusResponse)
async def delete_project_by_id(
project_service: ProjectServiceDep,
project_repository: ProjectRepositoryDep,
project_id: str = Path(..., description="Project external ID (UUID)"),
delete_notes: bool = Query(
False, description="If True, delete project directory from filesystem"
),
) -> ProjectStatusResponse:
"""Delete a project by external ID.
Args:
project_id: External ID (UUID string)
delete_notes: If True, delete the project directory from the filesystem
Returns:
Response confirming the project was deleted
Raises:
HTTPException: 400 if trying to delete default project, 404 if not found
Example:
DELETE /v2/projects/550e8400-e29b-41d4-a716-446655440000?delete_notes=false
"""
logger.info(
f"API v2 request: delete_project_by_id for project_id={project_id}, delete_notes={delete_notes}"
)
try:
old_project = await project_repository.get_by_external_id(project_id)
if not old_project:
raise HTTPException(
status_code=404, detail=f"Project with external_id '{project_id}' not found"
)
# Check if trying to delete the default project
# Use is_default from database, not ConfigManager (which doesn't work in cloud mode)
if old_project.is_default:
available_projects = await project_service.list_projects()
other_projects = [
p.name for p in available_projects if p.external_id != project_id
]
detail = f"Cannot delete default project '{old_project.name}'. "
if other_projects:
detail += ( # pragma: no cover
f"Set another project as default first. Available: {', '.join(other_projects)}"
)
else:
detail += "This is the only project in your configuration." # pragma: no cover
raise HTTPException(status_code=400, detail=detail)
# Delete using project name (service layer still uses names internally)
await project_service.remove_project(old_project.name, delete_notes=delete_notes)
return ProjectStatusResponse(
message=f"Project '{old_project.name}' removed successfully",
status="success",
default=False,
old_project=ProjectItem(
id=old_project.id,
external_id=old_project.external_id,
name=old_project.name,
path=old_project.path,
is_default=old_project.is_default or False,
),
new_project=None,
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e)) # pragma: no cover
@router.put("/{project_id}/default", response_model=ProjectStatusResponse)
async def set_default_project_by_id(
project_service: ProjectServiceDep,
project_repository: ProjectRepositoryDep,
project_id: str = Path(..., description="Project external ID (UUID)"),
) -> ProjectStatusResponse:
"""Set a project as the default project by external ID.
Args:
project_id: External ID (UUID string) to set as default
Returns:
Response confirming the project was set as default
Raises:
HTTPException: 404 if project not found
Example:
PUT /v2/projects/550e8400-e29b-41d4-a716-446655440000/default
"""
logger.info(f"API v2 request: set_default_project_by_id for project_id={project_id}")
try:
# Get the old default project from database
default_project = await project_repository.get_default_project()
if not default_project:
raise HTTPException( # pragma: no cover
status_code=404, detail="No default project is currently set"
)
# Get the new default project by external_id
new_default_project = await project_repository.get_by_external_id(project_id)
if not new_default_project:
raise HTTPException(
status_code=404, detail=f"Project with external_id '{project_id}' not found"
)
# Set as default using project name (service layer still uses names internally)
await project_service.set_default_project(new_default_project.name)
return ProjectStatusResponse(
message=f"Project '{new_default_project.name}' set as default successfully",
status="success",
default=True,
old_project=ProjectItem(
id=default_project.id,
external_id=default_project.external_id,
name=default_project.name,
path=default_project.path,
is_default=False,
),
new_project=ProjectItem(
id=new_default_project.id,
external_id=new_default_project.external_id,
name=new_default_project.name,
path=new_default_project.path,
is_default=True,
),
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e)) # pragma: no cover
@@ -0,0 +1,269 @@
"""V2 Prompt Router - ID-based prompt generation operations.
This router uses v2 dependencies for consistent project handling with external_id UUIDs.
Prompt endpoints are action-based (not resource-based), so they don't
have entity IDs in URLs - they generate formatted prompts from queries.
"""
from datetime import datetime, timezone
from fastapi import APIRouter, HTTPException, status, Path
from loguru import logger
from basic_memory.api.routers.utils import to_graph_context, to_search_results
from basic_memory.api.template_loader import template_loader
from basic_memory.schemas.base import parse_timeframe
from basic_memory.deps import (
ContextServiceV2ExternalDep,
EntityRepositoryV2ExternalDep,
SearchServiceV2ExternalDep,
EntityServiceV2ExternalDep,
)
from basic_memory.schemas.prompt import (
ContinueConversationRequest,
SearchPromptRequest,
PromptResponse,
PromptMetadata,
)
from basic_memory.schemas.search import SearchItemType, SearchQuery
router = APIRouter(prefix="/prompt", tags=["prompt-v2"])
@router.post("/continue-conversation", response_model=PromptResponse)
async def continue_conversation(
search_service: SearchServiceV2ExternalDep,
entity_service: EntityServiceV2ExternalDep,
context_service: ContextServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
request: ContinueConversationRequest,
project_id: str = Path(..., description="Project external UUID"),
) -> PromptResponse:
"""Generate a prompt for continuing a conversation.
This endpoint takes a topic and/or timeframe and generates a prompt with
relevant context from the knowledge base.
Args:
project_id: Project external UUID from URL path
request: The request parameters
Returns:
Formatted continuation prompt with context
"""
logger.info(
f"V2 Generating continue conversation prompt for project {project_id}, "
f"topic: {request.topic}, timeframe: {request.timeframe}"
)
since = parse_timeframe(request.timeframe) if request.timeframe else None
# Initialize search results
search_results = []
# Get data needed for template
if request.topic:
query = SearchQuery(text=request.topic, after_date=request.timeframe)
results = await search_service.search(query, limit=request.search_items_limit)
search_results = await to_search_results(entity_service, results)
# Build context from results
all_hierarchical_results = []
for result in search_results:
if hasattr(result, "permalink") and result.permalink:
# Get hierarchical context using the new dataclass-based approach
context_result = await context_service.build_context(
result.permalink,
depth=request.depth,
since=since,
max_related=request.related_items_limit,
include_observations=True, # Include observations for entities
)
# Process results into the schema format
graph_context = await to_graph_context(
context_result, entity_repository=entity_repository
)
# Add results to our collection (limit to top results for each permalink)
if graph_context.results:
all_hierarchical_results.extend(graph_context.results[:3])
# Limit to a reasonable number of total results
all_hierarchical_results = all_hierarchical_results[:10]
template_context = {
"topic": request.topic,
"timeframe": request.timeframe,
"hierarchical_results": all_hierarchical_results,
"has_results": len(all_hierarchical_results) > 0,
}
else:
# If no topic, get recent activity
context_result = await context_service.build_context(
types=[SearchItemType.ENTITY],
depth=request.depth,
since=since,
max_related=request.related_items_limit,
include_observations=True,
)
recent_context = await to_graph_context(context_result, entity_repository=entity_repository)
hierarchical_results = recent_context.results[:5] # Limit to top 5 recent items
template_context = {
"topic": f"Recent Activity from ({request.timeframe})",
"timeframe": request.timeframe,
"hierarchical_results": hierarchical_results,
"has_results": len(hierarchical_results) > 0,
}
try:
# Render template
rendered_prompt = await template_loader.render(
"prompts/continue_conversation.hbs", template_context
)
# Calculate metadata
# Count items of different types
observation_count = 0
relation_count = 0
entity_count = 0
# Get the hierarchical results from the template context
hierarchical_results_for_count = template_context.get("hierarchical_results", [])
# For topic-based search
if request.topic:
for item in hierarchical_results_for_count:
if hasattr(item, "observations"):
observation_count += len(item.observations) if item.observations else 0
if hasattr(item, "related_results"):
for related in item.related_results or []:
if hasattr(related, "type"):
if related.type == "relation":
relation_count += 1
elif related.type == "entity": # pragma: no cover
entity_count += 1 # pragma: no cover
# For recent activity
else:
for item in hierarchical_results_for_count:
if hasattr(item, "observations"):
observation_count += len(item.observations) if item.observations else 0
if hasattr(item, "related_results"):
for related in item.related_results or []:
if hasattr(related, "type"):
if related.type == "relation":
relation_count += 1
elif related.type == "entity": # pragma: no cover
entity_count += 1 # pragma: no cover
# Build metadata
metadata = {
"query": request.topic,
"timeframe": request.timeframe,
"search_count": len(search_results)
if request.topic
else 0, # Original search results count
"context_count": len(hierarchical_results_for_count),
"observation_count": observation_count,
"relation_count": relation_count,
"total_items": (
len(hierarchical_results_for_count)
+ observation_count
+ relation_count
+ entity_count
),
"search_limit": request.search_items_limit,
"context_depth": request.depth,
"related_limit": request.related_items_limit,
"generated_at": datetime.now(timezone.utc).isoformat(),
}
prompt_metadata = PromptMetadata(**metadata)
return PromptResponse(
prompt=rendered_prompt, context=template_context, metadata=prompt_metadata
)
except Exception as e:
logger.error(f"Error rendering continue conversation template: {e}")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Error rendering prompt template: {str(e)}",
)
@router.post("/search", response_model=PromptResponse)
async def search_prompt(
search_service: SearchServiceV2ExternalDep,
entity_service: EntityServiceV2ExternalDep,
request: SearchPromptRequest,
project_id: str = Path(..., description="Project external UUID"),
page: int = 1,
page_size: int = 10,
) -> PromptResponse:
"""Generate a prompt for search results.
This endpoint takes a search query and formats the results into a helpful
prompt with context and suggestions.
Args:
project_id: Project external UUID from URL path
request: The search parameters
page: The page number for pagination
page_size: The number of results per page, defaults to 10
Returns:
Formatted search results prompt with context
"""
logger.info(
f"V2 Generating search prompt for project {project_id}, "
f"query: {request.query}, timeframe: {request.timeframe}"
)
limit = page_size
offset = (page - 1) * page_size
query = SearchQuery(text=request.query, after_date=request.timeframe)
results = await search_service.search(query, limit=limit, offset=offset)
search_results = await to_search_results(entity_service, results)
template_context = {
"query": request.query,
"timeframe": request.timeframe,
"results": search_results,
"has_results": len(search_results) > 0,
"result_count": len(search_results),
}
try:
# Render template
rendered_prompt = await template_loader.render("prompts/search.hbs", template_context)
# Build metadata
metadata = {
"query": request.query,
"timeframe": request.timeframe,
"search_count": len(search_results),
"context_count": len(search_results),
"observation_count": 0, # Search results don't include observations
"relation_count": 0, # Search results don't include relations
"total_items": len(search_results),
"search_limit": limit,
"context_depth": 0, # No context depth for basic search
"related_limit": 0, # No related items for basic search
"generated_at": datetime.now(timezone.utc).isoformat(),
}
prompt_metadata = PromptMetadata(**metadata)
return PromptResponse(
prompt=rendered_prompt, context=template_context, metadata=prompt_metadata
)
except Exception as e:
logger.error(f"Error rendering search template: {e}")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Error rendering prompt template: {str(e)}",
)
@@ -0,0 +1,286 @@
"""V2 Resource Router - ID-based resource content operations.
This router uses entity external_ids (UUIDs) for all operations, with file paths
in request bodies when needed. This is consistent with v2's external_id-first design.
Key differences from v1:
- Uses UUID external_ids in URL paths instead of integer IDs or file paths
- File paths are in request bodies for create/update operations
- More RESTful: POST for create, PUT for update, GET for read
"""
from pathlib import Path as PathLib
from fastapi import APIRouter, HTTPException, Response, Path
from loguru import logger
from basic_memory.deps import (
ProjectConfigV2ExternalDep,
FileServiceV2ExternalDep,
EntityRepositoryV2ExternalDep,
SearchServiceV2ExternalDep,
)
from basic_memory.models.knowledge import Entity as EntityModel
from basic_memory.schemas.v2.resource import (
CreateResourceRequest,
UpdateResourceRequest,
ResourceResponse,
)
from basic_memory.utils import validate_project_path
router = APIRouter(prefix="/resource", tags=["resources-v2"])
@router.get("/{entity_id}")
async def get_resource_content(
config: ProjectConfigV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
file_service: FileServiceV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
entity_id: str = Path(..., description="Entity external UUID"),
) -> Response:
"""Get raw resource content by entity external_id.
Args:
project_id: Project external UUID from URL path
entity_id: Entity external UUID
config: Project configuration
entity_repository: Entity repository for fetching entity data
file_service: File service for reading file content
Returns:
Response with entity content
Raises:
HTTPException: 404 if entity or file not found
"""
logger.debug(f"V2 Getting content for project {project_id}, entity_id: {entity_id}")
# Get entity by external_id
entity = await entity_repository.get_by_external_id(entity_id)
if not entity:
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
# Validate entity file path to prevent path traversal
project_path = PathLib(config.home)
if not validate_project_path(entity.file_path, project_path):
logger.error( # pragma: no cover
f"Invalid file path in entity {entity.id}: {entity.file_path}"
)
raise HTTPException( # pragma: no cover
status_code=500,
detail="Entity contains invalid file path",
)
# Check file exists via file_service (for cloud compatibility)
if not await file_service.exists(entity.file_path):
raise HTTPException( # pragma: no cover
status_code=404,
detail=f"File not found: {entity.file_path}",
)
# Read content via file_service as bytes (works with both local and S3)
content = await file_service.read_file_bytes(entity.file_path)
content_type = file_service.content_type(entity.file_path)
return Response(content=content, media_type=content_type)
@router.post("", response_model=ResourceResponse)
async def create_resource(
data: CreateResourceRequest,
config: ProjectConfigV2ExternalDep,
file_service: FileServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
) -> ResourceResponse:
"""Create a new resource file.
Args:
project_id: Project external UUID from URL path
data: Create resource request with file_path and content
config: Project configuration
file_service: File service for writing files
entity_repository: Entity repository for creating entities
search_service: Search service for indexing
Returns:
ResourceResponse with file information including entity_id and external_id
Raises:
HTTPException: 400 for invalid file paths, 409 if file already exists
"""
try:
# Validate path to prevent path traversal attacks
project_path = PathLib(config.home)
if not validate_project_path(data.file_path, project_path):
logger.warning(
f"Invalid file path attempted: {data.file_path} in project {config.name}"
)
raise HTTPException(
status_code=400,
detail=f"Invalid file path: {data.file_path}. "
"Path must be relative and stay within project boundaries.",
)
# Check if entity already exists
existing_entity = await entity_repository.get_by_file_path(data.file_path)
if existing_entity:
raise HTTPException(
status_code=409,
detail=f"Resource already exists at {data.file_path} with entity_id {existing_entity.external_id}. "
f"Use PUT /resource/{existing_entity.external_id} to update it.",
)
# Cloud compatibility: avoid assuming a local filesystem path.
# Delegate directory creation + writes to FileService (local or S3).
await file_service.ensure_directory(PathLib(data.file_path).parent)
checksum = await file_service.write_file(data.file_path, data.content)
# Get file info
file_metadata = await file_service.get_file_metadata(data.file_path)
# Determine file details
file_name = PathLib(data.file_path).name
content_type = file_service.content_type(data.file_path)
entity_type = "canvas" if data.file_path.endswith(".canvas") else "file"
# Create a new entity model
entity = EntityModel(
title=file_name,
entity_type=entity_type,
content_type=content_type,
file_path=data.file_path,
checksum=checksum,
created_at=file_metadata.created_at,
updated_at=file_metadata.modified_at,
)
entity = await entity_repository.add(entity)
# Index the file for search
await search_service.index_entity(entity) # pyright: ignore
# Return success response
return ResourceResponse(
entity_id=entity.id,
external_id=entity.external_id,
file_path=data.file_path,
checksum=checksum,
size=file_metadata.size,
created_at=file_metadata.created_at.timestamp(),
modified_at=file_metadata.modified_at.timestamp(),
)
except HTTPException:
# Re-raise HTTP exceptions without wrapping
raise
except Exception as e: # pragma: no cover
logger.error(f"Error creating resource {data.file_path}: {e}")
raise HTTPException(status_code=500, detail=f"Failed to create resource: {str(e)}")
@router.put("/{entity_id}", response_model=ResourceResponse)
async def update_resource(
data: UpdateResourceRequest,
config: ProjectConfigV2ExternalDep,
file_service: FileServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
entity_id: str = Path(..., description="Entity external UUID"),
) -> ResourceResponse:
"""Update an existing resource by entity external_id.
Can update content and optionally move the file to a new path.
Args:
project_id: Project external UUID from URL path
entity_id: Entity external UUID of the resource to update
data: Update resource request with content and optional new file_path
config: Project configuration
file_service: File service for writing files
entity_repository: Entity repository for updating entities
search_service: Search service for indexing
Returns:
ResourceResponse with updated file information
Raises:
HTTPException: 404 if entity not found, 400 for invalid paths
"""
try:
# Get existing entity by external_id
entity = await entity_repository.get_by_external_id(entity_id)
if not entity:
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
# Determine target file path
target_file_path = data.file_path if data.file_path else entity.file_path
# Validate path to prevent path traversal attacks
project_path = PathLib(config.home)
if not validate_project_path(target_file_path, project_path):
logger.warning(
f"Invalid file path attempted: {target_file_path} in project {config.name}"
)
raise HTTPException(
status_code=400,
detail=f"Invalid file path: {target_file_path}. "
"Path must be relative and stay within project boundaries.",
)
# If moving file, handle the move
if data.file_path and data.file_path != entity.file_path:
# Ensure new parent directory exists (no-op for S3)
await file_service.ensure_directory(PathLib(target_file_path).parent)
# If old file exists, remove it via file_service (for cloud compatibility)
if await file_service.exists(entity.file_path):
await file_service.delete_file(entity.file_path)
else:
# Ensure directory exists for in-place update
await file_service.ensure_directory(PathLib(target_file_path).parent)
# Write content to target file
checksum = await file_service.write_file(target_file_path, data.content)
# Get file info
file_metadata = await file_service.get_file_metadata(target_file_path)
# Determine file details
file_name = PathLib(target_file_path).name
content_type = file_service.content_type(target_file_path)
entity_type = "canvas" if target_file_path.endswith(".canvas") else "file"
# Update entity using internal ID
updated_entity = await entity_repository.update(
entity.id,
{
"title": file_name,
"entity_type": entity_type,
"content_type": content_type,
"file_path": target_file_path,
"checksum": checksum,
"updated_at": file_metadata.modified_at,
},
)
# Index the updated file for search
await search_service.index_entity(updated_entity) # pyright: ignore
# Return success response
return ResourceResponse(
entity_id=entity.id,
external_id=entity.external_id,
file_path=target_file_path,
checksum=checksum,
size=file_metadata.size,
created_at=file_metadata.created_at.timestamp(),
modified_at=file_metadata.modified_at.timestamp(),
)
except HTTPException:
# Re-raise HTTP exceptions without wrapping
raise
except Exception as e: # pragma: no cover
logger.error(f"Error updating resource {entity_id}: {e}")
raise HTTPException(status_code=500, detail=f"Failed to update resource: {str(e)}")
@@ -0,0 +1,73 @@
"""V2 router for search operations.
This router uses external_id UUIDs for stable, API-friendly routing.
V1 uses string-based project names which are less efficient and less stable.
"""
from fastapi import APIRouter, BackgroundTasks, Path
from basic_memory.api.routers.utils import to_search_results
from basic_memory.schemas.search import SearchQuery, SearchResponse
from basic_memory.deps import SearchServiceV2ExternalDep, EntityServiceV2ExternalDep
# Note: No prefix here - it's added during registration as /v2/{project_id}/search
router = APIRouter(tags=["search"])
@router.post("/search/", response_model=SearchResponse)
async def search(
query: SearchQuery,
search_service: SearchServiceV2ExternalDep,
entity_service: EntityServiceV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
page: int = 1,
page_size: int = 10,
):
"""Search across all knowledge and documents in a project.
V2 uses external_id UUIDs for stable API references.
Args:
project_id: Project external UUID from URL path
query: Search query parameters (text, filters, etc.)
search_service: Search service scoped to project
entity_service: Entity service scoped to project
page: Page number for pagination
page_size: Number of results per page
Returns:
SearchResponse with paginated search results
"""
limit = page_size
offset = (page - 1) * page_size
results = await search_service.search(query, limit=limit, offset=offset)
search_results = await to_search_results(entity_service, results)
return SearchResponse(
results=search_results,
current_page=page,
page_size=page_size,
)
@router.post("/search/reindex")
async def reindex(
background_tasks: BackgroundTasks,
search_service: SearchServiceV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
):
"""Recreate and populate the search index for a project.
This is a background operation that rebuilds the search index
from scratch. Useful after bulk updates or if the index becomes
corrupted.
Args:
project_id: Project external UUID from URL path
background_tasks: FastAPI background tasks handler
search_service: Search service scoped to project
Returns:
Status message indicating reindex has been initiated
"""
await search_service.reindex_all(background_tasks=background_tasks)
return {"status": "ok", "message": "Reindex initiated"}
+43 -7
View File
@@ -1,8 +1,22 @@
from typing import Optional
# Suppress Logfire "not configured" warning - we only use Logfire in cloud/server contexts
import os
import typer
os.environ.setdefault("LOGFIRE_IGNORE_NO_CONFIG", "1")
from basic_memory.config import ConfigManager
# Remove loguru's default handler IMMEDIATELY, before any other imports.
# This prevents DEBUG logs from appearing on stdout during module-level
# initialization (e.g., template_loader.TemplateLoader() logs at DEBUG level).
from loguru import logger
logger.remove()
from typing import Optional # noqa: E402
import typer # noqa: E402
from basic_memory.cli.container import CliContainer, set_container # noqa: E402
from basic_memory.config import init_cli_logging # noqa: E402
from basic_memory.telemetry import show_notice_if_needed, track_app_started # noqa: E402
def version_callback(value: bool) -> None:
@@ -31,12 +45,34 @@ def app_callback(
) -> None:
"""Basic Memory - Local-first personal knowledge management."""
# Run initialization for every command unless --version was specified
if not version and ctx.invoked_subcommand is not None:
# Initialize logging for CLI (file only, no stdout)
init_cli_logging()
# --- Composition Root ---
# Create container and read config (single point of config access)
container = CliContainer.create()
set_container(container)
# Show telemetry notice and track CLI startup
# Skip for 'mcp' command - it handles its own telemetry in lifespan
# Skip for 'telemetry' command - avoid issues when user is managing telemetry
if ctx.invoked_subcommand not in {"mcp", "telemetry"}:
show_notice_if_needed()
track_app_started("cli")
# Run initialization for commands that don't use the API
# Skip for 'mcp' command - it has its own lifespan that handles initialization
# Skip for API-using commands (status, sync, etc.) - they handle initialization via deps.py
# Skip for 'reset' command - it manages its own database lifecycle
skip_init_commands = {"mcp", "status", "sync", "project", "tool", "reset"}
if (
not version
and ctx.invoked_subcommand is not None
and ctx.invoked_subcommand not in skip_init_commands
):
from basic_memory.services.initialization import ensure_initialization
app_config = ConfigManager().config
ensure_initialization(app_config)
ensure_initialization(container.config)
## import
+27 -4
View File
@@ -7,6 +7,9 @@ import os
import secrets
import time
import webbrowser
from contextlib import asynccontextmanager
from collections.abc import AsyncIterator, Callable
from typing import AsyncContextManager
import httpx
from rich.console import Console
@@ -19,7 +22,12 @@ console = Console()
class CLIAuth:
"""Handles WorkOS OAuth Device Authorization for CLI tools."""
def __init__(self, client_id: str, authkit_domain: str):
def __init__(
self,
client_id: str,
authkit_domain: str,
http_client_factory: Callable[[], AsyncContextManager[httpx.AsyncClient]] | None = None,
):
self.client_id = client_id
self.authkit_domain = authkit_domain
app_config = ConfigManager().config
@@ -28,6 +36,21 @@ class CLIAuth:
# PKCE parameters
self.code_verifier = None
self.code_challenge = None
self._http_client_factory = http_client_factory
@asynccontextmanager
async def _get_http_client(self) -> AsyncIterator[httpx.AsyncClient]:
"""Create an AsyncClient, optionally via injected factory.
Why: enables reliable tests without monkeypatching httpx internals while
still using real httpx request/response objects.
"""
if self._http_client_factory:
async with self._http_client_factory() as client:
yield client
else:
async with httpx.AsyncClient() as client:
yield client
def generate_pkce_pair(self) -> tuple[str, str]:
"""Generate PKCE code verifier and challenge."""
@@ -57,7 +80,7 @@ class CLIAuth:
}
try:
async with httpx.AsyncClient() as client:
async with self._get_http_client() as client:
response = await client.post(device_auth_url, data=data)
if response.status_code == 200:
@@ -111,7 +134,7 @@ class CLIAuth:
for _attempt in range(max_attempts):
try:
async with httpx.AsyncClient() as client:
async with self._get_http_client() as client:
response = await client.post(token_url, data=data)
if response.status_code == 200:
@@ -201,7 +224,7 @@ class CLIAuth:
}
try:
async with httpx.AsyncClient() as client:
async with self._get_http_client() as client:
response = await client.post(token_url, data=data)
if response.status_code == 200:
+3 -1
View File
@@ -1,7 +1,7 @@
"""CLI commands for basic-memory."""
from . import status, db, import_memory_json, mcp, import_claude_conversations
from . import import_claude_projects, import_chatgpt, tool, project
from . import import_claude_projects, import_chatgpt, tool, project, format, telemetry
__all__ = [
"status",
@@ -13,4 +13,6 @@ __all__ = [
"import_chatgpt",
"tool",
"project",
"format",
"telemetry",
]
@@ -1,6 +1,9 @@
"""Cloud API client utilities."""
from collections.abc import AsyncIterator
from typing import Optional
from contextlib import asynccontextmanager
from typing import AsyncContextManager, Callable
import httpx
import typer
@@ -11,6 +14,8 @@ from basic_memory.config import ConfigManager
console = Console()
HttpClientFactory = Callable[[], AsyncContextManager[httpx.AsyncClient]]
class CloudAPIError(Exception):
"""Exception raised for cloud API errors."""
@@ -38,14 +43,14 @@ def get_cloud_config() -> tuple[str, str, str]:
return config.cloud_client_id, config.cloud_domain, config.cloud_host
async def get_authenticated_headers() -> dict[str, str]:
async def get_authenticated_headers(auth: CLIAuth | None = None) -> dict[str, str]:
"""
Get authentication headers with JWT token.
handles jwt refresh if needed.
"""
client_id, domain, _ = get_cloud_config()
auth = CLIAuth(client_id=client_id, authkit_domain=domain)
token = await auth.get_valid_token()
auth_obj = auth or CLIAuth(client_id=client_id, authkit_domain=domain)
token = await auth_obj.get_valid_token()
if not token:
console.print("[red]Not authenticated. Please run 'basic-memory cloud login' first.[/red]")
raise typer.Exit(1)
@@ -53,21 +58,31 @@ async def get_authenticated_headers() -> dict[str, str]:
return {"Authorization": f"Bearer {token}"}
@asynccontextmanager
async def _default_http_client(timeout: float) -> AsyncIterator[httpx.AsyncClient]:
async with httpx.AsyncClient(timeout=timeout) as client:
yield client
async def make_api_request(
method: str,
url: str,
headers: Optional[dict] = None,
json_data: Optional[dict] = None,
timeout: float = 30.0,
*,
auth: CLIAuth | None = None,
http_client_factory: HttpClientFactory | None = None,
) -> httpx.Response:
"""Make an API request to the cloud service."""
headers = headers or {}
auth_headers = await get_authenticated_headers()
auth_headers = await get_authenticated_headers(auth=auth)
headers.update(auth_headers)
# Add debug headers to help with compression issues
headers.setdefault("Accept-Encoding", "identity") # Disable compression for debugging
async with httpx.AsyncClient(timeout=timeout) as client:
client_factory = http_client_factory or (lambda: _default_http_client(timeout))
async with client_factory() as client:
try:
response = await client.request(method=method, url=url, headers=headers, json=json_data)
response.raise_for_status()
@@ -16,7 +16,10 @@ class CloudUtilsError(Exception):
pass
async def fetch_cloud_projects() -> CloudProjectList:
async def fetch_cloud_projects(
*,
api_request=make_api_request,
) -> CloudProjectList:
"""Fetch list of projects from cloud API.
Returns:
@@ -27,14 +30,18 @@ async def fetch_cloud_projects() -> CloudProjectList:
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
response = await make_api_request(method="GET", url=f"{host_url}/proxy/projects/projects")
response = await api_request(method="GET", url=f"{host_url}/proxy/projects/projects")
return CloudProjectList.model_validate(response.json())
except Exception as e:
raise CloudUtilsError(f"Failed to fetch cloud projects: {e}") from e
async def create_cloud_project(project_name: str) -> CloudProjectCreateResponse:
async def create_cloud_project(
project_name: str,
*,
api_request=make_api_request,
) -> CloudProjectCreateResponse:
"""Create a new project on cloud.
Args:
@@ -57,7 +64,7 @@ async def create_cloud_project(project_name: str) -> CloudProjectCreateResponse:
set_default=False,
)
response = await make_api_request(
response = await api_request(
method="POST",
url=f"{host_url}/proxy/projects/projects",
headers={"Content-Type": "application/json"},
@@ -84,7 +91,7 @@ async def sync_project(project_name: str, force_full: bool = False) -> None:
raise CloudUtilsError(f"Failed to sync project '{project_name}': {e}") from e
async def project_exists(project_name: str) -> bool:
async def project_exists(project_name: str, *, api_request=make_api_request) -> bool:
"""Check if a project exists on cloud.
Args:
@@ -94,7 +101,7 @@ async def project_exists(project_name: str) -> bool:
True if project exists, False otherwise
"""
try:
projects = await fetch_cloud_projects()
projects = await fetch_cloud_projects(api_request=api_request)
project_names = {p.name for p in projects.projects}
return project_name in project_names
except Exception:
@@ -9,11 +9,14 @@ This module provides simplified, project-scoped rclone operations:
Replaces tenant-wide sync with project-scoped workflows.
"""
import re
import subprocess
from dataclasses import dataclass
from functools import lru_cache
from pathlib import Path
from typing import Optional
from typing import Callable, Optional, Protocol
from loguru import logger
from rich.console import Console
from basic_memory.cli.commands.cloud.rclone_installer import is_rclone_installed
@@ -21,6 +24,17 @@ from basic_memory.utils import normalize_project_path
console = Console()
# Minimum rclone version for --create-empty-src-dirs support
MIN_RCLONE_VERSION_EMPTY_DIRS = (1, 64, 0)
class RunResult(Protocol):
returncode: int
stdout: str
RunFunc = Callable[..., RunResult]
IsInstalledFunc = Callable[[], bool]
class RcloneError(Exception):
"""Exception raised for rclone command errors."""
@@ -28,13 +42,13 @@ class RcloneError(Exception):
pass
def check_rclone_installed() -> None:
def check_rclone_installed(is_installed: IsInstalledFunc = is_rclone_installed) -> None:
"""Check if rclone is installed and raise helpful error if not.
Raises:
RcloneError: If rclone is not installed with installation instructions
"""
if not is_rclone_installed():
if not is_installed():
raise RcloneError(
"rclone is not installed.\n\n"
"Install rclone by running: bm cloud setup\n"
@@ -43,6 +57,41 @@ def check_rclone_installed() -> None:
)
@lru_cache(maxsize=1)
def get_rclone_version(run: RunFunc = subprocess.run) -> tuple[int, int, int] | None:
"""Get rclone version as (major, minor, patch) tuple.
Returns:
Version tuple like (1, 64, 2), or None if version cannot be determined.
Note:
Result is cached since rclone version won't change during runtime.
"""
try:
result = run(["rclone", "version"], capture_output=True, text=True, timeout=10)
# Parse "rclone v1.64.2" or "rclone v1.60.1-DEV"
match = re.search(r"v(\d+)\.(\d+)\.(\d+)", result.stdout)
if match:
version = (int(match.group(1)), int(match.group(2)), int(match.group(3)))
logger.debug(f"Detected rclone version: {version}")
return version
except Exception as e:
logger.warning(f"Could not determine rclone version: {e}")
return None
def supports_create_empty_src_dirs(version: tuple[int, int, int] | None) -> bool:
"""Check if installed rclone supports --create-empty-src-dirs flag.
Returns:
True if rclone version >= 1.64.0, False otherwise.
"""
if version is None:
# If we can't determine version, assume older and skip the flag
return False
return version >= MIN_RCLONE_VERSION_EMPTY_DIRS
@dataclass
class SyncProject:
"""Project configured for cloud sync.
@@ -125,6 +174,10 @@ def project_sync(
bucket_name: str,
dry_run: bool = False,
verbose: bool = False,
*,
run: RunFunc = subprocess.run,
is_installed: IsInstalledFunc = is_rclone_installed,
filter_path: Path | None = None,
) -> bool:
"""One-way sync: local → cloud.
@@ -142,14 +195,14 @@ def project_sync(
Raises:
RcloneError: If project has no local_sync_path configured or rclone not installed
"""
check_rclone_installed()
check_rclone_installed(is_installed=is_installed)
if not project.local_sync_path:
raise RcloneError(f"Project {project.name} has no local_sync_path configured")
local_path = Path(project.local_sync_path).expanduser()
remote_path = get_project_remote(project, bucket_name)
filter_path = get_bmignore_filter_path()
filter_path = filter_path or get_bmignore_filter_path()
cmd = [
"rclone",
@@ -168,7 +221,7 @@ def project_sync(
if dry_run:
cmd.append("--dry-run")
result = subprocess.run(cmd, text=True)
result = run(cmd, text=True)
return result.returncode == 0
@@ -178,6 +231,13 @@ def project_bisync(
dry_run: bool = False,
resync: bool = False,
verbose: bool = False,
*,
run: RunFunc = subprocess.run,
is_installed: IsInstalledFunc = is_rclone_installed,
version: tuple[int, int, int] | None = None,
filter_path: Path | None = None,
state_path: Path | None = None,
is_initialized: Callable[[str], bool] = bisync_initialized,
) -> bool:
"""Two-way sync: local ↔ cloud.
@@ -200,15 +260,15 @@ def project_bisync(
Raises:
RcloneError: If project has no local_sync_path, needs --resync, or rclone not installed
"""
check_rclone_installed()
check_rclone_installed(is_installed=is_installed)
if not project.local_sync_path:
raise RcloneError(f"Project {project.name} has no local_sync_path configured")
local_path = Path(project.local_sync_path).expanduser()
remote_path = get_project_remote(project, bucket_name)
filter_path = get_bmignore_filter_path()
state_path = get_project_bisync_state(project.name)
filter_path = filter_path or get_bmignore_filter_path()
state_path = state_path or get_project_bisync_state(project.name)
# Ensure state directory exists
state_path.mkdir(parents=True, exist_ok=True)
@@ -218,7 +278,6 @@ def project_bisync(
"bisync",
str(local_path),
remote_path,
"--create-empty-src-dirs",
"--resilient",
"--conflict-resolve=newer",
"--max-delete=25",
@@ -229,6 +288,11 @@ def project_bisync(
str(state_path),
]
# Add --create-empty-src-dirs if rclone version supports it (v1.64+)
version = version if version is not None else get_rclone_version(run=run)
if supports_create_empty_src_dirs(version):
cmd.append("--create-empty-src-dirs")
if verbose:
cmd.append("--verbose")
else:
@@ -241,13 +305,13 @@ def project_bisync(
cmd.append("--resync")
# Check if first run requires resync
if not resync and not bisync_initialized(project.name) and not dry_run:
if not resync and not is_initialized(project.name) and not dry_run:
raise RcloneError(
f"First bisync for {project.name} requires --resync to establish baseline.\n"
f"Run: bm project bisync --name {project.name} --resync"
)
result = subprocess.run(cmd, text=True)
result = run(cmd, text=True)
return result.returncode == 0
@@ -255,6 +319,10 @@ def project_check(
project: SyncProject,
bucket_name: str,
one_way: bool = False,
*,
run: RunFunc = subprocess.run,
is_installed: IsInstalledFunc = is_rclone_installed,
filter_path: Path | None = None,
) -> bool:
"""Check integrity between local and cloud.
@@ -271,14 +339,14 @@ def project_check(
Raises:
RcloneError: If project has no local_sync_path configured or rclone not installed
"""
check_rclone_installed()
check_rclone_installed(is_installed=is_installed)
if not project.local_sync_path:
raise RcloneError(f"Project {project.name} has no local_sync_path configured")
local_path = Path(project.local_sync_path).expanduser()
remote_path = get_project_remote(project, bucket_name)
filter_path = get_bmignore_filter_path()
filter_path = filter_path or get_bmignore_filter_path()
cmd = [
"rclone",
@@ -292,7 +360,7 @@ def project_check(
if one_way:
cmd.append("--one-way")
result = subprocess.run(cmd, capture_output=True, text=True)
result = run(cmd, capture_output=True, text=True)
return result.returncode == 0
@@ -300,6 +368,9 @@ def project_ls(
project: SyncProject,
bucket_name: str,
path: Optional[str] = None,
*,
run: RunFunc = subprocess.run,
is_installed: IsInstalledFunc = is_rclone_installed,
) -> list[str]:
"""List files in remote project.
@@ -315,12 +386,12 @@ def project_ls(
subprocess.CalledProcessError: If rclone command fails
RcloneError: If rclone is not installed
"""
check_rclone_installed()
check_rclone_installed(is_installed=is_installed)
remote_path = get_project_remote(project, bucket_name)
if path:
remote_path = f"{remote_path}/{path}"
cmd = ["rclone", "ls", remote_path]
result = subprocess.run(cmd, capture_output=True, text=True, check=True)
result = run(cmd, capture_output=True, text=True, check=True)
return result.stdout.splitlines()
+10 -3
View File
@@ -2,6 +2,8 @@
import os
from pathlib import Path
from contextlib import AbstractAsyncContextManager
from typing import Callable
import aiofiles
import httpx
@@ -20,6 +22,9 @@ async def upload_path(
verbose: bool = False,
use_gitignore: bool = True,
dry_run: bool = False,
*,
client_cm_factory: Callable[[], AbstractAsyncContextManager[httpx.AsyncClient]] | None = None,
put_func=call_put,
) -> bool:
"""
Upload a file or directory to cloud project via WebDAV.
@@ -85,8 +90,10 @@ async def upload_path(
size_str = f"{size / (1024 * 1024):.1f} MB"
print(f" {relative_path} ({size_str})")
else:
# Upload files using httpx
async with get_client() as client:
# Upload files using httpx.
# Allow injection for tests (MockTransport) while keeping production default.
cm_factory = client_cm_factory or get_client
async with cm_factory() as client:
for i, (file_path, relative_path) in enumerate(files_to_upload, 1):
# Skip archive files (zip, tar, gz, etc.)
if _is_archive_file(file_path):
@@ -110,7 +117,7 @@ async def upload_path(
# Upload via HTTP PUT to WebDAV endpoint with mtime header
# Using X-OC-Mtime (ownCloud/Nextcloud standard)
response = await call_put(
response = await put_func(
client, remote_path, content=content, headers={"X-OC-Mtime": str(mtime)}
)
response.raise_for_status()
+52 -4
View File
@@ -1,12 +1,14 @@
"""utility functions for commands"""
from typing import Optional
import asyncio
from typing import Optional, TypeVar, Coroutine, Any
from mcp.server.fastmcp.exceptions import ToolError
import typer
from rich.console import Console
from basic_memory import db
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.tools.utils import call_post, call_get
@@ -15,24 +17,70 @@ from basic_memory.schemas import ProjectInfoResponse
console = Console()
T = TypeVar("T")
async def run_sync(project: Optional[str] = None, force_full: bool = False):
def run_with_cleanup(coro: Coroutine[Any, Any, T]) -> T:
"""Run an async coroutine with proper database cleanup.
This helper ensures database connections are cleaned up before the event
loop closes, preventing process hangs in CLI commands.
Args:
coro: The coroutine to run
Returns:
The result of the coroutine
"""
async def _with_cleanup() -> T:
try:
return await coro
finally:
await db.shutdown_db()
return asyncio.run(_with_cleanup())
async def run_sync(
project: Optional[str] = None,
force_full: bool = False,
run_in_background: bool = True,
):
"""Run sync operation via API endpoint.
Args:
project: Optional project name
force_full: If True, force a full scan bypassing watermark optimization
run_in_background: If True, return immediately; if False, wait for completion
"""
try:
async with get_client() as client:
project_item = await get_active_project(client, project, None)
url = f"{project_item.project_url}/project/sync"
params = []
if force_full:
url += "?force_full=true"
params.append("force_full=true")
if not run_in_background:
params.append("run_in_background=false")
if params:
url += "?" + "&".join(params)
response = await call_post(client, url)
data = response.json()
console.print(f"[green]{data['message']}[/green]")
# Background mode returns {"message": "..."}, foreground returns SyncReportResponse
if "message" in data:
console.print(f"[green]{data['message']}[/green]")
else:
# Foreground mode - show summary of sync results
total = data.get("total", 0)
new_count = len(data.get("new", []))
modified_count = len(data.get("modified", []))
deleted_count = len(data.get("deleted", []))
console.print(
f"[green]Synced {total} files[/green] "
f"(new: {new_count}, modified: {modified_count}, deleted: {deleted_count})"
)
except (ToolError, ValueError) as e:
console.print(f"[red]Sync failed: {e}[/red]")
raise typer.Exit(1)
+78 -19
View File
@@ -1,13 +1,50 @@
"""Database management commands."""
import asyncio
from pathlib import Path
import typer
from loguru import logger
from rich.console import Console
from sqlalchemy.exc import OperationalError
from basic_memory import db
from basic_memory.cli.app import app
from basic_memory.config import ConfigManager, BasicMemoryConfig, save_basic_memory_config
from basic_memory.config import ConfigManager
from basic_memory.repository import ProjectRepository
from basic_memory.services.initialization import reconcile_projects_with_config
from basic_memory.sync.sync_service import get_sync_service
console = Console()
async def _reindex_projects(app_config):
"""Reindex all projects in a single async context.
This ensures all database operations use the same event loop,
and proper cleanup happens when the function completes.
"""
try:
await reconcile_projects_with_config(app_config)
# Get database session (migrations already run if needed)
_, session_maker = await db.get_or_create_db(
db_path=app_config.database_path,
db_type=db.DatabaseType.FILESYSTEM,
)
project_repository = ProjectRepository(session_maker)
projects = await project_repository.get_active_projects()
for project in projects:
console.print(f" Indexing [cyan]{project.name}[/cyan]...")
logger.info(f"Starting sync for project: {project.name}")
sync_service = await get_sync_service(project)
sync_dir = Path(project.path)
await sync_service.sync(sync_dir, project_name=project.name)
logger.info(f"Sync completed for project: {project.name}")
finally:
# Clean up database connections before event loop closes
await db.shutdown_db()
@app.command()
@@ -15,30 +52,52 @@ def reset(
reindex: bool = typer.Option(False, "--reindex", help="Rebuild db index from filesystem"),
): # pragma: no cover
"""Reset database (drop all tables and recreate)."""
if typer.confirm("This will delete all data in your db. Are you sure?"):
console.print(
"[yellow]Note:[/yellow] This only deletes the index database. "
"Your markdown note files will not be affected.\n"
"Use [green]bm reset --reindex[/green] to automatically rebuild the index afterward."
)
if typer.confirm("Reset the database index?"):
logger.info("Resetting database...")
config_manager = ConfigManager()
app_config = config_manager.config
# Get database path
db_path = app_config.app_database_path
# Delete the database file if it exists
if db_path.exists():
db_path.unlink()
logger.info(f"Database file deleted: {db_path}")
# Delete the database file and WAL files if they exist
for suffix in ["", "-shm", "-wal"]:
path = db_path.parent / f"{db_path.name}{suffix}"
if path.exists():
try:
path.unlink()
logger.info(f"Deleted: {path}")
except OSError as e:
console.print(
f"[red]Error:[/red] Cannot delete {path.name}: {e}\n"
"The database may be in use by another process (e.g., MCP server).\n"
"Please close Claude Desktop or any other Basic Memory clients and try again."
)
raise typer.Exit(1)
# Reset project configuration
config = BasicMemoryConfig()
save_basic_memory_config(config_manager.config_file, config)
logger.info("Project configuration reset to default")
# Create a new empty database
asyncio.run(db.run_migrations(app_config))
logger.info("Database reset complete")
# Create a new empty database (preserves project configuration)
try:
asyncio.run(db.run_migrations(app_config))
except OperationalError as e:
if "disk I/O error" in str(e) or "database is locked" in str(e):
console.print(
"[red]Error:[/red] Cannot access database. "
"It may be in use by another process (e.g., MCP server).\n"
"Please close Claude Desktop or any other Basic Memory clients and try again."
)
raise typer.Exit(1)
raise
console.print("[green]Database reset complete[/green]")
if reindex:
# Run database sync directly
from basic_memory.cli.commands.command_utils import run_sync
logger.info("Rebuilding search index from filesystem...")
asyncio.run(run_sync(project=None))
projects = list(app_config.projects)
if not projects:
console.print("[yellow]No projects configured. Skipping reindex.[/yellow]")
else:
console.print(f"Rebuilding search index for {len(projects)} project(s)...")
asyncio.run(_reindex_projects(app_config))
console.print("[green]Reindex complete[/green]")
+198
View File
@@ -0,0 +1,198 @@
"""Format command for basic-memory CLI."""
import asyncio
from pathlib import Path
from typing import Annotated, Optional
import typer
from loguru import logger
from rich.console import Console
from rich.progress import Progress, SpinnerColumn, TextColumn
from basic_memory.cli.app import app
from basic_memory.config import ConfigManager, get_project_config
from basic_memory.file_utils import format_file
console = Console()
def is_markdown_extension(path: Path) -> bool:
"""Check if file has a markdown extension."""
return path.suffix.lower() in (".md", ".markdown")
async def format_single_file(file_path: Path, app_config) -> tuple[Path, bool, Optional[str]]:
"""Format a single file.
Returns:
Tuple of (path, success, error_message)
"""
try:
result = await format_file(
file_path, app_config, is_markdown=is_markdown_extension(file_path)
)
if result is not None:
return (file_path, True, None)
else:
return (file_path, False, "No formatter configured or formatting skipped")
except Exception as e:
return (file_path, False, str(e))
async def format_files(
paths: list[Path], app_config, show_progress: bool = True
) -> tuple[int, int, list[tuple[Path, str]]]:
"""Format multiple files.
Returns:
Tuple of (formatted_count, skipped_count, errors)
"""
formatted = 0
skipped = 0
errors: list[tuple[Path, str]] = []
if show_progress:
with Progress(
SpinnerColumn(),
TextColumn("[progress.description]{task.description}"),
console=console,
) as progress:
task = progress.add_task("Formatting files...", total=len(paths))
for file_path in paths:
path, success, error = await format_single_file(file_path, app_config)
if success:
formatted += 1
elif error and "No formatter configured" not in error:
errors.append((path, error))
else:
skipped += 1
progress.update(task, advance=1)
else:
for file_path in paths:
path, success, error = await format_single_file(file_path, app_config)
if success:
formatted += 1
elif error and "No formatter configured" not in error:
errors.append((path, error))
else:
skipped += 1
return formatted, skipped, errors
async def run_format(
path: Optional[Path] = None,
project: Optional[str] = None,
) -> None:
"""Run the format command."""
app_config = ConfigManager().config
# Check if formatting is enabled
if (
not app_config.format_on_save
and not app_config.formatter_command
and not app_config.formatters
):
console.print(
"[yellow]No formatters configured. Set format_on_save=true and "
"formatter_command or formatters in your config.[/yellow]"
)
console.print(
"\nExample config (~/.basic-memory/config.json):\n"
' "format_on_save": true,\n'
' "formatter_command": "prettier --write {file}"\n'
)
raise typer.Exit(1)
# Temporarily enable format_on_save for this command
# (so format_file actually runs the formatter)
original_format_on_save = app_config.format_on_save
app_config.format_on_save = True
try:
# Determine which files to format
if path:
# Format specific file or directory
if path.is_file():
files = [path]
elif path.is_dir():
# Find all markdown and json files
files = (
list(path.rglob("*.md"))
+ list(path.rglob("*.json"))
+ list(path.rglob("*.canvas"))
)
else:
console.print(f"[red]Path not found: {path}[/red]")
raise typer.Exit(1)
else:
# Format all files in project
project_config = get_project_config(project)
project_path = Path(project_config.home)
if not project_path.exists():
console.print(f"[red]Project path not found: {project_path}[/red]")
raise typer.Exit(1)
# Find all markdown and json files
files = (
list(project_path.rglob("*.md"))
+ list(project_path.rglob("*.json"))
+ list(project_path.rglob("*.canvas"))
)
if not files:
console.print("[yellow]No files found to format.[/yellow]")
return
console.print(f"Found {len(files)} file(s) to format...")
formatted, skipped, errors = await format_files(files, app_config)
# Print summary
console.print()
if formatted > 0:
console.print(f"[green]Formatted: {formatted} file(s)[/green]")
if skipped > 0:
console.print(f"[dim]Skipped: {skipped} file(s) (no formatter for extension)[/dim]")
if errors:
console.print(f"[red]Errors: {len(errors)} file(s)[/red]")
for path, error in errors:
console.print(f" [red]{path}[/red]: {error}")
finally:
# Restore original setting
app_config.format_on_save = original_format_on_save
@app.command()
def format(
path: Annotated[
Optional[Path],
typer.Argument(help="File or directory to format. Defaults to current project."),
] = None,
project: Annotated[
Optional[str],
typer.Option("--project", "-p", help="Project name to format."),
] = None,
) -> None:
"""Format files using configured formatters.
Uses the formatter_command or formatters settings from your config.
By default, formats all .md, .json, and .canvas files in the current project.
Examples:
basic-memory format # Format all files in current project
basic-memory format --project research # Format files in specific project
basic-memory format notes/meeting.md # Format a specific file
basic-memory format notes/ # Format all files in directory
"""
try:
asyncio.run(run_format(path, project))
except Exception as e:
if not isinstance(e, typer.Exit):
logger.error(f"Error formatting files: {e}")
console.print(f"[red]Error formatting files: {e}[/red]")
raise typer.Exit(code=1)
raise
@@ -3,13 +3,14 @@
import asyncio
import json
from pathlib import Path
from typing import Annotated
from typing import Annotated, Tuple
import typer
from basic_memory.cli.app import import_app
from basic_memory.config import get_project_config
from basic_memory.config import ConfigManager, get_project_config
from basic_memory.importers import ChatGPTImporter
from basic_memory.markdown import EntityParser, MarkdownProcessor
from basic_memory.services.file_service import FileService
from loguru import logger
from rich.console import Console
from rich.panel import Panel
@@ -17,11 +18,14 @@ from rich.panel import Panel
console = Console()
async def get_markdown_processor() -> MarkdownProcessor:
"""Get MarkdownProcessor instance."""
async def get_importer_dependencies() -> Tuple[MarkdownProcessor, FileService]:
"""Get MarkdownProcessor and FileService instances for importers."""
config = get_project_config()
app_config = ConfigManager().config
entity_parser = EntityParser(config.home)
return MarkdownProcessor(entity_parser)
markdown_processor = MarkdownProcessor(entity_parser, app_config=app_config)
file_service = FileService(config.home, markdown_processor, app_config=app_config)
return markdown_processor, file_service
@import_app.command(name="chatgpt", help="Import conversations from ChatGPT JSON export.")
@@ -48,15 +52,15 @@ def import_chatgpt(
typer.echo(f"Error: File not found: {conversations_json}", err=True)
raise typer.Exit(1)
# Get markdown processor
markdown_processor = asyncio.run(get_markdown_processor())
# Get importer dependencies
markdown_processor, file_service = asyncio.run(get_importer_dependencies())
config = get_project_config()
# Process the file
base_path = config.home / folder
console.print(f"\nImporting chats from {conversations_json}...writing to {base_path}")
# Create importer and run import
importer = ChatGPTImporter(config.home, markdown_processor)
importer = ChatGPTImporter(config.home, markdown_processor, file_service)
with conversations_json.open("r", encoding="utf-8") as file:
json_data = json.load(file)
result = asyncio.run(importer.import_data(json_data, folder))
@@ -3,13 +3,14 @@
import asyncio
import json
from pathlib import Path
from typing import Annotated
from typing import Annotated, Tuple
import typer
from basic_memory.cli.app import claude_app
from basic_memory.config import get_project_config
from basic_memory.config import ConfigManager, get_project_config
from basic_memory.importers.claude_conversations_importer import ClaudeConversationsImporter
from basic_memory.markdown import EntityParser, MarkdownProcessor
from basic_memory.services.file_service import FileService
from loguru import logger
from rich.console import Console
from rich.panel import Panel
@@ -17,11 +18,14 @@ from rich.panel import Panel
console = Console()
async def get_markdown_processor() -> MarkdownProcessor:
"""Get MarkdownProcessor instance."""
async def get_importer_dependencies() -> Tuple[MarkdownProcessor, FileService]:
"""Get MarkdownProcessor and FileService instances for importers."""
config = get_project_config()
app_config = ConfigManager().config
entity_parser = EntityParser(config.home)
return MarkdownProcessor(entity_parser)
markdown_processor = MarkdownProcessor(entity_parser, app_config=app_config)
file_service = FileService(config.home, markdown_processor, app_config=app_config)
return markdown_processor, file_service
@claude_app.command(name="conversations", help="Import chat conversations from Claude.ai.")
@@ -49,11 +53,11 @@ def import_claude(
typer.echo(f"Error: File not found: {conversations_json}", err=True)
raise typer.Exit(1)
# Get markdown processor
markdown_processor = asyncio.run(get_markdown_processor())
# Get importer dependencies
markdown_processor, file_service = asyncio.run(get_importer_dependencies())
# Create the importer
importer = ClaudeConversationsImporter(config.home, markdown_processor)
importer = ClaudeConversationsImporter(config.home, markdown_processor, file_service)
# Process the file
base_path = config.home / folder
@@ -3,13 +3,14 @@
import asyncio
import json
from pathlib import Path
from typing import Annotated
from typing import Annotated, Tuple
import typer
from basic_memory.cli.app import claude_app
from basic_memory.config import get_project_config
from basic_memory.config import ConfigManager, get_project_config
from basic_memory.importers.claude_projects_importer import ClaudeProjectsImporter
from basic_memory.markdown import EntityParser, MarkdownProcessor
from basic_memory.services.file_service import FileService
from loguru import logger
from rich.console import Console
from rich.panel import Panel
@@ -17,11 +18,14 @@ from rich.panel import Panel
console = Console()
async def get_markdown_processor() -> MarkdownProcessor:
"""Get MarkdownProcessor instance."""
async def get_importer_dependencies() -> Tuple[MarkdownProcessor, FileService]:
"""Get MarkdownProcessor and FileService instances for importers."""
config = get_project_config()
app_config = ConfigManager().config
entity_parser = EntityParser(config.home)
return MarkdownProcessor(entity_parser)
markdown_processor = MarkdownProcessor(entity_parser, app_config=app_config)
file_service = FileService(config.home, markdown_processor, app_config=app_config)
return markdown_processor, file_service
@claude_app.command(name="projects", help="Import projects from Claude.ai.")
@@ -48,11 +52,11 @@ def import_projects(
typer.echo(f"Error: File not found: {projects_json}", err=True)
raise typer.Exit(1)
# Get markdown processor
markdown_processor = asyncio.run(get_markdown_processor())
# Get importer dependencies
markdown_processor, file_service = asyncio.run(get_importer_dependencies())
# Create the importer
importer = ClaudeProjectsImporter(config.home, markdown_processor)
importer = ClaudeProjectsImporter(config.home, markdown_processor, file_service)
# Process the file
base_path = config.home / base_folder if base_folder else config.home
@@ -3,13 +3,14 @@
import asyncio
import json
from pathlib import Path
from typing import Annotated
from typing import Annotated, Tuple
import typer
from basic_memory.cli.app import import_app
from basic_memory.config import get_project_config
from basic_memory.config import ConfigManager, get_project_config
from basic_memory.importers.memory_json_importer import MemoryJsonImporter
from basic_memory.markdown import EntityParser, MarkdownProcessor
from basic_memory.services.file_service import FileService
from loguru import logger
from rich.console import Console
from rich.panel import Panel
@@ -17,11 +18,14 @@ from rich.panel import Panel
console = Console()
async def get_markdown_processor() -> MarkdownProcessor:
"""Get MarkdownProcessor instance."""
async def get_importer_dependencies() -> Tuple[MarkdownProcessor, FileService]:
"""Get MarkdownProcessor and FileService instances for importers."""
config = get_project_config()
app_config = ConfigManager().config
entity_parser = EntityParser(config.home)
return MarkdownProcessor(entity_parser)
markdown_processor = MarkdownProcessor(entity_parser, app_config=app_config)
file_service = FileService(config.home, markdown_processor, app_config=app_config)
return markdown_processor, file_service
@import_app.command()
@@ -47,11 +51,11 @@ def memory_json(
config = get_project_config()
try:
# Get markdown processor
markdown_processor = asyncio.run(get_markdown_processor())
# Get importer dependencies
markdown_processor, file_service = asyncio.run(get_importer_dependencies())
# Create the importer
importer = MemoryJsonImporter(config.home, markdown_processor)
importer = MemoryJsonImporter(config.home, markdown_processor, file_service)
# Process the file
base_path = config.home if not destination_folder else config.home / destination_folder
+8 -26
View File
@@ -1,14 +1,13 @@
"""MCP server command with streamable HTTP transport."""
import asyncio
import os
import typer
from typing import Optional
from basic_memory.cli.app import app
from basic_memory.config import ConfigManager
from basic_memory.config import ConfigManager, init_mcp_logging
# Import mcp instance
# Import mcp instance (has lifespan that handles initialization and file sync)
from basic_memory.mcp.server import mcp as mcp_server # pragma: no cover
# Import mcp tools to register them
@@ -17,8 +16,6 @@ import basic_memory.mcp.tools # noqa: F401 # pragma: no cover
# Import prompts to register them
import basic_memory.mcp.prompts # noqa: F401 # pragma: no cover
from loguru import logger
import threading
from basic_memory.services.initialization import initialize_file_sync
config = ConfigManager().config
@@ -43,7 +40,11 @@ if not config.cloud_mode_enabled:
- stdio: Standard I/O (good for local usage)
- streamable-http: Recommended for web deployments (default)
- sse: Server-Sent Events (for compatibility with existing clients)
Initialization, file sync, and cleanup are handled by the MCP server's lifespan.
"""
# Initialize logging for MCP (file only, stdout breaks protocol)
init_mcp_logging()
# Validate and set project constraint if specified
if project:
@@ -57,27 +58,8 @@ if not config.cloud_mode_enabled:
os.environ["BASIC_MEMORY_MCP_PROJECT"] = project_name
logger.info(f"MCP server constrained to project: {project_name}")
app_config = ConfigManager().config
def run_file_sync():
"""Run file sync in a separate thread with its own event loop."""
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
loop.run_until_complete(initialize_file_sync(app_config))
except Exception as e:
logger.error(f"File sync error: {e}", err=True)
finally:
loop.close()
logger.info(f"Sync changes enabled: {app_config.sync_changes}")
if app_config.sync_changes:
# Start the sync thread
sync_thread = threading.Thread(target=run_file_sync, daemon=True)
sync_thread.start()
logger.info("Started file sync in background")
# Now run the MCP server (blocks)
# Run the MCP server (blocks)
# Lifespan handles: initialization, migrations, file sync, cleanup
logger.info(f"Starting MCP server with {transport.upper()} transport")
if transport == "stdio":
+22 -9
View File
@@ -16,14 +16,9 @@ from datetime import datetime
from rich.panel import Panel
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.tools.utils import call_get
from basic_memory.schemas.project_info import ProjectList
from basic_memory.mcp.tools.utils import call_post
from basic_memory.schemas.project_info import ProjectStatusResponse
from basic_memory.mcp.tools.utils import call_delete
from basic_memory.mcp.tools.utils import call_put
from basic_memory.mcp.tools.utils import call_get, call_post, call_delete, call_put, call_patch
from basic_memory.schemas.project_info import ProjectList, ProjectStatusResponse
from basic_memory.utils import generate_permalink, normalize_project_path
from basic_memory.mcp.tools.utils import call_patch
# Import rclone commands for project sync
from basic_memory.cli.commands.cloud.rclone_commands import (
@@ -254,9 +249,17 @@ def remove_project(
async def _remove_project():
async with get_client() as client:
# Convert name to permalink for efficient resolution
project_permalink = generate_permalink(name)
# Use v2 project resolver to find project ID by permalink
resolve_data = {"identifier": project_permalink}
response = await call_post(client, "/v2/projects/resolve", json=resolve_data)
target_project = response.json()
# Use v2 API with project ID
response = await call_delete(
client, f"/projects/{project_permalink}?delete_notes={delete_notes}"
client, f"/v2/projects/{target_project['external_id']}?delete_notes={delete_notes}"
)
return ProjectStatusResponse.model_validate(response.json())
@@ -329,8 +332,18 @@ def set_default_project(
async def _set_default():
async with get_client() as client:
# Convert name to permalink for efficient resolution
project_permalink = generate_permalink(name)
response = await call_put(client, f"/projects/{project_permalink}/default")
# Use v2 project resolver to find project ID by permalink
resolve_data = {"identifier": project_permalink}
response = await call_post(client, "/v2/projects/resolve", json=resolve_data)
target_project = response.json()
# Use v2 API with project ID
response = await call_put(
client, f"/v2/projects/{target_project['external_id']}/default"
)
return ProjectStatusResponse.model_validate(response.json())
try:
+3 -2
View File
@@ -1,6 +1,5 @@
"""Status command for basic-memory CLI."""
import asyncio
from typing import Set, Dict
from typing import Annotated, Optional
@@ -165,8 +164,10 @@ def status(
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed file information"),
):
"""Show sync status between files and database."""
from basic_memory.cli.commands.command_utils import run_with_cleanup
try:
asyncio.run(run_status(project, verbose)) # pragma: no cover
run_with_cleanup(run_status(project, verbose)) # pragma: no cover
except Exception as e:
logger.error(f"Error checking status: {e}")
typer.echo(f"Error checking status: {e}", err=True)
@@ -0,0 +1,81 @@
"""Telemetry commands for basic-memory CLI."""
import typer
from rich.console import Console
from rich.panel import Panel
from basic_memory.cli.app import app
from basic_memory.config import ConfigManager
console = Console()
# Create telemetry subcommand group
telemetry_app = typer.Typer(help="Manage anonymous telemetry settings")
app.add_typer(telemetry_app, name="telemetry")
@telemetry_app.command("enable")
def enable() -> None:
"""Enable anonymous telemetry.
Telemetry helps improve Basic Memory by collecting anonymous usage data.
No personal data, note content, or file paths are ever collected.
"""
config_manager = ConfigManager()
config = config_manager.config
config.telemetry_enabled = True
config_manager.save_config(config)
console.print("[green]Telemetry enabled[/green]")
console.print("[dim]Thank you for helping improve Basic Memory![/dim]")
@telemetry_app.command("disable")
def disable() -> None:
"""Disable anonymous telemetry.
You can re-enable telemetry anytime with: bm telemetry enable
"""
config_manager = ConfigManager()
config = config_manager.config
config.telemetry_enabled = False
config_manager.save_config(config)
console.print("[yellow]Telemetry disabled[/yellow]")
@telemetry_app.command("status")
def status() -> None:
"""Show current telemetry status and what's collected."""
from basic_memory.telemetry import get_install_id, TELEMETRY_DOCS_URL
config = ConfigManager().config
status_text = (
"[green]enabled[/green]" if config.telemetry_enabled else "[yellow]disabled[/yellow]"
)
console.print(f"\nTelemetry: {status_text}")
console.print(f"Install ID: [dim]{get_install_id()}[/dim]")
console.print()
what_we_collect = """
[bold]What we collect:[/bold]
- App version, Python version, OS, architecture
- Feature usage (which MCP tools and CLI commands)
- Sync statistics (entity count, duration)
- Error types (sanitized, no file paths)
[bold]What we NEVER collect:[/bold]
- Note content, file names, or paths
- Personal information
- IP addresses
"""
console.print(
Panel(
what_we_collect.strip(),
title="Telemetry Details",
border_style="blue",
expand=False,
)
)
console.print(f"[dim]Details: {TELEMETRY_DOCS_URL}[/dim]")
+84
View File
@@ -0,0 +1,84 @@
"""CLI composition root for Basic Memory.
This container owns reading ConfigManager and environment variables for the
CLI entrypoint. Downstream modules receive config/dependencies explicitly
rather than reading globals.
Design principles:
- Only this module reads ConfigManager directly
- Runtime mode (cloud/local/test) is resolved here
- Different CLI commands may need different initialization
"""
from dataclasses import dataclass
from basic_memory.config import BasicMemoryConfig, ConfigManager
from basic_memory.runtime import RuntimeMode, resolve_runtime_mode
@dataclass
class CliContainer:
"""Composition root for the CLI entrypoint.
Holds resolved configuration and runtime context.
Created once at CLI startup, then used by subcommands.
"""
config: BasicMemoryConfig
mode: RuntimeMode
@classmethod
def create(cls) -> "CliContainer":
"""Create container by reading ConfigManager.
This is the single point where CLI reads global config.
"""
config = ConfigManager().config
mode = resolve_runtime_mode(
cloud_mode_enabled=config.cloud_mode_enabled,
is_test_env=config.is_test_env,
)
return cls(config=config, mode=mode)
# --- Runtime Mode Properties ---
@property
def is_cloud_mode(self) -> bool:
"""Whether running in cloud mode."""
return self.mode.is_cloud
# Module-level container instance (set by app callback)
_container: CliContainer | None = None
def get_container() -> CliContainer:
"""Get the current CLI container.
Returns:
The CLI container
Raises:
RuntimeError: If container hasn't been initialized
"""
if _container is None:
raise RuntimeError("CLI container not initialized. Call set_container() first.")
return _container
def set_container(container: CliContainer) -> None:
"""Set the CLI container (called by app callback)."""
global _container
_container = container
def get_or_create_container() -> CliContainer:
"""Get existing container or create new one.
This is useful for CLI commands that might be called before
the main app callback runs (e.g., eager options).
"""
global _container
if _container is None:
_container = CliContainer.create()
return _container
+7
View File
@@ -13,9 +13,16 @@ from basic_memory.cli.commands import ( # noqa: F401 # pragma: no cover
mcp,
project,
status,
telemetry,
tool,
)
# Re-apply warning filter AFTER all imports
# (authlib adds a DeprecationWarning filter that overrides ours)
import warnings # pragma: no cover
warnings.filterwarnings("ignore") # pragma: no cover
if __name__ == "__main__": # pragma: no cover
# start the app
app()
+169 -72
View File
@@ -6,12 +6,12 @@ from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from typing import Any, Dict, Literal, Optional, List, Tuple
from enum import Enum
from loguru import logger
from pydantic import BaseModel, Field, field_validator
from pydantic import BaseModel, Field, model_validator
from pydantic_settings import BaseSettings, SettingsConfigDict
import basic_memory
from basic_memory.utils import setup_logging, generate_permalink
@@ -24,6 +24,13 @@ WATCH_STATUS_JSON = "watch-status.json"
Environment = Literal["test", "dev", "user"]
class DatabaseBackend(str, Enum):
"""Supported database backends."""
SQLITE = "sqlite"
POSTGRES = "postgres"
@dataclass
class ProjectConfig:
"""Configuration for a specific basic-memory project."""
@@ -33,7 +40,7 @@ class ProjectConfig:
@property
def project(self):
return self.name
return self.name # pragma: no cover
@property
def project_url(self) -> str: # pragma: no cover
@@ -81,13 +88,43 @@ class BasicMemoryConfig(BaseSettings):
# overridden by ~/.basic-memory/config.json
log_level: str = "INFO"
# Database configuration
database_backend: DatabaseBackend = Field(
default=DatabaseBackend.SQLITE,
description="Database backend to use (sqlite or postgres)",
)
database_url: Optional[str] = Field(
default=None,
description="Database connection URL. For Postgres, use postgresql+asyncpg://user:pass@host:port/db. If not set, SQLite will use default path.",
)
# Database connection pool configuration (Postgres only)
db_pool_size: int = Field(
default=20,
description="Number of connections to keep in the pool (Postgres only)",
gt=0,
)
db_pool_overflow: int = Field(
default=40,
description="Max additional connections beyond pool_size under load (Postgres only)",
gt=0,
)
db_pool_recycle: int = Field(
default=180,
description="Recycle connections after N seconds to prevent stale connections. Default 180s works well with Neon's ~5 minute scale-to-zero (Postgres only)",
gt=0,
)
# Watch service configuration
sync_delay: int = Field(
default=1000, description="Milliseconds to wait after changes before syncing", gt=0
)
watch_project_reload_interval: int = Field(
default=30, description="Seconds between reloading project list in watch service", gt=0
default=300,
description="Seconds between reloading project list in watch service. Higher values reduce CPU usage by minimizing watcher restarts. Default 300s (5 min) balances efficiency with responsiveness to new projects.",
gt=0,
)
# update permalinks on move
@@ -128,6 +165,28 @@ class BasicMemoryConfig(BaseSettings):
description="Skip expensive initialization synchronization. Useful for cloud/stateless deployments where project reconciliation is not needed.",
)
# File formatting configuration
format_on_save: bool = Field(
default=False,
description="Automatically format files after saving using configured formatter. Disabled by default.",
)
formatter_command: Optional[str] = Field(
default=None,
description="External formatter command. Use {file} as placeholder for file path. If not set, uses built-in mdformat (Python, no Node.js required). Set to 'npx prettier --write {file}' for Prettier.",
)
formatters: Dict[str, str] = Field(
default_factory=dict,
description="Per-extension formatters. Keys are extensions (without dot), values are commands. Example: {'md': 'prettier --write {file}', 'json': 'prettier --write {file}'}",
)
formatter_timeout: float = Field(
default=5.0,
description="Maximum seconds to wait for formatter to complete",
gt=0,
)
# Project path constraints
project_root: Optional[str] = Field(
default=None,
@@ -162,6 +221,34 @@ class BasicMemoryConfig(BaseSettings):
description="Cloud project sync configuration mapping project names to their local paths and sync state",
)
# Telemetry configuration (Homebrew-style opt-out)
telemetry_enabled: bool = Field(
default=True,
description="Send anonymous usage statistics to help improve Basic Memory. Disable with: bm telemetry disable",
)
telemetry_notice_shown: bool = Field(
default=False,
description="Whether the one-time telemetry notice has been shown to the user",
)
@property
def is_test_env(self) -> bool:
"""Check if running in a test environment.
Returns True if any of:
- env field is set to "test"
- BASIC_MEMORY_ENV environment variable is "test"
- PYTEST_CURRENT_TEST environment variable is set (pytest is running)
Used to disable features like telemetry and file watchers during tests.
"""
return (
self.env == "test"
or os.getenv("BASIC_MEMORY_ENV", "").lower() == "test"
or os.getenv("PYTEST_CURRENT_TEST") is not None
)
@property
def cloud_mode_enabled(self) -> bool:
"""Check if cloud mode is enabled.
@@ -178,6 +265,36 @@ class BasicMemoryConfig(BaseSettings):
# Fall back to config file value
return self.cloud_mode
@classmethod
def for_cloud_tenant(
cls,
database_url: str,
projects: Optional[Dict[str, str]] = None,
) -> "BasicMemoryConfig":
"""Create config for cloud tenant - no config.json, database is source of truth.
This factory method creates a BasicMemoryConfig suitable for cloud deployments
where:
- Database is Postgres (Neon), not SQLite
- Projects are discovered from the database, not config file
- Path validation is skipped (no local filesystem in cloud)
- Initialization sync is skipped (stateless deployment)
Args:
database_url: Postgres connection URL for tenant database
projects: Optional project mapping (usually empty, discovered from DB)
Returns:
BasicMemoryConfig configured for cloud mode
"""
return cls( # pragma: no cover
database_backend=DatabaseBackend.POSTGRES,
database_url=database_url,
projects=projects or {},
cloud_mode=True,
skip_initialization_sync=True,
)
model_config = SettingsConfigDict(
env_prefix="BASIC_MEMORY_",
extra="ignore",
@@ -194,6 +311,10 @@ class BasicMemoryConfig(BaseSettings):
def model_post_init(self, __context: Any) -> None:
"""Ensure configuration is valid after initialization."""
# Skip project initialization in cloud mode - projects are discovered from DB
if self.database_backend == DatabaseBackend.POSTGRES: # pragma: no cover
return # pragma: no cover
# Ensure at least one project exists; if none exist then create main
if not self.projects: # pragma: no cover
self.projects["main"] = str(
@@ -236,19 +357,26 @@ class BasicMemoryConfig(BaseSettings):
"""Get all configured projects as ProjectConfig objects."""
return [ProjectConfig(name=name, home=Path(path)) for name, path in self.projects.items()]
@field_validator("projects")
@classmethod
def ensure_project_paths_exists(cls, v: Dict[str, str]) -> Dict[str, str]: # pragma: no cover
"""Ensure project path exists."""
for name, path_value in v.items():
@model_validator(mode="after")
def ensure_project_paths_exists(self) -> "BasicMemoryConfig": # pragma: no cover
"""Ensure project paths exist.
Skips path creation when using Postgres backend (cloud mode) since
cloud tenants don't use local filesystem paths.
"""
# Skip path creation for cloud mode - no local filesystem
if self.database_backend == DatabaseBackend.POSTGRES:
return self
for name, path_value in self.projects.items():
path = Path(path_value)
if not Path(path).exists():
if not path.exists():
try:
path.mkdir(parents=True)
except Exception as e:
logger.error(f"Failed to create project path: {e}")
raise e
return v
return self
@property
def data_dir_path(self):
@@ -451,69 +579,38 @@ def save_basic_memory_config(file_path: Path, config: BasicMemoryConfig) -> None
logger.error(f"Failed to save config: {e}")
# setup logging to a single log file in user home directory
user_home = Path.home()
log_dir = user_home / DATA_DIR_NAME
log_dir.mkdir(parents=True, exist_ok=True)
# Logging initialization functions for different entry points
# Process info for logging
def get_process_name(): # pragma: no cover
def init_cli_logging() -> None: # pragma: no cover
"""Initialize logging for CLI commands - file only.
CLI commands should not log to stdout to avoid interfering with
command output and shell integration.
"""
get the type of process for logging
"""
import sys
log_level = os.getenv("BASIC_MEMORY_LOG_LEVEL", "INFO")
setup_logging(log_level=log_level, log_to_file=True)
if "sync" in sys.argv:
return "sync"
elif "mcp" in sys.argv:
return "mcp"
elif "cli" in sys.argv:
return "cli"
def init_mcp_logging() -> None: # pragma: no cover
"""Initialize logging for MCP server - file only.
MCP server must not log to stdout as it would corrupt the
JSON-RPC protocol communication.
"""
log_level = os.getenv("BASIC_MEMORY_LOG_LEVEL", "INFO")
setup_logging(log_level=log_level, log_to_file=True)
def init_api_logging() -> None: # pragma: no cover
"""Initialize logging for API server.
Cloud mode (BASIC_MEMORY_CLOUD_MODE=1): stdout with structured context
Local mode: file only
"""
log_level = os.getenv("BASIC_MEMORY_LOG_LEVEL", "INFO")
cloud_mode = os.getenv("BASIC_MEMORY_CLOUD_MODE", "").lower() in ("1", "true")
if cloud_mode:
setup_logging(log_level=log_level, log_to_stdout=True, structured_context=True)
else:
return "api"
process_name = get_process_name()
# Global flag to track if logging has been set up
_LOGGING_SETUP = False
# Logging
def setup_basic_memory_logging(): # pragma: no cover
"""Set up logging for basic-memory, ensuring it only happens once."""
global _LOGGING_SETUP
if _LOGGING_SETUP:
# We can't log before logging is set up
# print("Skipping duplicate logging setup")
return
# Check for console logging environment variable - accept more truthy values
console_logging_env = os.getenv("BASIC_MEMORY_CONSOLE_LOGGING", "false").lower()
console_logging = console_logging_env in ("true", "1", "yes", "on")
# Check for log level environment variable first, fall back to config
log_level = os.getenv("BASIC_MEMORY_LOG_LEVEL")
if not log_level:
config_manager = ConfigManager()
log_level = config_manager.config.log_level
config_manager = ConfigManager()
config = get_project_config()
setup_logging(
env=config_manager.config.env,
home_dir=user_home, # Use user home for logs
log_level=log_level,
log_file=f"{DATA_DIR_NAME}/basic-memory-{process_name}.log",
console=console_logging,
)
logger.info(f"Basic Memory {basic_memory.__version__} (Project: {config.project})")
_LOGGING_SETUP = True
# Set up logging
setup_basic_memory_logging()
setup_logging(log_level=log_level, log_to_file=True)
+183 -77
View File
@@ -1,11 +1,12 @@
import asyncio
import os
import sys
from contextlib import asynccontextmanager
from enum import Enum, auto
from pathlib import Path
from typing import AsyncGenerator, Optional
from basic_memory.config import BasicMemoryConfig, ConfigManager
from basic_memory.config import BasicMemoryConfig, ConfigManager, DatabaseBackend
from alembic import command
from alembic.config import Config
@@ -20,12 +21,27 @@ from sqlalchemy.ext.asyncio import (
)
from sqlalchemy.pool import NullPool
from basic_memory.repository.search_repository import SearchRepository
from basic_memory.repository.postgres_search_repository import PostgresSearchRepository
from basic_memory.repository.sqlite_search_repository import SQLiteSearchRepository
# -----------------------------------------------------------------------------
# Windows event loop policy
# -----------------------------------------------------------------------------
# On Windows, the default ProactorEventLoop has known rough edges with aiosqlite
# during shutdown/teardown (threads posting results to a loop that's closing),
# which can manifest as:
# - "RuntimeError: Event loop is closed"
# - "IndexError: pop from an empty deque"
#
# The SelectorEventLoop doesn't support subprocess operations, so code that uses
# asyncio.create_subprocess_shell() (like sync_service._quick_count_files) must
# detect Windows and use fallback implementations.
if sys.platform == "win32": # pragma: no cover
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
# Module level state
_engine: Optional[AsyncEngine] = None
_session_maker: Optional[async_sessionmaker[AsyncSession]] = None
_migrations_completed: bool = False
class DatabaseType(Enum):
@@ -33,10 +49,41 @@ class DatabaseType(Enum):
MEMORY = auto()
FILESYSTEM = auto()
POSTGRES = auto()
@classmethod
def get_db_url(cls, db_path: Path, db_type: "DatabaseType") -> str:
"""Get SQLAlchemy URL for database path."""
def get_db_url(
cls, db_path: Path, db_type: "DatabaseType", config: Optional[BasicMemoryConfig] = None
) -> str:
"""Get SQLAlchemy URL for database path.
Args:
db_path: Path to SQLite database file (ignored for Postgres)
db_type: Type of database (MEMORY, FILESYSTEM, or POSTGRES)
config: Optional config to check for database backend and URL
Returns:
SQLAlchemy connection URL
"""
# Load config if not provided
if config is None:
config = ConfigManager().config
# Handle explicit Postgres type
if db_type == cls.POSTGRES:
if not config.database_url:
raise ValueError("DATABASE_URL must be set when using Postgres backend")
logger.info(f"Using Postgres database: {config.database_url}")
return config.database_url
# Check if Postgres backend is configured (for backward compatibility)
if config.database_backend == DatabaseBackend.POSTGRES:
if not config.database_url:
raise ValueError("DATABASE_URL must be set when using Postgres backend")
logger.info(f"Using Postgres database: {config.database_url}")
return config.database_url
# SQLite databases
if db_type == cls.MEMORY:
logger.info("Using in-memory SQLite database")
return "sqlite+aiosqlite://"
@@ -64,7 +111,14 @@ async def scoped_session(
factory = get_scoped_session_factory(session_maker)
session = factory()
try:
await session.execute(text("PRAGMA foreign_keys=ON"))
# Only enable foreign keys for SQLite (Postgres has them enabled by default)
# Detect database type from session's bind (engine) dialect
engine = session.get_bind()
dialect_name = engine.dialect.name
if dialect_name == "sqlite":
await session.execute(text("PRAGMA foreign_keys=ON"))
yield session
await session.commit()
except Exception:
@@ -103,13 +157,16 @@ def _configure_sqlite_connection(dbapi_conn, enable_wal: bool = True) -> None:
cursor.close()
def _create_engine_and_session(
db_path: Path, db_type: DatabaseType = DatabaseType.FILESYSTEM
) -> tuple[AsyncEngine, async_sessionmaker[AsyncSession]]:
"""Internal helper to create engine and session maker."""
db_url = DatabaseType.get_db_url(db_path, db_type)
logger.debug(f"Creating engine for db_url: {db_url}")
def _create_sqlite_engine(db_url: str, db_type: DatabaseType) -> AsyncEngine:
"""Create SQLite async engine with appropriate configuration.
Args:
db_url: SQLite connection URL
db_type: Database type (MEMORY or FILESYSTEM)
Returns:
Configured async engine for SQLite
"""
# Configure connection args with Windows-specific settings
connect_args: dict[str, bool | float | None] = {"check_same_thread": False}
@@ -146,6 +203,73 @@ def _create_engine_and_session(
"""Enable WAL mode on each connection."""
_configure_sqlite_connection(dbapi_conn, enable_wal=enable_wal)
return engine
def _create_postgres_engine(db_url: str, config: BasicMemoryConfig) -> AsyncEngine:
"""Create Postgres async engine with appropriate configuration.
Args:
db_url: Postgres connection URL (postgresql+asyncpg://...)
config: BasicMemoryConfig with pool settings
Returns:
Configured async engine for Postgres
"""
# Use NullPool connection issues.
# Assume connection pooler like PgBouncer handles connection pooling.
engine = create_async_engine(
db_url,
echo=False,
poolclass=NullPool, # No pooling - fresh connection per request
connect_args={
# Disable statement cache to avoid issues with prepared statements on reconnect
"statement_cache_size": 0,
# Allow 30s for commands (Neon cold start can take 2-5s, sometimes longer)
"command_timeout": 30,
# Allow 30s for initial connection (Neon wake-up time)
"timeout": 30,
"server_settings": {
"application_name": "basic-memory",
# Statement timeout for queries (30s to allow for cold start)
"statement_timeout": "30s",
},
},
)
logger.debug("Created Postgres engine with NullPool (no connection pooling)")
return engine
def _create_engine_and_session(
db_path: Path,
db_type: DatabaseType = DatabaseType.FILESYSTEM,
config: Optional[BasicMemoryConfig] = None,
) -> tuple[AsyncEngine, async_sessionmaker[AsyncSession]]:
"""Internal helper to create engine and session maker.
Args:
db_path: Path to database file (used for SQLite, ignored for Postgres)
db_type: Type of database (MEMORY, FILESYSTEM, or POSTGRES)
config: Optional explicit config. If not provided, reads from ConfigManager.
Prefer passing explicitly from composition roots.
Returns:
Tuple of (engine, session_maker)
"""
# Prefer explicit parameter; fall back to ConfigManager for backwards compatibility
if config is None:
config = ConfigManager().config
db_url = DatabaseType.get_db_url(db_path, db_type, config)
logger.debug(f"Creating engine for db_url: {db_url}")
# Delegate to backend-specific engine creation
# Check explicit POSTGRES type first, then config setting
if db_type == DatabaseType.POSTGRES or config.database_backend == DatabaseBackend.POSTGRES:
engine = _create_postgres_engine(db_url, config)
else:
engine = _create_sqlite_engine(db_url, db_type)
session_maker = async_sessionmaker(engine, expire_on_commit=False)
return engine, session_maker
@@ -154,17 +278,29 @@ async def get_or_create_db(
db_path: Path,
db_type: DatabaseType = DatabaseType.FILESYSTEM,
ensure_migrations: bool = True,
config: Optional[BasicMemoryConfig] = None,
) -> tuple[AsyncEngine, async_sessionmaker[AsyncSession]]: # pragma: no cover
"""Get or create database engine and session maker."""
"""Get or create database engine and session maker.
Args:
db_path: Path to database file
db_type: Type of database
ensure_migrations: Whether to run migrations
config: Optional explicit config. If not provided, reads from ConfigManager.
Prefer passing explicitly from composition roots.
"""
global _engine, _session_maker
# Prefer explicit parameter; fall back to ConfigManager for backwards compatibility
if config is None:
config = ConfigManager().config
if _engine is None:
_engine, _session_maker = _create_engine_and_session(db_path, db_type)
_engine, _session_maker = _create_engine_and_session(db_path, db_type, config)
# Run migrations automatically unless explicitly disabled
if ensure_migrations:
app_config = ConfigManager().config
await run_migrations(app_config, db_type)
await run_migrations(config, db_type)
# These checks should never fail since we just created the engine and session maker
# if they were None, but we'll check anyway for the type checker
@@ -181,70 +317,37 @@ async def get_or_create_db(
async def shutdown_db() -> None: # pragma: no cover
"""Clean up database connections."""
global _engine, _session_maker, _migrations_completed
global _engine, _session_maker
if _engine:
await _engine.dispose()
_engine = None
_session_maker = None
_migrations_completed = False
@asynccontextmanager
async def engine_session_factory(
db_path: Path,
db_type: DatabaseType = DatabaseType.MEMORY,
config: Optional[BasicMemoryConfig] = None,
) -> AsyncGenerator[tuple[AsyncEngine, async_sessionmaker[AsyncSession]], None]:
"""Create engine and session factory.
Note: This is primarily used for testing where we want a fresh database
for each test. For production use, use get_or_create_db() instead.
Args:
db_path: Path to database file
db_type: Type of database
config: Optional explicit config. If not provided, reads from ConfigManager.
"""
global _engine, _session_maker, _migrations_completed
global _engine, _session_maker
db_url = DatabaseType.get_db_url(db_path, db_type)
logger.debug(f"Creating engine for db_url: {db_url}")
# Configure connection args with Windows-specific settings
connect_args: dict[str, bool | float | None] = {"check_same_thread": False}
# Add Windows-specific parameters to improve reliability
if os.name == "nt": # Windows
connect_args.update(
{
"timeout": 30.0, # Increase timeout to 30 seconds for Windows
"isolation_level": None, # Use autocommit mode
}
)
# Use NullPool for Windows filesystem databases to avoid connection pooling issues
# Important: Do NOT use NullPool for in-memory databases as it will destroy the database
# between connections
if db_type == DatabaseType.FILESYSTEM:
_engine = create_async_engine(
db_url,
connect_args=connect_args,
poolclass=NullPool, # Disable connection pooling on Windows
echo=False,
)
else:
# In-memory databases need connection pooling to maintain state
_engine = create_async_engine(db_url, connect_args=connect_args)
else:
_engine = create_async_engine(db_url, connect_args=connect_args)
# Enable WAL mode for better concurrency and reliability
# Note: WAL mode is not supported for in-memory databases
enable_wal = db_type != DatabaseType.MEMORY
@event.listens_for(_engine.sync_engine, "connect")
def enable_wal_mode(dbapi_conn, connection_record):
"""Enable WAL mode on each connection."""
_configure_sqlite_connection(dbapi_conn, enable_wal=enable_wal)
# Use the same helper function as production code
_engine, _session_maker = _create_engine_and_session(db_path, db_type, config)
try:
_session_maker = async_sessionmaker(_engine, expire_on_commit=False)
# Verify that engine and session maker are initialized
if _engine is None: # pragma: no cover
logger.error("Database engine is None in engine_session_factory")
@@ -260,20 +363,16 @@ async def engine_session_factory(
await _engine.dispose()
_engine = None
_session_maker = None
_migrations_completed = False
async def run_migrations(
app_config: BasicMemoryConfig, database_type=DatabaseType.FILESYSTEM, force: bool = False
app_config: BasicMemoryConfig, database_type=DatabaseType.FILESYSTEM
): # pragma: no cover
"""Run any pending alembic migrations."""
global _migrations_completed
# Skip if migrations already completed unless forced
if _migrations_completed and not force:
logger.debug("Migrations already completed in this session, skipping")
return
"""Run any pending alembic migrations.
Note: Alembic tracks which migrations have been applied via the alembic_version table,
so it's safe to call this multiple times - it will only run pending migrations.
"""
logger.info("Running database migrations...")
try:
# Get the absolute path to the alembic directory relative to this file
@@ -288,9 +387,11 @@ async def run_migrations(
)
config.set_main_option("timezone", "UTC")
config.set_main_option("revision_environment", "false")
config.set_main_option(
"sqlalchemy.url", DatabaseType.get_db_url(app_config.database_path, database_type)
)
# Get the correct database URL based on backend configuration
# No URL conversion needed - env.py now handles both async and sync engines
db_url = DatabaseType.get_db_url(app_config.database_path, database_type, app_config)
config.set_main_option("sqlalchemy.url", db_url)
command.upgrade(config, "head")
logger.info("Migrations completed successfully")
@@ -301,12 +402,17 @@ async def run_migrations(
else:
session_maker = _session_maker
# initialize the search Index schema
# the project_id is not used for init_search_index, so we pass a dummy value
await SearchRepository(session_maker, 1).init_search_index()
# Mark migrations as completed
_migrations_completed = True
# Initialize the search index schema
# For SQLite: Create FTS5 virtual table
# For Postgres: No-op (tsvector column added by migrations)
# The project_id is not used for init_search_index, so we pass a dummy value
if (
database_type == DatabaseType.POSTGRES
or app_config.database_backend == DatabaseBackend.POSTGRES
):
await PostgresSearchRepository(session_maker, 1).init_search_index()
else:
await SQLiteSearchRepository(session_maker, 1).init_search_index()
except Exception as e: # pragma: no cover
logger.error(f"Error running migrations: {e}")
raise
+12 -407
View File
@@ -1,411 +1,16 @@
"""Dependency injection functions for basic-memory services."""
from typing import Annotated
from loguru import logger
from fastapi import Depends, HTTPException, Path, status, Request
from sqlalchemy.ext.asyncio import (
AsyncSession,
AsyncEngine,
async_sessionmaker,
)
import pathlib
from basic_memory import db
from basic_memory.config import ProjectConfig, BasicMemoryConfig, ConfigManager
from basic_memory.importers import (
ChatGPTImporter,
ClaudeConversationsImporter,
ClaudeProjectsImporter,
MemoryJsonImporter,
)
from basic_memory.markdown import EntityParser
from basic_memory.markdown.markdown_processor import MarkdownProcessor
from basic_memory.repository.entity_repository import EntityRepository
from basic_memory.repository.observation_repository import ObservationRepository
from basic_memory.repository.project_repository import ProjectRepository
from basic_memory.repository.relation_repository import RelationRepository
from basic_memory.repository.search_repository import SearchRepository
from basic_memory.services import EntityService, ProjectService
from basic_memory.services.context_service import ContextService
from basic_memory.services.directory_service import DirectoryService
from basic_memory.services.file_service import FileService
from basic_memory.services.link_resolver import LinkResolver
from basic_memory.services.search_service import SearchService
from basic_memory.sync import SyncService
from basic_memory.utils import generate_permalink
def get_app_config() -> BasicMemoryConfig: # pragma: no cover
app_config = ConfigManager().config
return app_config
AppConfigDep = Annotated[BasicMemoryConfig, Depends(get_app_config)] # pragma: no cover
## project
async def get_project_config(
project: "ProjectPathDep", project_repository: "ProjectRepositoryDep"
) -> ProjectConfig: # pragma: no cover
"""Get the current project referenced from request state.
Args:
request: The current request object
project_repository: Repository for project operations
Returns:
The resolved project config
Raises:
HTTPException: If project is not found
"""
# Convert project name to permalink for lookup
project_permalink = generate_permalink(str(project))
project_obj = await project_repository.get_by_permalink(project_permalink)
if project_obj:
return ProjectConfig(name=project_obj.name, home=pathlib.Path(project_obj.path))
# Not found
raise HTTPException( # pragma: no cover
status_code=status.HTTP_404_NOT_FOUND, detail=f"Project '{project}' not found."
)
ProjectConfigDep = Annotated[ProjectConfig, Depends(get_project_config)] # pragma: no cover
## sqlalchemy
async def get_engine_factory(
request: Request,
) -> tuple[AsyncEngine, async_sessionmaker[AsyncSession]]: # pragma: no cover
"""Get cached engine and session maker from app state.
For API requests, returns cached connections from app.state for optimal performance.
For non-API contexts (CLI), falls back to direct database connection.
"""
# Try to get cached connections from app state (API context)
if (
hasattr(request, "app")
and hasattr(request.app.state, "engine")
and hasattr(request.app.state, "session_maker")
):
return request.app.state.engine, request.app.state.session_maker
# Fallback for non-API contexts (CLI)
logger.debug("Using fallback database connection for non-API context")
app_config = get_app_config()
engine, session_maker = await db.get_or_create_db(app_config.database_path)
return engine, session_maker
EngineFactoryDep = Annotated[
tuple[AsyncEngine, async_sessionmaker[AsyncSession]], Depends(get_engine_factory)
]
async def get_session_maker(engine_factory: EngineFactoryDep) -> async_sessionmaker[AsyncSession]:
"""Get session maker."""
_, session_maker = engine_factory
return session_maker
SessionMakerDep = Annotated[async_sessionmaker, Depends(get_session_maker)]
## repositories
async def get_project_repository(
session_maker: SessionMakerDep,
) -> ProjectRepository:
"""Get the project repository."""
return ProjectRepository(session_maker)
ProjectRepositoryDep = Annotated[ProjectRepository, Depends(get_project_repository)]
ProjectPathDep = Annotated[str, Path()] # Use Path dependency to extract from URL
async def get_project_id(
project_repository: ProjectRepositoryDep,
project: ProjectPathDep,
) -> int:
"""Get the current project ID from request state.
When using sub-applications with /{project} mounting, the project value
is stored in request.state by middleware.
Args:
request: The current request object
project_repository: Repository for project operations
Returns:
The resolved project ID
Raises:
HTTPException: If project is not found
"""
# Convert project name to permalink for lookup
project_permalink = generate_permalink(str(project))
project_obj = await project_repository.get_by_permalink(project_permalink)
if project_obj:
return project_obj.id
# Try by name if permalink lookup fails
project_obj = await project_repository.get_by_name(str(project)) # pragma: no cover
if project_obj: # pragma: no cover
return project_obj.id
# Not found
raise HTTPException( # pragma: no cover
status_code=status.HTTP_404_NOT_FOUND, detail=f"Project '{project}' not found."
)
"""Dependency injection functions for basic-memory services.
DEPRECATED: This module is a backwards-compatibility shim.
Import from basic_memory.deps package submodules instead:
- basic_memory.deps.config for configuration
- basic_memory.deps.db for database/session
- basic_memory.deps.projects for project resolution
- basic_memory.deps.repositories for data access
- basic_memory.deps.services for business logic
- basic_memory.deps.importers for import functionality
This file will be removed once all callers are migrated.
"""
The project_id dependency is used in the following:
- EntityRepository
- ObservationRepository
- RelationRepository
- SearchRepository
- ProjectInfoRepository
"""
ProjectIdDep = Annotated[int, Depends(get_project_id)]
async def get_entity_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
) -> EntityRepository:
"""Create an EntityRepository instance for the current project."""
return EntityRepository(session_maker, project_id=project_id)
EntityRepositoryDep = Annotated[EntityRepository, Depends(get_entity_repository)]
async def get_observation_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
) -> ObservationRepository:
"""Create an ObservationRepository instance for the current project."""
return ObservationRepository(session_maker, project_id=project_id)
ObservationRepositoryDep = Annotated[ObservationRepository, Depends(get_observation_repository)]
async def get_relation_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
) -> RelationRepository:
"""Create a RelationRepository instance for the current project."""
return RelationRepository(session_maker, project_id=project_id)
RelationRepositoryDep = Annotated[RelationRepository, Depends(get_relation_repository)]
async def get_search_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
) -> SearchRepository:
"""Create a SearchRepository instance for the current project."""
return SearchRepository(session_maker, project_id=project_id)
SearchRepositoryDep = Annotated[SearchRepository, Depends(get_search_repository)]
# ProjectInfoRepository is deprecated and will be removed in a future version.
# Use ProjectRepository instead, which has the same functionality plus more project-specific operations.
## services
async def get_entity_parser(project_config: ProjectConfigDep) -> EntityParser:
return EntityParser(project_config.home)
EntityParserDep = Annotated["EntityParser", Depends(get_entity_parser)]
async def get_markdown_processor(entity_parser: EntityParserDep) -> MarkdownProcessor:
return MarkdownProcessor(entity_parser)
MarkdownProcessorDep = Annotated[MarkdownProcessor, Depends(get_markdown_processor)]
async def get_file_service(
project_config: ProjectConfigDep, markdown_processor: MarkdownProcessorDep
) -> FileService:
logger.debug(
f"Creating FileService for project: {project_config.name}, base_path: {project_config.home}"
)
file_service = FileService(project_config.home, markdown_processor)
logger.debug(f"Created FileService for project: {file_service} ")
return file_service
FileServiceDep = Annotated[FileService, Depends(get_file_service)]
async def get_entity_service(
entity_repository: EntityRepositoryDep,
observation_repository: ObservationRepositoryDep,
relation_repository: RelationRepositoryDep,
entity_parser: EntityParserDep,
file_service: FileServiceDep,
link_resolver: "LinkResolverDep",
app_config: AppConfigDep,
) -> EntityService:
"""Create EntityService with repository."""
return EntityService(
entity_repository=entity_repository,
observation_repository=observation_repository,
relation_repository=relation_repository,
entity_parser=entity_parser,
file_service=file_service,
link_resolver=link_resolver,
app_config=app_config,
)
EntityServiceDep = Annotated[EntityService, Depends(get_entity_service)]
async def get_search_service(
search_repository: SearchRepositoryDep,
entity_repository: EntityRepositoryDep,
file_service: FileServiceDep,
) -> SearchService:
"""Create SearchService with dependencies."""
return SearchService(search_repository, entity_repository, file_service)
SearchServiceDep = Annotated[SearchService, Depends(get_search_service)]
async def get_link_resolver(
entity_repository: EntityRepositoryDep, search_service: SearchServiceDep
) -> LinkResolver:
return LinkResolver(entity_repository=entity_repository, search_service=search_service)
LinkResolverDep = Annotated[LinkResolver, Depends(get_link_resolver)]
async def get_context_service(
search_repository: SearchRepositoryDep,
entity_repository: EntityRepositoryDep,
observation_repository: ObservationRepositoryDep,
) -> ContextService:
return ContextService(
search_repository=search_repository,
entity_repository=entity_repository,
observation_repository=observation_repository,
)
ContextServiceDep = Annotated[ContextService, Depends(get_context_service)]
async def get_sync_service(
app_config: AppConfigDep,
entity_service: EntityServiceDep,
entity_parser: EntityParserDep,
entity_repository: EntityRepositoryDep,
relation_repository: RelationRepositoryDep,
project_repository: ProjectRepositoryDep,
search_service: SearchServiceDep,
file_service: FileServiceDep,
) -> SyncService: # pragma: no cover
"""
:rtype: object
"""
return SyncService(
app_config=app_config,
entity_service=entity_service,
entity_parser=entity_parser,
entity_repository=entity_repository,
relation_repository=relation_repository,
project_repository=project_repository,
search_service=search_service,
file_service=file_service,
)
SyncServiceDep = Annotated[SyncService, Depends(get_sync_service)]
async def get_project_service(
project_repository: ProjectRepositoryDep,
) -> ProjectService:
"""Create ProjectService with repository."""
return ProjectService(repository=project_repository)
ProjectServiceDep = Annotated[ProjectService, Depends(get_project_service)]
async def get_directory_service(
entity_repository: EntityRepositoryDep,
) -> DirectoryService:
"""Create DirectoryService with dependencies."""
return DirectoryService(
entity_repository=entity_repository,
)
DirectoryServiceDep = Annotated[DirectoryService, Depends(get_directory_service)]
# Import
async def get_chatgpt_importer(
project_config: ProjectConfigDep, markdown_processor: MarkdownProcessorDep
) -> ChatGPTImporter:
"""Create ChatGPTImporter with dependencies."""
return ChatGPTImporter(project_config.home, markdown_processor)
ChatGPTImporterDep = Annotated[ChatGPTImporter, Depends(get_chatgpt_importer)]
async def get_claude_conversations_importer(
project_config: ProjectConfigDep, markdown_processor: MarkdownProcessorDep
) -> ClaudeConversationsImporter:
"""Create ChatGPTImporter with dependencies."""
return ClaudeConversationsImporter(project_config.home, markdown_processor)
ClaudeConversationsImporterDep = Annotated[
ClaudeConversationsImporter, Depends(get_claude_conversations_importer)
]
async def get_claude_projects_importer(
project_config: ProjectConfigDep, markdown_processor: MarkdownProcessorDep
) -> ClaudeProjectsImporter:
"""Create ChatGPTImporter with dependencies."""
return ClaudeProjectsImporter(project_config.home, markdown_processor)
ClaudeProjectsImporterDep = Annotated[ClaudeProjectsImporter, Depends(get_claude_projects_importer)]
async def get_memory_json_importer(
project_config: ProjectConfigDep, markdown_processor: MarkdownProcessorDep
) -> MemoryJsonImporter:
"""Create ChatGPTImporter with dependencies."""
return MemoryJsonImporter(project_config.home, markdown_processor)
MemoryJsonImporterDep = Annotated[MemoryJsonImporter, Depends(get_memory_json_importer)]
# Re-export everything from the deps package for backwards compatibility
from basic_memory.deps import * # noqa: F401, F403 # pragma: no cover
+293
View File
@@ -0,0 +1,293 @@
"""Dependency injection for basic-memory.
This package provides FastAPI dependencies organized by feature:
- config: Application configuration
- db: Database/session management
- projects: Project resolution and config
- repositories: Data access layer
- services: Business logic layer
- importers: Import functionality
For backwards compatibility, all dependencies are re-exported from this module.
New code should import from specific submodules to reduce coupling.
"""
# Re-export everything for backwards compatibility
# Eventually, callers should import from specific submodules
from basic_memory.deps.config import (
get_app_config,
AppConfigDep,
)
from basic_memory.deps.db import (
get_engine_factory,
EngineFactoryDep,
get_session_maker,
SessionMakerDep,
)
from basic_memory.deps.projects import (
get_project_repository,
ProjectRepositoryDep,
ProjectPathDep,
get_project_id,
ProjectIdDep,
get_project_config,
ProjectConfigDep,
validate_project_id,
ProjectIdPathDep,
get_project_config_v2,
ProjectConfigV2Dep,
validate_project_external_id,
ProjectExternalIdPathDep,
get_project_config_v2_external,
ProjectConfigV2ExternalDep,
)
from basic_memory.deps.repositories import (
get_entity_repository,
EntityRepositoryDep,
get_entity_repository_v2,
EntityRepositoryV2Dep,
get_entity_repository_v2_external,
EntityRepositoryV2ExternalDep,
get_observation_repository,
ObservationRepositoryDep,
get_observation_repository_v2,
ObservationRepositoryV2Dep,
get_observation_repository_v2_external,
ObservationRepositoryV2ExternalDep,
get_relation_repository,
RelationRepositoryDep,
get_relation_repository_v2,
RelationRepositoryV2Dep,
get_relation_repository_v2_external,
RelationRepositoryV2ExternalDep,
get_search_repository,
SearchRepositoryDep,
get_search_repository_v2,
SearchRepositoryV2Dep,
get_search_repository_v2_external,
SearchRepositoryV2ExternalDep,
)
from basic_memory.deps.services import (
get_entity_parser,
EntityParserDep,
get_entity_parser_v2,
EntityParserV2Dep,
get_entity_parser_v2_external,
EntityParserV2ExternalDep,
get_markdown_processor,
MarkdownProcessorDep,
get_markdown_processor_v2,
MarkdownProcessorV2Dep,
get_markdown_processor_v2_external,
MarkdownProcessorV2ExternalDep,
get_file_service,
FileServiceDep,
get_file_service_v2,
FileServiceV2Dep,
get_file_service_v2_external,
FileServiceV2ExternalDep,
get_search_service,
SearchServiceDep,
get_search_service_v2,
SearchServiceV2Dep,
get_search_service_v2_external,
SearchServiceV2ExternalDep,
get_link_resolver,
LinkResolverDep,
get_link_resolver_v2,
LinkResolverV2Dep,
get_link_resolver_v2_external,
LinkResolverV2ExternalDep,
get_entity_service,
EntityServiceDep,
get_entity_service_v2,
EntityServiceV2Dep,
get_entity_service_v2_external,
EntityServiceV2ExternalDep,
get_context_service,
ContextServiceDep,
get_context_service_v2,
ContextServiceV2Dep,
get_context_service_v2_external,
ContextServiceV2ExternalDep,
get_sync_service,
SyncServiceDep,
get_sync_service_v2,
SyncServiceV2Dep,
get_sync_service_v2_external,
SyncServiceV2ExternalDep,
get_project_service,
ProjectServiceDep,
get_directory_service,
DirectoryServiceDep,
get_directory_service_v2,
DirectoryServiceV2Dep,
get_directory_service_v2_external,
DirectoryServiceV2ExternalDep,
)
from basic_memory.deps.importers import (
get_chatgpt_importer,
ChatGPTImporterDep,
get_chatgpt_importer_v2,
ChatGPTImporterV2Dep,
get_chatgpt_importer_v2_external,
ChatGPTImporterV2ExternalDep,
get_claude_conversations_importer,
ClaudeConversationsImporterDep,
get_claude_conversations_importer_v2,
ClaudeConversationsImporterV2Dep,
get_claude_conversations_importer_v2_external,
ClaudeConversationsImporterV2ExternalDep,
get_claude_projects_importer,
ClaudeProjectsImporterDep,
get_claude_projects_importer_v2,
ClaudeProjectsImporterV2Dep,
get_claude_projects_importer_v2_external,
ClaudeProjectsImporterV2ExternalDep,
get_memory_json_importer,
MemoryJsonImporterDep,
get_memory_json_importer_v2,
MemoryJsonImporterV2Dep,
get_memory_json_importer_v2_external,
MemoryJsonImporterV2ExternalDep,
)
__all__ = [
# Config
"get_app_config",
"AppConfigDep",
# Database
"get_engine_factory",
"EngineFactoryDep",
"get_session_maker",
"SessionMakerDep",
# Projects
"get_project_repository",
"ProjectRepositoryDep",
"ProjectPathDep",
"get_project_id",
"ProjectIdDep",
"get_project_config",
"ProjectConfigDep",
"validate_project_id",
"ProjectIdPathDep",
"get_project_config_v2",
"ProjectConfigV2Dep",
"validate_project_external_id",
"ProjectExternalIdPathDep",
"get_project_config_v2_external",
"ProjectConfigV2ExternalDep",
# Repositories
"get_entity_repository",
"EntityRepositoryDep",
"get_entity_repository_v2",
"EntityRepositoryV2Dep",
"get_entity_repository_v2_external",
"EntityRepositoryV2ExternalDep",
"get_observation_repository",
"ObservationRepositoryDep",
"get_observation_repository_v2",
"ObservationRepositoryV2Dep",
"get_observation_repository_v2_external",
"ObservationRepositoryV2ExternalDep",
"get_relation_repository",
"RelationRepositoryDep",
"get_relation_repository_v2",
"RelationRepositoryV2Dep",
"get_relation_repository_v2_external",
"RelationRepositoryV2ExternalDep",
"get_search_repository",
"SearchRepositoryDep",
"get_search_repository_v2",
"SearchRepositoryV2Dep",
"get_search_repository_v2_external",
"SearchRepositoryV2ExternalDep",
# Services
"get_entity_parser",
"EntityParserDep",
"get_entity_parser_v2",
"EntityParserV2Dep",
"get_entity_parser_v2_external",
"EntityParserV2ExternalDep",
"get_markdown_processor",
"MarkdownProcessorDep",
"get_markdown_processor_v2",
"MarkdownProcessorV2Dep",
"get_markdown_processor_v2_external",
"MarkdownProcessorV2ExternalDep",
"get_file_service",
"FileServiceDep",
"get_file_service_v2",
"FileServiceV2Dep",
"get_file_service_v2_external",
"FileServiceV2ExternalDep",
"get_search_service",
"SearchServiceDep",
"get_search_service_v2",
"SearchServiceV2Dep",
"get_search_service_v2_external",
"SearchServiceV2ExternalDep",
"get_link_resolver",
"LinkResolverDep",
"get_link_resolver_v2",
"LinkResolverV2Dep",
"get_link_resolver_v2_external",
"LinkResolverV2ExternalDep",
"get_entity_service",
"EntityServiceDep",
"get_entity_service_v2",
"EntityServiceV2Dep",
"get_entity_service_v2_external",
"EntityServiceV2ExternalDep",
"get_context_service",
"ContextServiceDep",
"get_context_service_v2",
"ContextServiceV2Dep",
"get_context_service_v2_external",
"ContextServiceV2ExternalDep",
"get_sync_service",
"SyncServiceDep",
"get_sync_service_v2",
"SyncServiceV2Dep",
"get_sync_service_v2_external",
"SyncServiceV2ExternalDep",
"get_project_service",
"ProjectServiceDep",
"get_directory_service",
"DirectoryServiceDep",
"get_directory_service_v2",
"DirectoryServiceV2Dep",
"get_directory_service_v2_external",
"DirectoryServiceV2ExternalDep",
# Importers
"get_chatgpt_importer",
"ChatGPTImporterDep",
"get_chatgpt_importer_v2",
"ChatGPTImporterV2Dep",
"get_chatgpt_importer_v2_external",
"ChatGPTImporterV2ExternalDep",
"get_claude_conversations_importer",
"ClaudeConversationsImporterDep",
"get_claude_conversations_importer_v2",
"ClaudeConversationsImporterV2Dep",
"get_claude_conversations_importer_v2_external",
"ClaudeConversationsImporterV2ExternalDep",
"get_claude_projects_importer",
"ClaudeProjectsImporterDep",
"get_claude_projects_importer_v2",
"ClaudeProjectsImporterV2Dep",
"get_claude_projects_importer_v2_external",
"ClaudeProjectsImporterV2ExternalDep",
"get_memory_json_importer",
"MemoryJsonImporterDep",
"get_memory_json_importer_v2",
"MemoryJsonImporterV2Dep",
"get_memory_json_importer_v2_external",
"MemoryJsonImporterV2ExternalDep",
]
+26
View File
@@ -0,0 +1,26 @@
"""Configuration dependency injection for basic-memory.
This module provides configuration-related dependencies.
Note: Long-term goal is to minimize direct ConfigManager access
and inject config from composition roots instead.
"""
from typing import Annotated
from fastapi import Depends
from basic_memory.config import BasicMemoryConfig, ConfigManager
def get_app_config() -> BasicMemoryConfig: # pragma: no cover
"""Get the application configuration.
Note: This is a transitional dependency. The goal is for composition roots
to read ConfigManager and inject config explicitly. During migration,
this provides the same behavior as before.
"""
app_config = ConfigManager().config
return app_config
AppConfigDep = Annotated[BasicMemoryConfig, Depends(get_app_config)]
+56
View File
@@ -0,0 +1,56 @@
"""Database dependency injection for basic-memory.
This module provides database-related dependencies:
- Engine and session maker factories
- Session dependencies for request handling
"""
from typing import Annotated
from fastapi import Depends, Request
from loguru import logger
from sqlalchemy.ext.asyncio import (
AsyncEngine,
AsyncSession,
async_sessionmaker,
)
from basic_memory import db
from basic_memory.deps.config import get_app_config
async def get_engine_factory(
request: Request,
) -> tuple[AsyncEngine, async_sessionmaker[AsyncSession]]: # pragma: no cover
"""Get cached engine and session maker from app state.
For API requests, returns cached connections from app.state for optimal performance.
For non-API contexts (CLI), falls back to direct database connection.
"""
# Try to get cached connections from app state (API context)
if (
hasattr(request, "app")
and hasattr(request.app.state, "engine")
and hasattr(request.app.state, "session_maker")
):
return request.app.state.engine, request.app.state.session_maker
# Fallback for non-API contexts (CLI)
logger.debug("Using fallback database connection for non-API context")
app_config = get_app_config()
engine, session_maker = await db.get_or_create_db(app_config.database_path)
return engine, session_maker
EngineFactoryDep = Annotated[
tuple[AsyncEngine, async_sessionmaker[AsyncSession]], Depends(get_engine_factory)
]
async def get_session_maker(engine_factory: EngineFactoryDep) -> async_sessionmaker[AsyncSession]:
"""Get session maker."""
_, session_maker = engine_factory
return session_maker
SessionMakerDep = Annotated[async_sessionmaker, Depends(get_session_maker)]
+200
View File
@@ -0,0 +1,200 @@
"""Importer dependency injection for basic-memory.
This module provides importer dependencies:
- ChatGPTImporter
- ClaudeConversationsImporter
- ClaudeProjectsImporter
- MemoryJsonImporter
"""
from typing import Annotated
from fastapi import Depends
from basic_memory.deps.projects import (
ProjectConfigDep,
ProjectConfigV2Dep,
ProjectConfigV2ExternalDep,
)
from basic_memory.deps.services import (
FileServiceDep,
FileServiceV2Dep,
FileServiceV2ExternalDep,
MarkdownProcessorDep,
MarkdownProcessorV2Dep,
MarkdownProcessorV2ExternalDep,
)
from basic_memory.importers import (
ChatGPTImporter,
ClaudeConversationsImporter,
ClaudeProjectsImporter,
MemoryJsonImporter,
)
# --- ChatGPT Importer ---
async def get_chatgpt_importer(
project_config: ProjectConfigDep,
markdown_processor: MarkdownProcessorDep,
file_service: FileServiceDep,
) -> ChatGPTImporter:
"""Create ChatGPTImporter with dependencies."""
return ChatGPTImporter(project_config.home, markdown_processor, file_service)
ChatGPTImporterDep = Annotated[ChatGPTImporter, Depends(get_chatgpt_importer)]
async def get_chatgpt_importer_v2( # pragma: no cover
project_config: ProjectConfigV2Dep,
markdown_processor: MarkdownProcessorV2Dep,
file_service: FileServiceV2Dep,
) -> ChatGPTImporter:
"""Create ChatGPTImporter with v2 dependencies."""
return ChatGPTImporter(project_config.home, markdown_processor, file_service)
ChatGPTImporterV2Dep = Annotated[ChatGPTImporter, Depends(get_chatgpt_importer_v2)]
async def get_chatgpt_importer_v2_external(
project_config: ProjectConfigV2ExternalDep,
markdown_processor: MarkdownProcessorV2ExternalDep,
file_service: FileServiceV2ExternalDep,
) -> ChatGPTImporter:
"""Create ChatGPTImporter with v2 external_id dependencies."""
return ChatGPTImporter(project_config.home, markdown_processor, file_service)
ChatGPTImporterV2ExternalDep = Annotated[ChatGPTImporter, Depends(get_chatgpt_importer_v2_external)]
# --- Claude Conversations Importer ---
async def get_claude_conversations_importer(
project_config: ProjectConfigDep,
markdown_processor: MarkdownProcessorDep,
file_service: FileServiceDep,
) -> ClaudeConversationsImporter:
"""Create ClaudeConversationsImporter with dependencies."""
return ClaudeConversationsImporter(project_config.home, markdown_processor, file_service)
ClaudeConversationsImporterDep = Annotated[
ClaudeConversationsImporter, Depends(get_claude_conversations_importer)
]
async def get_claude_conversations_importer_v2( # pragma: no cover
project_config: ProjectConfigV2Dep,
markdown_processor: MarkdownProcessorV2Dep,
file_service: FileServiceV2Dep,
) -> ClaudeConversationsImporter:
"""Create ClaudeConversationsImporter with v2 dependencies."""
return ClaudeConversationsImporter(project_config.home, markdown_processor, file_service)
ClaudeConversationsImporterV2Dep = Annotated[
ClaudeConversationsImporter, Depends(get_claude_conversations_importer_v2)
]
async def get_claude_conversations_importer_v2_external(
project_config: ProjectConfigV2ExternalDep,
markdown_processor: MarkdownProcessorV2ExternalDep,
file_service: FileServiceV2ExternalDep,
) -> ClaudeConversationsImporter:
"""Create ClaudeConversationsImporter with v2 external_id dependencies."""
return ClaudeConversationsImporter(project_config.home, markdown_processor, file_service)
ClaudeConversationsImporterV2ExternalDep = Annotated[
ClaudeConversationsImporter, Depends(get_claude_conversations_importer_v2_external)
]
# --- Claude Projects Importer ---
async def get_claude_projects_importer(
project_config: ProjectConfigDep,
markdown_processor: MarkdownProcessorDep,
file_service: FileServiceDep,
) -> ClaudeProjectsImporter:
"""Create ClaudeProjectsImporter with dependencies."""
return ClaudeProjectsImporter(project_config.home, markdown_processor, file_service)
ClaudeProjectsImporterDep = Annotated[ClaudeProjectsImporter, Depends(get_claude_projects_importer)]
async def get_claude_projects_importer_v2( # pragma: no cover
project_config: ProjectConfigV2Dep,
markdown_processor: MarkdownProcessorV2Dep,
file_service: FileServiceV2Dep,
) -> ClaudeProjectsImporter:
"""Create ClaudeProjectsImporter with v2 dependencies."""
return ClaudeProjectsImporter(project_config.home, markdown_processor, file_service)
ClaudeProjectsImporterV2Dep = Annotated[
ClaudeProjectsImporter, Depends(get_claude_projects_importer_v2)
]
async def get_claude_projects_importer_v2_external(
project_config: ProjectConfigV2ExternalDep,
markdown_processor: MarkdownProcessorV2ExternalDep,
file_service: FileServiceV2ExternalDep,
) -> ClaudeProjectsImporter:
"""Create ClaudeProjectsImporter with v2 external_id dependencies."""
return ClaudeProjectsImporter(project_config.home, markdown_processor, file_service)
ClaudeProjectsImporterV2ExternalDep = Annotated[
ClaudeProjectsImporter, Depends(get_claude_projects_importer_v2_external)
]
# --- Memory JSON Importer ---
async def get_memory_json_importer(
project_config: ProjectConfigDep,
markdown_processor: MarkdownProcessorDep,
file_service: FileServiceDep,
) -> MemoryJsonImporter:
"""Create MemoryJsonImporter with dependencies."""
return MemoryJsonImporter(project_config.home, markdown_processor, file_service)
MemoryJsonImporterDep = Annotated[MemoryJsonImporter, Depends(get_memory_json_importer)]
async def get_memory_json_importer_v2( # pragma: no cover
project_config: ProjectConfigV2Dep,
markdown_processor: MarkdownProcessorV2Dep,
file_service: FileServiceV2Dep,
) -> MemoryJsonImporter:
"""Create MemoryJsonImporter with v2 dependencies."""
return MemoryJsonImporter(project_config.home, markdown_processor, file_service)
MemoryJsonImporterV2Dep = Annotated[MemoryJsonImporter, Depends(get_memory_json_importer_v2)]
async def get_memory_json_importer_v2_external(
project_config: ProjectConfigV2ExternalDep,
markdown_processor: MarkdownProcessorV2ExternalDep,
file_service: FileServiceV2ExternalDep,
) -> MemoryJsonImporter:
"""Create MemoryJsonImporter with v2 external_id dependencies."""
return MemoryJsonImporter(project_config.home, markdown_processor, file_service)
MemoryJsonImporterV2ExternalDep = Annotated[
MemoryJsonImporter, Depends(get_memory_json_importer_v2_external)
]
+238
View File
@@ -0,0 +1,238 @@
"""Project dependency injection for basic-memory.
This module provides project-related dependencies:
- Project path extraction from URL
- Project config resolution
- Project ID validation
- Project repository
"""
import pathlib
from typing import Annotated
from fastapi import Depends, HTTPException, Path, status
from basic_memory.config import ProjectConfig
from basic_memory.deps.db import SessionMakerDep
from basic_memory.repository.project_repository import ProjectRepository
from basic_memory.utils import generate_permalink
# --- Project Repository ---
async def get_project_repository(
session_maker: SessionMakerDep,
) -> ProjectRepository:
"""Get the project repository."""
return ProjectRepository(session_maker)
ProjectRepositoryDep = Annotated[ProjectRepository, Depends(get_project_repository)]
# --- Path Extraction ---
# V1 API: Project name from URL path
ProjectPathDep = Annotated[str, Path()]
# --- Project ID Resolution (V1 API) ---
async def get_project_id(
project_repository: ProjectRepositoryDep,
project: ProjectPathDep,
) -> int:
"""Get the current project ID from request state.
When using sub-applications with /{project} mounting, the project value
is stored in request.state by middleware.
Args:
project_repository: Repository for project operations
project: The project name from URL path
Returns:
The resolved project ID
Raises:
HTTPException: If project is not found
"""
# Convert project name to permalink for lookup
project_permalink = generate_permalink(str(project))
project_obj = await project_repository.get_by_permalink(project_permalink)
if project_obj:
return project_obj.id
# Try by name if permalink lookup fails
project_obj = await project_repository.get_by_name(str(project)) # pragma: no cover
if project_obj: # pragma: no cover
return project_obj.id
# Not found
raise HTTPException( # pragma: no cover
status_code=status.HTTP_404_NOT_FOUND, detail=f"Project '{project}' not found."
)
ProjectIdDep = Annotated[int, Depends(get_project_id)]
# --- Project Config Resolution (V1 API) ---
async def get_project_config(
project: ProjectPathDep, project_repository: ProjectRepositoryDep
) -> ProjectConfig: # pragma: no cover
"""Get the current project referenced from request state.
Args:
project: The project name from URL path
project_repository: Repository for project operations
Returns:
The resolved project config
Raises:
HTTPException: If project is not found
"""
# Convert project name to permalink for lookup
project_permalink = generate_permalink(str(project))
project_obj = await project_repository.get_by_permalink(project_permalink)
if project_obj:
return ProjectConfig(name=project_obj.name, home=pathlib.Path(project_obj.path))
# Not found
raise HTTPException( # pragma: no cover
status_code=status.HTTP_404_NOT_FOUND, detail=f"Project '{project}' not found."
)
ProjectConfigDep = Annotated[ProjectConfig, Depends(get_project_config)]
# --- V2 API: Integer Project ID from Path ---
async def validate_project_id(
project_id: int,
project_repository: ProjectRepositoryDep,
) -> int:
"""Validate that a numeric project ID exists in the database.
This is used for v2 API endpoints that take project IDs as integers in the path.
The project_id parameter will be automatically extracted from the URL path by FastAPI.
Args:
project_id: The numeric project ID from the URL path
project_repository: Repository for project operations
Returns:
The validated project ID
Raises:
HTTPException: If project with that ID is not found
"""
project_obj = await project_repository.get_by_id(project_id)
if not project_obj:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail=f"Project with ID {project_id} not found.",
)
return project_id
ProjectIdPathDep = Annotated[int, Depends(validate_project_id)]
async def get_project_config_v2(
project_id: ProjectIdPathDep, project_repository: ProjectRepositoryDep
) -> ProjectConfig: # pragma: no cover
"""Get the project config for v2 API (uses integer project_id from path).
Args:
project_id: The validated numeric project ID from the URL path
project_repository: Repository for project operations
Returns:
The resolved project config
Raises:
HTTPException: If project is not found
"""
project_obj = await project_repository.get_by_id(project_id)
if project_obj:
return ProjectConfig(name=project_obj.name, home=pathlib.Path(project_obj.path))
# Not found (this should not happen since ProjectIdPathDep already validates existence)
raise HTTPException( # pragma: no cover
status_code=status.HTTP_404_NOT_FOUND, detail=f"Project with ID {project_id} not found."
)
ProjectConfigV2Dep = Annotated[ProjectConfig, Depends(get_project_config_v2)]
# --- V2 API: External UUID Project ID from Path ---
async def validate_project_external_id(
project_id: str,
project_repository: ProjectRepositoryDep,
) -> int:
"""Validate that a project external_id (UUID) exists in the database.
This is used for v2 API endpoints that take project external_ids as strings in the path.
The project_id parameter will be automatically extracted from the URL path by FastAPI.
Args:
project_id: The external UUID from the URL path (named project_id for URL consistency)
project_repository: Repository for project operations
Returns:
The internal numeric project ID (for use by repositories)
Raises:
HTTPException: If project with that external_id is not found
"""
project_obj = await project_repository.get_by_external_id(project_id)
if not project_obj:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail=f"Project with external_id '{project_id}' not found.",
)
return project_obj.id
ProjectExternalIdPathDep = Annotated[int, Depends(validate_project_external_id)]
async def get_project_config_v2_external(
project_id: ProjectExternalIdPathDep, project_repository: ProjectRepositoryDep
) -> ProjectConfig: # pragma: no cover
"""Get the project config for v2 API (uses external_id UUID from path).
Args:
project_id: The internal project ID resolved from external_id
project_repository: Repository for project operations
Returns:
The resolved project config
Raises:
HTTPException: If project is not found
"""
project_obj = await project_repository.get_by_id(project_id)
if project_obj:
return ProjectConfig(name=project_obj.name, home=pathlib.Path(project_obj.path))
# Not found (this should not happen since ProjectExternalIdPathDep already validates)
raise HTTPException( # pragma: no cover
status_code=status.HTTP_404_NOT_FOUND, detail=f"Project with ID {project_id} not found."
)
ProjectConfigV2ExternalDep = Annotated[
ProjectConfig, Depends(get_project_config_v2_external)
]
+179
View File
@@ -0,0 +1,179 @@
"""Repository dependency injection for basic-memory.
This module provides repository dependencies:
- EntityRepository
- ObservationRepository
- RelationRepository
- SearchRepository
Each repository is scoped to a project ID from the request.
"""
from typing import Annotated
from fastapi import Depends
from basic_memory.deps.db import SessionMakerDep
from basic_memory.deps.projects import (
ProjectIdDep,
ProjectIdPathDep,
ProjectExternalIdPathDep,
)
from basic_memory.repository.entity_repository import EntityRepository
from basic_memory.repository.observation_repository import ObservationRepository
from basic_memory.repository.relation_repository import RelationRepository
from basic_memory.repository.search_repository import SearchRepository, create_search_repository
# --- Entity Repository ---
async def get_entity_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
) -> EntityRepository:
"""Create an EntityRepository instance for the current project."""
return EntityRepository(session_maker, project_id=project_id)
EntityRepositoryDep = Annotated[EntityRepository, Depends(get_entity_repository)]
async def get_entity_repository_v2( # pragma: no cover
session_maker: SessionMakerDep,
project_id: ProjectIdPathDep,
) -> EntityRepository:
"""Create an EntityRepository instance for v2 API (uses integer project_id from path)."""
return EntityRepository(session_maker, project_id=project_id)
EntityRepositoryV2Dep = Annotated[EntityRepository, Depends(get_entity_repository_v2)]
async def get_entity_repository_v2_external(
session_maker: SessionMakerDep,
project_id: ProjectExternalIdPathDep,
) -> EntityRepository:
"""Create an EntityRepository instance for v2 API (uses external_id from path)."""
return EntityRepository(session_maker, project_id=project_id)
EntityRepositoryV2ExternalDep = Annotated[EntityRepository, Depends(get_entity_repository_v2_external)]
# --- Observation Repository ---
async def get_observation_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
) -> ObservationRepository:
"""Create an ObservationRepository instance for the current project."""
return ObservationRepository(session_maker, project_id=project_id)
ObservationRepositoryDep = Annotated[ObservationRepository, Depends(get_observation_repository)]
async def get_observation_repository_v2( # pragma: no cover
session_maker: SessionMakerDep,
project_id: ProjectIdPathDep,
) -> ObservationRepository:
"""Create an ObservationRepository instance for v2 API."""
return ObservationRepository(session_maker, project_id=project_id)
ObservationRepositoryV2Dep = Annotated[
ObservationRepository, Depends(get_observation_repository_v2)
]
async def get_observation_repository_v2_external(
session_maker: SessionMakerDep,
project_id: ProjectExternalIdPathDep,
) -> ObservationRepository:
"""Create an ObservationRepository instance for v2 API (uses external_id)."""
return ObservationRepository(session_maker, project_id=project_id)
ObservationRepositoryV2ExternalDep = Annotated[
ObservationRepository, Depends(get_observation_repository_v2_external)
]
# --- Relation Repository ---
async def get_relation_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
) -> RelationRepository:
"""Create a RelationRepository instance for the current project."""
return RelationRepository(session_maker, project_id=project_id)
RelationRepositoryDep = Annotated[RelationRepository, Depends(get_relation_repository)]
async def get_relation_repository_v2( # pragma: no cover
session_maker: SessionMakerDep,
project_id: ProjectIdPathDep,
) -> RelationRepository:
"""Create a RelationRepository instance for v2 API."""
return RelationRepository(session_maker, project_id=project_id)
RelationRepositoryV2Dep = Annotated[RelationRepository, Depends(get_relation_repository_v2)]
async def get_relation_repository_v2_external(
session_maker: SessionMakerDep,
project_id: ProjectExternalIdPathDep,
) -> RelationRepository:
"""Create a RelationRepository instance for v2 API (uses external_id)."""
return RelationRepository(session_maker, project_id=project_id)
RelationRepositoryV2ExternalDep = Annotated[
RelationRepository, Depends(get_relation_repository_v2_external)
]
# --- Search Repository ---
async def get_search_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
) -> SearchRepository:
"""Create a backend-specific SearchRepository instance for the current project.
Uses factory function to return SQLiteSearchRepository or PostgresSearchRepository
based on database backend configuration.
"""
return create_search_repository(session_maker, project_id=project_id)
SearchRepositoryDep = Annotated[SearchRepository, Depends(get_search_repository)]
async def get_search_repository_v2( # pragma: no cover
session_maker: SessionMakerDep,
project_id: ProjectIdPathDep,
) -> SearchRepository:
"""Create a SearchRepository instance for v2 API."""
return create_search_repository(session_maker, project_id=project_id)
SearchRepositoryV2Dep = Annotated[SearchRepository, Depends(get_search_repository_v2)]
async def get_search_repository_v2_external(
session_maker: SessionMakerDep,
project_id: ProjectExternalIdPathDep,
) -> SearchRepository:
"""Create a SearchRepository instance for v2 API (uses external_id)."""
return create_search_repository(session_maker, project_id=project_id)
SearchRepositoryV2ExternalDep = Annotated[SearchRepository, Depends(get_search_repository_v2_external)]
+480
View File
@@ -0,0 +1,480 @@
"""Service dependency injection for basic-memory.
This module provides service-layer dependencies:
- EntityParser, MarkdownProcessor
- FileService, EntityService
- SearchService, LinkResolver, ContextService
- SyncService, ProjectService, DirectoryService
"""
from typing import Annotated
from fastapi import Depends
from loguru import logger
from basic_memory.deps.config import AppConfigDep
from basic_memory.deps.projects import (
ProjectConfigDep,
ProjectConfigV2Dep,
ProjectConfigV2ExternalDep,
ProjectRepositoryDep,
)
from basic_memory.deps.repositories import (
EntityRepositoryDep,
EntityRepositoryV2Dep,
EntityRepositoryV2ExternalDep,
ObservationRepositoryDep,
ObservationRepositoryV2Dep,
ObservationRepositoryV2ExternalDep,
RelationRepositoryDep,
RelationRepositoryV2Dep,
RelationRepositoryV2ExternalDep,
SearchRepositoryDep,
SearchRepositoryV2Dep,
SearchRepositoryV2ExternalDep,
)
from basic_memory.markdown import EntityParser
from basic_memory.markdown.markdown_processor import MarkdownProcessor
from basic_memory.services import EntityService, ProjectService
from basic_memory.services.context_service import ContextService
from basic_memory.services.directory_service import DirectoryService
from basic_memory.services.file_service import FileService
from basic_memory.services.link_resolver import LinkResolver
from basic_memory.services.search_service import SearchService
from basic_memory.sync import SyncService
# --- Entity Parser ---
async def get_entity_parser(project_config: ProjectConfigDep) -> EntityParser:
return EntityParser(project_config.home)
EntityParserDep = Annotated["EntityParser", Depends(get_entity_parser)]
async def get_entity_parser_v2(project_config: ProjectConfigV2Dep) -> EntityParser: # pragma: no cover
return EntityParser(project_config.home)
EntityParserV2Dep = Annotated["EntityParser", Depends(get_entity_parser_v2)]
async def get_entity_parser_v2_external(project_config: ProjectConfigV2ExternalDep) -> EntityParser:
return EntityParser(project_config.home)
EntityParserV2ExternalDep = Annotated["EntityParser", Depends(get_entity_parser_v2_external)]
# --- Markdown Processor ---
async def get_markdown_processor(
entity_parser: EntityParserDep, app_config: AppConfigDep
) -> MarkdownProcessor:
return MarkdownProcessor(entity_parser, app_config=app_config)
MarkdownProcessorDep = Annotated[MarkdownProcessor, Depends(get_markdown_processor)]
async def get_markdown_processor_v2( # pragma: no cover
entity_parser: EntityParserV2Dep, app_config: AppConfigDep
) -> MarkdownProcessor:
return MarkdownProcessor(entity_parser, app_config=app_config)
MarkdownProcessorV2Dep = Annotated[MarkdownProcessor, Depends(get_markdown_processor_v2)]
async def get_markdown_processor_v2_external(
entity_parser: EntityParserV2ExternalDep, app_config: AppConfigDep
) -> MarkdownProcessor:
return MarkdownProcessor(entity_parser, app_config=app_config)
MarkdownProcessorV2ExternalDep = Annotated[
MarkdownProcessor, Depends(get_markdown_processor_v2_external)
]
# --- File Service ---
async def get_file_service(
project_config: ProjectConfigDep,
markdown_processor: MarkdownProcessorDep,
app_config: AppConfigDep,
) -> FileService:
file_service = FileService(project_config.home, markdown_processor, app_config=app_config)
logger.debug(
f"Created FileService for project: {project_config.name}, base_path: {project_config.home} "
)
return file_service
FileServiceDep = Annotated[FileService, Depends(get_file_service)]
async def get_file_service_v2( # pragma: no cover
project_config: ProjectConfigV2Dep,
markdown_processor: MarkdownProcessorV2Dep,
app_config: AppConfigDep,
) -> FileService:
file_service = FileService(project_config.home, markdown_processor, app_config=app_config)
logger.debug(
f"Created FileService for project: {project_config.name}, base_path: {project_config.home}"
)
return file_service
FileServiceV2Dep = Annotated[FileService, Depends(get_file_service_v2)]
async def get_file_service_v2_external(
project_config: ProjectConfigV2ExternalDep,
markdown_processor: MarkdownProcessorV2ExternalDep,
app_config: AppConfigDep,
) -> FileService:
file_service = FileService(project_config.home, markdown_processor, app_config=app_config)
logger.debug(
f"Created FileService for project: {project_config.name}, base_path: {project_config.home}"
)
return file_service
FileServiceV2ExternalDep = Annotated[FileService, Depends(get_file_service_v2_external)]
# --- Search Service ---
async def get_search_service(
search_repository: SearchRepositoryDep,
entity_repository: EntityRepositoryDep,
file_service: FileServiceDep,
) -> SearchService:
"""Create SearchService with dependencies."""
return SearchService(search_repository, entity_repository, file_service)
SearchServiceDep = Annotated[SearchService, Depends(get_search_service)]
async def get_search_service_v2( # pragma: no cover
search_repository: SearchRepositoryV2Dep,
entity_repository: EntityRepositoryV2Dep,
file_service: FileServiceV2Dep,
) -> SearchService:
"""Create SearchService for v2 API."""
return SearchService(search_repository, entity_repository, file_service)
SearchServiceV2Dep = Annotated[SearchService, Depends(get_search_service_v2)]
async def get_search_service_v2_external(
search_repository: SearchRepositoryV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
file_service: FileServiceV2ExternalDep,
) -> SearchService:
"""Create SearchService for v2 API (uses external_id)."""
return SearchService(search_repository, entity_repository, file_service)
SearchServiceV2ExternalDep = Annotated[SearchService, Depends(get_search_service_v2_external)]
# --- Link Resolver ---
async def get_link_resolver(
entity_repository: EntityRepositoryDep, search_service: SearchServiceDep
) -> LinkResolver:
return LinkResolver(entity_repository=entity_repository, search_service=search_service)
LinkResolverDep = Annotated[LinkResolver, Depends(get_link_resolver)]
async def get_link_resolver_v2( # pragma: no cover
entity_repository: EntityRepositoryV2Dep, search_service: SearchServiceV2Dep
) -> LinkResolver:
return LinkResolver(entity_repository=entity_repository, search_service=search_service)
LinkResolverV2Dep = Annotated[LinkResolver, Depends(get_link_resolver_v2)]
async def get_link_resolver_v2_external(
entity_repository: EntityRepositoryV2ExternalDep, search_service: SearchServiceV2ExternalDep
) -> LinkResolver:
return LinkResolver(entity_repository=entity_repository, search_service=search_service)
LinkResolverV2ExternalDep = Annotated[LinkResolver, Depends(get_link_resolver_v2_external)]
# --- Entity Service ---
async def get_entity_service(
entity_repository: EntityRepositoryDep,
observation_repository: ObservationRepositoryDep,
relation_repository: RelationRepositoryDep,
entity_parser: EntityParserDep,
file_service: FileServiceDep,
link_resolver: LinkResolverDep,
search_service: SearchServiceDep,
app_config: AppConfigDep,
) -> EntityService:
"""Create EntityService with repository."""
return EntityService(
entity_repository=entity_repository,
observation_repository=observation_repository,
relation_repository=relation_repository,
entity_parser=entity_parser,
file_service=file_service,
link_resolver=link_resolver,
search_service=search_service,
app_config=app_config,
)
EntityServiceDep = Annotated[EntityService, Depends(get_entity_service)]
async def get_entity_service_v2( # pragma: no cover
entity_repository: EntityRepositoryV2Dep,
observation_repository: ObservationRepositoryV2Dep,
relation_repository: RelationRepositoryV2Dep,
entity_parser: EntityParserV2Dep,
file_service: FileServiceV2Dep,
link_resolver: LinkResolverV2Dep,
search_service: SearchServiceV2Dep,
app_config: AppConfigDep,
) -> EntityService:
"""Create EntityService for v2 API."""
return EntityService(
entity_repository=entity_repository,
observation_repository=observation_repository,
relation_repository=relation_repository,
entity_parser=entity_parser,
file_service=file_service,
link_resolver=link_resolver,
search_service=search_service,
app_config=app_config,
)
EntityServiceV2Dep = Annotated[EntityService, Depends(get_entity_service_v2)]
async def get_entity_service_v2_external(
entity_repository: EntityRepositoryV2ExternalDep,
observation_repository: ObservationRepositoryV2ExternalDep,
relation_repository: RelationRepositoryV2ExternalDep,
entity_parser: EntityParserV2ExternalDep,
file_service: FileServiceV2ExternalDep,
link_resolver: LinkResolverV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
app_config: AppConfigDep,
) -> EntityService:
"""Create EntityService for v2 API (uses external_id)."""
return EntityService(
entity_repository=entity_repository,
observation_repository=observation_repository,
relation_repository=relation_repository,
entity_parser=entity_parser,
file_service=file_service,
link_resolver=link_resolver,
search_service=search_service,
app_config=app_config,
)
EntityServiceV2ExternalDep = Annotated[EntityService, Depends(get_entity_service_v2_external)]
# --- Context Service ---
async def get_context_service(
search_repository: SearchRepositoryDep,
entity_repository: EntityRepositoryDep,
observation_repository: ObservationRepositoryDep,
) -> ContextService:
return ContextService(
search_repository=search_repository,
entity_repository=entity_repository,
observation_repository=observation_repository,
)
ContextServiceDep = Annotated[ContextService, Depends(get_context_service)]
async def get_context_service_v2( # pragma: no cover
search_repository: SearchRepositoryV2Dep,
entity_repository: EntityRepositoryV2Dep,
observation_repository: ObservationRepositoryV2Dep,
) -> ContextService:
"""Create ContextService for v2 API."""
return ContextService(
search_repository=search_repository,
entity_repository=entity_repository,
observation_repository=observation_repository,
)
ContextServiceV2Dep = Annotated[ContextService, Depends(get_context_service_v2)]
async def get_context_service_v2_external(
search_repository: SearchRepositoryV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
observation_repository: ObservationRepositoryV2ExternalDep,
) -> ContextService:
"""Create ContextService for v2 API (uses external_id)."""
return ContextService(
search_repository=search_repository,
entity_repository=entity_repository,
observation_repository=observation_repository,
)
ContextServiceV2ExternalDep = Annotated[ContextService, Depends(get_context_service_v2_external)]
# --- Sync Service ---
async def get_sync_service(
app_config: AppConfigDep,
entity_service: EntityServiceDep,
entity_parser: EntityParserDep,
entity_repository: EntityRepositoryDep,
relation_repository: RelationRepositoryDep,
project_repository: ProjectRepositoryDep,
search_service: SearchServiceDep,
file_service: FileServiceDep,
) -> SyncService: # pragma: no cover
return SyncService(
app_config=app_config,
entity_service=entity_service,
entity_parser=entity_parser,
entity_repository=entity_repository,
relation_repository=relation_repository,
project_repository=project_repository,
search_service=search_service,
file_service=file_service,
)
SyncServiceDep = Annotated[SyncService, Depends(get_sync_service)]
async def get_sync_service_v2(
app_config: AppConfigDep,
entity_service: EntityServiceV2Dep,
entity_parser: EntityParserV2Dep,
entity_repository: EntityRepositoryV2Dep,
relation_repository: RelationRepositoryV2Dep,
project_repository: ProjectRepositoryDep,
search_service: SearchServiceV2Dep,
file_service: FileServiceV2Dep,
) -> SyncService: # pragma: no cover
"""Create SyncService for v2 API."""
return SyncService(
app_config=app_config,
entity_service=entity_service,
entity_parser=entity_parser,
entity_repository=entity_repository,
relation_repository=relation_repository,
project_repository=project_repository,
search_service=search_service,
file_service=file_service,
)
SyncServiceV2Dep = Annotated[SyncService, Depends(get_sync_service_v2)]
async def get_sync_service_v2_external(
app_config: AppConfigDep,
entity_service: EntityServiceV2ExternalDep,
entity_parser: EntityParserV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
relation_repository: RelationRepositoryV2ExternalDep,
project_repository: ProjectRepositoryDep,
search_service: SearchServiceV2ExternalDep,
file_service: FileServiceV2ExternalDep,
) -> SyncService: # pragma: no cover
"""Create SyncService for v2 API (uses external_id)."""
return SyncService(
app_config=app_config,
entity_service=entity_service,
entity_parser=entity_parser,
entity_repository=entity_repository,
relation_repository=relation_repository,
project_repository=project_repository,
search_service=search_service,
file_service=file_service,
)
SyncServiceV2ExternalDep = Annotated[SyncService, Depends(get_sync_service_v2_external)]
# --- Project Service ---
async def get_project_service(
project_repository: ProjectRepositoryDep,
) -> ProjectService:
"""Create ProjectService with repository."""
return ProjectService(repository=project_repository)
ProjectServiceDep = Annotated[ProjectService, Depends(get_project_service)]
# --- Directory Service ---
async def get_directory_service(
entity_repository: EntityRepositoryDep,
) -> DirectoryService:
"""Create DirectoryService with dependencies."""
return DirectoryService(
entity_repository=entity_repository,
)
DirectoryServiceDep = Annotated[DirectoryService, Depends(get_directory_service)]
async def get_directory_service_v2( # pragma: no cover
entity_repository: EntityRepositoryV2Dep,
) -> DirectoryService:
"""Create DirectoryService for v2 API (uses integer project_id from path)."""
return DirectoryService(
entity_repository=entity_repository,
)
DirectoryServiceV2Dep = Annotated[DirectoryService, Depends(get_directory_service_v2)]
async def get_directory_service_v2_external(
entity_repository: EntityRepositoryV2ExternalDep,
) -> DirectoryService:
"""Create DirectoryService for v2 API (uses external_id from path)."""
return DirectoryService(
entity_repository=entity_repository,
)
DirectoryServiceV2ExternalDep = Annotated[DirectoryService, Depends(get_directory_service_v2_external)]
+212 -3
View File
@@ -1,9 +1,13 @@
"""Utilities for file operations."""
import asyncio
import hashlib
import shlex
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
import re
from typing import Any, Dict, Union
from typing import TYPE_CHECKING, Any, Dict, Optional, Union
import aiofiles
import yaml
@@ -12,6 +16,23 @@ from loguru import logger
from basic_memory.utils import FilePath
if TYPE_CHECKING: # pragma: no cover
from basic_memory.config import BasicMemoryConfig
@dataclass
class FileMetadata:
"""File metadata for cloud-compatible file operations.
This dataclass provides a cloud-agnostic way to represent file metadata,
enabling S3FileService to return metadata from head_object responses
instead of mock stat_result with zeros.
"""
size: int
created_at: datetime
modified_at: datetime
class FileError(Exception):
"""Base exception for file operations."""
@@ -53,6 +74,28 @@ async def compute_checksum(content: Union[str, bytes]) -> str:
raise FileError(f"Failed to compute checksum: {e}")
# UTF-8 BOM character that can appear at the start of files
UTF8_BOM = "\ufeff"
def strip_bom(content: str) -> str:
"""Strip UTF-8 BOM from the start of content if present.
BOM (Byte Order Mark) characters can be present in files created on Windows
or copied from certain sources. They should be stripped before processing
frontmatter. See issue #452.
Args:
content: Content that may start with BOM
Returns:
Content with BOM removed if present
"""
if content and content.startswith(UTF8_BOM):
return content[1:]
return content
async def write_file_atomic(path: FilePath, content: str) -> None:
"""
Write file with atomic operation using temporary file.
@@ -84,6 +127,168 @@ async def write_file_atomic(path: FilePath, content: str) -> None:
raise FileWriteError(f"Failed to write file {path}: {e}")
async def format_markdown_builtin(path: Path) -> Optional[str]:
"""
Format a markdown file using the built-in mdformat formatter.
Uses mdformat with GFM (GitHub Flavored Markdown) support for consistent
formatting without requiring Node.js or external tools.
Args:
path: Path to the markdown file to format
Returns:
Formatted content if successful, None if formatting failed.
"""
try:
import mdformat
except ImportError: # pragma: no cover
logger.warning(
"mdformat not installed, skipping built-in formatting",
path=str(path),
)
return None
try:
# Read original content
async with aiofiles.open(path, mode="r", encoding="utf-8") as f:
content = await f.read()
# Format using mdformat with GFM and frontmatter extensions
# mdformat is synchronous, so we run it in a thread executor
loop = asyncio.get_event_loop()
formatted_content = await loop.run_in_executor(
None,
lambda: mdformat.text(
content,
extensions={"gfm", "frontmatter"}, # GFM + YAML frontmatter support
options={"wrap": "no"}, # Don't wrap lines
),
)
# Only write if content changed
if formatted_content != content:
async with aiofiles.open(path, mode="w", encoding="utf-8") as f:
await f.write(formatted_content)
logger.debug(
"Formatted file with mdformat",
path=str(path),
changed=formatted_content != content,
)
return formatted_content
except Exception as e: # pragma: no cover
logger.warning(
"mdformat formatting failed",
path=str(path),
error=str(e),
)
return None
async def format_file(
path: Path,
config: "BasicMemoryConfig",
is_markdown: bool = False,
) -> Optional[str]:
"""
Format a file using configured formatter.
By default, uses the built-in mdformat formatter for markdown files (pure Python,
no Node.js required). External formatters like Prettier can be configured via
formatter_command or per-extension formatters.
Args:
path: File to format
config: Configuration with formatter settings
is_markdown: Whether this is a markdown file (caller should use FileService.is_markdown)
Returns:
Formatted content if successful, None if formatting was skipped or failed.
Failures are logged as warnings but don't raise exceptions.
"""
if not config.format_on_save:
return None
extension = path.suffix.lstrip(".")
formatter = config.formatters.get(extension) or config.formatter_command
# Use built-in mdformat for markdown files when no external formatter configured
if not formatter:
if is_markdown:
return await format_markdown_builtin(path)
else:
logger.debug("No formatter configured for extension", extension=extension)
return None
# Use external formatter
# Replace {file} placeholder with the actual path
cmd = formatter.replace("{file}", str(path))
try:
# Parse command into args list for safer execution (no shell=True)
args = shlex.split(cmd)
proc = await asyncio.create_subprocess_exec(
*args,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
try:
stdout, stderr = await asyncio.wait_for(
proc.communicate(),
timeout=config.formatter_timeout,
)
except asyncio.TimeoutError:
proc.kill()
await proc.wait()
logger.warning(
"Formatter timed out",
path=str(path),
timeout=config.formatter_timeout,
)
return None
if proc.returncode != 0:
logger.warning(
"Formatter exited with non-zero status",
path=str(path),
returncode=proc.returncode,
stderr=stderr.decode("utf-8", errors="replace") if stderr else "",
)
# Still try to read the file - formatter may have partially worked
# or the file may be unchanged
# Read formatted content
async with aiofiles.open(path, mode="r", encoding="utf-8") as f:
formatted_content = await f.read()
logger.debug(
"Formatted file successfully",
path=str(path),
formatter=args[0] if args else formatter,
)
return formatted_content
except FileNotFoundError:
# Formatter executable not found
logger.warning(
"Formatter executable not found",
command=cmd.split()[0] if cmd else "",
path=str(path),
)
return None
except Exception as e: # pragma: no cover
logger.warning(
"Formatter failed",
path=str(path),
error=str(e),
)
return None
def has_frontmatter(content: str) -> bool:
"""
Check if content contains valid YAML frontmatter.
@@ -97,7 +302,8 @@ def has_frontmatter(content: str) -> bool:
if not content:
return False
content = content.strip()
# Strip BOM before checking for frontmatter markers
content = strip_bom(content).strip()
if not content.startswith("---"):
return False
@@ -118,6 +324,8 @@ def parse_frontmatter(content: str) -> Dict[str, Any]:
ParseError: If frontmatter is invalid or parsing fails
"""
try:
# Strip BOM before parsing frontmatter
content = strip_bom(content)
if not content.strip().startswith("---"):
raise ParseError("Content has no frontmatter")
@@ -159,7 +367,8 @@ def remove_frontmatter(content: str) -> str:
Raises:
ParseError: If content starts with frontmatter marker but is malformed
"""
content = content.strip()
# Strip BOM before processing
content = strip_bom(content).strip()
# Return as-is if no frontmatter marker
if not content.startswith("---"):
+5 -5
View File
@@ -161,13 +161,13 @@ def load_bmignore_patterns() -> Set[str]:
# Skip empty lines and comments
if line and not line.startswith("#"):
patterns.add(line)
except Exception:
except Exception: # pragma: no cover
# If we can't read .bmignore, fall back to defaults
return set(DEFAULT_IGNORE_PATTERNS)
return set(DEFAULT_IGNORE_PATTERNS) # pragma: no cover
# If no patterns were loaded, use defaults
if not patterns:
return set(DEFAULT_IGNORE_PATTERNS)
if not patterns: # pragma: no cover
return set(DEFAULT_IGNORE_PATTERNS) # pragma: no cover
return patterns
@@ -261,7 +261,7 @@ def should_ignore_path(file_path: Path, base_path: Path, ignore_patterns: Set[st
# Glob pattern match on full path
if fnmatch.fnmatch(relative_posix, pattern) or fnmatch.fnmatch(relative_str, pattern):
return True
return True # pragma: no cover
return False
except ValueError:
+41 -20
View File
@@ -3,28 +3,43 @@
import logging
from abc import abstractmethod
from pathlib import Path
from typing import Any, Optional, TypeVar
from typing import TYPE_CHECKING, Any, Optional, TypeVar
from basic_memory.markdown.markdown_processor import MarkdownProcessor
from basic_memory.markdown.schemas import EntityMarkdown
from basic_memory.schemas.importer import ImportResult
if TYPE_CHECKING: # pragma: no cover
from basic_memory.services.file_service import FileService
logger = logging.getLogger(__name__)
T = TypeVar("T", bound=ImportResult)
class Importer[T: ImportResult]:
"""Base class for all import services."""
"""Base class for all import services.
def __init__(self, base_path: Path, markdown_processor: MarkdownProcessor):
All file operations are delegated to FileService, which can be overridden
in cloud environments to use S3 or other storage backends.
"""
def __init__(
self,
base_path: Path,
markdown_processor: MarkdownProcessor,
file_service: "FileService",
):
"""Initialize the import service.
Args:
markdown_processor: MarkdownProcessor instance for writing markdown files.
base_path: Base path for the project.
markdown_processor: MarkdownProcessor instance for markdown serialization.
file_service: FileService instance for all file operations.
"""
self.base_path = base_path.resolve() # Get absolute path
self.markdown_processor = markdown_processor
self.file_service = file_service
@abstractmethod
async def import_data(self, source_data, destination_folder: str, **kwargs: Any) -> T:
@@ -40,28 +55,34 @@ class Importer[T: ImportResult]:
"""
pass # pragma: no cover
async def write_entity(self, entity: EntityMarkdown, file_path: Path) -> None:
"""Write entity to file using markdown processor.
async def write_entity(self, entity: EntityMarkdown, file_path: str | Path) -> str:
"""Write entity to file using FileService.
This method serializes the entity to markdown and writes it using
FileService, which handles directory creation and storage backend
abstraction (local filesystem vs cloud storage).
Args:
entity: EntityMarkdown instance to write.
file_path: Path to write the entity to.
"""
await self.markdown_processor.write_file(file_path, entity)
def ensure_folder_exists(self, folder: str) -> Path:
"""Ensure folder exists, create if it doesn't.
Args:
base_path: Base path of the project.
folder: Folder name or path within the project.
file_path: Relative path to write the entity to. FileService handles base_path.
Returns:
Path to the folder.
Checksum of written file.
"""
folder_path = self.base_path / folder
folder_path.mkdir(parents=True, exist_ok=True)
return folder_path
content = self.markdown_processor.to_markdown_string(entity)
# FileService.write_file handles directory creation and returns checksum
return await self.file_service.write_file(file_path, content)
async def ensure_folder_exists(self, folder: str) -> None:
"""Ensure folder exists using FileService.
For cloud storage (S3), this is essentially a no-op since S3 doesn't
have actual folders - they're just key prefixes.
Args:
folder: Relative folder path within the project. FileService handles base_path.
"""
await self.file_service.ensure_directory(folder)
@abstractmethod
def handle_error(
+17 -4
View File
@@ -15,6 +15,19 @@ logger = logging.getLogger(__name__)
class ChatGPTImporter(Importer[ChatImportResult]):
"""Service for importing ChatGPT conversations."""
def handle_error( # pragma: no cover
self, message: str, error: Optional[Exception] = None
) -> ChatImportResult:
"""Return a failed ChatImportResult with an error message."""
error_msg = f"{message}: {error}" if error else message
return ChatImportResult(
import_count={},
success=False,
error_message=error_msg,
conversations=0,
messages=0,
)
async def import_data(
self, source_data, destination_folder: str, **kwargs: Any
) -> ChatImportResult:
@@ -30,7 +43,7 @@ class ChatGPTImporter(Importer[ChatImportResult]):
"""
try: # pragma: no cover
# Ensure the destination folder exists
self.ensure_folder_exists(destination_folder)
await self.ensure_folder_exists(destination_folder)
conversations = source_data
# Process each conversation
@@ -41,8 +54,8 @@ class ChatGPTImporter(Importer[ChatImportResult]):
# Convert to entity
entity = self._format_chat_content(destination_folder, chat)
# Write file
file_path = self.base_path / f"{entity.frontmatter.metadata['permalink']}.md"
# Write file using relative path - FileService handles base_path
file_path = f"{entity.frontmatter.metadata['permalink']}.md"
await self.write_entity(entity, file_path)
# Count messages
@@ -67,7 +80,7 @@ class ChatGPTImporter(Importer[ChatImportResult]):
except Exception as e: # pragma: no cover
logger.exception("Failed to import ChatGPT conversations")
return self.handle_error("Failed to import ChatGPT conversations", e) # pyright: ignore [reportReturnType]
return self.handle_error("Failed to import ChatGPT conversations", e)
def _format_chat_content(
self, folder: str, conversation: Dict[str, Any]
@@ -2,8 +2,7 @@
import logging
from datetime import datetime
from pathlib import Path
from typing import Any, Dict, List
from typing import Any, Dict, List, Optional
from basic_memory.markdown.schemas import EntityFrontmatter, EntityMarkdown
from basic_memory.importers.base import Importer
@@ -16,6 +15,19 @@ logger = logging.getLogger(__name__)
class ClaudeConversationsImporter(Importer[ChatImportResult]):
"""Service for importing Claude conversations."""
def handle_error( # pragma: no cover
self, message: str, error: Optional[Exception] = None
) -> ChatImportResult:
"""Return a failed ChatImportResult with an error message."""
error_msg = f"{message}: {error}" if error else message
return ChatImportResult(
import_count={},
success=False,
error_message=error_msg,
conversations=0,
messages=0,
)
async def import_data(
self, source_data, destination_folder: str, **kwargs: Any
) -> ChatImportResult:
@@ -31,7 +43,7 @@ class ClaudeConversationsImporter(Importer[ChatImportResult]):
"""
try:
# Ensure the destination folder exists
folder_path = self.ensure_folder_exists(destination_folder)
await self.ensure_folder_exists(destination_folder)
conversations = source_data
@@ -40,17 +52,20 @@ class ClaudeConversationsImporter(Importer[ChatImportResult]):
chats_imported = 0
for chat in conversations:
# Get name, providing default for unnamed conversations
chat_name = chat.get("name") or f"Conversation {chat.get('uuid', 'untitled')}"
# Convert to entity
entity = self._format_chat_content(
base_path=folder_path,
name=chat["name"],
folder=destination_folder,
name=chat_name,
messages=chat["chat_messages"],
created_at=chat["created_at"],
modified_at=chat["updated_at"],
)
# Write file
file_path = self.base_path / Path(f"{entity.frontmatter.metadata['permalink']}.md")
# Write file using relative path - FileService handles base_path
file_path = f"{entity.frontmatter.metadata['permalink']}.md"
await self.write_entity(entity, file_path)
chats_imported += 1
@@ -65,11 +80,11 @@ class ClaudeConversationsImporter(Importer[ChatImportResult]):
except Exception as e: # pragma: no cover
logger.exception("Failed to import Claude conversations")
return self.handle_error("Failed to import Claude conversations", e) # pyright: ignore [reportReturnType]
return self.handle_error("Failed to import Claude conversations", e)
def _format_chat_content(
self,
base_path: Path,
folder: str,
name: str,
messages: List[Dict[str, Any]],
created_at: str,
@@ -78,7 +93,7 @@ class ClaudeConversationsImporter(Importer[ChatImportResult]):
"""Convert chat messages to Basic Memory entity format.
Args:
base_path: Base path for the entity.
folder: Destination folder name (relative path).
name: Chat name.
messages: List of chat messages.
created_at: Creation timestamp.
@@ -87,10 +102,10 @@ class ClaudeConversationsImporter(Importer[ChatImportResult]):
Returns:
EntityMarkdown instance representing the conversation.
"""
# Generate permalink
# Generate permalink using folder name (relative path)
date_prefix = datetime.fromisoformat(created_at.replace("Z", "+00:00")).strftime("%Y%m%d")
clean_title = clean_filename(name)
permalink = f"{base_path.name}/{date_prefix}-{clean_title}"
permalink = f"{folder}/{date_prefix}-{clean_title}"
# Format content
content = self._format_chat_markdown(
@@ -14,6 +14,19 @@ logger = logging.getLogger(__name__)
class ClaudeProjectsImporter(Importer[ProjectImportResult]):
"""Service for importing Claude projects."""
def handle_error( # pragma: no cover
self, message: str, error: Optional[Exception] = None
) -> ProjectImportResult:
"""Return a failed ProjectImportResult with an error message."""
error_msg = f"{message}: {error}" if error else message
return ProjectImportResult(
import_count={},
success=False,
error_message=error_msg,
documents=0,
prompts=0,
)
async def import_data(
self, source_data, destination_folder: str, **kwargs: Any
) -> ProjectImportResult:
@@ -29,9 +42,8 @@ class ClaudeProjectsImporter(Importer[ProjectImportResult]):
"""
try:
# Ensure the base folder exists
base_path = self.base_path
if destination_folder:
base_path = self.ensure_folder_exists(destination_folder)
await self.ensure_folder_exists(destination_folder)
projects = source_data
@@ -42,20 +54,26 @@ class ClaudeProjectsImporter(Importer[ProjectImportResult]):
for project in projects:
project_dir = clean_filename(project["name"])
# Create project directories
docs_dir = base_path / project_dir / "docs"
docs_dir.mkdir(parents=True, exist_ok=True)
# Create project directories using FileService with relative path
docs_dir = (
f"{destination_folder}/{project_dir}/docs"
if destination_folder
else f"{project_dir}/docs"
)
await self.file_service.ensure_directory(docs_dir)
# Import prompt template if it exists
if prompt_entity := self._format_prompt_markdown(project):
file_path = base_path / f"{prompt_entity.frontmatter.metadata['permalink']}.md"
if prompt_entity := self._format_prompt_markdown(project, destination_folder):
# Write file using relative path - FileService handles base_path
file_path = f"{prompt_entity.frontmatter.metadata['permalink']}.md"
await self.write_entity(prompt_entity, file_path)
prompts_imported += 1
# Import project documents
for doc in project.get("docs", []):
entity = self._format_project_markdown(project, doc)
file_path = base_path / f"{entity.frontmatter.metadata['permalink']}.md"
entity = self._format_project_markdown(project, doc, destination_folder)
# Write file using relative path - FileService handles base_path
file_path = f"{entity.frontmatter.metadata['permalink']}.md"
await self.write_entity(entity, file_path)
docs_imported += 1
@@ -68,16 +86,17 @@ class ClaudeProjectsImporter(Importer[ProjectImportResult]):
except Exception as e: # pragma: no cover
logger.exception("Failed to import Claude projects")
return self.handle_error("Failed to import Claude projects", e) # pyright: ignore [reportReturnType]
return self.handle_error("Failed to import Claude projects", e)
def _format_project_markdown(
self, project: Dict[str, Any], doc: Dict[str, Any]
self, project: Dict[str, Any], doc: Dict[str, Any], destination_folder: str = ""
) -> EntityMarkdown:
"""Format a project document as a Basic Memory entity.
Args:
project: Project data.
doc: Document data.
destination_folder: Optional destination folder prefix.
Returns:
EntityMarkdown instance representing the document.
@@ -90,6 +109,13 @@ class ClaudeProjectsImporter(Importer[ProjectImportResult]):
project_dir = clean_filename(project["name"])
doc_file = clean_filename(doc["filename"])
# Build permalink with optional destination folder prefix
permalink = (
f"{destination_folder}/{project_dir}/docs/{doc_file}"
if destination_folder
else f"{project_dir}/docs/{doc_file}"
)
# Create entity
entity = EntityMarkdown(
frontmatter=EntityFrontmatter(
@@ -98,7 +124,7 @@ class ClaudeProjectsImporter(Importer[ProjectImportResult]):
"title": doc["filename"],
"created": created_at,
"modified": modified_at,
"permalink": f"{project_dir}/docs/{doc_file}",
"permalink": permalink,
"project_name": project["name"],
"project_uuid": project["uuid"],
"doc_uuid": doc["uuid"],
@@ -109,11 +135,14 @@ class ClaudeProjectsImporter(Importer[ProjectImportResult]):
return entity
def _format_prompt_markdown(self, project: Dict[str, Any]) -> Optional[EntityMarkdown]:
def _format_prompt_markdown(
self, project: Dict[str, Any], destination_folder: str = ""
) -> Optional[EntityMarkdown]:
"""Format project prompt template as a Basic Memory entity.
Args:
project: Project data.
destination_folder: Optional destination folder prefix.
Returns:
EntityMarkdown instance representing the prompt template, or None if
@@ -129,6 +158,13 @@ class ClaudeProjectsImporter(Importer[ProjectImportResult]):
# Generate clean project directory name
project_dir = clean_filename(project["name"])
# Build permalink with optional destination folder prefix
permalink = (
f"{destination_folder}/{project_dir}/prompt-template"
if destination_folder
else f"{project_dir}/prompt-template"
)
# Create entity
entity = EntityMarkdown(
frontmatter=EntityFrontmatter(
@@ -137,7 +173,7 @@ class ClaudeProjectsImporter(Importer[ProjectImportResult]):
"title": f"Prompt Template: {project['name']}",
"created": created_at,
"modified": modified_at,
"permalink": f"{project_dir}/prompt-template",
"permalink": permalink,
"project_name": project["name"],
"project_uuid": project["uuid"],
}
@@ -1,9 +1,8 @@
"""Memory JSON import service for Basic Memory."""
import logging
from typing import Any, Dict, List
from typing import Any, Dict, List, Optional
from basic_memory.config import get_project_config
from basic_memory.markdown.schemas import EntityFrontmatter, EntityMarkdown, Observation, Relation
from basic_memory.importers.base import Importer
from basic_memory.schemas.importer import EntityImportResult
@@ -14,6 +13,20 @@ logger = logging.getLogger(__name__)
class MemoryJsonImporter(Importer[EntityImportResult]):
"""Service for importing memory.json format data."""
def handle_error( # pragma: no cover
self, message: str, error: Optional[Exception] = None
) -> EntityImportResult:
"""Return a failed EntityImportResult with an error message."""
error_msg = f"{message}: {error}" if error else message
return EntityImportResult(
import_count={},
success=False,
error_message=error_msg,
entities=0,
relations=0,
skipped_entities=0,
)
async def import_data(
self, source_data, destination_folder: str = "", **kwargs: Any
) -> EntityImportResult:
@@ -27,17 +40,15 @@ class MemoryJsonImporter(Importer[EntityImportResult]):
Returns:
EntityImportResult containing statistics and status of the import.
"""
config = get_project_config()
try:
# First pass - collect all relations by source entity
entity_relations: Dict[str, List[Relation]] = {}
entities: Dict[str, Dict[str, Any]] = {}
skipped_entities: int = 0
# Ensure the base path exists
base_path = config.home # pragma: no cover
# Ensure the destination folder exists if provided
if destination_folder: # pragma: no cover
base_path = self.ensure_folder_exists(destination_folder)
await self.ensure_folder_exists(destination_folder)
# First pass - collect entities and relations
for line in source_data:
@@ -46,9 +57,9 @@ class MemoryJsonImporter(Importer[EntityImportResult]):
# Handle different possible name keys
entity_name = data.get("name") or data.get("entityName") or data.get("id")
if not entity_name:
logger.warning(f"Entity missing name field: {data}")
skipped_entities += 1
continue
logger.warning(f"Entity missing name field: {data}") # pragma: no cover
skipped_entities += 1 # pragma: no cover
continue # pragma: no cover
entities[entity_name] = data
elif data["type"] == "relation":
# Store relation with its source entity
@@ -68,9 +79,18 @@ class MemoryJsonImporter(Importer[EntityImportResult]):
# Get entity type with fallback
entity_type = entity_data.get("entityType") or entity_data.get("type") or "entity"
# Ensure entity type directory exists
entity_type_dir = base_path / entity_type
entity_type_dir.mkdir(parents=True, exist_ok=True)
# Build permalink with optional destination folder prefix
permalink = (
f"{destination_folder}/{entity_type}/{name}"
if destination_folder
else f"{entity_type}/{name}"
)
# Ensure entity type directory exists using FileService with relative path
entity_type_dir = (
f"{destination_folder}/{entity_type}" if destination_folder else entity_type
)
await self.file_service.ensure_directory(entity_type_dir)
# Get observations with fallback to empty list
observations = entity_data.get("observations", [])
@@ -80,7 +100,7 @@ class MemoryJsonImporter(Importer[EntityImportResult]):
metadata={
"type": entity_type,
"title": name,
"permalink": f"{entity_type}/{name}",
"permalink": permalink,
}
),
content=f"# {name}\n",
@@ -88,8 +108,8 @@ class MemoryJsonImporter(Importer[EntityImportResult]):
relations=entity_relations.get(name, []),
)
# Write entity file
file_path = base_path / f"{entity_type}/{name}.md"
# Write file using relative path - FileService handles base_path
file_path = f"{entity.frontmatter.metadata['permalink']}.md"
await self.write_entity(entity, file_path)
entities_created += 1
@@ -105,4 +125,4 @@ class MemoryJsonImporter(Importer[EntityImportResult]):
except Exception as e: # pragma: no cover
logger.exception("Failed to import memory.json")
return self.handle_error("Failed to import memory.json", e) # pyright: ignore [reportReturnType]
return self.handle_error("Failed to import memory.json", e)
+5 -2
View File
@@ -5,15 +5,18 @@ from datetime import datetime
from typing import Any
def clean_filename(name: str) -> str: # pragma: no cover
def clean_filename(name: str | None) -> str: # pragma: no cover
"""Clean a string to be used as a filename.
Args:
name: The string to clean.
name: The string to clean (can be None).
Returns:
A cleaned string suitable for use as a filename.
"""
# Handle None or empty input
if not name:
return "untitled"
# Replace common punctuation and whitespace with underscores
name = re.sub(r"[\s\-,.:/\\\[\]\(\)]+", "_", name)
# Remove any non-alphanumeric or underscore characters
+65 -26
View File
@@ -23,6 +23,7 @@ from basic_memory.markdown.schemas import (
)
from basic_memory.utils import parse_tags
md = MarkdownIt().use(observation_plugin).use(relation_plugin)
@@ -189,35 +190,69 @@ class EntityParser:
return self.base_path / path
async def parse_file_content(self, absolute_path, file_content):
# Parse frontmatter with proper error handling for malformed YAML (issue #185)
try:
post = frontmatter.loads(file_content)
except yaml.YAMLError as e:
# Log the YAML parsing error with file context
logger.warning(
f"Failed to parse YAML frontmatter in {absolute_path}: {e}. "
f"Treating file as plain markdown without frontmatter."
)
# Create a post with no frontmatter - treat entire content as markdown
post = frontmatter.Post(file_content, metadata={})
"""Parse markdown content from file stats.
# Extract file stat info
Delegates to parse_markdown_content() for actual parsing logic.
Exists for backwards compatibility with code that passes file paths.
"""
# Extract file stat info for timestamps
file_stats = absolute_path.stat()
# Normalize frontmatter values to prevent AttributeError on date objects (issue #236)
# PyYAML automatically converts date strings like "2025-10-24" to datetime.date objects
# This normalization converts them back to ISO format strings to ensure compatibility
# with code that expects string values
# Delegate to parse_markdown_content with timestamps from file stats
return await self.parse_markdown_content(
file_path=absolute_path,
content=file_content,
mtime=file_stats.st_mtime,
ctime=file_stats.st_ctime,
)
async def parse_markdown_content(
self,
file_path: Path,
content: str,
mtime: Optional[float] = None,
ctime: Optional[float] = None,
) -> EntityMarkdown:
"""Parse markdown content without requiring file to exist on disk.
Useful for parsing content from S3 or other remote sources where the file
is not available locally.
Args:
file_path: Path for metadata (doesn't need to exist on disk)
content: Markdown content as string
mtime: Optional modification time (Unix timestamp)
ctime: Optional creation time (Unix timestamp)
Returns:
EntityMarkdown with parsed content
"""
# Strip BOM before parsing (can be present in files from Windows or certain sources)
# See issue #452
from basic_memory.file_utils import strip_bom
content = strip_bom(content)
# Parse frontmatter with proper error handling for malformed YAML
try:
post = frontmatter.loads(content)
except yaml.YAMLError as e:
logger.warning(
f"Failed to parse YAML frontmatter in {file_path}: {e}. "
f"Treating file as plain markdown without frontmatter."
)
post = frontmatter.Post(content, metadata={})
# Normalize frontmatter values
metadata = normalize_frontmatter_metadata(post.metadata)
# Ensure required fields have defaults (issue #184, #387)
# Handle title - use default if missing, None/null, empty, or string "None"
# Ensure required fields have defaults
title = metadata.get("title")
if not title or title == "None":
metadata["title"] = absolute_path.stem
metadata["title"] = file_path.stem
else:
metadata["title"] = title
# Handle type - use default if missing OR explicitly set to None/null
entity_type = metadata.get("type")
metadata["type"] = entity_type if entity_type is not None else "note"
@@ -225,16 +260,20 @@ class EntityParser:
if tags:
metadata["tags"] = tags
# frontmatter - use metadata with defaults applied
entity_frontmatter = EntityFrontmatter(
metadata=metadata,
)
# Parse content for observations and relations
entity_frontmatter = EntityFrontmatter(metadata=metadata)
entity_content = parse(post.content)
# Use provided timestamps or current time as fallback
now = datetime.now().astimezone()
created = datetime.fromtimestamp(ctime).astimezone() if ctime else now
modified = datetime.fromtimestamp(mtime).astimezone() if mtime else now
return EntityMarkdown(
frontmatter=entity_frontmatter,
content=post.content,
observations=entity_content.observations,
relations=entity_content.relations,
created=datetime.fromtimestamp(file_stats.st_ctime).astimezone(),
modified=datetime.fromtimestamp(file_stats.st_mtime).astimezone(),
created=created,
modified=modified,
)
@@ -1,15 +1,19 @@
from pathlib import Path
from typing import Optional
from typing import TYPE_CHECKING, Optional
from collections import OrderedDict
from frontmatter import Post
from loguru import logger
from basic_memory import file_utils
from basic_memory.file_utils import dump_frontmatter
from basic_memory.markdown.entity_parser import EntityParser
from basic_memory.markdown.schemas import EntityMarkdown, Observation, Relation
if TYPE_CHECKING: # pragma: no cover
from basic_memory.config import BasicMemoryConfig
class DirtyFileError(Exception):
"""Raised when attempting to write to a file that has been modified."""
@@ -35,9 +39,14 @@ class MarkdownProcessor:
3. Track schema changes (that's done by the database)
"""
def __init__(self, entity_parser: EntityParser):
"""Initialize processor with base path and parser."""
def __init__(
self,
entity_parser: EntityParser,
app_config: Optional["BasicMemoryConfig"] = None,
):
"""Initialize processor with parser and optional config."""
self.entity_parser = entity_parser
self.app_config = app_config
async def read_file(self, path: Path) -> EntityMarkdown:
"""Read and parse file into EntityMarkdown schema.
@@ -122,7 +131,61 @@ class MarkdownProcessor:
# Write atomically and return checksum of updated file
path.parent.mkdir(parents=True, exist_ok=True)
await file_utils.write_file_atomic(path, final_content)
return await file_utils.compute_checksum(final_content)
# Format file if configured (MarkdownProcessor always handles markdown files)
content_for_checksum = final_content
if self.app_config:
formatted_content = await file_utils.format_file( # pragma: no cover
path, self.app_config, is_markdown=True
)
if formatted_content is not None: # pragma: no cover
content_for_checksum = formatted_content # pragma: no cover
return await file_utils.compute_checksum(content_for_checksum)
def to_markdown_string(self, markdown: EntityMarkdown) -> str:
"""Convert EntityMarkdown to markdown string with frontmatter.
This method handles serialization only - it does not write to files.
Use FileService.write_file() to persist the output.
This enables cloud environments to override file operations via
dependency injection while reusing the serialization logic.
Args:
markdown: EntityMarkdown schema to serialize
Returns:
Complete markdown string with frontmatter, content, and structured sections
"""
# Convert frontmatter to dict
frontmatter_dict = OrderedDict()
frontmatter_dict["title"] = markdown.frontmatter.title
frontmatter_dict["type"] = markdown.frontmatter.type
frontmatter_dict["permalink"] = markdown.frontmatter.permalink
metadata = markdown.frontmatter.metadata or {}
for k, v in metadata.items():
frontmatter_dict[k] = v
# Start with user content (or minimal title for new files)
content = markdown.content or f"# {markdown.frontmatter.title}\n"
# Add structured sections with proper spacing
content = content.rstrip() # Remove trailing whitespace
# Add a blank line if we have semantic content
if markdown.observations or markdown.relations:
content += "\n"
if markdown.observations:
content += self.format_observations(markdown.observations)
if markdown.relations:
content += self.format_relations(markdown.relations)
# Create Post object for frontmatter
post = Post(content, **frontmatter_dict)
return dump_frontmatter(post)
def format_observations(self, observations: list[Observation]) -> str:
"""Format observations section in standard way.
+4 -2
View File
@@ -30,7 +30,9 @@ def is_observation(token: Token) -> bool:
# Check for proper observation format: [category] content
match = re.match(r"^\[([^\[\]()]+)\]\s+(.+)", content)
has_tags = "#" in content
# Check for standalone hashtags (words starting with #)
# This excludes # in HTML attributes like color="#4285F4"
has_tags = any(part.startswith("#") for part in content.split())
return bool(match) or has_tags
@@ -160,7 +162,7 @@ def parse_inline_relations(content: str) -> List[Dict[str, Any]]:
target = content[start + 2 : end].strip()
if target:
relations.append({"type": "links to", "target": target, "context": None})
relations.append({"type": "links_to", "target": target, "context": None})
start = end + 2
+10 -1
View File
@@ -3,6 +3,7 @@
from pathlib import Path
from typing import Any, Optional
from frontmatter import Post
from basic_memory.file_utils import has_frontmatter, remove_frontmatter, parse_frontmatter
@@ -12,7 +13,10 @@ from basic_memory.models import Observation as ObservationModel
def entity_model_from_markdown(
file_path: Path, markdown: EntityMarkdown, entity: Optional[Entity] = None
file_path: Path,
markdown: EntityMarkdown,
entity: Optional[Entity] = None,
project_id: Optional[int] = None,
) -> Entity:
"""
Convert markdown entity to model. Does not include relations.
@@ -21,6 +25,7 @@ def entity_model_from_markdown(
file_path: Path to the markdown file
markdown: Parsed markdown entity
entity: Optional existing entity to update
project_id: Project ID for new observations (uses entity.project_id if not provided)
Returns:
Entity model populated from markdown
@@ -50,9 +55,13 @@ def entity_model_from_markdown(
metadata = markdown.frontmatter.metadata or {}
model.entity_metadata = {k: str(v) for k, v in metadata.items() if v is not None}
# Get project_id from entity if not provided
obs_project_id = project_id or (model.project_id if hasattr(model, "project_id") else None)
# Convert observations
model.observations = [
ObservationModel(
project_id=obs_project_id,
content=obs.content,
category=obs.category,
context=obs.context,
+1
View File
@@ -95,6 +95,7 @@ async def get_client() -> AsyncIterator[AsyncClient]:
yield client
else:
# Local mode: ASGI transport for in-process calls
# Note: ASGI transport does NOT trigger FastAPI lifespan, so no special handling needed
logger.info("Creating ASGI client for local Basic Memory API")
async with AsyncClient(
transport=ASGITransport(app=fastapi_app), base_url="http://test", timeout=timeout
+28
View File
@@ -0,0 +1,28 @@
"""Typed internal API clients for MCP tools.
These clients encapsulate API paths, error handling, and response validation.
MCP tools become thin adapters that call these clients and format results.
Usage:
from basic_memory.mcp.clients import KnowledgeClient, SearchClient
async with get_client() as http_client:
knowledge = KnowledgeClient(http_client, project_id)
entity = await knowledge.create_entity(entity_data)
"""
from basic_memory.mcp.clients.knowledge import KnowledgeClient
from basic_memory.mcp.clients.search import SearchClient
from basic_memory.mcp.clients.memory import MemoryClient
from basic_memory.mcp.clients.directory import DirectoryClient
from basic_memory.mcp.clients.resource import ResourceClient
from basic_memory.mcp.clients.project import ProjectClient
__all__ = [
"KnowledgeClient",
"SearchClient",
"MemoryClient",
"DirectoryClient",
"ResourceClient",
"ProjectClient",
]
+70
View File
@@ -0,0 +1,70 @@
"""Typed client for directory API operations.
Encapsulates all /v2/projects/{project_id}/directory/* endpoints.
"""
from typing import Optional, Any
from httpx import AsyncClient
from basic_memory.mcp.tools.utils import call_get
class DirectoryClient:
"""Typed client for directory listing operations.
Centralizes:
- API path construction for /v2/projects/{project_id}/directory/*
- Response validation
- Consistent error handling through call_* utilities
Usage:
async with get_client() as http_client:
client = DirectoryClient(http_client, project_id)
nodes = await client.list("/", depth=2)
"""
def __init__(self, http_client: AsyncClient, project_id: str):
"""Initialize the directory client.
Args:
http_client: HTTPX AsyncClient for making requests
project_id: Project external_id (UUID) for API calls
"""
self.http_client = http_client
self.project_id = project_id
self._base_path = f"/v2/projects/{project_id}/directory"
async def list(
self,
dir_name: str = "/",
*,
depth: int = 1,
file_name_glob: Optional[str] = None,
) -> list[dict[str, Any]]:
"""List directory contents.
Args:
dir_name: Directory path to list (default: root)
depth: How deep to traverse (default: 1)
file_name_glob: Optional glob pattern to filter files
Returns:
List of directory nodes with their contents
Raises:
ToolError: If the request fails
"""
params: dict = {
"dir_name": dir_name,
"depth": depth,
}
if file_name_glob:
params["file_name_glob"] = file_name_glob
response = await call_get(
self.http_client,
f"{self._base_path}/list",
params=params,
)
return response.json()
+176
View File
@@ -0,0 +1,176 @@
"""Typed client for knowledge/entity API operations.
Encapsulates all /v2/projects/{project_id}/knowledge/* endpoints.
"""
from typing import Any
from httpx import AsyncClient
from basic_memory.mcp.tools.utils import call_get, call_post, call_put, call_patch, call_delete
from basic_memory.schemas.response import EntityResponse, DeleteEntitiesResponse
class KnowledgeClient:
"""Typed client for knowledge graph entity operations.
Centralizes:
- API path construction for /v2/projects/{project_id}/knowledge/*
- Response validation via Pydantic models
- Consistent error handling through call_* utilities
Usage:
async with get_client() as http_client:
client = KnowledgeClient(http_client, project_id)
entity = await client.create_entity(entity_data)
"""
def __init__(self, http_client: AsyncClient, project_id: str):
"""Initialize the knowledge client.
Args:
http_client: HTTPX AsyncClient for making requests
project_id: Project external_id (UUID) for API calls
"""
self.http_client = http_client
self.project_id = project_id
self._base_path = f"/v2/projects/{project_id}/knowledge"
# --- Entity CRUD Operations ---
async def create_entity(self, entity_data: dict[str, Any]) -> EntityResponse:
"""Create a new entity.
Args:
entity_data: Entity data including title, content, folder, etc.
Returns:
EntityResponse with created entity details
Raises:
ToolError: If the request fails
"""
response = await call_post(
self.http_client,
f"{self._base_path}/entities",
json=entity_data,
)
return EntityResponse.model_validate(response.json())
async def update_entity(self, entity_id: str, entity_data: dict[str, Any]) -> EntityResponse:
"""Update an existing entity (full replacement).
Args:
entity_id: Entity external_id (UUID)
entity_data: Complete entity data for replacement
Returns:
EntityResponse with updated entity details
Raises:
ToolError: If the request fails
"""
response = await call_put(
self.http_client,
f"{self._base_path}/entities/{entity_id}",
json=entity_data,
)
return EntityResponse.model_validate(response.json())
async def get_entity(self, entity_id: str) -> EntityResponse:
"""Get an entity by ID.
Args:
entity_id: Entity external_id (UUID)
Returns:
EntityResponse with entity details
Raises:
ToolError: If the entity is not found or request fails
"""
response = await call_get(
self.http_client,
f"{self._base_path}/entities/{entity_id}",
)
return EntityResponse.model_validate(response.json())
async def patch_entity(self, entity_id: str, patch_data: dict[str, Any]) -> EntityResponse:
"""Partially update an entity.
Args:
entity_id: Entity external_id (UUID)
patch_data: Partial entity data to update
Returns:
EntityResponse with updated entity details
Raises:
ToolError: If the request fails
"""
response = await call_patch(
self.http_client,
f"{self._base_path}/entities/{entity_id}",
json=patch_data,
)
return EntityResponse.model_validate(response.json())
async def delete_entity(self, entity_id: str) -> DeleteEntitiesResponse:
"""Delete an entity.
Args:
entity_id: Entity external_id (UUID)
Returns:
DeleteEntitiesResponse confirming deletion
Raises:
ToolError: If the entity is not found or request fails
"""
response = await call_delete(
self.http_client,
f"{self._base_path}/entities/{entity_id}",
)
return DeleteEntitiesResponse.model_validate(response.json())
async def move_entity(self, entity_id: str, destination_path: str) -> EntityResponse:
"""Move an entity to a new location.
Args:
entity_id: Entity external_id (UUID)
destination_path: New file path for the entity
Returns:
EntityResponse with updated entity details
Raises:
ToolError: If the request fails
"""
response = await call_put(
self.http_client,
f"{self._base_path}/entities/{entity_id}/move",
json={"destination_path": destination_path},
)
return EntityResponse.model_validate(response.json())
# --- Resolution ---
async def resolve_entity(self, identifier: str) -> str:
"""Resolve a string identifier to an entity external_id.
Args:
identifier: The identifier to resolve (permalink, title, or path)
Returns:
The resolved entity external_id (UUID)
Raises:
ToolError: If the identifier cannot be resolved
"""
response = await call_post(
self.http_client,
f"{self._base_path}/resolve",
json={"identifier": identifier},
)
data = response.json()
return data["external_id"]
+120
View File
@@ -0,0 +1,120 @@
"""Typed client for memory/context API operations.
Encapsulates all /v2/projects/{project_id}/memory/* endpoints.
"""
from typing import Optional
from httpx import AsyncClient
from basic_memory.mcp.tools.utils import call_get
from basic_memory.schemas.memory import GraphContext
class MemoryClient:
"""Typed client for memory context operations.
Centralizes:
- API path construction for /v2/projects/{project_id}/memory/*
- Response validation via Pydantic models
- Consistent error handling through call_* utilities
Usage:
async with get_client() as http_client:
client = MemoryClient(http_client, project_id)
context = await client.build_context("memory://specs/search")
"""
def __init__(self, http_client: AsyncClient, project_id: str):
"""Initialize the memory client.
Args:
http_client: HTTPX AsyncClient for making requests
project_id: Project external_id (UUID) for API calls
"""
self.http_client = http_client
self.project_id = project_id
self._base_path = f"/v2/projects/{project_id}/memory"
async def build_context(
self,
path: str,
*,
depth: int = 1,
timeframe: Optional[str] = None,
page: int = 1,
page_size: int = 10,
max_related: int = 10,
) -> GraphContext:
"""Build context from a memory path.
Args:
path: The path to build context for (without memory:// prefix)
depth: How deep to traverse relations
timeframe: Time filter (e.g., "7d", "1 week")
page: Page number (1-indexed)
page_size: Results per page
max_related: Maximum related items per result
Returns:
GraphContext with hierarchical results
Raises:
ToolError: If the request fails
"""
params: dict = {
"depth": depth,
"page": page,
"page_size": page_size,
"max_related": max_related,
}
if timeframe:
params["timeframe"] = timeframe
response = await call_get(
self.http_client,
f"{self._base_path}/{path}",
params=params,
)
return GraphContext.model_validate(response.json())
async def recent(
self,
*,
timeframe: str = "7d",
depth: int = 1,
types: Optional[list[str]] = None,
page: int = 1,
page_size: int = 10,
) -> GraphContext:
"""Get recent activity.
Args:
timeframe: Time filter (e.g., "7d", "1 week", "2 days ago")
depth: How deep to traverse relations
types: Filter by item types
page: Page number (1-indexed)
page_size: Results per page
Returns:
GraphContext with recent activity
Raises:
ToolError: If the request fails
"""
params: dict = {
"timeframe": timeframe,
"depth": depth,
"page": page,
"page_size": page_size,
}
if types:
# Join types as comma-separated string if provided
params["type"] = ",".join(types) if isinstance(types, list) else types
response = await call_get(
self.http_client,
f"{self._base_path}/recent",
params=params,
)
return GraphContext.model_validate(response.json())
+89
View File
@@ -0,0 +1,89 @@
"""Typed client for project API operations.
Encapsulates project-level endpoints.
"""
from typing import Any
from httpx import AsyncClient
from basic_memory.mcp.tools.utils import call_get, call_post, call_delete
from basic_memory.schemas.project_info import ProjectList, ProjectStatusResponse
class ProjectClient:
"""Typed client for project management operations.
Centralizes:
- API path construction for project endpoints
- Response validation via Pydantic models
- Consistent error handling through call_* utilities
Note: This client does not require a project_id since it operates
across projects.
Usage:
async with get_client() as http_client:
client = ProjectClient(http_client)
projects = await client.list_projects()
"""
def __init__(self, http_client: AsyncClient):
"""Initialize the project client.
Args:
http_client: HTTPX AsyncClient for making requests
"""
self.http_client = http_client
async def list_projects(self) -> ProjectList:
"""List all available projects.
Returns:
ProjectList with all projects and default project name
Raises:
ToolError: If the request fails
"""
response = await call_get(
self.http_client,
"/projects/projects",
)
return ProjectList.model_validate(response.json())
async def create_project(self, project_data: dict[str, Any]) -> ProjectStatusResponse:
"""Create a new project.
Args:
project_data: Project creation data (name, path, set_default)
Returns:
ProjectStatusResponse with creation result
Raises:
ToolError: If the request fails
"""
response = await call_post(
self.http_client,
"/projects/projects",
json=project_data,
)
return ProjectStatusResponse.model_validate(response.json())
async def delete_project(self, project_external_id: str) -> ProjectStatusResponse:
"""Delete a project by its external ID.
Args:
project_external_id: Project external ID (UUID)
Returns:
ProjectStatusResponse with deletion result
Raises:
ToolError: If the request fails
"""
response = await call_delete(
self.http_client,
f"/v2/projects/{project_external_id}",
)
return ProjectStatusResponse.model_validate(response.json())
+71
View File
@@ -0,0 +1,71 @@
"""Typed client for resource API operations.
Encapsulates all /v2/projects/{project_id}/resource/* endpoints.
"""
from typing import Optional
from httpx import AsyncClient, Response
from basic_memory.mcp.tools.utils import call_get
class ResourceClient:
"""Typed client for resource operations.
Centralizes:
- API path construction for /v2/projects/{project_id}/resource/*
- Consistent error handling through call_* utilities
Note: This client returns raw Response objects for resources since they
may be text, images, or other binary content that needs special handling.
Usage:
async with get_client() as http_client:
client = ResourceClient(http_client, project_id)
response = await client.read(entity_id)
text = response.text
"""
def __init__(self, http_client: AsyncClient, project_id: str):
"""Initialize the resource client.
Args:
http_client: HTTPX AsyncClient for making requests
project_id: Project external_id (UUID) for API calls
"""
self.http_client = http_client
self.project_id = project_id
self._base_path = f"/v2/projects/{project_id}/resource"
async def read(
self,
entity_id: str,
*,
page: Optional[int] = None,
page_size: Optional[int] = None,
) -> Response:
"""Read a resource by entity ID.
Args:
entity_id: Entity external_id (UUID)
page: Optional page number for paginated content
page_size: Optional page size for paginated content
Returns:
Raw HTTP Response (caller handles text/binary content)
Raises:
ToolError: If the resource is not found or request fails
"""
params: dict = {}
if page is not None:
params["page"] = page
if page_size is not None:
params["page_size"] = page_size
return await call_get(
self.http_client,
f"{self._base_path}/{entity_id}",
params=params if params else None,
)
+65
View File
@@ -0,0 +1,65 @@
"""Typed client for search API operations.
Encapsulates all /v2/projects/{project_id}/search/* endpoints.
"""
from typing import Any
from httpx import AsyncClient
from basic_memory.mcp.tools.utils import call_post
from basic_memory.schemas.search import SearchResponse
class SearchClient:
"""Typed client for search operations.
Centralizes:
- API path construction for /v2/projects/{project_id}/search/*
- Response validation via Pydantic models
- Consistent error handling through call_* utilities
Usage:
async with get_client() as http_client:
client = SearchClient(http_client, project_id)
results = await client.search(search_query.model_dump())
"""
def __init__(self, http_client: AsyncClient, project_id: str):
"""Initialize the search client.
Args:
http_client: HTTPX AsyncClient for making requests
project_id: Project external_id (UUID) for API calls
"""
self.http_client = http_client
self.project_id = project_id
self._base_path = f"/v2/projects/{project_id}/search"
async def search(
self,
query: dict[str, Any],
*,
page: int = 1,
page_size: int = 10,
) -> SearchResponse:
"""Search across all content in the knowledge base.
Args:
query: Search query dict (from SearchQuery.model_dump())
page: Page number (1-indexed)
page_size: Results per page
Returns:
SearchResponse with results and pagination
Raises:
ToolError: If the request fails
"""
response = await call_post(
self.http_client,
f"{self._base_path}/",
json=query,
params={"page": page, "page_size": page_size},
)
return SearchResponse.model_validate(response.json())
+110
View File
@@ -0,0 +1,110 @@
"""MCP composition root for Basic Memory.
This container owns reading ConfigManager and environment variables for the
MCP server entrypoint. Downstream modules receive config/dependencies explicitly
rather than reading globals.
Design principles:
- Only this module reads ConfigManager directly
- Runtime mode (cloud/local/test) is resolved here
- File sync decisions are centralized here
"""
from dataclasses import dataclass
from typing import TYPE_CHECKING
from basic_memory.config import BasicMemoryConfig, ConfigManager
from basic_memory.runtime import RuntimeMode, resolve_runtime_mode
if TYPE_CHECKING: # pragma: no cover
from basic_memory.sync import SyncCoordinator
@dataclass
class McpContainer:
"""Composition root for the MCP server entrypoint.
Holds resolved configuration and runtime context.
Created once at server startup, then used to wire dependencies.
"""
config: BasicMemoryConfig
mode: RuntimeMode
@classmethod
def create(cls) -> "McpContainer":
"""Create container by reading ConfigManager.
This is the single point where MCP reads global config.
"""
config = ConfigManager().config
mode = resolve_runtime_mode(
cloud_mode_enabled=config.cloud_mode_enabled,
is_test_env=config.is_test_env,
)
return cls(config=config, mode=mode)
# --- Runtime Mode Properties ---
@property
def should_sync_files(self) -> bool:
"""Whether local file sync should be started.
Sync is enabled when:
- sync_changes is True in config
- Not in test mode (tests manage their own sync)
- Not in cloud mode (cloud handles sync differently)
"""
return (
self.config.sync_changes and not self.mode.is_test and not self.mode.is_cloud
)
@property
def sync_skip_reason(self) -> str | None:
"""Reason why sync is skipped, or None if sync should run.
Useful for logging why sync was disabled.
"""
if self.mode.is_test:
return "Test environment detected"
if self.mode.is_cloud:
return "Cloud mode enabled"
if not self.config.sync_changes:
return "Sync changes disabled"
return None
def create_sync_coordinator(self) -> "SyncCoordinator":
"""Create a SyncCoordinator with this container's settings.
Returns:
SyncCoordinator configured for this runtime environment
"""
# Deferred import to avoid circular dependency
from basic_memory.sync import SyncCoordinator
return SyncCoordinator(
config=self.config,
should_sync=self.should_sync_files,
skip_reason=self.sync_skip_reason,
)
# Module-level container instance (set by lifespan)
_container: McpContainer | None = None
def get_container() -> McpContainer:
"""Get the current MCP container.
Raises:
RuntimeError: If container hasn't been initialized
"""
if _container is None:
raise RuntimeError("MCP container not initialized. Call set_container() first.")
return _container
def set_container(container: McpContainer) -> None:
"""Set the MCP container (called by lifespan)."""
global _container
_container = container
+46 -32
View File
@@ -2,9 +2,12 @@
Provides project lookup utilities for MCP tools.
Handles project validation and context management in one place.
Note: This module uses ProjectResolver for unified project resolution.
The resolve_project_parameter function is a thin wrapper for backwards
compatibility with existing MCP tools.
"""
import os
from typing import Optional, List
from httpx import AsyncClient
from httpx._types import (
@@ -14,16 +17,26 @@ from loguru import logger
from fastmcp import Context
from basic_memory.config import ConfigManager
from basic_memory.mcp.tools.utils import call_get
from basic_memory.project_resolver import ProjectResolver
from basic_memory.schemas.project_info import ProjectItem, ProjectList
from basic_memory.utils import generate_permalink
async def resolve_project_parameter(project: Optional[str] = None) -> Optional[str]:
async def resolve_project_parameter(
project: Optional[str] = None,
allow_discovery: bool = False,
cloud_mode: Optional[bool] = None,
default_project_mode: Optional[bool] = None,
default_project: Optional[str] = None,
) -> Optional[str]:
"""Resolve project parameter using three-tier hierarchy.
if config.cloud_mode:
project is required
This is a thin wrapper around ProjectResolver for backwards compatibility.
New code should consider using ProjectResolver directly for more detailed
resolution information.
if cloud_mode:
project is required (unless allow_discovery=True for tools that support discovery mode)
else:
Resolution order:
1. Single Project Mode (--project cli arg, or BASIC_MEMORY_MCP_PROJECT env var) - highest priority
@@ -32,41 +45,39 @@ async def resolve_project_parameter(project: Optional[str] = None) -> Optional[s
Args:
project: Optional explicit project parameter
allow_discovery: If True, allows returning None in cloud mode for discovery mode
(used by tools like recent_activity that can operate across all projects)
cloud_mode: Optional explicit cloud mode. If not provided, reads from ConfigManager.
default_project_mode: Optional explicit default project mode. If not provided, reads from ConfigManager.
default_project: Optional explicit default project. If not provided, reads from ConfigManager.
Returns:
Resolved project name or None if no resolution possible
"""
# Load config for any values not explicitly provided
if cloud_mode is None or default_project_mode is None or default_project is None:
config = ConfigManager().config
if cloud_mode is None:
cloud_mode = config.cloud_mode
if default_project_mode is None:
default_project_mode = config.default_project_mode
if default_project is None:
default_project = config.default_project
config = ConfigManager().config
# if cloud_mode, project is required
if config.cloud_mode:
if project:
logger.debug(f"project: {project}, cloud_mode: {config.cloud_mode}")
return project
else:
raise ValueError("No project specified. Project is required for cloud mode.")
# Priority 1: CLI constraint overrides everything (--project arg sets env var)
constrained_project = os.environ.get("BASIC_MEMORY_MCP_PROJECT")
if constrained_project:
logger.debug(f"Using CLI constrained project: {constrained_project}")
return constrained_project
# Priority 2: Explicit project parameter
if project:
logger.debug(f"Using explicit project parameter: {project}")
return project
# Priority 3: Default project mode
if config.default_project_mode:
logger.debug(f"Using default project from config: {config.default_project}")
return config.default_project
# No resolution possible
return None
# Create resolver with configuration and resolve
resolver = ProjectResolver.from_env(
cloud_mode=cloud_mode,
default_project_mode=default_project_mode,
default_project=default_project,
)
result = resolver.resolve(project=project, allow_discovery=allow_discovery)
return result.project
async def get_project_names(client: AsyncClient, headers: HeaderTypes | None = None) -> List[str]:
# Deferred import to avoid circular dependency with tools
from basic_memory.mcp.tools.utils import call_get
response = await call_get(client, "/projects/projects", headers=headers)
project_list = ProjectList.model_validate(response.json())
return [project.name for project in project_list.projects]
@@ -92,6 +103,9 @@ async def get_active_project(
ValueError: If no project can be resolved
HTTPError: If project doesn't exist or is inaccessible
"""
# Deferred import to avoid circular dependency with tools
from basic_memory.mcp.tools.utils import call_get
resolved_project = await resolve_project_parameter(project)
if not resolved_project:
project_names = await get_project_names(client, headers)
@@ -32,7 +32,7 @@ def ai_assistant_guide() -> str:
# Add mode-specific header
mode_info = ""
if config.default_project_mode:
if config.default_project_mode: # pragma: no cover
mode_info = f"""
# 🎯 Default Project Mode Active
@@ -46,7 +46,7 @@ def ai_assistant_guide() -> str:
"""
else:
else: # pragma: no cover
mode_info = """
# 🔧 Multi-Project Mode Active
@@ -64,7 +64,7 @@ async def recent_activity_prompt(
primary_results.append(item.primary_result)
# Add up to 1 related result per primary item
if item.related_results:
related_results.extend(item.related_results[:1])
related_results.extend(item.related_results[:1]) # pragma: no cover
# Limit total results for readability
primary_results = primary_results[:8]
@@ -78,7 +78,7 @@ async def recent_activity_prompt(
primary_results.append(item.primary_result)
# Add up to 2 related results per primary item
if item.related_results:
related_results.extend(item.related_results[:2])
related_results.extend(item.related_results[:2]) # pragma: no cover
# Set topic based on mode
if project:
+3 -3
View File
@@ -123,9 +123,9 @@ def format_prompt_context(context: PromptContext) -> str:
# Add content snippet
if hasattr(primary, "content") and primary.content: # pyright: ignore
content = primary.content or "" # pyright: ignore
if content:
section += f"\n**Excerpt**:\n{content}\n"
content = primary.content or "" # pyright: ignore # pragma: no cover
if content: # pragma: no cover
section += f"\n**Excerpt**:\n{content}\n" # pragma: no cover
section += dedent(f"""
+58
View File
@@ -2,8 +2,66 @@
Basic Memory FastMCP server.
"""
from contextlib import asynccontextmanager
from fastmcp import FastMCP
from loguru import logger
from basic_memory import db
from basic_memory.mcp.container import McpContainer, set_container
from basic_memory.services.initialization import initialize_app
from basic_memory.telemetry import show_notice_if_needed, track_app_started
@asynccontextmanager
async def lifespan(app: FastMCP):
"""Lifecycle manager for the MCP server.
Handles:
- Database initialization and migrations
- Telemetry notice and tracking
- File sync via SyncCoordinator (if enabled and not in cloud mode)
- Proper cleanup on shutdown
"""
# --- Composition Root ---
# Create container and read config (single point of config access)
container = McpContainer.create()
set_container(container)
logger.info(f"Starting Basic Memory MCP server (mode={container.mode.name})")
# Show telemetry notice (first run only) and track startup
show_notice_if_needed()
track_app_started("mcp")
# Track if we created the engine (vs test fixtures providing it)
# This prevents disposing an engine provided by test fixtures when
# multiple Client connections are made in the same test
engine_was_none = db._engine is None
# Initialize app (runs migrations, reconciles projects)
await initialize_app(container.config)
# Create and start sync coordinator (lifecycle centralized in coordinator)
sync_coordinator = container.create_sync_coordinator()
await sync_coordinator.start()
try:
yield
finally:
# Shutdown - coordinator handles clean task cancellation
logger.info("Shutting down Basic Memory MCP server")
await sync_coordinator.stop()
# Only shutdown DB if we created it (not if test fixture provided it)
if engine_was_none:
await db.shutdown_db()
logger.info("Database connections closed")
else: # pragma: no cover
logger.debug("Skipping DB shutdown - engine provided externally")
mcp = FastMCP(
name="Basic Memory",
lifespan=lifespan,
)
+13 -13
View File
@@ -8,7 +8,7 @@ from fastmcp import Context
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.project_context import get_active_project
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_get
from basic_memory.telemetry import track_mcp_tool
from basic_memory.schemas.base import TimeFrame
from basic_memory.schemas.memory import (
GraphContext,
@@ -87,6 +87,7 @@ async def build_context(
Raises:
ToolError: If project doesn't exist or depth parameter is invalid
"""
track_mcp_tool("build_context")
logger.info(f"Building context from {url} in project {project}")
# Convert string depth to integer if needed
@@ -104,17 +105,16 @@ async def build_context(
# Get the active project using the new stateless approach
active_project = await get_active_project(client, project, context)
project_url = active_project.project_url
# Import here to avoid circular import
from basic_memory.mcp.clients import MemoryClient
response = await call_get(
client,
f"{project_url}/memory/{memory_url_path(url)}",
params={
"depth": depth,
"timeframe": timeframe,
"page": page,
"page_size": page_size,
"max_related": max_related,
},
# Use typed MemoryClient for API calls
memory_client = MemoryClient(client, active_project.external_id)
return await memory_client.build_context(
memory_url_path(url),
depth=depth or 1,
timeframe=timeframe,
page=page,
page_size=page_size,
max_related=max_related,
)
return GraphContext.model_validate(response.json())
+34 -12
View File
@@ -12,7 +12,8 @@ from fastmcp import Context
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.project_context import get_active_project
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_put
from basic_memory.mcp.tools.utils import call_put, call_post, resolve_entity_id
from basic_memory.telemetry import track_mcp_tool
@mcp.tool(
@@ -94,9 +95,9 @@ async def canvas(
Raises:
ToolError: If project doesn't exist or folder path is invalid
"""
track_mcp_tool("canvas")
async with get_client() as client:
active_project = await get_active_project(client, project, context)
project_url = active_project.project_url
# Ensure path has .canvas extension
file_title = title if title.endswith(".canvas") else f"{title}.canvas"
@@ -108,23 +109,44 @@ async def canvas(
# Convert to JSON
canvas_json = json.dumps(canvas_data, indent=2)
# Write the file using the resource API
# Try to create the canvas file first (optimistic create)
logger.info(f"Creating canvas file: {file_path} in project {project}")
# Send canvas_json as content string, not as json parameter
# The resource endpoint expects Body() string content, not JSON-encoded data
response = await call_put(
client,
f"{project_url}/resource/{file_path}",
content=canvas_json,
headers={"Content-Type": "text/plain"},
)
try:
response = await call_post(
client,
f"/v2/projects/{active_project.external_id}/resource",
json={"file_path": file_path, "content": canvas_json},
)
action = "Created"
except Exception as e:
# If creation failed due to conflict (already exists), try to update
if (
"409" in str(e)
or "conflict" in str(e).lower()
or "already exists" in str(e).lower()
):
logger.info(f"Canvas file exists, updating instead: {file_path}")
try:
entity_id = await resolve_entity_id(client, active_project.external_id, file_path)
# For update, send content in JSON body
response = await call_put(
client,
f"/v2/projects/{active_project.external_id}/resource/{entity_id}",
json={"content": canvas_json},
)
action = "Updated"
except Exception as update_error: # pragma: no cover
# Re-raise the original error if update also fails
raise e from update_error # pragma: no cover
else:
# Re-raise if it's not a conflict error
raise # pragma: no cover
# Parse response
result = response.json()
logger.debug(result)
# Build summary
action = "Created" if response.status_code == 201 else "Updated"
summary = [f"# {action}: {file_path}", "\nThe canvas is ready to open in Obsidian."]
return "\n".join(summary)

Some files were not shown because too many files have changed in this diff Show More