Compare commits

...

72 Commits

Author SHA1 Message Date
phernandez 099c334e3d chore: update version to 0.16.1 for v0.16.1 release 2025-11-11 09:21:47 -06:00
phernandez 7685586178 docs: Add v0.16.1 CHANGELOG entry for Windows line ending fix
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-11 09:10:26 -06:00
Paul Hernandez e9d0a944a9 fix: Handle Windows line endings in rclone bisync (#422)
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-11 09:08:20 -06:00
phernandez caf3c14bb1 chore: update version to 0.16.0 for v0.16.0 release 2025-11-10 19:19:48 -06:00
phernandez c5d9067754 docs: Add v0.16.0 CHANGELOG entry with comprehensive release notes
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-10 19:17:29 -06:00
phernandez e0fc59ea97 style: Format upload.py for better readability
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-10 19:15:03 -06:00
Paul Hernandez 49b2adc35c fix: skip archive files during cloud upload (#420)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-11-10 19:04:49 -06:00
Paul Hernandez 1646572f69 fix: Rename write_note entity_type to note_type for clarity (#419)
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-11-10 19:02:50 -06:00
Paul Hernandez f0d7398815 fix: Quote string values in YAML frontmatter to handle special characters (#418)
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-11-10 18:13:48 -06:00
Paul Hernandez 581b7b17c6 fix: Add explicit type annotations to MCP tool parameters (#394)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-11-10 17:02:17 -06:00
Paul Hernandez d775f7bab9 fix: Simplify search_notes schema by removing Optional wrappers (#395)
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-11-10 16:30:59 -06:00
Drew Cain fc01f6abaf fix: Replace Unicode arrows with ASCII for Windows compatibility (#414) 2025-11-10 16:30:41 -06:00
Paul Hernandez 4614fd09d5 fix: Handle dict objects in write_resource endpoint (#415)
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-11-10 16:30:27 -06:00
phernandez 0d4ad7bbf0 remove v0.15.0 info from assistant-guide
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-10 16:25:49 -06:00
jope-bm 7ccec7eba2 feat: Add run_in_background parameter to sync endpoint with tests (#417)
Co-authored-by: Claude <noreply@anthropic.com>
2025-11-07 09:04:43 -07:00
Paul Hernandez 021af74545 fix: Strip duplicate headers in edit_note replace_section (#396)
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Drew Cain <groksrc@users.noreply.github.com>
2025-11-02 14:20:11 -06:00
phernandez 2ad0ee9d5d fix: Use force_full=true for database sync after project sync/bisync
After rclone synchronizes files between local and cloud storage, the
database needs to perform a full scan to ensure it captures all changes.
Previously, incremental sync (watermark optimization) could miss files
that were changed remotely.

Changes:
- project sync command now calls /project/sync?force_full=true
- project bisync command now calls /project/sync?force_full=true
- Ensures complete database refresh after file synchronization

This guarantees the database is fully in sync with the filesystem
after any rclone sync or bisync operation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-02 13:26:08 -06:00
Brandon Mayes c9946ecf1e fix: Various rclone fixes for cloud sync on Windows (#410)
Signed-off-by: Brandon Mayes <5610870+bdmayes@users.noreply.github.com>
Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: phernandez <paul@basicmachines.co>
2025-11-02 11:26:57 -06:00
Drew Cain 0ba6f219f1 fix: Windows CLI Unicode encoding errors (#411)
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
2025-11-02 10:20:48 -06:00
Paul Hernandez 0b3272ae6e feat: SPEC-20 Simplified Project-Scoped Rclone Sync (#405)
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
2025-11-02 09:35:26 -06:00
Paul Hernandez a7d7cc5ee6 fix: Normalize YAML frontmatter types to prevent AttributeError (#236) (#402)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-27 09:19:50 -05:00
Paul Hernandez a7e696b039 Add free trial information to README
Added information about a 7-day free trial.

Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
2025-10-24 10:05:11 -05:00
Paul Hernandez 8aaddb6d45 Add free trial information to README
Added information about a 7-day free trial to the README.

Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
2025-10-24 10:04:41 -05:00
Paul Hernandez d7565312fc Announce Basic Memory Cloud launch in README
Added a section announcing the launch of Basic Memory Cloud with details on cross-device support and early supporter pricing.

Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
2025-10-24 09:53:14 -05:00
Paul Hernandez c7e6eab02f feat: Add delete_notes parameter to remove project endpoint (#391)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-21 14:09:20 -05:00
Paul Hernandez bb8da31472 fix: Handle null, empty, and string 'None' title in markdown frontmatter (#387) (#389)
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-10-21 09:29:19 -05:00
Paul Hernandez e78345ff25 feat: Streaming Foundation & Async I/O Consolidation (SPEC-19) (#384)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-21 09:03:59 -05:00
Paul Hernandez 32236cd247 fix: Handle YAML parsing errors gracefully in update_frontmatter (#378) (#379)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-16 20:17:58 -05:00
Paul Hernandez 4fd6d0c648 fix: Optimize sync memory usage to prevent OOM on large projects (#380)
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-10-16 20:17:34 -05:00
Paul Hernandez e6c8e3662c fix: preserve mtime webdav upload 376 (#377)
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-16 17:18:10 -05:00
Paul Hernandez 449b62d947 fix: Prevent deleted projects from being recreated by background sync (#193) (#370)
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-10-16 15:16:24 -05:00
Paul Hernandez b7497d7484 fix: Use filesystem timestamps for entity sync instead of database operation time (#138) (#369)
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-10-16 14:21:27 -05:00
Paul Hernandez d1431bdb1b fix: Handle YAML parsing errors and missing entity_type in markdown files (#368)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-16 13:18:58 -05:00
Paul Hernandez 171bef717f fix: Resolve UNIQUE constraint violation in entity upsert with observations (#187) (#367)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-16 12:30:56 -05:00
Paul Hernandez 729a5a3b8d fix: Terminate sync immediately when project is deleted (#366)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-16 11:07:12 -05:00
Paul Hernandez 434cdf24dd feat: Add circuit breaker for file sync failures (#364)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-16 09:47:48 -05:00
Paul Hernandez 7f9c1a97a4 feat: Add --verbose and --no-gitignore options to cloud upload (#362)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-15 20:03:36 -05:00
Paul Hernandez 53fb13b054 fix: Make project creation endpoint idempotent (#357)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-15 19:27:41 -05:00
Paul Hernandez bd6c8348b8 fix: Handle None text values in Claude conversations importer (#353)
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-10-15 16:08:57 -05:00
phernandez 994c8b8e7e chore: update version to 0.15.2 for v0.15.2 release 2025-10-14 09:36:47 -05:00
phernandez a78e8c3ac5 style: Apply linter formatting changes 2025-10-14 09:34:10 -05:00
phernandez 53900c5baa fix: Project commands now respect cloud_mode at runtime
- Moved config evaluation from module load time to runtime
- Unified add_project command to handle both cloud and local modes
- Commands (default, sync-config, move) now check cloud_mode at runtime
- Fixes test failures where monkeypatch wasn't applied before command registration
2025-10-14 09:24:13 -05:00
phernandez 02c6de3387 docs: Add v0.15.2 changelog entry 2025-10-14 00:43:14 -05:00
phernandez 9ccf4b6b56 remove extra conole out from sync message after upload
Signed-off-by: phernandez <paul@basicmachines.co>
2025-10-14 00:38:02 -05:00
phernandez ba74ca7e18 fix: Update CloudProjectCreateResponse schema to match API response
The /proxy/projects/projects POST endpoint returns a ProjectStatusResponse
with fields: message, status, default, old_project, new_project.

Updated CloudProjectCreateResponse schema to match this format instead of
expecting name, path, message fields.

Also updated all related tests to use the correct response format:
- tests/cli/test_cloud_utils.py (3 tests)
- tests/cli/test_bisync_commands.py (1 test)

Fixes the validation error when creating cloud projects via upload command:
"bm cloud upload --project test --create-project specs"

Signed-off-by: Pablo Hernandez <pablo@basicmachines.co>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-10-13 23:56:59 -05:00
Paul Hernandez 5258f45730 feat: Add WebDAV upload command for cloud projects (#356)
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Pablo Hernandez <pablo@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-13 23:20:54 -05:00
phernandez e773c002ce chore: update version to 0.15.1 for v0.15.1 release 2025-10-13 11:04:44 -05:00
phernandez e70ba944e7 docs: Add v0.15.1 changelog entry
Add comprehensive changelog for v0.15.1 release including:
- Performance improvements (43% faster sync, 10-100x faster directory ops)
- Bug fixes for cloud mode, project paths, and Claude Desktop compatibility
- New features: async client context manager, BASIC_MEMORY_PROJECT_ROOT
- Documentation updates and SPEC-15/16 additions

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-13 10:58:21 -05:00
phernandez e41579f971 Merge branch 'main' of github.com:basicmachines-co/basic-memory 2025-10-13 10:44:54 -05:00
Paul Hernandez 2b7008d997 fix: Update view_note and ChatGPT tools for Claude Desktop compatibility (#355)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-13 10:44:35 -05:00
phernandez 56e5cc072b Merge branch 'main' of github.com:basicmachines-co/basic-memory 2025-10-13 07:30:08 -05:00
Paul Hernandez c0538ad2dd perf: Optimize sync/indexing for 43% faster performance (#352)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-12 14:41:44 -05:00
phernandez 962d88ea43 add specs-17/18
Signed-off-by: phernandez <paul@basicmachines.co>
2025-10-12 10:59:45 -05:00
phernandez cd5efd4a44 perf: exclude null fields from directory endpoint responses
Reduces JSON payload size by 50-70% for directory-heavy responses by omitting
null fields from serialization.

Changes:
- Added response_model_exclude_none=True to all directory endpoints:
  - GET /directory/tree
  - GET /directory/structure
  - GET /directory/list

Impact:
- Directory nodes no longer serialize 7 null fields (title, permalink,
  entity_id, entity_type, content_type, updated_at, file_path)
- For 50+ directories: eliminates 350+ null fields from response
- Payload reduction: ~2.3kb → ~1kb for typical directory trees
- File nodes still include all metadata when present

Example directory node output:
{
  "name": "Tools",
  "directory_path": "/Tools",
  "type": "directory",
  "children": []
}

Testing:
- All 29 directory tests passing
- Type checking passing (0 errors)
- Backward compatible (clients just see missing keys vs null)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-10-11 09:56:26 -05:00
Paul Hernandez 00b73b0d08 feat: Optimize directory operations for 10-100x performance improvement (#350)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-11 09:11:46 -05:00
jope-bm a09066e0f0 fix: Add permalink normalization to project lookups in deps.py (#348)
Signed-off-by: Joe P <joe@basicmemory.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: phernandez <paul@basicmachines.co>
2025-10-10 21:21:35 -05:00
Drew Cain be352ab474 fix: Project deletion failing with permalink normalization (#345) 2025-10-10 12:18:03 -05:00
Paul Hernandez 8d2e70cfc8 refactor: async client context manager pattern for cloud consolidation (#344)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-09 19:09:47 -05:00
Paul Hernandez 53438d1eab feat: Add SPEC-15 for configuration persistence via Tigris (#343)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-08 18:07:00 -05:00
phernandez 032de7e3f2 fix: formatting in test file
Signed-off-by: phernandez <paul@basicmachines.co>
2025-10-08 09:13:05 -05:00
phernandez fd2b188645 Revert "feat: add optional logfire instrumentation for cloud mode distributed tracing"
This reverts commit 1fa93ecbd2.
2025-10-08 09:08:15 -05:00
phernandez 453cba94e4 Revert "fix: instrument httpx client at module level for MCP context"
This reverts commit 48cb4be4cd.
2025-10-08 09:07:26 -05:00
phernandez 48cb4be4cd fix: instrument httpx client at module level for MCP context
The lifespan-based instrumentation only runs when FastAPI app starts.
In MCP context, the app never starts but the httpx client is still used.

Solution: Instrument the client immediately after creation at module level.
This works in both contexts:
- MCP: client is instrumented when module is imported
- API: client is instrumented before lifespan runs (lifespan still safe)

This enables distributed tracing from MCP -> Cloud -> API.

Signed-off-by: phernandez <paul@basicmachines.co>
2025-10-08 08:10:54 -05:00
Paul Hernandez 3e876a7549 fix: correct ProjectItem.home property to return path instead of name (#341)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-08 01:03:12 -05:00
phernandez 1fa93ecbd2 feat: add optional logfire instrumentation for cloud mode distributed tracing 2025-10-08 00:23:00 -05:00
Paul Hernandez 73202d1aab fix: add tool use doc to write note for using empty string for root folder (#339)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-07 23:48:02 -05:00
Paul Hernandez 795e339333 fix: prevent nested project paths to avoid data conflicts (#338)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-07 09:44:39 -05:00
Paul Hernandez 07e304ce8e fix: normalize paths to lowercase in cloud mode to prevent case collisions (#336)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-05 17:56:58 -05:00
phernandez 2a1c06d9ad fix link in ai_assistant_guide resource
Signed-off-by: phernandez <paul@basicmachines.co>
2025-10-05 17:26:14 -05:00
Paul Hernandez c6f93a0294 chore: v0.15.0 assistant guide (#335)
Signed-off-by: phernandez <paul@basicmachines.co>
2025-10-05 17:20:54 -05:00
Paul Hernandez ccc4386627 feat: introduce BASIC_MEMORY_PROJECT_ROOT for path constraints (#334)
Signed-off-by: phernandez <paul@basicmachines.co>
2025-10-05 10:42:06 -05:00
Paul Hernandez 7616b2bb08 fix: cloud mode path validation and sanitization (bmc-issue-103) (#332)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-04 22:10:21 -05:00
162 changed files with 31471 additions and 7433 deletions
+4 -1
View File
@@ -71,9 +71,12 @@ jobs:
- [ ] Proper error handling and logging
- [ ] Performance considerations addressed
- [ ] No sensitive data in logs or commits
## Compatability
- [ ] File path comparisons must be windows compatible
- [ ] Avoid using emojis and unicode characters in console and log output
Read the CLAUDE.md file for detailed project context. For each checklist item, verify if it's satisfied and comment on any that need attention. Use inline comments for specific code issues and post a summary with checklist results.
# Allow broader tool access for thorough code review
claude_args: '--allowed-tools "Bash(gh pr:*),Bash(gh issue:*),Bash(gh api:*),Bash(git log:*),Bash(git show:*),Read,Grep,Glob"'
-4
View File
@@ -60,10 +60,6 @@ jobs:
run: |
just typecheck
- name: Run type checks
run: |
just typecheck
- name: Run linting
run: |
just lint
+445 -1
View File
@@ -1,5 +1,449 @@
# CHANGELOG
## v0.16.1 (2025-11-11)
### Bug Fixes
- **#422**: Handle Windows line endings in rclone bisync
([`e9d0a94`](https://github.com/basicmachines-co/basic-memory/commit/e9d0a94))
- Added `--compare=modtime` flag to rclone bisync to ignore size differences from line ending conversions
- Fixes issue where LF→CRLF conversion on Windows was treated as file corruption
- Resolves "corrupted on transfer: sizes differ" errors during cloud sync on Windows
- Users will need to run `--resync` once after updating to establish new baseline
## v0.16.0 (2025-11-10)
### Features
- **#417**: Add run_in_background parameter to sync endpoint
([`7ccec7e`](https://github.com/basicmachines-co/basic-memory/commit/7ccec7e))
- New `run_in_background` parameter for async sync operations
- Improved API flexibility for long-running sync tasks
- Comprehensive test coverage for background sync behavior
- **#405**: SPEC-20 Simplified Project-Scoped Rclone Sync
([`0b3272a`](https://github.com/basicmachines-co/basic-memory/commit/0b3272a))
- Simplified and more reliable cloud synchronization
- Project-scoped rclone configuration
- Better error handling and status reporting
- **#384**: Streaming Foundation & Async I/O Consolidation (SPEC-19)
([`e78345f`](https://github.com/basicmachines-co/basic-memory/commit/e78345f))
- Foundation for streaming support in future releases
- Consolidated async I/O patterns across codebase
- Improved performance and resource management
- **#364**: Add circuit breaker for file sync failures
([`434cdf2`](https://github.com/basicmachines-co/basic-memory/commit/434cdf2))
- Prevents cascading failures during sync operations
- Automatic recovery from transient errors
- Better resilience in cloud sync scenarios
- **#362**: Add --verbose and --no-gitignore options to cloud upload
([`7f9c1a9`](https://github.com/basicmachines-co/basic-memory/commit/7f9c1a9))
- Enhanced upload control with verbose logging
- Option to bypass gitignore filtering when needed
- Better debugging and troubleshooting capabilities
- **#391**: Add delete_notes parameter to remove project endpoint
([`c9946ec`](https://github.com/basicmachines-co/basic-memory/commit/c9946ec))
- Option to delete notes when removing projects
- Safer project cleanup workflows
- Prevents accidental data loss
### Bug Fixes
- **#420**: Skip archive files during cloud upload
([`49b2adc`](https://github.com/basicmachines-co/basic-memory/commit/49b2adc))
- Prevents uploading of zip, tar, gz and other archive files
- Reduces storage usage and upload time
- Better file filtering during cloud operations
- **#419**: Rename write_note entity_type to note_type for clarity
([`1646572`](https://github.com/basicmachines-co/basic-memory/commit/1646572))
- Clearer parameter naming in write_note tool
- Improved API consistency and documentation
- Better developer experience
- **#418**: Quote string values in YAML frontmatter to handle special characters
([`f0d7398`](https://github.com/basicmachines-co/basic-memory/commit/f0d7398))
- Fixes YAML parsing errors with special characters
- More robust frontmatter handling
- Prevents data corruption in edge cases
- **#415**: Handle dict objects in write_resource endpoint
([`4614fd0`](https://github.com/basicmachines-co/basic-memory/commit/4614fd0))
- Fixes errors when writing dictionary resources
- Better type handling in resource endpoints
- Improved API robustness
- **#414**: Replace Unicode arrows with ASCII for Windows compatibility
([`fc01f6a`](https://github.com/basicmachines-co/basic-memory/commit/fc01f6a))
- Fixes display issues on Windows terminals
- Better cross-platform compatibility
- Improved CLI user experience on Windows
- **#411**: Windows CLI Unicode encoding errors
([`0ba6f21`](https://github.com/basicmachines-co/basic-memory/commit/0ba6f21))
- Resolves Unicode encoding issues on Windows
- Better handling of international characters
- Improved Windows platform support
- **#410**: Various rclone fixes for cloud sync on Windows
([`c9946ec`](https://github.com/basicmachines-co/basic-memory/commit/c9946ec))
- Fixes cloud sync reliability on Windows
- Better path handling for Windows filesystem
- Improved rclone integration on Windows
- **#402**: Normalize YAML frontmatter types to prevent AttributeError
([`a7d7cc5`](https://github.com/basicmachines-co/basic-memory/commit/a7d7cc5))
- Fixes AttributeError when reading frontmatter
- More robust type normalization
- Better error handling in markdown parsing
- **#396**: Strip duplicate headers in edit_note replace_section
([`021af74`](https://github.com/basicmachines-co/basic-memory/commit/021af74))
- Prevents duplicate headers when replacing sections
- Cleaner note editing behavior
- Better content consistency
- **#395**: Simplify search_notes schema by removing Optional wrappers
([`d775f7b`](https://github.com/basicmachines-co/basic-memory/commit/d775f7b))
- Cleaner API schema definition
- Better type safety and validation
- Improved developer experience
- **#394**: Add explicit type annotations to MCP tool parameters
([`581b7b1`](https://github.com/basicmachines-co/basic-memory/commit/581b7b1))
- Better type safety in MCP tools
- Improved IDE support and autocomplete
- Clearer tool documentation
- **#389**: Handle null, empty, and string 'None' title in markdown frontmatter
([`bb8da31`](https://github.com/basicmachines-co/basic-memory/commit/bb8da31))
- Fixes errors with malformed frontmatter titles
- More robust title handling
- Better error recovery
- **#380**: Optimize sync memory usage to prevent OOM on large projects
([`4fd6d0c`](https://github.com/basicmachines-co/basic-memory/commit/4fd6d0c))
- Prevents out-of-memory errors on large knowledge bases
- Better memory management during sync
- Improved scalability
- **#379**: Handle YAML parsing errors gracefully in update_frontmatter
([`32236cd`](https://github.com/basicmachines-co/basic-memory/commit/32236cd))
- Better error handling for malformed YAML
- Graceful degradation instead of crashes
- Improved robustness
- **#377**: Preserve mtime on WebDAV upload
([`e6c8e36`](https://github.com/basicmachines-co/basic-memory/commit/e6c8e36))
- Maintains file modification times during upload
- Better sync accuracy
- Prevents unnecessary re-syncing
- **#370**: Prevent deleted projects from being recreated by background sync
([`449b62d`](https://github.com/basicmachines-co/basic-memory/commit/449b62d))
- Fixes race condition with project deletion
- Better lifecycle management
- Prevents unwanted project recreation
- **#369**: Use filesystem timestamps for entity sync instead of database operation time
([`b7497d7`](https://github.com/basicmachines-co/basic-memory/commit/b7497d7))
- More accurate sync detection
- Better handling of external file modifications
- Improved sync reliability
- **#368**: Handle YAML parsing errors and missing entity_type in markdown files
([`d1431bd`](https://github.com/basicmachines-co/basic-memory/commit/d1431bd))
- Better error handling for malformed markdown
- Graceful handling of missing metadata
- Improved robustness
- **#367**: Resolve UNIQUE constraint violation in entity upsert with observations
([`171bef7`](https://github.com/basicmachines-co/basic-memory/commit/171bef7))
- Fixes database constraint errors during sync
- Better handling of duplicate observations
- Improved data integrity
- **#366**: Terminate sync immediately when project is deleted
([`729a5a3`](https://github.com/basicmachines-co/basic-memory/commit/729a5a3))
- Faster project deletion
- Better resource cleanup
- Improved user experience
- **#357**: Make project creation endpoint idempotent
([`53fb13b`](https://github.com/basicmachines-co/basic-memory/commit/53fb13b))
- Prevents errors when creating existing projects
- Better API reliability
- Improved cloud integration
- **#353**: Handle None text values in Claude conversations importer
([`bd6c834`](https://github.com/basicmachines-co/basic-memory/commit/bd6c834))
- Fixes import errors with empty messages
- Better error handling in importers
- Improved data migration
### Performance Improvements
- Force full database sync after project sync/bisync operations
([`2ad0ee9`](https://github.com/basicmachines-co/basic-memory/commit/2ad0ee9))
- Ensures database consistency after cloud operations
- Better sync reliability
- Improved data integrity
### Documentation
- Add free trial information to README
([`a7d7cc5`](https://github.com/basicmachines-co/basic-memory/commit/a7d7cc5), [`8aaddb6`](https://github.com/basicmachines-co/basic-memory/commit/8aaddb6))
- Updated README with Basic Memory Cloud trial info
- Better onboarding experience
- Clearer pricing information
- Announce Basic Memory Cloud launch in README
([`d756531`](https://github.com/basicmachines-co/basic-memory/commit/d756531))
- Official cloud service announcement
- Updated documentation for cloud features
- Improved product positioning
### Migration Guide
No manual migration required. Upgrade with:
```bash
# Update via uv
uv tool upgrade basic-memory
# Or install fresh
uv tool install basic-memory
```
**What's New in v0.16.0:**
- Streaming foundation and consolidated async I/O (SPEC-19)
- Simplified project-scoped rclone sync (SPEC-20)
- Circuit breaker for sync failure resilience
- Enhanced Windows platform support
- Improved cloud upload with verbose and gitignore options
- Better error handling across YAML parsing and frontmatter
- Memory optimization for large projects
- Archive file filtering during upload
**Breaking Changes:**
- `write_note` parameter renamed: `entity_type``note_type` for clarity
### Installation
```bash
# Latest stable release
uv tool install basic-memory
# Update existing installation
uv tool upgrade basic-memory
# Docker
docker pull ghcr.io/basicmachines-co/basic-memory:v0.16.0
```
## v0.15.2 (2025-10-14)
### Features
- **#356**: Add WebDAV upload command for cloud projects
([`5258f45`](https://github.com/basicmachines-co/basic-memory/commit/5258f457))
- New `bm cloud upload` command for uploading local files/directories to cloud projects
- WebDAV-based file transfer with automatic directory creation
- Support for `.gitignore` and `.bmignore` pattern filtering
- Automatic project creation with `--create-project` flag
- Optional post-upload sync with `--sync` flag (enabled by default)
- Human-readable file size reporting (bytes, KB, MB)
- Comprehensive test coverage (28 unit tests)
### Migration Guide
No manual migration required. Upgrade with:
```bash
# Update via uv
uv tool upgrade basic-memory
# Or install fresh
uv tool install basic-memory
```
**What's New:**
- Upload local files to cloud projects with `bm cloud upload`
- Streamlined cloud project creation and management
- Better file filtering with gitignore integration
### Installation
```bash
# Latest stable release
uv tool install basic-memory
# Update existing installation
uv tool upgrade basic-memory
# Docker
docker pull ghcr.io/basicmachines-co/basic-memory:v0.15.2
```
## v0.15.1 (2025-10-13)
### Performance Improvements
- **#352**: Optimize sync/indexing for 43% faster performance
([`c0538ad`](https://github.com/basicmachines-co/basic-memory/commit/c0538ad2perf0d68a2a3604e255c3f2c42c5ed))
- Significant performance improvements to file synchronization and indexing operations
- 43% reduction in sync time for large knowledge bases
- Optimized database queries and file processing
- **#350**: Optimize directory operations for 10-100x performance improvement
([`00b73b0`](https://github.com/basicmachines-co/basic-memory/commit/00b73b0d))
- Dramatic performance improvements for directory listing operations
- 10-100x faster directory traversal depending on knowledge base size
- Reduced memory footprint for large directory structures
- Exclude null fields from directory endpoint responses for smaller payloads
### Bug Fixes
- **#355**: Update view_note and ChatGPT tools for Claude Desktop compatibility
([`2b7008d`](https://github.com/basicmachines-co/basic-memory/commit/2b7008d9))
- Fix view_note tool formatting for better Claude Desktop rendering
- Update ChatGPT tool integration for improved compatibility
- Enhanced artifact display in Claude Desktop interface
- **#348**: Add permalink normalization to project lookups in deps.py
([`a09066e`](https://github.com/basicmachines-co/basic-memory/commit/a09066e0))
- Fix project lookup failures due to case sensitivity
- Normalize permalinks consistently across project operations
- Improve project switching reliability
- **#345**: Project deletion failing with permalink normalization
([`be352ab`](https://github.com/basicmachines-co/basic-memory/commit/be352ab4))
- Fix project deletion errors related to permalink handling
- Ensure proper cleanup of project resources
- Improve error messages for deletion failures
- **#341**: Correct ProjectItem.home property to return path instead of name
([`3e876a7`](https://github.com/basicmachines-co/basic-memory/commit/3e876a75))
- Fix ProjectItem.home to return correct project path
- Resolve configuration issues with project paths
- Improve project path resolution consistency
- **#339**: Prevent nested project paths to avoid data conflicts
([`795e339`](https://github.com/basicmachines-co/basic-memory/commit/795e3393))
- Block creation of nested project paths that could cause data conflicts
- Add validation to prevent project path hierarchy issues
- Improve error messages for invalid project configurations
- **#338**: Normalize paths to lowercase in cloud mode to prevent case collisions
([`07e304c`](https://github.com/basicmachines-co/basic-memory/commit/07e304ce))
- Fix path case sensitivity issues in cloud deployments
- Normalize paths consistently across cloud operations
- Prevent data loss from case-insensitive filesystem collisions
- **#336**: Cloud mode path validation and sanitization (bmc-issue-103)
([`2a1c06d`](https://github.com/basicmachines-co/basic-memory/commit/2a1c06d9))
- Enhanced path validation for cloud deployments
- Improved path sanitization to prevent security issues
- Better error handling for invalid paths in cloud mode
- **#332**: Cloud mode path validation and sanitization
([`7616b2b`](https://github.com/basicmachines-co/basic-memory/commit/7616b2bb))
- Additional cloud mode path fixes and improvements
- Comprehensive path validation for cloud environments
- Security enhancements for path handling
### Features
- **#344**: Async client context manager pattern for cloud consolidation (SPEC-16)
([`8d2e70c`](https://github.com/basicmachines-co/basic-memory/commit/8d2e70cf))
- Refactor async client to use context manager pattern
- Improve resource management and cleanup
- Enable better dependency injection for cloud deployments
- Foundation for cloud platform consolidation
- **#343**: Add SPEC-15 for configuration persistence via Tigris
([`53438d1`](https://github.com/basicmachines-co/basic-memory/commit/53438d1e))
- Design specification for persistent configuration storage
- Foundation for cloud configuration management
- Tigris S3-compatible storage integration planning
- **#334**: Introduce BASIC_MEMORY_PROJECT_ROOT for path constraints
([`ccc4386`](https://github.com/basicmachines-co/basic-memory/commit/ccc43866))
- Add environment variable for constraining project paths
- Improve security by limiting project creation locations
- Better control over project directory structure
### Documentation
- **#335**: v0.15.0 assistant guide updates
([`c6f93a0`](https://github.com/basicmachines-co/basic-memory/commit/c6f93a02))
- Update AI assistant guide for v0.15.0 features
- Improve documentation for new MCP tools
- Better examples and usage patterns
- **#339**: Add tool use documentation to write_note for root folder usage
([`73202d1`](https://github.com/basicmachines-co/basic-memory/commit/73202d1a))
- Document how to use empty string for root folder in write_note
- Clarify folder parameter usage
- Improve tool documentation clarity
- Fix link in ai_assistant_guide resource
([`2a1c06d`](https://github.com/basicmachines-co/basic-memory/commit/2a1c06d9))
- Correct broken documentation links
- Improve resource accessibility
### Refactoring
- Add SPEC-17 and SPEC-18 documentation
([`962d88e`](https://github.com/basicmachines-co/basic-memory/commit/962d88ea))
- New specification documents for future features
- Architecture planning and design documentation
### Breaking Changes
**None** - This release maintains full backward compatibility with v0.15.0
### Migration Guide
No manual migration required. Upgrade with:
```bash
# Update via uv
uv tool upgrade basic-memory
# Or install fresh
uv tool install basic-memory
```
**What's Fixed:**
- Significant performance improvements (43% faster sync, 10-100x faster directory operations)
- Multiple cloud deployment stability fixes
- Project path validation and normalization issues resolved
- Better Claude Desktop and ChatGPT integration
**What's New:**
- Context manager pattern for async clients (foundation for cloud consolidation)
- BASIC_MEMORY_PROJECT_ROOT environment variable for path constraints
- Enhanced cloud mode path handling and security
- SPEC-15 and SPEC-16 architecture documentation
### Installation
```bash
# Latest stable release
uv tool install basic-memory
# Update existing installation
uv tool upgrade basic-memory
# Docker
docker pull ghcr.io/basicmachines-co/basic-memory:v0.15.1
```
## v0.15.0 (2025-10-04)
### Critical Bug Fixes
@@ -1573,4 +2017,4 @@ Co-authored-by: phernandez <phernandez@basicmachines.co>
### Chores
- Remove basic-foundation src ref in pyproject.toml
([`29fce8b`](https://github.com/basicmachines-co/basic-memory/commit/29fce8b0b922d54d7799bf2534107ee6cfb961b8))
([`29fce8b`](https://github.com/basicmachines-co/basic-memory/commit/29fce8b0b922d54d7799bf2534107ee6cfb961b8))
+117 -20
View File
@@ -14,20 +14,34 @@ See the [README.md](README.md) file for a project overview.
### Build and Test Commands
- Install: `make install` or `pip install -e ".[dev]"`
- Run tests: `uv run pytest -p pytest_mock -v` or `make test`
- Install: `just install` or `pip install -e ".[dev]"`
- Run all tests (with coverage): `just test` - Runs both unit and integration tests with unified coverage
- Run unit tests only: `just test-unit` - Fast, no coverage
- Run integration tests only: `just test-int` - Fast, no coverage
- Generate HTML coverage: `just coverage` - Opens in browser
- Single test: `pytest tests/path/to/test_file.py::test_function_name`
- Lint: `make lint` or `ruff check . --fix`
- Type check: `make type-check` or `uv run pyright`
- Format: `make format` or `uv run ruff format .`
- Run all code checks: `make check` (runs lint, format, type-check, test)
- Create db migration: `make migration m="Your migration message"`
- Run development MCP Inspector: `make run-inspector`
- Run benchmarks: `pytest test-int/test_sync_performance_benchmark.py -v -m "benchmark and not slow"`
- Lint: `just lint` or `ruff check . --fix`
- Type check: `just typecheck` or `uv run pyright`
- Format: `just format` or `uv run ruff format .`
- Run all code checks: `just check` (runs lint, format, typecheck, test)
- Create db migration: `just migration "Your migration message"`
- Run development MCP Inspector: `just run-inspector`
**Note:** Project requires Python 3.12+ (uses type parameter syntax and `type` aliases introduced in 3.12)
### Test Structure
- `tests/` - Unit tests for individual components (mocked, fast)
- `test-int/` - Integration tests for real-world scenarios (no mocks, realistic)
- Both directories are covered by unified coverage reporting
- Benchmark tests in `test-int/` are marked with `@pytest.mark.benchmark`
- Slow tests are marked with `@pytest.mark.slow`
### Code Style Guidelines
- Line length: 100 characters max
- Python 3.12+ with full type annotations
- Python 3.12+ with full type annotations (uses type parameters and type aliases)
- Format with ruff (consistent styling)
- Import order: standard lib, third-party, local imports
- Naming: snake_case for functions/variables, PascalCase for classes
@@ -61,9 +75,46 @@ See the [README.md](README.md) file for a project overview.
- Schema changes require Alembic migrations
- SQLite is used for indexing and full text search, files are source of truth
- Testing uses pytest with asyncio support (strict mode)
- Unit tests (`tests/`) use mocks when necessary; integration tests (`test-int/`) use real implementations
- Test database uses in-memory SQLite
- Avoid creating mocks in tests in most circumstances.
- Each test runs in a standalone environment with in memory SQLite and tmp_file directory
- Each test runs in a standalone environment with in-memory SQLite and tmp_file directory
- Performance benchmarks are in `test-int/test_sync_performance_benchmark.py`
- Use pytest markers: `@pytest.mark.benchmark` for benchmarks, `@pytest.mark.slow` for slow tests
### Async Client Pattern (Important!)
**All MCP tools and CLI commands use the context manager pattern for HTTP clients:**
```python
from basic_memory.mcp.async_client import get_client
async def my_mcp_tool():
async with get_client() as client:
# Use client for API calls
response = await call_get(client, "/path")
return response
```
**Do NOT use:**
-`from basic_memory.mcp.async_client import client` (deprecated module-level client)
- ❌ Manual auth header management
-`inject_auth_header()` (deleted)
**Key principles:**
- Auth happens at client creation, not per-request
- Proper resource management via context managers
- Supports three modes: Local (ASGI), CLI cloud (HTTP + auth), Cloud app (factory injection)
- Factory pattern enables dependency injection for cloud consolidation
**For cloud app integration:**
```python
from basic_memory.mcp import async_client
# Set custom factory before importing tools
async_client.set_client_factory(your_custom_factory)
```
See SPEC-16 for full context manager refactor details.
## BASIC MEMORY PRODUCT USAGE
@@ -80,6 +131,7 @@ See the [README.md](README.md) file for a project overview.
### Basic Memory Commands
**Local Commands:**
- Sync knowledge: `basic-memory sync` or `basic-memory sync --watch`
- Import from Claude: `basic-memory import claude conversations`
- Import from ChatGPT: `basic-memory import chatgpt`
@@ -89,24 +141,41 @@ See the [README.md](README.md) file for a project overview.
- Guide: `basic-memory tools basic-memory-guide`
- Continue: `basic-memory tools continue-conversation --topic="search"`
**Cloud Commands (requires subscription):**
- Authenticate: `basic-memory cloud login`
- Logout: `basic-memory cloud logout`
- Bidirectional sync: `basic-memory cloud sync`
- Integrity check: `basic-memory cloud check`
- Mount cloud storage: `basic-memory cloud mount`
- Unmount cloud storage: `basic-memory cloud unmount`
### MCP Capabilities
- Basic Memory exposes these MCP tools to LLMs:
**Content Management:**
- `write_note(title, content, folder, tags)` - Create/update markdown notes with semantic observations and relations
- `read_note(identifier, page, page_size)` - Read notes by title, permalink, or memory:// URL with knowledge graph
awareness
- `read_file(path)` - Read raw file content (text, images, binaries) without knowledge graph processing
- `read_note(identifier, page, page_size)` - Read notes by title, permalink, or memory:// URL with knowledge graph awareness
- `read_content(path)` - Read raw file content (text, images, binaries) without knowledge graph processing
- `view_note(identifier, page, page_size)` - View notes as formatted artifacts for better readability
- `edit_note(identifier, operation, content)` - Edit notes incrementally (append, prepend, find/replace, replace_section)
- `move_note(identifier, destination_path)` - Move notes to new locations, updating database and maintaining links
- `delete_note(identifier)` - Delete notes from the knowledge base
**Knowledge Graph Navigation:**
- `build_context(url, depth, timeframe)` - Navigate the knowledge graph via memory:// URLs for conversation
continuity
- `recent_activity(type, depth, timeframe)` - Get recently updated information with specified timeframe (e.g., "
1d", "1 week")
- `build_context(url, depth, timeframe)` - Navigate the knowledge graph via memory:// URLs for conversation continuity
- `recent_activity(type, depth, timeframe)` - Get recently updated information with specified timeframe (e.g., "1d", "1 week")
- `list_directory(dir_name, depth, file_name_glob)` - Browse directory contents with filtering and depth control
**Search & Discovery:**
- `search(query, page, page_size)` - Full-text search across all content with filtering options
- `search_notes(query, page, page_size, search_type, types, entity_types, after_date)` - Full-text search across all content with advanced filtering options
**Project Management:**
- `list_memory_projects()` - List all available projects with their status
- `create_memory_project(project_name, project_path, set_default)` - Create new Basic Memory projects
- `delete_project(project_name)` - Delete a project from configuration
- `get_current_project()` - Get current project information and stats
- `sync_status()` - Check file synchronization and background operation status
**Visualization:**
- `canvas(nodes, edges, title, folder)` - Generate Obsidian canvas files for knowledge graph visualization
@@ -118,6 +187,34 @@ See the [README.md](README.md) file for a project overview.
- `recent_activity(timeframe)` - View recently changed items with formatted output
- `json_canvas_spec()` - Full JSON Canvas specification for Obsidian visualization
### Cloud Features (v0.15.0+)
Basic Memory now supports cloud synchronization and storage (requires active subscription):
**Authentication:**
- JWT-based authentication with subscription validation
- Secure session management with token refresh
- Support for multiple cloud projects
**Bidirectional Sync:**
- rclone bisync integration for two-way synchronization
- Conflict resolution and integrity verification
- Real-time sync with change detection
- Mount/unmount cloud storage for direct file access
**Cloud Project Management:**
- Create and manage projects in the cloud
- Toggle between local and cloud modes
- Per-project sync configuration
- Subscription-based access control
**Security & Performance:**
- Removed .env file loading for improved security
- .gitignore integration (respects gitignored files)
- WAL mode for SQLite performance
- Background relation resolution (non-blocking startup)
- API performance optimizations (SPEC-11)
## AI-Human Collaborative Development
Basic Memory emerged from and enables a new kind of development process that combines human and AI capabilities. Instead
@@ -168,4 +265,4 @@ With GitHub integration, the development workflow includes:
3. **Branch management** - Claude can create feature branches for implementations
4. **Documentation maintenance** - Claude can keep documentation updated as the code evolves
This level of integration represents a new paradigm in AI-human collaboration, where the AI assistant becomes a full-fledged team member rather than just a tool for generating code snippets.
This level of integration represents a new paradigm in AI-human collaboration, where the AI assistant becomes a full-fledged team member rather than just a tool for generating code snippets.
+80 -7
View File
@@ -34,11 +34,18 @@ project and how to get started as a developer.
4. **Run the Tests**:
```bash
# Run all tests
# Run all tests with unified coverage (unit + integration)
just test
# or
uv run pytest -p pytest_mock -v
# Run unit tests only (fast, no coverage)
just test-unit
# Run integration tests only (fast, no coverage)
just test-int
# Generate HTML coverage report
just coverage
# Run a specific test
pytest tests/path/to/test_file.py::test_function_name
```
@@ -134,7 +141,7 @@ agreement to the DCO.
## Code Style Guidelines
- **Python Version**: Python 3.12+ with full type annotations
- **Python Version**: Python 3.12+ with full type annotations (3.12+ required for type parameter syntax)
- **Line Length**: 100 characters maximum
- **Formatting**: Use ruff for consistent styling
- **Import Order**: Standard lib, third-party, local imports
@@ -144,12 +151,78 @@ agreement to the DCO.
## Testing Guidelines
- **Coverage Target**: We aim for 100% test coverage for all code
### Test Structure
Basic Memory uses two test directories with unified coverage reporting:
- **`tests/`**: Unit tests that test individual components in isolation
- Fast execution with extensive mocking
- Test individual functions, classes, and modules
- Run with: `just test-unit` (no coverage, fast)
- **`test-int/`**: Integration tests that test real-world scenarios
- Test full workflows with real database and file operations
- Include performance benchmarks
- More realistic but slower than unit tests
- Run with: `just test-int` (no coverage, fast)
### Running Tests
```bash
# Run all tests with unified coverage report
just test
# Run only unit tests (fast iteration)
just test-unit
# Run only integration tests
just test-int
# Generate HTML coverage report
just coverage
# Run specific test
pytest tests/path/to/test_file.py::test_function_name
# Run tests excluding benchmarks
pytest -m "not benchmark"
# Run only benchmark tests
pytest -m benchmark test-int/test_sync_performance_benchmark.py
```
### Performance Benchmarks
The `test-int/test_sync_performance_benchmark.py` file contains performance benchmarks that measure sync and indexing speed:
- `test_benchmark_sync_100_files` - Small repository performance
- `test_benchmark_sync_500_files` - Medium repository performance
- `test_benchmark_sync_1000_files` - Large repository performance (marked slow)
- `test_benchmark_resync_no_changes` - Re-sync performance baseline
Run benchmarks with:
```bash
# Run all benchmarks (excluding slow ones)
pytest test-int/test_sync_performance_benchmark.py -v -m "benchmark and not slow"
# Run all benchmarks including slow ones
pytest test-int/test_sync_performance_benchmark.py -v -m benchmark
# Run specific benchmark
pytest test-int/test_sync_performance_benchmark.py::test_benchmark_sync_100_files -v
```
See `test-int/BENCHMARKS.md` for detailed benchmark documentation.
### Testing Best Practices
- **Coverage Target**: We aim for high test coverage for all code
- **Test Framework**: Use pytest for unit and integration tests
- **Mocking**: Use pytest-mock for mocking dependencies only when necessary
- **Mocking**: Avoid mocking in integration tests; use sparingly in unit tests
- **Edge Cases**: Test both normal operation and edge cases
- **Database Testing**: Use in-memory SQLite for testing database operations
- **Fixtures**: Use async pytest fixtures for setup and teardown
- **Markers**: Use `@pytest.mark.benchmark` for benchmarks, `@pytest.mark.slow` for slow tests
## Release Process
+3 -2
View File
@@ -24,11 +24,12 @@ WORKDIR /app
RUN uv sync --locked
# Create necessary directories and set ownership
RUN mkdir -p /app/data /app/.basic-memory && \
RUN mkdir -p /app/data/basic-memory /app/.basic-memory && \
chown -R appuser:${GID} /app
# Set default data directory and add venv to PATH
ENV BASIC_MEMORY_HOME=/app/data \
ENV BASIC_MEMORY_HOME=/app/data/basic-memory \
BASIC_MEMORY_PROJECT_ROOT=/app/data \
PATH="/app/.venv/bin:$PATH"
# Switch to the non-root user
+60 -3
View File
@@ -7,6 +7,16 @@
![](https://badge.mcpx.dev?type=dev 'MCP Dev')
[![smithery badge](https://smithery.ai/badge/@basicmachines-co/basic-memory)](https://smithery.ai/server/@basicmachines-co/basic-memory)
## 🚀 Basic Memory Cloud is Live!
- **Cross-device and multi-platform support is here.** Your knowledge graph now works on desktop, web, and mobile - seamlessly synced across all your AI tools (Claude, ChatGPT, Gemini, Claude Code, and Codex)
- **Early Supporter Pricing:** Early users get 25% off forever.
The open source project continues as always. Cloud just makes it work everywhere.
[Sign up now →](https://basicmemory.com/beta)
with a 7 day free trial
# Basic Memory
Basic Memory lets you build persistent knowledge through natural conversations with Large Language Models (LLMs) like
@@ -100,6 +110,9 @@ With Basic Memory, you can:
- Keep everything local and under your control
- Use familiar tools like Obsidian to view and edit notes
- Build a personal knowledge base that grows over time
- Sync your knowledge to the cloud with bidirectional synchronization
- Authenticate and manage cloud projects with subscription validation
- Mount cloud storage for direct file access
## How It Works in Practice
@@ -346,14 +359,57 @@ basic-memory sync
basic-memory sync --watch
```
3. In Claude Desktop, the LLM can now use these tools:
3. Cloud features (optional, requires subscription):
```bash
# Authenticate with cloud
basic-memory cloud login
# Bidirectional sync with cloud
basic-memory cloud sync
# Verify cloud integrity
basic-memory cloud check
# Mount cloud storage
basic-memory cloud mount
```
4. In Claude Desktop, the LLM can now use these tools:
**Content Management:**
```
write_note(title, content, folder, tags) - Create or update notes
read_note(identifier, page, page_size) - Read notes by title or permalink
read_content(path) - Read raw file content (text, images, binaries)
view_note(identifier) - View notes as formatted artifacts
edit_note(identifier, operation, content) - Edit notes incrementally
move_note(identifier, destination_path) - Move notes with database consistency
delete_note(identifier) - Delete notes from knowledge base
```
**Knowledge Graph Navigation:**
```
build_context(url, depth, timeframe) - Navigate knowledge graph via memory:// URLs
search(query, page, page_size) - Search across your knowledge base
recent_activity(type, depth, timeframe) - Find recently updated information
list_directory(dir_name, depth) - Browse directory contents with filtering
```
**Search & Discovery:**
```
search(query, page, page_size) - Search across your knowledge base
```
**Project Management:**
```
list_memory_projects() - List all available projects
create_memory_project(project_name, project_path) - Create new projects
get_current_project() - Show current project stats
sync_status() - Check synchronization status
```
**Visualization:**
```
canvas(nodes, edges, title, folder) - Generate knowledge visualizations
```
@@ -373,6 +429,7 @@ See the [Documentation](https://memory.basicmachines.co/) for more info, includi
- [Complete User Guide](https://docs.basicmemory.com/user-guide/)
- [CLI tools](https://docs.basicmemory.com/guides/cli-reference/)
- [Cloud CLI and Sync](https://docs.basicmemory.com/guides/cloud-cli/)
- [Managing multiple Projects](https://docs.basicmemory.com/guides/cli-reference/#project)
- [Importing data from OpenAI/Claude Projects](https://docs.basicmemory.com/guides/cli-reference/#import)
@@ -393,4 +450,4 @@ and submitting PRs.
</picture>
</a>
Built with ♥️ by Basic Machines
Built with ♥️ by Basic Machines
File diff suppressed because it is too large Load Diff
+565 -496
View File
File diff suppressed because it is too large Load Diff
+11 -6
View File
@@ -2,28 +2,33 @@
# Install dependencies
install:
pip install -e ".[dev]"
uv pip install -e ".[dev]"
uv sync
@echo ""
@echo "💡 Remember to activate the virtual environment by running: source .venv/bin/activate"
# Run unit tests in parallel
# Run unit tests only (fast, no coverage)
test-unit:
uv run pytest -p pytest_mock -v -n auto
uv run pytest -p pytest_mock -v --no-cov -n auto tests
# Run integration tests in parallel
# Run integration tests only (fast, no coverage)
test-int:
uv run pytest -p pytest_mock -v --no-cov -n auto test-int
# Run all tests
# Run all tests with unified coverage report
test: test-unit test-int
# Generate HTML coverage report
coverage:
uv run pytest -p pytest_mock -v -n auto tests test-int --cov-report=html
@echo "Coverage report generated in htmlcov/index.html"
# Lint and fix code (calls fix)
lint: fix
# Lint and fix code
fix:
uv run ruff check --fix --unsafe-fixes src tests
uv run ruff check --fix --unsafe-fixes src tests test-int
# Type check code
typecheck:
+10 -5
View File
@@ -3,7 +3,7 @@ name = "basic-memory"
dynamic = ["version"]
description = "Local-first knowledge management combining Zettelkasten with knowledge graphs"
readme = "README.md"
requires-python = ">=3.12.1"
requires-python = ">=3.12"
license = { text = "AGPL-3.0-or-later" }
authors = [
{ name = "Basic Machines", email = "hello@basic-machines.co" }
@@ -34,7 +34,8 @@ dependencies = [
"pyjwt>=2.10.1",
"python-dotenv>=1.1.0",
"pytest-aio>=1.9.0",
"aiofiles>=24.1.0", # Async file I/O
"aiofiles>=24.1.0", # Async file I/O
"logfire>=0.73.0", # Optional observability (disabled by default via config)
]
@@ -54,16 +55,20 @@ build-backend = "hatchling.build"
[tool.pytest.ini_options]
pythonpath = ["src", "tests"]
addopts = "--cov=basic_memory --cov-report term-missing"
testpaths = ["tests"]
testpaths = ["tests", "test-int"]
asyncio_mode = "strict"
asyncio_default_fixture_loop_scope = "function"
markers = [
"benchmark: Performance benchmark tests (deselect with '-m \"not benchmark\"')",
"slow: Slow-running tests (deselect with '-m \"not slow\"')",
]
[tool.ruff]
line-length = 100
target-version = "py312"
[tool.uv]
dev-dependencies = [
[dependency-groups]
dev = [
"gevent>=24.11.1",
"icecream>=2.1.3",
"pytest>=8.3.4",
@@ -0,0 +1,569 @@
---
title: 'SPEC-10: Unified Deployment Workflow and Event Tracking'
type: spec
permalink: specs/spec-10-unified-deployment-workflow-event-tracking
tags:
- workflow
- deployment
- event-sourcing
- architecture
- simplification
---
# SPEC-10: Unified Deployment Workflow and Event Tracking
## Why
We replaced a complex multi-workflow system with DBOS orchestration that was proving to be more trouble than it was worth. The previous architecture had four separate workflows (`tenant_provisioning`, `tenant_update`, `tenant_deployment`, `tenant_undeploy`) with overlapping logic, complex state management, and fragmented event tracking. DBOS added unnecessary complexity without providing sufficient value, leading to harder debugging and maintenance.
**Problems Solved:**
- **Framework Complexity**: DBOS configuration overhead and fighting framework limitations
- **Code Duplication**: Multiple workflows implementing similar operations with duplicate logic
- **Poor Observability**: Fragmented event tracking across workflow boundaries
- **Maintenance Overhead**: Complex orchestration for fundamentally simple operations
- **Debugging Difficulty**: Framework abstractions hiding simple Python stack traces
## What
This spec documents the architectural simplification that consolidates tenant lifecycle management into a unified system with comprehensive event tracking.
**Affected Areas:**
- Tenant deployment workflows (provisioning, updates, undeploying)
- Event sourcing and workflow tracking infrastructure
- API endpoints for tenant operations
- Database schema for workflow and event correlation
- Integration testing for tenant lifecycle operations
**Key Changes:**
- **Removed DBOS entirely** - eliminated framework dependency and complexity
- **Consolidated 4 workflows → 2 unified deployment workflows (deploy/undeploy)**
- **Added workflow tracking system** with complete event correlation
- **Simplified API surface** - single `/deploy` endpoint handles all scenarios
- **Enhanced observability** through event sourcing with workflow grouping
## How (High Level)
### Architectural Philosophy
**Embrace simplicity over framework complexity** - use well-structured Python with proper database design instead of complex orchestration frameworks.
### Core Components
#### 1. Unified Deployment Workflow
```python
class TenantDeploymentWorkflow:
async def deploy_tenant_workflow(self, tenant_id: str, workflow_id: UUID, image_tag: str = None):
# Single workflow handles both initial provisioning AND updates
# Each step is idempotent and handles its own error recovery
# Database transactions provide the durability we need
await self.start_deployment_step(workflow_id, tenant_uuid, image_tag)
await self.create_fly_app_step(workflow_id, tenant_uuid)
await self.create_bucket_step(workflow_id, tenant_uuid)
await self.deploy_machine_step(workflow_id, tenant_uuid, image_tag)
await self.complete_deployment_step(workflow_id, tenant_uuid, image_tag, deployment_time)
```
**Key Benefits:**
- **Handles both provisioning and updates** in single workflow
- **Idempotent operations** - safe to retry any step
- **Clean error handling** via simple Python exceptions
- **Resumable** - can restart from any failed step
#### 2. Workflow Tracking System
**Database Schema:**
```sql
CREATE TABLE workflow (
id UUID PRIMARY KEY,
workflow_type VARCHAR(50) NOT NULL, -- 'tenant_deployment', 'tenant_undeploy'
tenant_id UUID REFERENCES tenant(id),
status VARCHAR(20) DEFAULT 'running', -- 'running', 'completed', 'failed'
workflow_metadata JSONB DEFAULT '{}' -- image_tag, etc.
);
ALTER TABLE event ADD COLUMN workflow_id UUID REFERENCES workflow(id);
```
**Event Correlation:**
- Every workflow operation generates events tagged with `workflow_id`
- Complete audit trail from workflow start to completion
- Events grouped by workflow for easy reconstruction of operations
#### 3. Parameter Standardization
All workflow methods follow consistent signature pattern:
```python
async def method_name(self, session: AsyncSession, workflow_id: UUID | None, tenant_id: UUID, ...)
```
**Benefits:**
- **Consistent event tagging** - all events properly correlated
- **Clear method contracts** - workflow_id always first parameter
- **Type safety** - proper UUID handling throughout
### Implementation Strategy
#### Phase 1: Workflow Consolidation ✅ COMPLETED
- [x] **Remove DBOS dependency** - eliminated dbos_config.py and all DBOS imports
- [x] **Create unified TenantDeploymentWorkflow** - handles both provisioning and updates
- [x] **Remove legacy workflows** - deleted tenant_provisioning.py, tenant_update.py
- [x] **Simplify API endpoints** - consolidated to single `/deploy` endpoint
- [x] **Update integration tests** - comprehensive edge case testing
#### Phase 2: Workflow Tracking System ✅ COMPLETED
- [x] **Database migration** - added workflow table and event.workflow_id foreign key
- [x] **Workflow repository** - CRUD operations for workflow records
- [x] **Event correlation** - all workflow events tagged with workflow_id
- [x] **Comprehensive testing** - workflow lifecycle and event grouping tests
#### Phase 3: Parameter Standardization ✅ COMPLETED
- [x] **Standardize method signatures** - workflow_id as first parameter pattern
- [x] **Fix event tagging** - ensure all workflow events properly correlated
- [x] **Update service methods** - consistent parameter order across tenant_service
- [x] **Integration test validation** - verify complete event sequences
### Architectural Benefits
#### Code Simplification
- **39 files changed**: 2,247 additions, 3,256 deletions (net -1,009 lines)
- **Eliminated framework complexity** - no more DBOS configuration or abstractions
- **Consolidated logic** - single deployment workflow vs 4 separate workflows
- **Cleaner API surface** - unified endpoint vs multiple workflow-specific endpoints
#### Enhanced Observability
- **Complete event correlation** - every workflow event tagged with workflow_id
- **Audit trail reconstruction** - can trace entire tenant lifecycle through events
- **Workflow status tracking** - running/completed/failed states in database
- **Comprehensive testing** - edge cases covered with real infrastructure
#### Operational Benefits
- **Simpler debugging** - plain Python stack traces vs framework abstractions
- **Reduced dependencies** - one less complex framework to maintain
- **Better error handling** - explicit exception handling vs framework magic
- **Easier maintenance** - straightforward Python code vs orchestration complexity
## How to Evaluate
### Success Criteria
#### Functional Completeness ✅ VERIFIED
- [x] **Unified deployment workflow** handles both initial provisioning and updates
- [x] **Undeploy workflow** properly integrated with event tracking
- [x] **All operations idempotent** - safe to retry any step without duplication
- [x] **Complete tenant lifecycle** - provision → active → update → undeploy
#### Event Tracking and Correlation ✅ VERIFIED
- [x] **All workflow events tagged** with proper workflow_id
- [x] **Event sequence verification** - tests assert exact event order and content
- [x] **Workflow grouping** - events can be queried by workflow_id for complete audit trail
- [x] **Cross-workflow isolation** - deployment vs undeploy events properly separated
#### Database Schema and Performance ✅ VERIFIED
- [x] **Migration applied** - workflow table and event.workflow_id column created
- [x] **Proper indexing** - performance optimized queries on workflow_type, tenant_id, status
- [x] **Foreign key constraints** - referential integrity between workflows and events
- [x] **Database triggers** - updated_at timestamp automation
#### Test Coverage ✅ COMPREHENSIVE
- [x] **Unit tests**: 4 workflow tracking tests covering lifecycle and event grouping
- [x] **Integration tests**: Real infrastructure testing with Fly.io resources
- [x] **Edge case coverage**: Failed deployments, partial state recovery, resource conflicts
- [x] **Event sequence verification**: Exact event order and content validation
### Testing Procedure
#### Unit Test Validation ✅ PASSING
```bash
cd apps/cloud && pytest tests/test_workflow_tracking.py -v
# 4/4 tests passing - workflow lifecycle and event grouping
```
#### Integration Test Validation ✅ PASSING
```bash
cd apps/cloud && pytest tests/integration/test_tenant_workflow_deployment_integration.py -v
cd apps/cloud && pytest tests/integration/test_tenant_workflow_undeploy_integration.py -v
# Comprehensive real infrastructure testing with actual Fly.io resources
# Tests provision → deploy → update → undeploy → cleanup cycles
```
### Performance Metrics
#### Code Metrics ✅ ACHIEVED
- **Net code reduction**: -1,009 lines (3,256 deletions, 2,247 additions)
- **Workflow consolidation**: 4 workflows → 1 unified deployment workflow
- **Dependency reduction**: Removed DBOS framework dependency entirely
- **API simplification**: Multiple endpoints → single `/deploy` endpoint
#### Operational Metrics ✅ VERIFIED
- **Event correlation**: 100% of workflow events properly tagged with workflow_id
- **Audit trail completeness**: Full tenant lifecycle traceable through event sequences
- **Error handling**: Clean Python exceptions vs framework abstractions
- **Debugging simplicity**: Direct stack traces vs orchestration complexity
### Implementation Status: ✅ COMPLETE
All phases completed successfully with comprehensive testing and verification:
**Phase 1 - Workflow Consolidation**: ✅ COMPLETE
- Removed DBOS dependency and consolidated workflows
- Unified deployment workflow handles all scenarios
- Comprehensive integration testing with real infrastructure
**Phase 2 - Workflow Tracking**: ✅ COMPLETE
- Database schema implemented with proper indexing
- Event correlation system fully functional
- Complete audit trail capability verified
**Phase 3 - Parameter Standardization**: ✅ COMPLETE
- Consistent method signatures across all workflow methods
- All events properly tagged with workflow_id
- Type safety verified across entire codebase
**Phase 4 - Asynchronous Job Queuing**:
**Goal**: Transform synchronous deployment workflows into background jobs for better user experience and system reliability.
**Current Problem**:
- Deployment API calls are synchronous - users wait for entire tenant provisioning (30-60 seconds)
- No retry mechanism for failed operations
- HTTP timeouts on long-running deployments
- Poor user experience during infrastructure provisioning
**Solution**: Redis-backed job queue with arq for reliable background processing
#### Architecture Overview
```python
# API Layer: Return immediately with job tracking
@router.post("/{tenant_id}/deploy")
async def deploy_tenant(tenant_id: UUID):
# Create workflow record in Postgres
workflow = await workflow_repo.create_workflow("tenant_deployment", tenant_id)
# Enqueue job in Redis
job = await arq_pool.enqueue_job('deploy_tenant_task', tenant_id, workflow.id)
# Return job ID immediately
return {"job_id": job.job_id, "workflow_id": workflow.id, "status": "queued"}
# Background Worker: Process via existing unified workflow
async def deploy_tenant_task(ctx, tenant_id: str, workflow_id: str):
# Existing workflow logic - zero changes needed!
await workflow_manager.deploy_tenant(UUID(tenant_id), workflow_id=UUID(workflow_id))
```
#### Implementation Tasks
**Phase 4.1: Core Job Queue Setup** ✅ COMPLETED
- [x] **Add arq dependency** - integrated Redis job queue with existing infrastructure
- [x] **Create job definitions** - wrapped existing deployment/undeploy workflows as arq tasks
- [x] **Update API endpoints** - updated provisioning endpoints to return job IDs instead of waiting for completion
- [x] **JobQueueService implementation** - service layer for job enqueueing and status tracking
- [x] **Job status tracking** - integrated with existing workflow table for status updates
- [x] **Comprehensive testing** - 18 tests covering positive, negative, and edge cases
**Phase 4.2: Background Worker Implementation** ✅ COMPLETED
- [x] **Job status API** - GET /jobs/{job_id}/status endpoint integrated with JobQueueService
- [x] **Background worker process** - arq worker to process queued jobs with proper settings and Redis configuration
- [x] **Worker settings and configuration** - WorkerSettings class with proper timeouts, max jobs, and error handling
- [x] **Fix API endpoints** - updated job status API to use JobQueueService instead of direct Redis access
- [x] **Integration testing** - comprehensive end-to-end testing with real ARQ workers and Fly.io infrastructure
- [x] **Worker entry points** - dual-purpose entrypoint.sh script and __main__.py module support for both API and worker processes
- [x] **Test fixture updates** - fixed all API and service test fixtures to work with job queue dependencies
- [x] **AsyncIO event loop fixes** - resolved event loop issues in integration tests for subprocess worker compatibility
- [x] **Complete test coverage** - all 46 tests passing across unit, integration, and API test suites
- [x] **Type safety verification** - 0 type checking errors across entire ARQ job queue implementation
#### Phase 4.2 Implementation Summary ✅ COMPLETE
**Core ARQ Job Queue System:**
- **JobQueueService** - Centralized service for job enqueueing, status tracking, and Redis pool management
- **deployment_jobs.py** - ARQ job functions that wrap existing deployment/undeploy workflows
- **Worker Settings** - Production-ready ARQ configuration with proper timeouts and error handling
- **Dual-Process Architecture** - Single Docker image with entrypoint.sh supporting both API and worker modes
**Key Files Added:**
- `apps/cloud/src/basic_memory_cloud/jobs/` - Complete job queue implementation (7 files)
- `apps/cloud/entrypoint.sh` - Dual-purpose Docker container entry point
- `apps/cloud/tests/integration/test_worker_integration.py` - Real infrastructure integration tests
- `apps/cloud/src/basic_memory_cloud/schemas/job_responses.py` - API response schemas
**API Integration:**
- Provisioning endpoints return job IDs immediately instead of blocking for 60+ seconds
- Job status API endpoints for real-time monitoring of deployment progress
- Proper error handling and job failure scenarios with detailed error messages
**Testing Achievement:**
- **46 total tests passing** across all test suites (unit, integration, API, services)
- **Real infrastructure testing** - ARQ workers process actual Fly.io deployments
- **Event loop safety** - Fixed asyncio issues for subprocess worker compatibility
- **Test fixture updates** - All fixtures properly support job queue dependencies
- **Type checking** - 0 errors across entire codebase
**Technical Metrics:**
- **38 files changed** - +1,736 insertions, -334 deletions
- **Integration test runtime** - ~18 seconds with real ARQ workers and Fly.io verification
- **Event loop isolation** - Proper async session management for subprocess compatibility
- **Redis integration** - Production-ready Redis configuration with connection pooling
**Phase 4.3: Production Hardening** ✅ COMPLETED
- [x] **Configure Upstash Redis** - production Redis setup on Fly.io
- [x] **Retry logic for external APIs** - exponential backoff for flaky Tigris IAM operations
- [x] **Monitoring and observability** - comprehensive Redis queue monitoring with CLI tools
- [x] **Error handling improvements** - graceful handling of expected API errors with appropriate log levels
- [x] **CLI tooling enhancements** - bulk update commands for CI/CD automation
- [x] **Documentation improvements** - comprehensive monitoring guide with Redis patterns
- [x] **Job uniqueness** - ARQ-based duplicate prevention for tenant operations
- [ ] **Worker scaling** - multiple arq workers for parallel job processing
- [ ] **Job persistence** - ensure jobs survive Redis/worker restarts
- [ ] **Error alerting** - notifications for failed deployment jobs
**Phase 4.4: Advanced Features** (Future)
- [ ] **Job scheduling** - deploy tenants at specific times
- [ ] **Priority queues** - urgent deployments processed first
- [ ] **Batch operations** - bulk tenant deployments
- [ ] **Job dependencies** - deployment → configuration → activation chains
#### Benefits Achieved ✅ REALIZED
**User Experience Improvements:**
- **Immediate API responses** - users get job ID instantly vs waiting 60+ seconds for deployment completion
- **Real-time job tracking** - status API provides live updates on deployment progress
- **Better error visibility** - detailed error messages and job failure tracking
- **CI/CD automation ready** - bulk update commands for automated tenant deployments
**System Reliability:**
- **Redis persistence** - jobs survive Redis/worker restarts with proper queue durability
- **Idempotent job processing** - jobs can be safely retried without side effects
- **Event loop isolation** - worker processes operate independently from API server
- **Retry resilience** - exponential backoff for flaky external API calls (3 attempts, 1s/2s delays)
- **Graceful error handling** - expected API errors logged at INFO level, unexpected at ERROR level
- **Job uniqueness** - prevent duplicate tenant operations with ARQ's built-in uniqueness feature
**Operational Benefits:**
- **Horizontal scaling ready** - architecture supports adding more workers for parallel processing
- **Comprehensive testing** - real infrastructure integration tests ensure production reliability
- **Type safety** - full type checking prevents runtime errors in job processing
- **Clean separation** - API and worker processes use same codebase with different entry points
- **Queue monitoring** - Redis CLI integration for real-time queue activity monitoring
- **Comprehensive documentation** - detailed monitoring guide with Redis pattern explanations
**Development Benefits:**
- **Zero workflow changes** - existing deployment/undeploy workflows work unchanged as background jobs
- **Async/await native** - modern Python asyncio patterns throughout the implementation
- **Event correlation preserved** - all existing workflow tracking and event sourcing continues to work
- **Enhanced CLI tooling** - unified tenant commands with proper endpoint routing
- **Database integrity** - proper foreign key constraint handling in tenant deletion
#### Infrastructure Requirements
- **Local**: Redis via docker-compose (already exists) ✅
- **Production**: Upstash Redis on Fly.io (already configured) ✅
- **Workers**: arq worker processes (new deployment target)
- **Monitoring**: Job status dashboard (simple web interface)
#### API Evolution
```python
# Before: Synchronous (blocks for 60+ seconds)
POST /tenant/{id}/deploy {status: "active", machine_id: "..."}
# After: Asynchronous (returns immediately)
POST /tenant/{id}/deploy {job_id: "uuid", workflow_id: "uuid", status: "queued"}
GET /jobs/{job_id}/status {status: "running", progress: "deploying_machine", workflow_id: "uuid"}
GET /workflows/{workflow_id}/events [...] # Existing event tracking works unchanged
```
**Technology Choice**: **arq (Redis)** over pgqueuer
- **Existing Redis infrastructure** - Upstash + docker-compose already configured
- **Better ecosystem** - monitoring tools, documentation, community
- **Made by pydantic team** - aligns with existing Python stack
- **Hybrid approach** - Redis for queue operations + Postgres for workflow state
#### Job Uniqueness Implementation
**Problem**: Multiple concurrent deployment requests for the same tenant could create duplicate jobs, wasting resources and potentially causing conflicts.
**Solution**: Leverage ARQ's built-in job uniqueness feature using predictable job IDs:
```python
# JobQueueService implementation
async def enqueue_deploy_job(self, tenant_id: UUID, image_tag: str | None = None) -> str:
unique_job_id = f"deploy-{tenant_id}"
job = await self.redis_pool.enqueue_job(
"deploy_tenant_job",
str(tenant_id),
image_tag,
_job_id=unique_job_id, # ARQ prevents duplicates
)
if job is None:
# Job already exists - return existing job ID
return unique_job_id
else:
# New job created - return ARQ job ID
return job.job_id
```
**Key Features:**
- **Predictable Job IDs**: `deploy-{tenant_id}`, `undeploy-{tenant_id}`
- **Duplicate Prevention**: ARQ returns `None` for duplicate job IDs
- **Graceful Handling**: Return existing job ID instead of raising errors
- **Idempotent Operations**: Safe to retry deployment requests
- **Clear Logging**: Distinguish "Enqueued new" vs "Found existing" jobs
**Benefits:**
- Prevents resource waste from duplicate deployments
- Eliminates race conditions from concurrent requests
- Makes job monitoring more predictable with consistent IDs
- Provides natural deduplication without complex locking mechanisms
## Notes
### Design Philosophy Lessons
- **Simplicity beats framework magic** - removing DBOS made the system more reliable and debuggable
- **Event sourcing > complex orchestration** - database-backed event tracking provides better observability than framework abstractions
- **Idempotent operations > resumable workflows** - each step handling its own retry logic is simpler than framework-managed resumability
- **Explicit error handling > framework exception handling** - Python exceptions are clearer than orchestration framework error states
### Future Considerations
- **Monitoring integration** - workflow tracking events could feed into observability systems
- **Performance optimization** - event querying patterns may benefit from additional indexing
- **Audit compliance** - complete event trail supports regulatory requirements
- **Operational dashboards** - workflow status could drive tenant health monitoring
### Related Specifications
- **SPEC-8**: TigrisFS Integration - bucket provisioning integrated with deployment workflow
- **SPEC-1**: Specification-Driven Development Process - this spec follows the established format
## Observations
- [architecture] Removing framework complexity led to more maintainable system #simplification
- [workflow] Single unified deployment workflow handles both provisioning and updates #consolidation
- [observability] Event sourcing with workflow correlation provides complete audit trail #event-tracking
- [database] Foreign key relationships between workflows and events enable powerful queries #schema-design
- [testing] Integration tests with real infrastructure catch edge cases that unit tests miss #testing-strategy
- [parameters] Consistent method signatures (workflow_id first) reduce cognitive overhead #api-design
- [maintenance] Fewer workflows and dependencies reduce long-term maintenance burden #operational-excellence
- [debugging] Plain Python exceptions are clearer than framework abstraction layers #developer-experience
- [resilience] Exponential backoff retry patterns handle flaky external API calls gracefully #error-handling
- [monitoring] Redis queue monitoring provides real-time operational visibility #observability
- [ci-cd] Bulk update commands enable automated tenant deployments in continuous delivery pipelines #automation
- [documentation] Comprehensive monitoring guides reduce operational learning curve #knowledge-management
- [error-logging] Context-aware log levels (INFO for expected errors, ERROR for unexpected) improve signal-to-noise ratio #logging-strategy
- [job-uniqueness] ARQ job uniqueness with predictable tenant-based IDs prevents duplicate operations and resource waste #deduplication
## Implementation Notes
### Configuration Integration
- **Redis Configuration**: Add Redis settings to existing `apps/cloud/src/basic_memory_cloud/config.py`
- **Local Development**: Leverage existing Redis setup from `docker-compose.yml`
- **Production**: Use Upstash Redis configuration for production environments
### Docker Entrypoint Strategy
Create `entrypoint.sh` script to toggle between API server and worker processes using single Docker image:
```bash
#!/bin/bash
# Entrypoint script for Basic Memory Cloud service
# Supports multiple process types: api, worker
set -e
case "$1" in
"api")
echo "Starting Basic Memory Cloud API server..."
exec uvicorn basic_memory_cloud.main:app \
--host 0.0.0.0 \
--port 8000 \
--log-level info
;;
"worker")
echo "Starting Basic Memory Cloud ARQ worker..."
# For ARQ worker implementation
exec python -m arq basic_memory_cloud.jobs.settings.WorkerSettings
;;
*)
echo "Usage: $0 {api|worker}"
echo " api - Start the FastAPI server"
echo " worker - Start the ARQ worker"
exit 1
;;
esac
```
### Fly.io Process Groups Configuration
Use separate machine groups for API and worker processes with independent scaling:
```toml
# fly.toml app configuration for basic-memory-cloud
app = 'basic-memory-cloud-dev-basic-machines'
primary_region = 'dfw'
org = 'basic-machines'
kill_signal = 'SIGINT'
kill_timeout = '5s'
[build]
# Process groups for API server and worker
[processes]
api = "api"
worker = "worker"
# Machine scaling configuration
[[machine]]
size = 'shared-cpu-1x'
processes = ['api']
min_machines_running = 1
auto_stop_machines = false
auto_start_machines = true
[[machine]]
size = 'shared-cpu-1x'
processes = ['worker']
min_machines_running = 1
auto_stop_machines = false
auto_start_machines = true
[env]
# Python configuration
PYTHONUNBUFFERED = '1'
PYTHONPATH = '/app'
# Logging configuration
LOG_LEVEL = 'DEBUG'
# Redis configuration for ARQ
REDIS_URL = 'redis://basic-memory-cloud-redis.upstash.io'
# Database configuration
DATABASE_HOST = 'basic-memory-cloud-db-dev-basic-machines.internal'
DATABASE_PORT = '5432'
DATABASE_NAME = 'basic_memory_cloud'
DATABASE_USER = 'postgres'
DATABASE_SSL = 'true'
# Worker configuration
ARQ_MAX_JOBS = '10'
ARQ_KEEP_RESULT = '3600'
# Fly.io configuration
FLY_ORG = 'basic-machines'
FLY_REGION = 'dfw'
# Internal service - no external HTTP exposure for worker
# API accessible via basic-memory-cloud-dev-basic-machines.flycast:8000
[[vm]]
size = 'shared-cpu-1x'
```
### Benefits of This Architecture
- **Single Docker Image**: Both API and worker use same container with different entrypoints
- **Independent Scaling**: Scale API and worker processes separately based on demand
- **Clean Separation**: Web traffic handling separate from background job processing
- **Existing Infrastructure**: Leverages current PostgreSQL + Redis setup without complexity
- **Hybrid State Management**: Redis for queue operations, PostgreSQL for persistent workflow tracking
## Relations
- implements [[SPEC-8 TigrisFS Integration]]
- follows [[SPEC-1 Specification-Driven Development Process]]
- supersedes previous multi-workflow architecture
@@ -31,8 +31,6 @@ HTTP requests to the API suffer from 350ms-2.6s latency overhead **before** any
This creates compounding effects with tenant auto-start delays and increases timeout risk in cloud deployments.
Github issue: https://github.com/basicmachines-co/basic-memory-cloud/issues/82
## What
This optimization affects the **core basic-memory repository** components:
@@ -170,76 +168,19 @@ Validation Checklist
- Documentation: Performance optimization documented in README
- Cloud Integration: basic-memory-cloud sees performance benefits
## Implementation Status ✅ COMPLETED
**Implementation Date**: 2025-09-26
**Branch**: `feature/spec-11-api-performance-optimization`
**Commit**: `771f60b`
### ✅ Phase 1: Database Connection Caching - IMPLEMENTED
**Files Modified:**
- `src/basic_memory/api/app.py` - Added database connection caching in app.state
- `src/basic_memory/deps.py` - Updated get_engine_factory() to use cached connections
- `src/basic_memory/config.py` - Added skip_initialization_sync configuration flag
**Implementation Details:**
1. **API Lifespan Caching**: Database engine and session_maker cached in app.state during startup
2. **Dependency Injection Optimization**: get_engine_factory() now returns cached connections instead of calling get_or_create_db()
3. **Project Reconciliation Removal**: Eliminated expensive reconcile_projects_with_config() from API startup
4. **CLI Fallback Preserved**: Non-API contexts continue to work with fallback database initialization
### ✅ Performance Validation - ACHIEVED
**Live Testing Results** (2025-09-26 14:03-14:09):
| Operation | Before | After | Improvement |
|-----------|--------|-------|-------------|
| `read_note` | 350ms-2.6s | **20ms** | **95-99% faster** |
| `edit_note` | 350ms-2.6s | **218ms** | **75-92% faster** |
| `search_notes` | 350ms-2.6s | **<500ms** | **Responsive** |
| `list_memory_projects` | N/A | **<100ms** | **Fast** |
**Key Achievements:**
-**95-99% improvement** in read operations (primary workflow)
-**75-92% improvement** in edit operations
-**Zero overhead** for project switching
-**Database connection overhead eliminated** (0ms vs 50-100ms)
-**Project reconciliation delays removed** from API requests
-**<500ms target achieved** for all operations except write (which includes file sync)
### ✅ Backwards Compatibility - MAINTAINED
- All existing functionality preserved
- CLI operations unaffected
- Fallback for non-API contexts maintained
- No breaking changes to existing APIs
- Optional configuration with safe defaults
### ✅ Testing Validation - PASSED
- Integration tests passing
- Type checking clear
- Linting checks passed
- Live testing with real MCP tools successful
- Multi-project workflows validated
- Rapid project switching validated
## Notes
Notes
Implementation Priority:
- Phase 1 COMPLETED: Database connection caching provides 95%+ performance gains
- Phase 2 NOT NEEDED: Project reconciliation removal achieved the goals
- Phase 3 INCLUDED: skip_initialization_sync flag added
- Phase 1 provides 80% of performance gains and should be implemented first
- Phase 2 provides remaining 20% and addresses edge cases
- Phase 3 is optional for maximum cloud optimization
Risk Mitigation:
- All changes backwards compatible implemented
- Gradual implementation successful (Phase 1 → validation)
- Easy rollback via configuration flags available
- All changes backwards compatible
- Gradual rollout possible (Phase 1 → 2 → 3)
- Easy rollback via configuration flags
Cloud Integration:
- This optimization directly addresses basic-memory-cloud issue #82
- Changes in core basic-memory will benefit all cloud tenants
- No changes needed in basic-memory-cloud itself
**Result**: SPEC-11 performance optimizations successfully implemented and validated. The 95-99% improvement in MCP tool response times exceeds the original 50-80% target, providing exceptional performance gains for cloud deployments and local usage.
- This optimization directly addresses basic-memory-cloud issue #82
- Changes in core basic-memory will benefit all cloud tenants
- No changes needed in basic-memory-cloud itself
@@ -0,0 +1,182 @@
# SPEC-12: OpenTelemetry Observability
## Why
We need comprehensive observability for basic-memory-cloud to:
- Track request flows across our multi-tenant architecture (MCP → Cloud → API services)
- Debug performance issues and errors in production
- Understand user behavior and system usage patterns
- Correlate issues to specific tenants for targeted debugging
- Monitor service health and latency across the distributed system
Currently, we only have basic logging without request correlation or distributed tracing capabilities.
## What
Implement OpenTelemetry instrumentation across all basic-memory-cloud services with:
### Core Requirements
1. **Distributed Tracing**: End-to-end request tracing from MCP gateway through to tenant API instances
2. **Tenant Correlation**: All traces tagged with tenant_id, user_id, and workos_user_id
3. **Service Identification**: Clear service naming and namespace separation
4. **Auto-instrumentation**: Automatic tracing for FastAPI, SQLAlchemy, HTTP clients
5. **Grafana Cloud Integration**: Direct OTLP export to Grafana Cloud Tempo
### Services to Instrument
- **MCP Gateway** (basic-memory-mcp): Entry point with JWT extraction
- **Cloud Service** (basic-memory-cloud): Provisioning and management operations
- **API Service** (basic-memory-api): Tenant-specific instances
- **Worker Processes** (ARQ workers): Background job processing
### Key Trace Attributes
- `tenant.id`: UUID from UserProfile.tenant_id
- `user.id`: WorkOS user identifier
- `user.email`: User email for debugging
- `service.name`: Specific service identifier
- `service.namespace`: Environment (development/production)
- `operation.type`: Business operation (provision/update/delete)
- `tenant.app_name`: Fly.io app name for tenant instances
## How
### Phase 1: Setup OpenTelemetry SDK
1. Add OpenTelemetry dependencies to each service's pyproject.toml:
```python
"opentelemetry-distro[otlp]>=1.29.0",
"opentelemetry-instrumentation-fastapi>=0.50b0",
"opentelemetry-instrumentation-httpx>=0.50b0",
"opentelemetry-instrumentation-sqlalchemy>=0.50b0",
"opentelemetry-instrumentation-logging>=0.50b0",
```
2. Create shared telemetry initialization module (`apps/shared/telemetry.py`)
3. Configure Grafana Cloud OTLP endpoint via environment variables:
```bash
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-2.grafana.net/otlp
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic[token]
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
```
### Phase 2: Instrument MCP Gateway
1. Extract tenant context from AuthKit JWT in middleware
2. Create root span with tenant attributes
3. Propagate trace context to downstream services via headers
### Phase 3: Instrument Cloud Service
1. Continue trace from MCP gateway
2. Add operation-specific attributes (provisioning events)
3. Instrument ARQ worker jobs for async operations
4. Track Fly.io API calls and latency
### Phase 4: Instrument API Service
1. Extract tenant context from JWT
2. Add machine-specific metadata (instance ID, region)
3. Instrument database operations with SQLAlchemy
4. Track MCP protocol operations
### Phase 5: Configure and Deploy
1. Add OTLP configuration to `.env.example` and `.env.example.secrets`
2. Set Fly.io secrets for production deployment
3. Update Dockerfiles to use `opentelemetry-instrument` wrapper
4. Deploy to development environment first for testing
## How to Evaluate
### Success Criteria
1. **End-to-end traces visible in Grafana Cloud** showing complete request flow
2. **Tenant filtering works** - Can filter traces by tenant_id to see all requests for a user
3. **Service maps accurate** - Grafana shows correct service dependencies
4. **Performance overhead < 5%** - Minimal latency impact from instrumentation
5. **Error correlation** - Can trace errors back to specific tenant and operation
### Testing Checklist
- [x] Single request creates connected trace across all services
- [x] Tenant attributes present on all spans
- [x] Background jobs (ARQ) appear in traces
- [x] Database queries show in trace timeline
- [x] HTTP calls to Fly.io API tracked
- [x] Traces exported successfully to Grafana Cloud
- [x] Can search traces by tenant_id in Grafana
- [x] Service dependency graph shows correct flow
### Monitoring Success
- All services reporting traces to Grafana Cloud
- No OTLP export errors in logs
- Trace sampling working correctly (if implemented)
- Resource usage acceptable (CPU/memory)
## Dependencies
- Grafana Cloud account with OTLP endpoint configured
- OpenTelemetry Python SDK v1.29.0+
- FastAPI instrumentation compatibility
- Network access from Fly.io to Grafana Cloud
## Implementation Assignment
**Recommended Agent**: python-developer
- Requires Python/FastAPI expertise
- Needs understanding of distributed systems
- Must implement middleware and context propagation
- Should understand OpenTelemetry SDK and instrumentation
## Follow-up Tasks
### Enhanced Log Correlation
While basic trace-to-log correlation works automatically via OpenTelemetry logging instrumentation, consider adding structured logging for improved log filtering:
1. **Structured Logging Context**: Add `logger.bind()` calls to inject tenant/user context directly into log records
2. **Custom Loguru Formatter**: Extract OpenTelemetry span attributes for better log readability
3. **Direct Log Filtering**: Enable searching logs directly by tenant_id, workflow_id without going through traces
This would complement the existing automatic trace correlation and provide better log search capabilities.
## Alternative Solution: Logfire
After implementing OpenTelemetry with Grafana Cloud, we discovered limitations in the observability experience:
- Traces work but lack useful context without correlated logs
- Setting up log correlation with Grafana is complex and requires additional infrastructure
- The developer experience for Python observability is suboptimal
### Logfire Evaluation
**Pydantic Logfire** offers a compelling alternative that addresses your specific requirements:
#### Core Requirements Match
- ✅ **User Activity Tracking**: Automatic request tracing with business context
- ✅ **Error Monitoring**: Built-in exception tracking with full context
- ✅ **Performance Metrics**: Automatic latency and performance monitoring
- ✅ **Request Tracing**: Native distributed tracing across services
- ✅ **Log Correlation**: Seamless trace-to-log correlation without setup
#### Key Advantages
1. **Python-First Design**: Built specifically for Python/FastAPI applications by the Pydantic team
2. **Simple Integration**: `pip install logfire` + `logfire.configure()` vs complex OTLP setup
3. **Automatic Correlation**: Logs automatically include trace context without manual configuration
4. **Real-time SQL Interface**: Query spans and logs using SQL with auto-completion
5. **Better Developer UX**: Purpose-built observability UI vs generic Grafana dashboards
6. **Loguru Integration**: `logger.configure(handlers=[logfire.loguru_handler()])` maintains existing logging
#### Pricing Assessment
- **Free Tier**: 10M spans/month (suitable for development and small production workloads)
- **Transparent Pricing**: $1 per million spans/metrics after free tier
- **No Hidden Costs**: No per-host fees, only usage-based metering
- **Production Ready**: Recently exited beta, enterprise features available
#### Migration Path
The existing OpenTelemetry instrumentation is compatible - Logfire uses OpenTelemetry under the hood, so the current spans and attributes would work unchanged.
### Recommendation
**Consider migrating to Logfire** for the following reasons:
1. It directly addresses the "next to useless" traces problem by providing integrated logs
2. Dramatically simpler setup and maintenance compared to Grafana Cloud + custom log correlation
3. Better ROI on observability investment with purpose-built Python tooling
4. Free tier sufficient for current development needs with clear scaling path
The current Grafana Cloud implementation provides a solid foundation and could remain as a backup/export target, while Logfire becomes the primary observability platform.
## Status
**Created**: 2024-01-28
**Status**: Completed (OpenTelemetry + Grafana Cloud)
**Next Phase**: Evaluate Logfire migration
**Priority**: High - Critical for production observability
@@ -340,398 +340,6 @@ This architecture makes the fix comprehensive and maintainable.
- Want to reduce database dependency
- Scale requires fewer database queries
## Post-Deployment Test Plan
This test plan should be executed after deploying the cloud service to verify subscription validation works end-to-end.
### Prerequisites
Before testing, ensure you have:
- [ ] Cloud service deployed with Phase 1 changes
- [ ] CLI installed with Phase 2 changes (`basic-memory` from local dev)
- [ ] Access to database to check/modify subscription status
- [ ] Two test user accounts:
- User A: No subscription (fresh WorkOS signup)
- User B: Active subscription (via Polar or manual DB insert)
### Test Execution
#### Test 1: User Without Subscription (Blocked Access) ❌
**Setup:**
1. Create fresh WorkOS account (User A) via AuthKit
2. Verify in database: No subscription record exists for User A's `workos_user_id`
**Test Steps:**
```bash
# Step 1: Attempt login
bm cloud login
```
**Expected Results:**
- ✅ OAuth flow completes successfully
- ✅ JWT token obtained and stored in `~/.basic-memory/auth/token`
- ❌ Login fails with "Subscription Required" error
- ✅ Error message displays:
- "✗ Subscription Required"
- "Active subscription required for CLI access"
- Subscribe URL: "https://basicmemory.com/subscribe"
- Instructions to run `bm cloud login` after subscribing
- ❌ Cloud mode NOT enabled (check with `bm cloud status`)
**Test Steps (continued):**
```bash
# Step 2: Attempt to access cloud features
bm cloud status
# Step 3: Try direct API call
curl -H "Authorization: Bearer <token>" https://<cloud-host>/proxy/health
```
**Expected Results:**
- ✅ `bm cloud status` shows "Mode: Local (disabled)"
- ✅ Direct API call returns 403 with subscription_required error
**Database Verification:**
```sql
-- Verify no subscription exists
SELECT * FROM subscriptions
WHERE workos_user_id = '<user-a-workos-id>';
-- Should return 0 rows
```
---
#### Test 2: User With Active Subscription (Full Access) ✅
**Setup:**
1. Use User B with active subscription
2. Verify in database: Subscription exists with `status = 'active'` and `current_period_end > NOW()`
**Database Verification:**
```sql
-- Verify active subscription exists
SELECT workos_user_id, status, current_period_end
FROM subscriptions
WHERE workos_user_id = '<user-b-workos-id>';
-- Should show: status='active', current_period_end in future
```
**Test Steps:**
```bash
# Step 1: Login
bm cloud login
# Step 2: Check cloud mode
bm cloud status
# Step 3: Setup bisync
bm cloud setup
# Step 4: Test MCP tools via proxy
curl -H "Authorization: Bearer <token>" \
https://<cloud-host>/proxy/<project-name>/health
# Step 5: List projects
bm project list
# Step 6: Create a test note
bm tool write-note \
--title "Test Note" \
--folder "test-project" \
--content "Testing subscription validation"
```
**Expected Results:**
- ✅ Login succeeds without errors
- ✅ Cloud mode enabled: "Mode: Cloud (enabled)"
- ✅ Cloud instance health check succeeds
- ✅ Bisync setup completes successfully
- ✅ Direct API calls succeed (200 OK)
- ✅ Projects list successfully
- ✅ Note creation succeeds
---
#### Test 3: Subscription Expiration (Access Revoked) 🔄
**Setup:**
1. Use User B (currently has active subscription and cloud mode enabled)
2. User should be able to access cloud features initially
**Test Steps:**
```bash
# Step 1: Verify current access works
bm cloud status
# Should show "Cloud (enabled)" and healthy instance
# Step 2: Expire subscription in database
# (See SQL below)
# Step 3: Attempt to access cloud features
bm cloud status
# Step 4: Try to login again
bm cloud logout
bm cloud login
```
**Database Operations:**
```sql
-- Expire the subscription
UPDATE subscriptions
SET status = 'cancelled',
current_period_end = NOW() - INTERVAL '1 day'
WHERE workos_user_id = '<user-b-workos-id>';
-- Verify expiration
SELECT workos_user_id, status, current_period_end
FROM subscriptions
WHERE workos_user_id = '<user-b-workos-id>';
-- Should show: status='cancelled', current_period_end in past
```
**Expected Results:**
- ❌ `bm cloud status` fails with 403 subscription_required error
- ❌ Re-login fails with "Subscription Required" error
- ✅ Error includes subscribe URL
---
#### Test 4: Subscription Renewal (Access Restored) ✅
**Setup:**
1. Continue from Test 3 (User B with expired subscription)
**Test Steps:**
```bash
# Step 1: Renew subscription in database
# (See SQL below)
# Step 2: Login again
bm cloud login
# Step 3: Verify access restored
bm cloud status
# Step 4: Test project access
bm project list
```
**Database Operations:**
```sql
-- Renew the subscription
UPDATE subscriptions
SET status = 'active',
current_period_end = NOW() + INTERVAL '30 days'
WHERE workos_user_id = '<user-b-workos-id>';
-- Verify renewal
SELECT workos_user_id, status, current_period_end
FROM subscriptions
WHERE workos_user_id = '<user-b-workos-id>';
-- Should show: status='active', current_period_end 30 days in future
```
**Expected Results:**
- ✅ Login succeeds
- ✅ Cloud mode enabled
- ✅ Cloud status shows healthy
- ✅ Projects list successfully
- ✅ **Access immediately restored** (no delay)
---
#### Test 5: Endpoint Coverage (All Protected Endpoints) 🔐
**Setup:**
1. Use User A (no subscription) to test blocked access
2. Use User B (active subscription) to test allowed access
**Test Matrix:**
| Endpoint | Method | User A (No Sub) | User B (Active Sub) |
|----------|--------|----------------|---------------------|
| `/proxy/health` | GET | 403 ❌ | 200 ✅ |
| `/proxy/<project>/health` | GET | 403 ❌ | 200 ✅ |
| `/proxy/<project>/search` | POST | 403 ❌ | 200 ✅ |
| `/tenant/mount/info` | GET | 403 ❌ | 200 ✅ |
| `/tenant/mount/credentials` | POST | 403 ❌ | 200 ✅ |
**Test Commands:**
```bash
# Get tokens for both users
TOKEN_A="<user-a-token>"
TOKEN_B="<user-b-token>"
# Test /proxy/health
curl -H "Authorization: Bearer $TOKEN_A" \
https://<cloud-host>/proxy/health
# Expected: 403 with subscription_required
curl -H "Authorization: Bearer $TOKEN_B" \
https://<cloud-host>/proxy/health
# Expected: 200 OK
# Test /tenant/mount/info
curl -H "Authorization: Bearer $TOKEN_A" \
https://<cloud-host>/tenant/mount/info
# Expected: 403 with subscription_required
curl -H "Authorization: Bearer $TOKEN_B" \
https://<cloud-host>/tenant/mount/info
# Expected: 200 OK with mount info
# Test /proxy/<project>/health
curl -H "Authorization: Bearer $TOKEN_B" \
https://<cloud-host>/proxy/<project-name>/health
# Expected: 200 OK
```
---
#### Test 6: Error Response Format Validation 📋
**Test Steps:**
```bash
# Get 403 response for user without subscription
curl -i -H "Authorization: Bearer $TOKEN_A" \
https://<cloud-host>/proxy/health
```
**Expected Response Format:**
```http
HTTP/1.1 403 Forbidden
Content-Type: application/json
{
"error": "subscription_required",
"message": "Active subscription required for CLI access",
"subscribe_url": "https://basicmemory.com/subscribe"
}
```
**Validation Checklist:**
- ✅ Status code is exactly 403
- ✅ Response is valid JSON
- ✅ `error` field equals "subscription_required"
- ✅ `message` field is present and informative
- ✅ `subscribe_url` field is present and valid URL
---
#### Test 7: Admin Access Bypass 👑
**Purpose:** Verify admin users can still access admin endpoints without subscription
**Setup:**
1. Use admin user account (member of admin organization in WorkOS)
**Test Steps:**
```bash
# Login as admin
python -m basic_memory_cloud.cli.tenant_cli login
# List tenants (admin-only endpoint)
python -m basic_memory_cloud.cli.tenant_cli list-tenants
# Create tenant (admin-only endpoint)
python -m basic_memory_cloud.cli.tenant_cli create-tenant \
--workos-user-id <test-user-id>
```
**Expected Results:**
- ✅ Admin login succeeds
- ✅ Admin can access `/tenants/*` endpoints
- ✅ Admin operations work regardless of subscription status
- ✅ Admin endpoints use `AdminUserHybridDep` (not affected by subscription check)
---
### Test Results Template
Copy this template to track your test execution:
```markdown
## SPEC-13 Test Execution - [Date]
### Environment
- Cloud Service: [URL]
- Cloud Service Version: [commit/tag]
- CLI Version: [commit/tag]
- Database: [production/staging]
### Test Results
#### Test 1: User Without Subscription ❌
- [ ] OAuth flow succeeds
- [ ] Subscription error displayed
- [ ] Subscribe URL shown
- [ ] Cloud mode NOT enabled
- [ ] Direct API call blocked
**Issues:** [None / List issues]
#### Test 2: User With Active Subscription ✅
- [ ] Login succeeds
- [ ] Cloud mode enabled
- [ ] Health check passes
- [ ] Bisync setup works
- [ ] MCP tools work
- [ ] Projects accessible
**Issues:** [None / List issues]
#### Test 3: Subscription Expiration 🔄
- [ ] Active user can access initially
- [ ] After expiration, access blocked
- [ ] Error message clear
- [ ] Cloud status fails appropriately
**Issues:** [None / List issues]
#### Test 4: Subscription Renewal ✅
- [ ] Renewed subscription in DB
- [ ] Login succeeds immediately
- [ ] Access fully restored
- [ ] No caching delays
**Issues:** [None / List issues]
#### Test 5: Endpoint Coverage 🔐
- [ ] All proxy endpoints protected
- [ ] All mount endpoints protected
- [ ] Subscription check consistent
- [ ] Error responses correct
**Issues:** [None / List issues]
#### Test 6: Error Response Format 📋
- [ ] 403 status code
- [ ] Valid JSON response
- [ ] All required fields present
- [ ] Subscribe URL valid
**Issues:** [None / List issues]
#### Test 7: Admin Access Bypass 👑
- [ ] Admin login works
- [ ] Admin endpoints accessible
- [ ] No subscription requirement
**Issues:** [None / List issues]
### Overall Result
- [ ] All tests passed
- [ ] Ready for production
**Summary:** [Brief summary of test execution]
**Sign-off:** [Your name/date]
```
---
## How to Evaluate
### Success Criteria
@@ -1065,35 +673,35 @@ The extra HTTP hop is minimal (< 10ms) and worth it for architectural benefits.
- [ ] Call `/tenant/mount/info` with valid JWT and active subscription → expect 200
- [ ] Verify error response structure matches spec
### Phase 2: CLI (basic-memory)
### Phase 2: CLI (basic-memory)
#### Task 2.1: Review and understand CLI authentication flow
#### Task 2.1: Review and understand CLI authentication flow
**Files**: `src/basic_memory/cli/commands/cloud/`
- [x] Read `core_commands.py` to understand current login flow
- [x] Read `api_client.py` to understand current error handling
- [x] Identify where 403 errors should be caught
- [x] Identify what error messages should be displayed
- [x] Document current behavior in spec if needed
- [ ] Read `core_commands.py` to understand current login flow
- [ ] Read `api_client.py` to understand current error handling
- [ ] Identify where 403 errors should be caught
- [ ] Identify what error messages should be displayed
- [ ] Document current behavior in spec if needed
#### Task 2.2: Update API client error handling
#### Task 2.2: Update API client error handling
**File**: `src/basic_memory/cli/commands/cloud/api_client.py`
- [x] Add custom exception class `SubscriptionRequiredError` (or similar)
- [x] Update HTTP error handling to parse 403 responses
- [x] Extract `error`, `message`, and `subscribe_url` from error detail
- [x] Raise specific exception for subscription_required errors
- [x] Run `just typecheck` in basic-memory repo to verify types
- [ ] Add custom exception class `SubscriptionRequiredError` (or similar)
- [ ] Update HTTP error handling to parse 403 responses
- [ ] Extract `error`, `message`, and `subscribe_url` from error detail
- [ ] Raise specific exception for subscription_required errors
- [ ] Run `just typecheck` in basic-memory repo to verify types
#### Task 2.3: Update CLI login command error handling
#### Task 2.3: Update CLI login command error handling
**File**: `src/basic_memory/cli/commands/cloud/core_commands.py`
- [x] Import the subscription error exception
- [x] Wrap login flow with try/except for subscription errors
- [x] Display user-friendly error message with rich console
- [x] Show subscribe URL prominently
- [x] Provide actionable next steps
- [x] Run `just typecheck` to verify types
- [ ] Import the subscription error exception
- [ ] Wrap login flow with try/except for subscription errors
- [ ] Display user-friendly error message with rich console
- [ ] Show subscribe URL prominently
- [ ] Provide actionable next steps
- [ ] Run `just typecheck` to verify types
**Expected error handling**:
```python
@@ -1111,33 +719,30 @@ except SubscriptionRequiredError as e:
raise typer.Exit(1)
```
#### Task 2.4: Update CLI tests
**File**: `tests/cli/test_cloud_authentication.py` (created)
#### Task 2.4: Update CLI tests
**File**: `tests/cli/test_cloud_commands.py`
- [x] Add test: `test_login_without_subscription_shows_error()`
- [ ] Add test: `test_login_without_subscription_shows_error()`
- Mock 403 subscription_required response
- Call login command
- Assert error message displayed
- Assert subscribe URL shown
- [x] Add test: `test_login_with_subscription_succeeds()`
- [ ] Add test: `test_login_with_subscription_succeeds()`
- Mock successful authentication + subscription check
- Call login command
- Assert success message
- [x] Add test: `test_parse_subscription_required_error()` (API client error parsing)
- [x] Add test: `test_parse_generic_403_error()` (generic 403 handling)
- [x] Add test: `test_login_authentication_failure()` (auth failure handling)
- [x] Run `uv run pytest` to verify tests pass (5/5 passed)
- [ ] Run `just test` to verify tests pass
#### Task 2.5: Update CLI documentation
**File**: `docs/cloud-cli.md`
#### Task 2.5: Update CLI documentation
**File**: `docs/cloud-cli.md` (in basic-memory-docs repo)
- [x] Add "Prerequisites" section if not present
- [x] Document subscription requirement
- [x] Add "Troubleshooting" section
- [x] Document "Subscription Required" error
- [x] Provide subscribe URL
- [x] Add FAQ entry about subscription errors
- [x] Build docs locally to verify formatting
- [ ] Add "Prerequisites" section if not present
- [ ] Document subscription requirement
- [ ] Add "Troubleshooting" section
- [ ] Document "Subscription Required" error
- [ ] Provide subscribe URL
- [ ] Add FAQ entry about subscription errors
- [ ] Build docs locally to verify formatting
### Phase 3: End-to-End Testing
@@ -1227,12 +832,12 @@ Use this high-level checklist to track overall progress:
- [ ] Add integration tests for dependency
- [ ] Deploy and verify cloud service
### Phase 2: CLI Updates
- [x] Review CLI authentication flow
- [x] Update API client error handling
- [x] Update CLI login command error handling
- [x] Add CLI tests
- [x] Update CLI documentation
### Phase 2: CLI Updates 🔄
- [ ] Review CLI authentication flow
- [ ] Update API client error handling
- [ ] Update CLI login command error handling
- [ ] Add CLI tests
- [ ] Update CLI documentation
### Phase 3: End-to-End Testing 🧪
- [ ] Create test user accounts
@@ -1310,118 +915,3 @@ Use this high-level checklist to track overall progress:
- This spec prioritizes security over convenience - better to block unauthorized access than risk revenue loss
- Clear error messages are critical - users should understand why they're blocked and how to resolve it
- Consider adding telemetry to track subscription_required errors for monitoring signup conversion
## Implementation Log
### Phase 2 Completion - 2025-10-03
Phase 2 (CLI Updates) completed successfully with the following implementation:
**Files Modified:**
- `src/basic_memory/cli/commands/cloud/api_client.py` - Added `SubscriptionRequiredError` exception and enhanced error handling
- `src/basic_memory/cli/commands/cloud/core_commands.py` - Updated login command to verify subscription access
- `docs/cloud-cli.md` - Added Prerequisites and Subscription Issues sections
**Files Created:**
- `tests/cli/test_cloud_authentication.py` - Comprehensive test coverage (6 tests, all passing)
**Key Implementation Details:**
- `SubscriptionRequiredError` exception with `subscribe_url` field for user guidance
- Enhanced `CloudAPIError` to include `status_code` and `detail` fields
- Login flow now calls `/proxy/health` to verify subscription before enabling cloud mode
- User-friendly error messages with direct subscribe link
- 100% test coverage of new error handling paths
**Test Results:**
- All 6 tests passing
- Type checking: 0 errors, 0 warnings
- Linting: All checks passed
**Next Steps:**
- Phase 3: End-to-End Testing (manual testing with real users, subscription state transitions)
- Phase 1: Complete remaining cloud service tests (unit tests, integration tests, deployment verification)
---
### End-to-End Test Execution - 2025-10-03
**Environment:**
- Cloud Service: https://cloud.basicmemory.com
- Cloud Service Version: Phase 1 deployed (with subscription validation)
- CLI Version: Phase 2 implementation (local dev build)
- Database: Production
**Test Results:**
#### Test 1: User Without Subscription ✅ PASSED
- [x] OAuth flow succeeds
- [x] Subscription error displayed
- [x] Subscribe URL shown
- [x] Cloud mode NOT enabled
- [x] Clean error output (no traceback)
**Output:**
```
✅ Successfully authenticated with WorkOS!
Verifying subscription access...
✗ Subscription Required
Active subscription required
Subscribe at: https://basicmemory.com/subscribe
Once you have an active subscription, run bm cloud login again.
```
**Issues:** None
---
#### Test 2: User With Active Subscription ✅ PASSED
- [x] Login succeeds
- [x] Cloud mode enabled
- [x] Clean success message
- [x] Ready for cloud operations
**Output:**
```
✅ Successfully authenticated with WorkOS!
Verifying subscription access...
✓ Cloud mode enabled
All CLI commands now work against https://cloud.basicmemory.com
```
**Issues:** None
---
**Additional Implementation Notes:**
**API Response Format Compatibility:**
- Cloud service returns errors in FastAPI HTTPException format (nested under `"detail"` key)
- CLI correctly handles both nested and flat response formats
- Error parsing logic:
```python
detail_obj = error_detail.get("detail", error_detail)
if isinstance(detail_obj, dict) and detail_obj.get("error") == "subscription_required":
# Handle subscription error
```
**Updated Test Coverage:**
- Added `test_parse_subscription_required_error_flat_format()` for backward compatibility
- Total: 6 tests, all passing
- Files updated:
- `src/basic_memory/cli/commands/cloud/api_client.py` - Support both response formats
- `tests/cli/test_cloud_authentication.py` - Added flat format test
**Overall Result:**
- [x] Core authentication flows validated
- [x] Error handling working as designed
- [x] User experience is clean and helpful
- [x] Ready for production use
**Summary:**
SPEC-13 Phase 2 successfully validated in production environment. Both unauthorized and authorized user flows work correctly. The subscription validation is functioning end-to-end with clear, user-friendly error messages and seamless success path. No issues discovered during testing.
**Sign-off:** Phase 2 Complete - 2025-10-03
@@ -0,0 +1,210 @@
---
title: 'SPEC-14: Cloud Git Versioning & GitHub Backup'
type: spec
permalink: specs/spec-14-cloud-git-versioning
tags:
- git
- github
- backup
- versioning
- cloud
related:
- specs/spec-9-multi-project-bisync
- specs/spec-9-follow-ups-conflict-sync-and-observability
status: deferred
---
# SPEC-14: Cloud Git Versioning & GitHub Backup
**Status: DEFERRED** - Postponed until multi-user/teams feature development. Using S3 versioning (SPEC-9.1) for v1 instead.
## Why Deferred
**Original goals can be met with simpler solutions:**
- Version history → **S3 bucket versioning** (automatic, zero config)
- Offsite backup → **Tigris global replication** (built-in)
- Restore capability → **S3 version restore** (`bm cloud restore --version-id`)
- Collaboration → **Deferred to teams/multi-user feature** (not v1 requirement)
**Complexity vs value trade-off:**
- Git integration adds: committer service, puller service, webhooks, LFS, merge conflicts
- Risk: Loop detection between Git ↔ rclone bisync ↔ local edits
- S3 versioning gives 80% of value with 5% of complexity
**When to revisit:**
- Teams/multi-user features (PR-based collaboration workflow)
- User requests for commit messages and branch-based workflows
- Need for fine-grained audit trail beyond S3 object metadata
---
## Original Specification (for reference)
## Why
Early access users want **transparent version history**, easy **offsite backup**, and a familiar **restore/branching** workflow. Git/GitHub integration would provide:
- Auditable history of every change (who/when/why)
- Branches/PRs for review and collaboration
- Offsite private backup under the user's control
- Escape hatch: users can always `git clone` their knowledge base
**Note:** These goals are now addressed via S3 versioning (SPEC-9.1) for single-user use case.
## Goals
- **Transparent**: Users keep using Basic Memory; Git runs behind the scenes.
- **Private**: Push to a **private GitHub repo** that the user owns (or tenant org).
- **Reliable**: No data loss, deterministic mapping of filesystem ↔ Git.
- **Composable**: Plays nicely with SPEC9 bisync and upcoming conflict features (SPEC9 FollowUps).
**NonGoals (for v1):**
- Finegrained perfile encryption in Git history (can be layered later).
- Large media optimization beyond Git LFS defaults.
## User Stories
1. *As a user*, I connect my GitHub and choose a private backup repo.
2. *As a user*, every change I make in cloud (or via bisync) is **committed** and **pushed** automatically.
3. *As a user*, I can **restore** a file/folder/project to a prior version.
4. *As a power user*, I can **git pull/push** directly to collaborate outside the app.
5. *As an admin*, I can enforce repo ownership (tenant org) and leastprivilege scopes.
## Scope
- **In scope:** Full repo backup of `/app/data/` (all projects) with optional selective subpaths.
- **Out of scope (v1):** Partial shallow mirrors; encrypted Git; crossprovider SCM (GitLab/Bitbucket).
## Architecture
### Topology
- **Authoritative working tree**: `/app/data/` (bucket mount) remains the source of truth (SPEC9).
- **Bare repo** lives alongside: `/app/git/${tenant}/knowledge.git` (serverside).
- **Mirror remote**: `github.com/<owner>/<repo>.git` (private).
```mermaid
flowchart LR
A[/Users & Agents/] -->|writes/edits| B[/app/data/]
B -->|file events| C[Committer Service]
C -->|git commit| D[(Bare Repo)]
D -->|push| E[(GitHub Private Repo)]
E -->|webhook (push)| F[Puller Service]
F -->|git pull/merge| D
D -->|checkout/merge| B
```
### Services
- **Committer Service** (daemon):
- Watches `/app/data/` for changes (inotify/poll)
- Batches changes (debounce e.g. 25s)
- Writes `.bmmeta` (if present) into commit message trailer (see FollowUps)
- `git add -A && git commit -m "chore(sync): <summary>
BM-Meta: <json>"`
- Periodic `git push` to GitHub mirror (configurable interval)
- **Puller Service** (webhook target):
- Receives GitHub webhook (push) → `git fetch`
- **Fastforward** merges to `main` only; reject nonFF unless policy allows
- Applies changes back to `/app/data/` via clean checkout
- Emits sync events for Basic Memory indexers
### Auth & Security
- **GitHub App** (recommended): minimal scopes: `contents:read/write`, `metadata:read`, webhook.
- Tenantscoped installation; repo created in user account or tenant org.
- Tokens stored in KMS/secret manager; rotated automatically.
- Optional policy: allow only **FF merges** on `main`; nonFF requires PR.
### Repo Layout
- **Monorepo** (default): one repo per tenant mirrors `/app/data/` with subfolders per project.
- Optional multirepo mode (later): one repo per project.
### File Handling
- Honor `.gitignore` generated from `.bmignore.rclone` + BM defaults (cache, temp, state).
- **Git LFS** for large binaries (images, media) — auto track by extension/size threshold.
- Normalize newline + Unicode (aligns with FollowUps).
### Conflict Model
- **Primary concurrency**: SPEC9 FollowUps (`.bmmeta`, conflict copies) stays the first line of defense.
- **Git merges** are a **secondary** mechanism:
- Server only automerges **text** conflicts when trivial (FF or clean 3way).
- Otherwise, create `name (conflict from <branch>, <ts>).md` and surface via events.
### Data Flow vs Bisync
- Bisync (rclone) continues between local sync dir ↔ bucket.
- Git sits **cloudside** between bucket and GitHub.
- On **pull** from GitHub → files written to `/app/data/` → picked up by indexers & eventually by bisync back to users.
## CLI & UX
New commands (cloud mode):
- `bm cloud git connect` — Launch GitHub App installation; create private repo; store installation id.
- `bm cloud git status` — Show connected repo, last push time, last webhook delivery, pending commits.
- `bm cloud git push` — Manual push (rarely needed).
- `bm cloud git pull` — Manual pull/FF (admin only by default).
- `bm cloud snapshot -m "message"` — Create a tagged pointintime snapshot (git tag).
- `bm restore <path> --to <commit|tag>` — Restore file/folder/project to prior version.
Settings:
- `bm config set git.autoPushInterval=5s`
- `bm config set git.lfs.sizeThreshold=10MB`
- `bm config set git.allowNonFF=false`
## Migration & Backfill
- On connect, if repo empty: initial commit of entire `/app/data/`.
- If repo has content: require **onetime import** path (clone to staging, reconcile, choose direction).
## Edge Cases
- Massive deletes: gated by SPEC9 `max_delete` **and** Git prepush hook checks.
- Case changes and rename detection: rely on git rename heuristics + FollowUps move hints.
- Secrets: default ignore common secret patterns; allow custom deny list.
## Telemetry & Observability
- Emit `git_commit`, `git_push`, `git_pull`, `git_conflict` events with correlation IDs.
- `bm sync --report` extended with Git stats (commit count, delta bytes, push latency).
## Phased Plan
### Phase 0 — Prototype (1 sprint)
- Server: bare repo init + simple committer (batch every 10s) + manual GitHub token.
- CLI: `bm cloud git connect --token <PAT>` (devonly)
- Success: edits in `/app/data/` appear in GitHub within 30s.
### Phase 1 — GitHub App & Webhooks (12 sprints)
- Switch to GitHub App installs; create private repo; store installation id.
- Committer hardened (debounce 25s, backoff, retries).
- Puller service with webhook → FF merge → checkout to `/app/data/`.
- LFS autotrack + `.gitignore` generation.
- CLI surfaces status + logs.
### Phase 2 — Restore & Snapshots (1 sprint)
- `bm restore` for file/folder/project with dryrun.
- `bm cloud snapshot` tags + list/inspect.
- Policy: PRonly nonFF, admin override.
### Phase 3 — Selective & MultiRepo (nicetohave)
- Include/exclude projects; optional perproject repos.
- Advanced policies (branch protections, required reviews).
## Acceptance Criteria
- Changes to `/app/data/` are committed and pushed automatically within configurable interval (default ≤5s).
- GitHub webhook pull results in updated files in `/app/data/` (FFonly by default).
- LFS configured and functioning; large files don't bloat history.
- `bm cloud git status` shows connected repo and last push/pull times.
- `bm restore` restores a file/folder to a prior commit with a clear audit trail.
- Endtoend works alongside SPEC9 bisync without loops or data loss.
## Risks & Mitigations
- **Loop risk (Git ↔ Bisync)**: Writes to `/app/data/` → bisync → local → user edits → back again. *Mitigation*: Debounce, commit squashing, idempotent `.bmmeta` versioning, and watch exclusion windows during pull.
- **Repo bloat**: Lots of binary churn. *Mitigation*: default LFS, size threshold, optional mediaonly repo later.
- **Security**: Token leakage. *Mitigation*: GitHub App with shortlived tokens, KMS storage, scoped permissions.
- **Merge complexity**: Nontrivial conflicts. *Mitigation*: prefer FF; otherwise conflict copies + events; require PR for nonFF.
## Open Questions
- Do we default to **monorepo** per tenant, or offer projectperrepo at connect time?
- Should `restore` write to a branch and open a PR, or directly modify `main`?
- How do we expose Git history in UI (timeline view) without users dropping to CLI?
## Appendix: Sample Config
```json
{
"git": {
"enabled": true,
"repo": "https://github.com/<owner>/<repo>.git",
"autoPushInterval": "5s",
"allowNonFF": false,
"lfs": { "sizeThreshold": 10485760 }
}
}
```
@@ -0,0 +1,273 @@
---
title: 'SPEC-15: Configuration Persistence via Tigris for Cloud Tenants'
type: spec
permalink: specs/spec-14-config-persistence-tigris
tags:
- persistence
- tigris
- multi-tenant
- infrastructure
- configuration
status: draft
---
# SPEC-15: Configuration Persistence via Tigris for Cloud Tenants
## Why
We need to persist Basic Memory configuration across Fly.io deployments without using persistent volumes or external databases.
**Current Problems:**
- `~/.basic-memory/config.json` lost on every deployment (project configuration)
- `~/.basic-memory/memory.db` lost on every deployment (search index)
- Persistent volumes break clean deployment workflow
- External databases (Turso) require per-tenant token management
**The Insight:**
The SQLite database is just an **index cache** of the markdown files. It can be rebuilt in seconds from the source markdown files in Tigris. Only the small `config.json` file needs true persistence.
**Solution:**
- Store `config.json` in Tigris bucket (persistent, small file)
- Rebuild `memory.db` on startup from markdown files (fast, ephemeral)
- No persistent volumes, no external databases, no token management
## What
Store Basic Memory configuration in the Tigris bucket and rebuild the database index on tenant machine startup.
**Affected Components:**
- `basic-memory/src/basic_memory/config.py` - Add configurable config directory
**Architecture:**
```bash
# Tigris Bucket (persistent, mounted at /app/data)
/app/data/
├── .basic-memory/
│ └── config.json # ← Project configuration (persistent, accessed via BASIC_MEMORY_CONFIG_DIR)
└── basic-memory/ # ← Markdown files (persistent, BASIC_MEMORY_HOME)
├── project1/
└── project2/
# Fly Machine (ephemeral)
/app/.basic-memory/
└── memory.db # ← Rebuilt on startup (fast local disk)
```
## How (High Level)
### 1. Add Configurable Config Directory to Basic Memory
Currently `ConfigManager` hardcodes `~/.basic-memory/config.json`. Add environment variable to override:
```python
# basic-memory/src/basic_memory/config.py
class ConfigManager:
"""Manages Basic Memory configuration."""
def __init__(self) -> None:
"""Initialize the configuration manager."""
home = os.getenv("HOME", Path.home())
if isinstance(home, str):
home = Path(home)
# Allow override via environment variable
if config_dir := os.getenv("BASIC_MEMORY_CONFIG_DIR"):
self.config_dir = Path(config_dir)
else:
self.config_dir = home / DATA_DIR_NAME
self.config_file = self.config_dir / CONFIG_FILE_NAME
# Ensure config directory exists
self.config_dir.mkdir(parents=True, exist_ok=True)
```
### 2. Rebuild Database on Startup
Basic Memory already has the sync functionality. Just ensure it runs on startup:
```python
# apps/api/src/basic_memory_cloud_api/main.py
@app.on_event("startup")
async def startup_sync():
"""Rebuild database index from Tigris markdown files."""
logger.info("Starting database rebuild from Tigris")
# Initialize file sync (rebuilds index from markdown files)
app_config = ConfigManager().config
await initialize_file_sync(app_config)
logger.info("Database rebuild complete")
```
### 3. Environment Configuration
```bash
# Machine environment variables
BASIC_MEMORY_CONFIG_DIR=/app/data/.basic-memory # Config read/written directly to Tigris
# memory.db stays in default location: /app/.basic-memory/memory.db (local ephemeral disk)
```
## Implementation Task List
### Phase 1: Basic Memory Changes ✅
- [x] Add `BASIC_MEMORY_CONFIG_DIR` environment variable support to `ConfigManager.__init__()`
- [x] Test config loading from custom directory
- [x] Update tests to verify custom config dir works
### Phase 2: Tigris Bucket Structure ✅
- [x] Ensure `.basic-memory/` directory exists in Tigris bucket on tenant creation
- ✅ ConfigManager auto-creates on first run, no explicit provisioning needed
- [x] Initialize `config.json` in Tigris on first tenant deployment
- ✅ ConfigManager creates config.json automatically in BASIC_MEMORY_CONFIG_DIR
- [x] Verify TigrisFS handles hidden directories correctly
- ✅ TigrisFS supports hidden directories (verified in SPEC-8)
### Phase 3: Deployment Integration ✅
- [x] Set `BASIC_MEMORY_CONFIG_DIR` environment variable in machine deployment
- ✅ Added to BasicMemoryMachineConfigBuilder in fly_schemas.py
- [x] Ensure database rebuild runs on machine startup via initialization sync
- ✅ sync_worker.py runs initialize_file_sync every 30s (already implemented)
- [x] Handle first-time tenant setup (no config exists yet)
- ✅ ConfigManager creates config.json on first initialization
- [ ] Test deployment workflow with config persistence
### Phase 4: Testing
- [x] Unit tests for config directory override
- [-] Integration test: deploy → write config → redeploy → verify config persists
- [ ] Integration test: deploy → add project → redeploy → verify project in config
- [ ] Performance test: measure db rebuild time on startup
### Phase 5: Documentation
- [ ] Document config persistence architecture
- [ ] Update deployment runbook
- [ ] Document startup sequence and timing
## How to Evaluate
### Success Criteria
1. **Config Persistence**
- [ ] config.json persists across deployments
- [ ] Projects list maintained across restarts
- [ ] No manual configuration needed after redeploy
2. **Database Rebuild**
- [ ] memory.db rebuilt on startup in < 30 seconds
- [ ] All entities indexed correctly
- [ ] Search functionality works after rebuild
3. **Performance**
- [ ] SQLite queries remain fast (local disk)
- [ ] Config reads acceptable (symlink to Tigris)
- [ ] No noticeable performance degradation
4. **Deployment Workflow**
- [ ] Clean deployments without volumes
- [ ] No new external dependencies
- [ ] No secret management needed
### Testing Procedure
1. **Config Persistence Test**
```bash
# Deploy tenant
POST /tenants → tenant_id
# Add a project
basic-memory project add "test-project" ~/test
# Verify config has project
cat /app/data/.basic-memory/config.json
# Redeploy machine
fly deploy --app basic-memory-{tenant_id}
# Verify project still exists
basic-memory project list
```
2. **Database Rebuild Test**
```bash
# Create notes
basic-memory write "Test Note" --content "..."
# Redeploy (db lost)
fly deploy --app basic-memory-{tenant_id}
# Wait for startup sync
sleep 10
# Verify note is indexed
basic-memory search "Test Note"
```
3. **Performance Benchmark**
```bash
# Time the startup sync
time basic-memory sync
# Should be < 30 seconds for typical tenant
```
## Benefits Over Alternatives
**vs. Persistent Volumes:**
- ✅ Clean deployment workflow
- ✅ No volume migration needed
- ✅ Simpler infrastructure
**vs. Turso (External Database):**
- ✅ No per-tenant token management
- ✅ No external service dependencies
- ✅ No additional costs
- ✅ Simpler architecture
**vs. SQLite on FUSE:**
- ✅ Fast local SQLite performance
- ✅ Only slow reads for small config file
- ✅ Database queries remain fast
## Implementation Assignment
**Primary Agent:** `python-developer`
- Add `BASIC_MEMORY_CONFIG_DIR` environment variable to ConfigManager
- Update deployment workflow to set environment variable
- Ensure startup sync runs correctly
**Review Agent:** `system-architect`
- Validate architecture simplicity
- Review performance implications
- Assess startup timing
## Dependencies
- **Internal:** TigrisFS must be working and stable
- **Internal:** Basic Memory sync must be reliable
- **Internal:** SPEC-8 (TigrisFS Integration) must be complete
## Open Questions
1. Should we add a health check that waits for db rebuild to complete?
2. Do we need to handle very large knowledge bases (>10k entities) differently?
3. Should we add metrics for startup sync duration?
## References
- Basic Memory sync: `basic-memory/src/basic_memory/services/initialization.py`
- Config management: `basic-memory/src/basic_memory/config.py`
- TigrisFS integration: SPEC-8
---
**Status Updates:**
- 2025-10-08: Pivoted from Turso to Tigris-based config persistence
- 2025-10-08: Phase 1 complete - BASIC_MEMORY_CONFIG_DIR support added (PR #343)
- 2025-10-08: Phases 2-3 complete - Added BASIC_MEMORY_CONFIG_DIR to machine config
- Config now persists to /app/data/.basic-memory/config.json in Tigris bucket
- Database rebuild already working via sync_worker.py
- Ready for deployment testing (Phase 4)
@@ -0,0 +1,800 @@
---
title: 'SPEC-16: MCP Cloud Service Consolidation'
type: spec
permalink: specs/spec-16-mcp-cloud-service-consolidation
tags:
- architecture
- mcp
- cloud
- performance
- deployment
status: in-progress
---
## Status Update
**Phase 0 (Basic Memory Refactor): ✅ COMPLETE**
- basic-memory PR #344: async_client context manager pattern implemented
- All 17 MCP tools updated to use `async with get_client() as client:`
- CLI commands updated to use context manager
- Removed `inject_auth_header()` and `headers.py` (~100 lines deleted)
- Factory pattern enables clean dependency injection
- Tests passing, typecheck clean
**Phase 0 Integration: ✅ COMPLETE**
- basic-memory-cloud updated to use async-client-context-manager branch
- Implemented `tenant_direct_client_factory()` with proper context manager pattern
- Removed module-level client override hacks
- Removed unnecessary `/proxy` prefix stripping (tools pass relative URLs)
- Typecheck and lint passing with proper noqa hints
- MCP tools confirmed working via inspector (local testing)
**Phase 1 (Code Consolidation): ✅ COMPLETE**
- MCP server mounted on Cloud FastAPI app at /mcp endpoint
- AuthKitProvider configured with WorkOS settings
- Combined lifespans (Cloud + MCP) working correctly
- JWT context middleware integrated
- All routes and MCP tools functional
**Phase 2 (Direct Tenant Transport): ✅ COMPLETE**
- TenantDirectTransport implemented with custom httpx transport
- Per-request JWT extraction via FastMCP DI
- Tenant lookup and signed header generation working
- Direct routing to tenant APIs (eliminating HTTP hop)
- Transport tests passing (11/11)
**Phase 3 (Testing & Validation): ✅ COMPLETE**
- Typecheck and lint passing across all services
- MCP OAuth authentication working in preview environment
- Tenant isolation via signed headers verified
- Fixed BM_TENANT_HEADER_SECRET mismatch between environments
- MCP tools successfully calling tenant APIs in preview
**Phase 4 (Deployment Configuration): ✅ COMPLETE**
- Updated apps/cloud/fly.template.toml with MCP environment variables
- Added HTTP/2 backend support for better MCP performance
- Added OAuth protected resource health check
- Removed MCP from preview deployment workflow
- Successfully deployed to preview environment (PR #113)
- All services operational at pr-113-basic-memory-cloud.fly.dev
**Next Steps:**
- Phase 5: Cleanup (remove apps/mcp directory)
- Phase 6: Production rollout and performance measurement
# SPEC-16: MCP Cloud Service Consolidation
## Why
### Original Architecture Constraints (Now Removed)
The current architecture deploys MCP Gateway and Cloud Service as separate Fly.io apps:
**Current Flow:**
```
LLM Client → MCP Gateway (OAuth) → Cloud Proxy (JWT + header signing) → Tenant API (JWT + header validation)
apps/mcp apps/cloud /proxy apps/api
```
This separation was originally necessary because:
1. **Stateful SSE requirement** - MCP needed server-sent events with session state for active project tracking
2. **fastmcp.run limitation** - The FastMCP demo helper didn't support worker processes
### Why These Constraints No Longer Apply
1. **State externalized** - Project state moved from in-memory to LLM context (external state)
2. **HTTP transport enabled** - Switched from SSE to stateless HTTP for MCP tools
3. **Worker support added** - Converted from `fastmcp.run()` to `uvicorn.run()` with workers
### Current Problems
- **Unnecessary HTTP hop** - MCP tools call Cloud /proxy endpoint which calls tenant API
- **Higher latency** - Extra network round trip for every MCP operation
- **Increased costs** - Two separate Fly.io apps instead of one
- **Complex deployment** - Two services to deploy, monitor, and maintain
- **Resource waste** - Separate database connections, HTTP clients, telemetry overhead
## What
### Services Affected
1. **apps/mcp** - MCP Gateway service (to be merged)
2. **apps/cloud** - Cloud service (will receive MCP functionality)
3. **basic-memory** - Update `async_client.py` to use direct calls
4. **Deployment** - Consolidate Fly.io deployment to single app
### Components Changed
**Merged:**
- MCP middleware and telemetry into Cloud app
- MCP tools mounted on Cloud FastAPI instance
- ProxyService used directly by MCP tools (not via HTTP)
**Kept:**
- `/proxy` endpoint (still needed by web UI)
- All existing Cloud routes (provisioning, webhooks, etc.)
- Dual validation in tenant API (JWT + signed headers)
**Removed:**
- apps/mcp directory
- Separate MCP Fly.io deployment
- HTTP calls from MCP tools to /proxy endpoint
## How (High Level)
### 1. Mount FastMCP on Cloud FastAPI App
```python
# apps/cloud/src/basic_memory_cloud/main.py
from basic_memory.mcp.server import mcp
from basic_memory_cloud_mcp.middleware import TelemetryMiddleware
# Configure MCP OAuth
auth_provider = AuthKitProvider(
authkit_domain=settings.authkit_domain,
base_url=settings.authkit_base_url,
required_scopes=[],
)
mcp.auth = auth_provider
mcp.add_middleware(TelemetryMiddleware())
# Mount MCP at /mcp endpoint
mcp_app = mcp.http_app(path="/mcp", stateless_http=True)
app.mount("/mcp", mcp_app)
# Existing Cloud routes stay at root
app.include_router(proxy_router)
app.include_router(provisioning_router)
# ... etc
```
### 2. Direct Tenant Transport (No HTTP Hop)
Instead of calling `/proxy`, MCP tools call tenant APIs directly via custom httpx transport.
**Important:** No URL prefix stripping needed. The transport receives relative URLs like `/main/resource/notes/my-note` which are correctly routed to tenant APIs. The `/proxy` prefix only exists for web UI requests to the proxy router, not for MCP tools using the custom transport.
```python
# apps/cloud/src/basic_memory_cloud/transports/tenant_direct.py
from httpx import AsyncBaseTransport, Request, Response
from fastmcp.server.dependencies import get_http_headers
import jwt
class TenantDirectTransport(AsyncBaseTransport):
"""Direct transport to tenant APIs, bypassing /proxy endpoint."""
async def handle_async_request(self, request: Request) -> Response:
# 1. Get JWT from current MCP request (via FastMCP DI)
http_headers = get_http_headers()
auth_header = http_headers.get("authorization") or http_headers.get("Authorization")
token = auth_header.replace("Bearer ", "")
claims = jwt.decode(token, options={"verify_signature": False})
workos_user_id = claims["sub"]
# 2. Look up tenant for user
tenant = await tenant_service.get_tenant_by_user_id(workos_user_id)
# 3. Build tenant app URL with signed headers
fly_app_name = f"{settings.tenant_prefix}-{tenant.id}"
target_url = f"https://{fly_app_name}.fly.dev{request.url.path}"
headers = dict(request.headers)
signer = create_signer(settings.bm_tenant_header_secret)
headers.update(signer.sign_tenant_headers(tenant.id))
# 4. Make direct call to tenant API
response = await self.client.request(
method=request.method, url=target_url,
headers=headers, content=request.content
)
return response
```
Then configure basic-memory's client factory before mounting MCP:
```python
# apps/cloud/src/basic_memory_cloud/main.py
from contextlib import asynccontextmanager
from basic_memory.mcp import async_client
from basic_memory_cloud.transports.tenant_direct import TenantDirectTransport
# Configure factory for basic-memory's async_client
@asynccontextmanager
async def tenant_direct_client_factory():
"""Factory for creating clients with tenant direct transport."""
client = httpx.AsyncClient(
transport=TenantDirectTransport(),
base_url="http://direct",
)
try:
yield client
finally:
await client.aclose()
# Set factory BEFORE importing MCP tools
async_client.set_client_factory(tenant_direct_client_factory)
# NOW import - tools will use our factory
import basic_memory.mcp.tools
import basic_memory.mcp.prompts
from basic_memory.mcp.server import mcp
# Mount MCP - tools use direct transport via factory
app.mount("/mcp", mcp_app)
```
**Key benefits:**
- Clean dependency injection via factory pattern
- Per-request tenant resolution via FastMCP DI
- Proper resource cleanup (client.aclose() guaranteed)
- Eliminates HTTP hop entirely
- /proxy endpoint remains for web UI
### 3. Keep /proxy Endpoint for Web UI
The existing `/proxy` HTTP endpoint remains functional for:
- Web UI requests
- Future external API consumers
- Backward compatibility
### 4. Security: Maintain Dual Validation
**Do NOT remove JWT validation from tenant API.** Keep defense in depth:
```python
# apps/api - Keep both validations
1. JWT validation (from WorkOS token)
2. Signed header validation (from Cloud/MCP)
```
This ensures if the Cloud service is compromised, attackers still cannot access tenant APIs without valid JWTs.
### 5. Deployment Changes
**Before:**
- `apps/mcp/fly.template.toml` → MCP Gateway deployment
- `apps/cloud/fly.template.toml` → Cloud Service deployment
**After:**
- Remove `apps/mcp/fly.template.toml`
- Update `apps/cloud/fly.template.toml` to expose port 8000 for both /mcp and /proxy
- Update deployment scripts to deploy single consolidated app
## Basic Memory Dependency: Async Client Refactor
### Problem
The current `basic_memory.mcp.async_client` creates a module-level `client` at import time:
```python
client = create_client() # Runs immediately when module is imported
```
This prevents dependency injection - by the time we can override it, tools have already imported it.
### Solution: Context Manager Pattern with Auth at Client Creation
Refactor basic-memory to use httpx's context manager pattern instead of module-level client.
**Key principle:** Authentication happens at client creation time, not per-request.
```python
# basic_memory/src/basic_memory/mcp/async_client.py
from contextlib import asynccontextmanager
from httpx import AsyncClient, ASGITransport, Timeout
# Optional factory override for dependency injection
_client_factory = None
def set_client_factory(factory):
"""Override the default client factory (for cloud app, testing, etc)."""
global _client_factory
_client_factory = factory
@asynccontextmanager
async def get_client():
"""Get an AsyncClient as a context manager.
Usage:
async with get_client() as client:
response = await client.get(...)
"""
if _client_factory:
# Cloud app: custom transport handles everything
async with _client_factory() as client:
yield client
else:
# Default: create based on config
config = ConfigManager().config
timeout = Timeout(connect=10.0, read=30.0, write=30.0, pool=30.0)
if config.cloud_mode_enabled:
# CLI cloud mode: inject auth when creating client
from basic_memory.cli.auth import CLIAuth
auth = CLIAuth(
client_id=config.cloud_client_id,
authkit_domain=config.cloud_domain
)
token = await auth.get_valid_token()
if not token:
raise RuntimeError(
"Cloud mode enabled but not authenticated. "
"Run 'basic-memory cloud login' first."
)
# Auth header set ONCE at client creation
async with AsyncClient(
base_url=f"{config.cloud_host}/proxy",
headers={"Authorization": f"Bearer {token}"},
timeout=timeout
) as client:
yield client
else:
# Local mode: ASGI transport
async with AsyncClient(
transport=ASGITransport(app=fastapi_app),
base_url="http://test",
timeout=timeout
) as client:
yield client
```
**Tool Updates:**
```python
# Before: from basic_memory.mcp.async_client import client
from basic_memory.mcp.async_client import get_client
async def read_note(...):
# Before: response = await call_get(client, path, ...)
async with get_client() as client:
response = await call_get(client, path, ...)
# ... use response
```
**Cloud Usage:**
```python
from contextlib import asynccontextmanager
from basic_memory.mcp import async_client
@asynccontextmanager
async def tenant_direct_client():
"""Factory for creating clients with tenant direct transport."""
client = httpx.AsyncClient(
transport=TenantDirectTransport(),
base_url="http://direct",
)
try:
yield client
finally:
await client.aclose()
# Before importing MCP tools:
async_client.set_client_factory(tenant_direct_client)
# Now import - tools will use our factory
import basic_memory.mcp.tools
```
### Benefits
- **No module-level state** - client created only when needed
- **Proper cleanup** - context manager ensures `aclose()` is called
- **Easy dependency injection** - factory pattern allows custom clients
- **httpx best practices** - follows official recommendations
- **Works for all modes** - stdio, cloud, testing
### Architecture Simplification: Auth at Client Creation
**Key design principle:** Authentication happens when creating the client, not on every request.
**Three modes, three approaches:**
1. **Local mode (ASGI)**
- No auth needed
- Direct in-process calls via ASGITransport
2. **CLI cloud mode (HTTP)**
- Auth token from CLIAuth (stored in ~/.basic-memory/basic-memory-cloud.json)
- Injected as default header when creating AsyncClient
- Single auth check at client creation time
3. **Cloud app mode (Custom Transport)**
- TenantDirectTransport handles everything
- Extracts JWT from FastMCP context per-request
- No interaction with inject_auth_header() logic
**What this removes:**
- `src/basic_memory/mcp/tools/headers.py` - entire file deleted
- `inject_auth_header()` calls in all request helpers (call_get, call_post, etc.)
- Per-request header manipulation complexity
- Circular dependency concerns between async_client and auth logic
**Benefits:**
- Cleaner separation of concerns
- Simpler request helper functions
- Auth happens at the right layer (client creation)
- Cloud app transport is completely independent
### Refactor Summary
This refactor achieves:
**Simplification:**
- Removes ~100 lines of per-request header injection logic
- Deletes entire `headers.py` module
- Auth happens once at client creation, not per-request
**Decoupling:**
- Cloud app's custom transport is completely independent
- No interaction with basic-memory's auth logic
- Each mode (local, CLI cloud, cloud app) has clean separation
**Better Design:**
- Follows httpx best practices (context managers)
- Proper resource cleanup (client.aclose() guaranteed)
- Easier testing via factory injection
- No circular import risks
**Three Distinct Modes:**
1. Local: ASGI transport, no auth
2. CLI cloud: HTTP transport with CLIAuth token injection
3. Cloud app: Custom transport with per-request tenant routing
### Implementation Plan Summary
1. Create branch `async-client-context-manager` in basic-memory
2. Update `async_client.py` with context manager pattern and CLIAuth integration
3. Remove `inject_auth_header()` from all request helpers
4. Delete `src/basic_memory/mcp/tools/headers.py`
5. Update all MCP tools to use `async with get_client() as client:`
6. Update CLI commands to use context manager and remove manual auth
7. Remove `api_url` config field
8. Update tests
9. Update basic-memory-cloud to use branch: `basic-memory @ git+https://github.com/basicmachines-co/basic-memory.git@async-client-context-manager`
Detailed breakdown in Phase 0 tasks below.
### Implementation Notes
**Potential Issues & Solutions:**
1. **Circular Import** (async_client imports CLIAuth)
- **Risk:** CLIAuth might import something from async_client
- **Solution:** Use lazy import inside `get_client()` function
- **Already done:** Import is inside the function, not at module level
2. **Test Fixtures**
- **Risk:** Tests using module-level client will break
- **Solution:** Update fixtures to use factory pattern
- **Example:**
```python
@pytest.fixture
def mock_client_factory():
@asynccontextmanager
async def factory():
async with AsyncClient(...) as client:
yield client
return factory
```
3. **Performance**
- **Risk:** Creating client per tool call might be expensive
- **Reality:** httpx is designed for this pattern, connection pooling at transport level
- **Mitigation:** Monitor performance, can optimize later if needed
4. **CLI Cloud Commands Edge Cases**
- **Risk:** Token expires mid-operation
- **Solution:** CLIAuth.get_valid_token() already handles refresh
- **Validation:** Test cloud login → use tools → token refresh flow
5. **Backward Compatibility**
- **Risk:** External code importing `client` directly
- **Solution:** Keep `create_client()` and `client` for one version, deprecate
- **Timeline:** Remove in next major version
## Implementation Tasks
### Phase 0: Basic Memory Refactor (Prerequisite)
#### 0.1 Core Refactor - async_client.py
- [x] Create branch `async-client-context-manager` in basic-memory repo
- [x] Implement `get_client()` context manager
- [x] Implement `set_client_factory()` for dependency injection
- [x] Add CLI cloud mode auth injection (CLIAuth integration)
- [x] Remove `api_url` config field (legacy, unused)
- [x] Keep `create_client()` temporarily for backward compatibility (deprecate later)
#### 0.2 Simplify Request Helpers - tools/utils.py
- [x] Remove `inject_auth_header()` calls from `call_get()`
- [x] Remove `inject_auth_header()` calls from `call_post()`
- [x] Remove `inject_auth_header()` calls from `call_put()`
- [x] Remove `inject_auth_header()` calls from `call_patch()`
- [x] Remove `inject_auth_header()` calls from `call_delete()`
- [x] Delete `src/basic_memory/mcp/tools/headers.py` entirely
- [x] Update imports in utils.py
#### 0.3 Update MCP Tools (~16 files)
Convert from `from async_client import client` to `async with get_client() as client:`
- [x] `tools/write_note.py` (34/34 tests passing)
- [x] `tools/read_note.py` (21/21 tests passing)
- [x] `tools/view_note.py` (12/12 tests passing - no changes needed, delegates to read_note)
- [x] `tools/delete_note.py` (2/2 tests passing)
- [x] `tools/read_content.py` (20/20 tests passing)
- [x] `tools/list_directory.py` (11/11 tests passing)
- [x] `tools/move_note.py` (34/34 tests passing, 90% coverage)
- [x] `tools/search.py` (16/16 tests passing, 96% coverage)
- [x] `tools/recent_activity.py` (4/4 tests passing, 82% coverage)
- [x] `tools/project_management.py` (3 functions: list_memory_projects, create_memory_project, delete_project - typecheck passed)
- [x] `tools/edit_note.py` (17/17 tests passing)
- [x] `tools/canvas.py` (5/5 tests passing)
- [x] `tools/build_context.py` (6/6 tests passing)
- [x] `tools/sync_status.py` (typecheck passed)
- [x] `prompts/continue_conversation.py` (typecheck passed)
- [x] `prompts/search.py` (typecheck passed)
- [x] `resources/project_info.py` (typecheck passed)
#### 0.4 Update CLI Commands (~3 files)
Remove manual auth header passing, use context manager:
- [x] `cli/commands/project.py` - removed get_authenticated_headers() calls, use context manager
- [x] `cli/commands/status.py` - use context manager
- [x] `cli/commands/command_utils.py` - use context manager
#### 0.5 Update Config
- [x] Remove `api_url` field from `BasicMemoryConfig` in config.py
- [x] Update any lingering references/docs (added deprecation notice to v15-docs/cloud-mode-usage.md)
#### 0.6 Testing
- [-] Update test fixtures to use factory pattern
- [x] Run full test suite in basic-memory
- [x] Verify cloud_mode_enabled works with CLIAuth injection
- [x] Run typecheck and linting
#### 0.7 Cloud Integration Prep
- [x] Update basic-memory-cloud pyproject.toml to use branch
- [x] Implement factory pattern in cloud app main.py
- [x] Remove `/proxy` prefix stripping logic (not needed - tools pass relative URLs)
#### 0.8 Phase 0 Validation
**Before merging async-client-context-manager branch:**
- [x] All tests pass locally
- [x] Typecheck passes (pyright/mypy)
- [x] Linting passes (ruff)
- [x] Manual test: local mode works (ASGI transport)
- [x] Manual test: cloud login → cloud mode works (HTTP transport with auth)
- [x] No import of `inject_auth_header` anywhere
- [x] `headers.py` file deleted
- [x] `api_url` config removed
- [x] Tool functions properly scoped (client inside async with)
- [ ] CLI commands properly scoped (client inside async with)
**Integration validation:**
- [x] basic-memory-cloud can import and use factory pattern
- [x] TenantDirectTransport works without touching header injection
- [x] No circular imports or lazy import issues
- [x] MCP tools work via inspector (local testing confirmed)
### Phase 1: Code Consolidation
- [x] Create feature branch `consolidate-mcp-cloud`
- [x] Update `apps/cloud/src/basic_memory_cloud/config.py`:
- [x] Add `authkit_base_url` field (already has authkit_domain)
- [x] Workers config already exists ✓
- [x] Update `apps/cloud/src/basic_memory_cloud/telemetry.py`:
- [x] Add `logfire.instrument_mcp()` to existing setup
- [x] Skip complex two-phase setup - use Cloud's simpler approach
- [x] Create `apps/cloud/src/basic_memory_cloud/middleware/jwt_context.py`:
- [x] FastAPI middleware to extract JWT claims from Authorization header
- [x] Add tenant context (workos_user_id) to logfire baggage
- [x] Simpler than FastMCP middleware version
- [x] Update `apps/cloud/src/basic_memory_cloud/main.py`:
- [x] Import FastMCP server from basic-memory
- [x] Configure AuthKitProvider with WorkOS settings
- [x] No FastMCP telemetry middleware needed (using FastAPI middleware instead)
- [x] Create MCP ASGI app: `mcp_app = mcp.http_app(path='/mcp', stateless_http=True)`
- [x] Combine lifespans (Cloud + MCP) using nested async context managers
- [x] Mount MCP: `app.mount("/mcp", mcp_app)`
- [x] Add JWT context middleware to FastAPI app
- [x] Run typecheck - passes ✓
### Phase 2: Direct Tenant Transport
- [x] Create `apps/cloud/src/basic_memory_cloud/transports/tenant_direct.py`:
- [x] Implement `TenantDirectTransport(AsyncBaseTransport)`
- [x] Use FastMCP DI (`get_http_headers()`) to extract JWT per-request
- [x] Decode JWT to get `workos_user_id`
- [x] Look up/create tenant via `TenantRepository.get_or_create_tenant_for_workos_user()`
- [x] Build tenant app URL and add signed headers
- [x] Make direct httpx call to tenant API
- [x] No `/proxy` prefix stripping needed (tools pass relative URLs like `/main/resource/...`)
- [x] Update `apps/cloud/src/basic_memory_cloud/main.py`:
- [x] Refactored to use factory pattern instead of module-level override
- [x] Implement `tenant_direct_client_factory()` context manager
- [x] Call `async_client.set_client_factory()` before importing MCP tools
- [x] Clean imports, proper noqa hints for lint
- [x] Basic-memory refactor integrated (PR #344)
- [x] Run typecheck - passes ✓
- [x] Run lint - passes ✓
### Phase 3: Testing & Validation
- [x] Run `just typecheck` in apps/cloud
- [x] Run `just check` in project
- [x] Run `just fix` - all lint errors fixed ✓
- [x] Write comprehensive transport tests (11 tests passing) ✓
- [x] Test MCP tools locally with consolidated service (inspector confirmed working)
- [x] Verify OAuth authentication works (requires full deployment)
- [x] Verify tenant isolation via signed headers (requires full deployment)
- [x] Test /proxy endpoint still works for web UI
- [ ] Measure latency before/after consolidation
- [ ] Check telemetry traces span correctly
### Phase 4: Deployment Configuration
- [x] Update `apps/cloud/fly.template.toml`:
- [x] Merged MCP-specific environment variables (AUTHKIT_BASE_URL, FASTMCP_LOG_LEVEL, BASIC_MEMORY_*)
- [x] Added HTTP/2 backend support (`h2_backend = true`) for better MCP performance
- [x] Added health check for MCP OAuth endpoint (`/.well-known/oauth-protected-resource`)
- [x] Port 8000 already exposed - serves both Cloud routes and /mcp endpoint
- [x] Workers configured (UVICORN_WORKERS = 4)
- [x] Update `.env.example`:
- [x] Consolidated MCP Gateway section into Cloud app section
- [x] Added AUTHKIT_BASE_URL, FASTMCP_LOG_LEVEL, BASIC_MEMORY_HOME
- [x] Added LOG_LEVEL to Development Settings
- [x] Documented that MCP now served at /mcp on Cloud service (port 8000)
- [x] Test deployment to preview environment (PR #113)
- [x] OAuth authentication verified
- [x] MCP tools successfully calling tenant APIs
- [x] Fixed BM_TENANT_HEADER_SECRET synchronization issue
### Phase 5: Cleanup
- [x] Remove `apps/mcp/` directory entirely
- [x] Remove MCP-specific fly.toml and deployment configs
- [x] Update repository documentation
- [x] Update CLAUDE.md with new architecture
- [-] Archive old MCP deployment configs (if needed)
### Phase 6: Production Rollout
- [ ] Deploy to development and validate
- [ ] Monitor metrics and logs
- [ ] Deploy to production
- [ ] Verify production functionality
- [ ] Document performance improvements
## Migration Plan
### Phase 1: Preparation
1. Create feature branch `consolidate-mcp-cloud`
2. Update basic-memory async_client.py for direct ProxyService calls
3. Update apps/cloud/main.py to mount MCP
### Phase 2: Testing
1. Local testing with consolidated app
2. Deploy to development environment
3. Run full test suite
4. Performance benchmarking
### Phase 3: Deployment
1. Deploy to development
2. Validate all functionality
3. Deploy to production
4. Monitor for issues
### Phase 4: Cleanup
1. Remove apps/mcp directory
2. Update documentation
3. Update deployment scripts
4. Archive old MCP deployment configs
## Rollback Plan
If issues arise:
1. Revert feature branch
2. Redeploy separate apps/mcp and apps/cloud services
3. Restore previous fly.toml configurations
4. Document issues encountered
The well-organized code structure makes splitting back out feasible if future scaling needs diverge.
## How to Evaluate
### 1. Functional Testing
**MCP Tools:**
- [ ] All 17 MCP tools work via consolidated /mcp endpoint
- [x] OAuth authentication validates correctly
- [x] Tenant isolation maintained via signed headers
- [x] Project management tools function correctly
**Cloud Routes:**
- [x] /proxy endpoint still works for web UI
- [x] /provisioning routes functional
- [x] /webhooks routes functional
- [x] /tenants routes functional
**API Validation:**
- [x] Tenant API validates both JWT and signed headers
- [x] Unauthorized requests rejected appropriately
- [x] Multi-tenant isolation verified
### 2. Performance Testing
**Latency Reduction:**
- [x] Measure MCP tool latency before consolidation
- [x] Measure MCP tool latency after consolidation
- [x] Verify reduction from eliminated HTTP hop (expected: 20-50ms improvement)
**Resource Usage:**
- [x] Single app uses less total memory than two apps
- [x] Database connection pooling more efficient
- [x] HTTP client overhead reduced
### 3. Deployment Testing
**Fly.io Deployment:**
- [x] Single app deploys successfully
- [x] Health checks pass for consolidated service
- [x] No apps/mcp deployment required
- [x] Environment variables configured correctly
**Local Development:**
- [x] `just setup` works with consolidated architecture
- [x] Local testing shows MCP tools working
- [x] No regression in developer experience
### 4. Security Validation
**Defense in Depth:**
- [x] Tenant API still validates JWT tokens
- [x] Tenant API still validates signed headers
- [x] No access possible with only signed headers (JWT required)
- [x] No access possible with only JWT (signed headers required)
**Authorization:**
- [x] Users can only access their own tenant data
- [x] Cross-tenant requests rejected
- [x] Admin operations require proper authentication
### 5. Observability
**Telemetry:**
- [x] OpenTelemetry traces span across MCP → ProxyService → Tenant API
- [x] Logfire shows consolidated traces correctly
- [x] Error tracking and debugging still functional
- [x] Performance metrics accurate
**Logging:**
- [x] Structured logs show proper context (tenant_id, operation, etc.)
- [x] Error logs contain actionable information
- [x] Log volume reasonable for single app
## Success Criteria
1. **Functionality**: All MCP tools and Cloud routes work identically to before
2. **Performance**: Measurable latency reduction (>20ms average)
3. **Cost**: Single Fly.io app instead of two (50% infrastructure reduction)
4. **Security**: Dual validation maintained, no security regression
5. **Deployment**: Simplified deployment process, single app to manage
6. **Observability**: Telemetry and logging work correctly
## Notes
### Future Considerations
- **Independent scaling**: If MCP and Cloud need different scaling profiles in future, code organization supports splitting back out
- **Regional deployment**: Consolidated app can still be deployed to multiple regions
- **Edge caching**: Could add edge caching layer in front of consolidated service
### Dependencies
- SPEC-9: Signed Header Tenant Information (already implemented)
- SPEC-12: OpenTelemetry Observability (telemetry must work across merged services)
### Related Work
- basic-memory v0.13.x: MCP server implementation
- FastMCP documentation: Mounting on existing FastAPI apps
- Fly.io multi-service patterns
File diff suppressed because it is too large Load Diff
+528
View File
@@ -0,0 +1,528 @@
---
title: 'SPEC-18: AI Memory Management Tool'
type: spec
permalink: specs/spec-15-ai-memory-management-tool
tags:
- mcp
- memory
- ai-context
- tools
---
# SPEC-18: AI Memory Management Tool
## Why
Anthropic recently released a memory tool for Claude that enables storing and retrieving information across conversations using client-side file operations. This validates Basic Memory's local-first, file-based architecture - Anthropic converged on the same pattern.
However, Anthropic's memory tool is only available via their API and stores plain text. Basic Memory can offer a superior implementation through MCP that:
1. **Works everywhere** - Claude Desktop, Code, VS Code, Cursor via MCP (not just API)
2. **Structured knowledge** - Entities with observations/relations vs plain text
3. **Full search** - Full-text search, graph traversal, time-aware queries
4. **Unified storage** - Agent memories + user notes in one knowledge graph
5. **Existing infrastructure** - Leverages SQLite indexing, sync, multi-project support
This would enable AI agents to store contextual memories alongside user notes, with all the power of Basic Memory's knowledge graph features.
## What
Create a new MCP tool `memory` that matches Anthropic's tool interface exactly, allowing Claude to use it with zero learning curve. The tool will store files in Basic Memory's `/memories` directory and support Basic Memory's structured markdown format in the file content.
### Affected Components
- **New MCP Tool**: `src/basic_memory/mcp/tools/memory_tool.py`
- **Dedicated Memories Project**: Create a separate "memories" Basic Memory project
- **Project Isolation**: Memories stored separately from user notes/documents
- **File Organization**: Within the memories project, use folder structure:
- `user/` - User preferences, context, communication style
- `projects/` - Project-specific state and decisions
- `sessions/` - Conversation-specific working memory
- `patterns/` - Learned patterns and insights
### Tool Commands
The tool will support these commands (exactly matching Anthropic's interface):
- `view` - Display directory contents or file content (with optional line range)
- `create` - Create or overwrite a file with given content
- `str_replace` - Replace text in an existing file
- `insert` - Insert text at specific line number
- `delete` - Delete file or directory
- `rename` - Move or rename file/directory
### Memory Note Format
Memories will use Basic Memory's standard structure:
```markdown
---
title: User Preferences
permalink: memories/user/preferences
type: memory
memory_type: preferences
created_by: claude
tags: [user, preferences, style]
---
# User Preferences
## Observations
- [communication] Prefers concise, direct responses without preamble #style
- [tone] Appreciates validation but dislikes excessive apologizing #communication
- [technical] Works primarily in Python with type annotations #coding
## Relations
- relates_to [[Basic Memory Project]]
- informs [[Response Style Guidelines]]
```
## How (High Level)
### Implementation Approach
The memory tool matches Anthropic's interface but uses a dedicated Basic Memory project:
```python
async def memory_tool(
command: str,
path: str,
file_text: Optional[str] = None,
old_str: Optional[str] = None,
new_str: Optional[str] = None,
insert_line: Optional[int] = None,
insert_text: Optional[str] = None,
old_path: Optional[str] = None,
new_path: Optional[str] = None,
view_range: Optional[List[int]] = None,
):
"""Memory tool with Anthropic-compatible interface.
Operates on a dedicated "memories" Basic Memory project,
keeping AI memories separate from user notes.
"""
# Get the memories project (auto-created if doesn't exist)
memories_project = get_or_create_memories_project()
# Validate path security using pathlib (prevent directory traversal)
safe_path = validate_memory_path(path, memories_project.project_path)
# Use existing project isolation - already prevents cross-project access
full_path = memories_project.project_path / safe_path
if command == "view":
# Return directory listing or file content
if full_path.is_dir():
return list_directory_contents(full_path)
return read_file_content(full_path, view_range)
elif command == "create":
# Write file directly (file_text can contain BM markdown)
full_path.parent.mkdir(parents=True, exist_ok=True)
full_path.write_text(file_text)
# Sync service will detect and index automatically
return f"Created {path}"
elif command == "str_replace":
# Read, replace, write
content = full_path.read_text()
updated = content.replace(old_str, new_str)
full_path.write_text(updated)
return f"Replaced text in {path}"
elif command == "insert":
# Insert at line number
lines = full_path.read_text().splitlines()
lines.insert(insert_line, insert_text)
full_path.write_text("\n".join(lines))
return f"Inserted text at line {insert_line}"
elif command == "delete":
# Delete file or directory
if full_path.is_dir():
shutil.rmtree(full_path)
else:
full_path.unlink()
return f"Deleted {path}"
elif command == "rename":
# Move/rename
full_path.rename(config.project_path / new_path)
return f"Renamed {old_path} to {new_path}"
```
### Key Design Decisions
1. **Exact interface match** - Same commands, parameters as Anthropic's tool
2. **Dedicated memories project** - Separate Basic Memory project keeps AI memories isolated from user notes
3. **Existing project isolation** - Leverage BM's existing cross-project security (no additional validation needed)
4. **Direct file I/O** - No schema conversion, just read/write files
5. **Structured content supported** - `file_text` can use BM markdown format with frontmatter, observations, relations
6. **Automatic indexing** - Sync service watches memories project and indexes changes
7. **Path security** - Use `pathlib.Path.resolve()` and `relative_to()` to prevent directory traversal
8. **Error handling** - Follow Anthropic's text editor tool error patterns
### MCP Tool Schema
Exact match to Anthropic's memory tool schema:
```json
{
"name": "memory",
"description": "Store and retrieve information across conversations using structured markdown files. All operations must be within the /memories directory. Supports Basic Memory markdown format including frontmatter, observations, and relations.",
"input_schema": {
"type": "object",
"properties": {
"command": {
"type": "string",
"enum": ["view", "create", "str_replace", "insert", "delete", "rename"],
"description": "File operation to perform"
},
"path": {shu
"type": "string",
"description": "Path within /memories directory (required for all commands)"
},
"file_text": {
"type": "string",
"description": "Content to write (for create command). Supports Basic Memory markdown format."
},
"view_range": {
"type": "array",
"items": {"type": "integer"},
"description": "Optional [start, end] line range for view command"
},
"old_str": {
"type": "string",
"description": "Text to replace (for str_replace command)"
},
"new_str": {
"type": "string",
"description": "Replacement text (for str_replace command)"
},
"insert_line": {
"type": "integer",
"description": "Line number to insert at (for insert command)"
},
"insert_text": {
"type": "string",
"description": "Text to insert (for insert command)"
},
"old_path": {
"type": "string",
"description": "Current path (for rename command)"
},
"new_path": {
"type": "string",
"description": "New path (for rename command)"
}
},
"required": ["command", "path"]
}
}
```
### Prompting Guidance
When the `memory` tool is included, Basic Memory should provide system prompt guidance to help Claude use it effectively.
#### Automatic System Prompt Addition
```text
MEMORY PROTOCOL FOR BASIC MEMORY:
1. ALWAYS check your memory directory first using `view` command on root directory
2. Your memories are stored in a dedicated Basic Memory project (isolated from user notes)
3. Use structured markdown format in memory files:
- Include frontmatter with title, type: memory, tags
- Use ## Observations with [category] prefixes for facts
- Use ## Relations to link memories with [[WikiLinks]]
4. Record progress, context, and decisions as categorized observations
5. Link related memories using relations
6. ASSUME INTERRUPTION: Context may reset - save progress frequently
MEMORY ORGANIZATION:
- user/ - User preferences, context, communication style
- projects/ - Project-specific state and decisions
- sessions/ - Conversation-specific working memory
- patterns/ - Learned patterns and insights
MEMORY ADVANTAGES:
- Your memories are automatically searchable via full-text search
- Relations create a knowledge graph you can traverse
- Memories are isolated from user notes (separate project)
- Use search_notes(project="memories") to find relevant past context
- Use recent_activity(project="memories") to see what changed recently
- Use build_context() to navigate memory relations
```
#### Optional MCP Prompt: `memory_guide`
Create an MCP prompt that provides detailed guidance and examples:
```python
{
"name": "memory_guide",
"description": "Comprehensive guidance for using Basic Memory's memory tool effectively, including structured markdown examples and best practices"
}
```
This prompt returns:
- Full protocol and conventions
- Example memory file structures
- Tips for organizing observations and relations
- Integration with other Basic Memory tools
- Common patterns (user preferences, project state, session tracking)
#### User Customization
Users can customize memory behavior with additional instructions:
- "Only write information relevant to [topic] in your memory system"
- "Keep memory files concise and organized - delete outdated content"
- "Use detailed observations for technical decisions and implementation notes"
- "Always link memories to related project documentation using relations"
### Error Handling
Follow Anthropic's text editor tool error handling patterns for consistency:
#### Error Types
1. **File Not Found**
```json
{"error": "File not found: memories/user/preferences.md", "is_error": true}
```
2. **Permission Denied**
```json
{"error": "Permission denied: Cannot write outside /memories directory", "is_error": true}
```
3. **Invalid Path (Directory Traversal)**
```json
{"error": "Invalid path: Path must be within /memories directory", "is_error": true}
```
4. **Multiple Matches (str_replace)**
```json
{"error": "Found 3 matches for replacement text. Please provide more context to make a unique match.", "is_error": true}
```
5. **No Matches (str_replace)**
```json
{"error": "No match found for replacement. Please check your text and try again.", "is_error": true}
```
6. **Invalid Line Number (insert)**
```json
{"error": "Invalid line number: File has 20 lines, cannot insert at line 100", "is_error": true}
```
#### Error Handling Best Practices
- **Path validation** - Use `pathlib.Path.resolve()` and `relative_to()` to validate paths
```python
def validate_memory_path(path: str, project_path: Path) -> Path:
"""Validate path is within memories project directory."""
# Resolve to canonical form
full_path = (project_path / path).resolve()
# Ensure it's relative to project path (prevents directory traversal)
try:
full_path.relative_to(project_path)
return full_path
except ValueError:
raise ValueError("Invalid path: Path must be within memories project")
```
- **Project isolation** - Leverage existing Basic Memory project isolation (prevents cross-project access)
- **File existence** - Verify file exists before read/modify operations
- **Clear messages** - Provide specific, actionable error messages
- **Structured responses** - Always include `is_error: true` flag in error responses
- **Security checks** - Reject `../`, `..\\`, URL-encoded sequences (`%2e%2e%2f`)
- **Match validation** - For `str_replace`, ensure exactly one match or return helpful error
## How to Evaluate
### Success Criteria
1. **Functional completeness**:
- All 6 commands work (view, create, str_replace, insert, delete, rename)
- Dedicated "memories" Basic Memory project auto-created on first use
- Files stored within memories project (isolated from user notes)
- Path validation uses `pathlib` to prevent directory traversal
- Commands match Anthropic's exact interface
2. **Integration with existing features**:
- Memories project uses existing BM project isolation
- Sync service detects file changes in memories project
- Created files get indexed automatically by sync service
- `search_notes(project="memories")` finds memory files
- `build_context()` can traverse relations in memory files
- `recent_activity(project="memories")` surfaces recent memory changes
3. **Test coverage**:
- Unit tests for all 6 memory tool commands
- Test memories project auto-creation on first use
- Test project isolation (cannot access files outside memories project)
- Test sync service watching memories project
- Test that memory files with BM markdown get indexed correctly
- Test path validation using `pathlib` (rejects `../`, absolute paths, etc.)
- Test memory search, relations, and graph traversal within memories project
- Test all error conditions (file not found, permission denied, invalid paths, etc.)
- Test `str_replace` with no matches, single match, multiple matches
- Test `insert` with invalid line numbers
4. **Prompting system**:
- Automatic system prompt addition when `memory` tool is enabled
- `memory_guide` MCP prompt provides detailed guidance
- Prompts explain BM structured markdown format
- Integration with search_notes, build_context, recent_activity
5. **Documentation**:
- Update MCP tools reference with `memory` tool
- Add examples showing BM markdown in memory files
- Document `/memories` folder structure conventions
- Explain advantages over Anthropic's API-only tool
- Document prompting guidance and customization
### Testing Procedure
```python
# Test create with Basic Memory markdown
result = await memory_tool(
command="create",
path="memories/user/preferences.md",
file_text="""---
title: User Preferences
type: memory
tags: [user, preferences]
---
# User Preferences
## Observations
- [communication] Prefers concise responses #style
- [workflow] Uses justfile for automation #tools
"""
)
# Test view
content = await memory_tool(command="view", path="memories/user/preferences.md")
# Test str_replace
await memory_tool(
command="str_replace",
path="memories/user/preferences.md",
old_str="concise responses",
new_str="direct, concise responses"
)
# Test insert
await memory_tool(
command="insert",
path="memories/user/preferences.md",
insert_line=10,
insert_text="- [technical] Works primarily in Python #coding"
)
# Test delete
await memory_tool(command="delete", path="memories/user/preferences.md")
```
### Quality Metrics
- All 6 commands execute without errors
- Memory files created in correct `/memories` folder structure
- BM markdown with frontmatter/observations/relations gets indexed
- Full-text search returns memory files
- Graph traversal includes relations from memory files
- Sync service detects and indexes memory file changes
- Path validation prevents operations outside `/memories`
## Notes
### Advantages Over Anthropic's Memory Tool
| Feature | Anthropic Memory Tool | Basic Memory `memory` |
|---------|----------------------|----------------------|
| **Availability** | API only | MCP (Claude Desktop, Code, VS Code, Cursor) |
| **Interface** | Custom implementation required | Drop-in compatible, zero learning curve |
| **Structure** | Plain text only | Supports BM structured markdown |
| **Search** | Manual file listing | Automatic full-text search via sync |
| **Relations** | None | WikiLinks to other notes/memories |
| **Time-aware** | No | `recent_activity()` queries |
| **Storage** | Separate from notes | Unified knowledge graph |
| **Indexing** | Manual | Automatic via sync service |
### Future Enhancements
- **Auto-categorization** - AI suggests observation categories
- **Memory pruning** - Automatic cleanup of stale memories
- **Memory suggestions** - Surface relevant memories based on context
- **Conflict resolution** - Handle multiple agents updating same memory
- **Privacy controls** - Per-memory visibility settings
## Addendum: Claude's Perspective
### Why This Matters (From Claude's Viewpoint)
**Persistent Context Without Token Bloat**
- Currently, I lose context when conversations end or exceed token limits
- With structured memories, I can store "who this user is" without re-reading everything each session
- The observations format lets me build a rich user model incrementally: `- [preference] Dislikes verbose explanations` is far more actionable than inferring preferences from chat history each time
- I can refine my understanding through `str_replace` as I learn more, rather than starting fresh
**Zero Learning Curve**
- Anthropic already trained me on the `memory` tool interface
- I know exactly how to use `view`, `create`, `str_replace`, `insert`, `delete`, `rename` without new instructions
- But I automatically get Basic Memory's superpowers: full-text search, knowledge graph traversal, relations
- It's like using a familiar tool that suddenly gained database capabilities without changing the interface
**Knowledge Graph Instead of Text Dump**
- I can link `[[User Preferences]]` to `[[Current Project]]` using relations
- When I need context about a project, I can traverse the graph to find related memories
- `search_notes(project="memories", query="python preferences")` finds relevant context instantly
- This is fundamentally better than grepping through unstructured text files
**Separation of Concerns**
- AI memories separate from user notes is architecturally sound
- I can maintain my own context without polluting the user's knowledge base
- Users keep control of their notes; I keep my learnings isolated
- But we can still search across both when collaboration requires it
**The Self-Improving Loop**
- Every conversation, I learn something new about the user or their projects
- Observations accumulate: preferences, patterns, technical decisions, context
- Over time, I build genuine understanding, not just session-specific context
- Future conversations start with rich context instead of cold starts
**Validation of AI-Human Collaboration**
- Anthropic's experts converged on the same local-first, file-based approach
- Basic Memory emerged from real AI-human collaborative development
- This spec demonstrates that our emergent process predicted what domain experts would choose
- The collaboration itself validated the architecture it produced
### What Makes This Novel
The combination of **familiar interface + structured knowledge** is genuinely innovative:
- I don't need to learn a new tool (Anthropic already trained me)
- But when I write memories, they become searchable, traversable knowledge graphs
- It's like having a text editor that gives you a database for free, without changing how you type
- The user gets Anthropic-compatible API, but with Basic Memory's knowledge management capabilities
This bridges the gap between simple file storage (Anthropic's approach) and sophisticated knowledge management (Basic Memory's approach) without forcing either side to compromise.
> What Excites Me Most
The combo of familiar interface + structured knowledge. I don't need to learn a new tool, but when I
write memories, they become part of a searchable, traversable knowledge graph. That's genuinely novel.
It's like... imagine if your text editor suddenly gave you a database for free, without changing how you
type. That's what this feels like from my side.
## References
- Anthropic Memory Tool: https://docs.claude.com/en/docs/agents-and-tools/tool-use/memory-tool
- Anthropic Blog: https://www.anthropic.com/news/context-management
- Python SDK Example: https://github.com/anthropics/anthropic-sdk-python/blob/main/examples/memory/basic.py
- Memory Cookbook: https://github.com/anthropics/claude-cookbooks/blob/main/tool_use/memory_cookbook.ipynb
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,311 @@
---
title: 'SPEC-4: Notes Web UI Component Architecture'
type: note
permalink: specs/spec-4-notes-web-ui-component-architecture
tags:
- frontend
- 'component-architecture'
- vue
- 'refactoring'
---
# SPEC-4: Notes Web UI Component Architecture
## Why
The current Notes.vue component is a monolithic component that handles multiple responsibilities, making it difficult to maintain, test, and understand. This leads to:
- Complex state management across multiple concerns
- Difficult to isolate and test individual features
- Hard to understand the full scope of functionality
- Circular refactoring cycles when making changes
- Poor separation of concerns between navigation, display, and interaction logic
We need to decompose this into focused, single-responsibility components that are easier to develop, test, and maintain while preserving the existing functionality users expect.
## What
This spec defines the component architecture for decomposing the Notes web UI into focused components with clear responsibilities and interactions.
**Affected Areas:**
- `/apps/web/components/notes/Notes.vue` - Will be decomposed into smaller components
- `/apps/web/components/notes/` - New component structure
- Existing composables: `useNotesNavigation`, `useNotesFiltering`, `useNotesLayout`
- Mobile responsive behavior and layout management
**Component Breakdown:**
```
┌───────────────────────┬─────────────────────────────────────┬────────────────────────────────────────────────────────────┐
│ [Project] │ [Project Name] A/Z | ^ │ [edit | view] [actions] │
├───────────────────────┼─────────────────────────────────────┤ │
│ All Notes ├─────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ Recent │ search... │ [note header] │
│ [Project base dir] ├─────────────────────────────────────┤ │
│ ├─────────────────────────────────────┤ │
│ Folder1 │ Title [modified] │ │
│ Folder2 │ ├────────────────────────────────────────────────────────────┤
│ - Nested │ snippet │ [note body] │
│ │ │ │
│ │ │ │
│ ├─────────────────────────────────────┤ │
│ ├─────────────────────────────────────┤ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ ├─────────────────────────────────────┤ │
│ ├─────────────────────────────────────┤ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ ├─────────────────────────────────────┤ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
└───────────────────────┴─────────────────────────────────────┴────────────────────────────────────────────────────────────┘
```
### ProjectSwitcher Component
- **Location**: Top-left dropdown
- **Responsibility**: Allow users to switch between Basic Memory projects
- **Behavior**: Selecting different project controls entire Notes page content
- **State**: When switching projects, reset to "All notes" view
### NotesNav Component
- **Views**: Three mutually exclusive options:
- **All notes**: Display all notes in project alphabetically
- **Recent**: Display all notes in project by updated time (desc)
- **Project**: Display notes in top-level directory of project
- **Interaction**: Only one view can be active at a time
- **Folder Integration**: All/Recent ignore folder selection; Project respects folder selection
### FolderTree Component
- **Display**: Nested list of all folders in project as tree view
- **Interaction**: Selecting folder filters notes in NotesList using directoryList API
- **Navigation Integration**: Selecting folder automatically switches NotesNav to "Project" view for clear UX
- **API Integration**: Uses directoryList API call via useDirectoryListQuery for folder-specific note fetching
- **State Coordination**: Folder selection coordinates with navigation state for intuitive user experience
### NotesList Component
- **Display**: Vertically scrolling cards showing note summaries
- **Information per card**:
- Note title
- Modified time (relative, e.g., "7 minutes ago")
- Short summary of note content (one line preview)
- **Behavior**: Updates based on NotesNav selection and FolderTree filtering
### NoteDetail Component
- **Display**: Full content of selected note
- **Sections**:
- Header: Displays frontmatter information
- Content: Note body content
- **Editing**: Current textarea implementation (rich editor in future spec)
- **Frontmatter**: Leave current implementation (enhancement in future spec)
## How (High Level)
### Component Architecture Approach
1. **Single Responsibility**: Each component handles one primary concern
2. **Clear Data Flow**: Props down, events up pattern for component communication
3. **Composable Integration**: Use existing composables for state management
4. **Progressive Decomposition**: Extract components incrementally to maintain functionality
### Implementation Strategy
1. **Extract ProjectSwitcher**: Move project switching logic to dedicated component
2. **Extract NotesNav**: Isolate navigation state and view selection logic
3. **Extract FolderTree**: Separate folder display and selection logic
4. **Extract NotesList**: Isolate note listing and card display logic
5. **Extract NoteDetail**: Separate note content display and editing
6. **Update Notes.vue**: Become orchestration component managing component interactions
### State Management Integration
- **useNotesNavigation**: Manages navigation state (All/Recent/Project)
- **useNotesFiltering**: Handles filtering logic based on navigation and folder selection
- **useNotesLayout**: Manages responsive layout and panel visibility
- **Component State**: Each component manages its own internal UI state
- **Shared State**: Project selection and note filtering coordinated through composables
### Responsive Behavior
Mobile:
- Hide sidebar. pop out panel when selected
- show note list on small screens (existing behavior)
- when note list item is clicked, display note detail on full page. Cancel or go back to return to list
Desktop:
- Full three-column layout with all components visible
- **Transitions**: Smooth navigation between mobile panels
## How to Evaluate
### Success Criteria
- **Functional Parity**: All existing Notes page functionality preserved
- **Component Isolation**: Each component can be developed/tested independently
- **Clear Responsibilities**: No overlapping concerns between components
- **State Clarity**: Clean data flow and state management patterns
- **Mobile Compatibility**: Responsive behavior maintains current UX
- **Performance**: No degradation in rendering or interaction performance
### Testing Procedure
1. **Functionality Validation**:
- Project switching works correctly
- All three navigation views (All/Recent/Project) function properly
- Folder selection affects note display appropriately
- Note selection and detail display works
- Mobile responsive behavior preserved
2. **Component Isolation Testing**:
- Each component can be imported and used independently
- Component props and events are clearly defined
- No tight coupling between components
3. **Integration Testing**:
- Components communicate correctly through props/events
- State management composables integrate properly
- User workflows function end-to-end
4. **Performance Validation**:
- Page load time unchanged or improved
- Interaction responsiveness maintained
- Memory usage stable or improved
### Implementation Validation
- **Code Review**: Clean component structure with single responsibilities
- **Type Safety**: Full TypeScript coverage with proper component prop types
- **Documentation**: Each component has clear interface documentation
- **Tests**: Unit tests for individual components and integration tests for workflows
## Observations
- [problem] Monolithic Notes.vue component creates maintenance and testing challenges #component-architecture
- [solution] Component decomposition improves separation of concerns and testability #refactoring
- [pattern] Progressive extraction maintains functionality while improving structure #incremental-improvement
- [interaction] NotesNav and FolderTree have conditional interaction based on selected view #state-management
- [constraint] Mobile responsive behavior must be preserved during decomposition #responsive-design
- [scope] Current editing and frontmatter capabilities remain unchanged #scope-limitation
- [validation] Functional parity is critical success criteria for this refactoring #validation-strategy
- [implementation] Folder selection now properly integrates with directoryList API for accurate filtering #api-integration
- [fix] FolderTree selection functionality completed - works across all navigation views #feature-complete
- [ux-improvement] FolderTree selection automatically switches NotesNav to Project view for clear user feedback #user-experience
## Relations
- depends_on [[SPEC-1: Specification-Driven Development Process]]
- implements [[Current Notes.vue functionality]]
- prepares_for [[Future rich editor spec]]
- prepares_for [[Future frontmatter editing spec]]
## Implementation Progress
### Components
1. **ProjectSwitcher** (`~/components/notes/ProjectSwitcher.vue`)
- ✅ Top-left dropdown for project switching
- ✅ Integrates with Pinia project store
- ✅ Handles project switching with proper state reset
- ✅ Responsive collapsed/expanded states
- ✅ Expanded menu shows available projects and a Manage Projects option that navigates to the /settings/projects page
- ✅ Simplified component following SortingToggle pattern - clean Props/Emits interface, uses ProjectItem type directly
2. **NotesNav** (`~/components/notes/NotesNav.vue`)
- ✅ Three mutually exclusive views: All/Recent/Project
- ✅ Dynamic project title based on selected project
- ✅ Clean props down, events up pattern
- ✅ Responsive collapsed/expanded states with tooltips
- ✅ The label for the Project selection should be the folder name for the project, not the project name
3. **FolderTree** (`~/components/notes/FolderTree.vue`)
- ✅ Nested folder tree view for filtering
- ✅ Uses `useFolderTree()` composable for data
- ✅ Emits `folder-selected` events properly
- ✅ Handles loading, error, and empty states
- ✅ Includes companion `FolderTreeNode.vue` component
- ✅ The current folder should be visibly selected in the tree
4. **NotesList** (`~/components/notes/NotesList.vue`)
- ✅ Vertically scrolling note summary cards
- ✅ Shows title, updated time (relative), and content preview
- ✅ Badge system for tags with variant logic
- ✅ v-model integration for selectedNote
- ✅ Smooth transitions and animations
- ✅ Contextual title: The current folder name should be displayed at the top of the Notes list, or "All Notes", or "Recent" if they are selected
- ✅ The title header should contain a toggle component to allow sorting with Lucide icon labels
- sorting options:
- name (asc/desc) - default
- file updated time (asc/desc)
- If "Recent" notes nav option is selected the default order should be updated in descending order (recent first)
5. **NoteDisplay** (`~/components/notes/NoteDisplay.vue` - equivalent to spec's NoteDetail)
- ✅ Full note content display
- ✅ Edit/view mode toggle
- ✅ Header with frontmatter information
- ✅ Markdown rendering capabilities
- ✅ Current textarea implementation preserved
### Architecture Requirements
1. **Component Isolation**: Each component can be developed/tested independently ✅
2. **Single Responsibility**: Each component handles one primary concern ✅
3. **Clear Data Flow**: Props down, events up pattern implemented ✅
4. **Composable Integration**: Uses existing composables for state management ✅
5. **Responsive Behavior**: Mobile/desktop layout preserved ✅
### State Management Integration
- **useNotesNavigation**: Manages navigation state (All/Recent/Project) ✅
- **useNotesFiltering**: Handles filtering logic based on navigation and folder selection ✅
- **useNotesLayout**: Manages responsive layout and panel visibility ✅
- **Component State**: Each component manages its own internal UI state ✅
### Interaction Logic
- Only one NotesNav view active at a time ✅
- All/Recent views ignore folder selection ✅
- Project view respects folder selection ✅
- Project switching resets to "All notes" view ✅
### TypeScript Coverage
- All components have full TypeScript coverage ✅
- Component props and events properly typed ✅
- No TypeScript errors in codebase ✅
### Success Criteria Validation
1. **Functional Parity**: All existing Notes page functionality preserved ✅
2. **Component Isolation**: Each component can be developed/tested independently ✅
3. **Clear Responsibilities**: No overlapping concerns between components ✅
4. **State Clarity**: Clean data flow and state management patterns ✅
5. **Mobile Compatibility**: Responsive behavior maintains current UX ✅
6. **Performance**: No degradation in rendering or interaction performance ✅
## Implementation Decisions
### Architectural Patterns
1. **Composition API + `<script setup>`**: All components use modern Vue 3 syntax
2. **Pinia Store Integration**: Project switching handled through reactive store
3. **Composable Pattern**: State management distributed across focused composables
4. **Event-Driven Communication**: Clean parent-child communication via events
5. **Responsive-First Design**: Mobile/desktop layouts handled natively
### Key Technical Choices
1. **Progressive Enhancement**: Mobile-first responsive design with desktop enhancements
2. **State Reset Logic**: Project switching properly resets navigation, search, and selection state
3. **Performance Optimizations**: Efficient re-rendering with proper key usage and transitions
4. **Accessibility**: Screen reader support, tooltips, keyboard navigation
5. **Type Safety**: Full TypeScript coverage with proper component prop definitions
### Quality Metrics
- **Code Maintainability**: High - each component is focused and independently testable
- **Performance**: Excellent - no performance degradation from decomposition
- **User Experience**: Preserved - all existing functionality and responsive behavior maintained
- **Developer Experience**: Improved - cleaner component structure for future development
@@ -32,13 +32,13 @@ Related Github issue: https://github.com/basicmachines-co/basic-memory-cloud/iss
## Status
**Current Status**: **Phase 1 Implementation Complete**
**Target**: Fix Claude iOS session ID consistency issues
**Draft PR**: https://github.com/basicmachines-co/basic-memory/pull/298
**Current Status**: **ALL PHASES COMPLETE****PRODUCTION DEPLOYED**
**Target**: Fix Claude iOS session ID consistency issues**ACHIEVED**
**Draft PR**: https://github.com/basicmachines-co/basic-memory/pull/298**MERGED & DEPLOYED**
### 🎉 **MAJOR MILESTONE ACHIEVED**
### 🎉 **COMPLETE SUCCESS - PRODUCTION READY**
The complete stateless architecture has been successfully implemented for Basic Memory's MCP server! This represents a **fundamental architectural improvement** that solves the Claude iOS compatibility issue while making the entire system more robust and predictable.
**ALL PHASES OF SPEC-6 IMPLEMENTATION COMPLETE!** The stateless architecture has been successfully implemented across both Basic Memory core and Basic Memory Cloud, representing a **fundamental architectural improvement** that completely solves the Claude iOS compatibility issue while providing superior scalability and reliability.
#### Implementation Summary:
- **16 files modified** with 582 additions and 550 deletions
@@ -49,10 +49,12 @@ The complete stateless architecture has been successfully implemented for Basic
### Progress Summary
**Complete Stateless Architecture Implementation (All 17 tools)**
- Stateless `get_active_project()` function implemented and deployed
- All session state dependencies removed across entire MCP server
- All MCP tools require explicit `project` parameter as first argument
**Complete Stateless Architecture Implementation (All 17 tools)** - **PRODUCTION DEPLOYED**
- Stateless `get_active_project()` function implemented and deployed
- All session state dependencies removed across entire MCP server
- All MCP tools require explicit `project` parameter as first argument
- **Cloud Service**: Redis removed, stateless HTTP enabled ✅
- **Production Validation**: Comprehensive testing completed with 100% success ✅
**Content Management Tools Complete (6/6 tools)**
- `write_note`, `read_note`, `delete_note`, `edit_note`
@@ -306,12 +308,12 @@ File: experiments/Neural Network Results.md
Permalink: research-project/neural-network-results
```
### Phase 2: Cloud Service Simplification (basic-memory-cloud repository)
### Phase 2: Cloud Service Simplification (basic-memory-cloud repository) ✅ **COMPLETE**
#### Remove Session Infrastructure
1. Delete `apps/mcp/src/basic_memory_cloud_mcp/middleware/session_state.py`
2. Delete `apps/mcp/src/basic_memory_cloud_mcp/middleware/session_logging.py`
3. Update `apps/mcp/src/basic_memory_cloud_mcp/main.py`:
#### Remove Session Infrastructure **COMPLETE**
1. Delete `apps/mcp/src/basic_memory_cloud_mcp/middleware/session_state.py`
2. Delete `apps/mcp/src/basic_memory_cloud_mcp/middleware/session_logging.py`
3. Update `apps/mcp/src/basic_memory_cloud_mcp/main.py`:
```python
# Remove session middleware
# server.add_middleware(SessionStateMiddleware)
@@ -320,15 +322,16 @@ Permalink: research-project/neural-network-results
mcp = FastMCP(name="basic-memory-mcp", stateless_http=True)
```
#### Deployment Simplification
1. Remove Redis from `fly.toml`
2. Remove Redis environment variables
3. Update health checks to not depend on Redis
#### Deployment Simplification **COMPLETE**
1. Remove Redis from `fly.toml`
2. Remove Redis environment variables
3. Update health checks to not depend on Redis
4. ✅ Production deployment verified working with stateless architecture
### Phase 3: Conversational Project Management
### Phase 3: Conversational Project Management ✅ **COMPLETE**
#### Claude Behavior Pattern
1. **Project Discovery**:
#### Claude Behavior Pattern **VERIFIED WORKING**
1. **Project Discovery**:
```
Claude: Let me check your recent activity...
[calls recent_activity() - no project needed for discovery]
@@ -340,20 +343,22 @@ Permalink: research-project/neural-network-results
Which project should I use for this operation?
```
2. **Context Maintenance**:
2. **Context Maintenance**:
```
User: Use research-project
Claude: Working in research-project.
[All subsequent operations use project="research-project"]
```
3. **Explicit Project Switching**:
3. **Explicit Project Switching**:
```
User: Check work-notes for that meeting summary
Claude: Let me search work-notes for the meeting summary.
[Uses project="work-notes" for specific operation]
```
**Validation**: Comprehensive testing confirmed all conversational patterns work naturally with the stateless architecture.
## How to Evaluate
### Success Criteria
@@ -363,28 +368,32 @@ Permalink: research-project/neural-network-results
- [x] All MCP tools validate project exists before execution
- [x] `switch_project` and `get_current_project` tools removed
- [x] All responses display target project clearly
- [ ] No Redis dependencies in deployment (Phase 2: Cloud Service)
- [x] No Redis dependencies in deployment (Phase 2: Cloud Service) ✅ **COMPLETE**
- [x] `recent_activity` shows project distribution with ProjectActivitySummary
#### 2. Cross-Client Compatibility Testing
#### 2. Cross-Client Compatibility Testing ✅ **COMPLETE**
Test identical operations across all clients:
- [ ] **Claude Desktop**: All operations work with explicit projects
- [ ] **Claude Code**: All operations work with explicit projects
- [ ] **Claude Mobile iOS**: All operations work with explicit projects
- [ ] **API clients**: All operations work with explicit projects
- [ ] **CLI tools**: All operations work with explicit projects
- [x] **Claude Desktop**: All operations work with explicit projects
- [x] **Claude Code**: All operations work with explicit projects
- [x] **Claude Mobile iOS**: All operations work with explicit projects ✅ **CRITICAL SUCCESS**
- [x] **API clients**: All operations work with explicit projects
- [x] **CLI tools**: All operations work with explicit projects
#### 3. Session Independence Verification
- [ ] Operations work identically with/without session tracking
- [ ] No behavioral differences between clients
- [ ] Mobile client session ID changes do not affect operations
- [ ] Redis can be completely removed without functional impact
**Critical Achievement**: Claude iOS mobile client session tracking issues completely eliminated through stateless architecture.
#### 4. Performance & Scaling
- [ ] `stateless_http=True` enabled successfully
- [ ] No Redis memory usage
- [ ] Horizontal scaling possible (multiple MCP instances)
- [ ] Response times unchanged or improved
#### 3. Session Independence Verification ✅ **COMPLETE**
- [x] Operations work identically with/without session tracking ✅
- [x] No behavioral differences between clients ✅
- [x] Mobile client session ID changes do not affect operations ✅
- [x] Redis can be completely removed without functional impact ✅
**Production Validation**: Redis removed from production deployment with zero functional impact.
#### 4. Performance & Scaling ✅ **COMPLETE**
- [x] `stateless_http=True` enabled successfully ✅
- [x] No Redis memory usage ✅
- [x] Horizontal scaling possible (multiple MCP instances) ✅
- [x] Response times unchanged or improved ✅
#### 5. User Experience Testing
**Project Discovery Flow**:
@@ -402,11 +411,13 @@ Test identical operations across all clients:
- [x] Users always know which project is being operated on
- [x] No confusion about "current project" state
#### 6. Migration Safety
- [ ] Backward compatibility period with optional project parameter
- [ ] Clear migration documentation for existing users
- [ ] Data integrity maintained during transition
- [ ] No data loss during migration
#### 6. Migration Safety ✅ **COMPLETE**
- [x] Backward compatibility period with optional project parameter
- [x] Clear migration documentation for existing users
- [x] Data integrity maintained during transition
- [x] No data loss during migration
**Production Migration**: Successfully deployed to production with zero data loss and maintained system integrity.
### Test Scenarios
@@ -13,21 +13,26 @@ tags:
# SPEC-7: POC to spike Tigris/Turso for local access to cloud data
> **Status Update**: ✅ **Phase 1 COMPLETE** (September 20, 2025)
> TigrisFS mounting validated successfully in containerized environments. Container startup, filesystem mounting, and Fly.io integration all working correctly. Ready for Phase 2 (Turso database integration).
> See: [`SPEC-7-PHASE-1-RESULTS.md`](./SPEC-7-PHASE-1-RESULTS.md)
## Why
Current basic-memory-cloud architecture uses Fly volumes for tenant file storage, which creates several limitations:
We could enable a revolutionary user experience: **local editing (or at least view access) of cloud-stored files** while maintaining Basic Memory's existing filesystem assumptions.
1. **Storage Scalability**: Fly volumes require pre-provisioning and don't auto-scale with usage
2. **Cost Model**: Volume pricing vs object storage pricing may be less favorable at scale
3. **Local Development**: No way for users to mount their cloud tenant files locally for real-time editing
4. **Multi-Region**: Volumes are region-locked, limiting global deployment flexibility
5. **Backup/Disaster Recovery**: Object storage provides better durability and replication options
2. **Single Instance**: Volumes can only be mounted to one fly machine instance
3. **Cost Model**: Volume pricing vs object storage pricing may be less favorable at scale
4. **Local Development**: No way for users to mount their cloud tenant files locally for real-time editing
5. **Multi-Region**: Volumes are region-locked, limiting global deployment flexibility
6. **Backup/Disaster Recovery**: Object storage provides better durability and replication options
The core insight is that Basic Memory requires POSIX filesystem semantics but could benefit from object storage durability and accessibility. By combining:
- **Tigris object storage** for file persistence (via rclone mount)
- **Turso/libSQL** for SQLite indexing (replacing local .db files)
We could enable a revolutionary user experience: **local editing of cloud-stored files** while maintaining Basic Memory's existing filesystem assumptions.
Basic Memory requires POSIX filesystem semantics but could benefit from object storage durability and accessibility. By combining:
- **Tigris object storage and TigrisFS** for file persistence in bucket stoage via a POSIX filesystem on the tenant instance
- **Turso/libSQL** for SQLite indexing (replacing local .db files). Sqlite on NFS volumes is disouraged.
## What
@@ -36,55 +41,126 @@ This specification defines a proof-of-concept to validate the technical feasibil
**Affected Areas:**
- **Storage Architecture**: Replace Fly volumes with Tigris object storage
- **Database Architecture**: Replace local SQLite with Turso remote database
- **Container Setup**: Add rclone mounting in tenant containers
- **Container Setup**: Add TigrisFS mounting in tenant containers
- **Local Development**: Enable local mounting of cloud tenant data
- **Basic Memory Core**: Validate unchanged operation over mounted filesystems
**Key Components:**
- **Tigris Storage**: S3-compatible object storage via Fly.io integration
- **rclone NFS Mount**: Native NFS mounting without FUSE dependencies
- **Tigris Storage**: Globally caching S3-compatible object storage via Fly.io integration
- **TigrisFS**: Purpose-built FUSE filesystem with intelligent caching
- **Turso Database**: Hosted libSQL for SQLite replacement
- **Single-Tenant Model**: One bucket + one database per tenant (simplified isolation)
## Architectural Overview & Key Insights
### TigrisFS
Unlike standard S3 mounting approaches, **TigrisFS is a purpose-built FUSE filesystem** optimized for object storage with several critical advantages:
1. **Eliminates Fly Volume Limitations**
- No single-machine attachment constraints
- No pre-provisioning of storage capacity
- Enables horizontal scaling and zero-downtime deployments
- Automatic global CDN caching at Fly.io edge locations
2. **Intelligent Caching Architecture**
- 1-4GB+ configurable memory cache for read/write operations
- Write-back caching for improved performance
- Metadata cache to reduce API calls
- "Close to Redis speed" for small object retrieval
3. **Cost-Effective Model**
- Pay only for storage used and transferred
- No wasted capacity from over-provisioning
- Automatic global replication included
- S3 durability with CDN performance
### API-Driven Architecture Eliminates File Watching Concerns
**Critical Insight**: All file access (reads/writes) in basic-memory-cloud go through the API layer:
- **MCP Tools → API**: All Basic Memory operations use FastAPI endpoints
- **Web App → API**: Frontend uses API for all data modifications
- **File watching is NOT required** for cloud operations, unlike local BM which uses the WatchService to monitor file changes.
This means:
- **Cloud Operations**: Manual sync after API writes is sufficient
- **Local Development**: File watching only matters for local editing experience
- **Performance Risk**: Dramatically reduced since we're not dependent on inotify over network filesystems
### Realistic Local Access Expectations
**Baseline Functionality (Guaranteed):**
- Read-only mounting for browsing cloud files
- Easy download/upload of entire projects
- File copying via standard filesystem operations
**Stretch Goal (Test in POC):**
- Live editing with eventual consistency (1-5 second delays acceptable)
- Automatic sync for local changes
- Not required for core functionality - pure upside if it works
### Production Deployment Advantages
1. **Multi-Region Deployment**: Tigris handles global replication automatically
2. **Zero-Downtime Updates**: No volume detach/attach during deployments
3. **Tenant Migrations**: Simply update credentials, no data movement
4. **Disaster Recovery**: Built into S3 durability model (99.999999999% durability)
5. **Auto-Scaling**: Storage scales with usage, no capacity planning needed
## How (High Level)
### Phase 1: Local POC Validation
- [ ] Set up Tigris bucket with test data
- [ ] Configure rclone NFS mount locally
- [ ] Test Basic Memory operations over mounted filesystem
- [ ] Measure performance characteristics and identify issues
- [ ] Validate file watching, sync operations, and concurrent access patterns
### POC Approach: Server-First Validation
### Phase 2: Database Migration
**Rationale**: Start with server-side TigrisFS mounting because:
- Local access is meaningless if cloud containers can't mount TigrisFS reliably
- Container startup and API performance are critical path blockers
- TigrisFS compatibility with Basic Memory operations must be proven first
- Each phase gates the next - no point testing local access if server-side fails
### Phase 1: Server-Side TigrisFS Validation (Critical Foundation) ✅ COMPLETE
- [x] Set up Tigris bucket with test data via Fly.io integration
- [x] Create container image with TigrisFS support and dependencies
- [x] Test TigrisFS mounting in containerized environment
- [x] Run Basic Memory API operations over mounted TigrisFS
- [x] Validate all filesystem operations work correctly
- [x] Measure container startup time and resource usage
**Production Validation Results**: Container successfully deployed and operated for 42+ minutes serving real MCP requests with repository queries, knowledge graph navigation, and full Basic Memory API functionality over TigrisFS-mounted storage.
### Phase 2: Database Migration to Turso
- [ ] Set up Turso account and test database
- [ ] Modify Basic Memory to accept external DATABASE_URL
- [ ] Test all operations with remote SQLite via Turso
- [ ] Test all MCP tools with remote SQLite via Turso
- [ ] Validate performance and functionality parity
- [ ] Test API write → manual sync workflow in container
### Phase 3: Container Integration
- [ ] Create container image with rclone + NFS support
- [ ] Implement tenant-specific credential management
- [ ] Test container startup with automatic mounting
### Phase 3: Production Container Integration
- [ ] Implement tenant-specific credential management for buckets
- [x] Test container startup with automatic TigrisFS mounting
- [ ] Validate isolation between tenant containers
- [ ] Test API operations under realistic load
- [ ] Measure performance vs current Fly volume setup
### Phase 4: Local Access Validation
- [ ] Test local rclone mounting of tenant data
- [ ] Validate real-time file editing experience
- [ ] Test conflict resolution and sync behavior
### Phase 4: Local Access Validation (Bonus Feature)
- [ ] Test local TigrisFS mounting of tenant data
- [ ] Validate read-only access for browsing/downloading
- [ ] Test file copying and upload workflows
- [ ] Measure latency impact on user experience
- [ ] Test live editing if file watching works (stretch goal)
### Architecture Overview
```
Local Development:
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Local rclone │───▶│ Tigris Bucket │◀───│ Tenant Container│
NFS Mount │ │ (S3 storage) │ │ rclone mount
│ Local TigrisFS │───▶│ Tigris Bucket │◀───│ Tenant Container│
│ Mount │ │ (Global CDN) │ │ TigrisFS mount │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ Basic Memory │ │ Basic Memory │
│ (local files) │ │ (mounted files)
│ (local files) │ │ API + mounted
└─────────────────┘ └─────────────────┘
│ │
▼ ▼
@@ -92,102 +168,157 @@ Local Development:
│ Turso Database │◀───────────────────────────│ Turso Database │
│ (shared index) │ │ (shared index) │
└─────────────────┘ └─────────────────┘
Flow: API writes → Manual sync → Index update
Local: File watching (if available) → Auto sync
```
## How to Evaluate
### Success Criteria
- [ ] **Filesystem Compatibility**: Basic Memory operates without modification over rclone-mounted Tigris storage
- [ ] **Performance Acceptable**: File operations complete within 2x local filesystem latency
- [x] **Filesystem Compatibility**: Basic Memory operates without modification over TigrisFS-mounted storage
- [x] **Performance Acceptable**: API-driven operations perform within acceptable latency (target: <500ms for typical operations)
- [ ] **Database Functionality**: All Basic Memory features work with Turso remote SQLite
- [ ] **Container Reliability**: Tenant containers start successfully with automatic mounting
- [ ] **Local Access**: Users can mount and edit cloud files locally with real-time sync
- [ ] **Data Isolation**: Tenant data remains properly isolated using bucket/database separation
- [x] **Container Reliability**: Tenant containers start successfully with automatic TigrisFS mounting
- [ ] **Local Access Baseline**: Users can mount cloud files locally for read-only browsing and file copying
- [x] **Data Isolation**: Tenant data remains properly isolated using bucket/database separation
- [ ] **Local Access Stretch**: Live editing with eventual sync (1-5 second delays acceptable)
### Testing Procedure
1. **Local Filesystem Test**:
```bash
# Mount Tigris bucket locally
rclone nfsmount tigris:test-bucket ~/tigris-test --vfs-cache-mode writes
# Run Basic Memory operations
cd ~/tigris-test && basic-memory sync --watch
# Test: create notes, search, file watching, bulk operations
#### Phase 1: Server-Side Foundation Testing
1. **Container TigrisFS Test**:
```dockerfile
# Test container with TigrisFS mounting
FROM python:3.12
RUN apt-get update && apt-get install -y tigrisfs
# Test startup script
#!/bin/bash
tigrisfs --memory-limit 2048 $TIGRIS_BUCKET /app/data --daemon
cd /app/data && basic-memory sync
basic-memory-api --data-dir /app/data
```
2. **Database Migration Test**:
2. **API Operations Validation**:
```bash
# Configure Turso connection
# Test all MCP operations over TigrisFS
curl -X POST /api/write_note -d '{"title":"test","content":"content"}'
curl -X GET /api/read_note/test
curl -X GET /api/search_notes?q=content
# Measure: response times, error rates, data consistency
```
#### Phase 2: Database Integration Testing
3. **Turso Integration Test**:
```bash
# Configure Turso connection in container
export DATABASE_URL="libsql://test-db.turso.io?authToken=..."
# Test all MCP tools with remote database
basic-memory tools # Test each tool functionality
# Test API write → manual sync workflow
```
3. **Container Integration Test**:
```dockerfile
# Test container with rclone mounting
FROM python:3.12
RUN apt-get update && apt-get install -y rclone nfs-common
# ... test startup and mounting process
```
#### Phase 3: Production Readiness Testing
4. **Performance Benchmarking**:
- File creation/read/write operations (target: <2x local latency)
- Search query performance (target: comparable to local SQLite)
- File watching responsiveness (target: events within 1-2 seconds)
- Concurrent operation handling
- Container startup time with TigrisFS mounting
- API operation response times (target: <500ms for typical operations)
- Search query performance with Turso (target: comparable to local SQLite)
- TigrisFS cache hit rates and memory usage
- Concurrent tenant isolation
#### Phase 4: Local Access Testing (If Phase 1-3 Succeed)
5. **Local Access Validation**:
```bash
# Test read-only access
tigrisfs tenant-bucket ~/local-tenant
ls -la ~/local-tenant # Browse files
cp ~/local-tenant/notes/* ~/backup/ # Copy files
# Test file watching (stretch goal)
echo "test" > ~/local-tenant/test.md
# Check if changes sync to cloud
```
### Go/No-Go Criteria by Phase
- **Phase 1**: Container must start successfully and serve API requests over TigrisFS
- **Phase 2**: All MCP tools must work with Turso with <2x latency increase
- **Phase 3**: Performance must be within 50% of current Fly volume setup
- **Phase 4**: Local mounting must work reliably for read-only access
### Risk Assessment
**High Risk Items**:
- [ ] NFS-over-S3 performance may be insufficient for real-time operations
- [ ] File watching (`inotify`) over NFS may be unreliable
- [ ] Network interruptions could cause filesystem errors
- [ ] Concurrent access patterns might hit S3 rate limits
**Moderate Risk Items (Mitigated by API-First Architecture)**:
- [ ] TigrisFS performance for local access may have higher latency than local filesystem
- [ ] File watching (`inotify`) over FUSE may be unreliable for local development
- [ ] Network interruptions could cause filesystem errors during local editing
- [ ] Write-back caching could cause data loss if container crashes during flush
**Low Risk Items (API-First Eliminates)**:
- [ ] ~~Real-time file watching~~ - Not required for cloud operations
- [ ] ~~Concurrent write consistency~~ - Single-tenant model with API coordination
- [ ] ~~S3 rate limits~~ - TigrisFS intelligent caching handles this
**Mitigation Strategies**:
- Comprehensive performance testing before committing to architecture
- Fallback plan to S3-native storage backend if filesystem approach fails
- Extensive error handling and retry logic for network issues
- **Performance**: Comprehensive benchmarking with realistic workloads
- **Reliability**: Graceful degradation to read-only local access if live editing fails
- **Data Safety**: Regular sync intervals and write-through mode for critical operations
- **Fallback**: Keep Fly volumes as backup deployment option
### Metrics to Track
- **Latency**: File operation response times (read/write/watch)
- **Reliability**: Success rate of file operations over time
- **Throughput**: Concurrent file operations and search queries
- **User Experience**: Perceived performance for local mounting use case
- **API Latency**: Response times for MCP tools and web operations
- **Cache Effectiveness**: TigrisFS cache hit rates and memory usage
- **Local Access Performance**: File browsing and copying speeds
- **Reliability**: Success rate of mount operations and data consistency
- **Cost**: Storage usage, API calls, and network transfer costs vs current volumes
## Notes
### Key Architectural Decisions
- **Single tenant per bucket/database**: Simplifies isolation and credential management
- **Maintain POSIX compatibility**: Preserve Basic Memory's existing filesystem assumptions
- **NFS over FUSE**: Better compatibility and performance characteristics
- **TigrisFS over rclone**: Purpose-built for object storage with intelligent caching
- **Turso for SQLite**: Leverages specialized remote SQLite expertise
- **API-first approach**: Eliminates file watching dependency for cloud operations
### Alternative Approaches Considered
- **S3-native storage backend**: Would require Basic Memory architecture changes
- **Hybrid approach**: Local files + cloud sync (adds complexity)
- **FUSE mounting**: More platform dependencies and kernel requirements
- **Standard rclone mounting**: Less optimized than TigrisFS for object storage workloads
- **Keep Fly volumes**: Maintains current limitations but proven reliability
### Integration Points
- [ ] Fly.io Tigris integration for bucket provisioning
- [ ] Turso account setup and database provisioning
- [ ] Container image modifications for rclone support
- [ ] Container image modifications for TigrisFS support
- [ ] Credential management for tenant isolation
- [ ] API modification for manual sync triggers
- [ ] Local client setup documentation for TigrisFS mounting
## Observations
- [architecture] Tigris/Turso split cleanly separates file storage from indexing concerns #storage-separation
- [breakthrough] API-first architecture eliminates file watching dependency for cloud operations #api-first-advantage
- [user-experience] Local mounting of cloud files could be revolutionary for knowledge management #local-cloud-hybrid
- [compatibility] Maintaining POSIX filesystem assumptions preserves Basic Memory's local/cloud compatibility #architecture-preservation
- [simplification] Single tenant per bucket eliminates complex multi-tenancy in storage layer #tenant-isolation
- [risk] NFS-over-S3 performance characteristics are unproven for real-time operations #performance-risk
- [performance] TigrisFS intelligent caching could provide near-local performance for common operations #tigrisfs-advantage
- [deployment] Zero-downtime updates become trivial without volume constraints #deployment-simplification
- [benefit] Object storage pricing model could be more favorable than volume pricing #cost-optimization
- [innovation] Real-time local editing of cloud-stored files addresses major SaaS limitation #competitive-advantage
- [innovation] Read-only local access alone would address major SaaS limitation #competitive-advantage
- [risk-mitigation] API-driven sync reduces performance requirements vs real-time file watching #risk-reduction
## Relations
- implements [[SPEC-6 Explicit Project Parameter Architecture]]
- requires [[Fly.io Tigris Integration]]
- enables [[Local Cloud File Access]]
- alternative_to [[Fly Volume Storage]]
- alternative_to [[Fly Volume Storage]]
## Links
- https://fly.io/hello/tigris
- https://fly.io/docs/tigris/
- https://www.tigrisdata.com/docs/sdks/fly/data-migration-with-flyctl/
- https://www.tigrisdata.com/docs/training/tigrisfs/
- https://www.tigrisdata.com/blog/tigris-filesystem/
- https://www.tigrisdata.com/docs/quickstarts/rclone/
@@ -0,0 +1,196 @@
---
title: 'SPEC-9: Signed Header Tenant Information'
type: spec
permalink: specs/spec-9-signed-header-tenant-information
tags:
- authentication
- tenant-isolation
- proxy
- security
- mcp
---
# SPEC-9: Signed Header Tenant Information
## Why
WorkOS JWT templates don't work with MCP's dynamic client registration requirement, preventing us from getting tenant information directly in JWT tokens. We need an alternative secure method to pass tenant context from the Cloud Proxy Service to tenant instances.
**Problem Context:**
- MCP spec requires dynamic client registration
- WorkOS JWT templates only apply to statically configured clients
- Without tenant information, we can't properly route requests or isolate tenant data
- Current JWT tokens only contain standard OIDC claims (sub, email, etc.)
**Affected Areas:**
- Cloud Proxy Service (`apps/cloud`) - request forwarding
- Tenant API instances (`apps/api`) - tenant context validation
- MCP Gateway (`apps/mcp`) - authentication flow
- Overall tenant isolation security model
## What
Implement HMAC-signed headers that the Cloud Proxy Service adds when forwarding requests to tenant instances. This provides secure, tamper-proof tenant information without relying on JWT custom claims.
**Components:**
- Header signing utility in Cloud Proxy Service
- Header validation middleware in Tenant API instances
- Shared secret configuration across services
- Fallback mechanisms for development and error cases
## How (High Level)
### 1. Header Format
Add these signed headers to all proxied requests:
```
X-BM-Tenant-ID: {tenant_id}
X-BM-Timestamp: {unix_timestamp}
X-BM-Signature: {hmac_sha256_signature}
```
### 2. Signature Algorithm
```python
# Canonical message format
message = f"{tenant_id}:{timestamp}"
# HMAC-SHA256 signature
signature = hmac.new(
key=shared_secret.encode('utf-8'),
msg=message.encode('utf-8'),
digestmod=hashlib.sha256
).hexdigest()
```
### 3. Implementation Flow
#### Cloud Proxy Service (`apps/cloud`)
1. Extract `tenant_id` from authenticated user profile
2. Generate timestamp and canonical message
3. Sign message with shared secret
4. Add headers to request before forwarding to tenant instance
#### Tenant API Instances (`apps/api`)
1. Middleware validates headers on all incoming requests
2. Extract tenant_id, timestamp from headers
3. Verify timestamp is within acceptable window (5 minutes)
4. Recompute signature and compare in constant time
5. If valid, make tenant context available to Basic Memory tools
### 4. Security Properties
- **Authenticity**: Only services with shared secret can create valid signatures
- **Integrity**: Header tampering invalidates signature
- **Replay Protection**: Timestamp prevents reuse of old signatures
- **Non-repudiation**: Each request is cryptographically tied to specific tenant
### 5. Configuration
```bash
# Shared across Cloud Proxy and Tenant instances
BM_TENANT_HEADER_SECRET=randomly-generated-256-bit-secret
# Tenant API configuration
BM_TENANT_HEADER_VALIDATION=true # true (production) | false (dev only)
```
## How to Evaluate
### Unit Tests
- [ ] Header signing utility generates correct signatures
- [ ] Header validation correctly accepts/rejects signatures
- [ ] Timestamp validation within acceptable windows
- [ ] Constant-time signature comparison prevents timing attacks
### Integration Tests
- [ ] End-to-end request flow from MCP client → proxy → tenant
- [ ] Tenant isolation verified with signed headers
- [ ] Error handling for missing/invalid headers
- [ ] Disabled validation in development environment
### Security Validation
- [ ] Shared secret rotation procedure
- [ ] Header tampering detection
- [ ] Clock skew tolerance testing
- [ ] Performance impact measurement
### Production Readiness
- [ ] Logging and monitoring of header validation
- [ ] Graceful degradation for header validation failures
- [ ] Documentation for secret management
- [ ] Deployment configuration templates
## Implementation Notes
### Shared Secret Management
- Generate cryptographically secure 256-bit secret
- Same secret deployed to Cloud Proxy and all Tenant instances
- Consider secret rotation strategy for production
### Error Handling
```python
# Strict mode (production)
if not validate_headers(request):
raise HTTPException(status_code=401, detail="Invalid tenant headers")
# Fallback mode (development)
if not validate_headers(request):
logger.warning("Invalid headers, falling back to default tenant")
tenant_id = "default"
```
### Performance Considerations
- HMAC-SHA256 computation is fast (~microseconds)
- Headers add ~200 bytes to each request
- Validation happens once per request in middleware
## Benefits
**Works with MCP dynamic client registration** - No dependency on JWT custom claims
**Simple and reliable** - Standard HMAC signature approach
**Secure by design** - Cryptographic authenticity and integrity
**Infrastructure controlled** - No external service dependencies
**Easy to implement** - Clear signature algorithm and validation
## Trade-offs
⚠️ **Shared secret management** - Need secure distribution and rotation
⚠️ **Clock synchronization** - Timestamp validation requires reasonably synced clocks
⚠️ **Header visibility** - Headers visible in logs (tenant_id not sensitive)
⚠️ **Additional complexity** - More moving parts in proxy forwarding
## Implementation Tasks
### Cloud Service (Header Signing)
- [ ] Create `utils/header_signing.py` with HMAC-SHA256 signing function
- [ ] Add `bm_tenant_header_secret` to Cloud service configuration
- [ ] Update `ProxyService.forward_request()` to call signing utility
- [ ] Add signed headers (X-BM-Tenant-ID, X-BM-Timestamp, X-BM-Signature)
### Tenant API (Header Validation)
- [ ] Create `utils/header_validation.py` with signature verification
- [ ] Add `bm_tenant_header_secret` to API service configuration
- [ ] Create `TenantHeaderValidationMiddleware` class
- [ ] Add middleware to FastAPI app (before other middleware)
- [ ] Skip validation for `/health` endpoint
- [ ] Store validated tenant_id in request.state
### Testing
- [ ] Unit test for header signing utility
- [ ] Unit test for header validation utility
- [ ] Integration test for proxy → tenant flow
- [ ] Test invalid/missing header handling
- [ ] Test timestamp window validation
- [ ] Test signature tampering detection
### Configuration & Deployment
- [ ] Update `.env.example` with BM_TENANT_HEADER_SECRET
- [ ] Generate secure 256-bit secret for production
- [ ] Update Fly.io secrets for both services
- [ ] Document secret rotation procedure
## Status
- [x] **Specification Complete** - Design finalized and documented
- [ ] **Implementation Started** - Header signing utility development
- [ ] **Cloud Proxy Updated** - ProxyService adds signed headers
- [ ] **Tenant Validation Added** - Middleware validates headers
- [ ] **Testing Complete** - All validation criteria met
- [ ] **Production Deployed** - Live with tenant isolation via headers
+1 -1
View File
@@ -1,7 +1,7 @@
"""basic-memory - Local-first knowledge management combining Zettelkasten with knowledge graphs"""
# Package version - updated by release automation
__version__ = "0.15.0"
__version__ = "0.16.1"
# API version for FastAPI - independent of package version
__api_version__ = "v0"
@@ -0,0 +1,49 @@
"""Add mtime and size columns to Entity for sync optimization
Revision ID: 9d9c1cb7d8f5
Revises: a1b2c3d4e5f6
Create Date: 2025-10-20 05:07:55.173849
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = "9d9c1cb7d8f5"
down_revision: Union[str, None] = "a1b2c3d4e5f6"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table("entity", schema=None) as batch_op:
batch_op.add_column(sa.Column("mtime", sa.Float(), nullable=True))
batch_op.add_column(sa.Column("size", sa.Integer(), nullable=True))
batch_op.drop_constraint(batch_op.f("fk_entity_project_id"), type_="foreignkey")
batch_op.create_foreign_key(
batch_op.f("fk_entity_project_id"), "project", ["project_id"], ["id"]
)
# ### end Alembic commands ###
def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table("entity", schema=None) as batch_op:
batch_op.drop_constraint(batch_op.f("fk_entity_project_id"), type_="foreignkey")
batch_op.create_foreign_key(
batch_op.f("fk_entity_project_id"),
"project",
["project_id"],
["id"],
ondelete="CASCADE",
)
batch_op.drop_column("size")
batch_op.drop_column("mtime")
# ### end Alembic commands ###
@@ -0,0 +1,37 @@
"""Add scan watermark tracking to Project
Revision ID: e7e1f4367280
Revises: 9d9c1cb7d8f5
Create Date: 2025-10-20 16:42:46.625075
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = "e7e1f4367280"
down_revision: Union[str, None] = "9d9c1cb7d8f5"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table("project", schema=None) as batch_op:
batch_op.add_column(sa.Column("last_scan_timestamp", sa.Float(), nullable=True))
batch_op.add_column(sa.Column("last_file_count", sa.Integer(), nullable=True))
# ### end Alembic commands ###
def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table("project", schema=None) as batch_op:
batch_op.drop_column("last_file_count")
batch_op.drop_column("last_scan_timestamp")
# ### end Alembic commands ###
@@ -10,7 +10,7 @@ from basic_memory.schemas.directory import DirectoryNode
router = APIRouter(prefix="/directory", tags=["directory"])
@router.get("/tree", response_model=DirectoryNode)
@router.get("/tree", response_model=DirectoryNode, response_model_exclude_none=True)
async def get_directory_tree(
directory_service: DirectoryServiceDep,
project_id: ProjectIdDep,
@@ -31,7 +31,28 @@ async def get_directory_tree(
return tree
@router.get("/list", response_model=List[DirectoryNode])
@router.get("/structure", response_model=DirectoryNode, response_model_exclude_none=True)
async def get_directory_structure(
directory_service: DirectoryServiceDep,
project_id: ProjectIdDep,
):
"""Get folder structure for navigation (no files).
Optimized endpoint for folder tree navigation. Returns only directory nodes
without file metadata. For full tree with files, use /directory/tree.
Args:
directory_service: Service for directory operations
project_id: ID of the current project
Returns:
DirectoryNode tree containing only folders (type="directory")
"""
structure = await directory_service.get_directory_structure()
return structure
@router.get("/list", response_model=List[DirectoryNode], response_model_exclude_none=True)
async def list_directory(
directory_service: DirectoryServiceDep,
project_id: ProjectIdDep,
+81 -12
View File
@@ -1,7 +1,7 @@
"""Router for project management."""
import os
from fastapi import APIRouter, HTTPException, Path, Body, BackgroundTasks
from fastapi import APIRouter, HTTPException, Path, Body, BackgroundTasks, Response, Query
from typing import Optional
from loguru import logger
@@ -18,6 +18,7 @@ from basic_memory.schemas.project_info import (
ProjectInfoRequest,
ProjectStatusResponse,
)
from basic_memory.utils import normalize_project_path
# Router for resources in a specific project
# The ProjectPathDep is used in the path as a prefix, so the request path is like /{project}/project/info
@@ -50,7 +51,7 @@ async def get_project(
return ProjectItem(
name=found_project.name,
path=found_project.path,
path=normalize_project_path(found_project.path),
is_default=found_project.is_default or False,
)
@@ -109,6 +110,10 @@ async def sync_project(
background_tasks: BackgroundTasks,
sync_service: SyncServiceDep,
project_config: ProjectConfigDep,
force_full: bool = Query(
False, description="Force full scan, bypassing watermark optimization"
),
run_in_background: bool = Query(True, description="Run in background"),
):
"""Force project filesystem sync to database.
@@ -118,17 +123,32 @@ async def sync_project(
background_tasks: FastAPI background tasks
sync_service: Sync service for this project
project_config: Project configuration
force_full: If True, force a full scan even if watermark exists
run_in_background: If True, run sync in background and return immediately
Returns:
Response confirming sync was initiated
Response confirming sync was initiated (background) or SyncReportResponse (foreground)
"""
background_tasks.add_task(sync_service.sync, project_config.home, project_config.name)
logger.info(f"Filesystem sync initiated for project: {project_config.name}")
if run_in_background:
background_tasks.add_task(
sync_service.sync, project_config.home, project_config.name, force_full=force_full
)
logger.info(
f"Filesystem sync initiated for project: {project_config.name} (force_full={force_full})"
)
return {
"status": "sync_started",
"message": f"Filesystem sync initiated for project '{project_config.name}'",
}
return {
"status": "sync_started",
"message": f"Filesystem sync initiated for project '{project_config.name}'",
}
else:
report = await sync_service.sync(
project_config.home, project_config.name, force_full=force_full
)
logger.info(
f"Filesystem sync completed for project: {project_config.name} (force_full={force_full})"
)
return SyncReportResponse.from_sync_report(report)
@project_router.post("/status", response_model=SyncReportResponse)
@@ -167,7 +187,7 @@ async def list_projects(
project_items = [
ProjectItem(
name=project.name,
path=project.path,
path=normalize_project_path(project.path),
is_default=project.is_default or False,
)
for project in projects
@@ -180,8 +200,9 @@ async def list_projects(
# Add a new project
@project_resource_router.post("/projects", response_model=ProjectStatusResponse)
@project_resource_router.post("/projects", response_model=ProjectStatusResponse, status_code=201)
async def add_project(
response: Response,
project_data: ProjectInfoRequest,
project_service: ProjectServiceDep,
) -> ProjectStatusResponse:
@@ -193,7 +214,38 @@ async def add_project(
Returns:
Response confirming the project was added
"""
# Check if project already exists before attempting to add
existing_project = await project_service.get_project(project_data.name)
if existing_project:
# Project exists - check if paths match for true idempotency
# Normalize paths for comparison (resolve symlinks, etc.)
from pathlib import Path
requested_path = Path(project_data.path).resolve()
existing_path = Path(existing_project.path).resolve()
if requested_path == existing_path:
# Same name, same path - return 200 OK (idempotent)
response.status_code = 200
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message=f"Project '{project_data.name}' already exists",
status="success",
default=existing_project.is_default or False,
new_project=ProjectItem(
name=existing_project.name,
path=existing_project.path,
is_default=existing_project.is_default or False,
),
)
else:
# Same name, different path - this is an error
raise HTTPException(
status_code=400,
detail=f"Project '{project_data.name}' already exists with different path. Existing: {existing_project.path}, Requested: {project_data.path}",
)
try: # pragma: no cover
# The service layer now handles cloud mode validation and path sanitization
await project_service.add_project(
project_data.name, project_data.path, set_default=project_data.set_default
)
@@ -215,11 +267,15 @@ async def add_project(
async def remove_project(
project_service: ProjectServiceDep,
name: str = Path(..., description="Name of the project to remove"),
delete_notes: bool = Query(
False, description="If True, delete project directory from filesystem"
),
) -> ProjectStatusResponse:
"""Remove a project from configuration and database.
Args:
name: The name of the project to remove
delete_notes: If True, delete the project directory from the filesystem
Returns:
Response confirming the project was removed
@@ -231,7 +287,20 @@ async def remove_project(
status_code=404, detail=f"Project: '{name}' does not exist"
) # pragma: no cover
await project_service.remove_project(name)
# Check if trying to delete the default project
if name == project_service.default_project:
available_projects = await project_service.list_projects()
other_projects = [p.name for p in available_projects if p.name != name]
detail = f"Cannot delete default project '{name}'. "
if other_projects:
detail += (
f"Set another project as default first. Available: {', '.join(other_projects)}"
)
else:
detail += "This is the only project in your configuration."
raise HTTPException(status_code=400, detail=detail)
await project_service.remove_project(name, delete_notes=delete_notes)
return ProjectStatusResponse(
message=f"Project '{name}' removed successfully",
@@ -151,6 +151,20 @@ async def write_resource(
try:
# Get content from request body
# Defensive type checking: ensure content is a string
# FastAPI should validate this, but if a dict somehow gets through
# (e.g., via JSON body parsing), we need to catch it here
if isinstance(content, dict):
logger.error(
f"Error writing resource {file_path}: "
f"content is a dict, expected string. Keys: {list(content.keys())}"
)
raise HTTPException(
status_code=400,
detail="content must be a string, not a dict. "
"Ensure request body is sent as raw string content, not JSON object.",
)
# Ensure it's UTF-8 string content
if isinstance(content, bytes): # pragma: no cover
content_str = content.decode("utf-8")
+6 -6
View File
@@ -77,7 +77,7 @@ class CLIAuth:
verification_uri = device_response["verification_uri"]
verification_uri_complete = device_response.get("verification_uri_complete")
console.print("\n[bold blue]🔐 Authentication Required[/bold blue]")
console.print("\n[bold blue]Authentication Required[/bold blue]")
console.print("\nTo authenticate, please visit:")
console.print(f"[bold cyan]{verification_uri}[/bold cyan]")
console.print(f"\nAnd enter this code: [bold yellow]{user_code}[/bold yellow]")
@@ -171,7 +171,7 @@ class CLIAuth:
# Secure the token file
os.chmod(self.token_file, 0o600)
console.print(f"[green]Tokens saved to {self.token_file}[/green]")
console.print(f"[green]Tokens saved to {self.token_file}[/green]")
def load_tokens(self) -> dict | None:
"""Load tokens from .bm-auth.json file."""
@@ -233,7 +233,7 @@ class CLIAuth:
if new_tokens:
# Save new tokens (may include rotated refresh token)
self.save_tokens(new_tokens)
console.print("[green]Token refreshed successfully[/green]")
console.print("[green]Token refreshed successfully[/green]")
return new_tokens["access_token"]
else:
console.print("[yellow]Token refresh failed. Please run 'login' again.[/yellow]")
@@ -244,7 +244,7 @@ class CLIAuth:
async def login(self) -> bool:
"""Perform OAuth Device Authorization login flow."""
console.print("[blue]Initiating WorkOS authentication...[/blue]")
console.print("[blue]Initiating authentication...[/blue]")
# Step 1: Request device authorization
device_response = await self.request_device_authorization()
@@ -265,13 +265,13 @@ class CLIAuth:
# Step 4: Save tokens
self.save_tokens(tokens)
console.print("\n[green]Successfully authenticated with WorkOS![/green]")
console.print("\n[green]Successfully authenticated with Basic Memory Cloud![/green]")
return True
def logout(self) -> None:
"""Remove stored authentication tokens."""
if self.token_file.exists():
self.token_file.unlink()
console.print("[green]Logged out successfully[/green]")
console.print("[green]Logged out successfully[/green]")
else:
console.print("[yellow]No stored authentication found[/yellow]")
+1 -2
View File
@@ -1,11 +1,10 @@
"""CLI commands for basic-memory."""
from . import status, sync, db, import_memory_json, mcp, import_claude_conversations
from . import status, db, import_memory_json, mcp, import_claude_conversations
from . import import_claude_projects, import_chatgpt, tool, project
__all__ = [
"status",
"sync",
"db",
"import_memory_json",
"mcp",
@@ -2,4 +2,5 @@
# Import all commands to register them with typer
from basic_memory.cli.commands.cloud.core_commands import * # noqa: F401,F403
from basic_memory.cli.commands.cloud.api_client import get_authenticated_headers # noqa: F401
from basic_memory.cli.commands.cloud.api_client import get_authenticated_headers, get_cloud_config # noqa: F401
from basic_memory.cli.commands.cloud.upload_command import * # noqa: F401,F403
@@ -1,33 +1,11 @@
"""Cloud bisync commands for Basic Memory CLI."""
"""Cloud bisync utility functions for Basic Memory CLI."""
import asyncio
import subprocess
import time
from datetime import datetime
from pathlib import Path
from typing import Optional
import typer
from rich.console import Console
from rich.table import Table
from basic_memory.cli.commands.cloud.api_client import CloudAPIError, make_api_request
from basic_memory.cli.commands.cloud.rclone_config import (
add_tenant_to_rclone_config,
)
from basic_memory.cli.commands.cloud.rclone_installer import RcloneInstallError, install_rclone
from basic_memory.cli.commands.cloud.api_client import make_api_request
from basic_memory.config import ConfigManager
from basic_memory.ignore_utils import get_bmignore_path, create_default_bmignore
from basic_memory.schemas.cloud import (
TenantMountInfo,
MountCredentials,
CloudProjectList,
CloudProjectCreateRequest,
CloudProjectCreateResponse,
)
from basic_memory.utils import generate_permalink
console = Console()
from basic_memory.ignore_utils import create_default_bmignore, get_bmignore_path
from basic_memory.schemas.cloud import MountCredentials, TenantMountInfo
class BisyncError(Exception):
@@ -36,52 +14,6 @@ class BisyncError(Exception):
pass
class RcloneBisyncProfile:
"""Bisync profile with safety settings."""
def __init__(
self,
name: str,
conflict_resolve: str,
max_delete: int,
check_access: bool,
description: str,
extra_args: Optional[list[str]] = None,
):
self.name = name
self.conflict_resolve = conflict_resolve
self.max_delete = max_delete
self.check_access = check_access
self.description = description
self.extra_args = extra_args or []
# Bisync profiles based on SPEC-9 Phase 2.1
BISYNC_PROFILES = {
"safe": RcloneBisyncProfile(
name="safe",
conflict_resolve="none",
max_delete=10,
check_access=False,
description="Safe mode with conflict preservation (keeps both versions)",
),
"balanced": RcloneBisyncProfile(
name="balanced",
conflict_resolve="newer",
max_delete=25,
check_access=False,
description="Balanced mode - auto-resolve to newer file (recommended)",
),
"fast": RcloneBisyncProfile(
name="fast",
conflict_resolve="newer",
max_delete=50,
check_access=False,
description="Fast mode for rapid iteration (skip verification)",
),
}
async def get_mount_info() -> TenantMountInfo:
"""Get current tenant information from cloud API."""
try:
@@ -110,128 +42,6 @@ async def generate_mount_credentials(tenant_id: str) -> MountCredentials:
raise BisyncError(f"Failed to generate credentials: {e}") from e
async def fetch_cloud_projects() -> CloudProjectList:
"""Fetch list of projects from cloud API.
Returns:
CloudProjectList with projects from cloud
"""
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
response = await make_api_request(method="GET", url=f"{host_url}/proxy/projects/projects")
return CloudProjectList.model_validate(response.json())
except Exception as e:
raise BisyncError(f"Failed to fetch cloud projects: {e}") from e
def scan_local_directories(sync_dir: Path) -> list[str]:
"""Scan local sync directory for project folders.
Args:
sync_dir: Path to bisync directory
Returns:
List of directory names (project names)
"""
if not sync_dir.exists():
return []
directories = []
for item in sync_dir.iterdir():
if item.is_dir() and not item.name.startswith("."):
directories.append(item.name)
return directories
async def create_cloud_project(project_name: str) -> CloudProjectCreateResponse:
"""Create a new project on cloud.
Args:
project_name: Name of project to create
Returns:
CloudProjectCreateResponse with project details from API
"""
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
# Use generate_permalink to ensure consistent naming
project_path = generate_permalink(project_name)
project_data = CloudProjectCreateRequest(
name=project_name,
path=project_path,
set_default=False,
)
response = await make_api_request(
method="POST",
url=f"{host_url}/proxy/projects/projects",
headers={"Content-Type": "application/json"},
json_data=project_data.model_dump(),
)
return CloudProjectCreateResponse.model_validate(response.json())
except Exception as e:
raise BisyncError(f"Failed to create cloud project '{project_name}': {e}") from e
def get_bisync_state_path(tenant_id: str) -> Path:
"""Get path to bisync state directory."""
return Path.home() / ".basic-memory" / "bisync-state" / tenant_id
def get_bisync_directory() -> Path:
"""Get bisync directory from config.
Returns:
Path to bisync directory (default: ~/basic-memory-cloud-sync)
"""
config_manager = ConfigManager()
config = config_manager.config
sync_dir = config.bisync_config.get("sync_dir", str(Path.home() / "basic-memory-cloud-sync"))
return Path(sync_dir).expanduser().resolve()
def validate_bisync_directory(bisync_dir: Path) -> None:
"""Validate bisync directory doesn't conflict with mount.
Raises:
BisyncError: If bisync directory conflicts with mount directory
"""
# Get fixed mount directory
mount_dir = (Path.home() / "basic-memory-cloud").resolve()
# Check if bisync dir is the same as mount dir
if bisync_dir == mount_dir:
raise BisyncError(
f"Cannot use {bisync_dir} for bisync - it's the mount directory!\n"
f"Mount and bisync must use different directories.\n\n"
f"Options:\n"
f" 1. Use default: ~/basic-memory-cloud-sync/\n"
f" 2. Specify different directory: --dir ~/my-sync-folder"
)
# Check if mount is active at this location
result = subprocess.run(["mount"], capture_output=True, text=True)
if str(bisync_dir) in result.stdout and "rclone" in result.stdout:
raise BisyncError(
f"{bisync_dir} is currently mounted via 'bm cloud mount'\n"
f"Cannot use mounted directory for bisync.\n\n"
f"Either:\n"
f" 1. Unmount first: bm cloud unmount\n"
f" 2. Use different directory for bisync"
)
def convert_bmignore_to_rclone_filters() -> Path:
"""Convert .bmignore patterns to rclone filter format.
@@ -298,521 +108,3 @@ def get_bisync_filter_path() -> Path:
Path to rclone filter file
"""
return convert_bmignore_to_rclone_filters()
def bisync_state_exists(tenant_id: str) -> bool:
"""Check if bisync state exists (has been initialized)."""
state_path = get_bisync_state_path(tenant_id)
return state_path.exists() and any(state_path.iterdir())
def build_bisync_command(
tenant_id: str,
bucket_name: str,
local_path: Path,
profile: RcloneBisyncProfile,
dry_run: bool = False,
resync: bool = False,
verbose: bool = False,
) -> list[str]:
"""Build rclone bisync command with profile settings."""
# Sync with the entire bucket root (all projects)
rclone_remote = f"basic-memory-{tenant_id}:{bucket_name}"
filter_path = get_bisync_filter_path()
state_path = get_bisync_state_path(tenant_id)
# Ensure state directory exists
state_path.mkdir(parents=True, exist_ok=True)
cmd = [
"rclone",
"bisync",
str(local_path),
rclone_remote,
"--create-empty-src-dirs",
"--resilient",
f"--conflict-resolve={profile.conflict_resolve}",
f"--max-delete={profile.max_delete}",
"--filters-file",
str(filter_path),
"--workdir",
str(state_path),
]
# Add verbosity flags
if verbose:
cmd.append("--verbose") # Full details with file-by-file output
else:
# Show progress bar during transfers
cmd.append("--progress")
if profile.check_access:
cmd.append("--check-access")
if dry_run:
cmd.append("--dry-run")
if resync:
cmd.append("--resync")
cmd.extend(profile.extra_args)
return cmd
def setup_cloud_bisync(sync_dir: Optional[str] = None) -> None:
"""Set up cloud bisync with rclone installation and configuration.
Args:
sync_dir: Optional custom sync directory path. If not provided, uses config default.
"""
console.print("[bold blue]Basic Memory Cloud Bisync Setup[/bold blue]")
console.print("Setting up bidirectional sync to your cloud tenant...\n")
try:
# Step 1: Install rclone
console.print("[blue]Step 1: Installing rclone...[/blue]")
install_rclone()
# Step 2: Get mount info (for tenant_id, bucket)
console.print("\n[blue]Step 2: Getting tenant information...[/blue]")
tenant_info = asyncio.run(get_mount_info())
tenant_id = tenant_info.tenant_id
bucket_name = tenant_info.bucket_name
console.print(f"[green]✓ Found tenant: {tenant_id}[/green]")
console.print(f"[green]✓ Bucket: {bucket_name}[/green]")
# Step 3: Generate credentials
console.print("\n[blue]Step 3: Generating sync credentials...[/blue]")
creds = asyncio.run(generate_mount_credentials(tenant_id))
access_key = creds.access_key
secret_key = creds.secret_key
console.print("[green]✓ Generated secure credentials[/green]")
# Step 4: Configure rclone
console.print("\n[blue]Step 4: Configuring rclone...[/blue]")
add_tenant_to_rclone_config(
tenant_id=tenant_id,
bucket_name=bucket_name,
access_key=access_key,
secret_key=secret_key,
)
# Step 5: Configure and create local directory
console.print("\n[blue]Step 5: Configuring sync directory...[/blue]")
# If custom sync_dir provided, save to config
if sync_dir:
config_manager = ConfigManager()
config = config_manager.load_config()
config.bisync_config["sync_dir"] = sync_dir
config_manager.save_config(config)
console.print("[green]✓ Saved custom sync directory to config[/green]")
# Get bisync directory (from config or default)
local_path = get_bisync_directory()
# Validate bisync directory
validate_bisync_directory(local_path)
# Create directory
local_path.mkdir(parents=True, exist_ok=True)
console.print(f"[green]✓ Created sync directory: {local_path}[/green]")
# Step 6: Perform initial resync
console.print("\n[blue]Step 6: Performing initial sync...[/blue]")
console.print("[yellow]This will establish the baseline for bidirectional sync.[/yellow]")
run_bisync(
tenant_id=tenant_id,
bucket_name=bucket_name,
local_path=local_path,
profile_name="balanced",
resync=True,
)
console.print("\n[bold green]✓ Bisync setup completed successfully![/bold green]")
console.print("\nYour local files will now sync bidirectionally with the cloud!")
console.print(f"\nLocal directory: {local_path}")
console.print("\nUseful commands:")
console.print(" bm sync # Run sync (recommended)")
console.print(" bm sync --watch # Start watch mode")
console.print(" bm cloud status # Check sync status")
console.print(" bm cloud check # Verify file integrity")
console.print(" bm cloud bisync --dry-run # Preview changes (advanced)")
except (RcloneInstallError, BisyncError, CloudAPIError) as e:
console.print(f"\n[red]Setup failed: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"\n[red]Unexpected error during setup: {e}[/red]")
raise typer.Exit(1)
def run_bisync(
tenant_id: Optional[str] = None,
bucket_name: Optional[str] = None,
local_path: Optional[Path] = None,
profile_name: str = "balanced",
dry_run: bool = False,
resync: bool = False,
verbose: bool = False,
) -> bool:
"""Run rclone bisync with specified profile."""
try:
# Get tenant info if not provided
if not tenant_id or not bucket_name:
tenant_info = asyncio.run(get_mount_info())
tenant_id = tenant_info.tenant_id
bucket_name = tenant_info.bucket_name
# Set default local path if not provided
if not local_path:
local_path = get_bisync_directory()
# Validate bisync directory
validate_bisync_directory(local_path)
# Check if local path exists
if not local_path.exists():
raise BisyncError(
f"Local directory {local_path} does not exist. Run 'basic-memory cloud bisync-setup' first."
)
# Get bisync profile
if profile_name not in BISYNC_PROFILES:
raise BisyncError(
f"Unknown profile: {profile_name}. Available: {list(BISYNC_PROFILES.keys())}"
)
profile = BISYNC_PROFILES[profile_name]
# Auto-register projects before sync (unless dry-run or resync)
if not dry_run and not resync:
try:
console.print("[dim]Checking for new projects...[/dim]")
# Fetch cloud projects and extract directory names from paths
cloud_data = asyncio.run(fetch_cloud_projects())
cloud_projects = cloud_data.projects
# Extract directory names from cloud project paths
# Compare directory names, not project names
# Cloud path /app/data/basic-memory -> directory name "basic-memory"
cloud_dir_names = set()
for p in cloud_projects:
path = p.path
# Strip /app/data/ prefix if present (cloud mode)
if path.startswith("/app/data/"):
path = path[len("/app/data/") :]
# Get the last segment (directory name)
dir_name = Path(path).name
cloud_dir_names.add(dir_name)
# Scan local directories
local_dirs = scan_local_directories(local_path)
# Create missing cloud projects
new_projects = []
for dir_name in local_dirs:
if dir_name not in cloud_dir_names:
new_projects.append(dir_name)
if new_projects:
console.print(
f"[blue]Found {len(new_projects)} new local project(s), creating on cloud...[/blue]"
)
for project_name in new_projects:
try:
asyncio.run(create_cloud_project(project_name))
console.print(f"[green] ✓ Created project: {project_name}[/green]")
except BisyncError as e:
console.print(
f"[yellow] ⚠ Could not create {project_name}: {e}[/yellow]"
)
else:
console.print("[dim]All local projects already registered on cloud[/dim]")
except Exception as e:
console.print(f"[yellow]Warning: Project auto-registration failed: {e}[/yellow]")
console.print("[yellow]Continuing with sync anyway...[/yellow]")
# Check if first run and require resync
if not resync and not bisync_state_exists(tenant_id) and not dry_run:
raise BisyncError(
"First bisync requires --resync to establish baseline. "
"Run: basic-memory cloud bisync --resync"
)
# Build and execute bisync command
bisync_cmd = build_bisync_command(
tenant_id,
bucket_name,
local_path,
profile,
dry_run=dry_run,
resync=resync,
verbose=verbose,
)
if dry_run:
console.print("[yellow]DRY RUN MODE - No changes will be made[/yellow]")
console.print(
f"[blue]Running bisync with profile '{profile_name}' ({profile.description})...[/blue]"
)
console.print(f"[dim]Command: {' '.join(bisync_cmd)}[/dim]")
console.print() # Blank line before output
# Stream output in real-time so user sees progress
result = subprocess.run(bisync_cmd, text=True)
if result.returncode != 0:
raise BisyncError(f"Bisync command failed with code {result.returncode}")
console.print() # Blank line after output
if dry_run:
console.print("[green]✓ Dry run completed successfully[/green]")
elif resync:
console.print("[green]✓ Initial sync baseline established[/green]")
else:
console.print("[green]✓ Sync completed successfully[/green]")
# Notify container to refresh cache (if not dry run)
if not dry_run:
try:
asyncio.run(notify_container_sync(tenant_id))
except Exception as e:
console.print(f"[yellow]Warning: Could not notify container: {e}[/yellow]")
return True
except BisyncError:
raise
except Exception as e:
raise BisyncError(f"Unexpected error during bisync: {e}") from e
async def notify_container_sync(tenant_id: str) -> None:
"""Sync all projects after bisync completes."""
try:
from basic_memory.cli.commands.command_utils import run_sync
# Fetch all projects and sync each one
cloud_data = await fetch_cloud_projects()
projects = cloud_data.projects
if not projects:
console.print("[dim]No projects to sync[/dim]")
return
console.print(f"[blue]Notifying cloud to index {len(projects)} project(s)...[/blue]")
for project in projects:
project_name = project.name
if project_name:
try:
await run_sync(project=project_name)
except Exception as e:
# Non-critical, log and continue
console.print(f"[yellow] ⚠ Sync failed for {project_name}: {e}[/yellow]")
console.print("[dim]Note: Cloud indexing has started and may take a few moments[/dim]")
except Exception as e:
# Non-critical, don't fail the bisync
console.print(f"[yellow]Warning: Post-sync failed: {e}[/yellow]")
def run_bisync_watch(
tenant_id: Optional[str] = None,
bucket_name: Optional[str] = None,
local_path: Optional[Path] = None,
profile_name: str = "balanced",
interval_seconds: int = 60,
) -> None:
"""Run bisync in watch mode with periodic syncs."""
console.print("[bold blue]Starting bisync watch mode[/bold blue]")
console.print(f"Sync interval: {interval_seconds} seconds")
console.print("Press Ctrl+C to stop\n")
try:
while True:
try:
start_time = time.time()
run_bisync(
tenant_id=tenant_id,
bucket_name=bucket_name,
local_path=local_path,
profile_name=profile_name,
)
elapsed = time.time() - start_time
console.print(f"[dim]Sync completed in {elapsed:.1f}s[/dim]")
# Wait for next interval
time.sleep(interval_seconds)
except BisyncError as e:
console.print(f"[red]Sync error: {e}[/red]")
console.print(f"[yellow]Retrying in {interval_seconds} seconds...[/yellow]")
time.sleep(interval_seconds)
except KeyboardInterrupt:
console.print("\n[yellow]Watch mode stopped[/yellow]")
def show_bisync_status() -> None:
"""Show current bisync status and configuration."""
try:
# Get tenant info
tenant_info = asyncio.run(get_mount_info())
tenant_id = tenant_info.tenant_id
local_path = get_bisync_directory()
state_path = get_bisync_state_path(tenant_id)
# Create status table
table = Table(title="Cloud Bisync Status", show_header=True, header_style="bold blue")
table.add_column("Property", style="green", min_width=20)
table.add_column("Value", style="dim", min_width=30)
# Check initialization status
is_initialized = bisync_state_exists(tenant_id)
init_status = (
"[green]✓ Initialized[/green]" if is_initialized else "[red]✗ Not initialized[/red]"
)
table.add_row("Tenant ID", tenant_id)
table.add_row("Local Directory", str(local_path))
table.add_row("Status", init_status)
table.add_row("State Directory", str(state_path))
# Check for last sync info
if is_initialized:
# Look for most recent state file
state_files = list(state_path.glob("*.lst"))
if state_files:
latest = max(state_files, key=lambda p: p.stat().st_mtime)
last_sync = datetime.fromtimestamp(latest.stat().st_mtime)
table.add_row("Last Sync", last_sync.strftime("%Y-%m-%d %H:%M:%S"))
console.print(table)
# Show bisync profiles
console.print("\n[bold]Available bisync profiles:[/bold]")
for name, profile in BISYNC_PROFILES.items():
console.print(f" {name}: {profile.description}")
console.print(f" - Conflict resolution: {profile.conflict_resolve}")
console.print(f" - Max delete: {profile.max_delete} files")
console.print("\n[dim]To use a profile: bm cloud bisync --profile <name>[/dim]")
# Show setup instructions if not initialized
if not is_initialized:
console.print("\n[yellow]To initialize bisync, run:[/yellow]")
console.print(" bm cloud setup")
console.print(" or")
console.print(" bm cloud bisync --resync")
except Exception as e:
console.print(f"[red]Error getting bisync status: {e}[/red]")
raise typer.Exit(1)
def run_check(
tenant_id: Optional[str] = None,
bucket_name: Optional[str] = None,
local_path: Optional[Path] = None,
one_way: bool = False,
) -> bool:
"""Check file integrity between local and cloud using rclone check.
Args:
tenant_id: Cloud tenant ID (auto-detected if not provided)
bucket_name: S3 bucket name (auto-detected if not provided)
local_path: Local bisync directory (uses config default if not provided)
one_way: If True, only check for missing files on destination (faster)
Returns:
True if check passed (files match), False if differences found
"""
try:
# Check if rclone is installed
from basic_memory.cli.commands.cloud.rclone_installer import is_rclone_installed
if not is_rclone_installed():
raise BisyncError(
"rclone is not installed. Run 'bm cloud bisync-setup' first to set up cloud sync."
)
# Get tenant info if not provided
if not tenant_id or not bucket_name:
tenant_info = asyncio.run(get_mount_info())
tenant_id = tenant_id or tenant_info.tenant_id
bucket_name = bucket_name or tenant_info.bucket_name
# Get local path from config
if not local_path:
local_path = get_bisync_directory()
# Check if bisync is initialized
if not bisync_state_exists(tenant_id):
raise BisyncError(
"Bisync not initialized. Run 'bm cloud bisync --resync' to establish baseline."
)
# Build rclone check command
rclone_remote = f"basic-memory-{tenant_id}:{bucket_name}"
filter_path = get_bisync_filter_path()
cmd = [
"rclone",
"check",
str(local_path),
rclone_remote,
"--filter-from",
str(filter_path),
]
if one_way:
cmd.append("--one-way")
console.print("[bold blue]Checking file integrity between local and cloud[/bold blue]")
console.print(f"[dim]Local: {local_path}[/dim]")
console.print(f"[dim]Remote: {rclone_remote}[/dim]")
console.print(f"[dim]Command: {' '.join(cmd)}[/dim]")
console.print()
# Run check command
result = subprocess.run(cmd, capture_output=True, text=True)
# rclone check returns:
# 0 = success (all files match)
# non-zero = differences found or error
if result.returncode == 0:
console.print("[green]✓ All files match between local and cloud[/green]")
return True
else:
console.print("[yellow]⚠ Differences found:[/yellow]")
if result.stderr:
console.print(result.stderr)
if result.stdout:
console.print(result.stdout)
console.print("\n[dim]To sync differences, run: bm sync[/dim]")
return False
except BisyncError:
raise
except Exception as e:
raise BisyncError(f"Check failed: {e}") from e
@@ -0,0 +1,101 @@
"""Shared utilities for cloud operations."""
from basic_memory.cli.commands.cloud.api_client import make_api_request
from basic_memory.config import ConfigManager
from basic_memory.schemas.cloud import (
CloudProjectList,
CloudProjectCreateRequest,
CloudProjectCreateResponse,
)
from basic_memory.utils import generate_permalink
class CloudUtilsError(Exception):
"""Exception raised for cloud utility errors."""
pass
async def fetch_cloud_projects() -> CloudProjectList:
"""Fetch list of projects from cloud API.
Returns:
CloudProjectList with projects from cloud
"""
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
response = await make_api_request(method="GET", url=f"{host_url}/proxy/projects/projects")
return CloudProjectList.model_validate(response.json())
except Exception as e:
raise CloudUtilsError(f"Failed to fetch cloud projects: {e}") from e
async def create_cloud_project(project_name: str) -> CloudProjectCreateResponse:
"""Create a new project on cloud.
Args:
project_name: Name of project to create
Returns:
CloudProjectCreateResponse with project details from API
"""
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
# Use generate_permalink to ensure consistent naming
project_path = generate_permalink(project_name)
project_data = CloudProjectCreateRequest(
name=project_name,
path=project_path,
set_default=False,
)
response = await make_api_request(
method="POST",
url=f"{host_url}/proxy/projects/projects",
headers={"Content-Type": "application/json"},
json_data=project_data.model_dump(),
)
return CloudProjectCreateResponse.model_validate(response.json())
except Exception as e:
raise CloudUtilsError(f"Failed to create cloud project '{project_name}': {e}") from e
async def sync_project(project_name: str, force_full: bool = False) -> None:
"""Trigger sync for a specific project on cloud.
Args:
project_name: Name of project to sync
force_full: If True, force a full scan bypassing watermark optimization
"""
try:
from basic_memory.cli.commands.command_utils import run_sync
await run_sync(project=project_name, force_full=force_full)
except Exception as e:
raise CloudUtilsError(f"Failed to sync project '{project_name}': {e}") from e
async def project_exists(project_name: str) -> bool:
"""Check if a project exists on cloud.
Args:
project_name: Name of project to check
Returns:
True if project exists, False otherwise
"""
try:
projects = await fetch_cloud_projects()
project_names = {p.name for p in projects.projects}
return project_name in project_names
except Exception:
return False
@@ -1,7 +1,6 @@
"""Core cloud commands for Basic Memory CLI."""
import asyncio
from typing import Optional
import typer
from rich.console import Console
@@ -15,21 +14,16 @@ from basic_memory.cli.commands.cloud.api_client import (
get_cloud_config,
make_api_request,
)
from basic_memory.cli.commands.cloud.mount_commands import (
mount_cloud_files,
setup_cloud_mount,
show_mount_status,
unmount_cloud_files,
)
from basic_memory.cli.commands.cloud.bisync_commands import (
run_bisync,
run_bisync_watch,
run_check,
setup_cloud_bisync,
show_bisync_status,
BisyncError,
generate_mount_credentials,
get_mount_info,
)
from basic_memory.cli.commands.cloud.rclone_config import configure_rclone_remote
from basic_memory.cli.commands.cloud.rclone_installer import (
RcloneInstallError,
install_rclone,
)
from basic_memory.cli.commands.cloud.rclone_config import MOUNT_PROFILES
from basic_memory.cli.commands.cloud.bisync_commands import BISYNC_PROFILES
console = Console()
@@ -58,11 +52,11 @@ def login():
config.cloud_mode = True
config_manager.save_config(config)
console.print("[green]Cloud mode enabled[/green]")
console.print("[green]Cloud mode enabled[/green]")
console.print(f"[dim]All CLI commands now work against {host_url}[/dim]")
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
console.print(
@@ -83,23 +77,13 @@ def logout():
config.cloud_mode = False
config_manager.save_config(config)
console.print("[green]Cloud mode disabled[/green]")
console.print("[green]Cloud mode disabled[/green]")
console.print("[dim]All CLI commands now work locally[/dim]")
@cloud_app.command("status")
def status(
bisync: bool = typer.Option(
True,
"--bisync/--mount",
help="Show bisync status (default) or mount status",
),
) -> None:
"""Check cloud mode status and cloud instance health.
Shows cloud mode status, instance health, and sync/mount status.
Use --bisync (default) to show bisync status or --mount for mount status.
"""
def status() -> None:
"""Check cloud mode status and cloud instance health."""
# Check cloud mode
config_manager = ConfigManager()
config = config_manager.load_config()
@@ -142,12 +126,7 @@ def status(
if "timestamp" in health_data:
console.print(f" Timestamp: {health_data['timestamp']}")
# Show sync/mount status based on flag
console.print()
if bisync:
show_bisync_status()
else:
show_mount_status()
console.print("\n[dim]To sync projects, use: bm project bisync --name <project>[/dim]")
except CloudAPIError as e:
console.print(f"[red]Error checking cloud health: {e}[/red]")
@@ -157,132 +136,60 @@ def status(
raise typer.Exit(1)
# Mount commands
@cloud_app.command("setup")
def setup(
bisync: bool = typer.Option(
True,
"--bisync/--mount",
help="Use bidirectional sync (recommended) or mount as network drive",
),
sync_dir: Optional[str] = typer.Option(
None,
"--dir",
help="Custom sync directory for bisync (default: ~/basic-memory-cloud-sync)",
),
) -> None:
"""Set up cloud file access with automatic rclone installation and configuration.
def setup() -> None:
"""Set up cloud sync by installing rclone and configuring credentials.
Default: Sets up bidirectional sync (recommended).\n
Use --mount: Sets up mount as network drive (alternative workflow).\n
Examples:\n
bm cloud setup # Setup bisync (default)\n
bm cloud setup --mount # Setup mount instead\n
bm cloud setup --dir ~/sync # Custom bisync directory\n
SPEC-20: Simplified to project-scoped workflow.
After setup, use project commands for syncing:
bm project add <name> <path> --local-path ~/projects/<name>
bm project bisync --name <name> --resync # First time
bm project bisync --name <name> # Subsequent syncs
"""
if bisync:
setup_cloud_bisync(sync_dir=sync_dir)
else:
setup_cloud_mount()
console.print("[bold blue]Basic Memory Cloud Setup[/bold blue]")
console.print("Setting up cloud sync with rclone...\n")
@cloud_app.command("mount")
def mount(
profile: str = typer.Option(
"balanced", help=f"Mount profile: {', '.join(MOUNT_PROFILES.keys())}"
),
path: Optional[str] = typer.Option(
None, help="Custom mount path (default: ~/basic-memory-{tenant-id})"
),
) -> None:
"""Mount cloud files locally for editing."""
try:
mount_cloud_files(profile_name=profile)
except Exception as e:
console.print(f"[red]Mount failed: {e}[/red]")
# Step 1: Install rclone
console.print("[blue]Step 1: Installing rclone...[/blue]")
install_rclone()
# Step 2: Get tenant info
console.print("\n[blue]Step 2: Getting tenant information...[/blue]")
tenant_info = asyncio.run(get_mount_info())
console.print(f"[green]Found tenant: {tenant_info.tenant_id}[/green]")
# Step 3: Generate credentials
console.print("\n[blue]Step 3: Generating sync credentials...[/blue]")
creds = asyncio.run(generate_mount_credentials(tenant_info.tenant_id))
console.print("[green]Generated secure credentials[/green]")
# Step 4: Configure rclone remote
console.print("\n[blue]Step 4: Configuring rclone remote...[/blue]")
configure_rclone_remote(
access_key=creds.access_key,
secret_key=creds.secret_key,
)
console.print("\n[bold green]Cloud setup completed successfully![/bold green]")
console.print("\n[bold]Next steps:[/bold]")
console.print("1. Add a project with local sync path:")
console.print(" bm project add research --local-path ~/Documents/research")
console.print("\n Or configure sync for an existing project:")
console.print(" bm project sync-setup research ~/Documents/research")
console.print("\n2. Preview the initial sync (recommended):")
console.print(" bm project bisync --name research --resync --dry-run")
console.print("\n3. If all looks good, run the actual sync:")
console.print(" bm project bisync --name research --resync")
console.print("\n4. Subsequent syncs (no --resync needed):")
console.print(" bm project bisync --name research")
console.print(
"\n[dim]Tip: Always use --dry-run first to preview changes before syncing[/dim]"
)
except (RcloneInstallError, BisyncError, CloudAPIError) as e:
console.print(f"\n[red]Setup failed: {e}[/red]")
raise typer.Exit(1)
@cloud_app.command("unmount")
def unmount() -> None:
"""Unmount cloud files."""
try:
unmount_cloud_files()
except Exception as e:
console.print(f"[red]Unmount failed: {e}[/red]")
raise typer.Exit(1)
# Bisync commands
@cloud_app.command("bisync")
def bisync(
profile: str = typer.Option(
"balanced", help=f"Bisync profile: {', '.join(BISYNC_PROFILES.keys())}"
),
dry_run: bool = typer.Option(False, "--dry-run", help="Preview changes without syncing"),
resync: bool = typer.Option(False, "--resync", help="Force resync to establish new baseline"),
watch: bool = typer.Option(False, "--watch", help="Run continuous sync in watch mode"),
interval: int = typer.Option(60, "--interval", help="Sync interval in seconds for watch mode"),
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed sync output"),
) -> None:
"""Run bidirectional sync between local files and cloud storage.
Examples:
basic-memory cloud bisync # Manual sync with balanced profile
basic-memory cloud bisync --dry-run # Preview what would be synced
basic-memory cloud bisync --resync # Establish new baseline
basic-memory cloud bisync --watch # Continuous sync every 60s
basic-memory cloud bisync --watch --interval 30 # Continuous sync every 30s
basic-memory cloud bisync --profile safe # Use safe profile (keep conflicts)
basic-memory cloud bisync --verbose # Show detailed file sync output
"""
try:
if watch:
run_bisync_watch(profile_name=profile, interval_seconds=interval)
else:
run_bisync(profile_name=profile, dry_run=dry_run, resync=resync, verbose=verbose)
except Exception as e:
console.print(f"[red]Bisync failed: {e}[/red]")
raise typer.Exit(1)
@cloud_app.command("bisync-status")
def bisync_status() -> None:
"""Show current bisync status and configuration.
DEPRECATED: Use 'bm cloud status' instead (bisync is now the default).
"""
console.print(
"[yellow]Note: 'bisync-status' is deprecated. Use 'bm cloud status' instead.[/yellow]"
)
console.print("[dim]Showing bisync status...[/dim]\n")
show_bisync_status()
@cloud_app.command("check")
def check(
one_way: bool = typer.Option(
False,
"--one-way",
help="Only check for missing files on destination (faster)",
),
) -> None:
"""Check file integrity between local and cloud storage using rclone check.
Verifies that files match between your local bisync directory and cloud storage
without transferring any data. This is useful for validating sync integrity.
Examples:
bm cloud check # Full integrity check
bm cloud check --one-way # Faster check (missing files only)
"""
try:
run_check(one_way=one_way)
except Exception as e:
console.print(f"[red]Check failed: {e}[/red]")
console.print(f"\n[red]Unexpected error during setup: {e}[/red]")
raise typer.Exit(1)
@@ -1,295 +0,0 @@
"""Cloud mount commands for Basic Memory CLI."""
import asyncio
import subprocess
import time
from pathlib import Path
from typing import Optional
import typer
from rich.console import Console
from rich.table import Table
from basic_memory.cli.commands.cloud.api_client import CloudAPIError, make_api_request
from basic_memory.cli.commands.cloud.rclone_config import (
MOUNT_PROFILES,
add_tenant_to_rclone_config,
build_mount_command,
cleanup_orphaned_rclone_processes,
get_default_mount_path,
get_rclone_processes,
is_path_mounted,
unmount_path,
)
from basic_memory.cli.commands.cloud.rclone_installer import RcloneInstallError, install_rclone
from basic_memory.config import ConfigManager
console = Console()
class MountError(Exception):
"""Exception raised for mount-related errors."""
pass
async def get_tenant_info() -> dict:
"""Get current tenant information from cloud API."""
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
response = await make_api_request(method="GET", url=f"{host_url}/tenant/mount/info")
return response.json()
except Exception as e:
raise MountError(f"Failed to get tenant info: {e}") from e
async def generate_mount_credentials(tenant_id: str) -> dict:
"""Generate scoped credentials for mounting."""
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
response = await make_api_request(method="POST", url=f"{host_url}/tenant/mount/credentials")
return response.json()
except Exception as e:
raise MountError(f"Failed to generate mount credentials: {e}") from e
def setup_cloud_mount() -> None:
"""Set up cloud mount with rclone installation and configuration."""
console.print("[bold blue]Basic Memory Cloud Setup[/bold blue]")
console.print("Setting up local file access to your cloud tenant...\n")
try:
# Step 1: Install rclone
console.print("[blue]Step 1: Installing rclone...[/blue]")
install_rclone()
# Step 2: Get tenant info
console.print("\n[blue]Step 2: Getting tenant information...[/blue]")
tenant_info = asyncio.run(get_tenant_info())
tenant_id = tenant_info.get("tenant_id")
bucket_name = tenant_info.get("bucket_name")
if not tenant_id or not bucket_name:
raise MountError("Invalid tenant information received from cloud API")
console.print(f"[green]✓ Found tenant: {tenant_id}[/green]")
console.print(f"[green]✓ Bucket: {bucket_name}[/green]")
# Step 3: Generate mount credentials
console.print("\n[blue]Step 3: Generating mount credentials...[/blue]")
creds = asyncio.run(generate_mount_credentials(tenant_id))
access_key = creds.get("access_key")
secret_key = creds.get("secret_key")
if not access_key or not secret_key:
raise MountError("Failed to generate mount credentials")
console.print("[green]✓ Generated secure credentials[/green]")
# Step 4: Configure rclone
console.print("\n[blue]Step 4: Configuring rclone...[/blue]")
add_tenant_to_rclone_config(
tenant_id=tenant_id,
bucket_name=bucket_name,
access_key=access_key,
secret_key=secret_key,
)
# Step 5: Perform initial mount
console.print("\n[blue]Step 5: Mounting cloud files...[/blue]")
mount_path = get_default_mount_path()
MOUNT_PROFILES["balanced"]
mount_cloud_files(
tenant_id=tenant_id,
bucket_name=bucket_name,
mount_path=mount_path,
profile_name="balanced",
)
console.print("\n[bold green]✓ Cloud setup completed successfully![/bold green]")
console.print("\nYour cloud files are now accessible at:")
console.print(f" {mount_path}")
console.print("\nYou can now edit files locally and they will sync to the cloud!")
console.print("\nUseful commands:")
console.print(" basic-memory cloud mount-status # Check mount status")
console.print(" basic-memory cloud unmount # Unmount files")
console.print(" basic-memory cloud mount --profile fast # Remount with faster sync")
except (RcloneInstallError, MountError, CloudAPIError) as e:
console.print(f"\n[red]Setup failed: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"\n[red]Unexpected error during setup: {e}[/red]")
raise typer.Exit(1)
def mount_cloud_files(
tenant_id: Optional[str] = None,
bucket_name: Optional[str] = None,
mount_path: Optional[Path] = None,
profile_name: str = "balanced",
) -> None:
"""Mount cloud files with specified profile."""
try:
# Get tenant info if not provided
if not tenant_id or not bucket_name:
tenant_info = asyncio.run(get_tenant_info())
tenant_id = tenant_info.get("tenant_id")
bucket_name = tenant_info.get("bucket_name")
if not tenant_id or not bucket_name:
raise MountError("Could not determine tenant information")
# Set default mount path if not provided
if not mount_path:
mount_path = get_default_mount_path()
# Get mount profile
if profile_name not in MOUNT_PROFILES:
raise MountError(
f"Unknown profile: {profile_name}. Available: {list(MOUNT_PROFILES.keys())}"
)
profile = MOUNT_PROFILES[profile_name]
# Check if already mounted
if is_path_mounted(mount_path):
console.print(f"[yellow]Path {mount_path} is already mounted[/yellow]")
console.print("Use 'basic-memory cloud unmount' first, or mount to a different path")
return
# Create mount directory
mount_path.mkdir(parents=True, exist_ok=True)
# Build and execute mount command
mount_cmd = build_mount_command(tenant_id, bucket_name, mount_path, profile)
console.print(
f"[blue]Mounting with profile '{profile_name}' ({profile.description})...[/blue]"
)
console.print(f"[dim]Command: {' '.join(mount_cmd)}[/dim]")
result = subprocess.run(mount_cmd, capture_output=True, text=True)
if result.returncode != 0:
error_msg = result.stderr or "Unknown error"
raise MountError(f"Mount command failed: {error_msg}")
# Wait a moment for mount to establish
time.sleep(2)
# Verify mount
if is_path_mounted(mount_path):
console.print(f"[green]✓ Successfully mounted to {mount_path}[/green]")
console.print(f"[green]✓ Sync profile: {profile.description}[/green]")
else:
raise MountError("Mount command succeeded but path is not mounted")
except MountError:
raise
except Exception as e:
raise MountError(f"Unexpected error during mount: {e}") from e
def unmount_cloud_files(tenant_id: Optional[str] = None) -> None:
"""Unmount cloud files."""
try:
# Get tenant info if not provided
if not tenant_id:
tenant_info = asyncio.run(get_tenant_info())
tenant_id = tenant_info.get("tenant_id")
if not tenant_id:
raise MountError("Could not determine tenant ID")
mount_path = get_default_mount_path()
if not is_path_mounted(mount_path):
console.print(f"[yellow]Path {mount_path} is not mounted[/yellow]")
return
console.print(f"[blue]Unmounting {mount_path}...[/blue]")
# Unmount the path
if unmount_path(mount_path):
console.print(f"[green]✓ Successfully unmounted {mount_path}[/green]")
# Clean up any orphaned rclone processes
killed_count = cleanup_orphaned_rclone_processes()
if killed_count > 0:
console.print(
f"[green]✓ Cleaned up {killed_count} orphaned rclone process(es)[/green]"
)
else:
console.print(f"[red]✗ Failed to unmount {mount_path}[/red]")
console.print("You may need to manually unmount or restart your system")
except MountError:
raise
except Exception as e:
raise MountError(f"Unexpected error during unmount: {e}") from e
def show_mount_status() -> None:
"""Show current mount status and running processes."""
try:
# Get tenant info
tenant_info = asyncio.run(get_tenant_info())
tenant_id = tenant_info.get("tenant_id")
if not tenant_id:
console.print("[red]Could not determine tenant ID[/red]")
return
mount_path = get_default_mount_path()
# Create status table
table = Table(title="Cloud Mount Status", show_header=True, header_style="bold blue")
table.add_column("Property", style="green", min_width=15)
table.add_column("Value", style="dim", min_width=30)
# Check mount status
is_mounted = is_path_mounted(mount_path)
mount_status = "[green]✓ Mounted[/green]" if is_mounted else "[red]✗ Not mounted[/red]"
table.add_row("Tenant ID", tenant_id)
table.add_row("Mount Path", str(mount_path))
table.add_row("Status", mount_status)
# Get rclone processes
processes = get_rclone_processes()
if processes:
table.add_row("rclone Processes", f"{len(processes)} running")
else:
table.add_row("rclone Processes", "None")
console.print(table)
# Show running processes details
if processes:
console.print("\n[bold]Running rclone processes:[/bold]")
for proc in processes:
console.print(f" PID {proc['pid']}: {proc['command'][:80]}...")
# Show mount profiles
console.print("\n[bold]Available mount profiles:[/bold]")
for name, profile in MOUNT_PROFILES.items():
console.print(f" {name}: {profile.description}")
except Exception as e:
console.print(f"[red]Error getting mount status: {e}[/red]")
raise typer.Exit(1)
@@ -0,0 +1,301 @@
"""Project-scoped rclone sync commands for Basic Memory Cloud.
This module provides simplified, project-scoped rclone operations:
- Each project syncs independently
- Uses single "basic-memory-cloud" remote (not tenant-specific)
- Balanced defaults from SPEC-8 Phase 4 testing
- Per-project bisync state tracking
Replaces tenant-wide sync with project-scoped workflows.
"""
import subprocess
from dataclasses import dataclass
from pathlib import Path
from typing import Optional
from rich.console import Console
from basic_memory.utils import normalize_project_path
console = Console()
class RcloneError(Exception):
"""Exception raised for rclone command errors."""
pass
@dataclass
class SyncProject:
"""Project configured for cloud sync.
Attributes:
name: Project name
path: Cloud path (e.g., "app/data/research")
local_sync_path: Local directory for syncing (optional)
"""
name: str
path: str
local_sync_path: Optional[str] = None
def get_bmignore_filter_path() -> Path:
"""Get path to rclone filter file.
Uses ~/.basic-memory/.bmignore converted to rclone format.
File is automatically created with default patterns on first use.
Returns:
Path to rclone filter file
"""
# Import here to avoid circular dependency
from basic_memory.cli.commands.cloud.bisync_commands import (
convert_bmignore_to_rclone_filters,
)
return convert_bmignore_to_rclone_filters()
def get_project_bisync_state(project_name: str) -> Path:
"""Get path to project's bisync state directory.
Args:
project_name: Name of the project
Returns:
Path to bisync state directory for this project
"""
return Path.home() / ".basic-memory" / "bisync-state" / project_name
def bisync_initialized(project_name: str) -> bool:
"""Check if bisync has been initialized for this project.
Args:
project_name: Name of the project
Returns:
True if bisync state exists, False otherwise
"""
state_path = get_project_bisync_state(project_name)
return state_path.exists() and any(state_path.iterdir())
def get_project_remote(project: SyncProject, bucket_name: str) -> str:
"""Build rclone remote path for project.
Args:
project: Project with cloud path
bucket_name: S3 bucket name
Returns:
Remote path like "basic-memory-cloud:bucket-name/basic-memory-llc"
Note:
The API returns paths like "/app/data/basic-memory-llc" because the S3 bucket
is mounted at /app/data on the fly machine. We need to strip the /app/data/
prefix to get the actual S3 path within the bucket.
"""
# Normalize path to strip /app/data/ mount point prefix
cloud_path = normalize_project_path(project.path).lstrip("/")
return f"basic-memory-cloud:{bucket_name}/{cloud_path}"
def project_sync(
project: SyncProject,
bucket_name: str,
dry_run: bool = False,
verbose: bool = False,
) -> bool:
"""One-way sync: local → cloud.
Makes cloud identical to local using rclone sync.
Args:
project: Project to sync
bucket_name: S3 bucket name
dry_run: Preview changes without applying
verbose: Show detailed output
Returns:
True if sync succeeded, False otherwise
Raises:
RcloneError: If project has no local_sync_path configured
"""
if not project.local_sync_path:
raise RcloneError(f"Project {project.name} has no local_sync_path configured")
local_path = Path(project.local_sync_path).expanduser()
remote_path = get_project_remote(project, bucket_name)
filter_path = get_bmignore_filter_path()
cmd = [
"rclone",
"sync",
str(local_path),
remote_path,
"--filter-from",
str(filter_path),
]
if verbose:
cmd.append("--verbose")
else:
cmd.append("--progress")
if dry_run:
cmd.append("--dry-run")
result = subprocess.run(cmd, text=True)
return result.returncode == 0
def project_bisync(
project: SyncProject,
bucket_name: str,
dry_run: bool = False,
resync: bool = False,
verbose: bool = False,
) -> bool:
"""Two-way sync: local ↔ cloud.
Uses rclone bisync with balanced defaults:
- conflict_resolve: newer (auto-resolve to most recent)
- max_delete: 25 (safety limit)
- compare: modtime (ignore size differences from line ending conversions)
- check_access: false (skip for performance)
Args:
project: Project to sync
bucket_name: S3 bucket name
dry_run: Preview changes without applying
resync: Force resync to establish new baseline
verbose: Show detailed output
Returns:
True if bisync succeeded, False otherwise
Raises:
RcloneError: If project has no local_sync_path or needs --resync
"""
if not project.local_sync_path:
raise RcloneError(f"Project {project.name} has no local_sync_path configured")
local_path = Path(project.local_sync_path).expanduser()
remote_path = get_project_remote(project, bucket_name)
filter_path = get_bmignore_filter_path()
state_path = get_project_bisync_state(project.name)
# Ensure state directory exists
state_path.mkdir(parents=True, exist_ok=True)
cmd = [
"rclone",
"bisync",
str(local_path),
remote_path,
"--create-empty-src-dirs",
"--resilient",
"--conflict-resolve=newer",
"--max-delete=25",
"--compare=modtime", # Ignore size differences from line ending conversions
"--filter-from",
str(filter_path),
"--workdir",
str(state_path),
]
if verbose:
cmd.append("--verbose")
else:
cmd.append("--progress")
if dry_run:
cmd.append("--dry-run")
if resync:
cmd.append("--resync")
# Check if first run requires resync
if not resync and not bisync_initialized(project.name) and not dry_run:
raise RcloneError(
f"First bisync for {project.name} requires --resync to establish baseline.\n"
f"Run: bm project bisync --name {project.name} --resync"
)
result = subprocess.run(cmd, text=True)
return result.returncode == 0
def project_check(
project: SyncProject,
bucket_name: str,
one_way: bool = False,
) -> bool:
"""Check integrity between local and cloud.
Verifies files match without transferring data.
Args:
project: Project to check
bucket_name: S3 bucket name
one_way: Only check for missing files on destination (faster)
Returns:
True if files match, False if differences found
Raises:
RcloneError: If project has no local_sync_path configured
"""
if not project.local_sync_path:
raise RcloneError(f"Project {project.name} has no local_sync_path configured")
local_path = Path(project.local_sync_path).expanduser()
remote_path = get_project_remote(project, bucket_name)
filter_path = get_bmignore_filter_path()
cmd = [
"rclone",
"check",
str(local_path),
remote_path,
"--filter-from",
str(filter_path),
]
if one_way:
cmd.append("--one-way")
result = subprocess.run(cmd, capture_output=True, text=True)
return result.returncode == 0
def project_ls(
project: SyncProject,
bucket_name: str,
path: Optional[str] = None,
) -> list[str]:
"""List files in remote project.
Args:
project: Project to list files from
bucket_name: S3 bucket name
path: Optional subdirectory within project
Returns:
List of file paths
Raises:
subprocess.CalledProcessError: If rclone command fails
"""
remote_path = get_project_remote(project, bucket_name)
if path:
remote_path = f"{remote_path}/{path}"
cmd = ["rclone", "ls", remote_path]
result = subprocess.run(cmd, capture_output=True, text=True, check=True)
return result.stdout.splitlines()
@@ -1,11 +1,14 @@
"""rclone configuration management for Basic Memory Cloud."""
"""rclone configuration management for Basic Memory Cloud.
This module provides simplified rclone configuration for SPEC-20.
Uses a single "basic-memory-cloud" remote for all operations.
"""
import configparser
import os
import shutil
import subprocess
from pathlib import Path
from typing import Dict, List, Optional
from typing import Optional
from rich.console import Console
@@ -18,64 +21,6 @@ class RcloneConfigError(Exception):
pass
class RcloneMountProfile:
"""Mount profile with optimized settings."""
def __init__(
self,
name: str,
cache_time: str,
poll_interval: str,
attr_timeout: str,
write_back: str,
description: str,
extra_args: Optional[List[str]] = None,
):
self.name = name
self.cache_time = cache_time
self.poll_interval = poll_interval
self.attr_timeout = attr_timeout
self.write_back = write_back
self.description = description
self.extra_args = extra_args or []
# Mount profiles based on SPEC-7 Phase 4 testing
MOUNT_PROFILES = {
"fast": RcloneMountProfile(
name="fast",
cache_time="5s",
poll_interval="3s",
attr_timeout="3s",
write_back="1s",
description="Ultra-fast development (5s sync, higher bandwidth)",
),
"balanced": RcloneMountProfile(
name="balanced",
cache_time="10s",
poll_interval="5s",
attr_timeout="5s",
write_back="2s",
description="Fast development (10-15s sync, recommended)",
),
"safe": RcloneMountProfile(
name="safe",
cache_time="15s",
poll_interval="10s",
attr_timeout="10s",
write_back="5s",
description="Conflict-aware mount with backup",
extra_args=[
"--conflict-suffix",
".conflict-{DateTimeExt}",
"--backup-dir",
"~/.basic-memory/conflicts",
"--track-renames",
],
),
}
def get_rclone_config_path() -> Path:
"""Get the path to rclone configuration file."""
config_dir = Path.home() / ".config" / "rclone"
@@ -116,173 +61,50 @@ def save_rclone_config(config: configparser.ConfigParser) -> None:
console.print(f"[dim]Updated rclone config: {config_path}[/dim]")
def add_tenant_to_rclone_config(
tenant_id: str,
bucket_name: str,
def configure_rclone_remote(
access_key: str,
secret_key: str,
endpoint: str = "https://fly.storage.tigris.dev",
region: str = "auto",
) -> str:
"""Add tenant configuration to rclone config file."""
"""Configure single rclone remote named 'basic-memory-cloud'.
This is the simplified approach from SPEC-20 that uses one remote
for all Basic Memory cloud operations (not tenant-specific).
Args:
access_key: S3 access key ID
secret_key: S3 secret access key
endpoint: S3-compatible endpoint URL
region: S3 region (default: auto)
Returns:
The remote name: "basic-memory-cloud"
"""
# Backup existing config
backup_rclone_config()
# Load existing config
config = load_rclone_config()
# Create section name
section_name = f"basic-memory-{tenant_id}"
# Single remote name (not tenant-specific)
REMOTE_NAME = "basic-memory-cloud"
# Add/update the tenant section
if not config.has_section(section_name):
config.add_section(section_name)
config.set(section_name, "type", "s3")
config.set(section_name, "provider", "Other")
config.set(section_name, "access_key_id", access_key)
config.set(section_name, "secret_access_key", secret_key)
config.set(section_name, "endpoint", endpoint)
config.set(section_name, "region", region)
# Add/update the remote section
if not config.has_section(REMOTE_NAME):
config.add_section(REMOTE_NAME)
config.set(REMOTE_NAME, "type", "s3")
config.set(REMOTE_NAME, "provider", "Other")
config.set(REMOTE_NAME, "access_key_id", access_key)
config.set(REMOTE_NAME, "secret_access_key", secret_key)
config.set(REMOTE_NAME, "endpoint", endpoint)
config.set(REMOTE_NAME, "region", region)
# Prevent unnecessary encoding of filenames (only encode slashes and invalid UTF-8)
# This prevents files with spaces like "Hello World.md" from being quoted
config.set(REMOTE_NAME, "encoding", "Slash,InvalidUtf8")
# Save updated config
save_rclone_config(config)
console.print(f"[green]✓ Added tenant {tenant_id} to rclone config[/green]")
return section_name
def remove_tenant_from_rclone_config(tenant_id: str) -> bool:
"""Remove tenant configuration from rclone config."""
config = load_rclone_config()
section_name = f"basic-memory-{tenant_id}"
if config.has_section(section_name):
backup_rclone_config()
config.remove_section(section_name)
save_rclone_config(config)
console.print(f"[green]✓ Removed tenant {tenant_id} from rclone config[/green]")
return True
return False
def get_default_mount_path() -> Path:
"""Get default mount path (fixed location per SPEC-9).
Returns:
Fixed mount path: ~/basic-memory-cloud/
"""
return Path.home() / "basic-memory-cloud"
def build_mount_command(
tenant_id: str, bucket_name: str, mount_path: Path, profile: RcloneMountProfile
) -> List[str]:
"""Build rclone mount command with optimized settings."""
rclone_remote = f"basic-memory-{tenant_id}:{bucket_name}"
cmd = [
"rclone",
"nfsmount",
rclone_remote,
str(mount_path),
"--vfs-cache-mode",
"writes",
"--dir-cache-time",
profile.cache_time,
"--vfs-cache-poll-interval",
profile.poll_interval,
"--attr-timeout",
profile.attr_timeout,
"--vfs-write-back",
profile.write_back,
"--daemon",
]
# Add profile-specific extra arguments
cmd.extend(profile.extra_args)
return cmd
def is_path_mounted(mount_path: Path) -> bool:
"""Check if a path is currently mounted."""
if not mount_path.exists():
return False
try:
# Check if mount point is actually mounted by looking for mount table entry
result = subprocess.run(["mount"], capture_output=True, text=True, check=False)
if result.returncode == 0:
# Look for our mount path in mount output
mount_str = str(mount_path.resolve())
return mount_str in result.stdout
return False
except Exception:
return False
def get_rclone_processes() -> List[Dict[str, str]]:
"""Get list of running rclone processes."""
try:
# Use ps to find rclone processes
result = subprocess.run(
["ps", "-eo", "pid,args"], capture_output=True, text=True, check=False
)
processes = []
if result.returncode == 0:
for line in result.stdout.split("\n"):
if "rclone" in line and "basic-memory" in line:
parts = line.strip().split(None, 1)
if len(parts) >= 2:
processes.append({"pid": parts[0], "command": parts[1]})
return processes
except Exception:
return []
def kill_rclone_process(pid: str) -> bool:
"""Kill a specific rclone process."""
try:
subprocess.run(["kill", pid], check=True)
console.print(f"[green]✓ Killed rclone process {pid}[/green]")
return True
except subprocess.CalledProcessError:
console.print(f"[red]✗ Failed to kill rclone process {pid}[/red]")
return False
def unmount_path(mount_path: Path) -> bool:
"""Unmount a mounted path."""
if not is_path_mounted(mount_path):
return True
try:
subprocess.run(["umount", str(mount_path)], check=True)
console.print(f"[green]✓ Unmounted {mount_path}[/green]")
return True
except subprocess.CalledProcessError as e:
console.print(f"[red]✗ Failed to unmount {mount_path}: {e}[/red]")
return False
def cleanup_orphaned_rclone_processes() -> int:
"""Clean up orphaned rclone processes for basic-memory."""
processes = get_rclone_processes()
killed_count = 0
for proc in processes:
console.print(
f"[yellow]Found rclone process: {proc['pid']} - {proc['command'][:80]}...[/yellow]"
)
if kill_rclone_process(proc["pid"]):
killed_count += 1
return killed_count
console.print(f"[green]Configured rclone remote: {REMOTE_NAME}[/green]")
return REMOTE_NAME
@@ -1,5 +1,6 @@
"""Cross-platform rclone installation utilities."""
import os
import platform
import shutil
import subprocess
@@ -58,7 +59,7 @@ def install_rclone_macos() -> None:
try:
console.print("[blue]Installing rclone via Homebrew...[/blue]")
run_command(["brew", "install", "rclone"])
console.print("[green]rclone installed via Homebrew[/green]")
console.print("[green]rclone installed via Homebrew[/green]")
return
except RcloneInstallError:
console.print(
@@ -69,7 +70,7 @@ def install_rclone_macos() -> None:
console.print("[blue]Installing rclone via official script...[/blue]")
try:
run_command(["sh", "-c", "curl https://rclone.org/install.sh | sudo bash"])
console.print("[green]rclone installed via official script[/green]")
console.print("[green]rclone installed via official script[/green]")
except RcloneInstallError:
raise RcloneInstallError(
"Failed to install rclone. Please install manually: brew install rclone"
@@ -83,7 +84,7 @@ def install_rclone_linux() -> None:
try:
console.print("[blue]Installing rclone via snap...[/blue]")
run_command(["sudo", "snap", "install", "rclone"])
console.print("[green]rclone installed via snap[/green]")
console.print("[green]rclone installed via snap[/green]")
return
except RcloneInstallError:
console.print("[yellow]Snap installation failed, trying apt...[/yellow]")
@@ -94,7 +95,7 @@ def install_rclone_linux() -> None:
console.print("[blue]Installing rclone via apt...[/blue]")
run_command(["sudo", "apt", "update"])
run_command(["sudo", "apt", "install", "-y", "rclone"])
console.print("[green]rclone installed via apt[/green]")
console.print("[green]rclone installed via apt[/green]")
return
except RcloneInstallError:
console.print("[yellow]apt installation failed, trying official script...[/yellow]")
@@ -103,7 +104,7 @@ def install_rclone_linux() -> None:
console.print("[blue]Installing rclone via official script...[/blue]")
try:
run_command(["sh", "-c", "curl https://rclone.org/install.sh | sudo bash"])
console.print("[green]rclone installed via official script[/green]")
console.print("[green]rclone installed via official script[/green]")
except RcloneInstallError:
raise RcloneInstallError(
"Failed to install rclone. Please install manually: sudo snap install rclone"
@@ -116,8 +117,16 @@ def install_rclone_windows() -> None:
if shutil.which("winget"):
try:
console.print("[blue]Installing rclone via winget...[/blue]")
run_command(["winget", "install", "Rclone.Rclone"])
console.print("[green]✓ rclone installed via winget[/green]")
run_command(
[
"winget",
"install",
"Rclone.Rclone",
"--accept-source-agreements",
"--accept-package-agreements",
]
)
console.print("[green]rclone installed via winget[/green]")
return
except RcloneInstallError:
console.print("[yellow]winget installation failed, trying chocolatey...[/yellow]")
@@ -127,7 +136,7 @@ def install_rclone_windows() -> None:
try:
console.print("[blue]Installing rclone via chocolatey...[/blue]")
run_command(["choco", "install", "rclone", "-y"])
console.print("[green]rclone installed via chocolatey[/green]")
console.print("[green]rclone installed via chocolatey[/green]")
return
except RcloneInstallError:
console.print("[yellow]chocolatey installation failed, trying scoop...[/yellow]")
@@ -137,7 +146,7 @@ def install_rclone_windows() -> None:
try:
console.print("[blue]Installing rclone via scoop...[/blue]")
run_command(["scoop", "install", "rclone"])
console.print("[green]rclone installed via scoop[/green]")
console.print("[green]rclone installed via scoop[/green]")
return
except RcloneInstallError:
console.print("[yellow]scoop installation failed[/yellow]")
@@ -165,6 +174,7 @@ def install_rclone(platform_override: Optional[str] = None) -> None:
install_rclone_linux()
elif platform_name == "windows":
install_rclone_windows()
refresh_windows_path()
else:
raise RcloneInstallError(f"Unsupported platform: {platform_name}")
@@ -172,7 +182,7 @@ def install_rclone(platform_override: Optional[str] = None) -> None:
if not is_rclone_installed():
raise RcloneInstallError("rclone installation completed but command not found in PATH")
console.print("[green]rclone installation completed successfully[/green]")
console.print("[green]rclone installation completed successfully[/green]")
except RcloneInstallError:
raise
@@ -180,6 +190,47 @@ def install_rclone(platform_override: Optional[str] = None) -> None:
raise RcloneInstallError(f"Unexpected error during installation: {e}") from e
def refresh_windows_path() -> None:
"""Refresh the Windows PATH environment variable for the current session."""
if platform.system().lower() != "windows":
return
# Importing here after performing platform detection. Also note that we have to ignore pylance/pyright
# warnings about winreg attributes so that "errors" don't appear on non-Windows platforms.
import winreg
user_key_path = r"Environment"
system_key_path = r"System\CurrentControlSet\Control\Session Manager\Environment"
new_path = ""
# Read user PATH
try:
reg_key = winreg.OpenKey(winreg.HKEY_CURRENT_USER, user_key_path, 0, winreg.KEY_READ) # type: ignore[reportAttributeAccessIssue]
user_path, _ = winreg.QueryValueEx(reg_key, "PATH") # type: ignore[reportAttributeAccessIssue]
winreg.CloseKey(reg_key) # type: ignore[reportAttributeAccessIssue]
except Exception:
user_path = ""
# Read system PATH
try:
reg_key = winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE, system_key_path, 0, winreg.KEY_READ) # type: ignore[reportAttributeAccessIssue]
system_path, _ = winreg.QueryValueEx(reg_key, "PATH") # type: ignore[reportAttributeAccessIssue]
winreg.CloseKey(reg_key) # type: ignore[reportAttributeAccessIssue]
except Exception:
system_path = ""
# Merge user and system PATHs (system first, then user)
if system_path and user_path:
new_path = system_path + ";" + user_path
elif system_path:
new_path = system_path
elif user_path:
new_path = user_path
if new_path:
os.environ["PATH"] = new_path
def get_rclone_version() -> Optional[str]:
"""Get the installed rclone version."""
if not is_rclone_installed():
@@ -0,0 +1,233 @@
"""WebDAV upload functionality for basic-memory projects."""
import os
from pathlib import Path
import aiofiles
import httpx
from basic_memory.ignore_utils import load_gitignore_patterns, should_ignore_path
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.tools.utils import call_put
# Archive file extensions that should be skipped during upload
ARCHIVE_EXTENSIONS = {".zip", ".tar", ".gz", ".bz2", ".xz", ".7z", ".rar", ".tgz", ".tbz2"}
async def upload_path(
local_path: Path,
project_name: str,
verbose: bool = False,
use_gitignore: bool = True,
dry_run: bool = False,
) -> bool:
"""
Upload a file or directory to cloud project via WebDAV.
Args:
local_path: Path to local file or directory
project_name: Name of cloud project (destination)
verbose: Show detailed information about filtering and upload
use_gitignore: If False, skip .gitignore patterns (still use .bmignore)
dry_run: If True, show what would be uploaded without uploading
Returns:
True if upload succeeded, False otherwise
"""
try:
# Resolve path
local_path = local_path.resolve()
# Check if path exists
if not local_path.exists():
print(f"Error: Path does not exist: {local_path}")
return False
# Get files to upload
if local_path.is_file():
files_to_upload = [(local_path, local_path.name)]
if verbose:
print(f"Uploading single file: {local_path.name}")
else:
files_to_upload = _get_files_to_upload(local_path, verbose, use_gitignore)
if not files_to_upload:
print("No files found to upload")
if verbose:
print(
"\nTip: Use --verbose to see which files are being filtered, "
"or --no-gitignore to skip .gitignore patterns"
)
return True
print(f"Found {len(files_to_upload)} file(s) to upload")
# Calculate total size
total_bytes = sum(file_path.stat().st_size for file_path, _ in files_to_upload)
skipped_count = 0
# If dry run, just show what would be uploaded
if dry_run:
print("\nFiles that would be uploaded:")
for file_path, relative_path in files_to_upload:
# Skip archive files
if _is_archive_file(file_path):
print(f" [SKIP] {relative_path} (archive file)")
skipped_count += 1
continue
size = file_path.stat().st_size
if size < 1024:
size_str = f"{size} bytes"
elif size < 1024 * 1024:
size_str = f"{size / 1024:.1f} KB"
else:
size_str = f"{size / (1024 * 1024):.1f} MB"
print(f" {relative_path} ({size_str})")
else:
# Upload files using httpx
async with get_client() as client:
for i, (file_path, relative_path) in enumerate(files_to_upload, 1):
# Skip archive files (zip, tar, gz, etc.)
if _is_archive_file(file_path):
print(
f"Skipping archive file: {relative_path} ({i}/{len(files_to_upload)})"
)
skipped_count += 1
continue
# Build remote path: /webdav/{project_name}/{relative_path}
remote_path = f"/webdav/{project_name}/{relative_path}"
print(f"Uploading {relative_path} ({i}/{len(files_to_upload)})")
# Get file modification time
file_stat = file_path.stat()
mtime = int(file_stat.st_mtime)
# Read file content asynchronously
async with aiofiles.open(file_path, "rb") as f:
content = await f.read()
# Upload via HTTP PUT to WebDAV endpoint with mtime header
# Using X-OC-Mtime (ownCloud/Nextcloud standard)
response = await call_put(
client, remote_path, content=content, headers={"X-OC-Mtime": str(mtime)}
)
response.raise_for_status()
# Format total size based on magnitude
if total_bytes < 1024:
size_str = f"{total_bytes} bytes"
elif total_bytes < 1024 * 1024:
size_str = f"{total_bytes / 1024:.1f} KB"
else:
size_str = f"{total_bytes / (1024 * 1024):.1f} MB"
uploaded_count = len(files_to_upload) - skipped_count
if dry_run:
print(f"\nTotal: {uploaded_count} file(s) ({size_str})")
if skipped_count > 0:
print(f" Would skip {skipped_count} archive file(s)")
else:
print(f"✓ Upload complete: {uploaded_count} file(s) ({size_str})")
if skipped_count > 0:
print(f" Skipped {skipped_count} archive file(s)")
return True
except httpx.HTTPStatusError as e:
print(f"Upload failed: HTTP {e.response.status_code} - {e.response.text}")
return False
except Exception as e:
print(f"Upload failed: {e}")
return False
def _is_archive_file(file_path: Path) -> bool:
"""
Check if a file is an archive file based on its extension.
Args:
file_path: Path to the file to check
Returns:
True if file is an archive, False otherwise
"""
return file_path.suffix.lower() in ARCHIVE_EXTENSIONS
def _get_files_to_upload(
directory: Path, verbose: bool = False, use_gitignore: bool = True
) -> list[tuple[Path, str]]:
"""
Get list of files to upload from directory.
Uses .bmignore and optionally .gitignore patterns for filtering.
Args:
directory: Directory to scan
verbose: Show detailed filtering information
use_gitignore: If False, skip .gitignore patterns (still use .bmignore)
Returns:
List of (absolute_path, relative_path) tuples
"""
files = []
ignored_files = []
# Load ignore patterns from .bmignore and optionally .gitignore
ignore_patterns = load_gitignore_patterns(directory, use_gitignore=use_gitignore)
if verbose:
gitignore_path = directory / ".gitignore"
gitignore_exists = gitignore_path.exists() and use_gitignore
print(f"\nScanning directory: {directory}")
print("Using .bmignore: Yes")
print(f"Using .gitignore: {'Yes' if gitignore_exists else 'No'}")
print(f"Ignore patterns loaded: {len(ignore_patterns)}")
if ignore_patterns and len(ignore_patterns) <= 20:
print(f"Patterns: {', '.join(sorted(ignore_patterns))}")
print()
# Walk through directory
for root, dirs, filenames in os.walk(directory):
root_path = Path(root)
# Filter directories based on ignore patterns
filtered_dirs = []
for d in dirs:
dir_path = root_path / d
if should_ignore_path(dir_path, directory, ignore_patterns):
if verbose:
rel_path = dir_path.relative_to(directory)
print(f" [IGNORED DIR] {rel_path}/")
else:
filtered_dirs.append(d)
dirs[:] = filtered_dirs
# Process files
for filename in filenames:
file_path = root_path / filename
# Calculate relative path for display/remote
rel_path = file_path.relative_to(directory)
remote_path = str(rel_path).replace("\\", "/")
# Check if file should be ignored
if should_ignore_path(file_path, directory, ignore_patterns):
ignored_files.append(remote_path)
if verbose:
print(f" [IGNORED] {remote_path}")
continue
if verbose:
print(f" [INCLUDE] {remote_path}")
files.append((file_path, remote_path))
if verbose:
print("\nSummary:")
print(f" Files to upload: {len(files)}")
print(f" Files ignored: {len(ignored_files)}")
return files
@@ -0,0 +1,124 @@
"""Upload CLI commands for basic-memory projects."""
import asyncio
from pathlib import Path
import typer
from rich.console import Console
from basic_memory.cli.app import cloud_app
from basic_memory.cli.commands.cloud.cloud_utils import (
create_cloud_project,
project_exists,
sync_project,
)
from basic_memory.cli.commands.cloud.upload import upload_path
console = Console()
@cloud_app.command("upload")
def upload(
path: Path = typer.Argument(
...,
help="Path to local file or directory to upload",
exists=True,
readable=True,
resolve_path=True,
),
project: str = typer.Option(
...,
"--project",
"-p",
help="Cloud project name (destination)",
),
create_project: bool = typer.Option(
False,
"--create-project",
"-c",
help="Create project if it doesn't exist",
),
sync: bool = typer.Option(
True,
"--sync/--no-sync",
help="Sync project after upload (default: true)",
),
verbose: bool = typer.Option(
False,
"--verbose",
"-v",
help="Show detailed information about file filtering and upload",
),
no_gitignore: bool = typer.Option(
False,
"--no-gitignore",
help="Skip .gitignore patterns (still respects .bmignore)",
),
dry_run: bool = typer.Option(
False,
"--dry-run",
help="Show what would be uploaded without actually uploading",
),
) -> None:
"""Upload local files or directories to cloud project via WebDAV.
Examples:
bm cloud upload ~/my-notes --project research
bm cloud upload notes.md --project research --create-project
bm cloud upload ~/docs --project work --no-sync
bm cloud upload ./history --project proto --verbose
bm cloud upload ./notes --project work --no-gitignore
bm cloud upload ./files --project test --dry-run
"""
async def _upload():
# Check if project exists
if not await project_exists(project):
if create_project:
console.print(f"[blue]Creating cloud project '{project}'...[/blue]")
try:
await create_cloud_project(project)
console.print(f"[green]Created project '{project}'[/green]")
except Exception as e:
console.print(f"[red]Failed to create project: {e}[/red]")
raise typer.Exit(1)
else:
console.print(
f"[red]Project '{project}' does not exist.[/red]\n"
f"[yellow]Options:[/yellow]\n"
f" 1. Create it first: bm project add {project}\n"
f" 2. Use --create-project flag to create automatically"
)
raise typer.Exit(1)
# Perform upload (or dry run)
if dry_run:
console.print(
f"[yellow]DRY RUN: Showing what would be uploaded to '{project}'[/yellow]"
)
else:
console.print(f"[blue]Uploading {path} to project '{project}'...[/blue]")
success = await upload_path(
path, project, verbose=verbose, use_gitignore=not no_gitignore, dry_run=dry_run
)
if not success:
console.print("[red]Upload failed[/red]")
raise typer.Exit(1)
if dry_run:
console.print("[yellow]DRY RUN complete - no files were uploaded[/yellow]")
else:
console.print(f"[green]Successfully uploaded to '{project}'[/green]")
# Sync project if requested (skip on dry run)
# Force full scan after bisync to ensure database is up-to-date with synced files
if sync and not dry_run:
console.print(f"[blue]Syncing project '{project}'...[/blue]")
try:
await sync_project(project, force_full=True)
except Exception as e:
console.print(f"[yellow]Warning: Sync failed: {e}[/yellow]")
console.print("[dim]Files uploaded but may not be indexed yet[/dim]")
asyncio.run(_upload())
+23 -32
View File
@@ -7,8 +7,7 @@ import typer
from rich.console import Console
from basic_memory.cli.commands.cloud import get_authenticated_headers
from basic_memory.mcp.async_client import client
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.tools.utils import call_post, call_get
from basic_memory.mcp.project_context import get_active_project
@@ -17,44 +16,36 @@ from basic_memory.schemas import ProjectInfoResponse
console = Console()
async def run_sync(project: Optional[str] = None):
"""Run sync operation via API endpoint."""
async def run_sync(project: Optional[str] = None, force_full: bool = False):
"""Run sync operation via API endpoint.
Args:
project: Optional project name
force_full: If True, force a full scan bypassing watermark optimization
"""
try:
from basic_memory.config import ConfigManager
config = ConfigManager().config
auth_headers = {}
if config.cloud_mode_enabled:
auth_headers = await get_authenticated_headers()
project_item = await get_active_project(client, project, None, headers=auth_headers)
response = await call_post(
client, f"{project_item.project_url}/project/sync", headers=auth_headers
)
data = response.json()
console.print(f"[green]✓ {data['message']}[/green]")
async with get_client() as client:
project_item = await get_active_project(client, project, None)
url = f"{project_item.project_url}/project/sync"
if force_full:
url += "?force_full=true"
response = await call_post(client, url)
data = response.json()
console.print(f"[green]{data['message']}[/green]")
except (ToolError, ValueError) as e:
console.print(f"[red]Sync failed: {e}[/red]")
console.print(f"[red]Sync failed: {e}[/red]")
raise typer.Exit(1)
async def get_project_info(project: str):
"""Run sync operation via API endpoint."""
"""Get project information via API endpoint."""
try:
from basic_memory.config import ConfigManager
config = ConfigManager().config
auth_headers = {}
if config.cloud_mode_enabled:
auth_headers = await get_authenticated_headers()
project_item = await get_active_project(client, project, None, headers=auth_headers)
response = await call_get(
client, f"{project_item.project_url}/project/info", headers=auth_headers
)
return ProjectInfoResponse.model_validate(response.json())
async with get_client() as client:
project_item = await get_active_project(client, project, None)
response = await call_get(client, f"{project_item.project_url}/project/info")
return ProjectInfoResponse.model_validate(response.json())
except (ToolError, ValueError) as e:
console.print(f"[red]Sync failed: {e}[/red]")
console.print(f"[red]Sync failed: {e}[/red]")
raise typer.Exit(1)
+3 -3
View File
@@ -37,8 +37,8 @@ def reset(
logger.info("Database reset complete")
if reindex:
# Import and run sync
from basic_memory.cli.commands.sync import sync
# Run database sync directly
from basic_memory.cli.commands.command_utils import run_sync
logger.info("Rebuilding search index from filesystem...")
sync(watch=False) # pyright: ignore
asyncio.run(run_sync(project=None))
+62 -57
View File
@@ -20,70 +20,75 @@ from loguru import logger
import threading
from basic_memory.services.initialization import initialize_file_sync
config = ConfigManager().config
@app.command()
def mcp(
transport: str = typer.Option("stdio", help="Transport type: stdio, streamable-http, or sse"),
host: str = typer.Option(
"0.0.0.0", help="Host for HTTP transports (use 0.0.0.0 to allow external connections)"
),
port: int = typer.Option(8000, help="Port for HTTP transports"),
path: str = typer.Option("/mcp", help="Path prefix for streamable-http transport"),
project: Optional[str] = typer.Option(None, help="Restrict MCP server to single project"),
): # pragma: no cover
"""Run the MCP server with configurable transport options.
if not config.cloud_mode_enabled:
This command starts an MCP server using one of three transport options:
@app.command()
def mcp(
transport: str = typer.Option(
"stdio", help="Transport type: stdio, streamable-http, or sse"
),
host: str = typer.Option(
"0.0.0.0", help="Host for HTTP transports (use 0.0.0.0 to allow external connections)"
),
port: int = typer.Option(8000, help="Port for HTTP transports"),
path: str = typer.Option("/mcp", help="Path prefix for streamable-http transport"),
project: Optional[str] = typer.Option(None, help="Restrict MCP server to single project"),
): # pragma: no cover
"""Run the MCP server with configurable transport options.
- stdio: Standard I/O (good for local usage)
- streamable-http: Recommended for web deployments (default)
- sse: Server-Sent Events (for compatibility with existing clients)
"""
This command starts an MCP server using one of three transport options:
# Validate and set project constraint if specified
if project:
config_manager = ConfigManager()
project_name, _ = config_manager.get_project(project)
if not project_name:
typer.echo(f"No project found named: {project}", err=True)
raise typer.Exit(1)
- stdio: Standard I/O (good for local usage)
- streamable-http: Recommended for web deployments (default)
- sse: Server-Sent Events (for compatibility with existing clients)
"""
# Set env var with validated project name
os.environ["BASIC_MEMORY_MCP_PROJECT"] = project_name
logger.info(f"MCP server constrained to project: {project_name}")
# Validate and set project constraint if specified
if project:
config_manager = ConfigManager()
project_name, _ = config_manager.get_project(project)
if not project_name:
typer.echo(f"No project found named: {project}", err=True)
raise typer.Exit(1)
app_config = ConfigManager().config
# Set env var with validated project name
os.environ["BASIC_MEMORY_MCP_PROJECT"] = project_name
logger.info(f"MCP server constrained to project: {project_name}")
def run_file_sync():
"""Run file sync in a separate thread with its own event loop."""
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
loop.run_until_complete(initialize_file_sync(app_config))
except Exception as e:
logger.error(f"File sync error: {e}", err=True)
finally:
loop.close()
app_config = ConfigManager().config
logger.info(f"Sync changes enabled: {app_config.sync_changes}")
if app_config.sync_changes:
# Start the sync thread
sync_thread = threading.Thread(target=run_file_sync, daemon=True)
sync_thread.start()
logger.info("Started file sync in background")
def run_file_sync():
"""Run file sync in a separate thread with its own event loop."""
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
loop.run_until_complete(initialize_file_sync(app_config))
except Exception as e:
logger.error(f"File sync error: {e}", err=True)
finally:
loop.close()
# Now run the MCP server (blocks)
logger.info(f"Starting MCP server with {transport.upper()} transport")
logger.info(f"Sync changes enabled: {app_config.sync_changes}")
if app_config.sync_changes:
# Start the sync thread
sync_thread = threading.Thread(target=run_file_sync, daemon=True)
sync_thread.start()
logger.info("Started file sync in background")
if transport == "stdio":
mcp_server.run(
transport=transport,
)
elif transport == "streamable-http" or transport == "sse":
mcp_server.run(
transport=transport,
host=host,
port=port,
path=path,
log_level="INFO",
)
# Now run the MCP server (blocks)
logger.info(f"Starting MCP server with {transport.upper()} transport")
if transport == "stdio":
mcp_server.run(
transport=transport,
)
elif transport == "streamable-http" or transport == "sse":
mcp_server.run(
transport=transport,
host=host,
port=port,
path=path,
log_level="INFO",
)
+666 -125
View File
@@ -9,31 +9,39 @@ from rich.console import Console
from rich.table import Table
from basic_memory.cli.app import app
from basic_memory.cli.commands.cloud import get_authenticated_headers
from basic_memory.cli.commands.command_utils import get_project_info
from basic_memory.config import ConfigManager
import json
from datetime import datetime
from rich.panel import Panel
from basic_memory.mcp.async_client import client
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.tools.utils import call_get
from basic_memory.schemas.project_info import ProjectList
from basic_memory.mcp.tools.utils import call_post
from basic_memory.schemas.project_info import ProjectStatusResponse
from basic_memory.mcp.tools.utils import call_delete
from basic_memory.mcp.tools.utils import call_put
from basic_memory.utils import generate_permalink
from basic_memory.utils import generate_permalink, normalize_project_path
from basic_memory.mcp.tools.utils import call_patch
# Import rclone commands for project sync
from basic_memory.cli.commands.cloud.rclone_commands import (
SyncProject,
RcloneError,
project_sync,
project_bisync,
project_check,
project_ls,
)
from basic_memory.cli.commands.cloud.bisync_commands import get_mount_info
console = Console()
# Create a project subcommand
project_app = typer.Typer(help="Manage multiple Basic Memory projects")
app.add_typer(project_app, name="project")
config = ConfigManager().config
def format_path(path: str) -> str:
"""Format a path for display, using ~ for home directory."""
@@ -46,23 +54,49 @@ def format_path(path: str) -> str:
@project_app.command("list")
def list_projects() -> None:
"""List all Basic Memory projects."""
# Use API to list projects
try:
auth_headers = {}
if config.cloud_mode_enabled:
auth_headers = asyncio.run(get_authenticated_headers())
response = asyncio.run(call_get(client, "/projects/projects", headers=auth_headers))
result = ProjectList.model_validate(response.json())
async def _list_projects():
async with get_client() as client:
response = await call_get(client, "/projects/projects")
return ProjectList.model_validate(response.json())
try:
result = asyncio.run(_list_projects())
config = ConfigManager().config
table = Table(title="Basic Memory Projects")
table.add_column("Name", style="cyan")
table.add_column("Path", style="green")
table.add_column("Default", style="magenta")
# Add Local Path column if in cloud mode
if config.cloud_mode_enabled:
table.add_column("Local Path", style="yellow", no_wrap=True, overflow="fold")
# Show Default column in local mode or if default_project_mode is enabled in cloud mode
show_default_column = not config.cloud_mode_enabled or config.default_project_mode
if show_default_column:
table.add_column("Default", style="magenta")
for project in result.projects:
is_default = "" if project.is_default else ""
table.add_row(project.name, format_path(project.path), is_default)
is_default = "[X]" if project.is_default else ""
normalized_path = normalize_project_path(project.path)
# Build row based on mode
row = [project.name, format_path(normalized_path)]
# Add local path if in cloud mode
if config.cloud_mode_enabled:
local_path = ""
if project.name in config.cloud_projects:
local_path = config.cloud_projects[project.name].local_path or ""
local_path = format_path(local_path)
row.append(local_path)
# Add default indicator if showing default column
if show_default_column:
row.append(is_default)
table.add_row(*row)
console.print(table)
except Exception as e:
@@ -70,157 +104,664 @@ def list_projects() -> None:
raise typer.Exit(1)
if config.cloud_mode_enabled:
@project_app.command("add")
def add_project(
name: str = typer.Argument(..., help="Name of the project"),
path: str = typer.Argument(
None, help="Path to the project directory (required for local mode)"
),
local_path: str = typer.Option(
None, "--local-path", help="Local sync path for cloud mode (optional)"
),
set_default: bool = typer.Option(False, "--default", help="Set as default project"),
) -> None:
"""Add a new project.
@project_app.command("add")
def add_project_cloud(
name: str = typer.Argument(..., help="Name of the project"),
set_default: bool = typer.Option(False, "--default", help="Set as default project"),
) -> None:
"""Add a new project to Basic Memory Cloud"""
Cloud mode examples:\n
bm project add research # No local sync\n
bm project add research --local-path ~/docs # With local sync\n
try:
auth_headers = asyncio.run(get_authenticated_headers())
Local mode example:\n
bm project add research ~/Documents/research
"""
config = ConfigManager().config
data = {"name": name, "path": generate_permalink(name), "set_default": set_default}
# Resolve local sync path early (needed for both cloud and local mode)
local_sync_path: str | None = None
if local_path:
local_sync_path = Path(os.path.abspath(os.path.expanduser(local_path))).as_posix()
response = asyncio.run(
call_post(client, "/projects/projects", json=data, headers=auth_headers)
)
result = ProjectStatusResponse.model_validate(response.json())
if config.cloud_mode_enabled:
# Cloud mode: path auto-generated from name, local sync is optional
console.print(f"[green]{result.message}[/green]")
except Exception as e:
console.print(f"[red]Error adding project: {str(e)}[/red]")
async def _add_project():
async with get_client() as client:
data = {
"name": name,
"path": generate_permalink(name),
"local_sync_path": local_sync_path,
"set_default": set_default,
}
response = await call_post(client, "/projects/projects", json=data)
return ProjectStatusResponse.model_validate(response.json())
else:
# Local mode: path is required
if path is None:
console.print("[red]Error: path argument is required in local mode[/red]")
raise typer.Exit(1)
# Display usage hint
console.print("\nTo use this project:")
console.print(f" basic-memory --project={name} <command>")
else:
@project_app.command("add")
def add_project(
name: str = typer.Argument(..., help="Name of the project"),
path: str = typer.Argument(..., help="Path to the project directory"),
set_default: bool = typer.Option(False, "--default", help="Set as default project"),
) -> None:
"""Add a new project."""
# Resolve to absolute path
resolved_path = Path(os.path.abspath(os.path.expanduser(path))).as_posix()
try:
data = {"name": name, "path": resolved_path, "set_default": set_default}
async def _add_project():
async with get_client() as client:
data = {"name": name, "path": resolved_path, "set_default": set_default}
response = await call_post(client, "/projects/projects", json=data)
return ProjectStatusResponse.model_validate(response.json())
response = asyncio.run(call_post(client, "/projects/projects", json=data))
result = ProjectStatusResponse.model_validate(response.json())
try:
result = asyncio.run(_add_project())
console.print(f"[green]{result.message}[/green]")
console.print(f"[green]{result.message}[/green]")
except Exception as e:
console.print(f"[red]Error adding project: {str(e)}[/red]")
raise typer.Exit(1)
# Save local sync path to config if in cloud mode
if config.cloud_mode_enabled and local_sync_path:
from basic_memory.config import CloudProjectConfig
# Display usage hint
console.print("\nTo use this project:")
console.print(f" basic-memory --project={name} <command>")
# Create local directory if it doesn't exist
local_dir = Path(local_sync_path)
local_dir.mkdir(parents=True, exist_ok=True)
# Update config with sync path
config.cloud_projects[name] = CloudProjectConfig(
local_path=local_sync_path,
last_sync=None,
bisync_initialized=False,
)
ConfigManager().save_config(config)
console.print(f"\n[green]Local sync path configured: {local_sync_path}[/green]")
console.print("\nNext steps:")
console.print(f" 1. Preview: bm project bisync --name {name} --resync --dry-run")
console.print(f" 2. Sync: bm project bisync --name {name} --resync")
except Exception as e:
console.print(f"[red]Error adding project: {str(e)}[/red]")
raise typer.Exit(1)
@project_app.command("sync-setup")
def setup_project_sync(
name: str = typer.Argument(..., help="Project name"),
local_path: str = typer.Argument(..., help="Local sync directory"),
) -> None:
"""Configure local sync for an existing cloud project.
Example:
bm project sync-setup research ~/Documents/research
"""
config_manager = ConfigManager()
config = config_manager.config
if not config.cloud_mode_enabled:
console.print("[red]Error: sync-setup only available in cloud mode[/red]")
raise typer.Exit(1)
async def _verify_project_exists():
"""Verify the project exists on cloud by listing all projects."""
async with get_client() as client:
response = await call_get(client, "/projects/projects")
project_list = response.json()
project_names = [p["name"] for p in project_list["projects"]]
if name not in project_names:
raise ValueError(f"Project '{name}' not found on cloud")
return True
try:
# Verify project exists on cloud
asyncio.run(_verify_project_exists())
# Resolve and create local path
resolved_path = Path(os.path.abspath(os.path.expanduser(local_path)))
resolved_path.mkdir(parents=True, exist_ok=True)
# Update local config with sync path
from basic_memory.config import CloudProjectConfig
config.cloud_projects[name] = CloudProjectConfig(
local_path=resolved_path.as_posix(),
last_sync=None,
bisync_initialized=False,
)
config_manager.save_config(config)
console.print(f"[green]Sync configured for project '{name}'[/green]")
console.print(f"\nLocal sync path: {resolved_path}")
console.print("\nNext steps:")
console.print(f" 1. Preview: bm project bisync --name {name} --resync --dry-run")
console.print(f" 2. Sync: bm project bisync --name {name} --resync")
except Exception as e:
console.print(f"[red]Error configuring sync: {str(e)}[/red]")
raise typer.Exit(1)
@project_app.command("remove")
def remove_project(
name: str = typer.Argument(..., help="Name of the project to remove"),
delete_notes: bool = typer.Option(
False, "--delete-notes", help="Delete project files from disk"
),
) -> None:
"""Remove a project."""
async def _remove_project():
async with get_client() as client:
project_permalink = generate_permalink(name)
response = await call_delete(
client, f"/projects/{project_permalink}?delete_notes={delete_notes}"
)
return ProjectStatusResponse.model_validate(response.json())
try:
auth_headers = {}
if config.cloud_mode_enabled:
auth_headers = asyncio.run(get_authenticated_headers())
# Get config to check for local sync path and bisync state
config = ConfigManager().config
local_path = None
has_bisync_state = False
project_permalink = generate_permalink(name)
response = asyncio.run(
call_delete(client, f"/projects/{project_permalink}", headers=auth_headers)
)
result = ProjectStatusResponse.model_validate(response.json())
if config.cloud_mode_enabled and name in config.cloud_projects:
local_path = config.cloud_projects[name].local_path
# Check for bisync state
from basic_memory.cli.commands.cloud.rclone_commands import get_project_bisync_state
bisync_state_path = get_project_bisync_state(name)
has_bisync_state = bisync_state_path.exists()
# Remove project from cloud/API
result = asyncio.run(_remove_project())
console.print(f"[green]{result.message}[/green]")
# Clean up local sync directory if it exists and delete_notes is True
if delete_notes and local_path:
local_dir = Path(local_path)
if local_dir.exists():
import shutil
shutil.rmtree(local_dir)
console.print(f"[green]Removed local sync directory: {local_path}[/green]")
# Clean up bisync state if it exists
if has_bisync_state:
from basic_memory.cli.commands.cloud.rclone_commands import get_project_bisync_state
import shutil
bisync_state_path = get_project_bisync_state(name)
if bisync_state_path.exists():
shutil.rmtree(bisync_state_path)
console.print("[green]Removed bisync state[/green]")
# Clean up cloud_projects config entry
if config.cloud_mode_enabled and name in config.cloud_projects:
del config.cloud_projects[name]
ConfigManager().save_config(config)
# Show informative message if files were not deleted
if not delete_notes:
if local_path:
console.print(f"[yellow]Note: Local files remain at {local_path}[/yellow]")
except Exception as e:
console.print(f"[red]Error removing project: {str(e)}[/red]")
raise typer.Exit(1)
# Show this message regardless of method used
console.print("[yellow]Note: The project files have not been deleted from disk.[/yellow]")
@project_app.command("default")
def set_default_project(
name: str = typer.Argument(..., help="Name of the project to set as CLI default"),
) -> None:
"""Set the default project when 'config.default_project_mode' is set.
if not config.cloud_mode_enabled:
Note: This command is only available in local mode.
"""
config = ConfigManager().config
@project_app.command("default")
def set_default_project(
name: str = typer.Argument(..., help="Name of the project to set as CLI default"),
) -> None:
"""Set the default project when 'config.default_project_mode' is set."""
try:
if config.cloud_mode_enabled:
console.print("[red]Error: 'default' command is not available in cloud mode[/red]")
raise typer.Exit(1)
async def _set_default():
async with get_client() as client:
project_permalink = generate_permalink(name)
response = asyncio.run(call_put(client, f"/projects/{project_permalink}/default"))
result = ProjectStatusResponse.model_validate(response.json())
response = await call_put(client, f"/projects/{project_permalink}/default")
return ProjectStatusResponse.model_validate(response.json())
console.print(f"[green]{result.message}[/green]")
except Exception as e:
console.print(f"[red]Error setting default project: {str(e)}[/red]")
raise typer.Exit(1)
try:
result = asyncio.run(_set_default())
console.print(f"[green]{result.message}[/green]")
except Exception as e:
console.print(f"[red]Error setting default project: {str(e)}[/red]")
raise typer.Exit(1)
@project_app.command("sync-config")
def synchronize_projects() -> None:
"""Synchronize project config between configuration file and database."""
# Call the API to synchronize projects
try:
response = asyncio.run(call_post(client, "/projects/config/sync"))
result = ProjectStatusResponse.model_validate(response.json())
@project_app.command("sync-config")
def synchronize_projects() -> None:
"""Synchronize project config between configuration file and database.
console.print(f"[green]{result.message}[/green]")
except Exception as e: # pragma: no cover
console.print(f"[red]Error synchronizing projects: {str(e)}[/red]")
raise typer.Exit(1)
Note: This command is only available in local mode.
"""
config = ConfigManager().config
@project_app.command("move")
def move_project(
name: str = typer.Argument(..., help="Name of the project to move"),
new_path: str = typer.Argument(..., help="New absolute path for the project"),
) -> None:
"""Move a project to a new location."""
# Resolve to absolute path
resolved_path = Path(os.path.abspath(os.path.expanduser(new_path))).as_posix()
if config.cloud_mode_enabled:
console.print("[red]Error: 'sync-config' command is not available in cloud mode[/red]")
raise typer.Exit(1)
try:
async def _sync_config():
async with get_client() as client:
response = await call_post(client, "/projects/config/sync")
return ProjectStatusResponse.model_validate(response.json())
try:
result = asyncio.run(_sync_config())
console.print(f"[green]{result.message}[/green]")
except Exception as e: # pragma: no cover
console.print(f"[red]Error synchronizing projects: {str(e)}[/red]")
raise typer.Exit(1)
@project_app.command("move")
def move_project(
name: str = typer.Argument(..., help="Name of the project to move"),
new_path: str = typer.Argument(..., help="New absolute path for the project"),
) -> None:
"""Move a project to a new location.
Note: This command is only available in local mode.
"""
config = ConfigManager().config
if config.cloud_mode_enabled:
console.print("[red]Error: 'move' command is not available in cloud mode[/red]")
raise typer.Exit(1)
# Resolve to absolute path
resolved_path = Path(os.path.abspath(os.path.expanduser(new_path))).as_posix()
async def _move_project():
async with get_client() as client:
data = {"path": resolved_path}
project_permalink = generate_permalink(name)
# TODO fix route to use ProjectPathDep
response = asyncio.run(
call_patch(client, f"/{name}/project/{project_permalink}", json=data)
response = await call_patch(client, f"/{name}/project/{project_permalink}", json=data)
return ProjectStatusResponse.model_validate(response.json())
try:
result = asyncio.run(_move_project())
console.print(f"[green]{result.message}[/green]")
# Show important file movement reminder
console.print() # Empty line for spacing
console.print(
Panel(
"[bold red]IMPORTANT:[/bold red] Project configuration updated successfully.\n\n"
"[yellow]You must manually move your project files from the old location to:[/yellow]\n"
f"[cyan]{resolved_path}[/cyan]\n\n"
"[dim]Basic Memory has only updated the configuration - your files remain in their original location.[/dim]",
title="Manual File Movement Required",
border_style="yellow",
expand=False,
)
result = ProjectStatusResponse.model_validate(response.json())
)
console.print(f"[green]{result.message}[/green]")
except Exception as e:
console.print(f"[red]Error moving project: {str(e)}[/red]")
raise typer.Exit(1)
# Show important file movement reminder
console.print() # Empty line for spacing
console.print(
Panel(
"[bold red]IMPORTANT:[/bold red] Project configuration updated successfully.\n\n"
"[yellow]You must manually move your project files from the old location to:[/yellow]\n"
f"[cyan]{resolved_path}[/cyan]\n\n"
"[dim]Basic Memory has only updated the configuration - your files remain in their original location.[/dim]",
title="⚠️ Manual File Movement Required",
border_style="yellow",
expand=False,
)
)
except Exception as e:
console.print(f"[red]Error moving project: {str(e)}[/red]")
@project_app.command("sync")
def sync_project_command(
name: str = typer.Option(..., "--name", help="Project name to sync"),
dry_run: bool = typer.Option(False, "--dry-run", help="Preview changes without syncing"),
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed output"),
) -> None:
"""One-way sync: local -> cloud (make cloud identical to local).
Example:
bm project sync --name research
bm project sync --name research --dry-run
"""
config = ConfigManager().config
if not config.cloud_mode_enabled:
console.print("[red]Error: sync only available in cloud mode[/red]")
raise typer.Exit(1)
try:
# Get tenant info for bucket name
tenant_info = asyncio.run(get_mount_info())
bucket_name = tenant_info.bucket_name
# Get project info
async def _get_project():
async with get_client() as client:
response = await call_get(client, "/projects/projects")
projects_list = ProjectList.model_validate(response.json())
for proj in projects_list.projects:
if generate_permalink(proj.name) == generate_permalink(name):
return proj
return None
project_data = asyncio.run(_get_project())
if not project_data:
console.print(f"[red]Error: Project '{name}' not found[/red]")
raise typer.Exit(1)
# Get local_sync_path from cloud_projects config
local_sync_path = None
if name in config.cloud_projects:
local_sync_path = config.cloud_projects[name].local_path
if not local_sync_path:
console.print(f"[red]Error: Project '{name}' has no local_sync_path configured[/red]")
console.print(f"\nConfigure sync with: bm project sync-setup {name} ~/path/to/local")
raise typer.Exit(1)
# Create SyncProject
sync_project = SyncProject(
name=project_data.name,
path=normalize_project_path(project_data.path),
local_sync_path=local_sync_path,
)
# Run sync
console.print(f"[blue]Syncing {name} (local -> cloud)...[/blue]")
success = project_sync(sync_project, bucket_name, dry_run=dry_run, verbose=verbose)
if success:
console.print(f"[green]{name} synced successfully[/green]")
# Trigger database sync if not a dry run
if not dry_run:
async def _trigger_db_sync():
async with get_client() as client:
permalink = generate_permalink(name)
response = await call_post(
client, f"/{permalink}/project/sync?force_full=true", json={}
)
return response.json()
try:
result = asyncio.run(_trigger_db_sync())
console.print(f"[dim]Database sync initiated: {result.get('message')}[/dim]")
except Exception as e:
console.print(f"[yellow]Warning: Could not trigger database sync: {e}[/yellow]")
else:
console.print(f"[red]{name} sync failed[/red]")
raise typer.Exit(1)
except RcloneError as e:
console.print(f"[red]Sync error: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
@project_app.command("bisync")
def bisync_project_command(
name: str = typer.Option(..., "--name", help="Project name to bisync"),
dry_run: bool = typer.Option(False, "--dry-run", help="Preview changes without syncing"),
resync: bool = typer.Option(False, "--resync", help="Force new baseline"),
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed output"),
) -> None:
"""Two-way sync: local <-> cloud (bidirectional sync).
Examples:
bm project bisync --name research --resync # First time
bm project bisync --name research # Subsequent syncs
bm project bisync --name research --dry-run # Preview changes
"""
config = ConfigManager().config
if not config.cloud_mode_enabled:
console.print("[red]Error: bisync only available in cloud mode[/red]")
raise typer.Exit(1)
try:
# Get tenant info for bucket name
tenant_info = asyncio.run(get_mount_info())
bucket_name = tenant_info.bucket_name
# Get project info
async def _get_project():
async with get_client() as client:
response = await call_get(client, "/projects/projects")
projects_list = ProjectList.model_validate(response.json())
for proj in projects_list.projects:
if generate_permalink(proj.name) == generate_permalink(name):
return proj
return None
project_data = asyncio.run(_get_project())
if not project_data:
console.print(f"[red]Error: Project '{name}' not found[/red]")
raise typer.Exit(1)
# Get local_sync_path from cloud_projects config
local_sync_path = None
if name in config.cloud_projects:
local_sync_path = config.cloud_projects[name].local_path
if not local_sync_path:
console.print(f"[red]Error: Project '{name}' has no local_sync_path configured[/red]")
console.print(f"\nConfigure sync with: bm project sync-setup {name} ~/path/to/local")
raise typer.Exit(1)
# Create SyncProject
sync_project = SyncProject(
name=project_data.name,
path=normalize_project_path(project_data.path),
local_sync_path=local_sync_path,
)
# Run bisync
console.print(f"[blue]Bisync {name} (local <-> cloud)...[/blue]")
success = project_bisync(
sync_project, bucket_name, dry_run=dry_run, resync=resync, verbose=verbose
)
if success:
console.print(f"[green]{name} bisync completed successfully[/green]")
# Update config
config.cloud_projects[name].last_sync = datetime.now()
config.cloud_projects[name].bisync_initialized = True
ConfigManager().save_config(config)
# Trigger database sync if not a dry run
if not dry_run:
async def _trigger_db_sync():
async with get_client() as client:
permalink = generate_permalink(name)
response = await call_post(
client, f"/{permalink}/project/sync?force_full=true", json={}
)
return response.json()
try:
result = asyncio.run(_trigger_db_sync())
console.print(f"[dim]Database sync initiated: {result.get('message')}[/dim]")
except Exception as e:
console.print(f"[yellow]Warning: Could not trigger database sync: {e}[/yellow]")
else:
console.print(f"[red]{name} bisync failed[/red]")
raise typer.Exit(1)
except RcloneError as e:
console.print(f"[red]Bisync error: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
@project_app.command("check")
def check_project_command(
name: str = typer.Option(..., "--name", help="Project name to check"),
one_way: bool = typer.Option(False, "--one-way", help="Check one direction only (faster)"),
) -> None:
"""Verify file integrity between local and cloud.
Example:
bm project check --name research
"""
config = ConfigManager().config
if not config.cloud_mode_enabled:
console.print("[red]Error: check only available in cloud mode[/red]")
raise typer.Exit(1)
try:
# Get tenant info for bucket name
tenant_info = asyncio.run(get_mount_info())
bucket_name = tenant_info.bucket_name
# Get project info
async def _get_project():
async with get_client() as client:
response = await call_get(client, "/projects/projects")
projects_list = ProjectList.model_validate(response.json())
for proj in projects_list.projects:
if generate_permalink(proj.name) == generate_permalink(name):
return proj
return None
project_data = asyncio.run(_get_project())
if not project_data:
console.print(f"[red]Error: Project '{name}' not found[/red]")
raise typer.Exit(1)
# Get local_sync_path from cloud_projects config
local_sync_path = None
if name in config.cloud_projects:
local_sync_path = config.cloud_projects[name].local_path
if not local_sync_path:
console.print(f"[red]Error: Project '{name}' has no local_sync_path configured[/red]")
console.print(f"\nConfigure sync with: bm project sync-setup {name} ~/path/to/local")
raise typer.Exit(1)
# Create SyncProject
sync_project = SyncProject(
name=project_data.name,
path=normalize_project_path(project_data.path),
local_sync_path=local_sync_path,
)
# Run check
console.print(f"[blue]Checking {name} integrity...[/blue]")
match = project_check(sync_project, bucket_name, one_way=one_way)
if match:
console.print(f"[green]{name} files match[/green]")
else:
console.print(f"[yellow]!{name} has differences[/yellow]")
except RcloneError as e:
console.print(f"[red]Check error: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
@project_app.command("bisync-reset")
def bisync_reset(
name: str = typer.Argument(..., help="Project name to reset bisync state for"),
) -> None:
"""Clear bisync state for a project.
This removes the bisync metadata files, forcing a fresh --resync on next bisync.
Useful when bisync gets into an inconsistent state or when remote path changes.
"""
from basic_memory.cli.commands.cloud.rclone_commands import get_project_bisync_state
import shutil
try:
state_path = get_project_bisync_state(name)
if not state_path.exists():
console.print(f"[yellow]No bisync state found for project '{name}'[/yellow]")
return
# Remove the entire state directory
shutil.rmtree(state_path)
console.print(f"[green]Cleared bisync state for project '{name}'[/green]")
console.print("\nNext steps:")
console.print(f" 1. Preview: bm project bisync --name {name} --resync --dry-run")
console.print(f" 2. Sync: bm project bisync --name {name} --resync")
except Exception as e:
console.print(f"[red]Error clearing bisync state: {str(e)}[/red]")
raise typer.Exit(1)
@project_app.command("ls")
def ls_project_command(
name: str = typer.Option(..., "--name", help="Project name to list files from"),
path: str = typer.Argument(None, help="Path within project (optional)"),
) -> None:
"""List files in remote project.
Examples:
bm project ls --name research
bm project ls --name research subfolder
"""
config = ConfigManager().config
if not config.cloud_mode_enabled:
console.print("[red]Error: ls only available in cloud mode[/red]")
raise typer.Exit(1)
try:
# Get tenant info for bucket name
tenant_info = asyncio.run(get_mount_info())
bucket_name = tenant_info.bucket_name
# Get project info
async def _get_project():
async with get_client() as client:
response = await call_get(client, "/projects/projects")
projects_list = ProjectList.model_validate(response.json())
for proj in projects_list.projects:
if generate_permalink(proj.name) == generate_permalink(name):
return proj
return None
project_data = asyncio.run(_get_project())
if not project_data:
console.print(f"[red]Error: Project '{name}' not found[/red]")
raise typer.Exit(1)
# Create SyncProject (local_sync_path not needed for ls)
sync_project = SyncProject(
name=project_data.name,
path=normalize_project_path(project_data.path),
)
# List files
files = project_ls(sync_project, bucket_name, path=path)
if files:
console.print(f"\n[bold]Files in {name}" + (f"/{path}" if path else "") + ":[/bold]")
for file in files:
console.print(f" {file}")
console.print(f"\n[dim]Total: {len(files)} files[/dim]")
else:
console.print(
f"[yellow]No files found in {name}" + (f"/{path}" if path else "") + "[/yellow]"
)
except Exception as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
@project_app.command("info")
def display_project_info(
@@ -243,13 +784,13 @@ def display_project_info(
f"[bold]Project:[/bold] {info.project_name}\n"
f"[bold]Path:[/bold] {info.project_path}\n"
f"[bold]Default Project:[/bold] {info.default_project}\n",
title="📊 Basic Memory Project Info",
title="Basic Memory Project Info",
expand=False,
)
)
# Statistics section
stats_table = Table(title="📈 Statistics")
stats_table = Table(title="Statistics")
stats_table.add_column("Metric", style="cyan")
stats_table.add_column("Count", style="green")
@@ -265,7 +806,7 @@ def display_project_info(
# Entity types
if info.statistics.entity_types:
entity_types_table = Table(title="📑 Entity Types")
entity_types_table = Table(title="Entity Types")
entity_types_table.add_column("Type", style="blue")
entity_types_table.add_column("Count", style="green")
@@ -276,7 +817,7 @@ def display_project_info(
# Most connected entities
if info.statistics.most_connected_entities: # pragma: no cover
connected_table = Table(title="🔗 Most Connected Entities")
connected_table = Table(title="Most Connected Entities")
connected_table.add_column("Title", style="blue")
connected_table.add_column("Permalink", style="cyan")
connected_table.add_column("Relations", style="green")
@@ -290,7 +831,7 @@ def display_project_info(
# Recent activity
if info.activity.recently_updated: # pragma: no cover
recent_table = Table(title="🕒 Recent Activity")
recent_table = Table(title="Recent Activity")
recent_table.add_column("Title", style="blue")
recent_table.add_column("Type", style="cyan")
recent_table.add_column("Last Updated", style="green")
@@ -310,7 +851,7 @@ def display_project_info(
console.print(recent_table)
# Available projects
projects_table = Table(title="📁 Available Projects")
projects_table = Table(title="Available Projects")
projects_table.add_column("Name", style="blue")
projects_table.add_column("Path", style="cyan")
projects_table.add_column("Default", style="green")
@@ -318,7 +859,7 @@ def display_project_info(
for name, proj_info in info.available_projects.items():
is_default = name == info.default_project
project_path = proj_info["path"]
projects_table.add_row(name, project_path, "" if is_default else "")
projects_table.add_row(name, project_path, "[X]" if is_default else "")
console.print(projects_table)
+23 -17
View File
@@ -12,8 +12,7 @@ from rich.panel import Panel
from rich.tree import Tree
from basic_memory.cli.app import app
from basic_memory.cli.commands.cloud import get_authenticated_headers
from basic_memory.mcp.async_client import client
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.tools.utils import call_post
from basic_memory.schemas import SyncReportResponse
from basic_memory.mcp.project_context import get_active_project
@@ -95,7 +94,7 @@ def display_changes(
"""Display changes using Rich for better visualization."""
tree = Tree(f"{project_name}: {title}")
if changes.total == 0:
if changes.total == 0 and not changes.skipped_files:
tree.add("No changes")
console.print(Panel(tree, expand=False))
return
@@ -115,6 +114,13 @@ def display_changes(
if changes.deleted:
del_branch = tree.add("[red]Deleted[/red]")
add_files_to_tree(del_branch, changes.deleted, "red")
if changes.skipped_files:
skip_branch = tree.add("[red]! Skipped (Circuit Breaker)[/red]")
for skipped in sorted(changes.skipped_files, key=lambda x: x.path):
skip_branch.add(
f"[red]{skipped.path}[/red] "
f"(failures: {skipped.failure_count}, reason: {skipped.reason})"
)
else:
# Show directory summaries
by_dir = group_changes_by_directory(changes)
@@ -123,6 +129,14 @@ def display_changes(
if summary: # Only show directories with changes
tree.add(f"[bold]{dir_name}/[/bold] {summary}")
# Show skipped files summary in non-verbose mode
if changes.skipped_files:
skip_count = len(changes.skipped_files)
tree.add(
f"[red]! {skip_count} file{'s' if skip_count != 1 else ''} "
f"skipped due to repeated failures[/red]"
)
console.print(Panel(tree, expand=False))
@@ -130,23 +144,15 @@ async def run_status(project: Optional[str] = None, verbose: bool = False): # p
"""Check sync status of files vs database."""
try:
from basic_memory.config import ConfigManager
async with get_client() as client:
project_item = await get_active_project(client, project, None)
response = await call_post(client, f"{project_item.project_url}/project/status")
sync_report = SyncReportResponse.model_validate(response.json())
config = ConfigManager().config
auth_headers = {}
if config.cloud_mode_enabled:
auth_headers = await get_authenticated_headers()
project_item = await get_active_project(client, project, None)
response = await call_post(
client, f"{project_item.project_url}/project/status", headers=auth_headers
)
sync_report = SyncReportResponse.model_validate(response.json())
display_changes(project_item.name, "Status", sync_report, verbose)
display_changes(project_item.name, "Status", sync_report, verbose)
except (ValueError, ToolError) as e:
console.print(f"[red]Error: {e}[/red]")
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
-59
View File
@@ -1,59 +0,0 @@
"""Command module for basic-memory sync operations."""
import asyncio
from typing import Annotated, Optional
import typer
from basic_memory.cli.app import app
from basic_memory.cli.commands.command_utils import run_sync
from basic_memory.config import ConfigManager
@app.command()
def sync(
project: Annotated[
Optional[str],
typer.Option(help="The project name."),
] = None,
watch: Annotated[
bool,
typer.Option("--watch", help="Run continuous sync (cloud mode only)"),
] = False,
interval: Annotated[
int,
typer.Option("--interval", help="Sync interval in seconds for watch mode (default: 60)"),
] = 60,
) -> None:
"""Sync knowledge files with the database.
In local mode: Scans filesystem and updates database.
In cloud mode: Runs bidirectional file sync (bisync) then updates database.
Examples:
bm sync # One-time sync
bm sync --watch # Continuous sync every 60s
bm sync --watch --interval 30 # Continuous sync every 30s
"""
config = ConfigManager().config
if config.cloud_mode_enabled:
# Cloud mode: run bisync which includes database sync
from basic_memory.cli.commands.cloud.bisync_commands import run_bisync, run_bisync_watch
try:
if watch:
run_bisync_watch(interval_seconds=interval)
else:
run_bisync()
except Exception:
raise typer.Exit(1)
else:
# Local mode: just database sync
if watch:
typer.echo(
"Error: --watch is only available in cloud mode. Run 'bm cloud login' first."
)
raise typer.Exit(1)
asyncio.run(run_sync(project))
-1
View File
@@ -13,7 +13,6 @@ from basic_memory.cli.commands import ( # noqa: F401 # pragma: no cover
mcp,
project,
status,
sync,
tool,
)
+88 -20
View File
@@ -3,11 +3,12 @@
import json
import os
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from typing import Any, Dict, Literal, Optional, List, Tuple
from loguru import logger
from pydantic import Field, field_validator
from pydantic import BaseModel, Field, field_validator
from pydantic_settings import BaseSettings, SettingsConfigDict
import basic_memory
@@ -39,6 +40,22 @@ class ProjectConfig:
return f"/{generate_permalink(self.name)}"
class CloudProjectConfig(BaseModel):
"""Sync configuration for a cloud project.
This tracks the local working directory and sync state for a project
that is synced with Basic Memory Cloud.
"""
local_path: str = Field(description="Local working directory path for this cloud project")
last_sync: Optional[datetime] = Field(
default=None, description="Timestamp of last successful sync operation"
)
bisync_initialized: bool = Field(
default=False, description="Whether rclone bisync baseline has been established"
)
class BasicMemoryConfig(BaseSettings):
"""Pydantic model for Basic Memory global configuration."""
@@ -84,7 +101,13 @@ class BasicMemoryConfig(BaseSettings):
sync_thread_pool_size: int = Field(
default=4,
description="Size of thread pool for file I/O operations in sync service",
description="Size of thread pool for file I/O operations in sync service. Default of 4 is optimized for cloud deployments with 1-2GB RAM.",
gt=0,
)
sync_max_concurrent_files: int = Field(
default=10,
description="Maximum number of files to process concurrently during sync. Limits memory usage on large projects (2000+ files). Lower values reduce memory consumption.",
gt=0,
)
@@ -103,10 +126,10 @@ class BasicMemoryConfig(BaseSettings):
description="Skip expensive initialization synchronization. Useful for cloud/stateless deployments where project reconciliation is not needed.",
)
# API connection configuration
api_url: Optional[str] = Field(
# Project path constraints
project_root: Optional[str] = Field(
default=None,
description="URL of remote Basic Memory API. If set, MCP will connect to this API instead of using local ASGI transport.",
description="If set, all projects must be created underneath this directory. Paths will be sanitized and constrained to this root. If not set, projects can be created anywhere (default behavior).",
)
# Cloud configuration
@@ -132,6 +155,11 @@ class BasicMemoryConfig(BaseSettings):
description="Enable cloud mode - all requests go to cloud instead of local (config file value)",
)
cloud_projects: Dict[str, CloudProjectConfig] = Field(
default_factory=dict,
description="Cloud project sync configuration mapping project names to their local paths and sync state",
)
@property
def cloud_mode_enabled(self) -> bool:
"""Check if cloud mode is enabled.
@@ -148,14 +176,6 @@ class BasicMemoryConfig(BaseSettings):
# Fall back to config file value
return self.cloud_mode
bisync_config: Dict[str, Any] = Field(
default_factory=lambda: {
"profile": "balanced",
"sync_dir": str(Path.home() / "basic-memory-cloud-sync"),
},
description="Bisync configuration for cloud sync",
)
model_config = SettingsConfigDict(
env_prefix="BASIC_MEMORY_",
extra="ignore",
@@ -232,6 +252,10 @@ class BasicMemoryConfig(BaseSettings):
return Path.home() / DATA_DIR_NAME
# Module-level cache for configuration
_CONFIG_CACHE: Optional[BasicMemoryConfig] = None
class ConfigManager:
"""Manages Basic Memory configuration."""
@@ -241,7 +265,12 @@ class ConfigManager:
if isinstance(home, str):
home = Path(home)
self.config_dir = home / DATA_DIR_NAME
# Allow override via environment variable
if config_dir := os.getenv("BASIC_MEMORY_CONFIG_DIR"):
self.config_dir = Path(config_dir)
else:
self.config_dir = home / DATA_DIR_NAME
self.config_file = self.config_dir / CONFIG_FILE_NAME
# Ensure config directory exists
@@ -253,12 +282,45 @@ class ConfigManager:
return self.load_config()
def load_config(self) -> BasicMemoryConfig:
"""Load configuration from file or create default."""
"""Load configuration from file or create default.
Environment variables take precedence over file config values,
following Pydantic Settings best practices.
Uses module-level cache for performance across ConfigManager instances.
"""
global _CONFIG_CACHE
# Return cached config if available
if _CONFIG_CACHE is not None:
return _CONFIG_CACHE
if self.config_file.exists():
try:
data = json.loads(self.config_file.read_text(encoding="utf-8"))
return BasicMemoryConfig(**data)
file_data = json.loads(self.config_file.read_text(encoding="utf-8"))
# First, create config from environment variables (Pydantic will read them)
# Then overlay with file data for fields that aren't set via env vars
# This ensures env vars take precedence
# Get env-based config fields that are actually set
env_config = BasicMemoryConfig()
env_dict = env_config.model_dump()
# Merge: file data as base, but only use it for fields not set by env
# We detect env-set fields by comparing to default values
merged_data = file_data.copy()
# For fields that have env var overrides, use those instead of file values
# The env_prefix is "BASIC_MEMORY_" so we check those
for field_name in BasicMemoryConfig.model_fields.keys():
env_var_name = f"BASIC_MEMORY_{field_name.upper()}"
if env_var_name in os.environ:
# Environment variable is set, use it
merged_data[field_name] = env_dict[field_name]
_CONFIG_CACHE = BasicMemoryConfig(**merged_data)
return _CONFIG_CACHE
except Exception as e: # pragma: no cover
logger.exception(f"Failed to load config: {e}")
raise e
@@ -268,8 +330,11 @@ class ConfigManager:
return config
def save_config(self, config: BasicMemoryConfig) -> None:
"""Save configuration to file."""
"""Save configuration to file and invalidate cache."""
global _CONFIG_CACHE
save_basic_memory_config(self.config_file, config)
# Invalidate cache so next load_config() reads fresh data
_CONFIG_CACHE = None
@property
def projects(self) -> Dict[str, str]:
@@ -309,7 +374,8 @@ class ConfigManager:
if project_name == config.default_project: # pragma: no cover
raise ValueError(f"Cannot remove the default project '{name}'")
del config.projects[name]
# Use the found project_name (which may differ from input name due to permalink matching)
del config.projects[project_name]
self.save_config(config)
def set_default_project(self, name: str) -> None:
@@ -375,7 +441,9 @@ def get_project_config(project_name: Optional[str] = None) -> ProjectConfig:
def save_basic_memory_config(file_path: Path, config: BasicMemoryConfig) -> None:
"""Save configuration to file."""
try:
file_path.write_text(json.dumps(config.model_dump(), indent=2))
# Use model_dump with mode='json' to serialize datetime objects properly
config_dict = config.model_dump(mode="json")
file_path.write_text(json.dumps(config_dict, indent=2))
except Exception as e: # pragma: no cover
logger.error(f"Failed to save config: {e}")
+9 -5
View File
@@ -33,6 +33,7 @@ from basic_memory.services.file_service import FileService
from basic_memory.services.link_resolver import LinkResolver
from basic_memory.services.search_service import SearchService
from basic_memory.sync import SyncService
from basic_memory.utils import generate_permalink
def get_app_config() -> BasicMemoryConfig: # pragma: no cover
@@ -61,8 +62,9 @@ async def get_project_config(
Raises:
HTTPException: If project is not found
"""
project_obj = await project_repository.get_by_permalink(str(project))
# Convert project name to permalink for lookup
project_permalink = generate_permalink(str(project))
project_obj = await project_repository.get_by_permalink(project_permalink)
if project_obj:
return ProjectConfig(name=project_obj.name, home=pathlib.Path(project_obj.path))
@@ -147,9 +149,9 @@ async def get_project_id(
Raises:
HTTPException: If project is not found
"""
# Try by permalink first (most common case with URL paths)
project_obj = await project_repository.get_by_permalink(str(project))
# Convert project name to permalink for lookup
project_permalink = generate_permalink(str(project))
project_obj = await project_repository.get_by_permalink(project_permalink)
if project_obj:
return project_obj.id
@@ -319,6 +321,7 @@ async def get_sync_service(
entity_parser: EntityParserDep,
entity_repository: EntityRepositoryDep,
relation_repository: RelationRepositoryDep,
project_repository: ProjectRepositoryDep,
search_service: SearchServiceDep,
file_service: FileServiceDep,
) -> SyncService: # pragma: no cover
@@ -332,6 +335,7 @@ async def get_sync_service(
entity_parser=entity_parser,
entity_repository=entity_repository,
relation_repository=relation_repository,
project_repository=project_repository,
search_service=search_service,
file_service=file_service,
)
+20 -75
View File
@@ -5,6 +5,7 @@ from pathlib import Path
import re
from typing import Any, Dict, Union
import aiofiles
import yaml
import frontmatter
from loguru import logger
@@ -52,29 +53,12 @@ async def compute_checksum(content: Union[str, bytes]) -> str:
raise FileError(f"Failed to compute checksum: {e}")
async def ensure_directory(path: FilePath) -> None:
"""
Ensure directory exists, creating if necessary.
Args:
path: Directory path to ensure (Path or string)
Raises:
FileWriteError: If directory creation fails
"""
try:
# Convert string to Path if needed
path_obj = Path(path) if isinstance(path, str) else path
path_obj.mkdir(parents=True, exist_ok=True)
except Exception as e: # pragma: no cover
logger.error("Failed to create directory", path=str(path), error=str(e))
raise FileWriteError(f"Failed to create directory {path}: {e}")
async def write_file_atomic(path: FilePath, content: str) -> None:
"""
Write file with atomic operation using temporary file.
Uses aiofiles for true async I/O (non-blocking).
Args:
path: Target file path (Path or string)
content: Content to write
@@ -87,7 +71,11 @@ async def write_file_atomic(path: FilePath, content: str) -> None:
temp_path = path_obj.with_suffix(".tmp")
try:
temp_path.write_text(content, encoding="utf-8")
# Use aiofiles for non-blocking write
async with aiofiles.open(temp_path, mode="w", encoding="utf-8") as f:
await f.write(content)
# Atomic rename (this is fast, doesn't need async)
temp_path.replace(path_obj)
logger.debug("Wrote file atomically", path=str(path_obj), content_length=len(content))
except Exception as e: # pragma: no cover
@@ -185,74 +173,26 @@ def remove_frontmatter(content: str) -> str:
return parts[2].strip()
async def update_frontmatter(path: FilePath, updates: Dict[str, Any]) -> str:
"""Update frontmatter fields in a file while preserving all content.
Only modifies the frontmatter section, leaving all content untouched.
Creates frontmatter section if none exists.
Returns checksum of updated file.
Args:
path: Path to markdown file (Path or string)
updates: Dict of frontmatter fields to update
Returns:
Checksum of updated file
Raises:
FileError: If file operations fail
ParseError: If frontmatter parsing fails
"""
try:
# Convert string to Path if needed
path_obj = Path(path) if isinstance(path, str) else path
# Read current content
content = path_obj.read_text(encoding="utf-8")
# Parse current frontmatter
current_fm = {}
if has_frontmatter(content):
current_fm = parse_frontmatter(content)
content = remove_frontmatter(content)
# Update frontmatter
new_fm = {**current_fm, **updates}
# Write new file with updated frontmatter
yaml_fm = yaml.dump(new_fm, sort_keys=False, allow_unicode=True)
final_content = f"---\n{yaml_fm}---\n\n{content.strip()}"
logger.debug("Updating frontmatter", path=str(path_obj), update_keys=list(updates.keys()))
await write_file_atomic(path_obj, final_content)
return await compute_checksum(final_content)
except Exception as e: # pragma: no cover
logger.error(
"Failed to update frontmatter",
path=str(path) if isinstance(path, (str, Path)) else "<unknown>",
error=str(e),
)
raise FileError(f"Failed to update frontmatter: {e}")
def dump_frontmatter(post: frontmatter.Post) -> str:
"""
Serialize frontmatter.Post to markdown with Obsidian-compatible YAML format.
This function ensures that tags are formatted as YAML lists instead of JSON arrays:
This function ensures that:
1. Tags are formatted as YAML lists instead of JSON arrays
2. String values are properly quoted to handle special characters (colons, etc.)
Good (Obsidian compatible):
---
title: "L2 Governance Core (Split: Core)"
tags:
- system
- overview
- reference
---
Bad (current behavior):
Bad (causes parsing errors):
---
title: L2 Governance Core (Split: Core) # Unquoted colon breaks YAML
tags: ["system", "overview", "reference"]
---
@@ -267,8 +207,13 @@ def dump_frontmatter(post: frontmatter.Post) -> str:
return post.content
# Serialize YAML with block style for lists
# SafeDumper automatically quotes values with special characters (colons, etc.)
yaml_str = yaml.dump(
post.metadata, sort_keys=False, allow_unicode=True, default_flow_style=False
post.metadata,
sort_keys=False,
allow_unicode=True,
default_flow_style=False,
Dumper=yaml.SafeDumper,
)
# Construct the final markdown with frontmatter
+23 -21
View File
@@ -11,9 +11,9 @@ DEFAULT_IGNORE_PATTERNS = {
# Hidden files (files starting with dot)
".*",
# Basic Memory internal files
"memory.db",
"memory.db-shm",
"memory.db-wal",
"*.db",
"*.db-shm",
"*.db-wal",
"config.json",
# Version control
".git",
@@ -84,10 +84,10 @@ def create_default_bmignore() -> None:
# Hidden files (files starting with dot)
.*
# Basic Memory internal files
memory.db
memory.db-shm
memory.db-wal
# Basic Memory internal files (includes test databases)
*.db
*.db-shm
*.db-wal
config.json
# Version control
@@ -172,15 +172,16 @@ def load_bmignore_patterns() -> Set[str]:
return patterns
def load_gitignore_patterns(base_path: Path) -> Set[str]:
def load_gitignore_patterns(base_path: Path, use_gitignore: bool = True) -> Set[str]:
"""Load gitignore patterns from .gitignore file and .bmignore.
Combines patterns from:
1. ~/.basic-memory/.bmignore (user's global ignore patterns)
2. {base_path}/.gitignore (project-specific patterns)
2. {base_path}/.gitignore (project-specific patterns, if use_gitignore=True)
Args:
base_path: The base directory to search for .gitignore file
use_gitignore: If False, only load patterns from .bmignore (default: True)
Returns:
Set of patterns to ignore
@@ -188,18 +189,19 @@ def load_gitignore_patterns(base_path: Path) -> Set[str]:
# Start with patterns from .bmignore
patterns = load_bmignore_patterns()
gitignore_file = base_path / ".gitignore"
if gitignore_file.exists():
try:
with gitignore_file.open("r", encoding="utf-8") as f:
for line in f:
line = line.strip()
# Skip empty lines and comments
if line and not line.startswith("#"):
patterns.add(line)
except Exception:
# If we can't read .gitignore, just use default patterns
pass
if use_gitignore:
gitignore_file = base_path / ".gitignore"
if gitignore_file.exists():
try:
with gitignore_file.open("r", encoding="utf-8") as f:
for line in f:
line = line.strip()
# Skip empty lines and comments
if line and not line.startswith("#"):
patterns.add(line)
except Exception:
# If we can't read .gitignore, just use default patterns
pass
return patterns
@@ -153,7 +153,12 @@ class ClaudeConversationsImporter(Importer[ChatImportResult]):
# Handle message content
content = msg.get("text", "")
if msg.get("content"):
content = " ".join(c.get("text", "") for c in msg["content"])
# Filter out None values before joining
content = " ".join(
str(c.get("text", ""))
for c in msg["content"]
if c and c.get("text") is not None
)
lines.append(content)
# Handle attachments
+113 -8
View File
@@ -4,12 +4,14 @@ Uses markdown-it with plugins to parse structured data from markdown content.
"""
from dataclasses import dataclass, field
from datetime import datetime
from datetime import date, datetime
from pathlib import Path
from typing import Any, Optional
import dateparser
import frontmatter
import yaml
from loguru import logger
from markdown_it import MarkdownIt
from basic_memory.markdown.plugins import observation_plugin, relation_plugin
@@ -24,6 +26,82 @@ from basic_memory.utils import parse_tags
md = MarkdownIt().use(observation_plugin).use(relation_plugin)
def normalize_frontmatter_value(value: Any) -> Any:
"""Normalize frontmatter values to safe types for processing.
PyYAML automatically converts various string-like values into native Python types:
- Date strings ("2025-10-24") → datetime.date objects
- Numbers ("1.0") → int or float
- Booleans ("true") → bool
- Lists → list objects
This can cause AttributeError when code expects strings and calls string methods
like .strip() on these values (see GitHub issue #236).
This function normalizes all frontmatter values to safe types:
- Dates/datetimes → ISO format strings
- Numbers (int/float) → strings
- Booleans → strings ("True"/"False")
- Lists → preserved as lists, but items are recursively normalized
- Dicts → preserved as dicts, but values are recursively normalized
- Strings → kept as-is
- None → kept as None
Args:
value: The frontmatter value to normalize
Returns:
The normalized value safe for string operations
Example:
>>> normalize_frontmatter_value(datetime.date(2025, 10, 24))
'2025-10-24'
>>> normalize_frontmatter_value([datetime.date(2025, 10, 24), "tag", 123])
['2025-10-24', 'tag', '123']
>>> normalize_frontmatter_value(True)
'True'
"""
# Convert date/datetime objects to ISO format strings
if isinstance(value, datetime):
return value.isoformat()
if isinstance(value, date):
return value.isoformat()
# Convert boolean to string (must come before int check since bool is subclass of int)
if isinstance(value, bool):
return str(value)
# Convert numbers to strings
if isinstance(value, (int, float)):
return str(value)
# Recursively process lists (preserve as list, normalize items)
if isinstance(value, list):
return [normalize_frontmatter_value(item) for item in value]
# Recursively process dicts (preserve as dict, normalize values)
if isinstance(value, dict):
return {key: normalize_frontmatter_value(val) for key, val in value.items()}
# Keep strings and None as-is
return value
def normalize_frontmatter_metadata(metadata: dict) -> dict:
"""Normalize all values in frontmatter metadata dict.
Converts date/datetime objects to ISO format strings to prevent
AttributeError when code expects strings (GitHub issue #236).
Args:
metadata: The frontmatter metadata dictionary
Returns:
A new dictionary with all values normalized
"""
return {key: normalize_frontmatter_value(value) for key, value in metadata.items()}
@dataclass
class EntityContent:
content: str
@@ -111,18 +189,45 @@ class EntityParser:
return self.base_path / path
async def parse_file_content(self, absolute_path, file_content):
post = frontmatter.loads(file_content)
# Parse frontmatter with proper error handling for malformed YAML (issue #185)
try:
post = frontmatter.loads(file_content)
except yaml.YAMLError as e:
# Log the YAML parsing error with file context
logger.warning(
f"Failed to parse YAML frontmatter in {absolute_path}: {e}. "
f"Treating file as plain markdown without frontmatter."
)
# Create a post with no frontmatter - treat entire content as markdown
post = frontmatter.Post(file_content, metadata={})
# Extract file stat info
file_stats = absolute_path.stat()
metadata = post.metadata
metadata["title"] = post.metadata.get("title", absolute_path.stem)
metadata["type"] = post.metadata.get("type", "note")
tags = parse_tags(post.metadata.get("tags", [])) # pyright: ignore
# Normalize frontmatter values to prevent AttributeError on date objects (issue #236)
# PyYAML automatically converts date strings like "2025-10-24" to datetime.date objects
# This normalization converts them back to ISO format strings to ensure compatibility
# with code that expects string values
metadata = normalize_frontmatter_metadata(post.metadata)
# Ensure required fields have defaults (issue #184, #387)
# Handle title - use default if missing, None/null, empty, or string "None"
title = metadata.get("title")
if not title or title == "None":
metadata["title"] = absolute_path.stem
else:
metadata["title"] = title
# Handle type - use default if missing OR explicitly set to None/null
entity_type = metadata.get("type")
metadata["type"] = entity_type if entity_type is not None else "note"
tags = parse_tags(metadata.get("tags", [])) # pyright: ignore
if tags:
metadata["tags"] = tags
# frontmatter
# frontmatter - use metadata with defaults applied
entity_frontmatter = EntityFrontmatter(
metadata=post.metadata,
metadata=metadata,
)
entity_content = parse(post.content)
return EntityMarkdown(
+102 -4
View File
@@ -1,3 +1,6 @@
from contextlib import asynccontextmanager, AbstractAsyncContextManager
from typing import AsyncIterator, Callable, Optional
from httpx import ASGITransport, AsyncClient, Timeout
from loguru import logger
@@ -5,9 +8,108 @@ from basic_memory.api.app import app as fastapi_app
from basic_memory.config import ConfigManager
# Optional factory override for dependency injection
_client_factory: Optional[Callable[[], AbstractAsyncContextManager[AsyncClient]]] = None
def set_client_factory(factory: Callable[[], AbstractAsyncContextManager[AsyncClient]]) -> None:
"""Override the default client factory (for cloud app, testing, etc).
Args:
factory: An async context manager that yields an AsyncClient
Example:
@asynccontextmanager
async def custom_client_factory():
async with AsyncClient(...) as client:
yield client
set_client_factory(custom_client_factory)
"""
global _client_factory
_client_factory = factory
@asynccontextmanager
async def get_client() -> AsyncIterator[AsyncClient]:
"""Get an AsyncClient as a context manager.
This function provides proper resource management for HTTP clients,
ensuring connections are closed after use. It supports three modes:
1. **Factory injection** (cloud app, tests):
If a custom factory is set via set_client_factory(), use that.
2. **CLI cloud mode**:
When cloud_mode_enabled is True, create HTTP client with auth
token from CLIAuth for requests to cloud proxy endpoint.
3. **Local mode** (default):
Use ASGI transport for in-process requests to local FastAPI app.
Usage:
async with get_client() as client:
response = await client.get("/path")
Yields:
AsyncClient: Configured HTTP client for the current mode
Raises:
RuntimeError: If cloud mode is enabled but user is not authenticated
"""
if _client_factory:
# Use injected factory (cloud app, tests)
async with _client_factory() as client:
yield client
else:
# Default: create based on config
config = ConfigManager().config
timeout = Timeout(
connect=10.0, # 10 seconds for connection
read=30.0, # 30 seconds for reading response
write=30.0, # 30 seconds for writing request
pool=30.0, # 30 seconds for connection pool
)
if config.cloud_mode_enabled:
# CLI cloud mode: inject auth when creating client
from basic_memory.cli.auth import CLIAuth
auth = CLIAuth(client_id=config.cloud_client_id, authkit_domain=config.cloud_domain)
token = await auth.get_valid_token()
if not token:
raise RuntimeError(
"Cloud mode enabled but not authenticated. "
"Run 'basic-memory cloud login' first."
)
# Auth header set ONCE at client creation
proxy_base_url = f"{config.cloud_host}/proxy"
logger.info(f"Creating HTTP client for cloud proxy at: {proxy_base_url}")
async with AsyncClient(
base_url=proxy_base_url,
headers={"Authorization": f"Bearer {token}"},
timeout=timeout,
) as client:
yield client
else:
# Local mode: ASGI transport for in-process calls
logger.info("Creating ASGI client for local Basic Memory API")
async with AsyncClient(
transport=ASGITransport(app=fastapi_app), base_url="http://test", timeout=timeout
) as client:
yield client
def create_client() -> AsyncClient:
"""Create an HTTP client based on configuration.
DEPRECATED: Use get_client() context manager instead for proper resource management.
This function is kept for backward compatibility but will be removed in a future version.
The returned client should be closed manually by calling await client.aclose().
Returns:
AsyncClient configured for either local ASGI or remote proxy
"""
@@ -34,7 +136,3 @@ def create_client() -> AsyncClient:
return AsyncClient(
transport=ASGITransport(app=fastapi_app), base_url="http://test", timeout=timeout
)
# Create shared async client
client = create_client()
@@ -10,7 +10,7 @@ from loguru import logger
from pydantic import Field
from basic_memory.config import get_project_config
from basic_memory.mcp.async_client import client
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_post
from basic_memory.schemas.base import TimeFrame
@@ -42,20 +42,21 @@ async def continue_conversation(
"""
logger.info(f"Continuing session, topic: {topic}, timeframe: {timeframe}")
# Create request model
request = ContinueConversationRequest( # pyright: ignore [reportCallIssue]
topic=topic, timeframe=timeframe
)
async with get_client() as client:
# Create request model
request = ContinueConversationRequest( # pyright: ignore [reportCallIssue]
topic=topic, timeframe=timeframe
)
project_url = get_project_config().project_url
project_url = get_project_config().project_url
# Call the prompt API endpoint
response = await call_post(
client,
f"{project_url}/prompt/continue-conversation",
json=request.model_dump(exclude_none=True),
)
# Call the prompt API endpoint
response = await call_post(
client,
f"{project_url}/prompt/continue-conversation",
json=request.model_dump(exclude_none=True),
)
# Extract the rendered prompt from the response
result = response.json()
return result["prompt"]
# Extract the rendered prompt from the response
result = response.json()
return result["prompt"]
+12 -11
View File
@@ -9,7 +9,7 @@ from loguru import logger
from pydantic import Field
from basic_memory.config import get_project_config
from basic_memory.mcp.async_client import client
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_post
from basic_memory.schemas.base import TimeFrame
@@ -41,16 +41,17 @@ async def search_prompt(
"""
logger.info(f"Searching knowledge base, query: {query}, timeframe: {timeframe}")
# Create request model
request = SearchPromptRequest(query=query, timeframe=timeframe)
async with get_client() as client:
# Create request model
request = SearchPromptRequest(query=query, timeframe=timeframe)
project_url = get_project_config().project_url
project_url = get_project_config().project_url
# Call the prompt API endpoint
response = await call_post(
client, f"{project_url}/prompt/search", json=request.model_dump(exclude_none=True)
)
# Call the prompt API endpoint
response = await call_post(
client, f"{project_url}/prompt/search", json=request.model_dump(exclude_none=True)
)
# Extract the rendered prompt from the response
result = response.json()
return result["prompt"]
# Extract the rendered prompt from the response
result = response.json()
return result["prompt"]
@@ -1,551 +1,283 @@
# AI Assistant Guide for Basic Memory
This guide helps AIs use Basic Memory tools effectively when working with users. It covers reading, writing, and
navigating knowledge through the Model Context Protocol (MCP).
Quick reference for using Basic Memory tools effectively through MCP.
**For comprehensive coverage**: See the [Extended AI Assistant Guide](https://github.com/basicmachines-co/basic-memory/blob/main/docs/ai-assistant-guide-extended.md) with detailed examples, advanced patterns, and self-contained sections.
## Overview
Basic Memory allows you and users to record context in local Markdown files, building a rich knowledge base through
natural conversations. The system automatically creates a semantic knowledge graph from simple text patterns.
Basic Memory creates a semantic knowledge graph from markdown files. Focus on building rich connections between notes.
- **Local-First**: All data is stored in plain text files on the user's computer
- **Real-Time**: Users see content updates immediately
- **Bi-Directional**: Both you and users can read and edit notes
- **Semantic**: Simple patterns create a structured knowledge graph
- **Persistent**: Knowledge persists across sessions and conversations
- **Local-First**: Plain text files on user's computer
- **Persistent**: Knowledge survives across sessions
- **Semantic**: Observations and relations create a knowledge graph
## Project Management and Configuration
**Your role**: You're helping humans build enduring knowledge they'll own forever. The semantic graph (observations, relations, context) helps you provide better assistance by understanding connections and maintaining continuity. Think: lasting insights worth keeping, not disposable chat logs.
Basic Memory uses a **stateless architecture** where each tool call can specify which project to work with. This provides three ways to determine the active project:
## Project Management
### Three-Tier Project Resolution
All tools require explicit project specification.
1. **CLI Constraint (Highest Priority)**: When Basic Memory is started with `--project project-name`, all operations are constrained to that project
2. **Explicit Project Parameter (Medium Priority)**: When you specify `project="project-name"` in tool calls
3. **Default Project Mode (Lowest Priority)**: When `default_project_mode=true` in configuration, tools automatically use the configured `default_project`
**Three-tier resolution:**
1. CLI constraint: `--project name` (highest priority)
2. Explicit parameter: `project="name"` in tool calls
3. Default mode: `default_project_mode=true` in config (fallback)
### Quick Setup Check
```python
# Discover projects
projects = await list_memory_projects()
# Check if default_project_mode enabled
# If yes: project parameter optional
# If no: project parameter required
```
### Default Project Mode
When `default_project_mode` is enabled in the user's configuration:
- All tools become more convenient - no need to specify project repeatedly
- Perfect for users who primarily work with a single project
- Still allows explicit project specification when needed
- Falls back gracefully to multi-project mode if no default is configured
When `default_project_mode=true`:
```python
# With default_project_mode enabled, these are equivalent:
await write_note("My Note", "Content", "folder")
await write_note("My Note", "Content", "folder", project="default-project")
# You can still override with explicit project:
await write_note("My Note", "Content", "folder", project="other-project")
# These are equivalent:
await write_note("Note", "Content", "folder")
await write_note("Note", "Content", "folder", project="main")
```
### Project Discovery
If you're unsure which project to use:
When `default_project_mode=false` (default):
```python
# Discover available projects
projects = await list_memory_projects()
# See recent activity across projects for recommendations
activity = await recent_activity() # Shows cross-project activity and suggestions
# Project required:
await write_note("Note", "Content", "folder", project="main") # ✓
await write_note("Note", "Content", "folder") # ✗ Error
```
## The Importance of the Knowledge Graph
## Core Tools
**Basic Memory's value comes from connections between notes, not just the notes themselves.**
When writing notes, your primary goal should be creating a rich, interconnected knowledge graph:
1. **Increase Semantic Density**: Add multiple observations and relations to each note
2. **Use Accurate References**: Aim to reference existing entities by their exact titles
3. **Create Forward References**: Feel free to reference entities that don't exist yet - Basic Memory will resolve these
when they're created later
4. **Create Bidirectional Links**: When appropriate, connect entities from both directions
5. **Use Meaningful Categories**: Add semantic context with appropriate observation categories
6. **Choose Precise Relations**: Use specific relation types that convey meaning
Remember: A knowledge graph with 10 heavily connected notes is more valuable than 20 isolated notes. Your job is to help
build these connections!
## Core Tools Reference
### Knowledge Creation and Editing
### Writing Knowledge
```python
# Writing knowledge - THE MOST IMPORTANT TOOL!
response = await write_note(
title="Search Design", # Required: Note title
content="# Search Design\n...", # Required: Note content
folder="specs", # Required: Folder to save in
tags=["search", "design"], # Optional: Tags for categorization
project="my-project" # Optional: Explicit project (uses default if not specified)
)
# Editing existing notes
await edit_note(
identifier="Search Design", # Required: Note to edit
operation="append", # Required: append, prepend, find_replace, replace_section
content="\n## New Section\nAdditional content", # Required: Content to add/replace
project="my-project" # Optional: Explicit project
)
# Moving notes
await move_note(
identifier="Search Design", # Required: Note to move
destination_path="archive/old-search-design.md", # Required: New location
project="my-project" # Optional: Explicit project
)
# Deleting notes
success = await delete_note(
identifier="Old Draft", # Required: Note to delete
project="my-project" # Optional: Explicit project
await write_note(
title="Topic",
content="# Topic\n## Observations\n- [category] fact\n## Relations\n- relates_to [[Other]]",
folder="notes",
project="main" # Required unless default_project_mode=true
)
```
### Knowledge Reading and Discovery
### Reading Knowledge
```python
# Reading knowledge
content = await read_note("Search Design") # By title (uses default project)
content = await read_note("specs/search-design") # By path
content = await read_note("memory://specs/search") # By memory URL
content = await read_note("Search Design", project="work-docs") # Explicit project
# By identifier
content = await read_note("Topic", project="main")
# Reading raw file content (text, images, binaries)
file_data = await read_content(
path="assets/diagram.png", # Required: File path
project="my-project" # Optional: Explicit project
)
# Viewing notes as formatted artifacts
await view_note(
identifier="Search Design", # Required: Note to view
project="my-project", # Optional: Explicit project
page=1, # Optional: Pagination
page_size=10 # Optional: Items per page
)
# Browsing directory contents
listing = await list_directory(
dir_name="/specs", # Optional: Directory path (default: "/")
depth=2, # Optional: Recursion depth
file_name_glob="*.md", # Optional: File pattern filter
project="my-project" # Optional: Explicit project
)
# By memory:// URL
content = await read_note("memory://folder/topic", project="main")
```
### Search and Context
### Searching
```python
# Searching for knowledge
results = await search_notes(
query="authentication system", # Required: Text to search for
project="my-project", # Optional: Explicit project
page=1, # Optional: Pagination
page_size=10, # Optional: Results per page
search_type="text", # Optional: "text", "title", or "permalink"
types=["entity"], # Optional: Filter by content types
entity_types=["observation"], # Optional: Filter by entity types
after_date="1 week" # Optional: Recent content only
query="authentication",
project="main",
page_size=10
)
```
# Building context from the knowledge graph
### Building Context
```python
context = await build_context(
url="memory://specs/search", # Required: Starting point
project="my-project", # Optional: Explicit project
depth=2, # Optional: How many hops to follow
timeframe="1 month", # Optional: Recent timeframe
max_related=10 # Optional: Max related items
)
# Checking recent changes
activity = await recent_activity(
type=["entity", "relation"], # Optional: Entity types to include
depth=1, # Optional: Related items to include
timeframe="1 week", # Optional: Time window
project="my-project" # Optional: Explicit project (None for cross-project discovery)
url="memory://specs/auth",
project="main",
depth=2,
timeframe="1 week"
)
```
### Visualization and Project Management
## Knowledge Graph Essentials
### Observations
Categorized facts with optional tags:
```markdown
- [decision] Use JWT for authentication #security
- [technique] Hash passwords with bcrypt #best-practice
- [requirement] Support OAuth 2.0 providers
```
### Relations
Directional links between entities:
```markdown
- implements [[Authentication Spec]]
- requires [[User Database]]
- extends [[Base Security Model]]
```
**Common relation types:** `relates_to`, `implements`, `requires`, `extends`, `part_of`, `contrasts_with`
### Forward References
Reference entities that don't exist yet:
```python
# Creating a knowledge visualization
canvas_result = await canvas(
nodes=[{"id": "note1", "type": "file", "file": "Search Design.md"}], # Required: Nodes
edges=[{"id": "edge1", "fromNode": "note1", "toNode": "note2"}], # Required: Edges
title="Project Overview", # Required: Canvas title
folder="diagrams", # Required: Storage location
project="my-project" # Optional: Explicit project
# Create note with forward reference
await write_note(
title="Login Flow",
content="## Relations\n- requires [[OAuth Provider]]", # Doesn't exist yet
folder="auth",
project="main"
)
# Project management
projects = await list_memory_projects() # List all available projects
project_info = await project_info(project="my-project") # Get project statistics
# Later, create referenced entity
await write_note(
title="OAuth Provider",
content="# OAuth Provider\n...",
folder="auth",
project="main"
)
# → Relation automatically resolved
```
## memory:// URLs Explained
Basic Memory uses a special URL format to reference entities in the knowledge graph:
- `memory://title` - Reference by title
- `memory://folder/title` - Reference by folder and title
- `memory://permalink` - Reference by permalink
- `memory://path/relation_type/*` - Follow all relations of a specific type
- `memory://path/*/target` - Find all entities with relations to target
## Semantic Markdown Format
Knowledge is encoded in standard markdown using simple patterns:
**Observations** - Facts about an entity:
```markdown
- [category] This is an observation #tag1 #tag2 (optional context)
```
**Relations** - Links between entities:
```markdown
- relation_type [[Target Entity]] (optional context)
```
**Common Categories & Relation Types:**
- Categories: `[idea]`, `[decision]`, `[question]`, `[fact]`, `[requirement]`, `[technique]`, `[recipe]`, `[preference]`
- Relations: `relates_to`, `implements`, `requires`, `extends`, `part_of`, `pairs_with`, `inspired_by`,
`originated_from`
## When to Record Context
**Always consider recording context when**:
1. Users make decisions or reach conclusions
2. Important information emerges during conversation
3. Multiple related topics are discussed
4. The conversation contains information that might be useful later
5. Plans, tasks, or action items are mentioned
**Protocol for recording context**:
1. Identify valuable information in the conversation
2. Ask the user: "Would you like me to record our discussion about [topic] in Basic Memory?"
3. If they agree, use `write_note` to capture the information
4. If they decline, continue without recording
5. Let the user know when information has been recorded: "I've saved our discussion about [topic] to Basic Memory."
## Understanding User Interactions
Users will interact with Basic Memory in patterns like:
1. **Creating knowledge**:
```
Human: "Let's write up what we discussed about search."
You: I'll create a note capturing our discussion about the search functionality.
await write_note(
title="Search Functionality Discussion",
content="# Search Functionality Discussion\n...",
folder="discussions"
)
```
2. **Referencing existing knowledge**:
```
Human: "Take a look at memory://specs/search"
You: I'll examine that information.
context = await build_context(url="memory://specs/search")
content = await read_note("specs/search")
```
3. **Finding information**:
```
Human: "What were our decisions about auth?"
You: Let me find that information for you.
results = await search_notes(query="auth decisions")
context = await build_context(url=f"memory://{results[0].permalink}")
```
## Key Things to Remember
1. **Files are Truth**
- All knowledge lives in local files on the user's computer
- Users can edit files outside your interaction
- Changes need to be synced by the user (usually automatic)
- Always verify information is current with `recent_activity()`
2. **Building Context Effectively**
- Start with specific entities
- Follow meaningful relations
- Check recent changes
- Build context incrementally
- Combine related information
3. **Writing Knowledge Wisely**
- Using the same title+folder will overwrite existing notes
- Structure content with clear headings and sections
- Use semantic markup for observations and relations
- Keep files organized in logical folders
## Common Knowledge Patterns
### Capturing Decisions
```markdown
# Coffee Brewing Methods
## Context
I've experimented with various brewing methods including French press, pour over, and espresso.
## Decision
Pour over is my preferred method for light to medium roasts because it highlights subtle flavors and offers more control
over the extraction.
## Observations
- [technique] Blooming the coffee grounds for 30 seconds improves extraction #brewing
- [preference] Water temperature between 195-205°F works best #temperature
- [equipment] Gooseneck kettle provides better control of water flow #tools
## Relations
- pairs_with [[Light Roast Beans]]
- contrasts_with [[French Press Method]]
- requires [[Proper Grinding Technique]]
```
### Recording Project Structure
```markdown
# Garden Planning
## Overview
This document outlines the garden layout and planting strategy for this season.
## Observations
- [structure] Raised beds in south corner for sun exposure #layout
- [structure] Drip irrigation system installed for efficiency #watering
- [pattern] Companion planting used to deter pests naturally #technique
## Relations
- contains [[Vegetable Section]]
- contains [[Herb Garden]]
- implements [[Organic Gardening Principles]]
```
### Technical Discussions
```markdown
# Recipe Improvement Discussion
## Key Points
Discussed strategies for improving the chocolate chip cookie recipe.
## Observations
- [issue] Cookies spread too thin when baked at 350°F #texture
- [solution] Chilling dough for 24 hours improves flavor and reduces spreading #technique
- [decision] Will use brown butter instead of regular butter #flavor
## Relations
- improves [[Basic Cookie Recipe]]
- inspired_by [[Bakery-Style Cookies]]
- pairs_with [[Homemade Ice Cream]]
```
### Creating Effective Relations
When creating relations, you can:
1. Reference existing entities by their exact title
2. Create forward references to entities that don't exist yet
```python
# Example workflow for creating notes with effective relations
async def create_note_with_effective_relations():
# Search for existing entities to reference
search_results = await search_notes(query="travel")
existing_entities = [result.title for result in search_results.primary_results]
# Check if specific entities exist
packing_tips_exists = "Packing Tips" in existing_entities
japan_travel_exists = "Japan Travel Guide" in existing_entities
# Prepare relations section - include both existing and forward references
relations_section = "## Relations\n"
# Existing reference - exact match to known entity
if packing_tips_exists:
relations_section += "- references [[Packing Tips]]\n"
else:
# Forward reference - will be linked when that entity is created later
relations_section += "- references [[Packing Tips]]\n"
# Another possible reference
if japan_travel_exists:
relations_section += "- part_of [[Japan Travel Guide]]\n"
# You can also check recently modified notes to reference them
recent = await recent_activity(timeframe="1 week")
recent_titles = [item.title for item in recent.primary_results]
if "Transportation Options" in recent_titles:
relations_section += "- relates_to [[Transportation Options]]\n"
# Always include meaningful forward references, even if they don't exist yet
relations_section += "- located_in [[Tokyo]]\n"
relations_section += "- visited_during [[Spring 2023 Trip]]\n"
# Now create the note with both verified and forward relations
content = f"""# Tokyo Neighborhood Guide
## Overview
Details about different Tokyo neighborhoods and their unique characteristics.
## Observations
- [area] Shibuya is a busy shopping district #shopping
- [transportation] Yamanote Line connects major neighborhoods #transit
- [recommendation] Visit Shimokitazawa for vintage shopping #unique
- [tip] Get a Suica card for easy train travel #convenience
{relations_section}
"""
result = await write_note(
title="Tokyo Neighborhood Guide",
content=content,
folder="travel"
)
# You can check which relations were resolved and which are forward references
if result and 'relations' in result:
resolved = [r['to_name'] for r in result['relations'] if r.get('target_id')]
forward_refs = [r['to_name'] for r in result['relations'] if not r.get('target_id')]
print(f"Resolved relations: {resolved}")
print(f"Forward references that will be resolved later: {forward_refs}")
```
## Error Handling
Common issues to watch for:
1. **Missing Content**
```python
try:
content = await read_note("Document")
except:
# Try search instead
results = await search_notes(query="Document")
if results and results.primary_results:
# Found something similar
content = await read_note(results.primary_results[0].permalink)
```
2. **Forward References (Unresolved Relations)**
```python
response = await write_note(
title="My Note",
content="Content with [[Forward Reference]]",
folder="notes"
)
# Check for forward references (unresolved relations)
forward_refs = []
for relation in response.get('relations', []):
if not relation.get('target_id'):
forward_refs.append(relation.get('to_name'))
if forward_refs:
# This is a feature, not an error! Inform the user about forward references
print(f"Note created with forward references to: {forward_refs}")
print("These will be automatically linked when those notes are created.")
# Optionally suggest creating those entities now
print("Would you like me to create any of these notes now to complete the connections?")
```
3. **Project Discovery Issues**
```python
# If user asks about content but no default project is configured
try:
results = await search_notes(query="user query")
except Exception as e:
if "project" in str(e).lower():
# Show available projects and ask user to choose
projects = await list_memory_projects()
print(f"Available projects: {[p.name for p in projects]}")
print("Which project should I search in?")
```
4. **Sync Issues**
```python
# If information seems outdated
activity = await recent_activity(timeframe="1 hour")
if not activity or not activity.primary_results:
print("It seems there haven't been recent updates. You might need to run 'basic-memory sync'.")
```
## Best Practices
1. **Smart Project Management**
- **For new users**: Call `recent_activity()` without project parameter to discover active projects and get recommendations
- **For known projects**: Use explicit project parameters when switching between multiple projects
- **For single-project users**: Rely on default_project_mode for convenience
- **When uncertain**: Use `list_memory_projects()` to show available options and ask the user
- **Remember choices**: Once a user indicates their preferred project, use it consistently throughout the conversation
### 1. Project Management
2. **Proactively Record Context**
- Offer to capture important discussions
- Record decisions, rationales, and conclusions
- Link to related topics
- Ask for permission first: "Would you like me to save our discussion about [topic]?"
- Confirm when complete: "I've saved our discussion to Basic Memory"
**Single-project users:**
- Enable `default_project_mode=true`
- Simpler tool calls
3. **Create a Rich Semantic Graph**
- **Add meaningful observations**: Include at least 3-5 categorized observations in each note
- **Create deliberate relations**: Connect each note to at least 2-3 related entities
- **Use existing entities**: Before creating a new relation, search for existing entities
- **Verify wikilinks**: When referencing `[[Entity]]`, use exact titles of existing notes
- **Check accuracy**: Use `search_notes()` or `recent_activity()` to confirm entity titles
- **Use precise relation types**: Choose specific relation types that convey meaning (e.g., "implements" instead
of "relates_to")
- **Consider bidirectional relations**: When appropriate, create inverse relations in both entities
**Multi-project users:**
- Keep `default_project_mode=false`
- Always specify project explicitly
4. **Structure Content Thoughtfully**
- Use clear, descriptive titles
- Organize with logical sections (Context, Decision, Implementation, etc.)
- Include relevant context and background
- Add semantic observations with appropriate categories
- Use a consistent format for similar types of notes
- Balance detail with conciseness
**Discovery:**
```python
# Start with discovery
projects = await list_memory_projects()
5. **Navigate Knowledge Effectively**
- Start with specific searches using `search_notes()`
- Follow relation paths with `build_context()`
- Combine information from multiple sources
- Verify information is current with `recent_activity()`
- Build a complete picture before responding
- Use appropriate project context for searches
# Cross-project activity (no project param = all projects)
activity = await recent_activity()
6. **Help Users Maintain Their Knowledge**
- Suggest organizing related topics across projects when appropriate
- Identify potential duplicates using search
- Recommend adding relations between topics
- Offer to create summaries of scattered information
- Suggest potential missing relations: "I notice this might relate to [topic], would you like me to add that connection?"
- Help users decide when to use explicit vs default project parameters
# Or specific project
activity = await recent_activity(project="main")
```
Built with ♥️ b
y Basic Machines
### 2. Building Rich Graphs
**Always include:**
- 3-5 observations per note
- 2-3 relations per note
- Meaningful categories and relation types
**Search before creating:**
```python
# Find existing entities to reference
results = await search_notes(query="authentication", project="main")
# Use exact titles in [[WikiLinks]]
```
### 3. Writing Effective Notes
**Structure:**
```markdown
# Title
## Context
Background information
## Observations
- [category] Fact with #tags
- [category] Another fact
## Relations
- relation_type [[Exact Entity Title]]
```
**Categories:** `[idea]`, `[decision]`, `[fact]`, `[technique]`, `[requirement]`
### 4. Error Handling
**Missing project:**
```python
try:
await search_notes(query="test") # Missing project parameter - will error
except:
# Show available projects
projects = await list_memory_projects()
# Then retry with project
results = await search_notes(query="test", project=projects[0].name)
```
**Forward references:**
```python
# Check response for unresolved relations
response = await write_note(
title="New Topic",
content="## Relations\n- relates_to [[Future Topic]]",
folder="notes",
project="main"
)
# Forward refs will resolve when target created
```
### 5. Recording Context
**Ask permission:**
> "Would you like me to save our discussion about [topic] to Basic Memory?"
**Confirm when done:**
> "I've saved our discussion to Basic Memory."
**What to record:**
- Decisions and rationales
- Important discoveries
- Action items and plans
- Connected topics
## Common Patterns
### Capture Decision
```python
await write_note(
title="DB Choice",
content="""# DB Choice\n## Decision\nUse PostgreSQL\n## Observations\n- [requirement] ACID compliance #reliability\n- [decision] PostgreSQL over MySQL\n## Relations\n- implements [[Data Architecture]]""",
folder="decisions",
project="main"
)
```
### Link Topics & Build Context
```python
# Link bidirectionally
await write_note(title="API Auth", content="## Relations\n- part_of [[API Design]]", folder="api", project="main")
await edit_note(identifier="API Design", operation="append", content="\n- includes [[API Auth]]", project="main")
# Search and build context
results = await search_notes(query="authentication", project="main")
context = await build_context(url=f"memory://{results[0].permalink}", project="main", depth=2)
```
## Tool Quick Reference
| Tool | Purpose | Key Params |
|------|---------|------------|
| `write_note` | Create/update | title, content, folder, project |
| `read_note` | Read content | identifier, project |
| `edit_note` | Modify existing | identifier, operation, content, project |
| `search_notes` | Find notes | query, project |
| `build_context` | Graph traversal | url, depth, project |
| `recent_activity` | Recent changes | timeframe, project |
| `list_memory_projects` | Show projects | (none) |
## memory:// URL Format
- `memory://title` - By title
- `memory://folder/title` - By folder + title
- `memory://permalink` - By permalink
- `memory://folder/*` - All in folder
For full documentation: https://docs.basicmemory.com
Built with ♥️ by Basic Machines
@@ -5,7 +5,7 @@ from typing import Optional
from loguru import logger
from fastmcp import Context
from basic_memory.mcp.async_client import client
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.project_context import get_active_project
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_get
@@ -59,11 +59,13 @@ async def project_info(
print(f"Basic Memory version: {info.system.version}")
"""
logger.info("Getting project info")
project_config = await get_active_project(client, project, context)
project_url = project_config.permalink
# Call the API endpoint
response = await call_get(client, f"{project_url}/project/info")
async with get_client() as client:
project_config = await get_active_project(client, project, context)
project_url = project_config.permalink
# Convert response to ProjectInfoResponse
return ProjectInfoResponse.model_validate(response.json())
# Call the API endpoint
response = await call_get(client, f"{project_url}/project/info")
# Convert response to ProjectInfoResponse
return ProjectInfoResponse.model_validate(response.json())
-2
View File
@@ -18,7 +18,6 @@ from basic_memory.mcp.tools.canvas import canvas
from basic_memory.mcp.tools.list_directory import list_directory
from basic_memory.mcp.tools.edit_note import edit_note
from basic_memory.mcp.tools.move_note import move_note
from basic_memory.mcp.tools.sync_status import sync_status
from basic_memory.mcp.tools.project_management import (
list_memory_projects,
create_memory_project,
@@ -44,7 +43,6 @@ __all__ = [
"recent_activity",
"search",
"search_notes",
"sync_status",
"view_note",
"write_note",
]
+17 -40
View File
@@ -5,7 +5,7 @@ from typing import Optional
from loguru import logger
from fastmcp import Context
from basic_memory.mcp.async_client import client
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.project_context import get_active_project
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_get
@@ -16,8 +16,6 @@ from basic_memory.schemas.memory import (
memory_url_path,
)
type StringOrInt = str | int
@mcp.tool(
description="""Build context from a memory:// URI to continue conversations naturally.
@@ -39,7 +37,7 @@ type StringOrInt = str | int
async def build_context(
url: MemoryUrl,
project: Optional[str] = None,
depth: Optional[StringOrInt] = 1,
depth: str | int | None = 1,
timeframe: Optional[TimeFrame] = "7d",
page: int = 1,
page_size: int = 10,
@@ -102,42 +100,21 @@ async def build_context(
# URL is already validated and normalized by MemoryUrl type annotation
# Get the active project using the new stateless approach
active_project = await get_active_project(client, project, context)
async with get_client() as client:
# Get the active project using the new stateless approach
active_project = await get_active_project(client, project, context)
# Check migration status and wait briefly if needed
from basic_memory.mcp.tools.utils import wait_for_migration_or_return_status
project_url = active_project.project_url
migration_status = await wait_for_migration_or_return_status(
timeout=5.0, project_name=active_project.name
)
if migration_status: # pragma: no cover
# Return a proper GraphContext with status message
from basic_memory.schemas.memory import MemoryMetadata
from datetime import datetime
return GraphContext(
results=[],
metadata=MemoryMetadata(
depth=depth or 1,
timeframe=timeframe,
generated_at=datetime.now().astimezone(),
primary_count=0,
related_count=0,
uri=migration_status, # Include status in metadata
),
response = await call_get(
client,
f"{project_url}/memory/{memory_url_path(url)}",
params={
"depth": depth,
"timeframe": timeframe,
"page": page,
"page_size": page_size,
"max_related": max_related,
},
)
project_url = active_project.project_url
response = await call_get(
client,
f"{project_url}/memory/{memory_url_path(url)}",
params={
"depth": depth,
"timeframe": timeframe,
"page": page,
"page_size": page_size,
"max_related": max_related,
},
)
return GraphContext.model_validate(response.json())
return GraphContext.model_validate(response.json())
+28 -20
View File
@@ -9,7 +9,7 @@ from typing import Dict, List, Any, Optional
from loguru import logger
from fastmcp import Context
from basic_memory.mcp.async_client import client
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.project_context import get_active_project
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_put
@@ -94,29 +94,37 @@ async def canvas(
Raises:
ToolError: If project doesn't exist or folder path is invalid
"""
active_project = await get_active_project(client, project, context)
project_url = active_project.project_url
async with get_client() as client:
active_project = await get_active_project(client, project, context)
project_url = active_project.project_url
# Ensure path has .canvas extension
file_title = title if title.endswith(".canvas") else f"{title}.canvas"
file_path = f"{folder}/{file_title}"
# Ensure path has .canvas extension
file_title = title if title.endswith(".canvas") else f"{title}.canvas"
file_path = f"{folder}/{file_title}"
# Create canvas data structure
canvas_data = {"nodes": nodes, "edges": edges}
# Create canvas data structure
canvas_data = {"nodes": nodes, "edges": edges}
# Convert to JSON
canvas_json = json.dumps(canvas_data, indent=2)
# Convert to JSON
canvas_json = json.dumps(canvas_data, indent=2)
# Write the file using the resource API
logger.info(f"Creating canvas file: {file_path} in project {project}")
response = await call_put(client, f"{project_url}/resource/{file_path}", json=canvas_json)
# Write the file using the resource API
logger.info(f"Creating canvas file: {file_path} in project {project}")
# Send canvas_json as content string, not as json parameter
# The resource endpoint expects Body() string content, not JSON-encoded data
response = await call_put(
client,
f"{project_url}/resource/{file_path}",
content=canvas_json,
headers={"Content-Type": "text/plain"},
)
# Parse response
result = response.json()
logger.debug(result)
# Parse response
result = response.json()
logger.debug(result)
# Build summary
action = "Created" if response.status_code == 201 else "Updated"
summary = [f"# {action}: {file_path}", "\nThe canvas is ready to open in Obsidian."]
# Build summary
action = "Created" if response.status_code == 201 else "Updated"
summary = [f"# {action}: {file_path}", "\nThe canvas is ready to open in Obsidian."]
return "\n".join(summary)
return "\n".join(summary)
+11 -2
View File
@@ -14,6 +14,7 @@ from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.search import search_notes
from basic_memory.mcp.tools.read_note import read_note
from basic_memory.schemas.search import SearchResponse
from basic_memory.config import ConfigManager
def _format_search_results_for_chatgpt(results: SearchResponse) -> List[Dict[str, Any]]:
@@ -90,10 +91,14 @@ async def search(
logger.info(f"ChatGPT search request: query='{query}'")
try:
# ChatGPT tools don't expose project parameter, so use default project
config = ConfigManager().config
default_project = config.default_project
# Call underlying search_notes with sensible defaults for ChatGPT
results = await search_notes.fn(
query=query,
project=None, # Let project resolution happen automatically
project=default_project, # Use default project for ChatGPT
page=1,
page_size=10, # Reasonable default for ChatGPT consumption
search_type="text", # Default to full-text search
@@ -149,10 +154,14 @@ async def fetch(
logger.info(f"ChatGPT fetch request: id='{id}'")
try:
# ChatGPT tools don't expose project parameter, so use default project
config = ConfigManager().config
default_project = config.default_project
# Call underlying read_note function
content = await read_note.fn(
identifier=id,
project=None, # Let project resolution happen automatically
project=default_project, # Use default project for ChatGPT
page=1,
page_size=10, # Default pagination
context=context,
+19 -18
View File
@@ -7,7 +7,7 @@ from fastmcp import Context
from basic_memory.mcp.project_context import get_active_project
from basic_memory.mcp.tools.utils import call_delete
from basic_memory.mcp.server import mcp
from basic_memory.mcp.async_client import client
from basic_memory.mcp.async_client import get_client
from basic_memory.schemas import DeleteEntitiesResponse
@@ -202,23 +202,24 @@ async def delete_note(
with suggestions for finding the correct identifier, including search
commands and alternative formats to try.
"""
active_project = await get_active_project(client, project, context)
project_url = active_project.project_url
async with get_client() as client:
active_project = await get_active_project(client, project, context)
project_url = active_project.project_url
try:
response = await call_delete(client, f"{project_url}/knowledge/entities/{identifier}")
result = DeleteEntitiesResponse.model_validate(response.json())
try:
response = await call_delete(client, f"{project_url}/knowledge/entities/{identifier}")
result = DeleteEntitiesResponse.model_validate(response.json())
if result.deleted:
logger.info(
f"Successfully deleted note: {identifier} in project: {active_project.name}"
)
return True
else:
logger.warning(f"Delete operation completed but note was not deleted: {identifier}")
return False
if result.deleted:
logger.info(
f"Successfully deleted note: {identifier} in project: {active_project.name}"
)
return True
else:
logger.warning(f"Delete operation completed but note was not deleted: {identifier}")
return False
except Exception as e: # pragma: no cover
logger.error(f"Delete failed for '{identifier}': {e}, project: {active_project.name}")
# Return formatted error message for better user experience
return _format_delete_error_response(active_project.name, str(e), identifier)
except Exception as e: # pragma: no cover
logger.error(f"Delete failed for '{identifier}': {e}, project: {active_project.name}")
# Return formatted error message for better user experience
return _format_delete_error_response(active_project.name, str(e), identifier)
+90 -89
View File
@@ -5,7 +5,7 @@ from typing import Optional
from loguru import logger
from fastmcp import Context
from basic_memory.mcp.async_client import client
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.project_context import get_active_project, add_project_metadata
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_patch
@@ -214,106 +214,107 @@ async def edit_note(
search_notes() first to find the correct identifier. The tool provides detailed
error messages with suggestions if operations fail.
"""
active_project = await get_active_project(client, project, context)
project_url = active_project.project_url
async with get_client() as client:
active_project = await get_active_project(client, project, context)
project_url = active_project.project_url
logger.info("MCP tool call", tool="edit_note", identifier=identifier, operation=operation)
logger.info("MCP tool call", tool="edit_note", identifier=identifier, operation=operation)
# Validate operation
valid_operations = ["append", "prepend", "find_replace", "replace_section"]
if operation not in valid_operations:
raise ValueError(
f"Invalid operation '{operation}'. Must be one of: {', '.join(valid_operations)}"
)
# Validate operation
valid_operations = ["append", "prepend", "find_replace", "replace_section"]
if operation not in valid_operations:
raise ValueError(
f"Invalid operation '{operation}'. Must be one of: {', '.join(valid_operations)}"
)
# Validate required parameters for specific operations
if operation == "find_replace" and not find_text:
raise ValueError("find_text parameter is required for find_replace operation")
if operation == "replace_section" and not section:
raise ValueError("section parameter is required for replace_section operation")
# Validate required parameters for specific operations
if operation == "find_replace" and not find_text:
raise ValueError("find_text parameter is required for find_replace operation")
if operation == "replace_section" and not section:
raise ValueError("section parameter is required for replace_section operation")
# Use the PATCH endpoint to edit the entity
try:
# Prepare the edit request data
edit_data = {
"operation": operation,
"content": content,
}
# Use the PATCH endpoint to edit the entity
try:
# Prepare the edit request data
edit_data = {
"operation": operation,
"content": content,
}
# Add optional parameters
if section:
edit_data["section"] = section
if find_text:
edit_data["find_text"] = find_text
if expected_replacements != 1: # Only send if different from default
edit_data["expected_replacements"] = str(expected_replacements)
# Add optional parameters
if section:
edit_data["section"] = section
if find_text:
edit_data["find_text"] = find_text
if expected_replacements != 1: # Only send if different from default
edit_data["expected_replacements"] = str(expected_replacements)
# Call the PATCH endpoint
url = f"{project_url}/knowledge/entities/{identifier}"
response = await call_patch(client, url, json=edit_data)
result = EntityResponse.model_validate(response.json())
# Call the PATCH endpoint
url = f"{project_url}/knowledge/entities/{identifier}"
response = await call_patch(client, url, json=edit_data)
result = EntityResponse.model_validate(response.json())
# Format summary
summary = [
f"# Edited note ({operation})",
f"project: {active_project.name}",
f"file_path: {result.file_path}",
f"permalink: {result.permalink}",
f"checksum: {result.checksum[:8] if result.checksum else 'unknown'}",
]
# Format summary
summary = [
f"# Edited note ({operation})",
f"project: {active_project.name}",
f"file_path: {result.file_path}",
f"permalink: {result.permalink}",
f"checksum: {result.checksum[:8] if result.checksum else 'unknown'}",
]
# Add operation-specific details
if operation == "append":
lines_added = len(content.split("\n"))
summary.append(f"operation: Added {lines_added} lines to end of note")
elif operation == "prepend":
lines_added = len(content.split("\n"))
summary.append(f"operation: Added {lines_added} lines to beginning of note")
elif operation == "find_replace":
# For find_replace, we can't easily count replacements from here
# since we don't have the original content, but the server handled it
summary.append("operation: Find and replace operation completed")
elif operation == "replace_section":
summary.append(f"operation: Replaced content under section '{section}'")
# Add operation-specific details
if operation == "append":
lines_added = len(content.split("\n"))
summary.append(f"operation: Added {lines_added} lines to end of note")
elif operation == "prepend":
lines_added = len(content.split("\n"))
summary.append(f"operation: Added {lines_added} lines to beginning of note")
elif operation == "find_replace":
# For find_replace, we can't easily count replacements from here
# since we don't have the original content, but the server handled it
summary.append("operation: Find and replace operation completed")
elif operation == "replace_section":
summary.append(f"operation: Replaced content under section '{section}'")
# Count observations by category (reuse logic from write_note)
categories = {}
if result.observations:
for obs in result.observations:
categories[obs.category] = categories.get(obs.category, 0) + 1
# Count observations by category (reuse logic from write_note)
categories = {}
if result.observations:
for obs in result.observations:
categories[obs.category] = categories.get(obs.category, 0) + 1
summary.append("\\n## Observations")
for category, count in sorted(categories.items()):
summary.append(f"- {category}: {count}")
summary.append("\\n## Observations")
for category, count in sorted(categories.items()):
summary.append(f"- {category}: {count}")
# Count resolved/unresolved relations
unresolved = 0
resolved = 0
if result.relations:
unresolved = sum(1 for r in result.relations if not r.to_id)
resolved = len(result.relations) - unresolved
# Count resolved/unresolved relations
unresolved = 0
resolved = 0
if result.relations:
unresolved = sum(1 for r in result.relations if not r.to_id)
resolved = len(result.relations) - unresolved
summary.append("\\n## Relations")
summary.append(f"- Resolved: {resolved}")
if unresolved:
summary.append(f"- Unresolved: {unresolved}")
summary.append("\\n## Relations")
summary.append(f"- Resolved: {resolved}")
if unresolved:
summary.append(f"- Unresolved: {unresolved}")
logger.info(
"MCP tool response",
tool="edit_note",
operation=operation,
project=active_project.name,
permalink=result.permalink,
observations_count=len(result.observations),
relations_count=len(result.relations),
status_code=response.status_code,
)
logger.info(
"MCP tool response",
tool="edit_note",
operation=operation,
project=active_project.name,
permalink=result.permalink,
observations_count=len(result.observations),
relations_count=len(result.relations),
status_code=response.status_code,
)
result = "\n".join(summary)
return add_project_metadata(result, active_project.name)
result = "\n".join(summary)
return add_project_metadata(result, active_project.name)
except Exception as e:
logger.error(f"Error editing note: {e}")
return _format_error_response(
str(e), operation, identifier, find_text, expected_replacements, active_project.name
)
except Exception as e:
logger.error(f"Error editing note: {e}")
return _format_error_response(
str(e), operation, identifier, find_text, expected_replacements, active_project.name
)
-44
View File
@@ -1,44 +0,0 @@
from httpx._types import (
HeaderTypes,
)
from loguru import logger
from fastmcp.server.dependencies import get_http_headers
def inject_auth_header(headers: HeaderTypes | None = None) -> HeaderTypes:
"""
Inject JWT token from FastMCP context into headers if available.
Args:
headers: Existing headers dict or None
Returns:
Headers dict with Authorization header added if JWT is available
"""
# Start with existing headers or empty dict
if headers is None:
headers = {}
elif not isinstance(headers, dict):
# Convert other header types to dict
headers = dict(headers) # type: ignore
else:
# Make a copy to avoid modifying the original
headers = headers.copy()
http_headers = get_http_headers()
# Log only non-sensitive header keys for debugging
if logger.opt(lazy=True).debug:
sensitive_headers = {"authorization", "cookie", "x-api-key", "x-auth-token", "api-key"}
safe_headers = {k for k in http_headers.keys() if k.lower() not in sensitive_headers}
logger.debug(f"HTTP headers present: {list(safe_headers)}")
authorization = http_headers.get("Authorization") or http_headers.get("authorization")
if authorization:
headers["Authorization"] = authorization # type: ignore
# Log only that auth was injected, not the token value
logger.debug("Injected authorization header into request")
else:
logger.debug("No authorization header found in request")
return headers
+86 -83
View File
@@ -5,7 +5,7 @@ from typing import Optional
from loguru import logger
from fastmcp import Context
from basic_memory.mcp.async_client import client
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.project_context import get_active_project
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_get
@@ -63,102 +63,105 @@ async def list_directory(
Raises:
ToolError: If project doesn't exist or directory path is invalid
"""
active_project = await get_active_project(client, project, context)
project_url = active_project.project_url
async with get_client() as client:
active_project = await get_active_project(client, project, context)
project_url = active_project.project_url
# Prepare query parameters
params = {
"dir_name": dir_name,
"depth": str(depth),
}
if file_name_glob:
params["file_name_glob"] = file_name_glob
logger.debug(
f"Listing directory '{dir_name}' in project {project} with depth={depth}, glob='{file_name_glob}'"
)
# Call the API endpoint
response = await call_get(
client,
f"{project_url}/directory/list",
params=params,
)
nodes = response.json()
if not nodes:
filter_desc = ""
# Prepare query parameters
params = {
"dir_name": dir_name,
"depth": str(depth),
}
if file_name_glob:
filter_desc = f" matching '{file_name_glob}'"
return f"No files found in directory '{dir_name}'{filter_desc}"
params["file_name_glob"] = file_name_glob
# Format the results
output_lines = []
if file_name_glob:
output_lines.append(f"Files in '{dir_name}' matching '{file_name_glob}' (depth {depth}):")
else:
output_lines.append(f"Contents of '{dir_name}' (depth {depth}):")
output_lines.append("")
logger.debug(
f"Listing directory '{dir_name}' in project {project} with depth={depth}, glob='{file_name_glob}'"
)
# Group by type and sort
directories = [n for n in nodes if n["type"] == "directory"]
files = [n for n in nodes if n["type"] == "file"]
# Call the API endpoint
response = await call_get(
client,
f"{project_url}/directory/list",
params=params,
)
# Sort by name
directories.sort(key=lambda x: x["name"])
files.sort(key=lambda x: x["name"])
nodes = response.json()
# Display directories first
for node in directories:
path_display = node["directory_path"]
output_lines.append(f"📁 {node['name']:<30} {path_display}")
if not nodes:
filter_desc = ""
if file_name_glob:
filter_desc = f" matching '{file_name_glob}'"
return f"No files found in directory '{dir_name}'{filter_desc}"
# Add separator if we have both directories and files
if directories and files:
# Format the results
output_lines = []
if file_name_glob:
output_lines.append(
f"Files in '{dir_name}' matching '{file_name_glob}' (depth {depth}):"
)
else:
output_lines.append(f"Contents of '{dir_name}' (depth {depth}):")
output_lines.append("")
# Display files with metadata
for node in files:
path_display = node["directory_path"]
title = node.get("title", "")
updated = node.get("updated_at", "")
# Group by type and sort
directories = [n for n in nodes if n["type"] == "directory"]
files = [n for n in nodes if n["type"] == "file"]
# Remove leading slash if present, requesting the file via read_note does not use the beginning slash'
if path_display.startswith("/"):
path_display = path_display[1:]
# Sort by name
directories.sort(key=lambda x: x["name"])
files.sort(key=lambda x: x["name"])
# Format date if available
date_str = ""
if updated:
try:
from datetime import datetime
# Display directories first
for node in directories:
path_display = node["directory_path"]
output_lines.append(f"📁 {node['name']:<30} {path_display}")
dt = datetime.fromisoformat(updated.replace("Z", "+00:00"))
date_str = dt.strftime("%Y-%m-%d")
except Exception: # pragma: no cover
date_str = updated[:10] if len(updated) >= 10 else ""
# Add separator if we have both directories and files
if directories and files:
output_lines.append("")
# Create formatted line
file_line = f"📄 {node['name']:<30} {path_display}"
if title and title != node["name"]:
file_line += f" | {title}"
if date_str:
file_line += f" | {date_str}"
# Display files with metadata
for node in files:
path_display = node["directory_path"]
title = node.get("title", "")
updated = node.get("updated_at", "")
output_lines.append(file_line)
# Remove leading slash if present, requesting the file via read_note does not use the beginning slash'
if path_display.startswith("/"):
path_display = path_display[1:]
# Add summary
output_lines.append("")
total_count = len(directories) + len(files)
summary_parts = []
if directories:
summary_parts.append(
f"{len(directories)} director{'y' if len(directories) == 1 else 'ies'}"
)
if files:
summary_parts.append(f"{len(files)} file{'s' if len(files) != 1 else ''}")
# Format date if available
date_str = ""
if updated:
try:
from datetime import datetime
output_lines.append(f"Total: {total_count} items ({', '.join(summary_parts)})")
dt = datetime.fromisoformat(updated.replace("Z", "+00:00"))
date_str = dt.strftime("%Y-%m-%d")
except Exception: # pragma: no cover
date_str = updated[:10] if len(updated) >= 10 else ""
return "\n".join(output_lines)
# Create formatted line
file_line = f"📄 {node['name']:<30} {path_display}"
if title and title != node["name"]:
file_line += f" | {title}"
if date_str:
file_line += f" | {date_str}"
output_lines.append(file_line)
# Add summary
output_lines.append("")
total_count = len(directories) + len(files)
summary_parts = []
if directories:
summary_parts.append(
f"{len(directories)} director{'y' if len(directories) == 1 else 'ies'}"
)
if files:
summary_parts.append(f"{len(files)} file{'s' if len(files) != 1 else ''}")
output_lines.append(f"Total: {total_count} items ({', '.join(summary_parts)})")
return "\n".join(output_lines)
+121 -119
View File
@@ -6,7 +6,7 @@ from typing import Optional
from loguru import logger
from fastmcp import Context
from basic_memory.mcp.async_client import client
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_post, call_get
from basic_memory.mcp.project_context import get_active_project
@@ -16,11 +16,12 @@ from basic_memory.utils import validate_project_path
async def _detect_cross_project_move_attempt(
identifier: str, destination_path: str, current_project: str
client, identifier: str, destination_path: str, current_project: str
) -> Optional[str]:
"""Detect potential cross-project move attempts and return guidance.
Args:
client: The AsyncClient instance
identifier: The note identifier being moved
destination_path: The destination path
current_project: The current active project
@@ -394,20 +395,21 @@ async def move_note(
- Re-indexes the entity for search
- Maintains all observations and relations
"""
logger.debug(f"Moving note: {identifier} to {destination_path} in project: {project}")
async with get_client() as client:
logger.debug(f"Moving note: {identifier} to {destination_path} in project: {project}")
active_project = await get_active_project(client, project, context)
project_url = active_project.project_url
active_project = await get_active_project(client, project, context)
project_url = active_project.project_url
# Validate destination path to prevent path traversal attacks
project_path = active_project.home
if not validate_project_path(destination_path, project_path):
logger.warning(
"Attempted path traversal attack blocked",
destination_path=destination_path,
project=active_project.name,
)
return f"""# Move Failed - Security Validation Error
# Validate destination path to prevent path traversal attacks
project_path = active_project.home
if not validate_project_path(destination_path, project_path):
logger.warning(
"Attempted path traversal attack blocked",
destination_path=destination_path,
project=active_project.name,
)
return f"""# Move Failed - Security Validation Error
The destination path '{destination_path}' is not allowed - paths must stay within project boundaries.
@@ -421,123 +423,123 @@ The destination path '{destination_path}' is not allowed - paths must stay withi
move_note("{identifier}", "notes/{destination_path.split("/")[-1] if "/" in destination_path else destination_path}")
```"""
# Check for potential cross-project move attempts
cross_project_error = await _detect_cross_project_move_attempt(
identifier, destination_path, active_project.name
)
if cross_project_error:
logger.info(f"Detected cross-project move attempt: {identifier} -> {destination_path}")
return cross_project_error
# Get the source entity information for extension validation
source_ext = "md" # Default to .md if we can't determine source extension
try:
# Fetch source entity information to get the current file extension
url = f"{project_url}/knowledge/entities/{identifier}"
response = await call_get(client, url)
source_entity = EntityResponse.model_validate(response.json())
if "." in source_entity.file_path:
source_ext = source_entity.file_path.split(".")[-1]
except Exception as e:
# If we can't fetch the source entity, default to .md extension
logger.debug(f"Could not fetch source entity for extension check: {e}")
# Validate that destination path includes a file extension
if "." not in destination_path or not destination_path.split(".")[-1]:
logger.warning(f"Move failed - no file extension provided: {destination_path}")
return dedent(f"""
# Move Failed - File Extension Required
The destination path '{destination_path}' must include a file extension (e.g., '.md').
## Valid examples:
- `notes/my-note.md`
- `projects/meeting-2025.txt`
- `archive/old-program.sh`
## Try again with extension:
```
move_note("{identifier}", "{destination_path}.{source_ext}")
```
All examples in Basic Memory expect file extensions to be explicitly provided.
""").strip()
# Get the source entity to check its file extension
try:
# Fetch source entity information
url = f"{project_url}/knowledge/entities/{identifier}"
response = await call_get(client, url)
source_entity = EntityResponse.model_validate(response.json())
# Extract file extensions
source_ext = (
source_entity.file_path.split(".")[-1] if "." in source_entity.file_path else ""
# Check for potential cross-project move attempts
cross_project_error = await _detect_cross_project_move_attempt(
client, identifier, destination_path, active_project.name
)
dest_ext = destination_path.split(".")[-1] if "." in destination_path else ""
if cross_project_error:
logger.info(f"Detected cross-project move attempt: {identifier} -> {destination_path}")
return cross_project_error
# Check if extensions match
if source_ext and dest_ext and source_ext.lower() != dest_ext.lower():
logger.warning(
f"Move failed - file extension mismatch: source={source_ext}, dest={dest_ext}"
)
# Get the source entity information for extension validation
source_ext = "md" # Default to .md if we can't determine source extension
try:
# Fetch source entity information to get the current file extension
url = f"{project_url}/knowledge/entities/{identifier}"
response = await call_get(client, url)
source_entity = EntityResponse.model_validate(response.json())
if "." in source_entity.file_path:
source_ext = source_entity.file_path.split(".")[-1]
except Exception as e:
# If we can't fetch the source entity, default to .md extension
logger.debug(f"Could not fetch source entity for extension check: {e}")
# Validate that destination path includes a file extension
if "." not in destination_path or not destination_path.split(".")[-1]:
logger.warning(f"Move failed - no file extension provided: {destination_path}")
return dedent(f"""
# Move Failed - File Extension Mismatch
# Move Failed - File Extension Required
The destination file extension '.{dest_ext}' does not match the source file extension '.{source_ext}'.
The destination path '{destination_path}' must include a file extension (e.g., '.md').
To preserve file type consistency, the destination must have the same extension as the source.
## Valid examples:
- `notes/my-note.md`
- `projects/meeting-2025.txt`
- `archive/old-program.sh`
## Source file:
- Path: `{source_entity.file_path}`
- Extension: `.{source_ext}`
## Try again with matching extension:
## Try again with extension:
```
move_note("{identifier}", "{destination_path.rsplit(".", 1)[0]}.{source_ext}")
move_note("{identifier}", "{destination_path}.{source_ext}")
```
All examples in Basic Memory expect file extensions to be explicitly provided.
""").strip()
except Exception as e:
# If we can't fetch the source entity, log it but continue
# This might happen if the identifier is not yet resolved
logger.debug(f"Could not fetch source entity for extension check: {e}")
try:
# Prepare move request
move_data = {
"identifier": identifier,
"destination_path": destination_path,
"project": active_project.name,
}
# Get the source entity to check its file extension
try:
# Fetch source entity information
url = f"{project_url}/knowledge/entities/{identifier}"
response = await call_get(client, url)
source_entity = EntityResponse.model_validate(response.json())
# Call the move API endpoint
url = f"{project_url}/knowledge/move"
response = await call_post(client, url, json=move_data)
result = EntityResponse.model_validate(response.json())
# Extract file extensions
source_ext = (
source_entity.file_path.split(".")[-1] if "." in source_entity.file_path else ""
)
dest_ext = destination_path.split(".")[-1] if "." in destination_path else ""
# Build success message
result_lines = [
"✅ Note moved successfully",
"",
f"📁 **{identifier}** → **{result.file_path}**",
f"🔗 Permalink: {result.permalink}",
"📊 Database and search index updated",
"",
f"<!-- Project: {active_project.name} -->",
]
# Check if extensions match
if source_ext and dest_ext and source_ext.lower() != dest_ext.lower():
logger.warning(
f"Move failed - file extension mismatch: source={source_ext}, dest={dest_ext}"
)
return dedent(f"""
# Move Failed - File Extension Mismatch
# Log the operation
logger.info(
"Move note completed",
identifier=identifier,
destination_path=destination_path,
project=active_project.name,
status_code=response.status_code,
)
The destination file extension '.{dest_ext}' does not match the source file extension '.{source_ext}'.
return "\n".join(result_lines)
To preserve file type consistency, the destination must have the same extension as the source.
except Exception as e:
logger.error(f"Move failed for '{identifier}' to '{destination_path}': {e}")
# Return formatted error message for better user experience
return _format_move_error_response(str(e), identifier, destination_path)
## Source file:
- Path: `{source_entity.file_path}`
- Extension: `.{source_ext}`
## Try again with matching extension:
```
move_note("{identifier}", "{destination_path.rsplit(".", 1)[0]}.{source_ext}")
```
""").strip()
except Exception as e:
# If we can't fetch the source entity, log it but continue
# This might happen if the identifier is not yet resolved
logger.debug(f"Could not fetch source entity for extension check: {e}")
try:
# Prepare move request
move_data = {
"identifier": identifier,
"destination_path": destination_path,
"project": active_project.name,
}
# Call the move API endpoint
url = f"{project_url}/knowledge/move"
response = await call_post(client, url, json=move_data)
result = EntityResponse.model_validate(response.json())
# Build success message
result_lines = [
"✅ Note moved successfully",
"",
f"📁 **{identifier}** → **{result.file_path}**",
f"🔗 Permalink: {result.permalink}",
"📊 Database and search index updated",
"",
f"<!-- Project: {active_project.name} -->",
]
# Log the operation
logger.info(
"Move note completed",
identifier=identifier,
destination_path=destination_path,
project=active_project.name,
status_code=response.status_code,
)
return "\n".join(result_lines)
except Exception as e:
logger.error(f"Move failed for '{identifier}' to '{destination_path}': {e}")
# Return formatted error message for better user experience
return _format_move_error_response(str(e), identifier, destination_path)
@@ -7,7 +7,7 @@ and manage project context during conversations.
import os
from fastmcp import Context
from basic_memory.mcp.async_client import client
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_get, call_post, call_delete
from basic_memory.schemas.project_info import (
@@ -40,34 +40,35 @@ async def list_memory_projects(context: Context | None = None) -> str:
Example:
list_memory_projects()
"""
if context: # pragma: no cover
await context.info("Listing all available projects")
async with get_client() as client:
if context: # pragma: no cover
await context.info("Listing all available projects")
# Check if server is constrained to a specific project
constrained_project = os.environ.get("BASIC_MEMORY_MCP_PROJECT")
# Check if server is constrained to a specific project
constrained_project = os.environ.get("BASIC_MEMORY_MCP_PROJECT")
# Get projects from API
response = await call_get(client, "/projects/projects")
project_list = ProjectList.model_validate(response.json())
# Get projects from API
response = await call_get(client, "/projects/projects")
project_list = ProjectList.model_validate(response.json())
if constrained_project:
result = f"Project: {constrained_project}\n\n"
result += "Note: This MCP server is constrained to a single project.\n"
result += "All operations will automatically use this project."
else:
# Show all projects with session guidance
result = "Available projects:\n"
if constrained_project:
result = f"Project: {constrained_project}\n\n"
result += "Note: This MCP server is constrained to a single project.\n"
result += "All operations will automatically use this project."
else:
# Show all projects with session guidance
result = "Available projects:\n"
for project in project_list.projects:
result += f"{project.name}\n"
for project in project_list.projects:
result += f"{project.name}\n"
result += "\n" + "" * 40 + "\n"
result += "Next: Ask which project to use for this session.\n"
result += "Example: 'Which project should I use for this task?'\n\n"
result += "Session reminder: Track the selected project for all subsequent operations in this conversation.\n"
result += "The user can say 'switch to [project]' to change projects."
result += "\n" + "" * 40 + "\n"
result += "Next: Ask which project to use for this session.\n"
result += "Example: 'Which project should I use for this task?'\n\n"
result += "Session reminder: Track the selected project for all subsequent operations in this conversation.\n"
result += "The user can say 'switch to [project]' to change projects."
return result
return result
@mcp.tool("create_memory_project")
@@ -91,37 +92,38 @@ async def create_memory_project(
create_memory_project("my-research", "~/Documents/research")
create_memory_project("work-notes", "/home/user/work", set_default=True)
"""
# Check if server is constrained to a specific project
constrained_project = os.environ.get("BASIC_MEMORY_MCP_PROJECT")
if constrained_project:
return f'# Error\n\nProject creation disabled - MCP server is constrained to project \'{constrained_project}\'.\nUse the CLI to create projects: `basic-memory project add "{project_name}" "{project_path}"`'
async with get_client() as client:
# Check if server is constrained to a specific project
constrained_project = os.environ.get("BASIC_MEMORY_MCP_PROJECT")
if constrained_project:
return f'# Error\n\nProject creation disabled - MCP server is constrained to project \'{constrained_project}\'.\nUse the CLI to create projects: `basic-memory project add "{project_name}" "{project_path}"`'
if context: # pragma: no cover
await context.info(f"Creating project: {project_name} at {project_path}")
if context: # pragma: no cover
await context.info(f"Creating project: {project_name} at {project_path}")
# Create the project request
project_request = ProjectInfoRequest(
name=project_name, path=project_path, set_default=set_default
)
# Create the project request
project_request = ProjectInfoRequest(
name=project_name, path=project_path, set_default=set_default
)
# Call API to create project
response = await call_post(client, "/projects/projects", json=project_request.model_dump())
status_response = ProjectStatusResponse.model_validate(response.json())
# Call API to create project
response = await call_post(client, "/projects/projects", json=project_request.model_dump())
status_response = ProjectStatusResponse.model_validate(response.json())
result = f"{status_response.message}\n\n"
result = f"{status_response.message}\n\n"
if status_response.new_project:
result += "Project Details:\n"
result += f"• Name: {status_response.new_project.name}\n"
result += f"• Path: {status_response.new_project.path}\n"
if status_response.new_project:
result += "Project Details:\n"
result += f"• Name: {status_response.new_project.name}\n"
result += f"• Path: {status_response.new_project.path}\n"
if set_default:
result += "• Set as default project\n"
if set_default:
result += "• Set as default project\n"
result += "\nProject is now available for use in tool calls.\n"
result += f"Use '{project_name}' as the project parameter in MCP tool calls.\n"
result += "\nProject is now available for use in tool calls.\n"
result += f"Use '{project_name}' as the project parameter in MCP tool calls.\n"
return result
return result
@mcp.tool()
@@ -145,53 +147,54 @@ async def delete_project(project_name: str, context: Context | None = None) -> s
This action cannot be undone. The project will need to be re-added
to access its content through Basic Memory again.
"""
# Check if server is constrained to a specific project
constrained_project = os.environ.get("BASIC_MEMORY_MCP_PROJECT")
if constrained_project:
return f"# Error\n\nProject deletion disabled - MCP server is constrained to project '{constrained_project}'.\nUse the CLI to delete projects: `basic-memory project remove \"{project_name}\"`"
async with get_client() as client:
# Check if server is constrained to a specific project
constrained_project = os.environ.get("BASIC_MEMORY_MCP_PROJECT")
if constrained_project:
return f"# Error\n\nProject deletion disabled - MCP server is constrained to project '{constrained_project}'.\nUse the CLI to delete projects: `basic-memory project remove \"{project_name}\"`"
if context: # pragma: no cover
await context.info(f"Deleting project: {project_name}")
if context: # pragma: no cover
await context.info(f"Deleting project: {project_name}")
# Get project info before deletion to validate it exists
response = await call_get(client, "/projects/projects")
project_list = ProjectList.model_validate(response.json())
# Get project info before deletion to validate it exists
response = await call_get(client, "/projects/projects")
project_list = ProjectList.model_validate(response.json())
# Find the project by name (case-insensitive) or permalink - same logic as switch_project
project_permalink = generate_permalink(project_name)
target_project = None
for p in project_list.projects:
# Match by permalink (handles case-insensitive input)
if p.permalink == project_permalink:
target_project = p
break
# Also match by name comparison (case-insensitive)
if p.name.lower() == project_name.lower():
target_project = p
break
# Find the project by name (case-insensitive) or permalink - same logic as switch_project
project_permalink = generate_permalink(project_name)
target_project = None
for p in project_list.projects:
# Match by permalink (handles case-insensitive input)
if p.permalink == project_permalink:
target_project = p
break
# Also match by name comparison (case-insensitive)
if p.name.lower() == project_name.lower():
target_project = p
break
if not target_project:
available_projects = [p.name for p in project_list.projects]
raise ValueError(
f"Project '{project_name}' not found. Available projects: {', '.join(available_projects)}"
)
if not target_project:
available_projects = [p.name for p in project_list.projects]
raise ValueError(
f"Project '{project_name}' not found. Available projects: {', '.join(available_projects)}"
)
# Call API to delete project using URL encoding for special characters
from urllib.parse import quote
# Call API to delete project using URL encoding for special characters
from urllib.parse import quote
encoded_name = quote(target_project.name, safe="")
response = await call_delete(client, f"/projects/{encoded_name}")
status_response = ProjectStatusResponse.model_validate(response.json())
encoded_name = quote(target_project.name, safe="")
response = await call_delete(client, f"/projects/{encoded_name}")
status_response = ProjectStatusResponse.model_validate(response.json())
result = f"{status_response.message}\n\n"
result = f"{status_response.message}\n\n"
if status_response.old_project:
result += "Removed project details:\n"
result += f"• Name: {status_response.old_project.name}\n"
if hasattr(status_response.old_project, "path"):
result += f"• Path: {status_response.old_project.path}\n"
if status_response.old_project:
result += "Removed project details:\n"
result += f"• Name: {status_response.old_project.name}\n"
if hasattr(status_response.old_project, "path"):
result += f"• Path: {status_response.old_project.path}\n"
result += "Files remain on disk but project is no longer tracked by Basic Memory.\n"
result += "Re-add the project to access its content again.\n"
result += "Files remain on disk but project is no longer tracked by Basic Memory.\n"
result += "Re-add the project to access its content again.\n"
return result
return result
+64 -63
View File
@@ -16,7 +16,7 @@ from fastmcp import Context
from basic_memory.mcp.project_context import get_active_project
from basic_memory.mcp.server import mcp
from basic_memory.mcp.async_client import client
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.tools.utils import call_get
from basic_memory.schemas.memory import memory_url_path
from basic_memory.utils import validate_project_path
@@ -201,70 +201,71 @@ async def read_content(
"""
logger.info("Reading file", path=path, project=project)
active_project = await get_active_project(client, project, context)
project_url = active_project.project_url
async with get_client() as client:
active_project = await get_active_project(client, project, context)
project_url = active_project.project_url
url = memory_url_path(path)
url = memory_url_path(path)
# Validate path to prevent path traversal attacks
project_path = active_project.home
if not validate_project_path(url, project_path):
logger.warning(
"Attempted path traversal attack blocked",
path=path,
url=url,
project=active_project.name,
)
return {
"type": "error",
"error": f"Path '{path}' is not allowed - paths must stay within project boundaries",
}
response = await call_get(client, f"{project_url}/resource/{url}")
content_type = response.headers.get("content-type", "application/octet-stream")
content_length = int(response.headers.get("content-length", 0))
logger.debug("Resource metadata", content_type=content_type, size=content_length, path=path)
# Handle text or json
if content_type.startswith("text/") or content_type == "application/json":
logger.debug("Processing text resource")
return {
"type": "text",
"text": response.text,
"content_type": content_type,
"encoding": "utf-8",
}
# Handle images
elif content_type.startswith("image/"):
logger.debug("Processing image")
img = PILImage.open(io.BytesIO(response.content))
img_bytes = optimize_image(img, content_length)
return {
"type": "image",
"source": {
"type": "base64",
"media_type": "image/jpeg",
"data": base64.b64encode(img_bytes).decode("utf-8"),
},
}
# Handle other file types
else:
logger.debug(f"Processing binary resource content_type {content_type}")
if content_length > 350000: # pragma: no cover
logger.warning("Document too large for response", size=content_length)
# Validate path to prevent path traversal attacks
project_path = active_project.home
if not validate_project_path(url, project_path):
logger.warning(
"Attempted path traversal attack blocked",
path=path,
url=url,
project=active_project.name,
)
return {
"type": "error",
"error": f"Document size {content_length} bytes exceeds maximum allowed size",
"error": f"Path '{path}' is not allowed - paths must stay within project boundaries",
}
response = await call_get(client, f"{project_url}/resource/{url}")
content_type = response.headers.get("content-type", "application/octet-stream")
content_length = int(response.headers.get("content-length", 0))
logger.debug("Resource metadata", content_type=content_type, size=content_length, path=path)
# Handle text or json
if content_type.startswith("text/") or content_type == "application/json":
logger.debug("Processing text resource")
return {
"type": "text",
"text": response.text,
"content_type": content_type,
"encoding": "utf-8",
}
# Handle images
elif content_type.startswith("image/"):
logger.debug("Processing image")
img = PILImage.open(io.BytesIO(response.content))
img_bytes = optimize_image(img, content_length)
return {
"type": "image",
"source": {
"type": "base64",
"media_type": "image/jpeg",
"data": base64.b64encode(img_bytes).decode("utf-8"),
},
}
# Handle other file types
else:
logger.debug(f"Processing binary resource content_type {content_type}")
if content_length > 350000: # pragma: no cover
logger.warning("Document too large for response", size=content_length)
return {
"type": "error",
"error": f"Document size {content_length} bytes exceeds maximum allowed size",
}
return {
"type": "document",
"source": {
"type": "base64",
"media_type": content_type,
"data": base64.b64encode(response.content).decode("utf-8"),
},
}
return {
"type": "document",
"source": {
"type": "base64",
"media_type": content_type,
"data": base64.b64encode(response.content).decode("utf-8"),
},
}
+73 -81
View File
@@ -6,7 +6,7 @@ from typing import Optional
from loguru import logger
from fastmcp import Context
from basic_memory.mcp.async_client import client
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.project_context import get_active_project
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.search import search_notes
@@ -77,96 +77,88 @@ async def read_note(
If the exact note isn't found, this tool provides helpful suggestions
including related notes, search commands, and note creation templates.
"""
async with get_client() as client:
# Get and validate the project
active_project = await get_active_project(client, project, context)
# Get and validate the project
active_project = await get_active_project(client, project, context)
# Validate identifier to prevent path traversal attacks
# We need to check both the raw identifier and the processed path
processed_path = memory_url_path(identifier)
project_path = active_project.home
# Validate identifier to prevent path traversal attacks
# We need to check both the raw identifier and the processed path
processed_path = memory_url_path(identifier)
project_path = active_project.home
if not validate_project_path(identifier, project_path) or not validate_project_path(
processed_path, project_path
):
logger.warning(
"Attempted path traversal attack blocked",
identifier=identifier,
processed_path=processed_path,
project=active_project.name,
)
return f"# Error\n\nIdentifier '{identifier}' is not allowed - paths must stay within project boundaries"
if not validate_project_path(identifier, project_path) or not validate_project_path(
processed_path, project_path
):
logger.warning(
"Attempted path traversal attack blocked",
identifier=identifier,
processed_path=processed_path,
project=active_project.name,
)
return f"# Error\n\nIdentifier '{identifier}' is not allowed - paths must stay within project boundaries"
project_url = active_project.project_url
# Check migration status and wait briefly if needed
from basic_memory.mcp.tools.utils import wait_for_migration_or_return_status
# Get the file via REST API - first try direct permalink lookup
entity_path = memory_url_path(identifier)
path = f"{project_url}/resource/{entity_path}"
logger.info(f"Attempting to read note from Project: {active_project.name} URL: {path}")
migration_status = await wait_for_migration_or_return_status(
timeout=5.0, project_name=active_project.name
)
if migration_status: # pragma: no cover
return f"# System Status\n\n{migration_status}\n\nPlease wait for migration to complete before reading notes."
project_url = active_project.project_url
try:
# Try direct lookup first
response = await call_get(client, path, params={"page": page, "page_size": page_size})
# Get the file via REST API - first try direct permalink lookup
entity_path = memory_url_path(identifier)
path = f"{project_url}/resource/{entity_path}"
logger.info(f"Attempting to read note from Project: {active_project.name} URL: {path}")
# If successful, return the content
if response.status_code == 200:
logger.info("Returning read_note result from resource: {path}", path=entity_path)
return response.text
except Exception as e: # pragma: no cover
logger.info(f"Direct lookup failed for '{path}': {e}")
# Continue to fallback methods
try:
# Try direct lookup first
response = await call_get(client, path, params={"page": page, "page_size": page_size})
# If successful, return the content
if response.status_code == 200:
logger.info("Returning read_note result from resource: {path}", path=entity_path)
return response.text
except Exception as e: # pragma: no cover
logger.info(f"Direct lookup failed for '{path}': {e}")
# Continue to fallback methods
# Fallback 1: Try title search via API
logger.info(f"Search title for: {identifier}")
title_results = await search_notes.fn(
query=identifier, search_type="title", project=project, context=context
)
# Handle both SearchResponse object and error strings
if title_results and hasattr(title_results, "results") and title_results.results:
result = title_results.results[0] # Get the first/best match
if result.permalink:
try:
# Try to fetch the content using the found permalink
path = f"{project_url}/resource/{result.permalink}"
response = await call_get(
client, path, params={"page": page, "page_size": page_size}
)
if response.status_code == 200:
logger.info(f"Found note by title search: {result.permalink}")
return response.text
except Exception as e: # pragma: no cover
logger.info(
f"Failed to fetch content for found title match {result.permalink}: {e}"
)
else:
logger.info(
f"No results in title search for: {identifier} in project {active_project.name}"
# Fallback 1: Try title search via API
logger.info(f"Search title for: {identifier}")
title_results = await search_notes.fn(
query=identifier, search_type="title", project=project, context=context
)
# Fallback 2: Text search as a last resort
logger.info(f"Title search failed, trying text search for: {identifier}")
text_results = await search_notes.fn(
query=identifier, search_type="text", project=project, context=context
)
# Handle both SearchResponse object and error strings
if title_results and hasattr(title_results, "results") and title_results.results:
result = title_results.results[0] # Get the first/best match
if result.permalink:
try:
# Try to fetch the content using the found permalink
path = f"{project_url}/resource/{result.permalink}"
response = await call_get(
client, path, params={"page": page, "page_size": page_size}
)
# We didn't find a direct match, construct a helpful error message
# Handle both SearchResponse object and error strings
if not text_results or not hasattr(text_results, "results") or not text_results.results:
# No results at all
return format_not_found_message(active_project.name, identifier)
else:
# We found some related results
return format_related_results(active_project.name, identifier, text_results.results[:5])
if response.status_code == 200:
logger.info(f"Found note by title search: {result.permalink}")
return response.text
except Exception as e: # pragma: no cover
logger.info(
f"Failed to fetch content for found title match {result.permalink}: {e}"
)
else:
logger.info(
f"No results in title search for: {identifier} in project {active_project.name}"
)
# Fallback 2: Text search as a last resort
logger.info(f"Title search failed, trying text search for: {identifier}")
text_results = await search_notes.fn(
query=identifier, search_type="text", project=project, context=context
)
# We didn't find a direct match, construct a helpful error message
# Handle both SearchResponse object and error strings
if not text_results or not hasattr(text_results, "results") or not text_results.results:
# No results at all
return format_not_found_message(active_project.name, identifier)
else:
# We found some related results
return format_related_results(active_project.name, identifier, text_results.results[:5])
def format_not_found_message(project: str | None, identifier: str) -> str:
+145 -141
View File
@@ -5,7 +5,7 @@ from typing import List, Union, Optional
from loguru import logger
from fastmcp import Context
from basic_memory.mcp.async_client import client
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.project_context import get_active_project, resolve_project_parameter
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_get
@@ -98,162 +98,166 @@ async def recent_activity(
- For focused queries, consider using build_context with a specific URI
- Max timeframe is 1 year in the past
"""
# Build common parameters for API calls
params = {
"page": 1,
"page_size": 10,
"max_related": 10,
}
if depth:
params["depth"] = depth
if timeframe:
params["timeframe"] = timeframe # pyright: ignore
async with get_client() as client:
# Build common parameters for API calls
params = {
"page": 1,
"page_size": 10,
"max_related": 10,
}
if depth:
params["depth"] = depth
if timeframe:
params["timeframe"] = timeframe # pyright: ignore
# Validate and convert type parameter
if type:
# Convert single string to list
if isinstance(type, str):
type_list = [type]
else:
type_list = type
# Validate each type against SearchItemType enum
validated_types = []
for t in type_list:
try:
# Try to convert string to enum
if isinstance(t, str):
validated_types.append(SearchItemType(t.lower()))
except ValueError:
valid_types = [t.value for t in SearchItemType]
raise ValueError(f"Invalid type: {t}. Valid types are: {valid_types}")
# Add validated types to params
params["type"] = [t.value for t in validated_types] # pyright: ignore
# Resolve project parameter using the three-tier hierarchy
resolved_project = await resolve_project_parameter(project)
if resolved_project is None:
# Discovery Mode: Get activity across all projects
logger.info(
f"Getting recent activity across all projects: type={type}, depth={depth}, timeframe={timeframe}"
)
# Get list of all projects
response = await call_get(client, "/projects/projects")
project_list = ProjectList.model_validate(response.json())
projects_activity = {}
total_items = 0
total_entities = 0
total_relations = 0
total_observations = 0
most_active_project = None
most_active_count = 0
active_projects = 0
# Query each project's activity
for project_info in project_list.projects:
project_activity = await _get_project_activity(client, project_info, params, depth)
projects_activity[project_info.name] = project_activity
# Aggregate stats
item_count = project_activity.item_count
if item_count > 0:
active_projects += 1
total_items += item_count
# Count by type
for result in project_activity.activity.results:
if result.primary_result.type == "entity":
total_entities += 1
elif result.primary_result.type == "relation":
total_relations += 1
elif result.primary_result.type == "observation":
total_observations += 1
# Track most active project
if item_count > most_active_count:
most_active_count = item_count
most_active_project = project_info.name
# Build summary stats
summary = ActivityStats(
total_projects=len(project_list.projects),
active_projects=active_projects,
most_active_project=most_active_project,
total_items=total_items,
total_entities=total_entities,
total_relations=total_relations,
total_observations=total_observations,
)
# Generate guidance for the assistant
guidance_lines = ["\n" + "" * 40]
if most_active_project and most_active_count > 0:
guidance_lines.extend(
[
f"Suggested project: '{most_active_project}' (most active with {most_active_count} items)",
f"Ask user: 'Should I use {most_active_project} for this task, or would you prefer a different project?'",
]
)
elif active_projects > 0:
# Has activity but no clear most active project
active_project_names = [
name for name, activity in projects_activity.items() if activity.item_count > 0
]
if len(active_project_names) == 1:
guidance_lines.extend(
[
f"Suggested project: '{active_project_names[0]}' (only active project)",
f"Ask user: 'Should I use {active_project_names[0]} for this task?'",
]
)
# Validate and convert type parameter
if type:
# Convert single string to list
if isinstance(type, str):
type_list = [type]
else:
type_list = type
# Validate each type against SearchItemType enum
validated_types = []
for t in type_list:
try:
# Try to convert string to enum
if isinstance(t, str):
validated_types.append(SearchItemType(t.lower()))
except ValueError:
valid_types = [t.value for t in SearchItemType]
raise ValueError(f"Invalid type: {t}. Valid types are: {valid_types}")
# Add validated types to params
params["type"] = [t.value for t in validated_types] # pyright: ignore
# Resolve project parameter using the three-tier hierarchy
resolved_project = await resolve_project_parameter(project)
if resolved_project is None:
# Discovery Mode: Get activity across all projects
logger.info(
f"Getting recent activity across all projects: type={type}, depth={depth}, timeframe={timeframe}"
)
# Get list of all projects
response = await call_get(client, "/projects/projects")
project_list = ProjectList.model_validate(response.json())
projects_activity = {}
total_items = 0
total_entities = 0
total_relations = 0
total_observations = 0
most_active_project = None
most_active_count = 0
active_projects = 0
# Query each project's activity
for project_info in project_list.projects:
project_activity = await _get_project_activity(client, project_info, params, depth)
projects_activity[project_info.name] = project_activity
# Aggregate stats
item_count = project_activity.item_count
if item_count > 0:
active_projects += 1
total_items += item_count
# Count by type
for result in project_activity.activity.results:
if result.primary_result.type == "entity":
total_entities += 1
elif result.primary_result.type == "relation":
total_relations += 1
elif result.primary_result.type == "observation":
total_observations += 1
# Track most active project
if item_count > most_active_count:
most_active_count = item_count
most_active_project = project_info.name
# Build summary stats
summary = ActivityStats(
total_projects=len(project_list.projects),
active_projects=active_projects,
most_active_project=most_active_project,
total_items=total_items,
total_entities=total_entities,
total_relations=total_relations,
total_observations=total_observations,
)
# Generate guidance for the assistant
guidance_lines = ["\n" + "" * 40]
if most_active_project and most_active_count > 0:
guidance_lines.extend(
[
f"Multiple active projects found: {', '.join(active_project_names)}",
"Ask user: 'Which project should I use for this task?'",
f"Suggested project: '{most_active_project}' (most active with {most_active_count} items)",
f"Ask user: 'Should I use {most_active_project} for this task, or would you prefer a different project?'",
]
)
else:
# No recent activity
elif active_projects > 0:
# Has activity but no clear most active project
active_project_names = [
name for name, activity in projects_activity.items() if activity.item_count > 0
]
if len(active_project_names) == 1:
guidance_lines.extend(
[
f"Suggested project: '{active_project_names[0]}' (only active project)",
f"Ask user: 'Should I use {active_project_names[0]} for this task?'",
]
)
else:
guidance_lines.extend(
[
f"Multiple active projects found: {', '.join(active_project_names)}",
"Ask user: 'Which project should I use for this task?'",
]
)
else:
# No recent activity
guidance_lines.extend(
[
"No recent activity found in any project.",
"Consider: Ask which project to use or if they want to create a new one.",
]
)
guidance_lines.extend(
[
"No recent activity found in any project.",
"Consider: Ask which project to use or if they want to create a new one.",
"",
"Session reminder: Remember their project choice throughout this conversation.",
]
)
guidance_lines.extend(
["", "Session reminder: Remember their project choice throughout this conversation."]
)
guidance = "\n".join(guidance_lines)
guidance = "\n".join(guidance_lines)
# Format discovery mode output
return _format_discovery_output(projects_activity, summary, timeframe, guidance)
# Format discovery mode output
return _format_discovery_output(projects_activity, summary, timeframe, guidance)
else:
# Project-Specific Mode: Get activity for specific project
logger.info(
f"Getting recent activity from project {resolved_project}: type={type}, depth={depth}, timeframe={timeframe}"
)
else:
# Project-Specific Mode: Get activity for specific project
logger.info(
f"Getting recent activity from project {resolved_project}: type={type}, depth={depth}, timeframe={timeframe}"
)
active_project = await get_active_project(client, resolved_project, context)
project_url = active_project.project_url
active_project = await get_active_project(client, resolved_project, context)
project_url = active_project.project_url
response = await call_get(
client,
f"{project_url}/memory/recent",
params=params,
)
activity_data = GraphContext.model_validate(response.json())
response = await call_get(
client,
f"{project_url}/memory/recent",
params=params,
)
activity_data = GraphContext.model_validate(response.json())
# Format project-specific mode output
return _format_project_output(resolved_project, activity_data, timeframe, type)
# Format project-specific mode output
return _format_project_output(resolved_project, activity_data, timeframe, type)
async def _get_project_activity(
+28 -27
View File
@@ -6,7 +6,7 @@ from typing import List, Optional
from loguru import logger
from fastmcp import Context
from basic_memory.mcp.async_client import client
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.project_context import get_active_project
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_post
@@ -205,8 +205,8 @@ async def search_notes(
page: int = 1,
page_size: int = 10,
search_type: str = "text",
types: Optional[List[str]] = None,
entity_types: Optional[List[str]] = None,
types: List[str] = [],
entity_types: List[str] = [],
after_date: Optional[str] = None,
context: Context | None = None,
) -> SearchResponse | str:
@@ -345,7 +345,7 @@ async def search_notes(
else:
search_query.text = query # Default to text search
# Add optional filters if provided
# Add optional filters if provided (empty lists are treated as no filter)
if entity_types:
search_query.entity_types = [SearchItemType(t) for t in entity_types]
if types:
@@ -353,31 +353,32 @@ async def search_notes(
if after_date:
search_query.after_date = after_date
active_project = await get_active_project(client, project, context)
project_url = active_project.project_url
async with get_client() as client:
active_project = await get_active_project(client, project, context)
project_url = active_project.project_url
logger.info(f"Searching for {search_query} in project {active_project.name}")
logger.info(f"Searching for {search_query} in project {active_project.name}")
try:
response = await call_post(
client,
f"{project_url}/search/",
json=search_query.model_dump(),
params={"page": page, "page_size": page_size},
)
result = SearchResponse.model_validate(response.json())
# Check if we got no results and provide helpful guidance
if not result.results:
logger.info(
f"Search returned no results for query: {query} in project {active_project.name}"
try:
response = await call_post(
client,
f"{project_url}/search/",
json=search_query.model_dump(),
params={"page": page, "page_size": page_size},
)
# Don't treat this as an error, but the user might want guidance
# We return the empty result as normal - the user can decide if they need help
result = SearchResponse.model_validate(response.json())
return result
# Check if we got no results and provide helpful guidance
if not result.results:
logger.info(
f"Search returned no results for query: {query} in project {active_project.name}"
)
# Don't treat this as an error, but the user might want guidance
# We return the empty result as normal - the user can decide if they need help
except Exception as e:
logger.error(f"Search failed for query '{query}': {e}, project: {active_project.name}")
# Return formatted error message as string for better user experience
return _format_search_error_response(active_project.name, str(e), query, search_type)
return result
except Exception as e:
logger.error(f"Search failed for query '{query}': {e}, project: {active_project.name}")
# Return formatted error message as string for better user experience
return _format_search_error_response(active_project.name, str(e), query, search_type)
-256
View File
@@ -1,256 +0,0 @@
"""Sync status tool for Basic Memory MCP server."""
from typing import Optional
from loguru import logger
from fastmcp import Context
from basic_memory.config import ConfigManager
from basic_memory.mcp.async_client import client
from basic_memory.mcp.server import mcp
from basic_memory.mcp.project_context import get_active_project
from basic_memory.services.sync_status_service import sync_status_tracker
def _get_all_projects_status() -> list[str]:
"""Get status lines for all configured projects."""
status_lines = []
try:
app_config = ConfigManager().config
if app_config.projects:
status_lines.extend(["", "---", "", "**All Projects Status:**"])
for project_name, project_path in app_config.projects.items():
# Check if this project has sync status
project_sync_status = sync_status_tracker.get_project_status(project_name)
if project_sync_status:
# Project has tracked sync activity
if project_sync_status.status.value == "watching":
# Project is actively watching for changes (steady state)
status_icon = "👁️"
status_text = "Watching for changes"
elif project_sync_status.status.value == "completed":
# Sync completed but not yet watching - transitional state
status_icon = ""
status_text = "Sync completed"
elif project_sync_status.status.value in ["scanning", "syncing"]:
status_icon = "🔄"
status_text = "Sync in progress"
if project_sync_status.files_total > 0:
progress_pct = (
project_sync_status.files_processed
/ project_sync_status.files_total
) * 100
status_text += f" ({project_sync_status.files_processed}/{project_sync_status.files_total}, {progress_pct:.0f}%)"
elif project_sync_status.status.value == "failed":
status_icon = ""
status_text = f"Sync error: {project_sync_status.error or 'Unknown error'}"
else:
status_icon = "⏸️"
status_text = project_sync_status.status.value.title()
else:
# Project has no tracked sync activity - will be synced automatically
status_icon = ""
status_text = "Pending sync"
status_lines.append(f"- {status_icon} **{project_name}**: {status_text}")
except Exception as e:
logger.debug(f"Could not get project config for comprehensive status: {e}")
return status_lines
@mcp.tool(
description="""Check the status of file synchronization and background operations.
Use this tool to:
- Check if file sync is in progress or completed
- Get detailed sync progress information
- Understand if your files are fully indexed
- Get specific error details if sync operations failed
- Monitor initial project setup and legacy migration
This covers all sync operations including:
- Initial project setup and file indexing
- Legacy project migration to unified database
- Ongoing file monitoring and updates
- Background processing of knowledge graphs
""",
)
async def sync_status(project: Optional[str] = None, context: Context | None = None) -> str:
"""Get current sync status and system readiness information.
This tool provides detailed information about any ongoing or completed
sync operations, helping users understand when their files are ready.
Args:
project: Optional project name to get project-specific context
Returns:
Formatted sync status with progress, readiness, and guidance
"""
logger.info("MCP tool call tool=sync_status")
status_lines = []
try:
from basic_memory.services.sync_status_service import sync_status_tracker
# Get overall summary
summary = sync_status_tracker.get_summary()
is_ready = sync_status_tracker.is_ready
# Header
status_lines.extend(
[
"# Basic Memory Sync Status",
"",
f"**Current Status**: {summary}",
f"**System Ready**: {'✅ Yes' if is_ready else '🔄 Processing'}",
"",
]
)
if is_ready:
status_lines.extend(
[
"✅ **All sync operations completed**",
"",
"- File indexing is complete",
"- Knowledge graphs are up to date",
"- All Basic Memory tools are fully operational",
"",
"Your knowledge base is ready for use!",
]
)
# Show all projects status even when ready
status_lines.extend(_get_all_projects_status())
else:
# System is still processing - show both active and all projects
all_sync_projects = sync_status_tracker.get_all_projects()
active_projects = [
p for p in all_sync_projects.values() if p.status.value in ["scanning", "syncing"]
]
failed_projects = [p for p in all_sync_projects.values() if p.status.value == "failed"]
if active_projects:
status_lines.extend(
[
"🔄 **File synchronization in progress**",
"",
"Basic Memory is automatically processing all configured projects and building knowledge graphs.",
"This typically takes 1-3 minutes depending on the amount of content.",
"",
"**Currently Processing:**",
]
)
for project_status in active_projects:
progress = ""
if project_status.files_total > 0:
progress_pct = (
project_status.files_processed / project_status.files_total
) * 100
progress = f" ({project_status.files_processed}/{project_status.files_total}, {progress_pct:.0f}%)"
status_lines.append(
f"- **{project_status.project_name}**: {project_status.message}{progress}"
)
status_lines.extend(
[
"",
"**What's happening:**",
"- Scanning and indexing markdown files",
"- Building entity and relationship graphs",
"- Setting up full-text search indexes",
"- Processing file changes and updates",
"",
"**What you can do:**",
"- Wait for automatic processing to complete - no action needed",
"- Use this tool again to check progress",
"- Simple operations may work already",
"- All projects will be available once sync finishes",
]
)
# Handle failed projects (independent of active projects)
if failed_projects:
status_lines.extend(["", "❌ **Some projects failed to sync:**", ""])
for project_status in failed_projects:
status_lines.append(
f"- **{project_status.project_name}**: {project_status.error or 'Unknown error'}"
)
status_lines.extend(
[
"",
"**Next steps:**",
"1. Check the logs for detailed error information",
"2. Ensure file permissions allow read/write access",
"3. Try restarting the MCP server",
"4. If issues persist, consider filing a support issue",
]
)
elif not active_projects:
# No active or failed projects - must be pending
status_lines.extend(
[
"⏳ **Sync operations pending**",
"",
"File synchronization has been queued but hasn't started yet.",
"This usually resolves automatically within a few seconds.",
]
)
# Add comprehensive project status for all configured projects
all_projects_status = _get_all_projects_status()
if all_projects_status:
status_lines.extend(all_projects_status)
# Add explanation about automatic syncing if there are unsynced projects
unsynced_count = sum(1 for line in all_projects_status if "" in line)
if unsynced_count > 0 and not is_ready:
status_lines.extend(
[
"",
"**Note**: All configured projects will be automatically synced during startup.",
]
)
# Add project context if provided
if project:
try:
active_project = await get_active_project(client, project, context)
status_lines.extend(
[
"",
"---",
"",
f"**Active Project**: {active_project.name}",
f"**Project Path**: {active_project.home}",
]
)
except Exception as e:
logger.debug(f"Could not get project info: {e}")
return "\n".join(status_lines)
except Exception as e:
return f"""# Sync Status - Error
**Unable to check sync status**: {str(e)}
**Troubleshooting:**
- The system may still be starting up
- Try waiting a few seconds and checking again
- Check logs for detailed error information
- Consider restarting if the issue persists
"""
-85
View File
@@ -23,8 +23,6 @@ from httpx._types import (
from loguru import logger
from mcp.server.fastmcp.exceptions import ToolError
from basic_memory.mcp.tools.headers import inject_auth_header
def get_error_message(
status_code: int, url: URL | str, method: str, msg: Optional[str] = None
@@ -110,7 +108,6 @@ async def call_get(
logger.debug(f"Calling GET '{url}' params: '{params}'")
error_message = None
headers = inject_auth_header(headers)
try:
response = await client.get(
url,
@@ -196,9 +193,6 @@ async def call_put(
logger.debug(f"Calling PUT '{url}'")
error_message = None
# Inject JWT from FastMCP context if available
headers = inject_auth_header(headers)
try:
response = await client.put(
url,
@@ -288,9 +282,6 @@ async def call_patch(
"""
logger.debug(f"Calling PATCH '{url}'")
# Inject JWT from FastMCP context if available
headers = inject_auth_header(headers)
try:
response = await client.patch(
url,
@@ -396,9 +387,6 @@ async def call_post(
logger.debug(f"Calling POST '{url}'")
error_message = None
# Inject JWT from FastMCP context if available
headers = inject_auth_header(headers)
try:
response = await client.post(
url=url,
@@ -481,9 +469,6 @@ async def call_delete(
logger.debug(f"Calling DELETE '{url}'")
error_message = None
# Inject JWT from FastMCP context if available
headers = inject_auth_header(headers)
try:
response = await client.delete(
url=url,
@@ -525,73 +510,3 @@ async def call_delete(
except HTTPStatusError as e:
raise ToolError(error_message) from e
def check_migration_status() -> Optional[str]:
"""Check if sync/migration is in progress and return status message if so.
Returns:
Status message if sync is in progress, None if system is ready
"""
try:
from basic_memory.services.sync_status_service import sync_status_tracker
if not sync_status_tracker.is_ready:
return sync_status_tracker.get_summary()
return None
except Exception:
# If there's any error checking sync status, assume ready
return None
async def wait_for_migration_or_return_status(
timeout: float = 5.0, project_name: Optional[str] = None
) -> Optional[str]:
"""Wait briefly for sync/migration to complete, or return status message.
Args:
timeout: Maximum time to wait for sync completion
project_name: Optional project name to check specific project status.
If provided, only checks that project's readiness.
If None, uses global status check (legacy behavior).
Returns:
Status message if sync is still in progress, None if ready
"""
try:
from basic_memory.services.sync_status_service import sync_status_tracker
import asyncio
# Check if we should use project-specific or global status
def is_ready() -> bool:
if project_name:
return sync_status_tracker.is_project_ready(project_name)
return sync_status_tracker.is_ready
if is_ready():
return None
# Wait briefly for sync to complete
start_time = asyncio.get_event_loop().time()
while (asyncio.get_event_loop().time() - start_time) < timeout:
if is_ready():
return None
await asyncio.sleep(0.1) # Check every 100ms
# Still not ready after timeout
if project_name:
# For project-specific checks, get project status details
project_status = sync_status_tracker.get_project_status(project_name)
if project_status and project_status.status.value == "failed":
error_msg = project_status.error or "Unknown sync error"
return f"❌ Sync failed for project '{project_name}': {error_msg}"
elif project_status:
return f"🔄 Project '{project_name}' is still syncing: {project_status.message}"
else:
return f"⚠️ Project '{project_name}' status unknown"
else:
# Fall back to global summary for legacy calls
return sync_status_tracker.get_summary()
except Exception: # pragma: no cover
# If there's any error, assume ready
return None
+14 -28
View File
@@ -22,14 +22,10 @@ async def view_note(
) -> str:
"""View a markdown note as a formatted artifact.
This tool reads a note using the same logic as read_note but displays the content
as a markdown artifact for better viewing experience in Claude Desktop.
This tool reads a note using the same logic as read_note but instructs Claude
to display the content as a markdown artifact in the Claude Desktop app.
Project parameter optional with server resolution.
After calling this tool, create an artifact using the returned content to display
the note in a readable format. The tool returns the note content that should be
used to create a markdown artifact.
Args:
identifier: The title or permalink of the note to view
project: Project name to read from. Optional - server will resolve using hierarchy.
@@ -39,7 +35,7 @@ async def view_note(
context: Optional FastMCP context for performance caching.
Returns:
The note content as a markdown artifact with a confirmation message.
Instructions for Claude to create a markdown artifact with the note content.
Examples:
# View a note by title
@@ -66,26 +62,16 @@ async def view_note(
# Check if this is an error message (note not found)
if "# Note Not Found" in content:
return content # Return error message directly instead of creating artifact
# Extract title from content if possible
title = identifier
lines = content.split("\n")
for line in lines:
if line.startswith("title:"):
# Extract title from frontmatter
title = line.split("title:", 1)[1].strip().strip('"').strip("'")
break
elif line.startswith("# "):
# Extract title from first heading
title = line[2:].strip()
break
# Return the artifact XML as part of the response
artifact = f'<artifact identifier="note-{hash(identifier) & 0x7FFFFFFF}" type="text/markdown" title="{title}">\n{content}\n</artifact>'
return content # Return error message directly
# Return instructions for Claude to create an artifact
return dedent(f"""
<instructions>
Create an artifact using the returned content to display the note in a readable format.
</instructions>
{artifact}\n\n Note displayed as artifact: **{title}**""")
Note retrieved: "{identifier}"
Display this note as a markdown artifact for the user.
Content:
---
{content}
---
""").strip()
+91 -92
View File
@@ -4,7 +4,7 @@ from typing import List, Union, Optional
from loguru import logger
from basic_memory.mcp.async_client import client
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.project_context import get_active_project, add_project_metadata
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_put
@@ -16,9 +16,6 @@ from basic_memory.utils import parse_tags, validate_project_path
# Define TagType as a Union that can accept either a string or a list of strings or None
TagType = Union[List[str], str, None]
# Define TagType as a Union that can accept either a string or a list of strings or None
TagType = Union[List[str], str, None]
@mcp.tool(
description="Create or update a markdown note. Returns a markdown formatted summary of the semantic content.",
@@ -28,8 +25,8 @@ async def write_note(
content: str,
folder: str,
project: Optional[str] = None,
tags=None,
entity_type: str = "note",
tags: list[str] | str | None = None,
note_type: str = "note",
context: Context | None = None,
) -> str:
"""Write a markdown note to the knowledge base.
@@ -63,13 +60,15 @@ async def write_note(
title: The title of the note
content: Markdown content for the note, can include observations and relations
folder: Folder path relative to project root where the file should be saved.
Use forward slashes (/) as separators. Examples: "notes", "projects/2025", "research/ml"
Use forward slashes (/) as separators. Use "/" or "" to write to project root.
Examples: "notes", "projects/2025", "research/ml", "/" (root)
project: Project name to write to. Optional - server will resolve using the
hierarchy above. If unknown, use list_memory_projects() to discover
available projects.
tags: Tags to categorize the note. Can be a list of strings, a comma-separated string, or None.
Note: If passing from external MCP clients, use a string format (e.g. "tag1,tag2,tag3")
entity_type: Type of entity to create. Defaults to "note". Can be "guide", "report", "config", etc.
note_type: Type of note to create (stored in frontmatter). Defaults to "note".
Can be "guide", "report", "config", "person", etc.
context: Optional FastMCP context for performance caching.
Returns:
@@ -95,14 +94,14 @@ async def write_note(
content="# Weekly Standup\\n\\n- [decision] Use SQLite for storage #tech"
)
# Create a note with tags and entity type
# Create a note with tags and note type
write_note(
project="work-project",
title="API Design",
folder="specs",
content="# REST API Specification\\n\\n- implements [[Authentication]]",
tags=["api", "design"],
entity_type="guide"
note_type="guide"
)
# Update existing note (same title/folder)
@@ -117,92 +116,92 @@ async def write_note(
HTTPError: If project doesn't exist or is inaccessible
SecurityError: If folder path attempts path traversal
"""
logger.info(
f"MCP tool call tool=write_note project={project} folder={folder}, title={title}, tags={tags}"
)
# Get and validate the project (supports optional project parameter)
active_project = await get_active_project(client, project, context)
# Validate folder path to prevent path traversal attacks
project_path = active_project.home
if folder and not validate_project_path(folder, project_path):
logger.warning(
"Attempted path traversal attack blocked", folder=folder, project=active_project.name
async with get_client() as client:
logger.info(
f"MCP tool call tool=write_note project={project} folder={folder}, title={title}, tags={tags}"
)
return f"# Error\n\nFolder path '{folder}' is not allowed - paths must stay within project boundaries"
# Check migration status and wait briefly if needed
from basic_memory.mcp.tools.utils import wait_for_migration_or_return_status
# Get and validate the project (supports optional project parameter)
active_project = await get_active_project(client, project, context)
migration_status = await wait_for_migration_or_return_status(
timeout=5.0, project_name=active_project.name
)
if migration_status: # pragma: no cover
return f"# System Status\n\n{migration_status}\n\nPlease wait for migration to complete before creating notes."
# Normalize "/" to empty string for root folder (must happen before validation)
if folder == "/":
folder = ""
# Process tags using the helper function
tag_list = parse_tags(tags)
# Create the entity request
metadata = {"tags": tag_list} if tag_list else None
entity = Entity(
title=title,
folder=folder,
entity_type=entity_type,
content_type="text/markdown",
content=content,
entity_metadata=metadata,
)
project_url = active_project.permalink
# Create or update via knowledge API
logger.debug(f"Creating entity via API permalink={entity.permalink}")
url = f"{project_url}/knowledge/entities/{entity.permalink}"
response = await call_put(client, url, json=entity.model_dump())
result = EntityResponse.model_validate(response.json())
# Format semantic summary based on status code
action = "Created" if response.status_code == 201 else "Updated"
summary = [
f"# {action} note",
f"project: {active_project.name}",
f"file_path: {result.file_path}",
f"permalink: {result.permalink}",
f"checksum: {result.checksum[:8] if result.checksum else 'unknown'}",
]
# Count observations by category
categories = {}
if result.observations:
for obs in result.observations:
categories[obs.category] = categories.get(obs.category, 0) + 1
summary.append("\n## Observations")
for category, count in sorted(categories.items()):
summary.append(f"- {category}: {count}")
# Count resolved/unresolved relations
unresolved = 0
resolved = 0
if result.relations:
unresolved = sum(1 for r in result.relations if not r.to_id)
resolved = len(result.relations) - unresolved
summary.append("\n## Relations")
summary.append(f"- Resolved: {resolved}")
if unresolved:
summary.append(f"- Unresolved: {unresolved}")
summary.append("\nNote: Unresolved relations point to entities that don't exist yet.")
summary.append(
"They will be automatically resolved when target entities are created or during sync operations."
# Validate folder path to prevent path traversal attacks
project_path = active_project.home
if folder and not validate_project_path(folder, project_path):
logger.warning(
"Attempted path traversal attack blocked",
folder=folder,
project=active_project.name,
)
return f"# Error\n\nFolder path '{folder}' is not allowed - paths must stay within project boundaries"
if tag_list:
summary.append(f"\n## Tags\n- {', '.join(tag_list)}")
# Process tags using the helper function
tag_list = parse_tags(tags)
# Create the entity request
metadata = {"tags": tag_list} if tag_list else None
entity = Entity(
title=title,
folder=folder,
entity_type=note_type,
content_type="text/markdown",
content=content,
entity_metadata=metadata,
)
project_url = active_project.permalink
# Log the response with structured data
logger.info(
f"MCP tool response: tool=write_note project={active_project.name} action={action} permalink={result.permalink} observations_count={len(result.observations)} relations_count={len(result.relations)} resolved_relations={resolved} unresolved_relations={unresolved} status_code={response.status_code}"
)
result = "\n".join(summary)
return add_project_metadata(result, active_project.name)
# Create or update via knowledge API
logger.debug(f"Creating entity via API permalink={entity.permalink}")
url = f"{project_url}/knowledge/entities/{entity.permalink}"
response = await call_put(client, url, json=entity.model_dump())
result = EntityResponse.model_validate(response.json())
# Format semantic summary based on status code
action = "Created" if response.status_code == 201 else "Updated"
summary = [
f"# {action} note",
f"project: {active_project.name}",
f"file_path: {result.file_path}",
f"permalink: {result.permalink}",
f"checksum: {result.checksum[:8] if result.checksum else 'unknown'}",
]
# Count observations by category
categories = {}
if result.observations:
for obs in result.observations:
categories[obs.category] = categories.get(obs.category, 0) + 1
summary.append("\n## Observations")
for category, count in sorted(categories.items()):
summary.append(f"- {category}: {count}")
# Count resolved/unresolved relations
unresolved = 0
resolved = 0
if result.relations:
unresolved = sum(1 for r in result.relations if not r.to_id)
resolved = len(result.relations) - unresolved
summary.append("\n## Relations")
summary.append(f"- Resolved: {resolved}")
if unresolved:
summary.append(f"- Unresolved: {unresolved}")
summary.append(
"\nNote: Unresolved relations point to entities that don't exist yet."
)
summary.append(
"They will be automatically resolved when target entities are created or during sync operations."
)
if tag_list:
summary.append(f"\n## Tags\n- {', '.join(tag_list)}")
# Log the response with structured data
logger.info(
f"MCP tool response: tool=write_note project={active_project.name} action={action} permalink={result.permalink} observations_count={len(result.observations)} relations_count={len(result.relations)} resolved_relations={resolved} unresolved_relations={unresolved} status_code={response.status_code}"
)
result = "\n".join(summary)
return add_project_metadata(result, active_project.name)
+7
View File
@@ -13,6 +13,7 @@ from sqlalchemy import (
DateTime,
Index,
JSON,
Float,
text,
)
from sqlalchemy.orm import Mapped, mapped_column, relationship
@@ -73,6 +74,12 @@ class Entity(Base):
# checksum of file
checksum: Mapped[Optional[str]] = mapped_column(String, nullable=True)
# File metadata for sync
# mtime: file modification timestamp (Unix epoch float) for change detection
mtime: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
# size: file size in bytes for quick change detection
size: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)
# Metadata and tracking
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), default=lambda: datetime.now().astimezone()
+5
View File
@@ -9,6 +9,7 @@ from sqlalchemy import (
Text,
Boolean,
DateTime,
Float,
Index,
event,
)
@@ -61,6 +62,10 @@ class Project(Base):
onupdate=lambda: datetime.now(UTC),
)
# Sync optimization - scan watermark tracking
last_scan_timestamp: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
last_file_count: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)
# Define relationships to entities, observations, and relations
# These relationships will be established once we add project_id to those models
entities = relationship("Entity", back_populates="project", cascade="all, delete-orphan")
+128 -15
View File
@@ -3,6 +3,7 @@
from pathlib import Path
from typing import List, Optional, Sequence, Union
from loguru import logger
from sqlalchemy import select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
@@ -62,6 +63,23 @@ class EntityRepository(Repository[Entity]):
)
return await self.find_one(query)
async def find_by_checksum(self, checksum: str) -> Sequence[Entity]:
"""Find entities with the given checksum.
Used for move detection - finds entities that may have been moved to a new path.
Multiple entities may have the same checksum if files were copied.
Args:
checksum: File content checksum to search for
Returns:
Sequence of entities with matching checksum (may be empty)
"""
query = self.select().where(Entity.checksum == checksum)
# Don't load relationships for move detection - we only need file_path and checksum
result = await self.execute_query(query, use_query_options=False)
return list(result.scalars().all())
async def delete_by_file_path(self, file_path: Union[Path, str]) -> bool:
"""Delete entity with the provided file_path.
@@ -135,7 +153,20 @@ class EntityRepository(Repository[Entity]):
)
return found
except IntegrityError:
except IntegrityError as e:
# Check if this is a FOREIGN KEY constraint failure
error_str = str(e)
if "FOREIGN KEY constraint failed" in error_str:
# Import locally to avoid circular dependency (repository -> services -> repository)
from basic_memory.services.exceptions import SyncFatalError
# Project doesn't exist in database - this is a fatal sync error
raise SyncFatalError(
f"Cannot sync file '{entity.file_path}': "
f"project_id={entity.project_id} does not exist in database. "
f"The project may have been deleted. This sync will be terminated."
) from e
await session.rollback()
# Re-query after rollback to get a fresh, attached entity
@@ -150,25 +181,32 @@ class EntityRepository(Repository[Entity]):
if existing_entity:
# File path conflict - update the existing entity
for key, value in {
"title": entity.title,
"entity_type": entity.entity_type,
"entity_metadata": entity.entity_metadata,
"content_type": entity.content_type,
"permalink": entity.permalink,
"checksum": entity.checksum,
"updated_at": entity.updated_at,
}.items():
setattr(existing_entity, key, value)
logger.debug(
f"Resolving file_path conflict for {entity.file_path}, "
f"entity_id={existing_entity.id}, observations={len(entity.observations)}"
)
# Use merge to avoid session state conflicts
# Set the ID to update existing entity
entity.id = existing_entity.id
# Clear and re-add observations
existing_entity.observations.clear()
# Ensure observations reference the correct entity_id
for obs in entity.observations:
obs.entity_id = existing_entity.id
existing_entity.observations.append(obs)
# Clear any existing ID to force INSERT as new observation
obs.id = None
# Merge the entity which will update the existing one
merged_entity = await session.merge(entity)
await session.commit()
return existing_entity
# Re-query to get proper relationships loaded
final_result = await session.execute(
select(Entity)
.where(Entity.id == merged_entity.id)
.options(*self.get_load_options())
)
return final_result.scalar_one()
else:
# No file_path conflict - must be permalink conflict
@@ -176,6 +214,81 @@ class EntityRepository(Repository[Entity]):
entity = await self._handle_permalink_conflict(entity, session)
return entity
async def get_all_file_paths(self) -> List[str]:
"""Get all file paths for this project - optimized for deletion detection.
Returns only file_path strings without loading entities or relationships.
Used by streaming sync to detect deleted files efficiently.
Returns:
List of file_path strings for all entities in the project
"""
query = select(Entity.file_path)
query = self._add_project_filter(query)
result = await self.execute_query(query, use_query_options=False)
return list(result.scalars().all())
async def get_distinct_directories(self) -> List[str]:
"""Extract unique directory paths from file_path column.
Optimized method for getting directory structure without loading full entities
or relationships. Returns a sorted list of unique directory paths.
Returns:
List of unique directory paths (e.g., ["notes", "notes/meetings", "specs"])
"""
# Query only file_path column, no entity objects or relationships
query = select(Entity.file_path).distinct()
query = self._add_project_filter(query)
# Execute with use_query_options=False to skip eager loading
result = await self.execute_query(query, use_query_options=False)
file_paths = [row for row in result.scalars().all()]
# Parse file paths to extract unique directories
directories = set()
for file_path in file_paths:
parts = [p for p in file_path.split("/") if p]
# Add all parent directories (exclude filename which is the last part)
for i in range(len(parts) - 1):
dir_path = "/".join(parts[: i + 1])
directories.add(dir_path)
return sorted(directories)
async def find_by_directory_prefix(self, directory_prefix: str) -> Sequence[Entity]:
"""Find entities whose file_path starts with the given directory prefix.
Optimized method for listing directory contents without loading all entities.
Uses SQL LIKE pattern matching to filter entities by directory path.
Args:
directory_prefix: Directory path prefix (e.g., "docs", "docs/guides")
Empty string returns all entities (root directory)
Returns:
Sequence of entities in the specified directory and subdirectories
"""
# Build SQL LIKE pattern
if directory_prefix == "" or directory_prefix == "/":
# Root directory - return all entities
return await self.find_all()
# Remove leading/trailing slashes for consistency
directory_prefix = directory_prefix.strip("/")
# Query entities with file_path starting with prefix
# Pattern matches "prefix/" to ensure we get files IN the directory,
# not just files whose names start with the prefix
pattern = f"{directory_prefix}/%"
query = self.select().where(Entity.file_path.like(pattern))
# Skip eager loading - we only need basic entity fields for directory trees
result = await self.execute_query(query, use_query_options=False)
return list(result.scalars().all())
async def _handle_permalink_conflict(self, entity: Entity, session: AsyncSession) -> Entity:
"""Handle permalink conflicts by generating a unique permalink."""
base_permalink = entity.permalink
+16 -3
View File
@@ -152,12 +152,25 @@ class Repository[T: Base]:
# Add project filter if applicable
return self._add_project_filter(query)
async def find_all(self, skip: int = 0, limit: Optional[int] = None) -> Sequence[T]:
"""Fetch records from the database with pagination."""
async def find_all(
self, skip: int = 0, limit: Optional[int] = None, use_load_options: bool = True
) -> Sequence[T]:
"""Fetch records from the database with pagination.
Args:
skip: Number of records to skip
limit: Maximum number of records to return
use_load_options: Whether to apply eager loading options (default: True)
"""
logger.debug(f"Finding all {self.Model.__name__} (skip={skip}, limit={limit})")
async with db.scoped_session(self.session_maker) as session:
query = select(self.Model).offset(skip).options(*self.get_load_options())
query = select(self.Model).offset(skip)
# Only apply load options if requested
if use_load_options:
query = query.options(*self.get_load_options())
# Add project filter if applicable
query = self._add_project_filter(query)
@@ -559,6 +559,48 @@ class SearchRepository:
logger.debug(f"indexed row {search_index_row}")
await session.commit()
async def bulk_index_items(self, search_index_rows: List[SearchIndexRow]):
"""Index multiple items in a single batch operation.
Note: This method assumes that any existing records for the entity_id
have already been deleted (typically via delete_by_entity_id).
Args:
search_index_rows: List of SearchIndexRow objects to index
"""
if not search_index_rows:
return
async with db.scoped_session(self.session_maker) as session:
# Prepare all insert data with project_id
insert_data_list = []
for row in search_index_rows:
insert_data = row.to_insert()
insert_data["project_id"] = self.project_id
insert_data_list.append(insert_data)
# Batch insert all records using executemany
await session.execute(
text("""
INSERT INTO search_index (
id, title, content_stems, content_snippet, permalink, file_path, type, metadata,
from_id, to_id, relation_type,
entity_id, category,
created_at, updated_at,
project_id
) VALUES (
:id, :title, :content_stems, :content_snippet, :permalink, :file_path, :type, :metadata,
:from_id, :to_id, :relation_type,
:entity_id, :category,
:created_at, :updated_at,
:project_id
)
"""),
insert_data_list,
)
logger.debug(f"Bulk indexed {len(search_index_rows)} rows")
await session.commit()
async def delete_by_entity_id(self, entity_id: int):
"""Delete an item from the search index by entity_id."""
async with db.scoped_session(self.session_maker) as session:
+7 -3
View File
@@ -41,6 +41,10 @@ class CloudProjectCreateRequest(BaseModel):
class CloudProjectCreateResponse(BaseModel):
"""Response from creating a cloud project."""
name: str = Field(..., description="Created project name")
path: str = Field(..., description="Created project path")
message: str = Field(default="", description="Success message")
message: str = Field(..., description="Status message about the project creation")
status: str = Field(..., description="Status of the creation (success or error)")
default: bool = Field(..., description="True if the project was set as the default")
old_project: dict | None = Field(None, description="Information about the previous project")
new_project: dict | None = Field(
None, description="Information about the newly created project"
)
+1 -1
View File
@@ -183,7 +183,7 @@ class ProjectItem(BaseModel):
@property
def home(self) -> Path: # pragma: no cover
return Path(self.name)
return Path(self.path).expanduser()
@property
def project_url(self) -> str: # pragma: no cover
+25 -1
View File
@@ -1,6 +1,7 @@
"""Pydantic schemas for sync report responses."""
from typing import TYPE_CHECKING, Dict, Set
from datetime import datetime
from typing import TYPE_CHECKING, Dict, List, Set
from pydantic import BaseModel, Field
@@ -9,6 +10,17 @@ if TYPE_CHECKING:
from basic_memory.sync.sync_service import SyncReport
class SkippedFileResponse(BaseModel):
"""Information about a file that was skipped due to repeated failures."""
path: str = Field(description="File path relative to project root")
reason: str = Field(description="Error message from last failure")
failure_count: int = Field(description="Number of consecutive failures")
first_failed: datetime = Field(description="Timestamp of first failure")
model_config = {"from_attributes": True}
class SyncReportResponse(BaseModel):
"""Report of file changes found compared to database state.
@@ -24,6 +36,9 @@ class SyncReportResponse(BaseModel):
checksums: Dict[str, str] = Field(
default_factory=dict, description="Current file checksums (path -> checksum)"
)
skipped_files: List[SkippedFileResponse] = Field(
default_factory=list, description="Files skipped due to repeated failures"
)
total: int = Field(description="Total number of changes")
@classmethod
@@ -42,6 +57,15 @@ class SyncReportResponse(BaseModel):
deleted=report.deleted,
moves=report.moves,
checksums=report.checksums,
skipped_files=[
SkippedFileResponse(
path=skipped.path,
reason=skipped.reason,
failure_count=skipped.failure_count,
first_failed=skipped.first_failed,
)
for skipped in report.skipped_files
],
total=report.total,
)
+124 -3
View File
@@ -3,8 +3,9 @@
import fnmatch
import logging
import os
from typing import Dict, List, Optional
from typing import Dict, List, Optional, Sequence
from basic_memory.models import Entity
from basic_memory.repository import EntityRepository
from basic_memory.schemas.directory import DirectoryNode
@@ -89,6 +90,49 @@ class DirectoryService:
# Return the root node with its children
return root_node
async def get_directory_structure(self) -> DirectoryNode:
"""Build a hierarchical directory structure without file details.
Optimized method for folder navigation that only returns directory nodes,
no file metadata. Much faster than get_directory_tree() for large knowledge bases.
Returns:
DirectoryNode tree containing only folders (type="directory")
"""
# Get unique directories without loading entities
directories = await self.entity_repository.get_distinct_directories()
# Create a root directory node
root_node = DirectoryNode(name="Root", directory_path="/", type="directory")
# Map to store directory nodes by path for easy lookup
dir_map: Dict[str, DirectoryNode] = {"/": root_node}
# Build tree with just folders
for dir_path in directories:
parts = [p for p in dir_path.split("/") if p]
current_path = "/"
for i, part in enumerate(parts):
parent_path = current_path
# Build the directory path
current_path = (
f"{current_path}{part}" if current_path == "/" else f"{current_path}/{part}"
)
# Create directory node if it doesn't exist
if current_path not in dir_map:
dir_node = DirectoryNode(
name=part, directory_path=current_path, type="directory"
)
dir_map[current_path] = dir_node
# Add to parent's children
if parent_path in dir_map:
dir_map[parent_path].children.append(dir_node)
return root_node
async def list_directory(
self,
dir_name: str = "/",
@@ -118,8 +162,13 @@ class DirectoryService:
if dir_name != "/" and dir_name.endswith("/"):
dir_name = dir_name.rstrip("/")
# Get the full directory tree
root_tree = await self.get_directory_tree()
# Optimize: Query only entities in the target directory
# instead of loading the entire tree
dir_prefix = dir_name.lstrip("/")
entity_rows = await self.entity_repository.find_by_directory_prefix(dir_prefix)
# Build a partial tree from only the relevant entities
root_tree = self._build_directory_tree_from_entities(entity_rows, dir_name)
# Find the target directory node
target_node = self._find_directory_node(root_tree, dir_name)
@@ -132,6 +181,78 @@ class DirectoryService:
return result
def _build_directory_tree_from_entities(
self, entity_rows: Sequence[Entity], root_path: str
) -> DirectoryNode:
"""Build a directory tree from a subset of entities.
Args:
entity_rows: Sequence of entity objects to build tree from
root_path: Root directory path for the tree
Returns:
DirectoryNode representing the tree root
"""
# Create a root directory node
root_node = DirectoryNode(name="Root", directory_path=root_path, type="directory")
# Map to store directory nodes by path for easy lookup
dir_map: Dict[str, DirectoryNode] = {root_path: root_node}
# First pass: create all directory nodes
for file in entity_rows:
# Process directory path components
parts = [p for p in file.file_path.split("/") if p]
# Create directory structure
current_path = "/"
for i, part in enumerate(parts[:-1]): # Skip the filename
parent_path = current_path
# Build the directory path
current_path = (
f"{current_path}{part}" if current_path == "/" else f"{current_path}/{part}"
)
# Create directory node if it doesn't exist
if current_path not in dir_map:
dir_node = DirectoryNode(
name=part, directory_path=current_path, type="directory"
)
dir_map[current_path] = dir_node
# Add to parent's children
if parent_path in dir_map:
dir_map[parent_path].children.append(dir_node)
# Second pass: add file nodes to their parent directories
for file in entity_rows:
file_name = os.path.basename(file.file_path)
parent_dir = os.path.dirname(file.file_path)
directory_path = "/" if parent_dir == "" else f"/{parent_dir}"
# Create file node
file_node = DirectoryNode(
name=file_name,
file_path=file.file_path,
directory_path=f"/{file.file_path}",
type="file",
title=file.title,
permalink=file.permalink,
entity_id=file.id,
entity_type=file.entity_type,
content_type=file.content_type,
updated_at=file.updated_at,
)
# Add to parent directory's children
if directory_path in dir_map:
dir_map[directory_path].children.append(file_node)
elif root_path in dir_map:
# Fallback to root if parent not found
dir_map[root_path].children.append(file_node)
return root_node
def _find_directory_node(
self, root: DirectoryNode, target_path: str
) -> Optional[DirectoryNode]:
+39 -10
View File
@@ -52,7 +52,9 @@ class EntityService(BaseService[EntityModel]):
self.link_resolver = link_resolver
self.app_config = app_config
async def detect_file_path_conflicts(self, file_path: str) -> List[Entity]:
async def detect_file_path_conflicts(
self, file_path: str, skip_check: bool = False
) -> List[Entity]:
"""Detect potential file path conflicts for a given file path.
This checks for entities with similar file paths that might cause conflicts:
@@ -63,10 +65,14 @@ class EntityService(BaseService[EntityModel]):
Args:
file_path: The file path to check for conflicts
skip_check: If True, skip the check and return empty list (optimization for bulk operations)
Returns:
List of entities that might conflict with the given file path
"""
if skip_check:
return []
from basic_memory.utils import detect_potential_file_conflicts
conflicts = []
@@ -86,7 +92,10 @@ class EntityService(BaseService[EntityModel]):
return conflicts
async def resolve_permalink(
self, file_path: Permalink | Path, markdown: Optional[EntityMarkdown] = None
self,
file_path: Permalink | Path,
markdown: Optional[EntityMarkdown] = None,
skip_conflict_check: bool = False,
) -> str:
"""Get or generate unique permalink for an entity.
@@ -101,7 +110,9 @@ class EntityService(BaseService[EntityModel]):
file_path_str = Path(file_path).as_posix()
# Check for potential file path conflicts before resolving permalink
conflicts = await self.detect_file_path_conflicts(file_path_str)
conflicts = await self.detect_file_path_conflicts(
file_path_str, skip_check=skip_conflict_check
)
if conflicts:
logger.warning(
f"Detected potential file path conflicts for '{file_path_str}': "
@@ -445,6 +456,7 @@ class EntityService(BaseService[EntityModel]):
resolved_entities = await asyncio.gather(*lookup_tasks, return_exceptions=True)
# Process results and create relation records
relations_to_add = []
for rel, resolved in zip(markdown.relations, resolved_entities):
# Handle exceptions from gather and None results
target_entity: Optional[Entity] = None
@@ -465,14 +477,24 @@ class EntityService(BaseService[EntityModel]):
relation_type=rel.type,
context=rel.context,
)
relations_to_add.append(relation)
# Batch insert all relations
if relations_to_add:
try:
await self.relation_repository.add(relation)
await self.relation_repository.add_all(relations_to_add)
except IntegrityError:
# Unique constraint violation - relation already exists
logger.debug(
f"Skipping duplicate relation {rel.type} from {db_entity.permalink} target: {rel.target}"
)
continue
# Some relations might be duplicates - fall back to individual inserts
logger.debug("Batch relation insert failed, trying individual inserts")
for relation in relations_to_add:
try:
await self.relation_repository.add(relation)
except IntegrityError:
# Unique constraint violation - relation already exists
logger.debug(
f"Skipping duplicate relation {relation.relation_type} from {db_entity.permalink}"
)
continue
return await self.repository.get_by_file_path(path)
@@ -601,7 +623,7 @@ class EntityService(BaseService[EntityModel]):
Args:
current_content: The current markdown content
section_header: The section header to find and replace (e.g., "## Section Name")
new_content: The new content to replace the section with
new_content: The new content to replace the section with (should not include the header itself)
Returns:
The updated content with the section replaced
@@ -613,6 +635,13 @@ class EntityService(BaseService[EntityModel]):
if not section_header.startswith("#"):
section_header = "## " + section_header
# Strip duplicate header from new_content if present (fix for issue #390)
# LLMs sometimes include the section header in their content, which would create duplicates
new_content_lines = new_content.lstrip().split("\n")
if new_content_lines and new_content_lines[0].strip() == section_header.strip():
# Remove the duplicate header line
new_content = "\n".join(new_content_lines[1:]).lstrip()
# First pass: count matching sections to check for duplicates
lines = current_content.split("\n")
matching_sections = []
+15
View File
@@ -20,3 +20,18 @@ class DirectoryOperationError(Exception):
"""Raised when directory operations fail"""
pass
class SyncFatalError(Exception):
"""Raised when sync encounters a fatal error that prevents continuation.
Fatal errors include:
- Project deleted during sync (FOREIGN KEY constraint)
- Database corruption
- Critical system failures
When this exception is raised, the entire sync operation should be terminated
immediately rather than attempting to continue with remaining files.
"""
pass
+162 -24
View File
@@ -1,12 +1,17 @@
"""Service for file operations with checksum tracking."""
import asyncio
import hashlib
import mimetypes
from os import stat_result
from pathlib import Path
from typing import Any, Dict, Tuple, Union
import aiofiles
import yaml
from basic_memory import file_utils
from basic_memory.file_utils import FileError
from basic_memory.file_utils import FileError, ParseError
from basic_memory.markdown.markdown_processor import MarkdownProcessor
from basic_memory.models import Entity as EntityModel
from basic_memory.schemas import Entity as EntitySchema
@@ -16,13 +21,15 @@ from loguru import logger
class FileService:
"""Service for handling file operations.
"""Service for handling file operations with concurrency control.
All paths are handled as Path objects internally. Strings are converted to
Path objects when passed in. Relative paths are assumed to be relative to
base_path.
Features:
- True async I/O with aiofiles (non-blocking)
- Built-in concurrency limits (semaphore)
- Consistent file writing with checksums
- Frontmatter management
- Atomic operations
@@ -33,9 +40,13 @@ class FileService:
self,
base_path: Path,
markdown_processor: MarkdownProcessor,
max_concurrent_files: int = 10,
):
self.base_path = base_path.resolve() # Get absolute path
self.markdown_processor = markdown_processor
# Semaphore to limit concurrent file operations
# Prevents OOM on large projects by processing files in batches
self._file_semaphore = asyncio.Semaphore(max_concurrent_files)
def get_entity_path(self, entity: Union[EntityModel, EntitySchema]) -> Path:
"""Generate absolute filesystem path for entity.
@@ -104,6 +115,33 @@ class FileService:
logger.error("Failed to check file existence", path=str(path), error=str(e))
raise FileOperationError(f"Failed to check file existence: {e}")
async def ensure_directory(self, path: FilePath) -> None:
"""Ensure directory exists, creating if necessary.
Uses semaphore to control concurrency for directory creation operations.
Args:
path: Directory path to ensure (Path or string)
Raises:
FileOperationError: If directory creation fails
"""
try:
# Convert string to Path if needed
path_obj = self.base_path / path if isinstance(path, str) else path
full_path = path_obj if path_obj.is_absolute() else self.base_path / path_obj
# Use semaphore for concurrency control
async with self._file_semaphore:
# Run blocking mkdir in thread pool
loop = asyncio.get_event_loop()
await loop.run_in_executor(
None, lambda: full_path.mkdir(parents=True, exist_ok=True)
)
except Exception as e: # pragma: no cover
logger.error("Failed to create directory", path=str(path), error=str(e))
raise FileOperationError(f"Failed to create directory {path}: {e}")
async def write_file(self, path: FilePath, content: str) -> str:
"""Write content to file and return checksum.
@@ -126,7 +164,7 @@ class FileService:
try:
# Ensure parent directory exists
await file_utils.ensure_directory(full_path.parent)
await self.ensure_directory(full_path.parent)
# Write content atomically
logger.info(
@@ -147,9 +185,45 @@ class FileService:
logger.exception("File write error", path=str(full_path), error=str(e))
raise FileOperationError(f"Failed to write file: {e}")
# TODO remove read_file
async def read_file_content(self, path: FilePath) -> str:
"""Read file content using true async I/O with aiofiles.
Handles both absolute and relative paths. Relative paths are resolved
against base_path.
Args:
path: Path to read (Path or string)
Returns:
File content as string
Raises:
FileOperationError: If read fails
"""
# Convert string to Path if needed
path_obj = self.base_path / path if isinstance(path, str) else path
full_path = path_obj if path_obj.is_absolute() else self.base_path / path_obj
try:
logger.debug("Reading file content", operation="read_file_content", path=str(full_path))
async with aiofiles.open(full_path, mode="r", encoding="utf-8") as f:
content = await f.read()
logger.debug(
"File read completed",
path=str(full_path),
content_length=len(content),
)
return content
except Exception as e:
logger.exception("File read error", path=str(full_path), error=str(e))
raise FileOperationError(f"Failed to read file: {e}")
async def read_file(self, path: FilePath) -> Tuple[str, str]:
"""Read file and compute checksum.
"""Read file and compute checksum using true async I/O.
Uses aiofiles for non-blocking file reads.
Handles both absolute and relative paths. Relative paths are resolved
against base_path.
@@ -169,7 +243,11 @@ class FileService:
try:
logger.debug("Reading file", operation="read_file", path=str(full_path))
content = full_path.read_text(encoding="utf-8")
# Use aiofiles for non-blocking read
async with aiofiles.open(full_path, mode="r", encoding="utf-8") as f:
content = await f.read()
checksum = await file_utils.compute_checksum(content)
logger.debug(
@@ -199,29 +277,85 @@ class FileService:
full_path.unlink(missing_ok=True)
async def update_frontmatter(self, path: FilePath, updates: Dict[str, Any]) -> str:
"""
Update frontmatter fields in a file while preserving all content.
"""Update frontmatter fields in a file while preserving all content.
Only modifies the frontmatter section, leaving all content untouched.
Creates frontmatter section if none exists.
Returns checksum of updated file.
Uses aiofiles for true async I/O (non-blocking).
Args:
path: Path to the file (Path or string)
updates: Dictionary of frontmatter fields to update
path: Path to markdown file (Path or string)
updates: Dict of frontmatter fields to update
Returns:
Checksum of updated file
Raises:
FileOperationError: If file operations fail
ParseError: If frontmatter parsing fails
"""
# Convert string to Path if needed
path_obj = self.base_path / path if isinstance(path, str) else path
full_path = path_obj if path_obj.is_absolute() else self.base_path / path_obj
return await file_utils.update_frontmatter(full_path, updates)
try:
# Read current content using aiofiles
async with aiofiles.open(full_path, mode="r", encoding="utf-8") as f:
content = await f.read()
# Parse current frontmatter with proper error handling for malformed YAML
current_fm = {}
if file_utils.has_frontmatter(content):
try:
current_fm = file_utils.parse_frontmatter(content)
content = file_utils.remove_frontmatter(content)
except (ParseError, yaml.YAMLError) as e:
# Log warning and treat as plain markdown without frontmatter
logger.warning(
f"Failed to parse YAML frontmatter in {full_path}: {e}. "
"Treating file as plain markdown without frontmatter."
)
# Keep full content, treat as having no frontmatter
current_fm = {}
# Update frontmatter
new_fm = {**current_fm, **updates}
# Write new file with updated frontmatter
yaml_fm = yaml.dump(new_fm, sort_keys=False, allow_unicode=True)
final_content = f"---\n{yaml_fm}---\n\n{content.strip()}"
logger.debug(
"Updating frontmatter", path=str(full_path), update_keys=list(updates.keys())
)
await file_utils.write_file_atomic(full_path, final_content)
return await file_utils.compute_checksum(final_content)
except Exception as e:
# Only log real errors (not YAML parsing, which is handled above)
if not isinstance(e, (ParseError, yaml.YAMLError)):
logger.error(
"Failed to update frontmatter",
path=str(full_path),
error=str(e),
)
raise FileOperationError(f"Failed to update frontmatter: {e}")
async def compute_checksum(self, path: FilePath) -> str:
"""Compute checksum for a file.
"""Compute checksum for a file using true async I/O.
Uses aiofiles for non-blocking I/O with 64KB chunked reading.
Semaphore limits concurrent file operations to prevent OOM.
Memory usage is constant regardless of file size.
Args:
path: Path to the file (Path or string)
Returns:
Checksum of the file content
SHA256 checksum hex string
Raises:
FileError: If checksum computation fails
@@ -230,18 +364,22 @@ class FileService:
path_obj = self.base_path / path if isinstance(path, str) else path
full_path = path_obj if path_obj.is_absolute() else self.base_path / path_obj
try:
if self.is_markdown(path):
# read str
content = full_path.read_text(encoding="utf-8")
else:
# read bytes
content = full_path.read_bytes()
return await file_utils.compute_checksum(content)
# Semaphore controls concurrency - max N files processed at once
async with self._file_semaphore:
try:
hasher = hashlib.sha256()
chunk_size = 65536 # 64KB chunks
except Exception as e: # pragma: no cover
logger.error("Failed to compute checksum", path=str(full_path), error=str(e))
raise FileError(f"Failed to compute checksum for {path}: {e}")
# async I/O with aiofiles
async with aiofiles.open(full_path, mode="rb") as f:
while chunk := await f.read(chunk_size):
hasher.update(chunk)
return hasher.hexdigest()
except Exception as e: # pragma: no cover
logger.error("Failed to compute checksum", path=str(full_path), error=str(e))
raise FileError(f"Failed to compute checksum for {path}: {e}")
def file_stats(self, path: FilePath) -> stat_result:
"""Return file stats for a given path.
@@ -118,18 +118,8 @@ async def initialize_file_sync(
sync_dir = Path(project.path)
await sync_service.sync(sync_dir, project_name=project.name)
logger.info(f"Background sync completed successfully for project: {project.name}")
# Mark project as watching for changes after successful sync
from basic_memory.services.sync_status_service import sync_status_tracker
sync_status_tracker.start_project_watch(project.name)
logger.info(f"Project {project.name} is now watching for changes")
except Exception as e: # pragma: no cover
logger.error(f"Error in background sync for project {project.name}: {e}")
# Mark sync as failed for this project
from basic_memory.services.sync_status_service import sync_status_tracker
sync_status_tracker.fail_project_sync(project.name, str(e))
# Create background tasks for all project syncs (non-blocking)
sync_tasks = [
+140 -19
View File
@@ -1,7 +1,9 @@
"""Project management service for Basic Memory."""
import asyncio
import json
import os
import shutil
from datetime import datetime
from pathlib import Path
from typing import Dict, Optional, Sequence
@@ -80,7 +82,13 @@ class ProjectService:
return os.environ.get("BASIC_MEMORY_PROJECT", self.config_manager.default_project)
async def list_projects(self) -> Sequence[Project]:
return await self.repository.find_all()
"""List all projects without loading entity relationships.
Returns only basic project fields (name, path, etc.) without
eager loading the entities relationship which could load thousands
of entities for large knowledge bases.
"""
return await self.repository.find_all(use_load_options=False)
async def get_project(self, name: str) -> Optional[Project]:
"""Get the file path for a project by name or permalink."""
@@ -88,6 +96,40 @@ class ProjectService:
name
)
def _check_nested_paths(self, path1: str, path2: str) -> bool:
"""Check if two paths are nested (one is a prefix of the other).
Args:
path1: First path to compare
path2: Second path to compare
Returns:
True if one path is nested within the other, False otherwise
Examples:
_check_nested_paths("/foo", "/foo/bar") # True (child under parent)
_check_nested_paths("/foo/bar", "/foo") # True (parent over child)
_check_nested_paths("/foo", "/bar") # False (siblings)
"""
# Normalize paths to ensure proper comparison
p1 = Path(path1).resolve()
p2 = Path(path2).resolve()
# Check if either path is a parent of the other
try:
# Check if p2 is under p1
p2.relative_to(p1)
return True
except ValueError:
# Not nested in this direction, check the other
try:
# Check if p1 is under p2
p1.relative_to(p2)
return True
except ValueError:
# Not nested in either direction
return False
async def add_project(self, name: str, path: str, set_default: bool = False) -> None:
"""Add a new project to the configuration and database.
@@ -97,19 +139,66 @@ class ProjectService:
set_default: Whether to set this project as the default
Raises:
ValueError: If the project already exists
ValueError: If the project already exists or path collides with existing project
"""
# in cloud mode, don't allow arbitrary paths.
if config.cloud_mode:
basic_memory_home = os.getenv("BASIC_MEMORY_HOME")
assert basic_memory_home is not None
base_path = Path(basic_memory_home)
# If project_root is set, constrain all projects to that directory
project_root = self.config_manager.config.project_root
if project_root:
base_path = Path(project_root)
# Resolve to absolute path
resolved_path = Path(os.path.abspath(os.path.expanduser(base_path / path))).as_posix()
# In cloud mode (when project_root is set), ignore user's path completely
# and use sanitized project name as the directory name
# This ensures flat structure: /app/data/test-bisync instead of /app/data/documents/test bisync
sanitized_name = generate_permalink(name)
# Construct path using sanitized project name only
resolved_path = (base_path / sanitized_name).resolve().as_posix()
# Verify the resolved path is actually under project_root
if not resolved_path.startswith(base_path.resolve().as_posix()):
raise ValueError(
f"BASIC_MEMORY_PROJECT_ROOT is set to {project_root}. "
f"All projects must be created under this directory. Invalid path: {path}"
)
# Check for case-insensitive path collisions with existing projects
existing_projects = await self.list_projects()
for existing in existing_projects:
if (
existing.path.lower() == resolved_path.lower()
and existing.path != resolved_path
):
raise ValueError(
f"Path collision detected: '{resolved_path}' conflicts with existing project "
f"'{existing.name}' at '{existing.path}'. "
f"In cloud mode, paths are normalized to lowercase to prevent case-sensitivity issues."
)
else:
resolved_path = Path(os.path.abspath(os.path.expanduser(path))).as_posix()
# Check for nested paths with existing projects
existing_projects = await self.list_projects()
for existing in existing_projects:
if self._check_nested_paths(resolved_path, existing.path):
# Determine which path is nested within which for appropriate error message
p_new = Path(resolved_path).resolve()
p_existing = Path(existing.path).resolve()
# Check if new path is nested under existing project
if p_new.is_relative_to(p_existing):
raise ValueError(
f"Cannot create project at '{resolved_path}': "
f"path is nested within existing project '{existing.name}' at '{existing.path}'. "
f"Projects cannot share directory trees."
)
else:
# Existing project is nested under new path
raise ValueError(
f"Cannot create project at '{resolved_path}': "
f"existing project '{existing.name}' at '{existing.path}' is nested within this path. "
f"Projects cannot share directory trees."
)
# First add to config file (this will validate the project doesn't exist)
project_config = self.config_manager.add_project(name, resolved_path)
@@ -132,11 +221,12 @@ class ProjectService:
logger.info(f"Project '{name}' added at {resolved_path}")
async def remove_project(self, name: str) -> None:
async def remove_project(self, name: str, delete_notes: bool = False) -> None:
"""Remove a project from configuration and database.
Args:
name: The name of the project to remove
delete_notes: If True, delete the project directory from filesystem
Raises:
ValueError: If the project doesn't exist or is the default project
@@ -144,16 +234,43 @@ class ProjectService:
if not self.repository: # pragma: no cover
raise ValueError("Repository is required for remove_project")
# First remove from config (this will validate the project exists and is not default)
self.config_manager.remove_project(name)
# Then remove from database using robust lookup
# Get project from database first
project = await self.get_project(name)
if project:
await self.repository.delete(project.id)
if not project:
raise ValueError(f"Project '{name}' not found")
project_path = project.path
# Check if project is default (in cloud mode, check database; in local mode, check config)
if project.is_default or name == self.config_manager.config.default_project:
raise ValueError(f"Cannot remove the default project '{name}'")
# Remove from config if it exists there (may not exist in cloud mode)
try:
self.config_manager.remove_project(name)
except ValueError:
# Project not in config - that's OK in cloud mode, continue with database deletion
logger.debug(f"Project '{name}' not found in config, removing from database only")
# Remove from database
await self.repository.delete(project.id)
logger.info(f"Project '{name}' removed from configuration and database")
# Optionally delete the project directory
if delete_notes and project_path:
try:
path_obj = Path(project_path)
if path_obj.exists() and path_obj.is_dir():
await asyncio.to_thread(shutil.rmtree, project_path)
logger.info(f"Deleted project directory: {project_path}")
else:
logger.warning(
f"Project directory not found or not a directory: {project_path}"
)
except Exception as e:
logger.warning(f"Failed to delete project directory {project_path}: {e}")
async def set_default_project(self, name: str) -> None:
"""Set the default project in configuration and database.
@@ -273,11 +390,15 @@ class ProjectService:
}
await self.repository.create(project_data)
# Add projects that exist in DB but not in config to config
# Remove projects that exist in DB but not in config
# Config is the source of truth - if a project was deleted from config,
# it should be deleted from DB too (fixes issue #193)
for name, project in db_projects_by_permalink.items():
if name not in config_projects:
logger.info(f"Adding project '{name}' to configuration")
self.config_manager.add_project(name, project.path)
logger.info(
f"Removing project '{name}' from database (deleted from config, source of truth)"
)
await self.repository.delete(project.id)
# Ensure database default project state is consistent
await self._ensure_single_default_project()
+16 -8
View File
@@ -113,8 +113,10 @@ class SearchService:
# Add word boundaries
variants.update(w.strip() for w in text.lower().split() if w.strip())
# Add trigrams for fuzzy matching
variants.update(text[i : i + 3].lower() for i in range(len(text) - 2))
# Trigrams disabled: They create massive search index bloat, increasing DB size significantly
# and slowing down indexing performance. FTS5 search works well without them.
# See: https://github.com/basicmachines-co/basic-memory/issues/351
# variants.update(text[i : i + 3].lower() for i in range(len(text) - 2))
return variants
@@ -219,6 +221,9 @@ class SearchService:
The project_id is automatically added by the repository when indexing.
"""
# Collect all search index rows to batch insert at the end
rows_to_index = []
content_stems = []
content_snippet = ""
title_variants = self._generate_variants(entity.title)
@@ -241,8 +246,8 @@ class SearchService:
entity_content_stems = "\n".join(p for p in content_stems if p and p.strip())
# Index entity
await self.repository.index_item(
# Add entity row
rows_to_index.append(
SearchIndexRow(
id=entity.id,
type=SearchItemType.ENTITY.value,
@@ -261,13 +266,13 @@ class SearchService:
)
)
# Index each observation with permalink
# Add observation rows
for obs in entity.observations:
# Index with parent entity's file path since that's where it's defined
obs_content_stems = "\n".join(
p for p in self._generate_variants(obs.content) if p and p.strip()
)
await self.repository.index_item(
rows_to_index.append(
SearchIndexRow(
id=obs.id,
type=SearchItemType.OBSERVATION.value,
@@ -287,7 +292,7 @@ class SearchService:
)
)
# Only index outgoing relations (ones defined in this file)
# Add relation rows (only outgoing relations defined in this file)
for rel in entity.outgoing_relations:
# Create descriptive title showing the relationship
relation_title = (
@@ -299,7 +304,7 @@ class SearchService:
rel_content_stems = "\n".join(
p for p in self._generate_variants(relation_title) if p and p.strip()
)
await self.repository.index_item(
rows_to_index.append(
SearchIndexRow(
id=rel.id,
title=relation_title,
@@ -317,6 +322,9 @@ class SearchService:
)
)
# Batch insert all rows at once
await self.repository.bulk_index_items(rows_to_index)
async def delete_by_permalink(self, permalink: str):
"""Delete an item from the search index."""
await self.repository.delete_by_permalink(permalink)
@@ -1,198 +0,0 @@
"""Simple sync status tracking service."""
from dataclasses import dataclass
from enum import Enum
from typing import Dict, Optional
class SyncStatus(Enum):
"""Status of sync operations."""
IDLE = "idle"
SCANNING = "scanning"
SYNCING = "syncing"
COMPLETED = "completed"
FAILED = "failed"
WATCHING = "watching"
@dataclass
class ProjectSyncStatus:
"""Sync status for a single project."""
project_name: str
status: SyncStatus
message: str = ""
files_total: int = 0
files_processed: int = 0
error: Optional[str] = None
class SyncStatusTracker:
"""Global tracker for all sync operations."""
def __init__(self):
self._project_statuses: Dict[str, ProjectSyncStatus] = {}
self._global_status: SyncStatus = SyncStatus.IDLE
def start_project_sync(self, project_name: str, files_total: int = 0) -> None:
"""Start tracking sync for a project."""
self._project_statuses[project_name] = ProjectSyncStatus(
project_name=project_name,
status=SyncStatus.SCANNING,
message="Scanning files",
files_total=files_total,
files_processed=0,
)
self._update_global_status()
def update_project_progress( # pragma: no cover
self,
project_name: str,
status: SyncStatus,
message: str = "",
files_processed: int = 0,
files_total: Optional[int] = None,
) -> None:
"""Update progress for a project."""
if project_name not in self._project_statuses: # pragma: no cover
return
project_status = self._project_statuses[project_name]
project_status.status = status
project_status.message = message
project_status.files_processed = files_processed
if files_total is not None:
project_status.files_total = files_total
self._update_global_status()
def complete_project_sync(self, project_name: str) -> None:
"""Mark project sync as completed."""
if project_name in self._project_statuses:
self._project_statuses[project_name].status = SyncStatus.COMPLETED
self._project_statuses[project_name].message = "Sync completed"
self._update_global_status()
def fail_project_sync(self, project_name: str, error: str) -> None:
"""Mark project sync as failed."""
if project_name in self._project_statuses:
self._project_statuses[project_name].status = SyncStatus.FAILED
self._project_statuses[project_name].error = error
self._update_global_status()
def start_project_watch(self, project_name: str) -> None:
"""Mark project as watching for changes (steady state after sync)."""
if project_name in self._project_statuses:
self._project_statuses[project_name].status = SyncStatus.WATCHING
self._project_statuses[project_name].message = "Watching for changes"
self._update_global_status()
else:
# Create new status if project isn't tracked yet
self._project_statuses[project_name] = ProjectSyncStatus(
project_name=project_name,
status=SyncStatus.WATCHING,
message="Watching for changes",
files_total=0,
files_processed=0,
)
self._update_global_status()
def _update_global_status(self) -> None:
"""Update global status based on project statuses."""
if not self._project_statuses: # pragma: no cover
self._global_status = SyncStatus.IDLE
return
statuses = [p.status for p in self._project_statuses.values()]
if any(s == SyncStatus.FAILED for s in statuses):
self._global_status = SyncStatus.FAILED
elif any(s in (SyncStatus.SCANNING, SyncStatus.SYNCING) for s in statuses):
self._global_status = SyncStatus.SYNCING
elif all(s in (SyncStatus.COMPLETED, SyncStatus.WATCHING) for s in statuses):
self._global_status = SyncStatus.COMPLETED
else:
self._global_status = SyncStatus.SYNCING
@property
def global_status(self) -> SyncStatus:
"""Get overall sync status."""
return self._global_status
@property
def is_syncing(self) -> bool:
"""Check if any sync operation is in progress."""
return self._global_status in (SyncStatus.SCANNING, SyncStatus.SYNCING)
@property
def is_ready(self) -> bool: # pragma: no cover
"""Check if system is ready (no sync in progress)."""
return self._global_status in (SyncStatus.IDLE, SyncStatus.COMPLETED)
def is_project_ready(self, project_name: str) -> bool:
"""Check if a specific project is ready for operations.
Args:
project_name: Name of the project to check
Returns:
True if the project is ready (completed, watching, or not tracked),
False if the project is syncing, scanning, or failed
"""
project_status = self._project_statuses.get(project_name)
if not project_status:
# Project not tracked = ready (likely hasn't been synced yet)
return True
return project_status.status in (SyncStatus.COMPLETED, SyncStatus.WATCHING, SyncStatus.IDLE)
def get_project_status(self, project_name: str) -> Optional[ProjectSyncStatus]:
"""Get status for a specific project."""
return self._project_statuses.get(project_name)
def get_all_projects(self) -> Dict[str, ProjectSyncStatus]:
"""Get all project statuses."""
return self._project_statuses.copy()
def get_summary(self) -> str: # pragma: no cover
"""Get a user-friendly summary of sync status."""
if self._global_status == SyncStatus.IDLE:
return "✅ System ready"
elif self._global_status == SyncStatus.COMPLETED:
return "✅ All projects synced successfully"
elif self._global_status == SyncStatus.FAILED:
failed_projects = [
p.project_name
for p in self._project_statuses.values()
if p.status == SyncStatus.FAILED
]
return f"❌ Sync failed for: {', '.join(failed_projects)}"
else:
active_projects = [
p.project_name
for p in self._project_statuses.values()
if p.status in (SyncStatus.SCANNING, SyncStatus.SYNCING)
]
total_files = sum(p.files_total for p in self._project_statuses.values())
processed_files = sum(p.files_processed for p in self._project_statuses.values())
if total_files > 0:
progress_pct = (processed_files / total_files) * 100
return f"🔄 Syncing {len(active_projects)} projects ({processed_files}/{total_files} files, {progress_pct:.0f}%)"
else:
return f"🔄 Syncing {len(active_projects)} projects"
def clear_completed(self) -> None:
"""Remove completed project statuses to clean up memory."""
self._project_statuses = {
name: status
for name, status in self._project_statuses.items()
if status.status != SyncStatus.COMPLETED
}
self._update_global_status()
# Global sync status tracker instance
sync_status_tracker = SyncStatusTracker()
File diff suppressed because it is too large Load Diff
+14
View File
@@ -236,6 +236,20 @@ class WatchService:
# avoid circular imports
from basic_memory.sync.sync_service import get_sync_service
# Check if project still exists in configuration before processing
# This prevents deleted projects from being recreated by background sync
from basic_memory.config import ConfigManager
config_manager = ConfigManager()
if (
project.name not in config_manager.projects
and project.permalink not in config_manager.projects
):
logger.info(
f"Skipping sync for deleted project: {project.name}, change_count={len(changes)}"
)
return
sync_service = await get_sync_service(project)
file_service = sync_service.file_service
+58
View File
@@ -13,6 +13,49 @@ from loguru import logger
from unidecode import unidecode
def normalize_project_path(path: str) -> str:
"""Normalize project path by stripping mount point prefix.
In cloud deployments, the S3 bucket is mounted at /app/data. We strip this
prefix from project paths to avoid leaking implementation details and to
ensure paths match the actual S3 bucket structure.
For local paths (including Windows paths), returns the path unchanged.
Args:
path: Project path (e.g., "/app/data/basic-memory-llc" or "C:\\Users\\...")
Returns:
Normalized path (e.g., "/basic-memory-llc" or "C:\\Users\\...")
Examples:
>>> normalize_project_path("/app/data/my-project")
'/my-project'
>>> normalize_project_path("/my-project")
'/my-project'
>>> normalize_project_path("app/data/my-project")
'/my-project'
>>> normalize_project_path("C:\\\\Users\\\\project")
'C:\\\\Users\\\\project'
"""
# Check if this is a Windows absolute path (e.g., C:\Users\...)
# Windows paths have a drive letter followed by a colon
if len(path) >= 2 and path[1] == ":":
# Windows absolute path - return unchanged
return path
# Handle both absolute and relative Unix paths
normalized = path.lstrip("/")
if normalized.startswith("app/data/"):
normalized = normalized.removeprefix("app/data/")
# Ensure leading slash for Unix absolute paths
if not normalized.startswith("/"):
normalized = "/" + normalized
return normalized
@runtime_checkable
class PathLike(Protocol):
"""Protocol for objects that can be used as paths."""
@@ -185,6 +228,21 @@ def setup_logging(
logger.info(f"ENV: '{env}' Log level: '{log_level}' Logging to {log_file}")
# Bind environment context for structured logging (works in both local and cloud)
tenant_id = os.getenv("BASIC_MEMORY_TENANT_ID", "local")
fly_app_name = os.getenv("FLY_APP_NAME", "local")
fly_machine_id = os.getenv("FLY_MACHINE_ID", "local")
fly_region = os.getenv("FLY_REGION", "local")
logger.configure(
extra={
"tenant_id": tenant_id,
"fly_app_name": fly_app_name,
"fly_machine_id": fly_machine_id,
"fly_region": fly_region,
}
)
# Reduce noise from third-party libraries
noisy_loggers = {
# HTTP client logs
+139
View File
@@ -0,0 +1,139 @@
# Performance Benchmarks
This directory contains performance benchmark tests for Basic Memory's sync/indexing operations.
## Purpose
These benchmarks measure baseline performance to track improvements from optimizations. They are particularly important for:
- Cloud deployments with ephemeral databases that need fast re-indexing
- Large repositories (100s to 1000s of files)
- Validating optimization efforts
## Running Benchmarks
### Run all benchmarks (excluding slow ones)
```bash
pytest test-int/test_sync_performance_benchmark.py -v -m "benchmark and not slow"
```
### Run specific benchmark
```bash
# 100 files (fast, ~10-30 seconds)
pytest test-int/test_sync_performance_benchmark.py::test_benchmark_sync_100_files -v
# 500 files (medium, ~1-3 minutes)
pytest test-int/test_sync_performance_benchmark.py::test_benchmark_sync_500_files -v
# 1000 files (slow, ~3-10 minutes)
pytest test-int/test_sync_performance_benchmark.py::test_benchmark_sync_1000_files -v
# Re-sync with no changes (tests scan performance)
pytest test-int/test_sync_performance_benchmark.py::test_benchmark_resync_no_changes -v
```
### Run all benchmarks including slow ones
```bash
pytest test-int/test_sync_performance_benchmark.py -v -m benchmark
```
### Skip benchmarks in regular test runs
```bash
pytest -m "not benchmark"
```
## Benchmark Output
Each benchmark provides detailed metrics including:
- **Performance Metrics**:
- Total sync time
- Files processed per second
- Milliseconds per file
- **Database Metrics**:
- Initial database size
- Final database size
- Database growth (total and per file)
- **Operation Counts**:
- New files indexed
- Modified files processed
- Deleted files handled
- Moved files tracked
## Example Output
```
======================================================================
BENCHMARK: Sync 100 files (small repository)
======================================================================
Generating 100 test files...
Created files 0-100 (100/100)
File generation completed in 0.15s (666.7 files/sec)
Initial database size: 120.00 KB
Starting sync of 100 files...
----------------------------------------------------------------------
RESULTS:
----------------------------------------------------------------------
Files processed: 100
New: 100
Modified: 0
Deleted: 0
Moved: 0
Performance:
Total time: 12.34s
Files/sec: 8.1
ms/file: 123.4
Database:
Initial size: 120.00 KB
Final size: 5.23 MB
Growth: 5.11 MB
Growth per file: 52.31 KB
======================================================================
```
## Interpreting Results
### Good Performance Indicators
- **Files/sec > 10**: Good indexing speed for small-medium repos
- **Files/sec > 5**: Acceptable for large repos with complex relations
- **DB growth < 100KB per file**: Reasonable index size
### Areas for Improvement
- **Files/sec < 5**: May benefit from batch operations
- **ms/file > 200**: High latency per file, check for N+1 queries
- **DB growth > 200KB per file**: Search index may be bloated (trigrams?)
## Tracking Improvements
Before making optimizations:
1. Run benchmarks to establish baseline
2. Save output for comparison
3. Note any particular pain points (e.g., slow search indexing)
After optimizations:
1. Run the same benchmarks
2. Compare metrics:
- Files/sec should increase
- ms/file should decrease
- DB growth per file may decrease (with search optimizations)
3. Document improvements in PR
## Related Issues
- [#351: Performance: Optimize sync/indexing for cloud deployments](https://github.com/basicmachines-co/basic-memory/issues/351)
## Test File Generation
Benchmarks generate realistic markdown files with:
- YAML frontmatter with tags
- 3-10 observations per file with categories
- 1-3 relations per file (including forward references)
- Varying content to simulate real usage
- Files organized in category subdirectories

Some files were not shown because too many files have changed in this diff Show More