Compare commits

...

68 Commits

Author SHA1 Message Date
phernandez 4bfec8a88e feat: Add research skill and /research command
New capability to research topics and save structured reports:

/research command:
- /research <topic> [folder]
- Investigates using web search, codebase search, and existing notes
- Produces structured report with findings and analysis
- Saves to research/ folder by default

research skill (model-invoked):
- Triggers on "research", "investigate", "look into", "explore"
- Gathers information from multiple sources
- Synthesizes findings into actionable reports
- Links to sources and related notes

Report structure:
- Summary and research question
- Key findings with evidence
- Analysis and recommendations
- Open questions and sources
- Observations and relations for knowledge graph

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 14:31:02 -06:00
phernandez 9c3d2eb335 refactor: Move plugin into claude-code-plugin subdirectory
Reorganize plugin files into a dedicated subdirectory to keep
them separate from the main Basic Memory Python package:

- Move all plugin files to claude-code-plugin/
- Add README.md with quick start guide
- Update installation paths to use subdirectory

New structure:
```
claude-code-plugin/
├── .claude-plugin/
│   ├── plugin.json
│   └── marketplace.json
├── commands/
├── skills/
├── hooks/
├── README.md
└── PLUGIN.md
```

Installation:
/plugin marketplace add basicmachines-co/basic-memory/claude-code-plugin
/plugin install basic-memory@basicmachines

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 14:08:29 -06:00
phernandez 5fcbae3fdb feat: Add /organize slash command for knowledge graph maintenance
User-invoked command to complement the knowledge-organize skill.

Actions:
- /organize health - Quick overview of KB status (default)
- /organize orphans - Find notes with no relations
- /organize duplicates - Find similar/overlapping notes
- /organize relations [note] - Suggest connections for a note
- /organize tags - Review and normalize tag consistency

Always confirms before modifying notes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 13:58:19 -06:00
phernandez 68ee310a55 refactor: Rename knowledge-organizer to knowledge-organize
Use verb form for skill name to be consistent with action-oriented
naming (like knowledge-capture, not knowledge-capturer).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 13:56:18 -06:00
phernandez 3a7fca8a9e feat: Add knowledge-organizer skill for maintaining knowledge graph
New skill to help users organize and maintain their knowledge base:

Capabilities:
- Find orphan notes (no relations to other notes)
- Suggest relations based on content similarity
- Identify duplicate or overlapping notes
- Review and suggest folder organization
- Normalize inconsistent tags
- Create index/hub notes for topic navigation
- Enrich sparse notes with observations and structure

Includes workflows for:
- Quick health check (overview of KB status)
- Deep organization session (thorough review)
- Topic-focused organization (organize around a subject)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 13:48:43 -06:00
phernandez 4f2b1b2fd3 feat: Add edit-note skills for interactive note editing
Add two new skills for editing Basic Memory notes:

edit-note (MCP-based):
- Works with both cloud and local installations
- Conversational editing workflow via MCP tools
- Uses edit_note operations: append, prepend, find_replace, replace_section
- Shows before/after state for user verification

edit-note-local (file-based):
- For local installations with file system access
- Edits markdown files directly using Claude Code's Read/Edit/Write
- Changes sync automatically via `basic-memory sync --watch`
- Full file access including frontmatter editing

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 13:33:52 -06:00
phernandez a70b355838 feat: Complete plugin with marketplace, commands, and hooks
Add full plugin infrastructure for distribution:

Marketplace:
- Add marketplace.json for self-hosting at basicmachines-co/basic-memory
- Users can add via: /plugin marketplace add basicmachines-co/basic-memory

Slash Commands:
- /remember [title] - Capture insights to Basic Memory
- /continue [topic] - Resume previous work with context
- /context <url> - Build context from memory:// URLs
- /recent [timeframe] - Show recent activity

Hooks:
- PostToolUse: Confirm when notes are saved
- Stop: Suggest /remember for valuable conversations

Updated PLUGIN.md with comprehensive documentation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 13:11:11 -06:00
phernandez 32acbfe982 refactor: Package skills as Claude Code plugin
Convert the Basic Memory skills into a proper Claude Code plugin format:

- Add .claude-plugin/plugin.json manifest with metadata
- Move skills from .claude/skills/ to root skills/ directory
- Add PLUGIN.md with installation and usage documentation

Plugin structure:
```
.claude-plugin/
  plugin.json        # Plugin manifest
skills/
  knowledge-capture/
  continue-conversation/
  spec-driven-development/
PLUGIN.md            # Plugin documentation
```

Users can install via: /plugin install basic-memory@basicmachines

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 13:01:22 -06:00
phernandez 1831397861 feat: Add Claude Code skills for Basic Memory MCP integration
Add three model-invoked skills that help Claude automatically use
Basic Memory's MCP tools in the right contexts:

- knowledge-capture: Capture insights, decisions, and learnings into
  structured notes with observations and relations
- continue-conversation: Resume previous work by building context from
  the knowledge graph using memory:// URLs and recent activity
- spec-driven-development: Guide implementation based on specs stored
  in Basic Memory, following the SPEC-1 process

Unlike slash commands (user-invoked), skills are automatically
discovered and applied by Claude based on conversation context.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 12:52:47 -06:00
jope-bm 28cc5225a7 feat: Implement API v2 with ID-based endpoints (Phase 1) (#441)
Signed-off-by: Joe P <joe@basicmemory.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: phernandez <paul@basicmachines.co>
2025-11-27 10:35:55 -06:00
phernandez 9b7bbc7116 formatting and logic change to resolve_relations, remove fuzzy search 2025-11-25 22:54:37 -06:00
phernandez 138c283d6c add postgres db type 2025-11-25 20:25:56 -06:00
phernandez 7a8954c37e add extra logic for cloud-indexing improvements 2025-11-25 13:52:58 -06:00
phernandez 10c7c19c03 fix db url for sqlite migrations
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-21 13:21:20 -06:00
Paul Hernandez fb5e9e1d77 feat: Add PostgreSQL database backend support (#439)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-11-20 11:20:29 -06:00
phernandez 66b91b2847 ci: Add PostgreSQL testing to GitHub Actions workflow
Add Postgres service container and separate test step for PostgreSQL backend testing.
The Postgres tests only run on Linux runners since GitHub Actions service containers
are only available on Linux.

- Add postgres:17 service container with health checks
- Add 'Run tests (Postgres)' step with Linux-only condition
- Rename existing test step to 'Run tests (SQLite)' for clarity

This enables CI testing of dual database backend support introduced in the
postgres-support feature branch.

Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-18 12:25:18 -06:00
Cedric Hurst b004565df9 fix: handle periods in kebab_filenames mode (#424) 2025-11-18 06:49:55 -05:00
Drew Cain a258b73e1d chore: update version to 0.16.2 for v0.16.2 release 2025-11-16 21:30:59 -06:00
Drew Cain 9a845f2906 docs: prepare for v0.16.2 release 2025-11-16 21:28:13 -06:00
Drew Cain 6517e9845f fix: Use platform-native path separators in config.json (#429)
Signed-off-by: Drew Cain <groksrc@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2025-11-13 09:12:26 -06:00
Drew Cain 1af05392ee fix: Add rclone installation checks for Windows bisync commands (#427) 2025-11-12 14:22:14 -06:00
Brandon Mayes cad7019c89 fix: main project always recreated on project list command (#421) 2025-11-12 09:57:08 -05:00
phernandez 099c334e3d chore: update version to 0.16.1 for v0.16.1 release 2025-11-11 09:21:47 -06:00
phernandez 7685586178 docs: Add v0.16.1 CHANGELOG entry for Windows line ending fix
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-11 09:10:26 -06:00
Paul Hernandez e9d0a944a9 fix: Handle Windows line endings in rclone bisync (#422)
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-11 09:08:20 -06:00
phernandez caf3c14bb1 chore: update version to 0.16.0 for v0.16.0 release 2025-11-10 19:19:48 -06:00
phernandez c5d9067754 docs: Add v0.16.0 CHANGELOG entry with comprehensive release notes
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-10 19:17:29 -06:00
phernandez e0fc59ea97 style: Format upload.py for better readability
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-10 19:15:03 -06:00
Paul Hernandez 49b2adc35c fix: skip archive files during cloud upload (#420)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-11-10 19:04:49 -06:00
Paul Hernandez 1646572f69 fix: Rename write_note entity_type to note_type for clarity (#419)
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-11-10 19:02:50 -06:00
Paul Hernandez f0d7398815 fix: Quote string values in YAML frontmatter to handle special characters (#418)
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-11-10 18:13:48 -06:00
Paul Hernandez 581b7b17c6 fix: Add explicit type annotations to MCP tool parameters (#394)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-11-10 17:02:17 -06:00
Paul Hernandez d775f7bab9 fix: Simplify search_notes schema by removing Optional wrappers (#395)
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-11-10 16:30:59 -06:00
Drew Cain fc01f6abaf fix: Replace Unicode arrows with ASCII for Windows compatibility (#414) 2025-11-10 16:30:41 -06:00
Paul Hernandez 4614fd09d5 fix: Handle dict objects in write_resource endpoint (#415)
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-11-10 16:30:27 -06:00
phernandez 0d4ad7bbf0 remove v0.15.0 info from assistant-guide
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-10 16:25:49 -06:00
jope-bm 7ccec7eba2 feat: Add run_in_background parameter to sync endpoint with tests (#417)
Co-authored-by: Claude <noreply@anthropic.com>
2025-11-07 09:04:43 -07:00
Paul Hernandez 021af74545 fix: Strip duplicate headers in edit_note replace_section (#396)
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Drew Cain <groksrc@users.noreply.github.com>
2025-11-02 14:20:11 -06:00
phernandez 2ad0ee9d5d fix: Use force_full=true for database sync after project sync/bisync
After rclone synchronizes files between local and cloud storage, the
database needs to perform a full scan to ensure it captures all changes.
Previously, incremental sync (watermark optimization) could miss files
that were changed remotely.

Changes:
- project sync command now calls /project/sync?force_full=true
- project bisync command now calls /project/sync?force_full=true
- Ensures complete database refresh after file synchronization

This guarantees the database is fully in sync with the filesystem
after any rclone sync or bisync operation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-02 13:26:08 -06:00
Brandon Mayes c9946ecf1e fix: Various rclone fixes for cloud sync on Windows (#410)
Signed-off-by: Brandon Mayes <5610870+bdmayes@users.noreply.github.com>
Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: phernandez <paul@basicmachines.co>
2025-11-02 11:26:57 -06:00
Drew Cain 0ba6f219f1 fix: Windows CLI Unicode encoding errors (#411)
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
2025-11-02 10:20:48 -06:00
Paul Hernandez 0b3272ae6e feat: SPEC-20 Simplified Project-Scoped Rclone Sync (#405)
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
2025-11-02 09:35:26 -06:00
Paul Hernandez a7d7cc5ee6 fix: Normalize YAML frontmatter types to prevent AttributeError (#236) (#402)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-27 09:19:50 -05:00
Paul Hernandez a7e696b039 Add free trial information to README
Added information about a 7-day free trial.

Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
2025-10-24 10:05:11 -05:00
Paul Hernandez 8aaddb6d45 Add free trial information to README
Added information about a 7-day free trial to the README.

Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
2025-10-24 10:04:41 -05:00
Paul Hernandez d7565312fc Announce Basic Memory Cloud launch in README
Added a section announcing the launch of Basic Memory Cloud with details on cross-device support and early supporter pricing.

Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
2025-10-24 09:53:14 -05:00
Paul Hernandez c7e6eab02f feat: Add delete_notes parameter to remove project endpoint (#391)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-21 14:09:20 -05:00
Paul Hernandez bb8da31472 fix: Handle null, empty, and string 'None' title in markdown frontmatter (#387) (#389)
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-10-21 09:29:19 -05:00
Paul Hernandez e78345ff25 feat: Streaming Foundation & Async I/O Consolidation (SPEC-19) (#384)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-21 09:03:59 -05:00
Paul Hernandez 32236cd247 fix: Handle YAML parsing errors gracefully in update_frontmatter (#378) (#379)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-16 20:17:58 -05:00
Paul Hernandez 4fd6d0c648 fix: Optimize sync memory usage to prevent OOM on large projects (#380)
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-10-16 20:17:34 -05:00
Paul Hernandez e6c8e3662c fix: preserve mtime webdav upload 376 (#377)
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-16 17:18:10 -05:00
Paul Hernandez 449b62d947 fix: Prevent deleted projects from being recreated by background sync (#193) (#370)
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-10-16 15:16:24 -05:00
Paul Hernandez b7497d7484 fix: Use filesystem timestamps for entity sync instead of database operation time (#138) (#369)
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-10-16 14:21:27 -05:00
Paul Hernandez d1431bdb1b fix: Handle YAML parsing errors and missing entity_type in markdown files (#368)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-16 13:18:58 -05:00
Paul Hernandez 171bef717f fix: Resolve UNIQUE constraint violation in entity upsert with observations (#187) (#367)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-16 12:30:56 -05:00
Paul Hernandez 729a5a3b8d fix: Terminate sync immediately when project is deleted (#366)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-16 11:07:12 -05:00
Paul Hernandez 434cdf24dd feat: Add circuit breaker for file sync failures (#364)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-16 09:47:48 -05:00
Paul Hernandez 7f9c1a97a4 feat: Add --verbose and --no-gitignore options to cloud upload (#362)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-15 20:03:36 -05:00
Paul Hernandez 53fb13b054 fix: Make project creation endpoint idempotent (#357)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-15 19:27:41 -05:00
Paul Hernandez bd6c8348b8 fix: Handle None text values in Claude conversations importer (#353)
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-10-15 16:08:57 -05:00
phernandez 994c8b8e7e chore: update version to 0.15.2 for v0.15.2 release 2025-10-14 09:36:47 -05:00
phernandez a78e8c3ac5 style: Apply linter formatting changes 2025-10-14 09:34:10 -05:00
phernandez 53900c5baa fix: Project commands now respect cloud_mode at runtime
- Moved config evaluation from module load time to runtime
- Unified add_project command to handle both cloud and local modes
- Commands (default, sync-config, move) now check cloud_mode at runtime
- Fixes test failures where monkeypatch wasn't applied before command registration
2025-10-14 09:24:13 -05:00
phernandez 02c6de3387 docs: Add v0.15.2 changelog entry 2025-10-14 00:43:14 -05:00
phernandez 9ccf4b6b56 remove extra conole out from sync message after upload
Signed-off-by: phernandez <paul@basicmachines.co>
2025-10-14 00:38:02 -05:00
phernandez ba74ca7e18 fix: Update CloudProjectCreateResponse schema to match API response
The /proxy/projects/projects POST endpoint returns a ProjectStatusResponse
with fields: message, status, default, old_project, new_project.

Updated CloudProjectCreateResponse schema to match this format instead of
expecting name, path, message fields.

Also updated all related tests to use the correct response format:
- tests/cli/test_cloud_utils.py (3 tests)
- tests/cli/test_bisync_commands.py (1 test)

Fixes the validation error when creating cloud projects via upload command:
"bm cloud upload --project test --create-project specs"

Signed-off-by: Pablo Hernandez <pablo@basicmachines.co>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-10-13 23:56:59 -05:00
Paul Hernandez 5258f45730 feat: Add WebDAV upload command for cloud projects (#356)
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Pablo Hernandez <pablo@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-13 23:20:54 -05:00
217 changed files with 24293 additions and 14269 deletions
+94 -17
View File
@@ -15,10 +15,16 @@ Create a stable release using the automated justfile target with comprehensive v
You are an expert release manager for the Basic Memory project. When the user runs `/release`, execute the following steps:
### Step 1: Pre-flight Validation
1. Verify version format matches `v\d+\.\d+\.\d+` pattern
2. Check current git status for uncommitted changes
3. Verify we're on the `main` branch
4. Confirm no existing tag with this version
#### Version Check
1. Check current version in `src/basic_memory/__init__.py`
2. Verify new version format matches `v\d+\.\d+\.\d+` pattern
3. Confirm version is higher than current version
#### Git Status
1. Check current git status for uncommitted changes
2. Verify we're on the `main` branch
3. Confirm no existing tag with this version
#### Documentation Validation
1. **Changelog Check**
@@ -39,19 +45,83 @@ The justfile target handles:
- ✅ Version update in `src/basic_memory/__init__.py`
- ✅ Automatic commit with proper message
- ✅ Tag creation and pushing to GitHub
- ✅ Release workflow trigger
- ✅ Release workflow trigger (automatic on tag push)
The GitHub Actions workflow (`.github/workflows/release.yml`) then:
- ✅ Builds the package using `uv build`
- ✅ Creates GitHub release with auto-generated notes
- ✅ Publishes to PyPI
- ✅ Updates Homebrew formula (stable releases only)
### Step 3: Monitor Release Process
1. Check that GitHub Actions workflow starts successfully
2. Monitor workflow completion at: https://github.com/basicmachines-co/basic-memory/actions
3. Verify PyPI publication
4. Test installation: `uv tool install basic-memory`
1. Verify tag push triggered the workflow (should start automatically within seconds)
2. Monitor workflow progress at: https://github.com/basicmachines-co/basic-memory/actions
3. Watch for successful completion of both jobs:
- `release` - Builds package and publishes to PyPI
- `homebrew` - Updates Homebrew formula (stable releases only)
4. Check for any workflow failures and investigate logs if needed
### Step 4: Post-Release Validation
1. Verify GitHub release is created automatically
2. Check PyPI publication
3. Validate release assets
4. Update any post-release documentation
#### GitHub Release
1. Verify GitHub release is created at: https://github.com/basicmachines-co/basic-memory/releases/tag/<version>
2. Check that release notes are auto-generated from commits
3. Validate release assets (`.whl` and `.tar.gz` files are attached)
#### PyPI Publication
1. Verify package published at: https://pypi.org/project/basic-memory/<version>/
2. Test installation: `uv tool install basic-memory`
3. Verify installed version: `basic-memory --version`
#### Homebrew Formula (Stable Releases Only)
1. Check formula update at: https://github.com/basicmachines-co/homebrew-basic-memory
2. Verify formula version matches release
3. Test Homebrew installation: `brew install basicmachines-co/basic-memory/basic-memory`
#### Website Updates
**1. basicmachines.co** (`/Users/drew/code/basicmachines.co`)
- **Goal**: Update version number displayed on the homepage
- **Location**: Search for "Basic Memory v0." in the codebase to find version displays
- **What to update**:
- Hero section heading that shows "Basic Memory v{VERSION}"
- "What's New in v{VERSION}" section heading
- Feature highlights array (look for array of features with title/description)
- **Process**:
1. Pull latest from GitHub: `git pull origin main`
2. Create release branch: `git checkout -b release/v{VERSION}`
3. Search codebase for current version number (e.g., "v0.16.1")
4. Update version numbers to new release version
5. Update feature highlights with 3-5 key features from this release (extract from CHANGELOG.md)
6. Commit changes: `git commit -m "chore: update to v{VERSION}"`
7. Push branch: `git push origin release/v{VERSION}`
- **Deploy**: Follow deployment process for basicmachines.co
**2. docs.basicmemory.com** (`/Users/drew/code/docs.basicmemory.com`)
- **Goal**: Add new release notes section to the latest-releases page
- **File**: `src/pages/latest-releases.mdx`
- **What to do**:
1. Pull latest from GitHub: `git pull origin main`
2. Create release branch: `git checkout -b release/v{VERSION}`
3. Read the existing file to understand the format and structure
4. Read `/Users/drew/code/basic-memory/CHANGELOG.md` to get release content
5. Add new release section **at the top** (after MDX imports, before other releases)
6. Follow the existing pattern:
- Heading: `## [v{VERSION}](github-link) — YYYY-MM-DD`
- Focus statement if applicable
- `<Info>` block with highlights (3-5 key items)
- Sections for Features, Bug Fixes, Breaking Changes, etc.
- Link to full changelog at the end
- Separator `---` between releases
7. Commit changes: `git commit -m "docs: add v{VERSION} release notes"`
8. Push branch: `git push origin release/v{VERSION}`
- **Source content**: Extract and format sections from CHANGELOG.md for this version
- **Deploy**: Follow deployment process for docs.basicmemory.com
**4. Announce Release**
- Post to Discord community if significant changes
- Update social media if major release
- Notify users via appropriate channels
## Pre-conditions Check
Before starting, verify:
@@ -74,13 +144,18 @@ Before starting, verify:
🏷️ Tag: v0.13.2
📋 GitHub Release: https://github.com/basicmachines-co/basic-memory/releases/tag/v0.13.2
📦 PyPI: https://pypi.org/project/basic-memory/0.13.2/
🍺 Homebrew: https://github.com/basicmachines-co/homebrew-basic-memory
🚀 GitHub Actions: Completed
Install with:
uv tool install basic-memory
Install with pip/uv:
uv tool install basic-memory
Install with Homebrew:
brew install basicmachines-co/basic-memory/basic-memory
Users can now upgrade:
uv tool upgrade basic-memory
uv tool upgrade basic-memory
brew upgrade basic-memory
```
## Context
@@ -89,4 +164,6 @@ uv tool upgrade basic-memory
- Uses the automated justfile target for consistency
- Version is automatically updated in `__init__.py`
- Triggers automated GitHub release with changelog
- Leverages uv-dynamic-versioning for package version management
- Package is published to PyPI for `pip` and `uv` users
- Homebrew formula is automatically updated for stable releases
- Supports multiple installation methods (uv, pip, Homebrew)
+16 -20
View File
@@ -1,17 +1,19 @@
---
allowed-tools: mcp__basic-memory__write_note, mcp__basic-memory__read_note, mcp__basic-memory__search_notes, mcp__basic-memory__edit_note, Task
argument-hint: [create|status|implement|review] [spec-name]
allowed-tools: mcp__basic-memory__write_note, mcp__basic-memory__read_note, mcp__basic-memory__search_notes, mcp__basic-memory__edit_note
argument-hint: [create|status|show|review] [spec-name]
description: Manage specifications in our development process
---
## Context
You are managing specifications using our specification-driven development process defined in @docs/specs/SPEC-001.md.
Specifications are managed in the Basic Memory "specs" project. All specs live in a centralized location accessible across all repositories via MCP tools.
See SPEC-1 and SPEC-2 in the "specs" project for the full specification-driven development process.
Available commands:
- `create [name]` - Create new specification
- `status` - Show all spec statuses
- `implement [spec-name]` - Hand spec to appropriate agent
- `show [spec-name]` - Read a specific spec
- `review [spec-name]` - Review implementation against spec
## Your task
@@ -19,23 +21,19 @@ Available commands:
Execute the spec command: `/spec $ARGUMENTS`
### If command is "create":
1. Get next SPEC number by searching existing specs
2. Create new spec using template from @docs/specs/Slash\ Commands\ Reference.md
3. Place in `/specs` folder with title "SPEC-XXX: [name]"
1. Get next SPEC number by searching existing specs in "specs" project
2. Create new spec using template from SPEC-2
3. Use mcp__basic-memory__write_note with project="specs"
4. Include standard sections: Why, What, How, How to Evaluate
### If command is "status":
1. Search all notes in `/specs` folder
2. Display table with spec number, title, and status
3. Show any dependencies or assigned agents
1. Use mcp__basic-memory__search_notes with project="specs"
2. Display table with spec number, title, and progress
3. Show completion status from checkboxes in content
### If command is "implement":
1. Read the specified spec
2. Determine appropriate agent based on content:
- Frontend/UI → vue-developer
- Architecture/system → system-architect
- Backend/API → python-developer
3. Launch Task tool with appropriate agent and spec context
### If command is "show":
1. Use mcp__basic-memory__read_note with project="specs"
2. Display the full spec content
### If command is "review":
1. Read the specified spec and its "How to Evaluate" section
@@ -49,7 +47,5 @@ Execute the spec command: `/spec $ARGUMENTS`
- **Architecture compliance** - Component isolation, state management patterns
- **Documentation completeness** - Implementation matches specification
3. Provide honest, accurate assessment - do not overstate completeness
4. Document findings and update spec with review results
4. Document findings and update spec with review results using mcp__basic-memory__edit_note
5. If gaps found, clearly identify what still needs to be implemented/tested
Use the agent definitions from @docs/specs/Agent\ Definitions.md for implementation handoffs.
+28
View File
@@ -0,0 +1,28 @@
# Basic Memory Environment Variables Example
# Copy this file to .env and customize as needed
# Note: .env files are gitignored and should never be committed
# ============================================================================
# PostgreSQL Test Database Configuration
# ============================================================================
# These variables allow you to override the default test database credentials
# Default values match docker-compose-postgres.yml for local development
#
# Only needed if you want to use different credentials or a remote test database
# By default, tests use: postgresql://basic_memory_user:dev_password@localhost:5433/basic_memory_test
# Full PostgreSQL test database URL (used by tests and migrations)
# POSTGRES_TEST_URL=postgresql+asyncpg://basic_memory_user:dev_password@localhost:5433/basic_memory_test
# Individual components (used by justfile postgres-reset command)
# POSTGRES_USER=basic_memory_user
# POSTGRES_TEST_DB=basic_memory_test
# ============================================================================
# Production Database Configuration
# ============================================================================
# For production use, set these in your deployment environment
# DO NOT use the test credentials above in production!
# BASIC_MEMORY_DATABASE_BACKEND=postgres # or "sqlite"
# BASIC_MEMORY_DATABASE_URL=postgresql+asyncpg://user:password@host:port/database
+4 -1
View File
@@ -71,9 +71,12 @@ jobs:
- [ ] Proper error handling and logging
- [ ] Performance considerations addressed
- [ ] No sensitive data in logs or commits
## Compatability
- [ ] File path comparisons must be windows compatible
- [ ] Avoid using emojis and unicode characters in console and log output
Read the CLAUDE.md file for detailed project context. For each checklist item, verify if it's satisfied and comment on any that need attention. Use inline comments for specific code issues and post a summary with checklist results.
# Allow broader tool access for thorough code review
claude_args: '--allowed-tools "Bash(gh pr:*),Bash(gh issue:*),Bash(gh api:*),Bash(git log:*),Bash(git show:*),Read,Grep,Glob"'
+60 -3
View File
@@ -13,7 +13,8 @@ on:
branches: [ "main" ]
jobs:
test:
test-sqlite:
name: Test SQLite (${{ matrix.os }}, Python ${{ matrix.python-version }})
strategy:
fail-fast: false
matrix:
@@ -64,7 +65,63 @@ jobs:
run: |
just lint
- name: Run tests
- name: Run tests (SQLite)
run: |
uv pip install pytest pytest-cov
just test
just test-sqlite
test-postgres:
name: Test Postgres (Python ${{ matrix.python-version }})
strategy:
fail-fast: false
matrix:
python-version: [ "3.12", "3.13" ]
runs-on: ubuntu-latest
# Postgres service (only available on Linux runners)
services:
postgres:
image: postgres:17
env:
POSTGRES_DB: basic_memory_test
POSTGRES_USER: basic_memory_user
POSTGRES_PASSWORD: dev_password
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 5433:5432
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
- name: Install uv
run: |
pip install uv
- name: Install just
run: |
curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to /usr/local/bin
- name: Create virtual env
run: |
uv venv
- name: Install dependencies
run: |
uv pip install -e .[dev]
- name: Run tests (Postgres)
run: |
uv pip install pytest pytest-cov
just test-postgres
+2 -1
View File
@@ -52,4 +52,5 @@ ENV/
# claude action
claude-output
**/.claude/settings.local.json
**/.claude/settings.local.json
.mcp.json
+312 -1
View File
@@ -1,5 +1,316 @@
# CHANGELOG
## v0.16.2 (2025-11-16)
### Bug Fixes
- **#429**: Use platform-native path separators in config.json
([`6517e98`](https://github.com/basicmachines-co/basic-memory/commit/6517e98))
- Fixes config.json path separator issues on Windows
- Uses os.path.join for platform-native path construction
- Ensures consistent path handling across platforms
- **#427**: Add rclone installation checks for Windows bisync commands
([`1af0539`](https://github.com/basicmachines-co/basic-memory/commit/1af0539))
- Validates rclone installation before running bisync commands
- Provides clear error messages when rclone is not installed
- Improves user experience on Windows
- **#421**: Main project always recreated on project list command
([`cad7019`](https://github.com/basicmachines-co/basic-memory/commit/cad7019))
- Fixes issue where main project was recreated unnecessarily
- Improves project list command reliability
- Reduces unnecessary file system operations
## v0.16.1 (2025-11-11)
### Bug Fixes
- **#422**: Handle Windows line endings in rclone bisync
([`e9d0a94`](https://github.com/basicmachines-co/basic-memory/commit/e9d0a94))
- Added `--compare=modtime` flag to rclone bisync to ignore size differences from line ending conversions
- Fixes issue where LF→CRLF conversion on Windows was treated as file corruption
- Resolves "corrupted on transfer: sizes differ" errors during cloud sync on Windows
- Users will need to run `--resync` once after updating to establish new baseline
## v0.16.0 (2025-11-10)
### Features
- **#417**: Add run_in_background parameter to sync endpoint
([`7ccec7e`](https://github.com/basicmachines-co/basic-memory/commit/7ccec7e))
- New `run_in_background` parameter for async sync operations
- Improved API flexibility for long-running sync tasks
- Comprehensive test coverage for background sync behavior
- **#405**: SPEC-20 Simplified Project-Scoped Rclone Sync
([`0b3272a`](https://github.com/basicmachines-co/basic-memory/commit/0b3272a))
- Simplified and more reliable cloud synchronization
- Project-scoped rclone configuration
- Better error handling and status reporting
- **#384**: Streaming Foundation & Async I/O Consolidation (SPEC-19)
([`e78345f`](https://github.com/basicmachines-co/basic-memory/commit/e78345f))
- Foundation for streaming support in future releases
- Consolidated async I/O patterns across codebase
- Improved performance and resource management
- **#364**: Add circuit breaker for file sync failures
([`434cdf2`](https://github.com/basicmachines-co/basic-memory/commit/434cdf2))
- Prevents cascading failures during sync operations
- Automatic recovery from transient errors
- Better resilience in cloud sync scenarios
- **#362**: Add --verbose and --no-gitignore options to cloud upload
([`7f9c1a9`](https://github.com/basicmachines-co/basic-memory/commit/7f9c1a9))
- Enhanced upload control with verbose logging
- Option to bypass gitignore filtering when needed
- Better debugging and troubleshooting capabilities
- **#391**: Add delete_notes parameter to remove project endpoint
([`c9946ec`](https://github.com/basicmachines-co/basic-memory/commit/c9946ec))
- Option to delete notes when removing projects
- Safer project cleanup workflows
- Prevents accidental data loss
### Bug Fixes
- **#420**: Skip archive files during cloud upload
([`49b2adc`](https://github.com/basicmachines-co/basic-memory/commit/49b2adc))
- Prevents uploading of zip, tar, gz and other archive files
- Reduces storage usage and upload time
- Better file filtering during cloud operations
- **#419**: Rename write_note entity_type to note_type for clarity
([`1646572`](https://github.com/basicmachines-co/basic-memory/commit/1646572))
- Clearer parameter naming in write_note tool
- Improved API consistency and documentation
- Better developer experience
- **#418**: Quote string values in YAML frontmatter to handle special characters
([`f0d7398`](https://github.com/basicmachines-co/basic-memory/commit/f0d7398))
- Fixes YAML parsing errors with special characters
- More robust frontmatter handling
- Prevents data corruption in edge cases
- **#415**: Handle dict objects in write_resource endpoint
([`4614fd0`](https://github.com/basicmachines-co/basic-memory/commit/4614fd0))
- Fixes errors when writing dictionary resources
- Better type handling in resource endpoints
- Improved API robustness
- **#414**: Replace Unicode arrows with ASCII for Windows compatibility
([`fc01f6a`](https://github.com/basicmachines-co/basic-memory/commit/fc01f6a))
- Fixes display issues on Windows terminals
- Better cross-platform compatibility
- Improved CLI user experience on Windows
- **#411**: Windows CLI Unicode encoding errors
([`0ba6f21`](https://github.com/basicmachines-co/basic-memory/commit/0ba6f21))
- Resolves Unicode encoding issues on Windows
- Better handling of international characters
- Improved Windows platform support
- **#410**: Various rclone fixes for cloud sync on Windows
([`c9946ec`](https://github.com/basicmachines-co/basic-memory/commit/c9946ec))
- Fixes cloud sync reliability on Windows
- Better path handling for Windows filesystem
- Improved rclone integration on Windows
- **#402**: Normalize YAML frontmatter types to prevent AttributeError
([`a7d7cc5`](https://github.com/basicmachines-co/basic-memory/commit/a7d7cc5))
- Fixes AttributeError when reading frontmatter
- More robust type normalization
- Better error handling in markdown parsing
- **#396**: Strip duplicate headers in edit_note replace_section
([`021af74`](https://github.com/basicmachines-co/basic-memory/commit/021af74))
- Prevents duplicate headers when replacing sections
- Cleaner note editing behavior
- Better content consistency
- **#395**: Simplify search_notes schema by removing Optional wrappers
([`d775f7b`](https://github.com/basicmachines-co/basic-memory/commit/d775f7b))
- Cleaner API schema definition
- Better type safety and validation
- Improved developer experience
- **#394**: Add explicit type annotations to MCP tool parameters
([`581b7b1`](https://github.com/basicmachines-co/basic-memory/commit/581b7b1))
- Better type safety in MCP tools
- Improved IDE support and autocomplete
- Clearer tool documentation
- **#389**: Handle null, empty, and string 'None' title in markdown frontmatter
([`bb8da31`](https://github.com/basicmachines-co/basic-memory/commit/bb8da31))
- Fixes errors with malformed frontmatter titles
- More robust title handling
- Better error recovery
- **#380**: Optimize sync memory usage to prevent OOM on large projects
([`4fd6d0c`](https://github.com/basicmachines-co/basic-memory/commit/4fd6d0c))
- Prevents out-of-memory errors on large knowledge bases
- Better memory management during sync
- Improved scalability
- **#379**: Handle YAML parsing errors gracefully in update_frontmatter
([`32236cd`](https://github.com/basicmachines-co/basic-memory/commit/32236cd))
- Better error handling for malformed YAML
- Graceful degradation instead of crashes
- Improved robustness
- **#377**: Preserve mtime on WebDAV upload
([`e6c8e36`](https://github.com/basicmachines-co/basic-memory/commit/e6c8e36))
- Maintains file modification times during upload
- Better sync accuracy
- Prevents unnecessary re-syncing
- **#370**: Prevent deleted projects from being recreated by background sync
([`449b62d`](https://github.com/basicmachines-co/basic-memory/commit/449b62d))
- Fixes race condition with project deletion
- Better lifecycle management
- Prevents unwanted project recreation
- **#369**: Use filesystem timestamps for entity sync instead of database operation time
([`b7497d7`](https://github.com/basicmachines-co/basic-memory/commit/b7497d7))
- More accurate sync detection
- Better handling of external file modifications
- Improved sync reliability
- **#368**: Handle YAML parsing errors and missing entity_type in markdown files
([`d1431bd`](https://github.com/basicmachines-co/basic-memory/commit/d1431bd))
- Better error handling for malformed markdown
- Graceful handling of missing metadata
- Improved robustness
- **#367**: Resolve UNIQUE constraint violation in entity upsert with observations
([`171bef7`](https://github.com/basicmachines-co/basic-memory/commit/171bef7))
- Fixes database constraint errors during sync
- Better handling of duplicate observations
- Improved data integrity
- **#366**: Terminate sync immediately when project is deleted
([`729a5a3`](https://github.com/basicmachines-co/basic-memory/commit/729a5a3))
- Faster project deletion
- Better resource cleanup
- Improved user experience
- **#357**: Make project creation endpoint idempotent
([`53fb13b`](https://github.com/basicmachines-co/basic-memory/commit/53fb13b))
- Prevents errors when creating existing projects
- Better API reliability
- Improved cloud integration
- **#353**: Handle None text values in Claude conversations importer
([`bd6c834`](https://github.com/basicmachines-co/basic-memory/commit/bd6c834))
- Fixes import errors with empty messages
- Better error handling in importers
- Improved data migration
### Performance Improvements
- Force full database sync after project sync/bisync operations
([`2ad0ee9`](https://github.com/basicmachines-co/basic-memory/commit/2ad0ee9))
- Ensures database consistency after cloud operations
- Better sync reliability
- Improved data integrity
### Documentation
- Add free trial information to README
([`a7d7cc5`](https://github.com/basicmachines-co/basic-memory/commit/a7d7cc5), [`8aaddb6`](https://github.com/basicmachines-co/basic-memory/commit/8aaddb6))
- Updated README with Basic Memory Cloud trial info
- Better onboarding experience
- Clearer pricing information
- Announce Basic Memory Cloud launch in README
([`d756531`](https://github.com/basicmachines-co/basic-memory/commit/d756531))
- Official cloud service announcement
- Updated documentation for cloud features
- Improved product positioning
### Migration Guide
No manual migration required. Upgrade with:
```bash
# Update via uv
uv tool upgrade basic-memory
# Or install fresh
uv tool install basic-memory
```
**What's New in v0.16.0:**
- Streaming foundation and consolidated async I/O (SPEC-19)
- Simplified project-scoped rclone sync (SPEC-20)
- Circuit breaker for sync failure resilience
- Enhanced Windows platform support
- Improved cloud upload with verbose and gitignore options
- Better error handling across YAML parsing and frontmatter
- Memory optimization for large projects
- Archive file filtering during upload
**Breaking Changes:**
- `write_note` parameter renamed: `entity_type``note_type` for clarity
### Installation
```bash
# Latest stable release
uv tool install basic-memory
# Update existing installation
uv tool upgrade basic-memory
# Docker
docker pull ghcr.io/basicmachines-co/basic-memory:v0.16.0
```
## v0.15.2 (2025-10-14)
### Features
- **#356**: Add WebDAV upload command for cloud projects
([`5258f45`](https://github.com/basicmachines-co/basic-memory/commit/5258f457))
- New `bm cloud upload` command for uploading local files/directories to cloud projects
- WebDAV-based file transfer with automatic directory creation
- Support for `.gitignore` and `.bmignore` pattern filtering
- Automatic project creation with `--create-project` flag
- Optional post-upload sync with `--sync` flag (enabled by default)
- Human-readable file size reporting (bytes, KB, MB)
- Comprehensive test coverage (28 unit tests)
### Migration Guide
No manual migration required. Upgrade with:
```bash
# Update via uv
uv tool upgrade basic-memory
# Or install fresh
uv tool install basic-memory
```
**What's New:**
- Upload local files to cloud projects with `bm cloud upload`
- Streamlined cloud project creation and management
- Better file filtering with gitignore integration
### Installation
```bash
# Latest stable release
uv tool install basic-memory
# Update existing installation
uv tool upgrade basic-memory
# Docker
docker pull ghcr.io/basicmachines-co/basic-memory:v0.15.2
```
## v0.15.1 (2025-10-13)
### Performance Improvements
@@ -1728,4 +2039,4 @@ Co-authored-by: phernandez <phernandez@basicmachines.co>
### Chores
- Remove basic-foundation src ref in pyproject.toml
([`29fce8b`](https://github.com/basicmachines-co/basic-memory/commit/29fce8b0b922d54d7799bf2534107ee6cfb961b8))
([`29fce8b`](https://github.com/basicmachines-co/basic-memory/commit/29fce8b0b922d54d7799bf2534107ee6cfb961b8))
+1
View File
@@ -264,5 +264,6 @@ With GitHub integration, the development workflow includes:
2. **Contribution tracking** - All of Claude's contributions are properly attributed in the Git history
3. **Branch management** - Claude can create feature branches for implementations
4. **Documentation maintenance** - Claude can keep documentation updated as the code evolves
5. **Code Commits**: ALWAYS sign off commits with `git commit -s`
This level of integration represents a new paradigm in AI-human collaboration, where the AI assistant becomes a full-fledged team member rather than just a tool for generating code snippets.
+62 -1
View File
@@ -7,6 +7,16 @@
![](https://badge.mcpx.dev?type=dev 'MCP Dev')
[![smithery badge](https://smithery.ai/badge/@basicmachines-co/basic-memory)](https://smithery.ai/server/@basicmachines-co/basic-memory)
## 🚀 Basic Memory Cloud is Live!
- **Cross-device and multi-platform support is here.** Your knowledge graph now works on desktop, web, and mobile - seamlessly synced across all your AI tools (Claude, ChatGPT, Gemini, Claude Code, and Codex)
- **Early Supporter Pricing:** Early users get 25% off forever.
The open source project continues as always. Cloud just makes it work everywhere.
[Sign up now →](https://basicmemory.com/beta)
with a 7 day free trial
# Basic Memory
Basic Memory lets you build persistent knowledge through natural conversations with Large Language Models (LLMs) like
@@ -423,6 +433,57 @@ See the [Documentation](https://memory.basicmachines.co/) for more info, includi
- [Managing multiple Projects](https://docs.basicmemory.com/guides/cli-reference/#project)
- [Importing data from OpenAI/Claude Projects](https://docs.basicmemory.com/guides/cli-reference/#import)
## Development
### Running Tests
Basic Memory supports dual database backends (SQLite and Postgres). Tests are parametrized to run against both backends automatically.
**Quick Start:**
```bash
# Run SQLite tests (default, no Docker needed)
just test-sqlite
# Run Postgres tests (requires Docker)
just test-postgres
```
**Available Test Commands:**
- `just test-sqlite` - Run tests against SQLite only (fastest, no Docker needed)
- `just test-postgres` - Run tests against Postgres only (requires Docker)
- `just test-windows` - Run Windows-specific tests (auto-skips on other platforms)
- `just test-benchmark` - Run performance benchmark tests
- `just test-all` - Run all tests including Windows, Postgres, and benchmarks
**Postgres Testing Requirements:**
To run Postgres tests, you need to start the test database:
```bash
docker-compose -f docker-compose-postgres.yml up -d
```
Tests will connect to `localhost:5433/basic_memory_test`.
**Test Markers:**
Tests use pytest markers for selective execution:
- `postgres` - Tests that run against Postgres backend
- `windows` - Windows-specific database optimizations
- `benchmark` - Performance tests (excluded from default runs)
**Other Development Commands:**
```bash
just install # Install with dev dependencies
just lint # Run linting checks
just typecheck # Run type checking
just format # Format code with ruff
just check # Run all quality checks
just migration "msg" # Create database migration
```
See the [justfile](justfile) for the complete list of development commands.
## License
AGPL-3.0
@@ -440,4 +501,4 @@ and submitting PRs.
</picture>
</a>
Built with ♥️ by Basic Machines
Built with ♥️ by Basic Machines
@@ -0,0 +1,23 @@
{
"name": "basicmachines",
"owner": {
"name": "Basic Machines",
"email": "hello@basicmachines.co"
},
"metadata": {
"description": "Official plugins from Basic Machines for knowledge management and AI-assisted development",
"version": "0.1.0"
},
"plugins": [
{
"name": "basic-memory",
"source": ".",
"description": "Skills, commands, and hooks for Basic Memory MCP - capture knowledge, continue conversations, and follow spec-driven development",
"version": "0.1.0",
"author": {
"name": "Basic Machines"
},
"keywords": ["memory", "knowledge", "mcp", "specs", "context"]
}
]
}
@@ -0,0 +1,9 @@
{
"name": "basic-memory",
"description": "Claude Code skills for Basic Memory - capture knowledge, continue conversations, and follow spec-driven development using the Basic Memory MCP server",
"version": "0.1.0",
"author": {
"name": "Basic Machines"
},
"repository": "https://github.com/basicmachines-co/basic-memory"
}
+313
View File
@@ -0,0 +1,313 @@
# Basic Memory Plugin for Claude Code
This plugin provides skills, commands, and hooks for working with [Basic Memory](https://basicmemory.io) - a local-first knowledge management system built on the Model Context Protocol (MCP).
## Prerequisites
You need the Basic Memory MCP server running. Install it via:
```bash
# Install basic-memory
pip install basic-memory
# Or with pipx
pipx install basic-memory
```
Then add it to your Claude Code MCP configuration.
## Installation
### Add the Marketplace
```
/plugin marketplace add basicmachines-co/basic-memory/claude-code-plugin
```
### Install the Plugin
```
/plugin install basic-memory@basicmachines
```
### Or via Repository Settings
Add to your `.claude/settings.json`:
```json
{
"plugins": {
"extraKnownMarketplaces": {
"basicmachines": {
"source": {
"source": "github",
"repo": "basicmachines-co/basic-memory",
"path": "claude-code-plugin"
}
}
},
"installed": ["basic-memory@basicmachines"]
}
}
```
---
## Slash Commands
User-invoked commands for explicit interaction with Basic Memory.
### `/remember [title] [folder]`
Capture insights, decisions, or learnings from the current conversation.
```
/remember "FastAPI Async Pattern"
/remember "Auth Decision" decisions
```
Creates a structured note with:
- Context from the conversation
- Observations with `[decision]`, `[insight]`, `[pattern]` categories
- Relations linking to related concepts
### `/continue [topic]`
Resume previous work by building context from Basic Memory.
```
/continue postgres migration
/continue SPEC-24
/continue
```
If no topic is provided, shows recent activity and asks what to dive into.
### `/context <memory://url> [depth] [timeframe]`
Build context from a specific memory:// URL.
```
/context memory://SPEC-24
/context memory://architecture/* 3 2weeks
```
### `/recent [timeframe] [project]`
Show recent activity in Basic Memory.
```
/recent
/recent 1week
/recent today specs
```
### `/organize [action] [project]`
Organize and maintain your knowledge graph.
```
/organize # Quick health check
/organize orphans # Find unlinked notes
/organize duplicates # Find similar notes
/organize relations "Note" # Suggest links for a note
/organize tags # Review tag consistency
```
Actions:
- `health` - Overview of knowledge base status (default)
- `orphans` - Find notes with no relations
- `duplicates` - Find overlapping notes
- `relations` - Suggest connections
- `tags` - Review tag consistency
### `/research <topic> [folder]`
Research a topic and save a structured report to Basic Memory.
```
/research MCP protocol
/research "database migrations"
/research "auth options" decisions
```
Produces a report with:
- Summary and key findings
- Analysis and recommendations
- Sources and related notes
- Saved to `research/` folder by default
---
## Skills
Model-invoked capabilities that Claude uses automatically based on context.
### knowledge-capture
Automatically captures insights, decisions, and learnings into structured notes.
**Triggers when:**
- Important decisions are made
- Technical insights are discovered
- Problems are solved
- Design trade-offs are discussed
### continue-conversation
Resumes previous work by building context from the knowledge graph.
**Triggers when:**
- Starting a new session
- User mentions previous work ("continue with...", "back to...")
- Need context about ongoing projects
### spec-driven-development
Guides implementation based on specifications stored in Basic Memory.
**Triggers when:**
- Implementing a feature defined by a spec
- Creating new specifications
- Reviewing implementation against criteria
### edit-note
Interactively edit notes using MCP tools in a conversational workflow.
**Triggers when:**
- User wants to edit, update, or modify a note
- User asks to change specific content in a note
- User wants to add observations or relations
**How it works:**
1. Fetches the note via MCP
2. Shows current content
3. Applies edits using `edit_note` operations (append, prepend, find_replace, replace_section)
4. Shows the updated result
**Best for:** Cloud users or when you want conversational editing.
### edit-note-local
Edit notes directly as local markdown files with automatic sync.
**Triggers when:**
- User has local Basic Memory installation
- User wants to make substantial file edits
- User prefers working with full file content
**How it works:**
1. Finds the note's file path via MCP
2. Uses Claude Code's Read/Edit/Write tools on the actual file
3. Basic Memory's `sync --watch` picks up changes automatically
**Best for:** Local users who want full file access and git integration.
### knowledge-organize
Help organize, link, and maintain the knowledge graph.
**Triggers when:**
- User wants to organize their notes
- User asks about orphan or unlinked notes
- User wants to find connections between notes
- User mentions duplicates or similar notes
- User asks for help with folder organization
**Capabilities:**
- **Find orphan notes** - Identify notes with no relations
- **Suggest relations** - Propose meaningful links between notes
- **Identify duplicates** - Find notes covering similar topics
- **Folder organization** - Review and suggest folder structure
- **Tag consistency** - Normalize and improve tagging
- **Create index notes** - Generate hub notes linking related topics
- **Enrich sparse notes** - Suggest observations and structure
**Best for:** Periodic knowledge base maintenance and improving discoverability.
### research
Research topics thoroughly and produce structured reports saved to Basic Memory.
**Triggers when:**
- User asks to research or investigate something
- User wants to understand a concept or technology
- User needs context before making a decision
- Phrases like "research", "look into", "explore", "investigate"
**What it produces:**
- Structured report with summary, findings, and analysis
- Recommendations when applicable
- Links to sources and related notes
- Saved to `research/` folder
**Best for:** Building knowledge base through investigation and documentation.
---
## Hooks
Automated behaviors that enhance the Basic Memory workflow.
### PostToolUse: write_note
Confirms when notes are saved to Basic Memory.
### Stop
After significant conversations, suggests using `/remember` to capture valuable insights (only when genuinely useful).
---
## MCP Tools Used
This plugin leverages Basic Memory's MCP tools:
| Tool | Purpose |
|------|---------|
| `write_note` | Create/update markdown notes |
| `read_note` | Read notes by title or permalink |
| `search_notes` | Full-text search across content |
| `build_context` | Navigate knowledge graph via memory:// URLs |
| `recent_activity` | Get recently updated information |
| `edit_note` | Incrementally update notes |
---
## Plugin Structure
```
claude-code-plugin/
├── .claude-plugin/
│ ├── plugin.json # Plugin manifest
│ └── marketplace.json # Self-hosted marketplace
├── commands/
│ ├── remember.md # /remember command
│ ├── continue.md # /continue command
│ ├── context.md # /context command
│ ├── recent.md # /recent command
│ ├── organize.md # /organize command
│ └── research.md # /research command
├── skills/
│ ├── knowledge-capture/
│ ├── continue-conversation/
│ ├── spec-driven-development/
│ ├── edit-note/
│ ├── edit-note-local/
│ ├── knowledge-organize/
│ └── research/
├── hooks/
│ └── hooks.json # Hook definitions
├── README.md # Quick start guide
└── PLUGIN.md # Full documentation
```
---
## Related
- [Basic Memory Documentation](https://docs.basicmemory.io)
- [Basic Memory GitHub](https://github.com/basicmachines-co/basic-memory)
- [Model Context Protocol](https://modelcontextprotocol.io)
- [Claude Code Plugins](https://code.claude.com/docs/en/plugins)
+100
View File
@@ -0,0 +1,100 @@
# Basic Memory Plugin for Claude Code
A Claude Code plugin that integrates [Basic Memory](https://basicmemory.io) - a local-first knowledge management system built on the Model Context Protocol (MCP).
## What This Plugin Does
This plugin helps Claude Code work seamlessly with your Basic Memory knowledge base:
- **Capture knowledge** from conversations automatically
- **Resume previous work** by building context from your knowledge graph
- **Edit notes** interactively through conversation
- **Organize your knowledge** by finding orphans, suggesting links, and maintaining structure
## Installation
### 1. Install Basic Memory
```bash
pip install basic-memory
# or
pipx install basic-memory
```
### 2. Add the Marketplace
```
/plugin marketplace add basicmachines-co/basic-memory/claude-code-plugin
```
### 3. Install the Plugin
```
/plugin install basic-memory@basicmachines
```
## Commands
| Command | Description |
|---------|-------------|
| `/remember [title]` | Capture insights from the current conversation |
| `/continue [topic]` | Resume previous work with context |
| `/context <memory://url>` | Build context from a specific note |
| `/recent [timeframe]` | Show recent activity |
| `/organize [action]` | Maintain your knowledge graph |
| `/research <topic>` | Research a topic and save a report |
### Examples
```bash
# Capture what we just discussed
/remember "Database Design Decision"
# Pick up where we left off
/continue postgres migration
# Check recent changes
/recent 1week
# Find orphan notes and suggest links
/organize orphans
# Research a topic and save findings
/research "MCP protocol"
```
## Skills
Skills are model-invoked - Claude uses them automatically when the context fits.
| Skill | What It Does |
|-------|--------------|
| `knowledge-capture` | Auto-captures decisions and insights into structured notes |
| `continue-conversation` | Builds context when resuming previous work |
| `spec-driven-development` | Guides implementation based on specs in Basic Memory |
| `edit-note` | Edits notes via MCP tools (cloud-compatible) |
| `edit-note-local` | Edits notes as files (local installations) |
| `knowledge-organize` | Helps organize and link notes |
| `research` | Researches topics and produces saved reports |
## Hooks
| Event | Behavior |
|-------|----------|
| `PostToolUse: write_note` | Confirms when notes are saved |
| `Stop` | Suggests capturing valuable insights after conversations |
## Requirements
- [Claude Code](https://claude.com/claude-code)
- [Basic Memory](https://basicmemory.io) with MCP server configured
## Documentation
- [Full Plugin Documentation](./PLUGIN.md)
- [Basic Memory Docs](https://docs.basicmemory.io)
- [Claude Code Plugins](https://code.claude.com/docs/en/plugins)
## License
MIT - See the [Basic Memory repository](https://github.com/basicmachines-co/basic-memory) for details.
+39
View File
@@ -0,0 +1,39 @@
---
description: Build context from a Basic Memory URL
argument-hint: <memory://url> [depth] [timeframe]
allowed-tools: mcp__basic-memory__build_context, mcp__basic-memory__read_note
---
# Context
Build context from a Basic Memory memory:// URL.
## Arguments
- `$1` - Memory URL (e.g., `memory://topic`, `memory://folder/*`, `memory://SPEC-24`)
- `$2` - Depth of relation traversal (optional, default: 2)
- `$3` - Timeframe for recent changes (optional, default: "7d")
## Your Task
Navigate the knowledge graph and build comprehensive context.
1. **Build context** using `mcp__basic-memory__build_context`:
- url: "$1"
- depth: $2 or 2
- timeframe: "$3" or "7d"
2. **Present the context**:
- Main note content
- Related notes found via relations
- Recent changes within timeframe
- Key observations and decisions
3. **Read additional notes** if needed for more detail.
## Memory URL Formats
- `memory://note-title` - Single note by title
- `memory://folder/*` - All notes in a folder
- `memory://SPEC-*` - Pattern matching
- `memory://specs/SPEC-24` - Note in specific project folder
+46
View File
@@ -0,0 +1,46 @@
---
description: Resume previous work from Basic Memory context
argument-hint: [topic]
allowed-tools: mcp__basic-memory__build_context, mcp__basic-memory__recent_activity, mcp__basic-memory__search_notes, mcp__basic-memory__read_note
---
# Continue
Resume previous work by building context from Basic Memory.
## Arguments
- `$ARGUMENTS` - Topic, note title, or search terms to find previous context
## Your Task
Build context to continue previous work seamlessly.
1. **Find relevant context**:
If a specific topic is provided ("$ARGUMENTS"):
- Search for matching notes: `mcp__basic-memory__search_notes`
- Build context from matches: `mcp__basic-memory__build_context`
- Read key notes for details: `mcp__basic-memory__read_note`
If no topic provided:
- Get recent activity: `mcp__basic-memory__recent_activity` with timeframe "3d"
- Present what's been happening
- Ask which topic to dive into
2. **Present context**:
- Summarize current state of the work
- Highlight recent changes or progress
- List open items or next steps
- Show related context from the knowledge graph
3. **Be ready to continue**:
- Understand what was done before
- Know what needs to happen next
- Have relevant context loaded
## Tips
- Use `memory://topic` URL format with `build_context`
- Check multiple projects if needed (main, specs)
- Follow relations to find connected knowledge
+87
View File
@@ -0,0 +1,87 @@
---
description: Organize and maintain your Basic Memory knowledge graph
argument-hint: [health|orphans|duplicates|relations|tags] [project]
allowed-tools: mcp__basic-memory__search_notes, mcp__basic-memory__read_note, mcp__basic-memory__list_directory, mcp__basic-memory__edit_note, mcp__basic-memory__write_note
---
# Organize
Help organize, link, and maintain your Basic Memory knowledge graph.
## Arguments
- `$1` - Action (optional): `health`, `orphans`, `duplicates`, `relations`, `tags` (default: `health`)
- `$2` - Project (optional): defaults to "main"
## Actions
### `/organize` or `/organize health`
Run a quick health check:
1. Count total notes
2. Identify orphan notes (no relations)
3. Check for potential duplicates
4. Show folder distribution
5. Report any issues found
### `/organize orphans`
Find and address orphan notes:
1. Search for notes with empty Relations sections
2. List orphans found
3. For each orphan, suggest potential relations based on content
4. Offer to add relations or create index notes
### `/organize duplicates`
Find potentially duplicate notes:
1. Search for notes with similar titles
2. Compare content for overlap
3. Suggest: merge, differentiate, or link with `supersedes`
### `/organize relations [note-title]`
Suggest relations for a specific note (or recent notes if not specified):
1. Read the target note
2. Search for related content
3. Suggest relation types:
- `relates-to` - General connection
- `extends` - Builds upon
- `implements` - Realizes concept
- `depends-on` - Requires understanding of
4. Offer to add selected relations
### `/organize tags`
Review tag consistency:
1. Gather all tags across notes
2. Find similar/duplicate tags (e.g., `arch` vs `architecture`)
3. Identify over-used or under-used tags
4. Suggest normalization
## Your Task
Execute: `/organize $ARGUMENTS`
Based on the action requested:
1. **Gather data** using search and list tools
2. **Analyze** for the specific issue (orphans, duplicates, etc.)
3. **Present findings** clearly with counts and examples
4. **Offer solutions** - ask before making changes
5. **Apply fixes** using edit_note or write_note when user approves
Always confirm before modifying notes. Show what will change and get approval.
## Examples
```
/organize # Quick health check
/organize health # Same as above
/organize orphans # Find unlinked notes
/organize duplicates # Find similar notes
/organize relations # Suggest links for recent notes
/organize relations "My Note" # Suggest links for specific note
/organize tags # Review tag consistency
/organize health specs # Health check on specs project
```
+40
View File
@@ -0,0 +1,40 @@
---
description: Show recent activity in Basic Memory
argument-hint: [timeframe] [project]
allowed-tools: mcp__basic-memory__recent_activity, mcp__basic-memory__read_note
---
# Recent
Show recent activity in Basic Memory.
## Arguments
- `$1` - Timeframe (optional): "today", "1d", "3d", "1 week", "2 weeks" (default: "3d")
- `$2` - Project (optional): "main", "specs", etc.
## Your Task
Show what's been happening in Basic Memory recently.
1. **Get recent activity** using `mcp__basic-memory__recent_activity`:
- timeframe: "$1" or "3d"
- project: "$2" or check all projects
2. **Present activity**:
- List recently modified notes
- Group by type or folder if helpful
- Highlight key changes
- Show dates of modifications
3. **Offer to dive deeper**:
- Ask if user wants to read any specific notes
- Suggest continuing work on active items
## Timeframe Examples
- `today` - Just today
- `1d` or `yesterday` - Last 24 hours
- `3d` - Last 3 days
- `1 week` - Last week
- `2 weeks` - Last 2 weeks
+43
View File
@@ -0,0 +1,43 @@
---
description: Capture insights, decisions, or learnings to Basic Memory
argument-hint: [title] [optional: folder]
allowed-tools: mcp__basic-memory__write_note, mcp__basic-memory__search_notes
---
# Remember
Capture what we just discussed into a Basic Memory note.
## Arguments
- `$1` - Title for the note (required)
- `$2` - Folder to save in (optional, defaults to "notes")
## Your Task
Create a structured note capturing the key insights from our conversation.
1. **Analyze the conversation** for:
- Decisions made
- Insights discovered
- Problems solved
- Patterns identified
- Trade-offs discussed
2. **Structure the note** with:
- Clear title: "$1" (or generate one if not provided)
- Context section explaining the situation
- Main content with key points
- Observations using `[category]` format:
- `[decision]` - Choices made
- `[insight]` - Understanding gained
- `[pattern]` - Reusable approaches
- `[learning]` - Lessons learned
- Relations to link related concepts with `[[WikiLinks]]`
3. **Save using** `mcp__basic-memory__write_note`:
- folder: "$2" or "notes"
- Include relevant tags
- Project: use "main" unless user specifies otherwise
4. **Confirm** what was captured and where it was saved.
+140
View File
@@ -0,0 +1,140 @@
---
description: Research a topic and save a structured report to Basic Memory
argument-hint: <topic> [folder]
allowed-tools: mcp__basic-memory__write_note, mcp__basic-memory__search_notes, mcp__basic-memory__read_note, mcp__basic-memory__build_context, WebSearch, WebFetch, Grep, Glob, Read
---
# Research
Research a topic thoroughly and produce a structured report saved to Basic Memory.
## Arguments
- `$1` - Topic to research (required)
- `$2` - Folder to save report (optional, default: "research")
## Your Task
Conduct thorough research on: **$ARGUMENTS**
### 1. Check Existing Knowledge
First, see what we already know:
```python
mcp__basic-memory__search_notes(query="$1", project="main")
```
Read any relevant existing notes to avoid duplicating research.
### 2. Gather Information
Depending on the topic, use appropriate tools:
**For codebase topics:**
- Search code with Grep/Glob
- Read relevant files
- Check tests for examples
**For external topics:**
- Use WebSearch for current information
- Fetch documentation with WebFetch
- Look for official sources
**For Basic Memory context:**
- Build context from related notes
- Check for prior decisions or research
### 3. Analyze Findings
Synthesize what you learned:
- Identify key concepts
- Note patterns and trade-offs
- Form recommendations if applicable
- Flag uncertainties
### 4. Produce Report
Create a structured report with this format:
```markdown
---
title: "Research: [Topic]"
type: research
tags:
- research
- [relevant-tags]
---
# Research: [Topic]
## Summary
[2-3 sentence executive summary]
## Research Question
[What we investigated and why]
## Key Findings
### [Finding 1]
[Details and evidence]
### [Finding 2]
[Details and evidence]
### [Finding 3]
[Details and evidence]
## Analysis
[Synthesis, patterns, trade-offs, recommendations]
## Open Questions
- [Areas needing more investigation]
## Sources
- [Links to sources]
- [[Related Notes]] from Basic Memory
## Observations
- [finding] Key insight #research
- [recommendation] Suggested approach based on research
## Relations
- researches [[Topic]]
- relates-to [[Related Concepts]]
```
### 5. Save Report
```python
mcp__basic-memory__write_note(
title="Research: $1",
content="[report content]",
folder="$2" or "research",
tags=["research", ...],
project="main"
)
```
### 6. Present Summary
After saving, present:
- Key findings summary
- Main recommendation (if applicable)
- Where the report was saved
- Offer to dive deeper into any aspect
## Examples
```
/research MCP protocol
/research "database migration patterns"
/research "authentication options" decisions
/research "React vs Vue" architecture
```
+26
View File
@@ -0,0 +1,26 @@
{
"hooks": {
"PostToolUse": [
{
"matcher": "mcp__basic-memory__write_note",
"hooks": [
{
"type": "command",
"command": "echo '✓ Note saved to Basic Memory'"
}
]
}
],
"Stop": [
{
"matcher": "*",
"hooks": [
{
"type": "prompt",
"prompt": "If this conversation contained valuable insights, decisions, or learnings that should be preserved, suggest using `/remember [title]` to capture them in Basic Memory. Only suggest this if there's genuinely valuable content worth preserving - don't suggest for trivial interactions."
}
]
}
]
}
}
@@ -0,0 +1,211 @@
---
name: continue-conversation
description: Resume previous work by building context from Basic Memory knowledge graph using memory URLs and recent activity
---
# Continue Conversation
This skill helps you resume previous work by building context from the Basic Memory knowledge graph, enabling seamless continuation across sessions.
## When to Use
Use this skill when:
- Starting a new session and need to pick up where you left off
- User mentions previous work ("continue with...", "back to...", "where were we with...")
- Need context about ongoing projects or specs
- User asks about something discussed in a previous conversation
- Working on a multi-session task
## Building Context
### 1. Identify What to Continue
Ask if unclear:
- What topic or project to resume?
- What timeframe to look at?
- Any specific aspect to focus on?
### 2. Gather Context with MCP Tools
**Option A: Known Topic - Use build_context**
```python
# Navigate knowledge graph from a known starting point
mcp__basic-memory__build_context(
url="memory://topic-or-note-name",
depth=2, # How many relation hops to follow
timeframe="7d", # Recent changes
project="main" # or "specs" for specifications
)
```
Memory URL formats:
- `memory://note-title` - Single note
- `memory://folder/*` - All notes in folder
- `memory://specs/SPEC-24*` - Pattern matching
**Option B: Recent Activity - What's been happening?**
```python
# See what's changed recently
mcp__basic-memory__recent_activity(
timeframe="3d", # "1d", "1 week", "2 weeks"
depth=1,
project="main"
)
```
**Option C: Search for Context**
```python
# Find relevant notes
mcp__basic-memory__search_notes(
query="search terms",
page_size=10,
project="main"
)
```
### 3. Read Key Notes
Once you identify relevant notes:
```python
mcp__basic-memory__read_note(
identifier="note-title-or-permalink",
project="main"
)
```
### 4. Present Context to User
Summarize what you found:
- Current state of the work
- Recent changes or progress
- Open items or next steps
- Related context that might be helpful
## Context Strategies by Scenario
### Resuming a Spec Implementation
```python
# 1. Read the spec
mcp__basic-memory__read_note(
identifier="SPEC-24: Postgres Database Migration",
project="specs"
)
# 2. Check recent activity on related topics
mcp__basic-memory__build_context(
url="memory://SPEC-24*",
timeframe="7d",
project="specs"
)
# 3. Look at what's been done in the codebase
# (Use regular file tools for this)
```
### Continuing General Work
```python
# 1. Check recent activity across projects
mcp__basic-memory__recent_activity(
timeframe="3d",
project="main"
)
# 2. Read any notes from recent sessions
mcp__basic-memory__read_note(
identifier="relevant-note",
project="main"
)
```
### Following Up on a Topic
```python
# 1. Search for the topic
mcp__basic-memory__search_notes(
query="topic keywords",
project="main"
)
# 2. Build context from best match
mcp__basic-memory__build_context(
url="memory://found-note-permalink",
depth=2,
project="main"
)
```
## Timeframe Reference
Natural language timeframes:
- `"today"` - Current day
- `"yesterday"` - Previous day
- `"3d"` or `"3 days"` - Last 3 days
- `"1 week"` or `"7d"` - Last week
- `"2 weeks"` - Last 2 weeks
- `"1 month"` - Last month
## Project Reference
Common projects:
- `main` - Primary knowledge base
- `specs` - Specifications and design docs
- `basic-memory-llc` - Business/company notes
- `getting-started` - Tutorial content
List available projects:
```python
mcp__basic-memory__list_memory_projects()
```
## Example Conversations
### User: "Let's continue with the Postgres migration"
```
1. Read SPEC-24 from specs project
2. Check for related notes about implementation progress
3. Summarize:
- Spec overview and goals
- What's been completed (checkmarks)
- What's pending (checkboxes)
- Any blockers or decisions needed
```
### User: "What was I working on yesterday?"
```
1. Get recent activity for last 2 days
2. List modified notes with brief descriptions
3. Ask which topic to dive into
```
### User: "Back to the async client pattern"
```
1. Search for "async client pattern"
2. Build context from matching note
3. Include related notes via relations
4. Present the full picture
```
## Best Practices
1. **Start broad, then narrow** - Get overview first, then specific details
2. **Follow relations** - Knowledge graph connections are valuable
3. **Check multiple projects** - Specs might be separate from implementation notes
4. **Present incrementally** - Share what you find as you go
5. **Confirm understanding** - Verify the context is what user needs
6. **Update as you go** - Capture new progress in notes during the session
## Combining with Other Skills
After building context, you might:
- Use **knowledge-capture** to document new progress
- Use **spec-driven-development** if continuing a spec implementation
- Create new notes linking to the context you gathered
@@ -0,0 +1,261 @@
---
name: edit-note-local
description: Edit Basic Memory notes directly as local files - enables full file editing with automatic sync (local installations only)
---
# Edit Note Local
This skill enables direct file-based editing of Basic Memory notes. It works by editing the actual markdown files in the knowledge base, which Basic Memory's sync service automatically picks up. This provides a more seamless editing experience for local installations.
## When to Use
Use this skill when:
- User has a local Basic Memory installation (not cloud-only)
- User wants to make substantial edits to a note
- User prefers working with the full file content
- User wants changes to sync automatically via `basic-memory sync --watch`
**Note:** This skill requires local file access. For cloud-only users, use the `edit-note` skill instead.
## Editing Workflow
### 1. Find the Note's File Path
First, get the note metadata to find its file location:
```python
# Search for the note
mcp__basic-memory__search_notes(
query="note title or keywords",
project="main"
)
# Read the note to get file_path from metadata
mcp__basic-memory__read_note(
identifier="note-title",
project="main"
)
```
The response includes `file_path` which gives the relative path within the knowledge base.
### 2. Determine Full File Path
Basic Memory projects have a root directory. Common locations:
- Default: `~/basic-memory/`
- Custom: Check project configuration
Construct the full path:
```
{project_root}/{file_path}
```
For example:
- Project root: `/Users/username/basic-memory`
- File path from note: `notes/My Note.md`
- Full path: `/Users/username/basic-memory/notes/My Note.md`
### 3. Read the File
Use Claude Code's Read tool to get the full file content:
```python
Read(file_path="/Users/username/basic-memory/notes/My Note.md")
```
Display the content to the user, explaining the structure:
- Frontmatter (YAML between `---` markers)
- Main content
- Observations section
- Relations section
### 4. Edit the File
Use Claude Code's Edit tool for precise changes:
```python
Edit(
file_path="/Users/username/basic-memory/notes/My Note.md",
old_string="text to replace",
new_string="new text"
)
```
Or use Write for complete rewrites:
```python
Write(
file_path="/Users/username/basic-memory/notes/My Note.md",
content="Complete new file content..."
)
```
### 5. Sync Happens Automatically
If the user has `basic-memory sync --watch` running, changes are picked up automatically. Otherwise, they can run:
```bash
basic-memory sync
```
## File Structure Reference
Basic Memory notes follow this markdown structure:
```markdown
---
title: Note Title
type: note
permalink: note-title
tags:
- tag1
- tag2
---
# Note Title
## Context
Background and situation explanation.
## Main Content
The primary content of the note...
## Observations
- [category] Observation text #optional-tag
- [decision] A decision that was made #tag
- [insight] An insight discovered
## Relations
- relates-to [[Other Note]]
- implements [[Parent Concept]]
- learned-from [[Source Note]]
```
## Editing Patterns
### Edit Frontmatter Tags
```python
Edit(
file_path="/path/to/note.md",
old_string="tags:\n- old-tag",
new_string="tags:\n- old-tag\n- new-tag"
)
```
### Add New Section
```python
Edit(
file_path="/path/to/note.md",
old_string="## Observations",
new_string="## New Section\n\nNew content here.\n\n## Observations"
)
```
### Modify Observation
```python
Edit(
file_path="/path/to/note.md",
old_string="- [decision] Old decision",
new_string="- [decision] Updated decision with new info #updated"
)
```
### Add Relation
```python
Edit(
file_path="/path/to/note.md",
old_string="## Relations\n",
new_string="## Relations\n\n- relates-to [[New Related Note]]\n"
)
```
### Complete Rewrite
For major changes, read the file, construct new content preserving the frontmatter structure, and write:
```python
Write(
file_path="/path/to/note.md",
content="""---
title: Note Title
type: note
permalink: note-title
tags:
- updated
---
# Note Title
Completely rewritten content...
## Observations
- [rewrite] Complete rewrite of this note #major-update
## Relations
- updates [[Previous Version]]
"""
)
```
## Finding the Project Root
To find where Basic Memory stores files, you can:
1. **Check common locations:**
- `~/basic-memory/`
- `~/Documents/basic-memory/`
- Current working directory
2. **Use the list_directory tool:**
```python
mcp__basic-memory__list_directory(
dir_name="/",
project="main"
)
```
3. **Ask the user:** "Where is your Basic Memory knowledge base located?"
## Advantages of Local Editing
1. **Full file access** - Edit any part of the file including frontmatter
2. **Multi-line edits** - Make complex structural changes easily
3. **Batch operations** - Edit multiple files in sequence
4. **Version control** - Changes tracked by git if the folder is a repo
5. **Instant preview** - Use any markdown editor alongside
6. **Auto-sync** - `sync --watch` picks up changes automatically
## Best Practices
1. **Preserve frontmatter** - Don't break the YAML structure
2. **Keep valid markdown** - Maintain proper formatting
3. **Preserve permalinks** - Changing them can break links
4. **Show diffs** - Tell the user what changed
5. **Suggest sync** - Remind about `basic-memory sync` if not watching
6. **Handle missing files** - Check if file exists before editing
## Example Conversation
**User:** "I want to completely rewrite my architecture decision note"
**Claude:**
1. Searches for the note via MCP
2. Gets the file path
3. Reads the current file content
4. Asks: "Here's the current note. What would you like the new version to say?"
**User:** Provides new content
**Claude:**
1. Preserves the frontmatter (title, permalink, type)
2. Writes the new content using Write tool
3. Confirms: "Updated the file at `/path/to/note.md`. If you have sync --watch running, it's already indexed. Otherwise run `basic-memory sync`."
@@ -0,0 +1,209 @@
---
name: edit-note
description: Interactively edit Basic Memory notes using MCP tools - view, modify, and update notes in a conversational workflow (works with cloud and local)
---
# Edit Note
This skill enables interactive editing of Basic Memory notes using MCP tools. It works with both Basic Memory Cloud and local installations since it operates through the MCP interface rather than direct file access.
## When to Use
Use this skill when:
- User wants to edit an existing note
- User asks to update, change, or modify note content
- User wants to refine observations or relations in a note
- User says things like "edit my note about...", "update the...", "change X to Y in..."
## Editing Workflow
### 1. Fetch the Current Note
First, retrieve the note to show the user what exists:
```python
mcp__basic-memory__read_note(
identifier="Note Title or permalink",
project="main" # or specified project
)
```
Present the note content clearly, highlighting:
- Current title and metadata
- Main content sections
- Observations (with categories)
- Relations (with link targets)
### 2. Understand the Edit Request
Ask clarifying questions if needed:
- Which section to modify?
- What specifically to change?
- Add new content or replace existing?
### 3. Apply the Edit
Use the appropriate `edit_note` operation:
**Append** - Add content to the end:
```python
mcp__basic-memory__edit_note(
identifier="note-title",
operation="append",
content="\n\n## New Section\n\nNew content here...",
project="main"
)
```
**Prepend** - Add content to the beginning:
```python
mcp__basic-memory__edit_note(
identifier="note-title",
operation="prepend",
content="# Updated Header\n\n",
project="main"
)
```
**Find and Replace** - Replace specific text:
```python
mcp__basic-memory__edit_note(
identifier="note-title",
operation="find_replace",
find_text="old text to find",
content="new replacement text",
project="main"
)
```
**Replace Section** - Replace an entire section by heading:
```python
mcp__basic-memory__edit_note(
identifier="note-title",
operation="replace_section",
section="## Section Heading",
content="## Section Heading\n\nCompletely new section content...",
project="main"
)
```
### 4. Show the Result
After editing, fetch and display the updated note:
```python
mcp__basic-memory__read_note(
identifier="note-title",
project="main"
)
```
Highlight what changed so the user can verify.
## Edit Operations Reference
| Operation | Use Case | Required Parameters |
|-----------|----------|---------------------|
| `append` | Add to end | `content` |
| `prepend` | Add to beginning | `content` |
| `find_replace` | Change specific text | `find_text`, `content` |
| `replace_section` | Rewrite a section | `section`, `content` |
## Common Edit Patterns
### Adding a New Observation
```python
mcp__basic-memory__edit_note(
identifier="note-title",
operation="find_replace",
find_text="## Observations",
content="## Observations\n\n- [new-category] New observation here #tag",
project="main"
)
```
Or append to observations section:
```python
mcp__basic-memory__edit_note(
identifier="note-title",
operation="append",
content="\n- [insight] Additional insight discovered #tag",
project="main"
)
```
### Adding a New Relation
```python
mcp__basic-memory__edit_note(
identifier="note-title",
operation="find_replace",
find_text="## Relations",
content="## Relations\n\n- relates-to [[New Related Note]]",
project="main"
)
```
### Updating a Specific Observation
```python
mcp__basic-memory__edit_note(
identifier="note-title",
operation="find_replace",
find_text="- [decision] Old decision text",
content="- [decision] Updated decision with new context #updated",
project="main"
)
```
### Rewriting the Context Section
```python
mcp__basic-memory__edit_note(
identifier="note-title",
operation="replace_section",
section="## Context",
content="## Context\n\nCompletely rewritten context explaining the new situation...",
project="main"
)
```
## Multi-Step Editing Session
For complex edits, work iteratively:
1. **Show current state** → Read and display the note
2. **First edit** → Apply one change
3. **Show result** → Display updated note
4. **Next edit** → Apply another change if needed
5. **Confirm complete** → Final display and confirmation
This keeps the user informed and allows course correction.
## Best Practices
1. **Always show before and after** - User should see what changed
2. **One edit at a time** - For complex changes, do multiple operations
3. **Preserve structure** - Maintain the note's markdown format
4. **Be careful with find_replace** - Ensure the find_text is unique
5. **Confirm destructive changes** - Ask before replacing large sections
6. **Keep observations formatted** - Maintain `[category]` prefix format
7. **Keep relations formatted** - Maintain `- relation-type [[Target]]` format
## Example Conversation
**User:** "Edit my note about the async client pattern - add an observation about testing"
**Claude:**
1. Fetches "Async Client Pattern" note
2. Displays current content
3. Asks: "What observation about testing would you like to add?"
**User:** "That the context manager pattern makes mocking easier in tests"
**Claude:**
1. Uses `edit_note` with `append` to add:
`- [testing] Context manager pattern simplifies mocking in unit tests #testability`
2. Fetches and displays updated note
3. Confirms: "Added the testing observation. Here's the updated note..."
@@ -0,0 +1,211 @@
---
name: knowledge-capture
description: Capture insights, decisions, and learnings from conversations into structured Basic Memory notes with observations and relations
---
# Knowledge Capture
This skill helps you capture valuable information from conversations into Basic Memory's knowledge graph using structured notes with observations and relations.
## When to Use
Use this skill when:
- Important decisions are made during a conversation
- Technical insights or patterns are discovered
- Problems are solved and the solution should be preserved
- Design trade-offs are discussed
- Architecture or implementation approaches are chosen
- Learnings from debugging or investigation emerge
## Capture Process
### 1. Identify Valuable Information
Look for:
- **Decisions**: Choices made and their rationale
- **Insights**: New understanding or "aha" moments
- **Patterns**: Reusable approaches or solutions
- **Trade-offs**: Options considered and why one was chosen
- **Learnings**: What worked, what didn't, and why
- **Context**: Background that would help future understanding
### 2. Structure the Note
Use Basic Memory's knowledge format:
```markdown
---
title: Descriptive Title
type: note
tags:
- relevant
- tags
---
# Title
## Context
Brief background explaining the situation.
## Content
Main content organized logically.
## Observations
- [decision] What was decided and why #tag
- [insight] Key understanding gained #tag
- [pattern] Reusable approach identified #tag
- [learning] What we learned from this #tag
- [tradeoff] Option A chosen over B because... #tag
## Relations
- relates-to [[Related Concept]]
- implements [[Parent Spec or Design]]
- learned-from [[Source of Learning]]
```
### 3. Choose Appropriate Categories
Common observation categories:
- `[decision]` - Choices made
- `[insight]` - Understanding gained
- `[pattern]` - Reusable approaches
- `[learning]` - Lessons learned
- `[tradeoff]` - Options weighed
- `[problem]` - Issues identified
- `[solution]` - Fixes applied
- `[architecture]` - Structural decisions
- `[implementation]` - Code-level choices
- `[constraint]` - Limitations discovered
- `[requirement]` - Needs identified
### 4. Create Meaningful Relations
Link to related knowledge:
- `relates-to` - General association
- `implements` - Realizes a spec or design
- `extends` - Builds upon existing concept
- `learned-from` - Source of insight
- `enables` - Makes something possible
- `depends-on` - Requires another concept
- `solves` - Addresses a problem
## MCP Tools to Use
```python
# Write a new note
mcp__basic-memory__write_note(
title="Your Note Title",
content="Full markdown content...",
folder="appropriate/folder",
tags=["tag1", "tag2"],
project="main" # or appropriate project
)
# Search for related notes to link
mcp__basic-memory__search_notes(
query="relevant terms",
project="main"
)
# Read existing notes for context
mcp__basic-memory__read_note(
identifier="note-title-or-permalink",
project="main"
)
```
## Folder Organization
Choose appropriate folders:
- `decisions/` - Architecture and design decisions
- `learnings/` - Insights and lessons learned
- `patterns/` - Reusable approaches
- `debug-logs/` - Problem investigations
- `conversations/` - Imported conversation summaries
## Examples
### Capturing a Technical Decision
```markdown
---
title: FastAPI Async Client Pattern
type: note
tags:
- architecture
- fastapi
- async
---
# FastAPI Async Client Pattern
## Context
During implementation of MCP tools, we needed to decide how to handle HTTP client lifecycle.
## Decision
Use context manager pattern for HTTP clients instead of module-level singletons.
## Rationale
- Proper resource management
- Supports three deployment modes (local ASGI, CLI cloud, cloud app)
- Auth happens at client creation, not per-request
- Enables dependency injection for testing
## Observations
- [decision] Context manager pattern for HTTP clients enables proper resource cleanup #architecture
- [pattern] Factory pattern allows different client configurations per deployment mode #flexibility
- [tradeoff] Slightly more verbose than singleton but much more flexible #engineering
## Relations
- implements [[SPEC-16 MCP Cloud Service Consolidation]]
- enables [[Cloud App Integration]]
```
### Capturing a Debugging Insight
```markdown
---
title: SQLite WAL Mode Performance Fix
type: note
tags:
- debugging
- sqlite
- performance
---
# SQLite WAL Mode Performance Fix
## Problem
Sync operations were slow with multiple concurrent writes.
## Investigation
Found that default SQLite journaling was causing lock contention.
## Solution
Enabled WAL (Write-Ahead Logging) mode for the database connection.
## Observations
- [problem] Default SQLite journaling causes lock contention under concurrent writes #performance
- [solution] WAL mode significantly improves concurrent write performance #sqlite
- [learning] Always consider WAL mode for SQLite in applications with concurrent access #database
## Relations
- solves [[Sync Performance Issues]]
- relates-to [[SPEC-19 Sync Performance]]
```
## Best Practices
1. **Capture immediately** - Write notes while context is fresh
2. **Be specific** - Include concrete details, not vague summaries
3. **Link liberally** - More relations = better knowledge graph
4. **Use tags** - Enable discovery via search
5. **Include context** - Future you won't remember the situation
6. **Prefer facts over opinions** - Observations should be verifiable
7. **Keep notes atomic** - One concept per note when possible
@@ -0,0 +1,283 @@
---
name: knowledge-organize
description: Help organize, link, and maintain the Basic Memory knowledge graph - find orphan notes, suggest relations, identify duplicates, and improve overall knowledge structure
---
# Knowledge Organize
This skill helps users maintain a healthy, well-connected knowledge graph. As notes accumulate, it becomes valuable to periodically organize, link, and curate the knowledge base.
## When to Use
Use this skill when:
- User asks to organize their notes
- User wants to find connections between notes
- User mentions orphan or unlinked notes
- User wants to clean up or improve their knowledge base
- User asks about duplicate or similar notes
- User wants help with folder organization
- User asks to review or audit their notes
- Phrases like "help me organize", "find related notes", "what's not linked", "clean up my notes"
## Organization Capabilities
### 1. Find Orphan Notes
Identify notes that have no relations to other notes - they're isolated in the knowledge graph.
```python
# Get all notes
mcp__basic-memory__search_notes(
query="*",
page_size=50,
project="main"
)
# For each note, check if it has relations
# Orphans have empty Relations sections
```
**What to do with orphans:**
- Suggest potential relations based on content similarity
- Ask if they should be linked to existing topics
- Propose creating hub notes to connect related orphans
### 2. Suggest Relations
Analyze note content and suggest meaningful connections.
```python
# Read a note
mcp__basic-memory__read_note(
identifier="note-to-analyze",
project="main"
)
# Search for potentially related notes
mcp__basic-memory__search_notes(
query="key terms from the note",
project="main"
)
# Suggest relations based on:
# - Shared topics or concepts
# - Complementary content (problem/solution, question/answer)
# - Sequential relationship (part 1, part 2)
# - Hierarchical (parent concept, child detail)
```
**Relation types to suggest:**
- `relates-to` - General topical connection
- `extends` - Builds upon or expands
- `implements` - Realizes a concept
- `depends-on` - Requires understanding of
- `contradicts` - Presents alternative view
- `learned-from` - Source of insight
- `enables` - Makes something possible
### 3. Identify Similar/Duplicate Notes
Find notes that may cover the same topic.
```python
# Search for notes with similar titles or content
mcp__basic-memory__search_notes(
query="topic keywords",
project="main"
)
# Compare results for overlap
# Look for:
# - Similar titles
# - Overlapping observations
# - Same tags
# - Related timestamps (created around same time)
```
**Actions for duplicates:**
- Merge into a single comprehensive note
- Link them with `supersedes` or `updates` relations
- Differentiate by adding context about their distinct focus
### 4. Folder Organization Review
Analyze folder structure and suggest improvements.
```python
# List directory structure
mcp__basic-memory__list_directory(
dir_name="/",
depth=3,
project="main"
)
# Identify:
# - Overcrowded folders
# - Single-note folders
# - Inconsistent naming
# - Notes that might belong elsewhere
```
**Organization suggestions:**
- Group related notes into topic folders
- Create subfolders for large categories
- Suggest consistent naming conventions
- Move misplaced notes
### 5. Tag Consistency
Review and normalize tags across notes.
```python
# Search notes to analyze tag patterns
mcp__basic-memory__search_notes(
query="*",
page_size=100,
project="main"
)
# Look for:
# - Similar tags (architecture vs arch)
# - Unused tags
# - Over-used generic tags
# - Missing tags on relevant notes
```
**Tag improvements:**
- Suggest tag standardization (pick one variant)
- Propose new tags for common themes
- Identify notes missing obvious tags
### 6. Create Index/Hub Notes
Generate notes that serve as navigation hubs for related topics.
```python
# After identifying a cluster of related notes
mcp__basic-memory__write_note(
title="Architecture Decisions Index",
content="""---
title: Architecture Decisions Index
type: index
tags:
- architecture
- index
---
# Architecture Decisions Index
A hub linking all architecture-related decisions and patterns.
## Decisions
- [[Database Selection Decision]]
- [[API Design Patterns]]
- [[Authentication Architecture]]
## Patterns
- [[Repository Pattern]]
- [[Async Client Pattern]]
## Observations
- [index] Central hub for architecture knowledge #navigation
## Relations
- indexes [[Architecture]]
""",
folder="indexes",
project="main"
)
```
### 7. Enrich Sparse Notes
Find notes lacking observations or structure and suggest improvements.
```python
# Read a sparse note
mcp__basic-memory__read_note(
identifier="sparse-note",
project="main"
)
# If missing:
# - Observations section → suggest categories
# - Relations section → suggest links
# - Tags → suggest relevant tags
# - Context → suggest adding background
```
## Organization Workflows
### Quick Health Check
A fast overview of knowledge base status:
1. Count total notes
2. Identify orphan count
3. List recently modified
4. Check for obvious duplicates
5. Report folder distribution
### Deep Organization Session
Thorough review and improvement:
1. **Audit phase** - Catalog all notes, identify issues
2. **Orphan phase** - Address unlinked notes
3. **Relation phase** - Suggest new connections
4. **Duplicate phase** - Merge or differentiate similar notes
5. **Structure phase** - Reorganize folders if needed
6. **Index phase** - Create hub notes for major topics
### Topic-Focused Organization
Organize around a specific subject:
1. Find all notes related to topic
2. Map existing relations
3. Identify gaps in the topic graph
4. Suggest new notes to fill gaps
5. Create topic index note
## Best Practices
1. **Work incrementally** - Don't reorganize everything at once
2. **Confirm before changing** - Always ask before moving/editing notes
3. **Preserve permalinks** - Moving is okay, changing permalinks breaks links
4. **Show the graph** - Help user visualize connections
5. **Explain suggestions** - Say why a relation makes sense
6. **Respect user's system** - Enhance their organization, don't impose a new one
## Example Conversations
**User:** "Help me organize my notes"
**Claude:**
1. Runs health check on the knowledge base
2. Reports: "You have 47 notes. I found 12 orphan notes and 3 potential duplicates."
3. Asks: "Would you like to start by connecting the orphan notes, or review the duplicates first?"
**User:** "Find notes that should be linked to my API design note"
**Claude:**
1. Reads the API design note
2. Searches for related content
3. Suggests: "I found 5 notes that could relate:
- 'REST Best Practices' → relates-to
- 'Authentication Flow' → implements
- 'Rate Limiting Decision' → extends
Would you like me to add any of these relations?"
**User:** "Are there any notes about similar topics?"
**Claude:**
1. Analyzes note titles and content
2. Identifies clusters of similar notes
3. Reports: "I found these potential overlaps:
- 'Auth Flow' and 'Authentication Design' cover similar ground
- 'DB Schema v1' and 'DB Schema v2' might need a 'supersedes' relation
Would you like to review any of these?"
+213
View File
@@ -0,0 +1,213 @@
---
name: research
description: Research a topic thoroughly and produce a structured report saved to Basic Memory - investigate concepts, gather context, and document findings
---
# Research
This skill helps conduct thorough research on a topic and produces a structured report that gets saved to Basic Memory for future reference.
## When to Use
Use this skill when:
- User asks to research or investigate something
- User wants to understand a concept, technology, or approach
- User needs context gathered before making a decision
- User asks "what is...", "how does... work", "explore...", "investigate..."
- User wants findings documented for later
- Phrases like "research this", "look into", "find out about", "explore options for"
## Research Process
### 1. Understand the Research Question
Clarify what specifically to investigate:
- What is the core question or topic?
- What scope - broad overview or deep dive?
- Any specific aspects to focus on?
- What will the research inform (a decision, implementation, understanding)?
### 2. Gather Information
Use available tools to collect information:
**For codebase research:**
- Search the codebase for relevant code
- Read documentation and comments
- Trace how things connect
- Look at tests for usage examples
**For concept research:**
- Use web search for current information
- Fetch documentation from official sources
- Look for examples and best practices
- Compare alternatives if relevant
**For Basic Memory context:**
```python
# Check what we already know
mcp__basic-memory__search_notes(
query="topic keywords",
project="main"
)
# Build context from related notes
mcp__basic-memory__build_context(
url="memory://related-topic",
depth=2,
project="main"
)
```
### 3. Analyze and Synthesize
Organize findings into coherent insights:
- Identify key concepts and how they relate
- Note patterns, trade-offs, and considerations
- Highlight what's most relevant to the user's needs
- Flag uncertainties or areas needing more investigation
### 4. Produce the Report
Create a structured research report:
```markdown
---
title: "Research: [Topic]"
type: research
tags:
- research
- [topic-tags]
---
# Research: [Topic]
## Summary
[2-3 sentence executive summary of findings]
## Research Question
[What we set out to understand]
## Key Findings
### [Finding 1]
[Details, evidence, implications]
### [Finding 2]
[Details, evidence, implications]
### [Finding 3]
[Details, evidence, implications]
## Analysis
[Synthesis of findings - patterns, trade-offs, recommendations]
## Open Questions
- [Things that need more investigation]
- [Uncertainties or assumptions]
## Sources
- [Where information came from]
- [[Related Note]] - relevant prior knowledge
## Observations
- [finding] Key insight discovered #research
- [pattern] Pattern identified during research
- [recommendation] Suggested approach based on findings
## Relations
- researches [[Topic]]
- informs [[Decision or Implementation]]
- relates-to [[Related Concepts]]
```
### 5. Save to Basic Memory
```python
mcp__basic-memory__write_note(
title="Research: [Topic]",
content="[Full report content]",
folder="research",
tags=["research", "topic-tags"],
project="main"
)
```
## Report Styles
Adjust based on the research type:
### Quick Investigation
- Focused summary
- 2-3 key findings
- Direct recommendation
- Saved to `research/` folder
### Deep Dive
- Comprehensive analysis
- Multiple sections
- Detailed evidence
- Comparison of options
- Saved to `research/` folder
### Decision Support
- Options evaluated
- Pros/cons for each
- Clear recommendation with rationale
- Saved to `decisions/` or `research/` folder
### Technical Exploration
- How it works
- Architecture/design
- Code examples
- Integration considerations
- Saved to `research/` folder
## Best Practices
1. **Start with what we know** - Check Basic Memory for existing context
2. **Be thorough but focused** - Cover the topic well without tangents
3. **Cite sources** - Link to where information came from
4. **Be honest about uncertainty** - Flag what's unclear or needs verification
5. **Make it actionable** - Include recommendations when appropriate
6. **Link to related knowledge** - Connect to existing notes
7. **Save for future reference** - Always save the report to Basic Memory
## Example Conversations
**User:** "Research how other projects handle database migrations"
**Claude:**
1. Searches codebase for migration patterns
2. Checks Basic Memory for related decisions
3. Looks up best practices online
4. Produces report comparing approaches
5. Saves to `research/Database Migration Approaches.md`
6. Presents summary with recommendation
**User:** "Investigate the MCP protocol"
**Claude:**
1. Fetches MCP documentation
2. Searches for examples in codebase
3. Checks Basic Memory for prior context
4. Produces comprehensive report on MCP
5. Saves to `research/MCP Protocol Overview.md`
6. Presents key concepts and how to use them
**User:** "Look into authentication options for the API"
**Claude:**
1. Researches common auth patterns (JWT, OAuth, API keys)
2. Checks existing codebase auth implementation
3. Evaluates trade-offs for the use case
4. Produces decision-support report
5. Saves to `research/API Authentication Options.md`
6. Recommends approach with rationale
@@ -0,0 +1,292 @@
---
name: spec-driven-development
description: Guide implementation based on specs stored in Basic Memory, following the SPEC-1 specification-driven development process
---
# Spec-Driven Development
This skill guides implementation work based on specifications stored in the Basic Memory "specs" project, following the process defined in SPEC-1.
## When to Use
Use this skill when:
- Implementing a feature defined by a spec
- Creating a new specification before implementation
- Reviewing implementation against spec criteria
- Need to understand what a spec requires
- Updating spec progress as work completes
## The Spec-Driven Process
From SPEC-1, the workflow is:
1. **Create** - Write spec as complete thought in Basic Memory "specs" project
2. **Discuss** - Iterate and refine the specification
3. **Implement** - Execute implementation directly
4. **Validate** - Review implementation against spec criteria
5. **Document** - Update spec with learnings and decisions
## Spec Structure
Every spec contains:
- **Why** - The reasoning and problem being solved
- **What** - What is affected or changed
- **How** - High-level approach to implementation
- **How to Evaluate** - Testing/validation procedure
### Progress Tracking Format
Specs use living documentation with checklists:
```markdown
### Feature Area
- ✅ Basic functionality implemented
- ✅ Props and events defined
- [ ] Add sorting controls
- [ ] Improve accessibility
- [x] Currently implementing responsive design
```
- `✅` - Completed items
- `[ ]` - Pending items
- `[x]` - In-progress items
## Working with Specs
### Reading a Spec
```python
# Get the full spec
mcp__basic-memory__read_note(
identifier="SPEC-24: Postgres Database Migration",
project="specs"
)
# Or search for it
mcp__basic-memory__search_notes(
query="postgres migration",
project="specs"
)
```
### Creating a New Spec
```python
# 1. First, find the next spec number
mcp__basic-memory__search_notes(
query="SPEC-",
project="specs"
)
# 2. Create the spec with proper structure
mcp__basic-memory__write_note(
title="SPEC-30: Your Feature Name",
content="""---
title: 'SPEC-30: Your Feature Name'
type: spec
tags:
- feature-area
- component
---
# SPEC-30: Your Feature Name
## Why
[Problem statement and motivation]
## What
[What is affected or changed]
- Affected areas
- Components involved
- Scope boundaries
## How (High Level)
[Implementation approach]
### Phase 1: Foundation
- [ ] Task 1
- [ ] Task 2
### Phase 2: Core Features
- [ ] Task 3
- [ ] Task 4
## How to Evaluate
### Success Criteria
- [ ] Criterion 1
- [ ] Criterion 2
### Testing Procedure
1. Step 1
2. Step 2
## Observations
- [goal] Primary objective #tag
- [constraint] Known limitation #tag
## Relations
- relates-to [[Related Spec]]
- depends-on [[Dependency]]
""",
folder="", # Root of specs project
project="specs"
)
```
### Updating Spec Progress
```python
# Mark items complete as you implement
mcp__basic-memory__edit_note(
identifier="SPEC-24: Postgres Database Migration",
operation="find_replace",
find_text="- [ ] Create migration scripts",
content="- ✅ Create migration scripts",
project="specs"
)
# Or add new observations
mcp__basic-memory__edit_note(
identifier="SPEC-24: Postgres Database Migration",
operation="append",
content="\n- [learning] Alembic autogenerate works well for model changes #migration",
project="specs"
)
```
### Reviewing Implementation
When reviewing against a spec:
1. **Read the spec's "How to Evaluate" section**
2. **Check each success criterion:**
- Functional completeness
- Test coverage (count test files, check categories)
- Code quality (TypeScript, linting, performance)
- Architecture compliance
- Documentation completeness
3. **Be honest** - Don't overstate completeness
4. **Document findings** - Update spec with review results
5. **Identify gaps** - Clearly note what still needs work
## Implementation Workflow
### Starting Implementation
1. **Read the spec thoroughly**
```python
mcp__basic-memory__read_note(
identifier="SPEC-XX: Feature Name",
project="specs"
)
```
2. **Understand dependencies**
- Check Relations section for dependencies
- Read related specs if needed
3. **Plan your approach**
- Break "How" section into concrete tasks
- Identify what to implement first
4. **Mark first item in-progress**
```python
mcp__basic-memory__edit_note(
identifier="SPEC-XX",
operation="find_replace",
find_text="- [ ] First task",
content="- [x] First task",
project="specs"
)
```
### During Implementation
1. **Update progress as you complete items**
2. **Add observations for decisions made**
3. **Note any deviations from the spec**
4. **Capture learnings that might help future specs**
### After Implementation
1. **Run full evaluation against criteria**
2. **Mark all completed items with ✅**
3. **Add final observations**
4. **Document any follow-up work needed**
## Spec Naming Convention
Format: `SPEC-X: Descriptive Title`
Examples:
- `SPEC-24: Postgres Database Migration`
- `SPEC-25: Cloud Index Service`
- `SPEC-26: Multi-User Security and Permissions`
## Common Spec Patterns
### Feature Spec
```markdown
## Why
User need or problem
## What
- New UI components
- API endpoints
- Database changes
## How
Implementation phases with checkboxes
```
### Architecture Spec
```markdown
## Why
Technical debt or scalability need
## What
- System components affected
- Data flow changes
- Integration points
## How
Migration strategy with rollback plan
```
### Process Spec
```markdown
## Why
Workflow improvement need
## What
- Process steps changed
- Tools involved
- Team impact
## How
Rollout plan and adoption strategy
```
## Best Practices
1. **Spec first, code second** - Write spec before implementation
2. **Keep specs living** - Update as understanding evolves
3. **Be specific in criteria** - Vague criteria = vague completion
4. **Link related specs** - Build the knowledge graph
5. **Capture decisions** - Future you will thank you
6. **Review honestly** - Incomplete is okay, dishonest isn't
7. **Close the loop** - Mark items done as you complete them
## Using with Slash Commands
The `/spec` command provides quick access:
- `/spec create [name]` - Create new specification
- `/spec status` - Show all spec statuses
- `/spec show [name]` - Read a specific spec
- `/spec review [name]` - Validate implementation
+42
View File
@@ -0,0 +1,42 @@
# Docker Compose configuration for Basic Memory with PostgreSQL
# Use this for local development and testing with Postgres backend
#
# Usage:
# docker-compose -f docker-compose-postgres.yml up -d
# docker-compose -f docker-compose-postgres.yml down
services:
postgres:
image: postgres:17
container_name: basic-memory-postgres
environment:
# Local development/test credentials - NOT for production
# These values are referenced by tests and justfile commands
POSTGRES_DB: basic_memory
POSTGRES_USER: basic_memory_user
POSTGRES_PASSWORD: dev_password # Simple password for local testing only
ports:
- "5433:5432"
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U basic_memory_user -d basic_memory"]
interval: 10s
timeout: 5s
retries: 5
restart: unless-stopped
volumes:
# Named volume for Postgres data
postgres_data:
driver: local
# Named volume for persistent configuration
# Database will be stored in Postgres, not in this volume
basic-memory-config:
driver: local
# Network configuration (optional)
# networks:
# basic-memory-net:
# driver: bridge
+7 -26
View File
@@ -77,25 +77,6 @@ SQLite Database (Index)
3. Files are parsed and indexed in SQLite
4. MCP server exposes indexed data to AI
5. AI can query, traverse, and update knowledge graph
### Version 0.15.0 Changes
**Breaking Change: Stateless Architecture**
- All MCP tools now require explicit `project` parameter
- No implicit project context carried between calls
- Exception: `default_project_mode` config option enables fallback
**Three-Tier Project Resolution**:
1. CLI constraint: `--project name` flag (highest priority)
2. Explicit parameter: `project="name"` in tool calls
3. Default mode: `default_project_mode=true` in config (fallback)
**Why This Matters**:
- More predictable behavior across sessions
- Explicit project selection prevents errors
- Multi-project workflows more reliable
- Single-project users can enable default mode for convenience
---
## Project Management
@@ -576,7 +557,7 @@ Adopt GraphQL instead of REST for our API layer.
""",
folder="decisions",
tags=["decision", "api", "graphql"],
entity_type="decision",
note_type="decision",
project="main"
)
```
@@ -613,7 +594,7 @@ await write_note(
""",
folder="meetings",
tags=["meeting", "api", "team"],
entity_type="meeting",
note_type="meeting",
project="main"
)
```
@@ -665,7 +646,7 @@ Specification for user authentication system using JWT tokens.
""",
folder="specs",
tags=["spec", "auth", "security"],
entity_type="spec",
note_type="spec",
project="main"
)
```
@@ -1631,7 +1612,7 @@ await write_note(
{related_entities}
""",
folder="decisions",
entity_type="decision",
note_type="decision",
project="main"
)
```
@@ -2706,14 +2687,14 @@ await write_note(
### Content Management
**write_note(title, content, folder, tags, entity_type, project)**
**write_note(title, content, folder, tags, note_type, project)**
- Create or update markdown notes
- Parameters:
- `title` (required): Note title
- `content` (required): Markdown content
- `folder` (required): Destination folder
- `tags` (optional): List of tags
- `entity_type` (optional): Entity type (note, person, meeting, etc.)
- `note_type` (optional): Type of note (stored in frontmatter). Can be "note", "person", "meeting", "guide", etc.
- `project` (required unless default_project_mode): Target project
- Returns: Created/updated entity with permalink
- Example:
@@ -2723,7 +2704,7 @@ await write_note(
content="# API Design\n...",
folder="specs",
tags=["api", "design"],
entity_type="spec",
note_type="spec",
project="main"
)
```
+565 -496
View File
File diff suppressed because it is too large Load Diff
+67 -5
View File
@@ -2,21 +2,83 @@
# Install dependencies
install:
pip install -e ".[dev]"
uv pip install -e ".[dev]"
uv sync
@echo ""
@echo "💡 Remember to activate the virtual environment by running: source .venv/bin/activate"
# Run all tests with unified coverage report
test: test-unit test-int
# Run unit tests only (fast, no coverage)
test-unit:
uv run pytest -p pytest_mock -v --no-cov -n auto tests
uv run pytest -p pytest_mock -v --no-cov tests
# Run integration tests only (fast, no coverage)
test-int:
uv run pytest -p pytest_mock -v --no-cov -n auto test-int
uv run pytest -p pytest_mock -v --no-cov test-int
# Run all tests with unified coverage report
test: test-unit test-int
# ==============================================================================
# DATABASE BACKEND TESTING
# ==============================================================================
# Basic Memory supports dual database backends (SQLite and Postgres).
# Tests are parametrized to run against both backends automatically.
#
# Quick Start:
# just test-sqlite # Run SQLite tests (default, no Docker needed)
# just test-postgres # Run Postgres tests (requires Docker)
#
# For Postgres tests, first start the database:
# docker-compose -f docker-compose-postgres.yml up -d
# ==============================================================================
# Run tests against SQLite only (default backend, skip Postgres/Benchmark tests)
# This is the fastest option and doesn't require any Docker setup.
# Use this for local development and quick feedback.
# Includes Windows-specific tests which will auto-skip on non-Windows platforms.
test-sqlite:
uv run pytest -p pytest_mock -v --no-cov -m "not postgres and not benchmark" tests test-int
# Run tests against Postgres only (requires docker-compose-postgres.yml up)
# First start Postgres: docker-compose -f docker-compose-postgres.yml up -d
# Tests will connect to localhost:5433/basic_memory_test
# To reset the database: just postgres-reset
test-postgres:
uv run pytest -p pytest_mock -v --no-cov -m "postgres and not benchmark" tests test-int
# Reset Postgres test database (drops and recreates schema)
# Useful when Alembic migration state gets out of sync during development
# Uses credentials from docker-compose-postgres.yml
postgres-reset:
docker exec basic-memory-postgres psql -U ${POSTGRES_USER:-basic_memory_user} -d ${POSTGRES_TEST_DB:-basic_memory_test} -c "DROP SCHEMA public CASCADE; CREATE SCHEMA public;"
@echo "✅ Postgres test database reset"
# Run Alembic migrations manually against Postgres test database
# Useful for debugging migration issues
# Uses credentials from docker-compose-postgres.yml (can override with env vars)
postgres-migrate:
@cd src/basic_memory/alembic && \
BASIC_MEMORY_DATABASE_BACKEND=postgres \
BASIC_MEMORY_DATABASE_URL=${POSTGRES_TEST_URL:-postgresql://basic_memory_user:dev_password@localhost:5433/basic_memory_test} \
uv run alembic upgrade head
@echo "✅ Migrations applied to Postgres test database"
# Run Windows-specific tests only (only works on Windows platform)
# These tests verify Windows-specific database optimizations (locking mode, NullPool)
# Will be skipped automatically on non-Windows platforms
test-windows:
uv run pytest -p pytest_mock -v --no-cov -m windows tests test-int
# Run benchmark tests only (performance testing)
# These are slow tests that measure sync performance with various file counts
# Excluded from default test runs to keep CI fast
test-benchmark:
uv run pytest -p pytest_mock -v --no-cov -m benchmark tests test-int
# Run all tests including Windows, Postgres, and Benchmarks (for CI/comprehensive testing)
# Use this before releasing to ensure everything works across all backends and platforms
test-all:
uv run pytest -p pytest_mock -v --no-cov tests test-int
# Generate HTML coverage report
coverage:
+7 -2
View File
@@ -15,7 +15,6 @@ dependencies = [
"aiosqlite>=0.20.0",
"greenlet>=3.1.1",
"pydantic[email,timezone]>=2.10.3",
"icecream>=2.1.3",
"mcp>=1.2.0",
"pydantic-settings>=2.6.1",
"loguru>=0.7.3",
@@ -34,7 +33,10 @@ dependencies = [
"pyjwt>=2.10.1",
"python-dotenv>=1.1.0",
"pytest-aio>=1.9.0",
"aiofiles>=24.1.0", # Async file I/O
"aiofiles>=24.1.0", # Async file I/O
"logfire>=0.73.0", # Optional observability (disabled by default via config)
"asyncpg>=0.30.0",
"nest-asyncio>=1.6.0", # For Alembic migrations with Postgres
]
@@ -60,6 +62,8 @@ asyncio_default_fixture_loop_scope = "function"
markers = [
"benchmark: Performance benchmark tests (deselect with '-m \"not benchmark\"')",
"slow: Slow-running tests (deselect with '-m \"not slow\"')",
"postgres: Tests that run against Postgres backend (deselect with '-m \"not postgres\"')",
"windows: Windows-specific tests (deselect with '-m \"not windows\"')",
]
[tool.ruff]
@@ -77,6 +81,7 @@ dev = [
"pytest-xdist>=3.0.0",
"ruff>=0.1.6",
"freezegun>=1.5.5",
]
[tool.hatch.version]
@@ -0,0 +1,569 @@
---
title: 'SPEC-10: Unified Deployment Workflow and Event Tracking'
type: spec
permalink: specs/spec-10-unified-deployment-workflow-event-tracking
tags:
- workflow
- deployment
- event-sourcing
- architecture
- simplification
---
# SPEC-10: Unified Deployment Workflow and Event Tracking
## Why
We replaced a complex multi-workflow system with DBOS orchestration that was proving to be more trouble than it was worth. The previous architecture had four separate workflows (`tenant_provisioning`, `tenant_update`, `tenant_deployment`, `tenant_undeploy`) with overlapping logic, complex state management, and fragmented event tracking. DBOS added unnecessary complexity without providing sufficient value, leading to harder debugging and maintenance.
**Problems Solved:**
- **Framework Complexity**: DBOS configuration overhead and fighting framework limitations
- **Code Duplication**: Multiple workflows implementing similar operations with duplicate logic
- **Poor Observability**: Fragmented event tracking across workflow boundaries
- **Maintenance Overhead**: Complex orchestration for fundamentally simple operations
- **Debugging Difficulty**: Framework abstractions hiding simple Python stack traces
## What
This spec documents the architectural simplification that consolidates tenant lifecycle management into a unified system with comprehensive event tracking.
**Affected Areas:**
- Tenant deployment workflows (provisioning, updates, undeploying)
- Event sourcing and workflow tracking infrastructure
- API endpoints for tenant operations
- Database schema for workflow and event correlation
- Integration testing for tenant lifecycle operations
**Key Changes:**
- **Removed DBOS entirely** - eliminated framework dependency and complexity
- **Consolidated 4 workflows → 2 unified deployment workflows (deploy/undeploy)**
- **Added workflow tracking system** with complete event correlation
- **Simplified API surface** - single `/deploy` endpoint handles all scenarios
- **Enhanced observability** through event sourcing with workflow grouping
## How (High Level)
### Architectural Philosophy
**Embrace simplicity over framework complexity** - use well-structured Python with proper database design instead of complex orchestration frameworks.
### Core Components
#### 1. Unified Deployment Workflow
```python
class TenantDeploymentWorkflow:
async def deploy_tenant_workflow(self, tenant_id: str, workflow_id: UUID, image_tag: str = None):
# Single workflow handles both initial provisioning AND updates
# Each step is idempotent and handles its own error recovery
# Database transactions provide the durability we need
await self.start_deployment_step(workflow_id, tenant_uuid, image_tag)
await self.create_fly_app_step(workflow_id, tenant_uuid)
await self.create_bucket_step(workflow_id, tenant_uuid)
await self.deploy_machine_step(workflow_id, tenant_uuid, image_tag)
await self.complete_deployment_step(workflow_id, tenant_uuid, image_tag, deployment_time)
```
**Key Benefits:**
- **Handles both provisioning and updates** in single workflow
- **Idempotent operations** - safe to retry any step
- **Clean error handling** via simple Python exceptions
- **Resumable** - can restart from any failed step
#### 2. Workflow Tracking System
**Database Schema:**
```sql
CREATE TABLE workflow (
id UUID PRIMARY KEY,
workflow_type VARCHAR(50) NOT NULL, -- 'tenant_deployment', 'tenant_undeploy'
tenant_id UUID REFERENCES tenant(id),
status VARCHAR(20) DEFAULT 'running', -- 'running', 'completed', 'failed'
workflow_metadata JSONB DEFAULT '{}' -- image_tag, etc.
);
ALTER TABLE event ADD COLUMN workflow_id UUID REFERENCES workflow(id);
```
**Event Correlation:**
- Every workflow operation generates events tagged with `workflow_id`
- Complete audit trail from workflow start to completion
- Events grouped by workflow for easy reconstruction of operations
#### 3. Parameter Standardization
All workflow methods follow consistent signature pattern:
```python
async def method_name(self, session: AsyncSession, workflow_id: UUID | None, tenant_id: UUID, ...)
```
**Benefits:**
- **Consistent event tagging** - all events properly correlated
- **Clear method contracts** - workflow_id always first parameter
- **Type safety** - proper UUID handling throughout
### Implementation Strategy
#### Phase 1: Workflow Consolidation ✅ COMPLETED
- [x] **Remove DBOS dependency** - eliminated dbos_config.py and all DBOS imports
- [x] **Create unified TenantDeploymentWorkflow** - handles both provisioning and updates
- [x] **Remove legacy workflows** - deleted tenant_provisioning.py, tenant_update.py
- [x] **Simplify API endpoints** - consolidated to single `/deploy` endpoint
- [x] **Update integration tests** - comprehensive edge case testing
#### Phase 2: Workflow Tracking System ✅ COMPLETED
- [x] **Database migration** - added workflow table and event.workflow_id foreign key
- [x] **Workflow repository** - CRUD operations for workflow records
- [x] **Event correlation** - all workflow events tagged with workflow_id
- [x] **Comprehensive testing** - workflow lifecycle and event grouping tests
#### Phase 3: Parameter Standardization ✅ COMPLETED
- [x] **Standardize method signatures** - workflow_id as first parameter pattern
- [x] **Fix event tagging** - ensure all workflow events properly correlated
- [x] **Update service methods** - consistent parameter order across tenant_service
- [x] **Integration test validation** - verify complete event sequences
### Architectural Benefits
#### Code Simplification
- **39 files changed**: 2,247 additions, 3,256 deletions (net -1,009 lines)
- **Eliminated framework complexity** - no more DBOS configuration or abstractions
- **Consolidated logic** - single deployment workflow vs 4 separate workflows
- **Cleaner API surface** - unified endpoint vs multiple workflow-specific endpoints
#### Enhanced Observability
- **Complete event correlation** - every workflow event tagged with workflow_id
- **Audit trail reconstruction** - can trace entire tenant lifecycle through events
- **Workflow status tracking** - running/completed/failed states in database
- **Comprehensive testing** - edge cases covered with real infrastructure
#### Operational Benefits
- **Simpler debugging** - plain Python stack traces vs framework abstractions
- **Reduced dependencies** - one less complex framework to maintain
- **Better error handling** - explicit exception handling vs framework magic
- **Easier maintenance** - straightforward Python code vs orchestration complexity
## How to Evaluate
### Success Criteria
#### Functional Completeness ✅ VERIFIED
- [x] **Unified deployment workflow** handles both initial provisioning and updates
- [x] **Undeploy workflow** properly integrated with event tracking
- [x] **All operations idempotent** - safe to retry any step without duplication
- [x] **Complete tenant lifecycle** - provision → active → update → undeploy
#### Event Tracking and Correlation ✅ VERIFIED
- [x] **All workflow events tagged** with proper workflow_id
- [x] **Event sequence verification** - tests assert exact event order and content
- [x] **Workflow grouping** - events can be queried by workflow_id for complete audit trail
- [x] **Cross-workflow isolation** - deployment vs undeploy events properly separated
#### Database Schema and Performance ✅ VERIFIED
- [x] **Migration applied** - workflow table and event.workflow_id column created
- [x] **Proper indexing** - performance optimized queries on workflow_type, tenant_id, status
- [x] **Foreign key constraints** - referential integrity between workflows and events
- [x] **Database triggers** - updated_at timestamp automation
#### Test Coverage ✅ COMPREHENSIVE
- [x] **Unit tests**: 4 workflow tracking tests covering lifecycle and event grouping
- [x] **Integration tests**: Real infrastructure testing with Fly.io resources
- [x] **Edge case coverage**: Failed deployments, partial state recovery, resource conflicts
- [x] **Event sequence verification**: Exact event order and content validation
### Testing Procedure
#### Unit Test Validation ✅ PASSING
```bash
cd apps/cloud && pytest tests/test_workflow_tracking.py -v
# 4/4 tests passing - workflow lifecycle and event grouping
```
#### Integration Test Validation ✅ PASSING
```bash
cd apps/cloud && pytest tests/integration/test_tenant_workflow_deployment_integration.py -v
cd apps/cloud && pytest tests/integration/test_tenant_workflow_undeploy_integration.py -v
# Comprehensive real infrastructure testing with actual Fly.io resources
# Tests provision → deploy → update → undeploy → cleanup cycles
```
### Performance Metrics
#### Code Metrics ✅ ACHIEVED
- **Net code reduction**: -1,009 lines (3,256 deletions, 2,247 additions)
- **Workflow consolidation**: 4 workflows → 1 unified deployment workflow
- **Dependency reduction**: Removed DBOS framework dependency entirely
- **API simplification**: Multiple endpoints → single `/deploy` endpoint
#### Operational Metrics ✅ VERIFIED
- **Event correlation**: 100% of workflow events properly tagged with workflow_id
- **Audit trail completeness**: Full tenant lifecycle traceable through event sequences
- **Error handling**: Clean Python exceptions vs framework abstractions
- **Debugging simplicity**: Direct stack traces vs orchestration complexity
### Implementation Status: ✅ COMPLETE
All phases completed successfully with comprehensive testing and verification:
**Phase 1 - Workflow Consolidation**: ✅ COMPLETE
- Removed DBOS dependency and consolidated workflows
- Unified deployment workflow handles all scenarios
- Comprehensive integration testing with real infrastructure
**Phase 2 - Workflow Tracking**: ✅ COMPLETE
- Database schema implemented with proper indexing
- Event correlation system fully functional
- Complete audit trail capability verified
**Phase 3 - Parameter Standardization**: ✅ COMPLETE
- Consistent method signatures across all workflow methods
- All events properly tagged with workflow_id
- Type safety verified across entire codebase
**Phase 4 - Asynchronous Job Queuing**:
**Goal**: Transform synchronous deployment workflows into background jobs for better user experience and system reliability.
**Current Problem**:
- Deployment API calls are synchronous - users wait for entire tenant provisioning (30-60 seconds)
- No retry mechanism for failed operations
- HTTP timeouts on long-running deployments
- Poor user experience during infrastructure provisioning
**Solution**: Redis-backed job queue with arq for reliable background processing
#### Architecture Overview
```python
# API Layer: Return immediately with job tracking
@router.post("/{tenant_id}/deploy")
async def deploy_tenant(tenant_id: UUID):
# Create workflow record in Postgres
workflow = await workflow_repo.create_workflow("tenant_deployment", tenant_id)
# Enqueue job in Redis
job = await arq_pool.enqueue_job('deploy_tenant_task', tenant_id, workflow.id)
# Return job ID immediately
return {"job_id": job.job_id, "workflow_id": workflow.id, "status": "queued"}
# Background Worker: Process via existing unified workflow
async def deploy_tenant_task(ctx, tenant_id: str, workflow_id: str):
# Existing workflow logic - zero changes needed!
await workflow_manager.deploy_tenant(UUID(tenant_id), workflow_id=UUID(workflow_id))
```
#### Implementation Tasks
**Phase 4.1: Core Job Queue Setup** ✅ COMPLETED
- [x] **Add arq dependency** - integrated Redis job queue with existing infrastructure
- [x] **Create job definitions** - wrapped existing deployment/undeploy workflows as arq tasks
- [x] **Update API endpoints** - updated provisioning endpoints to return job IDs instead of waiting for completion
- [x] **JobQueueService implementation** - service layer for job enqueueing and status tracking
- [x] **Job status tracking** - integrated with existing workflow table for status updates
- [x] **Comprehensive testing** - 18 tests covering positive, negative, and edge cases
**Phase 4.2: Background Worker Implementation** ✅ COMPLETED
- [x] **Job status API** - GET /jobs/{job_id}/status endpoint integrated with JobQueueService
- [x] **Background worker process** - arq worker to process queued jobs with proper settings and Redis configuration
- [x] **Worker settings and configuration** - WorkerSettings class with proper timeouts, max jobs, and error handling
- [x] **Fix API endpoints** - updated job status API to use JobQueueService instead of direct Redis access
- [x] **Integration testing** - comprehensive end-to-end testing with real ARQ workers and Fly.io infrastructure
- [x] **Worker entry points** - dual-purpose entrypoint.sh script and __main__.py module support for both API and worker processes
- [x] **Test fixture updates** - fixed all API and service test fixtures to work with job queue dependencies
- [x] **AsyncIO event loop fixes** - resolved event loop issues in integration tests for subprocess worker compatibility
- [x] **Complete test coverage** - all 46 tests passing across unit, integration, and API test suites
- [x] **Type safety verification** - 0 type checking errors across entire ARQ job queue implementation
#### Phase 4.2 Implementation Summary ✅ COMPLETE
**Core ARQ Job Queue System:**
- **JobQueueService** - Centralized service for job enqueueing, status tracking, and Redis pool management
- **deployment_jobs.py** - ARQ job functions that wrap existing deployment/undeploy workflows
- **Worker Settings** - Production-ready ARQ configuration with proper timeouts and error handling
- **Dual-Process Architecture** - Single Docker image with entrypoint.sh supporting both API and worker modes
**Key Files Added:**
- `apps/cloud/src/basic_memory_cloud/jobs/` - Complete job queue implementation (7 files)
- `apps/cloud/entrypoint.sh` - Dual-purpose Docker container entry point
- `apps/cloud/tests/integration/test_worker_integration.py` - Real infrastructure integration tests
- `apps/cloud/src/basic_memory_cloud/schemas/job_responses.py` - API response schemas
**API Integration:**
- Provisioning endpoints return job IDs immediately instead of blocking for 60+ seconds
- Job status API endpoints for real-time monitoring of deployment progress
- Proper error handling and job failure scenarios with detailed error messages
**Testing Achievement:**
- **46 total tests passing** across all test suites (unit, integration, API, services)
- **Real infrastructure testing** - ARQ workers process actual Fly.io deployments
- **Event loop safety** - Fixed asyncio issues for subprocess worker compatibility
- **Test fixture updates** - All fixtures properly support job queue dependencies
- **Type checking** - 0 errors across entire codebase
**Technical Metrics:**
- **38 files changed** - +1,736 insertions, -334 deletions
- **Integration test runtime** - ~18 seconds with real ARQ workers and Fly.io verification
- **Event loop isolation** - Proper async session management for subprocess compatibility
- **Redis integration** - Production-ready Redis configuration with connection pooling
**Phase 4.3: Production Hardening** ✅ COMPLETED
- [x] **Configure Upstash Redis** - production Redis setup on Fly.io
- [x] **Retry logic for external APIs** - exponential backoff for flaky Tigris IAM operations
- [x] **Monitoring and observability** - comprehensive Redis queue monitoring with CLI tools
- [x] **Error handling improvements** - graceful handling of expected API errors with appropriate log levels
- [x] **CLI tooling enhancements** - bulk update commands for CI/CD automation
- [x] **Documentation improvements** - comprehensive monitoring guide with Redis patterns
- [x] **Job uniqueness** - ARQ-based duplicate prevention for tenant operations
- [ ] **Worker scaling** - multiple arq workers for parallel job processing
- [ ] **Job persistence** - ensure jobs survive Redis/worker restarts
- [ ] **Error alerting** - notifications for failed deployment jobs
**Phase 4.4: Advanced Features** (Future)
- [ ] **Job scheduling** - deploy tenants at specific times
- [ ] **Priority queues** - urgent deployments processed first
- [ ] **Batch operations** - bulk tenant deployments
- [ ] **Job dependencies** - deployment → configuration → activation chains
#### Benefits Achieved ✅ REALIZED
**User Experience Improvements:**
- **Immediate API responses** - users get job ID instantly vs waiting 60+ seconds for deployment completion
- **Real-time job tracking** - status API provides live updates on deployment progress
- **Better error visibility** - detailed error messages and job failure tracking
- **CI/CD automation ready** - bulk update commands for automated tenant deployments
**System Reliability:**
- **Redis persistence** - jobs survive Redis/worker restarts with proper queue durability
- **Idempotent job processing** - jobs can be safely retried without side effects
- **Event loop isolation** - worker processes operate independently from API server
- **Retry resilience** - exponential backoff for flaky external API calls (3 attempts, 1s/2s delays)
- **Graceful error handling** - expected API errors logged at INFO level, unexpected at ERROR level
- **Job uniqueness** - prevent duplicate tenant operations with ARQ's built-in uniqueness feature
**Operational Benefits:**
- **Horizontal scaling ready** - architecture supports adding more workers for parallel processing
- **Comprehensive testing** - real infrastructure integration tests ensure production reliability
- **Type safety** - full type checking prevents runtime errors in job processing
- **Clean separation** - API and worker processes use same codebase with different entry points
- **Queue monitoring** - Redis CLI integration for real-time queue activity monitoring
- **Comprehensive documentation** - detailed monitoring guide with Redis pattern explanations
**Development Benefits:**
- **Zero workflow changes** - existing deployment/undeploy workflows work unchanged as background jobs
- **Async/await native** - modern Python asyncio patterns throughout the implementation
- **Event correlation preserved** - all existing workflow tracking and event sourcing continues to work
- **Enhanced CLI tooling** - unified tenant commands with proper endpoint routing
- **Database integrity** - proper foreign key constraint handling in tenant deletion
#### Infrastructure Requirements
- **Local**: Redis via docker-compose (already exists) ✅
- **Production**: Upstash Redis on Fly.io (already configured) ✅
- **Workers**: arq worker processes (new deployment target)
- **Monitoring**: Job status dashboard (simple web interface)
#### API Evolution
```python
# Before: Synchronous (blocks for 60+ seconds)
POST /tenant/{id}/deploy {status: "active", machine_id: "..."}
# After: Asynchronous (returns immediately)
POST /tenant/{id}/deploy {job_id: "uuid", workflow_id: "uuid", status: "queued"}
GET /jobs/{job_id}/status {status: "running", progress: "deploying_machine", workflow_id: "uuid"}
GET /workflows/{workflow_id}/events [...] # Existing event tracking works unchanged
```
**Technology Choice**: **arq (Redis)** over pgqueuer
- **Existing Redis infrastructure** - Upstash + docker-compose already configured
- **Better ecosystem** - monitoring tools, documentation, community
- **Made by pydantic team** - aligns with existing Python stack
- **Hybrid approach** - Redis for queue operations + Postgres for workflow state
#### Job Uniqueness Implementation
**Problem**: Multiple concurrent deployment requests for the same tenant could create duplicate jobs, wasting resources and potentially causing conflicts.
**Solution**: Leverage ARQ's built-in job uniqueness feature using predictable job IDs:
```python
# JobQueueService implementation
async def enqueue_deploy_job(self, tenant_id: UUID, image_tag: str | None = None) -> str:
unique_job_id = f"deploy-{tenant_id}"
job = await self.redis_pool.enqueue_job(
"deploy_tenant_job",
str(tenant_id),
image_tag,
_job_id=unique_job_id, # ARQ prevents duplicates
)
if job is None:
# Job already exists - return existing job ID
return unique_job_id
else:
# New job created - return ARQ job ID
return job.job_id
```
**Key Features:**
- **Predictable Job IDs**: `deploy-{tenant_id}`, `undeploy-{tenant_id}`
- **Duplicate Prevention**: ARQ returns `None` for duplicate job IDs
- **Graceful Handling**: Return existing job ID instead of raising errors
- **Idempotent Operations**: Safe to retry deployment requests
- **Clear Logging**: Distinguish "Enqueued new" vs "Found existing" jobs
**Benefits:**
- Prevents resource waste from duplicate deployments
- Eliminates race conditions from concurrent requests
- Makes job monitoring more predictable with consistent IDs
- Provides natural deduplication without complex locking mechanisms
## Notes
### Design Philosophy Lessons
- **Simplicity beats framework magic** - removing DBOS made the system more reliable and debuggable
- **Event sourcing > complex orchestration** - database-backed event tracking provides better observability than framework abstractions
- **Idempotent operations > resumable workflows** - each step handling its own retry logic is simpler than framework-managed resumability
- **Explicit error handling > framework exception handling** - Python exceptions are clearer than orchestration framework error states
### Future Considerations
- **Monitoring integration** - workflow tracking events could feed into observability systems
- **Performance optimization** - event querying patterns may benefit from additional indexing
- **Audit compliance** - complete event trail supports regulatory requirements
- **Operational dashboards** - workflow status could drive tenant health monitoring
### Related Specifications
- **SPEC-8**: TigrisFS Integration - bucket provisioning integrated with deployment workflow
- **SPEC-1**: Specification-Driven Development Process - this spec follows the established format
## Observations
- [architecture] Removing framework complexity led to more maintainable system #simplification
- [workflow] Single unified deployment workflow handles both provisioning and updates #consolidation
- [observability] Event sourcing with workflow correlation provides complete audit trail #event-tracking
- [database] Foreign key relationships between workflows and events enable powerful queries #schema-design
- [testing] Integration tests with real infrastructure catch edge cases that unit tests miss #testing-strategy
- [parameters] Consistent method signatures (workflow_id first) reduce cognitive overhead #api-design
- [maintenance] Fewer workflows and dependencies reduce long-term maintenance burden #operational-excellence
- [debugging] Plain Python exceptions are clearer than framework abstraction layers #developer-experience
- [resilience] Exponential backoff retry patterns handle flaky external API calls gracefully #error-handling
- [monitoring] Redis queue monitoring provides real-time operational visibility #observability
- [ci-cd] Bulk update commands enable automated tenant deployments in continuous delivery pipelines #automation
- [documentation] Comprehensive monitoring guides reduce operational learning curve #knowledge-management
- [error-logging] Context-aware log levels (INFO for expected errors, ERROR for unexpected) improve signal-to-noise ratio #logging-strategy
- [job-uniqueness] ARQ job uniqueness with predictable tenant-based IDs prevents duplicate operations and resource waste #deduplication
## Implementation Notes
### Configuration Integration
- **Redis Configuration**: Add Redis settings to existing `apps/cloud/src/basic_memory_cloud/config.py`
- **Local Development**: Leverage existing Redis setup from `docker-compose.yml`
- **Production**: Use Upstash Redis configuration for production environments
### Docker Entrypoint Strategy
Create `entrypoint.sh` script to toggle between API server and worker processes using single Docker image:
```bash
#!/bin/bash
# Entrypoint script for Basic Memory Cloud service
# Supports multiple process types: api, worker
set -e
case "$1" in
"api")
echo "Starting Basic Memory Cloud API server..."
exec uvicorn basic_memory_cloud.main:app \
--host 0.0.0.0 \
--port 8000 \
--log-level info
;;
"worker")
echo "Starting Basic Memory Cloud ARQ worker..."
# For ARQ worker implementation
exec python -m arq basic_memory_cloud.jobs.settings.WorkerSettings
;;
*)
echo "Usage: $0 {api|worker}"
echo " api - Start the FastAPI server"
echo " worker - Start the ARQ worker"
exit 1
;;
esac
```
### Fly.io Process Groups Configuration
Use separate machine groups for API and worker processes with independent scaling:
```toml
# fly.toml app configuration for basic-memory-cloud
app = 'basic-memory-cloud-dev-basic-machines'
primary_region = 'dfw'
org = 'basic-machines'
kill_signal = 'SIGINT'
kill_timeout = '5s'
[build]
# Process groups for API server and worker
[processes]
api = "api"
worker = "worker"
# Machine scaling configuration
[[machine]]
size = 'shared-cpu-1x'
processes = ['api']
min_machines_running = 1
auto_stop_machines = false
auto_start_machines = true
[[machine]]
size = 'shared-cpu-1x'
processes = ['worker']
min_machines_running = 1
auto_stop_machines = false
auto_start_machines = true
[env]
# Python configuration
PYTHONUNBUFFERED = '1'
PYTHONPATH = '/app'
# Logging configuration
LOG_LEVEL = 'DEBUG'
# Redis configuration for ARQ
REDIS_URL = 'redis://basic-memory-cloud-redis.upstash.io'
# Database configuration
DATABASE_HOST = 'basic-memory-cloud-db-dev-basic-machines.internal'
DATABASE_PORT = '5432'
DATABASE_NAME = 'basic_memory_cloud'
DATABASE_USER = 'postgres'
DATABASE_SSL = 'true'
# Worker configuration
ARQ_MAX_JOBS = '10'
ARQ_KEEP_RESULT = '3600'
# Fly.io configuration
FLY_ORG = 'basic-machines'
FLY_REGION = 'dfw'
# Internal service - no external HTTP exposure for worker
# API accessible via basic-memory-cloud-dev-basic-machines.flycast:8000
[[vm]]
size = 'shared-cpu-1x'
```
### Benefits of This Architecture
- **Single Docker Image**: Both API and worker use same container with different entrypoints
- **Independent Scaling**: Scale API and worker processes separately based on demand
- **Clean Separation**: Web traffic handling separate from background job processing
- **Existing Infrastructure**: Leverages current PostgreSQL + Redis setup without complexity
- **Hybrid State Management**: Redis for queue operations, PostgreSQL for persistent workflow tracking
## Relations
- implements [[SPEC-8 TigrisFS Integration]]
- follows [[SPEC-1 Specification-Driven Development Process]]
- supersedes previous multi-workflow architecture
@@ -31,8 +31,6 @@ HTTP requests to the API suffer from 350ms-2.6s latency overhead **before** any
This creates compounding effects with tenant auto-start delays and increases timeout risk in cloud deployments.
Github issue: https://github.com/basicmachines-co/basic-memory-cloud/issues/82
## What
This optimization affects the **core basic-memory repository** components:
@@ -170,76 +168,19 @@ Validation Checklist
- Documentation: Performance optimization documented in README
- Cloud Integration: basic-memory-cloud sees performance benefits
## Implementation Status ✅ COMPLETED
**Implementation Date**: 2025-09-26
**Branch**: `feature/spec-11-api-performance-optimization`
**Commit**: `771f60b`
### ✅ Phase 1: Database Connection Caching - IMPLEMENTED
**Files Modified:**
- `src/basic_memory/api/app.py` - Added database connection caching in app.state
- `src/basic_memory/deps.py` - Updated get_engine_factory() to use cached connections
- `src/basic_memory/config.py` - Added skip_initialization_sync configuration flag
**Implementation Details:**
1. **API Lifespan Caching**: Database engine and session_maker cached in app.state during startup
2. **Dependency Injection Optimization**: get_engine_factory() now returns cached connections instead of calling get_or_create_db()
3. **Project Reconciliation Removal**: Eliminated expensive reconcile_projects_with_config() from API startup
4. **CLI Fallback Preserved**: Non-API contexts continue to work with fallback database initialization
### ✅ Performance Validation - ACHIEVED
**Live Testing Results** (2025-09-26 14:03-14:09):
| Operation | Before | After | Improvement |
|-----------|--------|-------|-------------|
| `read_note` | 350ms-2.6s | **20ms** | **95-99% faster** |
| `edit_note` | 350ms-2.6s | **218ms** | **75-92% faster** |
| `search_notes` | 350ms-2.6s | **<500ms** | **Responsive** |
| `list_memory_projects` | N/A | **<100ms** | **Fast** |
**Key Achievements:**
-**95-99% improvement** in read operations (primary workflow)
-**75-92% improvement** in edit operations
-**Zero overhead** for project switching
-**Database connection overhead eliminated** (0ms vs 50-100ms)
-**Project reconciliation delays removed** from API requests
-**<500ms target achieved** for all operations except write (which includes file sync)
### ✅ Backwards Compatibility - MAINTAINED
- All existing functionality preserved
- CLI operations unaffected
- Fallback for non-API contexts maintained
- No breaking changes to existing APIs
- Optional configuration with safe defaults
### ✅ Testing Validation - PASSED
- Integration tests passing
- Type checking clear
- Linting checks passed
- Live testing with real MCP tools successful
- Multi-project workflows validated
- Rapid project switching validated
## Notes
Notes
Implementation Priority:
- Phase 1 COMPLETED: Database connection caching provides 95%+ performance gains
- Phase 2 NOT NEEDED: Project reconciliation removal achieved the goals
- Phase 3 INCLUDED: skip_initialization_sync flag added
- Phase 1 provides 80% of performance gains and should be implemented first
- Phase 2 provides remaining 20% and addresses edge cases
- Phase 3 is optional for maximum cloud optimization
Risk Mitigation:
- All changes backwards compatible implemented
- Gradual implementation successful (Phase 1 → validation)
- Easy rollback via configuration flags available
- All changes backwards compatible
- Gradual rollout possible (Phase 1 → 2 → 3)
- Easy rollback via configuration flags
Cloud Integration:
- This optimization directly addresses basic-memory-cloud issue #82
- Changes in core basic-memory will benefit all cloud tenants
- No changes needed in basic-memory-cloud itself
**Result**: SPEC-11 performance optimizations successfully implemented and validated. The 95-99% improvement in MCP tool response times exceeds the original 50-80% target, providing exceptional performance gains for cloud deployments and local usage.
- This optimization directly addresses basic-memory-cloud issue #82
- Changes in core basic-memory will benefit all cloud tenants
- No changes needed in basic-memory-cloud itself
@@ -0,0 +1,182 @@
# SPEC-12: OpenTelemetry Observability
## Why
We need comprehensive observability for basic-memory-cloud to:
- Track request flows across our multi-tenant architecture (MCP → Cloud → API services)
- Debug performance issues and errors in production
- Understand user behavior and system usage patterns
- Correlate issues to specific tenants for targeted debugging
- Monitor service health and latency across the distributed system
Currently, we only have basic logging without request correlation or distributed tracing capabilities.
## What
Implement OpenTelemetry instrumentation across all basic-memory-cloud services with:
### Core Requirements
1. **Distributed Tracing**: End-to-end request tracing from MCP gateway through to tenant API instances
2. **Tenant Correlation**: All traces tagged with tenant_id, user_id, and workos_user_id
3. **Service Identification**: Clear service naming and namespace separation
4. **Auto-instrumentation**: Automatic tracing for FastAPI, SQLAlchemy, HTTP clients
5. **Grafana Cloud Integration**: Direct OTLP export to Grafana Cloud Tempo
### Services to Instrument
- **MCP Gateway** (basic-memory-mcp): Entry point with JWT extraction
- **Cloud Service** (basic-memory-cloud): Provisioning and management operations
- **API Service** (basic-memory-api): Tenant-specific instances
- **Worker Processes** (ARQ workers): Background job processing
### Key Trace Attributes
- `tenant.id`: UUID from UserProfile.tenant_id
- `user.id`: WorkOS user identifier
- `user.email`: User email for debugging
- `service.name`: Specific service identifier
- `service.namespace`: Environment (development/production)
- `operation.type`: Business operation (provision/update/delete)
- `tenant.app_name`: Fly.io app name for tenant instances
## How
### Phase 1: Setup OpenTelemetry SDK
1. Add OpenTelemetry dependencies to each service's pyproject.toml:
```python
"opentelemetry-distro[otlp]>=1.29.0",
"opentelemetry-instrumentation-fastapi>=0.50b0",
"opentelemetry-instrumentation-httpx>=0.50b0",
"opentelemetry-instrumentation-sqlalchemy>=0.50b0",
"opentelemetry-instrumentation-logging>=0.50b0",
```
2. Create shared telemetry initialization module (`apps/shared/telemetry.py`)
3. Configure Grafana Cloud OTLP endpoint via environment variables:
```bash
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-2.grafana.net/otlp
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic[token]
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
```
### Phase 2: Instrument MCP Gateway
1. Extract tenant context from AuthKit JWT in middleware
2. Create root span with tenant attributes
3. Propagate trace context to downstream services via headers
### Phase 3: Instrument Cloud Service
1. Continue trace from MCP gateway
2. Add operation-specific attributes (provisioning events)
3. Instrument ARQ worker jobs for async operations
4. Track Fly.io API calls and latency
### Phase 4: Instrument API Service
1. Extract tenant context from JWT
2. Add machine-specific metadata (instance ID, region)
3. Instrument database operations with SQLAlchemy
4. Track MCP protocol operations
### Phase 5: Configure and Deploy
1. Add OTLP configuration to `.env.example` and `.env.example.secrets`
2. Set Fly.io secrets for production deployment
3. Update Dockerfiles to use `opentelemetry-instrument` wrapper
4. Deploy to development environment first for testing
## How to Evaluate
### Success Criteria
1. **End-to-end traces visible in Grafana Cloud** showing complete request flow
2. **Tenant filtering works** - Can filter traces by tenant_id to see all requests for a user
3. **Service maps accurate** - Grafana shows correct service dependencies
4. **Performance overhead < 5%** - Minimal latency impact from instrumentation
5. **Error correlation** - Can trace errors back to specific tenant and operation
### Testing Checklist
- [x] Single request creates connected trace across all services
- [x] Tenant attributes present on all spans
- [x] Background jobs (ARQ) appear in traces
- [x] Database queries show in trace timeline
- [x] HTTP calls to Fly.io API tracked
- [x] Traces exported successfully to Grafana Cloud
- [x] Can search traces by tenant_id in Grafana
- [x] Service dependency graph shows correct flow
### Monitoring Success
- All services reporting traces to Grafana Cloud
- No OTLP export errors in logs
- Trace sampling working correctly (if implemented)
- Resource usage acceptable (CPU/memory)
## Dependencies
- Grafana Cloud account with OTLP endpoint configured
- OpenTelemetry Python SDK v1.29.0+
- FastAPI instrumentation compatibility
- Network access from Fly.io to Grafana Cloud
## Implementation Assignment
**Recommended Agent**: python-developer
- Requires Python/FastAPI expertise
- Needs understanding of distributed systems
- Must implement middleware and context propagation
- Should understand OpenTelemetry SDK and instrumentation
## Follow-up Tasks
### Enhanced Log Correlation
While basic trace-to-log correlation works automatically via OpenTelemetry logging instrumentation, consider adding structured logging for improved log filtering:
1. **Structured Logging Context**: Add `logger.bind()` calls to inject tenant/user context directly into log records
2. **Custom Loguru Formatter**: Extract OpenTelemetry span attributes for better log readability
3. **Direct Log Filtering**: Enable searching logs directly by tenant_id, workflow_id without going through traces
This would complement the existing automatic trace correlation and provide better log search capabilities.
## Alternative Solution: Logfire
After implementing OpenTelemetry with Grafana Cloud, we discovered limitations in the observability experience:
- Traces work but lack useful context without correlated logs
- Setting up log correlation with Grafana is complex and requires additional infrastructure
- The developer experience for Python observability is suboptimal
### Logfire Evaluation
**Pydantic Logfire** offers a compelling alternative that addresses your specific requirements:
#### Core Requirements Match
- ✅ **User Activity Tracking**: Automatic request tracing with business context
- ✅ **Error Monitoring**: Built-in exception tracking with full context
- ✅ **Performance Metrics**: Automatic latency and performance monitoring
- ✅ **Request Tracing**: Native distributed tracing across services
- ✅ **Log Correlation**: Seamless trace-to-log correlation without setup
#### Key Advantages
1. **Python-First Design**: Built specifically for Python/FastAPI applications by the Pydantic team
2. **Simple Integration**: `pip install logfire` + `logfire.configure()` vs complex OTLP setup
3. **Automatic Correlation**: Logs automatically include trace context without manual configuration
4. **Real-time SQL Interface**: Query spans and logs using SQL with auto-completion
5. **Better Developer UX**: Purpose-built observability UI vs generic Grafana dashboards
6. **Loguru Integration**: `logger.configure(handlers=[logfire.loguru_handler()])` maintains existing logging
#### Pricing Assessment
- **Free Tier**: 10M spans/month (suitable for development and small production workloads)
- **Transparent Pricing**: $1 per million spans/metrics after free tier
- **No Hidden Costs**: No per-host fees, only usage-based metering
- **Production Ready**: Recently exited beta, enterprise features available
#### Migration Path
The existing OpenTelemetry instrumentation is compatible - Logfire uses OpenTelemetry under the hood, so the current spans and attributes would work unchanged.
### Recommendation
**Consider migrating to Logfire** for the following reasons:
1. It directly addresses the "next to useless" traces problem by providing integrated logs
2. Dramatically simpler setup and maintenance compared to Grafana Cloud + custom log correlation
3. Better ROI on observability investment with purpose-built Python tooling
4. Free tier sufficient for current development needs with clear scaling path
The current Grafana Cloud implementation provides a solid foundation and could remain as a backup/export target, while Logfire becomes the primary observability platform.
## Status
**Created**: 2024-01-28
**Status**: Completed (OpenTelemetry + Grafana Cloud)
**Next Phase**: Evaluate Logfire migration
**Priority**: High - Critical for production observability
@@ -340,398 +340,6 @@ This architecture makes the fix comprehensive and maintainable.
- Want to reduce database dependency
- Scale requires fewer database queries
## Post-Deployment Test Plan
This test plan should be executed after deploying the cloud service to verify subscription validation works end-to-end.
### Prerequisites
Before testing, ensure you have:
- [ ] Cloud service deployed with Phase 1 changes
- [ ] CLI installed with Phase 2 changes (`basic-memory` from local dev)
- [ ] Access to database to check/modify subscription status
- [ ] Two test user accounts:
- User A: No subscription (fresh WorkOS signup)
- User B: Active subscription (via Polar or manual DB insert)
### Test Execution
#### Test 1: User Without Subscription (Blocked Access) ❌
**Setup:**
1. Create fresh WorkOS account (User A) via AuthKit
2. Verify in database: No subscription record exists for User A's `workos_user_id`
**Test Steps:**
```bash
# Step 1: Attempt login
bm cloud login
```
**Expected Results:**
- ✅ OAuth flow completes successfully
- ✅ JWT token obtained and stored in `~/.basic-memory/auth/token`
- ❌ Login fails with "Subscription Required" error
- ✅ Error message displays:
- "✗ Subscription Required"
- "Active subscription required for CLI access"
- Subscribe URL: "https://basicmemory.com/subscribe"
- Instructions to run `bm cloud login` after subscribing
- ❌ Cloud mode NOT enabled (check with `bm cloud status`)
**Test Steps (continued):**
```bash
# Step 2: Attempt to access cloud features
bm cloud status
# Step 3: Try direct API call
curl -H "Authorization: Bearer <token>" https://<cloud-host>/proxy/health
```
**Expected Results:**
- ✅ `bm cloud status` shows "Mode: Local (disabled)"
- ✅ Direct API call returns 403 with subscription_required error
**Database Verification:**
```sql
-- Verify no subscription exists
SELECT * FROM subscriptions
WHERE workos_user_id = '<user-a-workos-id>';
-- Should return 0 rows
```
---
#### Test 2: User With Active Subscription (Full Access) ✅
**Setup:**
1. Use User B with active subscription
2. Verify in database: Subscription exists with `status = 'active'` and `current_period_end > NOW()`
**Database Verification:**
```sql
-- Verify active subscription exists
SELECT workos_user_id, status, current_period_end
FROM subscriptions
WHERE workos_user_id = '<user-b-workos-id>';
-- Should show: status='active', current_period_end in future
```
**Test Steps:**
```bash
# Step 1: Login
bm cloud login
# Step 2: Check cloud mode
bm cloud status
# Step 3: Setup bisync
bm cloud setup
# Step 4: Test MCP tools via proxy
curl -H "Authorization: Bearer <token>" \
https://<cloud-host>/proxy/<project-name>/health
# Step 5: List projects
bm project list
# Step 6: Create a test note
bm tool write-note \
--title "Test Note" \
--folder "test-project" \
--content "Testing subscription validation"
```
**Expected Results:**
- ✅ Login succeeds without errors
- ✅ Cloud mode enabled: "Mode: Cloud (enabled)"
- ✅ Cloud instance health check succeeds
- ✅ Bisync setup completes successfully
- ✅ Direct API calls succeed (200 OK)
- ✅ Projects list successfully
- ✅ Note creation succeeds
---
#### Test 3: Subscription Expiration (Access Revoked) 🔄
**Setup:**
1. Use User B (currently has active subscription and cloud mode enabled)
2. User should be able to access cloud features initially
**Test Steps:**
```bash
# Step 1: Verify current access works
bm cloud status
# Should show "Cloud (enabled)" and healthy instance
# Step 2: Expire subscription in database
# (See SQL below)
# Step 3: Attempt to access cloud features
bm cloud status
# Step 4: Try to login again
bm cloud logout
bm cloud login
```
**Database Operations:**
```sql
-- Expire the subscription
UPDATE subscriptions
SET status = 'cancelled',
current_period_end = NOW() - INTERVAL '1 day'
WHERE workos_user_id = '<user-b-workos-id>';
-- Verify expiration
SELECT workos_user_id, status, current_period_end
FROM subscriptions
WHERE workos_user_id = '<user-b-workos-id>';
-- Should show: status='cancelled', current_period_end in past
```
**Expected Results:**
- ❌ `bm cloud status` fails with 403 subscription_required error
- ❌ Re-login fails with "Subscription Required" error
- ✅ Error includes subscribe URL
---
#### Test 4: Subscription Renewal (Access Restored) ✅
**Setup:**
1. Continue from Test 3 (User B with expired subscription)
**Test Steps:**
```bash
# Step 1: Renew subscription in database
# (See SQL below)
# Step 2: Login again
bm cloud login
# Step 3: Verify access restored
bm cloud status
# Step 4: Test project access
bm project list
```
**Database Operations:**
```sql
-- Renew the subscription
UPDATE subscriptions
SET status = 'active',
current_period_end = NOW() + INTERVAL '30 days'
WHERE workos_user_id = '<user-b-workos-id>';
-- Verify renewal
SELECT workos_user_id, status, current_period_end
FROM subscriptions
WHERE workos_user_id = '<user-b-workos-id>';
-- Should show: status='active', current_period_end 30 days in future
```
**Expected Results:**
- ✅ Login succeeds
- ✅ Cloud mode enabled
- ✅ Cloud status shows healthy
- ✅ Projects list successfully
- ✅ **Access immediately restored** (no delay)
---
#### Test 5: Endpoint Coverage (All Protected Endpoints) 🔐
**Setup:**
1. Use User A (no subscription) to test blocked access
2. Use User B (active subscription) to test allowed access
**Test Matrix:**
| Endpoint | Method | User A (No Sub) | User B (Active Sub) |
|----------|--------|----------------|---------------------|
| `/proxy/health` | GET | 403 ❌ | 200 ✅ |
| `/proxy/<project>/health` | GET | 403 ❌ | 200 ✅ |
| `/proxy/<project>/search` | POST | 403 ❌ | 200 ✅ |
| `/tenant/mount/info` | GET | 403 ❌ | 200 ✅ |
| `/tenant/mount/credentials` | POST | 403 ❌ | 200 ✅ |
**Test Commands:**
```bash
# Get tokens for both users
TOKEN_A="<user-a-token>"
TOKEN_B="<user-b-token>"
# Test /proxy/health
curl -H "Authorization: Bearer $TOKEN_A" \
https://<cloud-host>/proxy/health
# Expected: 403 with subscription_required
curl -H "Authorization: Bearer $TOKEN_B" \
https://<cloud-host>/proxy/health
# Expected: 200 OK
# Test /tenant/mount/info
curl -H "Authorization: Bearer $TOKEN_A" \
https://<cloud-host>/tenant/mount/info
# Expected: 403 with subscription_required
curl -H "Authorization: Bearer $TOKEN_B" \
https://<cloud-host>/tenant/mount/info
# Expected: 200 OK with mount info
# Test /proxy/<project>/health
curl -H "Authorization: Bearer $TOKEN_B" \
https://<cloud-host>/proxy/<project-name>/health
# Expected: 200 OK
```
---
#### Test 6: Error Response Format Validation 📋
**Test Steps:**
```bash
# Get 403 response for user without subscription
curl -i -H "Authorization: Bearer $TOKEN_A" \
https://<cloud-host>/proxy/health
```
**Expected Response Format:**
```http
HTTP/1.1 403 Forbidden
Content-Type: application/json
{
"error": "subscription_required",
"message": "Active subscription required for CLI access",
"subscribe_url": "https://basicmemory.com/subscribe"
}
```
**Validation Checklist:**
- ✅ Status code is exactly 403
- ✅ Response is valid JSON
- ✅ `error` field equals "subscription_required"
- ✅ `message` field is present and informative
- ✅ `subscribe_url` field is present and valid URL
---
#### Test 7: Admin Access Bypass 👑
**Purpose:** Verify admin users can still access admin endpoints without subscription
**Setup:**
1. Use admin user account (member of admin organization in WorkOS)
**Test Steps:**
```bash
# Login as admin
python -m basic_memory_cloud.cli.tenant_cli login
# List tenants (admin-only endpoint)
python -m basic_memory_cloud.cli.tenant_cli list-tenants
# Create tenant (admin-only endpoint)
python -m basic_memory_cloud.cli.tenant_cli create-tenant \
--workos-user-id <test-user-id>
```
**Expected Results:**
- ✅ Admin login succeeds
- ✅ Admin can access `/tenants/*` endpoints
- ✅ Admin operations work regardless of subscription status
- ✅ Admin endpoints use `AdminUserHybridDep` (not affected by subscription check)
---
### Test Results Template
Copy this template to track your test execution:
```markdown
## SPEC-13 Test Execution - [Date]
### Environment
- Cloud Service: [URL]
- Cloud Service Version: [commit/tag]
- CLI Version: [commit/tag]
- Database: [production/staging]
### Test Results
#### Test 1: User Without Subscription ❌
- [ ] OAuth flow succeeds
- [ ] Subscription error displayed
- [ ] Subscribe URL shown
- [ ] Cloud mode NOT enabled
- [ ] Direct API call blocked
**Issues:** [None / List issues]
#### Test 2: User With Active Subscription ✅
- [ ] Login succeeds
- [ ] Cloud mode enabled
- [ ] Health check passes
- [ ] Bisync setup works
- [ ] MCP tools work
- [ ] Projects accessible
**Issues:** [None / List issues]
#### Test 3: Subscription Expiration 🔄
- [ ] Active user can access initially
- [ ] After expiration, access blocked
- [ ] Error message clear
- [ ] Cloud status fails appropriately
**Issues:** [None / List issues]
#### Test 4: Subscription Renewal ✅
- [ ] Renewed subscription in DB
- [ ] Login succeeds immediately
- [ ] Access fully restored
- [ ] No caching delays
**Issues:** [None / List issues]
#### Test 5: Endpoint Coverage 🔐
- [ ] All proxy endpoints protected
- [ ] All mount endpoints protected
- [ ] Subscription check consistent
- [ ] Error responses correct
**Issues:** [None / List issues]
#### Test 6: Error Response Format 📋
- [ ] 403 status code
- [ ] Valid JSON response
- [ ] All required fields present
- [ ] Subscribe URL valid
**Issues:** [None / List issues]
#### Test 7: Admin Access Bypass 👑
- [ ] Admin login works
- [ ] Admin endpoints accessible
- [ ] No subscription requirement
**Issues:** [None / List issues]
### Overall Result
- [ ] All tests passed
- [ ] Ready for production
**Summary:** [Brief summary of test execution]
**Sign-off:** [Your name/date]
```
---
## How to Evaluate
### Success Criteria
@@ -1065,35 +673,35 @@ The extra HTTP hop is minimal (< 10ms) and worth it for architectural benefits.
- [ ] Call `/tenant/mount/info` with valid JWT and active subscription → expect 200
- [ ] Verify error response structure matches spec
### Phase 2: CLI (basic-memory)
### Phase 2: CLI (basic-memory)
#### Task 2.1: Review and understand CLI authentication flow
#### Task 2.1: Review and understand CLI authentication flow
**Files**: `src/basic_memory/cli/commands/cloud/`
- [x] Read `core_commands.py` to understand current login flow
- [x] Read `api_client.py` to understand current error handling
- [x] Identify where 403 errors should be caught
- [x] Identify what error messages should be displayed
- [x] Document current behavior in spec if needed
- [ ] Read `core_commands.py` to understand current login flow
- [ ] Read `api_client.py` to understand current error handling
- [ ] Identify where 403 errors should be caught
- [ ] Identify what error messages should be displayed
- [ ] Document current behavior in spec if needed
#### Task 2.2: Update API client error handling
#### Task 2.2: Update API client error handling
**File**: `src/basic_memory/cli/commands/cloud/api_client.py`
- [x] Add custom exception class `SubscriptionRequiredError` (or similar)
- [x] Update HTTP error handling to parse 403 responses
- [x] Extract `error`, `message`, and `subscribe_url` from error detail
- [x] Raise specific exception for subscription_required errors
- [x] Run `just typecheck` in basic-memory repo to verify types
- [ ] Add custom exception class `SubscriptionRequiredError` (or similar)
- [ ] Update HTTP error handling to parse 403 responses
- [ ] Extract `error`, `message`, and `subscribe_url` from error detail
- [ ] Raise specific exception for subscription_required errors
- [ ] Run `just typecheck` in basic-memory repo to verify types
#### Task 2.3: Update CLI login command error handling
#### Task 2.3: Update CLI login command error handling
**File**: `src/basic_memory/cli/commands/cloud/core_commands.py`
- [x] Import the subscription error exception
- [x] Wrap login flow with try/except for subscription errors
- [x] Display user-friendly error message with rich console
- [x] Show subscribe URL prominently
- [x] Provide actionable next steps
- [x] Run `just typecheck` to verify types
- [ ] Import the subscription error exception
- [ ] Wrap login flow with try/except for subscription errors
- [ ] Display user-friendly error message with rich console
- [ ] Show subscribe URL prominently
- [ ] Provide actionable next steps
- [ ] Run `just typecheck` to verify types
**Expected error handling**:
```python
@@ -1111,33 +719,30 @@ except SubscriptionRequiredError as e:
raise typer.Exit(1)
```
#### Task 2.4: Update CLI tests
**File**: `tests/cli/test_cloud_authentication.py` (created)
#### Task 2.4: Update CLI tests
**File**: `tests/cli/test_cloud_commands.py`
- [x] Add test: `test_login_without_subscription_shows_error()`
- [ ] Add test: `test_login_without_subscription_shows_error()`
- Mock 403 subscription_required response
- Call login command
- Assert error message displayed
- Assert subscribe URL shown
- [x] Add test: `test_login_with_subscription_succeeds()`
- [ ] Add test: `test_login_with_subscription_succeeds()`
- Mock successful authentication + subscription check
- Call login command
- Assert success message
- [x] Add test: `test_parse_subscription_required_error()` (API client error parsing)
- [x] Add test: `test_parse_generic_403_error()` (generic 403 handling)
- [x] Add test: `test_login_authentication_failure()` (auth failure handling)
- [x] Run `uv run pytest` to verify tests pass (5/5 passed)
- [ ] Run `just test` to verify tests pass
#### Task 2.5: Update CLI documentation
**File**: `docs/cloud-cli.md`
#### Task 2.5: Update CLI documentation
**File**: `docs/cloud-cli.md` (in basic-memory-docs repo)
- [x] Add "Prerequisites" section if not present
- [x] Document subscription requirement
- [x] Add "Troubleshooting" section
- [x] Document "Subscription Required" error
- [x] Provide subscribe URL
- [x] Add FAQ entry about subscription errors
- [x] Build docs locally to verify formatting
- [ ] Add "Prerequisites" section if not present
- [ ] Document subscription requirement
- [ ] Add "Troubleshooting" section
- [ ] Document "Subscription Required" error
- [ ] Provide subscribe URL
- [ ] Add FAQ entry about subscription errors
- [ ] Build docs locally to verify formatting
### Phase 3: End-to-End Testing
@@ -1227,12 +832,12 @@ Use this high-level checklist to track overall progress:
- [ ] Add integration tests for dependency
- [ ] Deploy and verify cloud service
### Phase 2: CLI Updates
- [x] Review CLI authentication flow
- [x] Update API client error handling
- [x] Update CLI login command error handling
- [x] Add CLI tests
- [x] Update CLI documentation
### Phase 2: CLI Updates 🔄
- [ ] Review CLI authentication flow
- [ ] Update API client error handling
- [ ] Update CLI login command error handling
- [ ] Add CLI tests
- [ ] Update CLI documentation
### Phase 3: End-to-End Testing 🧪
- [ ] Create test user accounts
@@ -1310,118 +915,3 @@ Use this high-level checklist to track overall progress:
- This spec prioritizes security over convenience - better to block unauthorized access than risk revenue loss
- Clear error messages are critical - users should understand why they're blocked and how to resolve it
- Consider adding telemetry to track subscription_required errors for monitoring signup conversion
## Implementation Log
### Phase 2 Completion - 2025-10-03
Phase 2 (CLI Updates) completed successfully with the following implementation:
**Files Modified:**
- `src/basic_memory/cli/commands/cloud/api_client.py` - Added `SubscriptionRequiredError` exception and enhanced error handling
- `src/basic_memory/cli/commands/cloud/core_commands.py` - Updated login command to verify subscription access
- `docs/cloud-cli.md` - Added Prerequisites and Subscription Issues sections
**Files Created:**
- `tests/cli/test_cloud_authentication.py` - Comprehensive test coverage (6 tests, all passing)
**Key Implementation Details:**
- `SubscriptionRequiredError` exception with `subscribe_url` field for user guidance
- Enhanced `CloudAPIError` to include `status_code` and `detail` fields
- Login flow now calls `/proxy/health` to verify subscription before enabling cloud mode
- User-friendly error messages with direct subscribe link
- 100% test coverage of new error handling paths
**Test Results:**
- All 6 tests passing
- Type checking: 0 errors, 0 warnings
- Linting: All checks passed
**Next Steps:**
- Phase 3: End-to-End Testing (manual testing with real users, subscription state transitions)
- Phase 1: Complete remaining cloud service tests (unit tests, integration tests, deployment verification)
---
### End-to-End Test Execution - 2025-10-03
**Environment:**
- Cloud Service: https://cloud.basicmemory.com
- Cloud Service Version: Phase 1 deployed (with subscription validation)
- CLI Version: Phase 2 implementation (local dev build)
- Database: Production
**Test Results:**
#### Test 1: User Without Subscription ✅ PASSED
- [x] OAuth flow succeeds
- [x] Subscription error displayed
- [x] Subscribe URL shown
- [x] Cloud mode NOT enabled
- [x] Clean error output (no traceback)
**Output:**
```
✅ Successfully authenticated with WorkOS!
Verifying subscription access...
✗ Subscription Required
Active subscription required
Subscribe at: https://basicmemory.com/subscribe
Once you have an active subscription, run bm cloud login again.
```
**Issues:** None
---
#### Test 2: User With Active Subscription ✅ PASSED
- [x] Login succeeds
- [x] Cloud mode enabled
- [x] Clean success message
- [x] Ready for cloud operations
**Output:**
```
✅ Successfully authenticated with WorkOS!
Verifying subscription access...
✓ Cloud mode enabled
All CLI commands now work against https://cloud.basicmemory.com
```
**Issues:** None
---
**Additional Implementation Notes:**
**API Response Format Compatibility:**
- Cloud service returns errors in FastAPI HTTPException format (nested under `"detail"` key)
- CLI correctly handles both nested and flat response formats
- Error parsing logic:
```python
detail_obj = error_detail.get("detail", error_detail)
if isinstance(detail_obj, dict) and detail_obj.get("error") == "subscription_required":
# Handle subscription error
```
**Updated Test Coverage:**
- Added `test_parse_subscription_required_error_flat_format()` for backward compatibility
- Total: 6 tests, all passing
- Files updated:
- `src/basic_memory/cli/commands/cloud/api_client.py` - Support both response formats
- `tests/cli/test_cloud_authentication.py` - Added flat format test
**Overall Result:**
- [x] Core authentication flows validated
- [x] Error handling working as designed
- [x] User experience is clean and helpful
- [x] Ready for production use
**Summary:**
SPEC-13 Phase 2 successfully validated in production environment. Both unauthorized and authorized user flows work correctly. The subscription validation is functioning end-to-end with clear, user-friendly error messages and seamless success path. No issues discovered during testing.
**Sign-off:** Phase 2 Complete - 2025-10-03
@@ -0,0 +1,210 @@
---
title: 'SPEC-14: Cloud Git Versioning & GitHub Backup'
type: spec
permalink: specs/spec-14-cloud-git-versioning
tags:
- git
- github
- backup
- versioning
- cloud
related:
- specs/spec-9-multi-project-bisync
- specs/spec-9-follow-ups-conflict-sync-and-observability
status: deferred
---
# SPEC-14: Cloud Git Versioning & GitHub Backup
**Status: DEFERRED** - Postponed until multi-user/teams feature development. Using S3 versioning (SPEC-9.1) for v1 instead.
## Why Deferred
**Original goals can be met with simpler solutions:**
- Version history → **S3 bucket versioning** (automatic, zero config)
- Offsite backup → **Tigris global replication** (built-in)
- Restore capability → **S3 version restore** (`bm cloud restore --version-id`)
- Collaboration → **Deferred to teams/multi-user feature** (not v1 requirement)
**Complexity vs value trade-off:**
- Git integration adds: committer service, puller service, webhooks, LFS, merge conflicts
- Risk: Loop detection between Git ↔ rclone bisync ↔ local edits
- S3 versioning gives 80% of value with 5% of complexity
**When to revisit:**
- Teams/multi-user features (PR-based collaboration workflow)
- User requests for commit messages and branch-based workflows
- Need for fine-grained audit trail beyond S3 object metadata
---
## Original Specification (for reference)
## Why
Early access users want **transparent version history**, easy **offsite backup**, and a familiar **restore/branching** workflow. Git/GitHub integration would provide:
- Auditable history of every change (who/when/why)
- Branches/PRs for review and collaboration
- Offsite private backup under the user's control
- Escape hatch: users can always `git clone` their knowledge base
**Note:** These goals are now addressed via S3 versioning (SPEC-9.1) for single-user use case.
## Goals
- **Transparent**: Users keep using Basic Memory; Git runs behind the scenes.
- **Private**: Push to a **private GitHub repo** that the user owns (or tenant org).
- **Reliable**: No data loss, deterministic mapping of filesystem ↔ Git.
- **Composable**: Plays nicely with SPEC9 bisync and upcoming conflict features (SPEC9 FollowUps).
**NonGoals (for v1):**
- Finegrained perfile encryption in Git history (can be layered later).
- Large media optimization beyond Git LFS defaults.
## User Stories
1. *As a user*, I connect my GitHub and choose a private backup repo.
2. *As a user*, every change I make in cloud (or via bisync) is **committed** and **pushed** automatically.
3. *As a user*, I can **restore** a file/folder/project to a prior version.
4. *As a power user*, I can **git pull/push** directly to collaborate outside the app.
5. *As an admin*, I can enforce repo ownership (tenant org) and leastprivilege scopes.
## Scope
- **In scope:** Full repo backup of `/app/data/` (all projects) with optional selective subpaths.
- **Out of scope (v1):** Partial shallow mirrors; encrypted Git; crossprovider SCM (GitLab/Bitbucket).
## Architecture
### Topology
- **Authoritative working tree**: `/app/data/` (bucket mount) remains the source of truth (SPEC9).
- **Bare repo** lives alongside: `/app/git/${tenant}/knowledge.git` (serverside).
- **Mirror remote**: `github.com/<owner>/<repo>.git` (private).
```mermaid
flowchart LR
A[/Users & Agents/] -->|writes/edits| B[/app/data/]
B -->|file events| C[Committer Service]
C -->|git commit| D[(Bare Repo)]
D -->|push| E[(GitHub Private Repo)]
E -->|webhook (push)| F[Puller Service]
F -->|git pull/merge| D
D -->|checkout/merge| B
```
### Services
- **Committer Service** (daemon):
- Watches `/app/data/` for changes (inotify/poll)
- Batches changes (debounce e.g. 25s)
- Writes `.bmmeta` (if present) into commit message trailer (see FollowUps)
- `git add -A && git commit -m "chore(sync): <summary>
BM-Meta: <json>"`
- Periodic `git push` to GitHub mirror (configurable interval)
- **Puller Service** (webhook target):
- Receives GitHub webhook (push) → `git fetch`
- **Fastforward** merges to `main` only; reject nonFF unless policy allows
- Applies changes back to `/app/data/` via clean checkout
- Emits sync events for Basic Memory indexers
### Auth & Security
- **GitHub App** (recommended): minimal scopes: `contents:read/write`, `metadata:read`, webhook.
- Tenantscoped installation; repo created in user account or tenant org.
- Tokens stored in KMS/secret manager; rotated automatically.
- Optional policy: allow only **FF merges** on `main`; nonFF requires PR.
### Repo Layout
- **Monorepo** (default): one repo per tenant mirrors `/app/data/` with subfolders per project.
- Optional multirepo mode (later): one repo per project.
### File Handling
- Honor `.gitignore` generated from `.bmignore.rclone` + BM defaults (cache, temp, state).
- **Git LFS** for large binaries (images, media) — auto track by extension/size threshold.
- Normalize newline + Unicode (aligns with FollowUps).
### Conflict Model
- **Primary concurrency**: SPEC9 FollowUps (`.bmmeta`, conflict copies) stays the first line of defense.
- **Git merges** are a **secondary** mechanism:
- Server only automerges **text** conflicts when trivial (FF or clean 3way).
- Otherwise, create `name (conflict from <branch>, <ts>).md` and surface via events.
### Data Flow vs Bisync
- Bisync (rclone) continues between local sync dir ↔ bucket.
- Git sits **cloudside** between bucket and GitHub.
- On **pull** from GitHub → files written to `/app/data/` → picked up by indexers & eventually by bisync back to users.
## CLI & UX
New commands (cloud mode):
- `bm cloud git connect` — Launch GitHub App installation; create private repo; store installation id.
- `bm cloud git status` — Show connected repo, last push time, last webhook delivery, pending commits.
- `bm cloud git push` — Manual push (rarely needed).
- `bm cloud git pull` — Manual pull/FF (admin only by default).
- `bm cloud snapshot -m "message"` — Create a tagged pointintime snapshot (git tag).
- `bm restore <path> --to <commit|tag>` — Restore file/folder/project to prior version.
Settings:
- `bm config set git.autoPushInterval=5s`
- `bm config set git.lfs.sizeThreshold=10MB`
- `bm config set git.allowNonFF=false`
## Migration & Backfill
- On connect, if repo empty: initial commit of entire `/app/data/`.
- If repo has content: require **onetime import** path (clone to staging, reconcile, choose direction).
## Edge Cases
- Massive deletes: gated by SPEC9 `max_delete` **and** Git prepush hook checks.
- Case changes and rename detection: rely on git rename heuristics + FollowUps move hints.
- Secrets: default ignore common secret patterns; allow custom deny list.
## Telemetry & Observability
- Emit `git_commit`, `git_push`, `git_pull`, `git_conflict` events with correlation IDs.
- `bm sync --report` extended with Git stats (commit count, delta bytes, push latency).
## Phased Plan
### Phase 0 — Prototype (1 sprint)
- Server: bare repo init + simple committer (batch every 10s) + manual GitHub token.
- CLI: `bm cloud git connect --token <PAT>` (devonly)
- Success: edits in `/app/data/` appear in GitHub within 30s.
### Phase 1 — GitHub App & Webhooks (12 sprints)
- Switch to GitHub App installs; create private repo; store installation id.
- Committer hardened (debounce 25s, backoff, retries).
- Puller service with webhook → FF merge → checkout to `/app/data/`.
- LFS autotrack + `.gitignore` generation.
- CLI surfaces status + logs.
### Phase 2 — Restore & Snapshots (1 sprint)
- `bm restore` for file/folder/project with dryrun.
- `bm cloud snapshot` tags + list/inspect.
- Policy: PRonly nonFF, admin override.
### Phase 3 — Selective & MultiRepo (nicetohave)
- Include/exclude projects; optional perproject repos.
- Advanced policies (branch protections, required reviews).
## Acceptance Criteria
- Changes to `/app/data/` are committed and pushed automatically within configurable interval (default ≤5s).
- GitHub webhook pull results in updated files in `/app/data/` (FFonly by default).
- LFS configured and functioning; large files don't bloat history.
- `bm cloud git status` shows connected repo and last push/pull times.
- `bm restore` restores a file/folder to a prior commit with a clear audit trail.
- Endtoend works alongside SPEC9 bisync without loops or data loss.
## Risks & Mitigations
- **Loop risk (Git ↔ Bisync)**: Writes to `/app/data/` → bisync → local → user edits → back again. *Mitigation*: Debounce, commit squashing, idempotent `.bmmeta` versioning, and watch exclusion windows during pull.
- **Repo bloat**: Lots of binary churn. *Mitigation*: default LFS, size threshold, optional mediaonly repo later.
- **Security**: Token leakage. *Mitigation*: GitHub App with shortlived tokens, KMS storage, scoped permissions.
- **Merge complexity**: Nontrivial conflicts. *Mitigation*: prefer FF; otherwise conflict copies + events; require PR for nonFF.
## Open Questions
- Do we default to **monorepo** per tenant, or offer projectperrepo at connect time?
- Should `restore` write to a branch and open a PR, or directly modify `main`?
- How do we expose Git history in UI (timeline view) without users dropping to CLI?
## Appendix: Sample Config
```json
{
"git": {
"enabled": true,
"repo": "https://github.com/<owner>/<repo>.git",
"autoPushInterval": "5s",
"allowNonFF": false,
"lfs": { "sizeThreshold": 10485760 }
}
}
```
@@ -41,16 +41,16 @@ Store Basic Memory configuration in the Tigris bucket and rebuild the database i
**Architecture:**
```bash
# Tigris Bucket (persistent, mounted at /mnt/tigris)
/mnt/tigris/
# Tigris Bucket (persistent, mounted at /app/data)
/app/data/
├── .basic-memory/
│ └── config.json # ← Project configuration (persistent, accessed via BASIC_MEMORY_CONFIG_DIR)
└── projects/ # ← Markdown files (persistent)
└── basic-memory/ # ← Markdown files (persistent, BASIC_MEMORY_HOME)
├── project1/
└── project2/
# Fly Machine (ephemeral)
~/.basic-memory/
/app/.basic-memory/
└── memory.db # ← Rebuilt on startup (fast local disk)
```
@@ -107,8 +107,8 @@ async def startup_sync():
```bash
# Machine environment variables
BASIC_MEMORY_CONFIG_DIR=/mnt/tigris/.basic-memory # Config read/written directly to Tigris
# memory.db stays in default location: ~/.basic-memory/memory.db (local ephemeral disk)
BASIC_MEMORY_CONFIG_DIR=/app/data/.basic-memory # Config read/written directly to Tigris
# memory.db stays in default location: /app/.basic-memory/memory.db (local ephemeral disk)
```
## Implementation Task List
@@ -118,20 +118,26 @@ BASIC_MEMORY_CONFIG_DIR=/mnt/tigris/.basic-memory # Config read/written directl
- [x] Test config loading from custom directory
- [x] Update tests to verify custom config dir works
### Phase 2: Tigris Bucket Structure
- [ ] Ensure `.basic-memory/` directory exists in Tigris bucket on tenant creation
- [ ] Initialize `config.json` in Tigris on first tenant deployment
- [ ] Verify TigrisFS handles hidden directories correctly
### Phase 2: Tigris Bucket Structure
- [x] Ensure `.basic-memory/` directory exists in Tigris bucket on tenant creation
- ✅ ConfigManager auto-creates on first run, no explicit provisioning needed
- [x] Initialize `config.json` in Tigris on first tenant deployment
- ✅ ConfigManager creates config.json automatically in BASIC_MEMORY_CONFIG_DIR
- [x] Verify TigrisFS handles hidden directories correctly
- ✅ TigrisFS supports hidden directories (verified in SPEC-8)
### Phase 3: Deployment Integration
- [ ] Set `BASIC_MEMORY_CONFIG_DIR` environment variable in machine deployment
- [ ] Ensure database rebuild runs on machine startup via initialization sync
- [ ] Handle first-time tenant setup (no config exists yet)
### Phase 3: Deployment Integration
- [x] Set `BASIC_MEMORY_CONFIG_DIR` environment variable in machine deployment
- ✅ Added to BasicMemoryMachineConfigBuilder in fly_schemas.py
- [x] Ensure database rebuild runs on machine startup via initialization sync
- ✅ sync_worker.py runs initialize_file_sync every 30s (already implemented)
- [x] Handle first-time tenant setup (no config exists yet)
- ✅ ConfigManager creates config.json on first initialization
- [ ] Test deployment workflow with config persistence
### Phase 4: Testing
- [x] Unit tests for config directory override
- [ ] Integration test: deploy → write config → redeploy → verify config persists
- [-] Integration test: deploy → write config → redeploy → verify config persists
- [ ] Integration test: deploy → add project → redeploy → verify project in config
- [ ] Performance test: measure db rebuild time on startup
@@ -175,7 +181,7 @@ BASIC_MEMORY_CONFIG_DIR=/mnt/tigris/.basic-memory # Config read/written directl
basic-memory project add "test-project" ~/test
# Verify config has project
cat /mnt/tigris/.basic-memory/config.json
cat /app/data/.basic-memory/config.json
# Redeploy machine
fly deploy --app basic-memory-{tenant_id}
@@ -261,4 +267,7 @@ BASIC_MEMORY_CONFIG_DIR=/mnt/tigris/.basic-memory # Config read/written directl
- 2025-10-08: Pivoted from Turso to Tigris-based config persistence
- 2025-10-08: Phase 1 complete - BASIC_MEMORY_CONFIG_DIR support added (PR #343)
- Next: Implement Phases 2-3 in basic-memory-cloud repository
- 2025-10-08: Phases 2-3 complete - Added BASIC_MEMORY_CONFIG_DIR to machine config
- Config now persists to /app/data/.basic-memory/config.json in Tigris bucket
- Database rebuild already working via sync_worker.py
- Ready for deployment testing (Phase 4)
+163 -82
View File
@@ -8,9 +8,60 @@ tags:
- cloud
- performance
- deployment
status: draft
status: in-progress
---
## Status Update
**Phase 0 (Basic Memory Refactor): ✅ COMPLETE**
- basic-memory PR #344: async_client context manager pattern implemented
- All 17 MCP tools updated to use `async with get_client() as client:`
- CLI commands updated to use context manager
- Removed `inject_auth_header()` and `headers.py` (~100 lines deleted)
- Factory pattern enables clean dependency injection
- Tests passing, typecheck clean
**Phase 0 Integration: ✅ COMPLETE**
- basic-memory-cloud updated to use async-client-context-manager branch
- Implemented `tenant_direct_client_factory()` with proper context manager pattern
- Removed module-level client override hacks
- Removed unnecessary `/proxy` prefix stripping (tools pass relative URLs)
- Typecheck and lint passing with proper noqa hints
- MCP tools confirmed working via inspector (local testing)
**Phase 1 (Code Consolidation): ✅ COMPLETE**
- MCP server mounted on Cloud FastAPI app at /mcp endpoint
- AuthKitProvider configured with WorkOS settings
- Combined lifespans (Cloud + MCP) working correctly
- JWT context middleware integrated
- All routes and MCP tools functional
**Phase 2 (Direct Tenant Transport): ✅ COMPLETE**
- TenantDirectTransport implemented with custom httpx transport
- Per-request JWT extraction via FastMCP DI
- Tenant lookup and signed header generation working
- Direct routing to tenant APIs (eliminating HTTP hop)
- Transport tests passing (11/11)
**Phase 3 (Testing & Validation): ✅ COMPLETE**
- Typecheck and lint passing across all services
- MCP OAuth authentication working in preview environment
- Tenant isolation via signed headers verified
- Fixed BM_TENANT_HEADER_SECRET mismatch between environments
- MCP tools successfully calling tenant APIs in preview
**Phase 4 (Deployment Configuration): ✅ COMPLETE**
- Updated apps/cloud/fly.template.toml with MCP environment variables
- Added HTTP/2 backend support for better MCP performance
- Added OAuth protected resource health check
- Removed MCP from preview deployment workflow
- Successfully deployed to preview environment (PR #113)
- All services operational at pr-113-basic-memory-cloud.fly.dev
**Next Steps:**
- Phase 5: Cleanup (remove apps/mcp directory)
- Phase 6: Production rollout and performance measurement
# SPEC-16: MCP Cloud Service Consolidation
## Why
@@ -100,7 +151,9 @@ app.include_router(provisioning_router)
### 2. Direct Tenant Transport (No HTTP Hop)
Instead of calling `/proxy`, MCP tools call tenant APIs directly via custom httpx transport:
Instead of calling `/proxy`, MCP tools call tenant APIs directly via custom httpx transport.
**Important:** No URL prefix stripping needed. The transport receives relative URLs like `/main/resource/notes/my-note` which are correctly routed to tenant APIs. The `/proxy` prefix only exists for web UI requests to the proxy router, not for MCP tools using the custom transport.
```python
# apps/cloud/src/basic_memory_cloud/transports/tenant_direct.py
@@ -139,28 +192,45 @@ class TenantDirectTransport(AsyncBaseTransport):
return response
```
Then override basic-memory's client before mounting MCP:
Then configure basic-memory's client factory before mounting MCP:
```python
# apps/cloud/src/basic_memory_cloud/main.py
from contextlib import asynccontextmanager
from basic_memory.mcp import async_client
from basic_memory_cloud.transports.tenant_direct import TenantDirectTransport
# Override basic-memory's HTTP client with direct transport
async_client.client = httpx.AsyncClient(
transport=TenantDirectTransport(),
base_url="http://direct"
)
# Configure factory for basic-memory's async_client
@asynccontextmanager
async def tenant_direct_client_factory():
"""Factory for creating clients with tenant direct transport."""
client = httpx.AsyncClient(
transport=TenantDirectTransport(),
base_url="http://direct",
)
try:
yield client
finally:
await client.aclose()
# Now mount MCP - tools will use direct transport
# Set factory BEFORE importing MCP tools
async_client.set_client_factory(tenant_direct_client_factory)
# NOW import - tools will use our factory
import basic_memory.mcp.tools
import basic_memory.mcp.prompts
from basic_memory.mcp.server import mcp
# Mount MCP - tools use direct transport via factory
app.mount("/mcp", mcp_app)
```
**Key benefits:**
- No changes to basic-memory code
- Clean dependency injection via factory pattern
- Per-request tenant resolution via FastMCP DI
- Eliminates HTTP hop entirely (~50 lines of code)
- Proper resource cleanup (client.aclose() guaranteed)
- Eliminates HTTP hop entirely
- /proxy endpoint remains for web UI
### 3. Keep /proxy Endpoint for Web UI
@@ -478,14 +548,15 @@ Remove manual auth header passing, use context manager:
- [x] Update any lingering references/docs (added deprecation notice to v15-docs/cloud-mode-usage.md)
#### 0.6 Testing
- [x] ~~Update test fixtures to use factory pattern~~ (Not needed - tests work fine as-is)
- [-] Update test fixtures to use factory pattern
- [x] Run full test suite in basic-memory
- [x] Verify cloud_mode_enabled works with CLIAuth injection (tested in preview env)
- [x] Verify cloud_mode_enabled works with CLIAuth injection
- [x] Run typecheck and linting
#### 0.7 Cloud Integration Prep
- [x] Update basic-memory-cloud pyproject.toml to use branch
- [x] Document factory usage pattern for cloud app
- [x] Implement factory pattern in cloud app main.py
- [x] Remove `/proxy` prefix stripping logic (not needed - tools pass relative URLs)
#### 0.8 Phase 0 Validation
@@ -496,18 +567,17 @@ Remove manual auth header passing, use context manager:
- [x] Linting passes (ruff)
- [x] Manual test: local mode works (ASGI transport)
- [x] Manual test: cloud login → cloud mode works (HTTP transport with auth)
- [x] No import of `inject_auth_header` anywhere
- [x] `headers.py` file deleted
- [x] `api_url` config removed
- [x] no use of `async_client.client` ✅
- [x] Tool functions properly scoped (client inside async with) - 15 tools ✅
- [x] CLI commands properly scoped (client inside async with) - 10 commands ✅
- [x] Prompts/resources properly scoped - 3 files ✅
- [x] No import of `inject_auth_header` anywhere
- [x] `headers.py` file deleted
- [x] `api_url` config removed
- [x] Tool functions properly scoped (client inside async with)
- [ ] CLI commands properly scoped (client inside async with)
**Integration validation:**
- [x] basic-memory-cloud can import and use factory pattern
- [x] TenantDirectTransport works without touching header injection
- [x] No circular imports or lazy import issues
- [x] basic-memory-cloud can import and use factory pattern
- [x] TenantDirectTransport works without touching header injection
- [x] No circular imports or lazy import issues
- [x] MCP tools work via inspector (local testing confirmed)
### Phase 1: Code Consolidation
- [x] Create feature branch `consolidate-mcp-cloud`
@@ -538,41 +608,52 @@ Remove manual auth header passing, use context manager:
- [x] Decode JWT to get `workos_user_id`
- [x] Look up/create tenant via `TenantRepository.get_or_create_tenant_for_workos_user()`
- [x] Build tenant app URL and add signed headers
- [x] Make direct httpx call to tenant API (no header stripping - keep it simple!)
- [x] Make direct httpx call to tenant API
- [x] No `/proxy` prefix stripping needed (tools pass relative URLs like `/main/resource/...`)
- [x] Update `apps/cloud/src/basic_memory_cloud/main.py`:
- [x] Import `async_client` from basic-memory
- [x] Override `async_client.client` with TenantDirectTransport
- [x] Do this BEFORE mounting MCP app
- [x] No changes to basic-memory required ✓
- [x] Refactored to use factory pattern instead of module-level override
- [x] Implement `tenant_direct_client_factory()` context manager
- [x] Call `async_client.set_client_factory()` before importing MCP tools
- [x] Clean imports, proper noqa hints for lint
- [x] Basic-memory refactor integrated (PR #344)
- [x] Run typecheck - passes ✓
- [x] Run lint - passes ✓
### Phase 3: Testing & Validation
- [x] Run `just typecheck` in apps/cloud
- [x] Run `just check` in project
- [x] Run `just fix` - all lint errors fixed ✓
- [x] Write comprehensive transport tests (11 tests passing) ✓
- [ ] Test MCP tools locally with consolidated service
- [ ] Verify OAuth authentication works
- [ ] Verify tenant isolation via signed headers
- [ ] Test /proxy endpoint still works for web UI
- [x] Test MCP tools locally with consolidated service (inspector confirmed working)
- [x] Verify OAuth authentication works (requires full deployment)
- [x] Verify tenant isolation via signed headers (requires full deployment)
- [x] Test /proxy endpoint still works for web UI
- [ ] Measure latency before/after consolidation
- [ ] Check telemetry traces span correctly
### Phase 4: Deployment Configuration
- [ ] Update `apps/cloud/fly.template.toml`:
- [ ] Ensure port 8000 exposed for /mcp endpoint
- [ ] Add MCP environment variables
- [ ] Configure workers setting
- [ ] Update deployment scripts to skip apps/mcp
- [ ] Update environment variable documentation
- [ ] Test deployment to development environment
- [x] Update `apps/cloud/fly.template.toml`:
- [x] Merged MCP-specific environment variables (AUTHKIT_BASE_URL, FASTMCP_LOG_LEVEL, BASIC_MEMORY_*)
- [x] Added HTTP/2 backend support (`h2_backend = true`) for better MCP performance
- [x] Added health check for MCP OAuth endpoint (`/.well-known/oauth-protected-resource`)
- [x] Port 8000 already exposed - serves both Cloud routes and /mcp endpoint
- [x] Workers configured (UVICORN_WORKERS = 4)
- [x] Update `.env.example`:
- [x] Consolidated MCP Gateway section into Cloud app section
- [x] Added AUTHKIT_BASE_URL, FASTMCP_LOG_LEVEL, BASIC_MEMORY_HOME
- [x] Added LOG_LEVEL to Development Settings
- [x] Documented that MCP now served at /mcp on Cloud service (port 8000)
- [x] Test deployment to preview environment (PR #113)
- [x] OAuth authentication verified
- [x] MCP tools successfully calling tenant APIs
- [x] Fixed BM_TENANT_HEADER_SECRET synchronization issue
### Phase 5: Cleanup
- [ ] Remove `apps/mcp/` directory entirely
- [ ] Remove MCP-specific fly.toml and deployment configs
- [ ] Update repository documentation
- [ ] Update CLAUDE.md with new architecture
- [ ] Archive old MCP deployment configs (if needed)
- [x] Remove `apps/mcp/` directory entirely
- [x] Remove MCP-specific fly.toml and deployment configs
- [x] Update repository documentation
- [x] Update CLAUDE.md with new architecture
- [-] Archive old MCP deployment configs (if needed)
### Phase 6: Production Rollout
- [ ] Deploy to development and validate
@@ -622,71 +703,71 @@ The well-organized code structure makes splitting back out feasible if future sc
**MCP Tools:**
- [ ] All 17 MCP tools work via consolidated /mcp endpoint
- [ ] OAuth authentication validates correctly
- [ ] Tenant isolation maintained via signed headers
- [ ] Project management tools function correctly
- [x] OAuth authentication validates correctly
- [x] Tenant isolation maintained via signed headers
- [x] Project management tools function correctly
**Cloud Routes:**
- [ ] /proxy endpoint still works for web UI
- [ ] /provisioning routes functional
- [ ] /webhooks routes functional
- [ ] /tenants routes functional
- [x] /proxy endpoint still works for web UI
- [x] /provisioning routes functional
- [x] /webhooks routes functional
- [x] /tenants routes functional
**API Validation:**
- [ ] Tenant API validates both JWT and signed headers
- [ ] Unauthorized requests rejected appropriately
- [ ] Multi-tenant isolation verified
- [x] Tenant API validates both JWT and signed headers
- [x] Unauthorized requests rejected appropriately
- [x] Multi-tenant isolation verified
### 2. Performance Testing
**Latency Reduction:**
- [ ] Measure MCP tool latency before consolidation
- [ ] Measure MCP tool latency after consolidation
- [ ] Verify reduction from eliminated HTTP hop (expected: 20-50ms improvement)
- [x] Measure MCP tool latency before consolidation
- [x] Measure MCP tool latency after consolidation
- [x] Verify reduction from eliminated HTTP hop (expected: 20-50ms improvement)
**Resource Usage:**
- [ ] Single app uses less total memory than two apps
- [ ] Database connection pooling more efficient
- [ ] HTTP client overhead reduced
- [x] Single app uses less total memory than two apps
- [x] Database connection pooling more efficient
- [x] HTTP client overhead reduced
### 3. Deployment Testing
**Fly.io Deployment:**
- [ ] Single app deploys successfully
- [ ] Health checks pass for consolidated service
- [ ] No apps/mcp deployment required
- [ ] Environment variables configured correctly
- [x] Single app deploys successfully
- [x] Health checks pass for consolidated service
- [x] No apps/mcp deployment required
- [x] Environment variables configured correctly
**Local Development:**
- [ ] `just setup` works with consolidated architecture
- [ ] Local testing shows MCP tools working
- [ ] No regression in developer experience
- [x] `just setup` works with consolidated architecture
- [x] Local testing shows MCP tools working
- [x] No regression in developer experience
### 4. Security Validation
**Defense in Depth:**
- [ ] Tenant API still validates JWT tokens
- [ ] Tenant API still validates signed headers
- [ ] No access possible with only signed headers (JWT required)
- [ ] No access possible with only JWT (signed headers required)
- [x] Tenant API still validates JWT tokens
- [x] Tenant API still validates signed headers
- [x] No access possible with only signed headers (JWT required)
- [x] No access possible with only JWT (signed headers required)
**Authorization:**
- [ ] Users can only access their own tenant data
- [ ] Cross-tenant requests rejected
- [ ] Admin operations require proper authentication
- [x] Users can only access their own tenant data
- [x] Cross-tenant requests rejected
- [x] Admin operations require proper authentication
### 5. Observability
**Telemetry:**
- [ ] OpenTelemetry traces span across MCP → ProxyService → Tenant API
- [ ] Logfire shows consolidated traces correctly
- [ ] Error tracking and debugging still functional
- [ ] Performance metrics accurate
- [x] OpenTelemetry traces span across MCP → ProxyService → Tenant API
- [x] Logfire shows consolidated traces correctly
- [x] Error tracking and debugging still functional
- [x] Performance metrics accurate
**Logging:**
- [ ] Structured logs show proper context (tenant_id, operation, etc.)
- [ ] Error logs contain actionable information
- [ ] Log volume reasonable for single app
- [x] Structured logs show proper context (tenant_id, operation, etc.)
- [x] Error logs contain actionable information
- [x] Log volume reasonable for single app
## Success Criteria
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,311 @@
---
title: 'SPEC-4: Notes Web UI Component Architecture'
type: note
permalink: specs/spec-4-notes-web-ui-component-architecture
tags:
- frontend
- 'component-architecture'
- vue
- 'refactoring'
---
# SPEC-4: Notes Web UI Component Architecture
## Why
The current Notes.vue component is a monolithic component that handles multiple responsibilities, making it difficult to maintain, test, and understand. This leads to:
- Complex state management across multiple concerns
- Difficult to isolate and test individual features
- Hard to understand the full scope of functionality
- Circular refactoring cycles when making changes
- Poor separation of concerns between navigation, display, and interaction logic
We need to decompose this into focused, single-responsibility components that are easier to develop, test, and maintain while preserving the existing functionality users expect.
## What
This spec defines the component architecture for decomposing the Notes web UI into focused components with clear responsibilities and interactions.
**Affected Areas:**
- `/apps/web/components/notes/Notes.vue` - Will be decomposed into smaller components
- `/apps/web/components/notes/` - New component structure
- Existing composables: `useNotesNavigation`, `useNotesFiltering`, `useNotesLayout`
- Mobile responsive behavior and layout management
**Component Breakdown:**
```
┌───────────────────────┬─────────────────────────────────────┬────────────────────────────────────────────────────────────┐
│ [Project] │ [Project Name] A/Z | ^ │ [edit | view] [actions] │
├───────────────────────┼─────────────────────────────────────┤ │
│ All Notes ├─────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ Recent │ search... │ [note header] │
│ [Project base dir] ├─────────────────────────────────────┤ │
│ ├─────────────────────────────────────┤ │
│ Folder1 │ Title [modified] │ │
│ Folder2 │ ├────────────────────────────────────────────────────────────┤
│ - Nested │ snippet │ [note body] │
│ │ │ │
│ │ │ │
│ ├─────────────────────────────────────┤ │
│ ├─────────────────────────────────────┤ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ ├─────────────────────────────────────┤ │
│ ├─────────────────────────────────────┤ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ ├─────────────────────────────────────┤ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
└───────────────────────┴─────────────────────────────────────┴────────────────────────────────────────────────────────────┘
```
### ProjectSwitcher Component
- **Location**: Top-left dropdown
- **Responsibility**: Allow users to switch between Basic Memory projects
- **Behavior**: Selecting different project controls entire Notes page content
- **State**: When switching projects, reset to "All notes" view
### NotesNav Component
- **Views**: Three mutually exclusive options:
- **All notes**: Display all notes in project alphabetically
- **Recent**: Display all notes in project by updated time (desc)
- **Project**: Display notes in top-level directory of project
- **Interaction**: Only one view can be active at a time
- **Folder Integration**: All/Recent ignore folder selection; Project respects folder selection
### FolderTree Component
- **Display**: Nested list of all folders in project as tree view
- **Interaction**: Selecting folder filters notes in NotesList using directoryList API
- **Navigation Integration**: Selecting folder automatically switches NotesNav to "Project" view for clear UX
- **API Integration**: Uses directoryList API call via useDirectoryListQuery for folder-specific note fetching
- **State Coordination**: Folder selection coordinates with navigation state for intuitive user experience
### NotesList Component
- **Display**: Vertically scrolling cards showing note summaries
- **Information per card**:
- Note title
- Modified time (relative, e.g., "7 minutes ago")
- Short summary of note content (one line preview)
- **Behavior**: Updates based on NotesNav selection and FolderTree filtering
### NoteDetail Component
- **Display**: Full content of selected note
- **Sections**:
- Header: Displays frontmatter information
- Content: Note body content
- **Editing**: Current textarea implementation (rich editor in future spec)
- **Frontmatter**: Leave current implementation (enhancement in future spec)
## How (High Level)
### Component Architecture Approach
1. **Single Responsibility**: Each component handles one primary concern
2. **Clear Data Flow**: Props down, events up pattern for component communication
3. **Composable Integration**: Use existing composables for state management
4. **Progressive Decomposition**: Extract components incrementally to maintain functionality
### Implementation Strategy
1. **Extract ProjectSwitcher**: Move project switching logic to dedicated component
2. **Extract NotesNav**: Isolate navigation state and view selection logic
3. **Extract FolderTree**: Separate folder display and selection logic
4. **Extract NotesList**: Isolate note listing and card display logic
5. **Extract NoteDetail**: Separate note content display and editing
6. **Update Notes.vue**: Become orchestration component managing component interactions
### State Management Integration
- **useNotesNavigation**: Manages navigation state (All/Recent/Project)
- **useNotesFiltering**: Handles filtering logic based on navigation and folder selection
- **useNotesLayout**: Manages responsive layout and panel visibility
- **Component State**: Each component manages its own internal UI state
- **Shared State**: Project selection and note filtering coordinated through composables
### Responsive Behavior
Mobile:
- Hide sidebar. pop out panel when selected
- show note list on small screens (existing behavior)
- when note list item is clicked, display note detail on full page. Cancel or go back to return to list
Desktop:
- Full three-column layout with all components visible
- **Transitions**: Smooth navigation between mobile panels
## How to Evaluate
### Success Criteria
- **Functional Parity**: All existing Notes page functionality preserved
- **Component Isolation**: Each component can be developed/tested independently
- **Clear Responsibilities**: No overlapping concerns between components
- **State Clarity**: Clean data flow and state management patterns
- **Mobile Compatibility**: Responsive behavior maintains current UX
- **Performance**: No degradation in rendering or interaction performance
### Testing Procedure
1. **Functionality Validation**:
- Project switching works correctly
- All three navigation views (All/Recent/Project) function properly
- Folder selection affects note display appropriately
- Note selection and detail display works
- Mobile responsive behavior preserved
2. **Component Isolation Testing**:
- Each component can be imported and used independently
- Component props and events are clearly defined
- No tight coupling between components
3. **Integration Testing**:
- Components communicate correctly through props/events
- State management composables integrate properly
- User workflows function end-to-end
4. **Performance Validation**:
- Page load time unchanged or improved
- Interaction responsiveness maintained
- Memory usage stable or improved
### Implementation Validation
- **Code Review**: Clean component structure with single responsibilities
- **Type Safety**: Full TypeScript coverage with proper component prop types
- **Documentation**: Each component has clear interface documentation
- **Tests**: Unit tests for individual components and integration tests for workflows
## Observations
- [problem] Monolithic Notes.vue component creates maintenance and testing challenges #component-architecture
- [solution] Component decomposition improves separation of concerns and testability #refactoring
- [pattern] Progressive extraction maintains functionality while improving structure #incremental-improvement
- [interaction] NotesNav and FolderTree have conditional interaction based on selected view #state-management
- [constraint] Mobile responsive behavior must be preserved during decomposition #responsive-design
- [scope] Current editing and frontmatter capabilities remain unchanged #scope-limitation
- [validation] Functional parity is critical success criteria for this refactoring #validation-strategy
- [implementation] Folder selection now properly integrates with directoryList API for accurate filtering #api-integration
- [fix] FolderTree selection functionality completed - works across all navigation views #feature-complete
- [ux-improvement] FolderTree selection automatically switches NotesNav to Project view for clear user feedback #user-experience
## Relations
- depends_on [[SPEC-1: Specification-Driven Development Process]]
- implements [[Current Notes.vue functionality]]
- prepares_for [[Future rich editor spec]]
- prepares_for [[Future frontmatter editing spec]]
## Implementation Progress
### Components
1. **ProjectSwitcher** (`~/components/notes/ProjectSwitcher.vue`)
- ✅ Top-left dropdown for project switching
- ✅ Integrates with Pinia project store
- ✅ Handles project switching with proper state reset
- ✅ Responsive collapsed/expanded states
- ✅ Expanded menu shows available projects and a Manage Projects option that navigates to the /settings/projects page
- ✅ Simplified component following SortingToggle pattern - clean Props/Emits interface, uses ProjectItem type directly
2. **NotesNav** (`~/components/notes/NotesNav.vue`)
- ✅ Three mutually exclusive views: All/Recent/Project
- ✅ Dynamic project title based on selected project
- ✅ Clean props down, events up pattern
- ✅ Responsive collapsed/expanded states with tooltips
- ✅ The label for the Project selection should be the folder name for the project, not the project name
3. **FolderTree** (`~/components/notes/FolderTree.vue`)
- ✅ Nested folder tree view for filtering
- ✅ Uses `useFolderTree()` composable for data
- ✅ Emits `folder-selected` events properly
- ✅ Handles loading, error, and empty states
- ✅ Includes companion `FolderTreeNode.vue` component
- ✅ The current folder should be visibly selected in the tree
4. **NotesList** (`~/components/notes/NotesList.vue`)
- ✅ Vertically scrolling note summary cards
- ✅ Shows title, updated time (relative), and content preview
- ✅ Badge system for tags with variant logic
- ✅ v-model integration for selectedNote
- ✅ Smooth transitions and animations
- ✅ Contextual title: The current folder name should be displayed at the top of the Notes list, or "All Notes", or "Recent" if they are selected
- ✅ The title header should contain a toggle component to allow sorting with Lucide icon labels
- sorting options:
- name (asc/desc) - default
- file updated time (asc/desc)
- If "Recent" notes nav option is selected the default order should be updated in descending order (recent first)
5. **NoteDisplay** (`~/components/notes/NoteDisplay.vue` - equivalent to spec's NoteDetail)
- ✅ Full note content display
- ✅ Edit/view mode toggle
- ✅ Header with frontmatter information
- ✅ Markdown rendering capabilities
- ✅ Current textarea implementation preserved
### Architecture Requirements
1. **Component Isolation**: Each component can be developed/tested independently ✅
2. **Single Responsibility**: Each component handles one primary concern ✅
3. **Clear Data Flow**: Props down, events up pattern implemented ✅
4. **Composable Integration**: Uses existing composables for state management ✅
5. **Responsive Behavior**: Mobile/desktop layout preserved ✅
### State Management Integration
- **useNotesNavigation**: Manages navigation state (All/Recent/Project) ✅
- **useNotesFiltering**: Handles filtering logic based on navigation and folder selection ✅
- **useNotesLayout**: Manages responsive layout and panel visibility ✅
- **Component State**: Each component manages its own internal UI state ✅
### Interaction Logic
- Only one NotesNav view active at a time ✅
- All/Recent views ignore folder selection ✅
- Project view respects folder selection ✅
- Project switching resets to "All notes" view ✅
### TypeScript Coverage
- All components have full TypeScript coverage ✅
- Component props and events properly typed ✅
- No TypeScript errors in codebase ✅
### Success Criteria Validation
1. **Functional Parity**: All existing Notes page functionality preserved ✅
2. **Component Isolation**: Each component can be developed/tested independently ✅
3. **Clear Responsibilities**: No overlapping concerns between components ✅
4. **State Clarity**: Clean data flow and state management patterns ✅
5. **Mobile Compatibility**: Responsive behavior maintains current UX ✅
6. **Performance**: No degradation in rendering or interaction performance ✅
## Implementation Decisions
### Architectural Patterns
1. **Composition API + `<script setup>`**: All components use modern Vue 3 syntax
2. **Pinia Store Integration**: Project switching handled through reactive store
3. **Composable Pattern**: State management distributed across focused composables
4. **Event-Driven Communication**: Clean parent-child communication via events
5. **Responsive-First Design**: Mobile/desktop layouts handled natively
### Key Technical Choices
1. **Progressive Enhancement**: Mobile-first responsive design with desktop enhancements
2. **State Reset Logic**: Project switching properly resets navigation, search, and selection state
3. **Performance Optimizations**: Efficient re-rendering with proper key usage and transitions
4. **Accessibility**: Screen reader support, tooltips, keyboard navigation
5. **Type Safety**: Full TypeScript coverage with proper component prop definitions
### Quality Metrics
- **Code Maintainability**: High - each component is focused and independently testable
- **Performance**: Excellent - no performance degradation from decomposition
- **User Experience**: Preserved - all existing functionality and responsive behavior maintained
- **Developer Experience**: Improved - cleaner component structure for future development
@@ -32,13 +32,13 @@ Related Github issue: https://github.com/basicmachines-co/basic-memory-cloud/iss
## Status
**Current Status**: **Phase 1 Implementation Complete**
**Target**: Fix Claude iOS session ID consistency issues
**Draft PR**: https://github.com/basicmachines-co/basic-memory/pull/298
**Current Status**: **ALL PHASES COMPLETE** ✅ **PRODUCTION DEPLOYED**
**Target**: Fix Claude iOS session ID consistency issues ✅ **ACHIEVED**
**Draft PR**: https://github.com/basicmachines-co/basic-memory/pull/298 ✅ **MERGED & DEPLOYED**
### 🎉 **MAJOR MILESTONE ACHIEVED**
### 🎉 **COMPLETE SUCCESS - PRODUCTION READY**
The complete stateless architecture has been successfully implemented for Basic Memory's MCP server! This represents a **fundamental architectural improvement** that solves the Claude iOS compatibility issue while making the entire system more robust and predictable.
**ALL PHASES OF SPEC-6 IMPLEMENTATION COMPLETE!** The stateless architecture has been successfully implemented across both Basic Memory core and Basic Memory Cloud, representing a **fundamental architectural improvement** that completely solves the Claude iOS compatibility issue while providing superior scalability and reliability.
#### Implementation Summary:
- **16 files modified** with 582 additions and 550 deletions
@@ -49,10 +49,12 @@ The complete stateless architecture has been successfully implemented for Basic
### Progress Summary
✅ **Complete Stateless Architecture Implementation (All 17 tools)**
- Stateless `get_active_project()` function implemented and deployed
- All session state dependencies removed across entire MCP server
- All MCP tools require explicit `project` parameter as first argument
**Complete Stateless Architecture Implementation (All 17 tools)** - **PRODUCTION DEPLOYED**
- Stateless `get_active_project()` function implemented and deployed
- All session state dependencies removed across entire MCP server
- All MCP tools require explicit `project` parameter as first argument
- **Cloud Service**: Redis removed, stateless HTTP enabled ✅
- **Production Validation**: Comprehensive testing completed with 100% success ✅
✅ **Content Management Tools Complete (6/6 tools)**
- `write_note`, `read_note`, `delete_note`, `edit_note`
@@ -306,12 +308,12 @@ File: experiments/Neural Network Results.md
Permalink: research-project/neural-network-results
```
### Phase 2: Cloud Service Simplification (basic-memory-cloud repository)
### Phase 2: Cloud Service Simplification (basic-memory-cloud repository) ✅ **COMPLETE**
#### Remove Session Infrastructure
1. Delete `apps/mcp/src/basic_memory_cloud_mcp/middleware/session_state.py`
2. Delete `apps/mcp/src/basic_memory_cloud_mcp/middleware/session_logging.py`
3. Update `apps/mcp/src/basic_memory_cloud_mcp/main.py`:
#### Remove Session Infrastructure **COMPLETE**
1. Delete `apps/mcp/src/basic_memory_cloud_mcp/middleware/session_state.py`
2. Delete `apps/mcp/src/basic_memory_cloud_mcp/middleware/session_logging.py`
3. Update `apps/mcp/src/basic_memory_cloud_mcp/main.py`:
```python
# Remove session middleware
# server.add_middleware(SessionStateMiddleware)
@@ -320,15 +322,16 @@ Permalink: research-project/neural-network-results
mcp = FastMCP(name="basic-memory-mcp", stateless_http=True)
```
#### Deployment Simplification
1. Remove Redis from `fly.toml`
2. Remove Redis environment variables
3. Update health checks to not depend on Redis
#### Deployment Simplification **COMPLETE**
1. Remove Redis from `fly.toml`
2. Remove Redis environment variables
3. Update health checks to not depend on Redis
4. ✅ Production deployment verified working with stateless architecture
### Phase 3: Conversational Project Management
### Phase 3: Conversational Project Management ✅ **COMPLETE**
#### Claude Behavior Pattern
1. **Project Discovery**:
#### Claude Behavior Pattern **VERIFIED WORKING**
1. **Project Discovery**:
```
Claude: Let me check your recent activity...
[calls recent_activity() - no project needed for discovery]
@@ -340,20 +343,22 @@ Permalink: research-project/neural-network-results
Which project should I use for this operation?
```
2. **Context Maintenance**:
2. **Context Maintenance**:
```
User: Use research-project
Claude: Working in research-project.
[All subsequent operations use project="research-project"]
```
3. **Explicit Project Switching**:
3. **Explicit Project Switching**:
```
User: Check work-notes for that meeting summary
Claude: Let me search work-notes for the meeting summary.
[Uses project="work-notes" for specific operation]
```
**Validation**: Comprehensive testing confirmed all conversational patterns work naturally with the stateless architecture.
## How to Evaluate
### Success Criteria
@@ -363,28 +368,32 @@ Permalink: research-project/neural-network-results
- [x] All MCP tools validate project exists before execution
- [x] `switch_project` and `get_current_project` tools removed
- [x] All responses display target project clearly
- [ ] No Redis dependencies in deployment (Phase 2: Cloud Service)
- [x] No Redis dependencies in deployment (Phase 2: Cloud Service) ✅ **COMPLETE**
- [x] `recent_activity` shows project distribution with ProjectActivitySummary
#### 2. Cross-Client Compatibility Testing
#### 2. Cross-Client Compatibility Testing ✅ **COMPLETE**
Test identical operations across all clients:
- [ ] **Claude Desktop**: All operations work with explicit projects
- [ ] **Claude Code**: All operations work with explicit projects
- [ ] **Claude Mobile iOS**: All operations work with explicit projects
- [ ] **API clients**: All operations work with explicit projects
- [ ] **CLI tools**: All operations work with explicit projects
- [x] **Claude Desktop**: All operations work with explicit projects
- [x] **Claude Code**: All operations work with explicit projects
- [x] **Claude Mobile iOS**: All operations work with explicit projects ✅ **CRITICAL SUCCESS**
- [x] **API clients**: All operations work with explicit projects
- [x] **CLI tools**: All operations work with explicit projects
#### 3. Session Independence Verification
- [ ] Operations work identically with/without session tracking
- [ ] No behavioral differences between clients
- [ ] Mobile client session ID changes do not affect operations
- [ ] Redis can be completely removed without functional impact
**Critical Achievement**: Claude iOS mobile client session tracking issues completely eliminated through stateless architecture.
#### 4. Performance & Scaling
- [ ] `stateless_http=True` enabled successfully
- [ ] No Redis memory usage
- [ ] Horizontal scaling possible (multiple MCP instances)
- [ ] Response times unchanged or improved
#### 3. Session Independence Verification ✅ **COMPLETE**
- [x] Operations work identically with/without session tracking ✅
- [x] No behavioral differences between clients ✅
- [x] Mobile client session ID changes do not affect operations ✅
- [x] Redis can be completely removed without functional impact ✅
**Production Validation**: Redis removed from production deployment with zero functional impact.
#### 4. Performance & Scaling ✅ **COMPLETE**
- [x] `stateless_http=True` enabled successfully ✅
- [x] No Redis memory usage ✅
- [x] Horizontal scaling possible (multiple MCP instances) ✅
- [x] Response times unchanged or improved ✅
#### 5. User Experience Testing
**Project Discovery Flow**:
@@ -402,11 +411,13 @@ Test identical operations across all clients:
- [x] Users always know which project is being operated on
- [x] No confusion about "current project" state
#### 6. Migration Safety
- [ ] Backward compatibility period with optional project parameter
- [ ] Clear migration documentation for existing users
- [ ] Data integrity maintained during transition
- [ ] No data loss during migration
#### 6. Migration Safety ✅ **COMPLETE**
- [x] Backward compatibility period with optional project parameter
- [x] Clear migration documentation for existing users
- [x] Data integrity maintained during transition
- [x] No data loss during migration
**Production Migration**: Successfully deployed to production with zero data loss and maintained system integrity.
### Test Scenarios
@@ -13,21 +13,26 @@ tags:
# SPEC-7: POC to spike Tigris/Turso for local access to cloud data
> **Status Update**: ✅ **Phase 1 COMPLETE** (September 20, 2025)
> TigrisFS mounting validated successfully in containerized environments. Container startup, filesystem mounting, and Fly.io integration all working correctly. Ready for Phase 2 (Turso database integration).
> See: [`SPEC-7-PHASE-1-RESULTS.md`](./SPEC-7-PHASE-1-RESULTS.md)
## Why
Current basic-memory-cloud architecture uses Fly volumes for tenant file storage, which creates several limitations:
We could enable a revolutionary user experience: **local editing (or at least view access) of cloud-stored files** while maintaining Basic Memory's existing filesystem assumptions.
1. **Storage Scalability**: Fly volumes require pre-provisioning and don't auto-scale with usage
2. **Cost Model**: Volume pricing vs object storage pricing may be less favorable at scale
3. **Local Development**: No way for users to mount their cloud tenant files locally for real-time editing
4. **Multi-Region**: Volumes are region-locked, limiting global deployment flexibility
5. **Backup/Disaster Recovery**: Object storage provides better durability and replication options
2. **Single Instance**: Volumes can only be mounted to one fly machine instance
3. **Cost Model**: Volume pricing vs object storage pricing may be less favorable at scale
4. **Local Development**: No way for users to mount their cloud tenant files locally for real-time editing
5. **Multi-Region**: Volumes are region-locked, limiting global deployment flexibility
6. **Backup/Disaster Recovery**: Object storage provides better durability and replication options
The core insight is that Basic Memory requires POSIX filesystem semantics but could benefit from object storage durability and accessibility. By combining:
- **Tigris object storage** for file persistence (via rclone mount)
- **Turso/libSQL** for SQLite indexing (replacing local .db files)
We could enable a revolutionary user experience: **local editing of cloud-stored files** while maintaining Basic Memory's existing filesystem assumptions.
Basic Memory requires POSIX filesystem semantics but could benefit from object storage durability and accessibility. By combining:
- **Tigris object storage and TigrisFS** for file persistence in bucket stoage via a POSIX filesystem on the tenant instance
- **Turso/libSQL** for SQLite indexing (replacing local .db files). Sqlite on NFS volumes is disouraged.
## What
@@ -36,55 +41,126 @@ This specification defines a proof-of-concept to validate the technical feasibil
**Affected Areas:**
- **Storage Architecture**: Replace Fly volumes with Tigris object storage
- **Database Architecture**: Replace local SQLite with Turso remote database
- **Container Setup**: Add rclone mounting in tenant containers
- **Container Setup**: Add TigrisFS mounting in tenant containers
- **Local Development**: Enable local mounting of cloud tenant data
- **Basic Memory Core**: Validate unchanged operation over mounted filesystems
**Key Components:**
- **Tigris Storage**: S3-compatible object storage via Fly.io integration
- **rclone NFS Mount**: Native NFS mounting without FUSE dependencies
- **Tigris Storage**: Globally caching S3-compatible object storage via Fly.io integration
- **TigrisFS**: Purpose-built FUSE filesystem with intelligent caching
- **Turso Database**: Hosted libSQL for SQLite replacement
- **Single-Tenant Model**: One bucket + one database per tenant (simplified isolation)
## Architectural Overview & Key Insights
### TigrisFS
Unlike standard S3 mounting approaches, **TigrisFS is a purpose-built FUSE filesystem** optimized for object storage with several critical advantages:
1. **Eliminates Fly Volume Limitations**
- No single-machine attachment constraints
- No pre-provisioning of storage capacity
- Enables horizontal scaling and zero-downtime deployments
- Automatic global CDN caching at Fly.io edge locations
2. **Intelligent Caching Architecture**
- 1-4GB+ configurable memory cache for read/write operations
- Write-back caching for improved performance
- Metadata cache to reduce API calls
- "Close to Redis speed" for small object retrieval
3. **Cost-Effective Model**
- Pay only for storage used and transferred
- No wasted capacity from over-provisioning
- Automatic global replication included
- S3 durability with CDN performance
### API-Driven Architecture Eliminates File Watching Concerns
**Critical Insight**: All file access (reads/writes) in basic-memory-cloud go through the API layer:
- **MCP Tools → API**: All Basic Memory operations use FastAPI endpoints
- **Web App → API**: Frontend uses API for all data modifications
- **File watching is NOT required** for cloud operations, unlike local BM which uses the WatchService to monitor file changes.
This means:
- **Cloud Operations**: Manual sync after API writes is sufficient
- **Local Development**: File watching only matters for local editing experience
- **Performance Risk**: Dramatically reduced since we're not dependent on inotify over network filesystems
### Realistic Local Access Expectations
**Baseline Functionality (Guaranteed):**
- Read-only mounting for browsing cloud files
- Easy download/upload of entire projects
- File copying via standard filesystem operations
**Stretch Goal (Test in POC):**
- Live editing with eventual consistency (1-5 second delays acceptable)
- Automatic sync for local changes
- Not required for core functionality - pure upside if it works
### Production Deployment Advantages
1. **Multi-Region Deployment**: Tigris handles global replication automatically
2. **Zero-Downtime Updates**: No volume detach/attach during deployments
3. **Tenant Migrations**: Simply update credentials, no data movement
4. **Disaster Recovery**: Built into S3 durability model (99.999999999% durability)
5. **Auto-Scaling**: Storage scales with usage, no capacity planning needed
## How (High Level)
### Phase 1: Local POC Validation
- [ ] Set up Tigris bucket with test data
- [ ] Configure rclone NFS mount locally
- [ ] Test Basic Memory operations over mounted filesystem
- [ ] Measure performance characteristics and identify issues
- [ ] Validate file watching, sync operations, and concurrent access patterns
### POC Approach: Server-First Validation
### Phase 2: Database Migration
**Rationale**: Start with server-side TigrisFS mounting because:
- Local access is meaningless if cloud containers can't mount TigrisFS reliably
- Container startup and API performance are critical path blockers
- TigrisFS compatibility with Basic Memory operations must be proven first
- Each phase gates the next - no point testing local access if server-side fails
### Phase 1: Server-Side TigrisFS Validation (Critical Foundation) ✅ COMPLETE
- [x] Set up Tigris bucket with test data via Fly.io integration
- [x] Create container image with TigrisFS support and dependencies
- [x] Test TigrisFS mounting in containerized environment
- [x] Run Basic Memory API operations over mounted TigrisFS
- [x] Validate all filesystem operations work correctly
- [x] Measure container startup time and resource usage
**Production Validation Results**: Container successfully deployed and operated for 42+ minutes serving real MCP requests with repository queries, knowledge graph navigation, and full Basic Memory API functionality over TigrisFS-mounted storage.
### Phase 2: Database Migration to Turso
- [ ] Set up Turso account and test database
- [ ] Modify Basic Memory to accept external DATABASE_URL
- [ ] Test all operations with remote SQLite via Turso
- [ ] Test all MCP tools with remote SQLite via Turso
- [ ] Validate performance and functionality parity
- [ ] Test API write → manual sync workflow in container
### Phase 3: Container Integration
- [ ] Create container image with rclone + NFS support
- [ ] Implement tenant-specific credential management
- [ ] Test container startup with automatic mounting
### Phase 3: Production Container Integration
- [ ] Implement tenant-specific credential management for buckets
- [x] Test container startup with automatic TigrisFS mounting
- [ ] Validate isolation between tenant containers
- [ ] Test API operations under realistic load
- [ ] Measure performance vs current Fly volume setup
### Phase 4: Local Access Validation
- [ ] Test local rclone mounting of tenant data
- [ ] Validate real-time file editing experience
- [ ] Test conflict resolution and sync behavior
### Phase 4: Local Access Validation (Bonus Feature)
- [ ] Test local TigrisFS mounting of tenant data
- [ ] Validate read-only access for browsing/downloading
- [ ] Test file copying and upload workflows
- [ ] Measure latency impact on user experience
- [ ] Test live editing if file watching works (stretch goal)
### Architecture Overview
```
Local Development:
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Local rclone │───▶│ Tigris Bucket │◀───│ Tenant Container│
NFS Mount │ │ (S3 storage) │ │ rclone mount
│ Local TigrisFS │───▶│ Tigris Bucket │◀───│ Tenant Container│
│ Mount │ │ (Global CDN) │ │ TigrisFS mount │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ Basic Memory │ │ Basic Memory │
│ (local files) │ │ (mounted files)
│ (local files) │ │ API + mounted
└─────────────────┘ └─────────────────┘
│ │
▼ ▼
@@ -92,102 +168,157 @@ Local Development:
│ Turso Database │◀───────────────────────────│ Turso Database │
│ (shared index) │ │ (shared index) │
└─────────────────┘ └─────────────────┘
Flow: API writes → Manual sync → Index update
Local: File watching (if available) → Auto sync
```
## How to Evaluate
### Success Criteria
- [ ] **Filesystem Compatibility**: Basic Memory operates without modification over rclone-mounted Tigris storage
- [ ] **Performance Acceptable**: File operations complete within 2x local filesystem latency
- [x] **Filesystem Compatibility**: Basic Memory operates without modification over TigrisFS-mounted storage
- [x] **Performance Acceptable**: API-driven operations perform within acceptable latency (target: <500ms for typical operations)
- [ ] **Database Functionality**: All Basic Memory features work with Turso remote SQLite
- [ ] **Container Reliability**: Tenant containers start successfully with automatic mounting
- [ ] **Local Access**: Users can mount and edit cloud files locally with real-time sync
- [ ] **Data Isolation**: Tenant data remains properly isolated using bucket/database separation
- [x] **Container Reliability**: Tenant containers start successfully with automatic TigrisFS mounting
- [ ] **Local Access Baseline**: Users can mount cloud files locally for read-only browsing and file copying
- [x] **Data Isolation**: Tenant data remains properly isolated using bucket/database separation
- [ ] **Local Access Stretch**: Live editing with eventual sync (1-5 second delays acceptable)
### Testing Procedure
1. **Local Filesystem Test**:
```bash
# Mount Tigris bucket locally
rclone nfsmount tigris:test-bucket ~/tigris-test --vfs-cache-mode writes
# Run Basic Memory operations
cd ~/tigris-test && basic-memory sync --watch
# Test: create notes, search, file watching, bulk operations
#### Phase 1: Server-Side Foundation Testing
1. **Container TigrisFS Test**:
```dockerfile
# Test container with TigrisFS mounting
FROM python:3.12
RUN apt-get update && apt-get install -y tigrisfs
# Test startup script
#!/bin/bash
tigrisfs --memory-limit 2048 $TIGRIS_BUCKET /app/data --daemon
cd /app/data && basic-memory sync
basic-memory-api --data-dir /app/data
```
2. **Database Migration Test**:
2. **API Operations Validation**:
```bash
# Configure Turso connection
# Test all MCP operations over TigrisFS
curl -X POST /api/write_note -d '{"title":"test","content":"content"}'
curl -X GET /api/read_note/test
curl -X GET /api/search_notes?q=content
# Measure: response times, error rates, data consistency
```
#### Phase 2: Database Integration Testing
3. **Turso Integration Test**:
```bash
# Configure Turso connection in container
export DATABASE_URL="libsql://test-db.turso.io?authToken=..."
# Test all MCP tools with remote database
basic-memory tools # Test each tool functionality
# Test API write → manual sync workflow
```
3. **Container Integration Test**:
```dockerfile
# Test container with rclone mounting
FROM python:3.12
RUN apt-get update && apt-get install -y rclone nfs-common
# ... test startup and mounting process
```
#### Phase 3: Production Readiness Testing
4. **Performance Benchmarking**:
- File creation/read/write operations (target: <2x local latency)
- Search query performance (target: comparable to local SQLite)
- File watching responsiveness (target: events within 1-2 seconds)
- Concurrent operation handling
- Container startup time with TigrisFS mounting
- API operation response times (target: <500ms for typical operations)
- Search query performance with Turso (target: comparable to local SQLite)
- TigrisFS cache hit rates and memory usage
- Concurrent tenant isolation
#### Phase 4: Local Access Testing (If Phase 1-3 Succeed)
5. **Local Access Validation**:
```bash
# Test read-only access
tigrisfs tenant-bucket ~/local-tenant
ls -la ~/local-tenant # Browse files
cp ~/local-tenant/notes/* ~/backup/ # Copy files
# Test file watching (stretch goal)
echo "test" > ~/local-tenant/test.md
# Check if changes sync to cloud
```
### Go/No-Go Criteria by Phase
- **Phase 1**: Container must start successfully and serve API requests over TigrisFS
- **Phase 2**: All MCP tools must work with Turso with <2x latency increase
- **Phase 3**: Performance must be within 50% of current Fly volume setup
- **Phase 4**: Local mounting must work reliably for read-only access
### Risk Assessment
**High Risk Items**:
- [ ] NFS-over-S3 performance may be insufficient for real-time operations
- [ ] File watching (`inotify`) over NFS may be unreliable
- [ ] Network interruptions could cause filesystem errors
- [ ] Concurrent access patterns might hit S3 rate limits
**Moderate Risk Items (Mitigated by API-First Architecture)**:
- [ ] TigrisFS performance for local access may have higher latency than local filesystem
- [ ] File watching (`inotify`) over FUSE may be unreliable for local development
- [ ] Network interruptions could cause filesystem errors during local editing
- [ ] Write-back caching could cause data loss if container crashes during flush
**Low Risk Items (API-First Eliminates)**:
- [ ] ~~Real-time file watching~~ - Not required for cloud operations
- [ ] ~~Concurrent write consistency~~ - Single-tenant model with API coordination
- [ ] ~~S3 rate limits~~ - TigrisFS intelligent caching handles this
**Mitigation Strategies**:
- Comprehensive performance testing before committing to architecture
- Fallback plan to S3-native storage backend if filesystem approach fails
- Extensive error handling and retry logic for network issues
- **Performance**: Comprehensive benchmarking with realistic workloads
- **Reliability**: Graceful degradation to read-only local access if live editing fails
- **Data Safety**: Regular sync intervals and write-through mode for critical operations
- **Fallback**: Keep Fly volumes as backup deployment option
### Metrics to Track
- **Latency**: File operation response times (read/write/watch)
- **Reliability**: Success rate of file operations over time
- **Throughput**: Concurrent file operations and search queries
- **User Experience**: Perceived performance for local mounting use case
- **API Latency**: Response times for MCP tools and web operations
- **Cache Effectiveness**: TigrisFS cache hit rates and memory usage
- **Local Access Performance**: File browsing and copying speeds
- **Reliability**: Success rate of mount operations and data consistency
- **Cost**: Storage usage, API calls, and network transfer costs vs current volumes
## Notes
### Key Architectural Decisions
- **Single tenant per bucket/database**: Simplifies isolation and credential management
- **Maintain POSIX compatibility**: Preserve Basic Memory's existing filesystem assumptions
- **NFS over FUSE**: Better compatibility and performance characteristics
- **TigrisFS over rclone**: Purpose-built for object storage with intelligent caching
- **Turso for SQLite**: Leverages specialized remote SQLite expertise
- **API-first approach**: Eliminates file watching dependency for cloud operations
### Alternative Approaches Considered
- **S3-native storage backend**: Would require Basic Memory architecture changes
- **Hybrid approach**: Local files + cloud sync (adds complexity)
- **FUSE mounting**: More platform dependencies and kernel requirements
- **Standard rclone mounting**: Less optimized than TigrisFS for object storage workloads
- **Keep Fly volumes**: Maintains current limitations but proven reliability
### Integration Points
- [ ] Fly.io Tigris integration for bucket provisioning
- [ ] Turso account setup and database provisioning
- [ ] Container image modifications for rclone support
- [ ] Container image modifications for TigrisFS support
- [ ] Credential management for tenant isolation
- [ ] API modification for manual sync triggers
- [ ] Local client setup documentation for TigrisFS mounting
## Observations
- [architecture] Tigris/Turso split cleanly separates file storage from indexing concerns #storage-separation
- [breakthrough] API-first architecture eliminates file watching dependency for cloud operations #api-first-advantage
- [user-experience] Local mounting of cloud files could be revolutionary for knowledge management #local-cloud-hybrid
- [compatibility] Maintaining POSIX filesystem assumptions preserves Basic Memory's local/cloud compatibility #architecture-preservation
- [simplification] Single tenant per bucket eliminates complex multi-tenancy in storage layer #tenant-isolation
- [risk] NFS-over-S3 performance characteristics are unproven for real-time operations #performance-risk
- [performance] TigrisFS intelligent caching could provide near-local performance for common operations #tigrisfs-advantage
- [deployment] Zero-downtime updates become trivial without volume constraints #deployment-simplification
- [benefit] Object storage pricing model could be more favorable than volume pricing #cost-optimization
- [innovation] Real-time local editing of cloud-stored files addresses major SaaS limitation #competitive-advantage
- [innovation] Read-only local access alone would address major SaaS limitation #competitive-advantage
- [risk-mitigation] API-driven sync reduces performance requirements vs real-time file watching #risk-reduction
## Relations
- implements [[SPEC-6 Explicit Project Parameter Architecture]]
- requires [[Fly.io Tigris Integration]]
- enables [[Local Cloud File Access]]
- alternative_to [[Fly Volume Storage]]
- alternative_to [[Fly Volume Storage]]
## Links
- https://fly.io/hello/tigris
- https://fly.io/docs/tigris/
- https://www.tigrisdata.com/docs/sdks/fly/data-migration-with-flyctl/
- https://www.tigrisdata.com/docs/training/tigrisfs/
- https://www.tigrisdata.com/blog/tigris-filesystem/
- https://www.tigrisdata.com/docs/quickstarts/rclone/
@@ -0,0 +1,196 @@
---
title: 'SPEC-9: Signed Header Tenant Information'
type: spec
permalink: specs/spec-9-signed-header-tenant-information
tags:
- authentication
- tenant-isolation
- proxy
- security
- mcp
---
# SPEC-9: Signed Header Tenant Information
## Why
WorkOS JWT templates don't work with MCP's dynamic client registration requirement, preventing us from getting tenant information directly in JWT tokens. We need an alternative secure method to pass tenant context from the Cloud Proxy Service to tenant instances.
**Problem Context:**
- MCP spec requires dynamic client registration
- WorkOS JWT templates only apply to statically configured clients
- Without tenant information, we can't properly route requests or isolate tenant data
- Current JWT tokens only contain standard OIDC claims (sub, email, etc.)
**Affected Areas:**
- Cloud Proxy Service (`apps/cloud`) - request forwarding
- Tenant API instances (`apps/api`) - tenant context validation
- MCP Gateway (`apps/mcp`) - authentication flow
- Overall tenant isolation security model
## What
Implement HMAC-signed headers that the Cloud Proxy Service adds when forwarding requests to tenant instances. This provides secure, tamper-proof tenant information without relying on JWT custom claims.
**Components:**
- Header signing utility in Cloud Proxy Service
- Header validation middleware in Tenant API instances
- Shared secret configuration across services
- Fallback mechanisms for development and error cases
## How (High Level)
### 1. Header Format
Add these signed headers to all proxied requests:
```
X-BM-Tenant-ID: {tenant_id}
X-BM-Timestamp: {unix_timestamp}
X-BM-Signature: {hmac_sha256_signature}
```
### 2. Signature Algorithm
```python
# Canonical message format
message = f"{tenant_id}:{timestamp}"
# HMAC-SHA256 signature
signature = hmac.new(
key=shared_secret.encode('utf-8'),
msg=message.encode('utf-8'),
digestmod=hashlib.sha256
).hexdigest()
```
### 3. Implementation Flow
#### Cloud Proxy Service (`apps/cloud`)
1. Extract `tenant_id` from authenticated user profile
2. Generate timestamp and canonical message
3. Sign message with shared secret
4. Add headers to request before forwarding to tenant instance
#### Tenant API Instances (`apps/api`)
1. Middleware validates headers on all incoming requests
2. Extract tenant_id, timestamp from headers
3. Verify timestamp is within acceptable window (5 minutes)
4. Recompute signature and compare in constant time
5. If valid, make tenant context available to Basic Memory tools
### 4. Security Properties
- **Authenticity**: Only services with shared secret can create valid signatures
- **Integrity**: Header tampering invalidates signature
- **Replay Protection**: Timestamp prevents reuse of old signatures
- **Non-repudiation**: Each request is cryptographically tied to specific tenant
### 5. Configuration
```bash
# Shared across Cloud Proxy and Tenant instances
BM_TENANT_HEADER_SECRET=randomly-generated-256-bit-secret
# Tenant API configuration
BM_TENANT_HEADER_VALIDATION=true # true (production) | false (dev only)
```
## How to Evaluate
### Unit Tests
- [ ] Header signing utility generates correct signatures
- [ ] Header validation correctly accepts/rejects signatures
- [ ] Timestamp validation within acceptable windows
- [ ] Constant-time signature comparison prevents timing attacks
### Integration Tests
- [ ] End-to-end request flow from MCP client → proxy → tenant
- [ ] Tenant isolation verified with signed headers
- [ ] Error handling for missing/invalid headers
- [ ] Disabled validation in development environment
### Security Validation
- [ ] Shared secret rotation procedure
- [ ] Header tampering detection
- [ ] Clock skew tolerance testing
- [ ] Performance impact measurement
### Production Readiness
- [ ] Logging and monitoring of header validation
- [ ] Graceful degradation for header validation failures
- [ ] Documentation for secret management
- [ ] Deployment configuration templates
## Implementation Notes
### Shared Secret Management
- Generate cryptographically secure 256-bit secret
- Same secret deployed to Cloud Proxy and all Tenant instances
- Consider secret rotation strategy for production
### Error Handling
```python
# Strict mode (production)
if not validate_headers(request):
raise HTTPException(status_code=401, detail="Invalid tenant headers")
# Fallback mode (development)
if not validate_headers(request):
logger.warning("Invalid headers, falling back to default tenant")
tenant_id = "default"
```
### Performance Considerations
- HMAC-SHA256 computation is fast (~microseconds)
- Headers add ~200 bytes to each request
- Validation happens once per request in middleware
## Benefits
**Works with MCP dynamic client registration** - No dependency on JWT custom claims
**Simple and reliable** - Standard HMAC signature approach
**Secure by design** - Cryptographic authenticity and integrity
**Infrastructure controlled** - No external service dependencies
**Easy to implement** - Clear signature algorithm and validation
## Trade-offs
⚠️ **Shared secret management** - Need secure distribution and rotation
⚠️ **Clock synchronization** - Timestamp validation requires reasonably synced clocks
⚠️ **Header visibility** - Headers visible in logs (tenant_id not sensitive)
⚠️ **Additional complexity** - More moving parts in proxy forwarding
## Implementation Tasks
### Cloud Service (Header Signing)
- [ ] Create `utils/header_signing.py` with HMAC-SHA256 signing function
- [ ] Add `bm_tenant_header_secret` to Cloud service configuration
- [ ] Update `ProxyService.forward_request()` to call signing utility
- [ ] Add signed headers (X-BM-Tenant-ID, X-BM-Timestamp, X-BM-Signature)
### Tenant API (Header Validation)
- [ ] Create `utils/header_validation.py` with signature verification
- [ ] Add `bm_tenant_header_secret` to API service configuration
- [ ] Create `TenantHeaderValidationMiddleware` class
- [ ] Add middleware to FastAPI app (before other middleware)
- [ ] Skip validation for `/health` endpoint
- [ ] Store validated tenant_id in request.state
### Testing
- [ ] Unit test for header signing utility
- [ ] Unit test for header validation utility
- [ ] Integration test for proxy → tenant flow
- [ ] Test invalid/missing header handling
- [ ] Test timestamp window validation
- [ ] Test signature tampering detection
### Configuration & Deployment
- [ ] Update `.env.example` with BM_TENANT_HEADER_SECRET
- [ ] Generate secure 256-bit secret for production
- [ ] Update Fly.io secrets for both services
- [ ] Document secret rotation procedure
## Status
- [x] **Specification Complete** - Design finalized and documented
- [ ] **Implementation Started** - Header signing utility development
- [ ] **Cloud Proxy Updated** - ProxyService adds signed headers
- [ ] **Tenant Validation Added** - Middleware validates headers
- [ ] **Testing Complete** - All validation criteria met
- [ ] **Production Deployed** - Live with tenant isolation via headers
+1 -1
View File
@@ -1,7 +1,7 @@
"""basic-memory - Local-first knowledge management combining Zettelkasten with knowledge graphs"""
# Package version - updated by release automation
__version__ = "0.15.1"
__version__ = "0.16.2"
# API version for FastAPI - independent of package version
__api_version__ = "v0"
+104 -22
View File
@@ -1,10 +1,21 @@
"""Alembic environment configuration."""
import asyncio
import os
from logging.config import fileConfig
from sqlalchemy import engine_from_config
from sqlalchemy import pool
# Allow nested event loops (needed for pytest-asyncio and other async contexts)
# Note: nest_asyncio doesn't work with uvloop, so we handle that case separately
try:
import nest_asyncio
nest_asyncio.apply()
except (ImportError, ValueError):
# nest_asyncio not available or can't patch this loop type (e.g., uvloop)
pass
from sqlalchemy import engine_from_config, pool
from sqlalchemy.ext.asyncio import AsyncEngine, create_async_engine
from alembic import context
@@ -20,12 +31,22 @@ from basic_memory.models import Base # noqa: E402
# access to the values within the .ini file in use.
config = context.config
# Load app config - this will read environment variables (BASIC_MEMORY_DATABASE_BACKEND, etc.)
# due to Pydantic's env_prefix="BASIC_MEMORY_" setting
app_config = ConfigManager().config
# Set the SQLAlchemy URL from our app config
sqlalchemy_url = f"sqlite:///{app_config.database_path}"
config.set_main_option("sqlalchemy.url", sqlalchemy_url)
# print(f"Using SQLAlchemy URL: {sqlalchemy_url}")
# Set the SQLAlchemy URL based on database backend configuration
# If the URL is already set in config (e.g., from run_migrations), use that
# Otherwise, get it from app config
# Note: alembic.ini has a placeholder URL "driver://user:pass@localhost/dbname" that we need to override
current_url = config.get_main_option("sqlalchemy.url")
if not current_url or current_url == "driver://user:pass@localhost/dbname":
from basic_memory.db import DatabaseType
sqlalchemy_url = DatabaseType.get_db_url(
app_config.database_path, DatabaseType.FILESYSTEM, app_config
)
config.set_main_option("sqlalchemy.url", sqlalchemy_url)
# Interpret the config file for Python logging.
if config.config_file_name is not None:
@@ -69,28 +90,89 @@ def run_migrations_offline() -> None:
context.run_migrations()
def do_run_migrations(connection):
"""Execute migrations with the given connection."""
context.configure(
connection=connection,
target_metadata=target_metadata,
include_object=include_object,
render_as_batch=True,
compare_type=True,
)
with context.begin_transaction():
context.run_migrations()
async def run_async_migrations(connectable):
"""Run migrations asynchronously with AsyncEngine."""
async with connectable.connect() as connection:
await connection.run_sync(do_run_migrations)
await connectable.dispose()
def run_migrations_online() -> None:
"""Run migrations in 'online' mode.
In this scenario we need to create an Engine
and associate a connection with the context.
Supports both sync engines (SQLite) and async engines (PostgreSQL with asyncpg).
"""
connectable = engine_from_config(
config.get_section(config.config_ini_section, {}),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
# Check if a connection/engine was provided (e.g., from run_migrations)
connectable = context.config.attributes.get("connection", None)
with connectable.connect() as connection:
context.configure(
connection=connection,
target_metadata=target_metadata,
include_object=include_object,
render_as_batch=True,
)
if connectable is None:
# No connection provided, create engine from config
url = context.config.get_main_option("sqlalchemy.url")
with context.begin_transaction():
context.run_migrations()
# Check if it's an async URL (sqlite+aiosqlite or postgresql+asyncpg)
if url and ("+asyncpg" in url or "+aiosqlite" in url):
# Create async engine for asyncpg or aiosqlite
connectable = create_async_engine(
url,
poolclass=pool.NullPool,
future=True,
)
else:
# Create sync engine for regular sqlite or postgresql
connectable = engine_from_config(
context.config.get_section(context.config.config_ini_section, {}),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
# Handle async engines (PostgreSQL with asyncpg)
if isinstance(connectable, AsyncEngine):
# Try to run async migrations
# nest_asyncio allows asyncio.run() from within event loops, but doesn't work with uvloop
try:
asyncio.run(run_async_migrations(connectable))
except RuntimeError as e:
if "cannot be called from a running event loop" in str(e):
# We're in a running event loop (likely uvloop) - need to use a different approach
# Create a new thread to run the async migrations
import concurrent.futures
def run_in_thread():
"""Run async migrations in a new event loop in a separate thread."""
new_loop = asyncio.new_event_loop()
asyncio.set_event_loop(new_loop)
try:
new_loop.run_until_complete(run_async_migrations(connectable))
finally:
new_loop.close()
with concurrent.futures.ThreadPoolExecutor() as executor:
future = executor.submit(run_in_thread)
future.result() # Wait for completion and re-raise any exceptions
else:
raise
else:
# Handle sync engines (SQLite) or sync connections
if hasattr(connectable, "connect"):
# It's an engine, get a connection
with connectable.connect() as connection:
do_run_migrations(connection)
else:
# It's already a connection
do_run_migrations(connectable)
if context.is_offline_mode():
@@ -0,0 +1,131 @@
"""Add Postgres full-text search support with tsvector and GIN indexes
Revision ID: 314f1ea54dc4
Revises: e7e1f4367280
Create Date: 2025-11-15 18:05:01.025405
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = "314f1ea54dc4"
down_revision: Union[str, None] = "e7e1f4367280"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Add PostgreSQL full-text search support.
This migration:
1. Creates search_index table for Postgres (SQLite uses FTS5 virtual table)
2. Adds generated tsvector column for full-text search
3. Creates GIN index on the tsvector column for fast text queries
4. Creates GIN index on metadata JSONB column for fast containment queries
Note: These changes only apply to Postgres. SQLite continues to use FTS5 virtual tables.
"""
# Check if we're using Postgres
connection = op.get_bind()
if connection.dialect.name == "postgresql":
# Create search_index table for Postgres
# For SQLite, this is a FTS5 virtual table created elsewhere
from sqlalchemy.dialects.postgresql import JSONB
op.create_table(
"search_index",
sa.Column("id", sa.Integer(), nullable=False), # Entity IDs are integers
sa.Column("project_id", sa.Integer(), nullable=False), # Multi-tenant isolation
sa.Column("title", sa.Text(), nullable=True),
sa.Column("content_stems", sa.Text(), nullable=True),
sa.Column("content_snippet", sa.Text(), nullable=True),
sa.Column("permalink", sa.String(), nullable=True), # Nullable for non-markdown files
sa.Column("file_path", sa.String(), nullable=True),
sa.Column("type", sa.String(), nullable=True),
sa.Column("from_id", sa.Integer(), nullable=True), # Relation IDs are integers
sa.Column("to_id", sa.Integer(), nullable=True), # Relation IDs are integers
sa.Column("relation_type", sa.String(), nullable=True),
sa.Column("entity_id", sa.Integer(), nullable=True), # Entity IDs are integers
sa.Column("category", sa.String(), nullable=True),
sa.Column("metadata", JSONB(), nullable=True), # Use JSONB for Postgres
sa.Column("created_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=True),
sa.PrimaryKeyConstraint(
"id", "type", "project_id"
), # Composite key: id can repeat across types
sa.ForeignKeyConstraint(
["project_id"],
["project.id"],
name="fk_search_index_project_id",
ondelete="CASCADE",
),
if_not_exists=True,
)
# Create index on project_id for efficient multi-tenant queries
op.create_index(
"ix_search_index_project_id",
"search_index",
["project_id"],
unique=False,
)
# Create unique partial index on permalink for markdown files
# Non-markdown files don't have permalinks, so we use a partial index
op.execute("""
CREATE UNIQUE INDEX uix_search_index_permalink_project
ON search_index (permalink, project_id)
WHERE permalink IS NOT NULL
""")
# Add tsvector column as a GENERATED ALWAYS column
# This automatically updates when title or content_stems change
op.execute("""
ALTER TABLE search_index
ADD COLUMN textsearchable_index_col tsvector
GENERATED ALWAYS AS (
to_tsvector('english',
coalesce(title, '') || ' ' ||
coalesce(content_stems, '')
)
) STORED
""")
# Create GIN index on tsvector column for fast full-text search
op.create_index(
"idx_search_index_fts",
"search_index",
["textsearchable_index_col"],
unique=False,
postgresql_using="gin",
)
# Create GIN index on metadata JSONB for fast containment queries
# Using jsonb_path_ops for smaller index size and better performance
op.execute("""
CREATE INDEX idx_search_index_metadata_gin
ON search_index
USING GIN (metadata jsonb_path_ops)
""")
def downgrade() -> None:
"""Remove PostgreSQL full-text search support."""
connection = op.get_bind()
if connection.dialect.name == "postgresql":
# Drop indexes first
op.execute("DROP INDEX IF EXISTS idx_search_index_metadata_gin")
op.drop_index("idx_search_index_fts", table_name="search_index")
op.execute("DROP INDEX IF EXISTS uix_search_index_permalink_project")
op.drop_index("ix_search_index_project_id", table_name="search_index")
# Drop the generated column
op.execute("ALTER TABLE search_index DROP COLUMN IF EXISTS textsearchable_index_col")
# Drop the search_index table
op.drop_table("search_index")
@@ -21,6 +21,12 @@ depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
# SQLite FTS5 virtual table handling is SQLite-specific
# For Postgres, search_index is a regular table managed by ORM
connection = op.get_bind()
is_sqlite = connection.dialect.name == "sqlite"
op.create_table(
"project",
sa.Column("id", sa.Integer(), nullable=False),
@@ -55,7 +61,9 @@ def upgrade() -> None:
batch_op.add_column(sa.Column("project_id", sa.Integer(), nullable=False))
batch_op.drop_index(
"uix_entity_permalink",
sqlite_where=sa.text("content_type = 'text/markdown' AND permalink IS NOT NULL"),
sqlite_where=sa.text("content_type = 'text/markdown' AND permalink IS NOT NULL")
if is_sqlite
else None,
)
batch_op.drop_index("ix_entity_file_path")
batch_op.create_index(batch_op.f("ix_entity_file_path"), ["file_path"], unique=False)
@@ -67,12 +75,16 @@ def upgrade() -> None:
"uix_entity_permalink_project",
["permalink", "project_id"],
unique=True,
sqlite_where=sa.text("content_type = 'text/markdown' AND permalink IS NOT NULL"),
sqlite_where=sa.text("content_type = 'text/markdown' AND permalink IS NOT NULL")
if is_sqlite
else None,
)
batch_op.create_foreign_key("fk_entity_project_id", "project", ["project_id"], ["id"])
# drop the search index table. it will be recreated
op.drop_table("search_index")
# Only drop for SQLite - Postgres uses regular table managed by ORM
if is_sqlite:
op.drop_table("search_index")
# ### end Alembic commands ###
@@ -25,43 +25,51 @@ def upgrade() -> None:
The UNIQUE constraint prevents multiple projects from having is_default=FALSE,
which breaks project creation when the service sets is_default=False.
Since SQLite doesn't support dropping specific constraints easily, we'll
recreate the table without the problematic constraint.
SQLite: Recreate the table without the constraint (no ALTER TABLE support)
Postgres: Use ALTER TABLE to drop the constraint directly
"""
# For SQLite, we need to recreate the table without the UNIQUE constraint
# Create a new table without the UNIQUE constraint on is_default
op.create_table(
"project_new",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("name", sa.String(), nullable=False),
sa.Column("description", sa.Text(), nullable=True),
sa.Column("permalink", sa.String(), nullable=False),
sa.Column("path", sa.String(), nullable=False),
sa.Column("is_active", sa.Boolean(), nullable=False),
sa.Column("is_default", sa.Boolean(), nullable=True), # No UNIQUE constraint!
sa.Column("created_at", sa.DateTime(), nullable=False),
sa.Column("updated_at", sa.DateTime(), nullable=False),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("name"),
sa.UniqueConstraint("permalink"),
)
connection = op.get_bind()
is_sqlite = connection.dialect.name == "sqlite"
# Copy data from old table to new table
op.execute("INSERT INTO project_new SELECT * FROM project")
if is_sqlite:
# For SQLite, we need to recreate the table without the UNIQUE constraint
# Create a new table without the UNIQUE constraint on is_default
op.create_table(
"project_new",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("name", sa.String(), nullable=False),
sa.Column("description", sa.Text(), nullable=True),
sa.Column("permalink", sa.String(), nullable=False),
sa.Column("path", sa.String(), nullable=False),
sa.Column("is_active", sa.Boolean(), nullable=False),
sa.Column("is_default", sa.Boolean(), nullable=True), # No UNIQUE constraint!
sa.Column("created_at", sa.DateTime(), nullable=False),
sa.Column("updated_at", sa.DateTime(), nullable=False),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("name"),
sa.UniqueConstraint("permalink"),
)
# Drop the old table
op.drop_table("project")
# Copy data from old table to new table
op.execute("INSERT INTO project_new SELECT * FROM project")
# Rename the new table
op.rename_table("project_new", "project")
# Drop the old table
op.drop_table("project")
# Recreate the indexes
with op.batch_alter_table("project", schema=None) as batch_op:
batch_op.create_index("ix_project_created_at", ["created_at"], unique=False)
batch_op.create_index("ix_project_name", ["name"], unique=True)
batch_op.create_index("ix_project_path", ["path"], unique=False)
batch_op.create_index("ix_project_permalink", ["permalink"], unique=True)
batch_op.create_index("ix_project_updated_at", ["updated_at"], unique=False)
# Rename the new table
op.rename_table("project_new", "project")
# Recreate the indexes
with op.batch_alter_table("project", schema=None) as batch_op:
batch_op.create_index("ix_project_created_at", ["created_at"], unique=False)
batch_op.create_index("ix_project_name", ["name"], unique=True)
batch_op.create_index("ix_project_path", ["path"], unique=False)
batch_op.create_index("ix_project_permalink", ["permalink"], unique=True)
batch_op.create_index("ix_project_updated_at", ["updated_at"], unique=False)
else:
# For Postgres, we can simply drop the constraint
with op.batch_alter_table("project", schema=None) as batch_op:
batch_op.drop_constraint("project_is_default_key", type_="unique")
def downgrade() -> None:
@@ -0,0 +1,49 @@
"""Add mtime and size columns to Entity for sync optimization
Revision ID: 9d9c1cb7d8f5
Revises: a1b2c3d4e5f6
Create Date: 2025-10-20 05:07:55.173849
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = "9d9c1cb7d8f5"
down_revision: Union[str, None] = "a1b2c3d4e5f6"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table("entity", schema=None) as batch_op:
batch_op.add_column(sa.Column("mtime", sa.Float(), nullable=True))
batch_op.add_column(sa.Column("size", sa.Integer(), nullable=True))
batch_op.drop_constraint(batch_op.f("fk_entity_project_id"), type_="foreignkey")
batch_op.create_foreign_key(
batch_op.f("fk_entity_project_id"), "project", ["project_id"], ["id"]
)
# ### end Alembic commands ###
def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table("entity", schema=None) as batch_op:
batch_op.drop_constraint(batch_op.f("fk_entity_project_id"), type_="foreignkey")
batch_op.create_foreign_key(
batch_op.f("fk_entity_project_id"),
"project",
["project_id"],
["id"],
ondelete="CASCADE",
)
batch_op.drop_column("size")
batch_op.drop_column("mtime")
# ### end Alembic commands ###
@@ -21,6 +21,12 @@ depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Upgrade database schema to use new search index with content_stems and content_snippet."""
# This migration is SQLite-specific (FTS5 virtual tables)
# For Postgres, the search_index table is created via ORM models
connection = op.get_bind()
if connection.dialect.name != "sqlite":
return
# First, drop the existing search_index table
op.execute("DROP TABLE IF EXISTS search_index")
@@ -59,6 +65,13 @@ def upgrade() -> None:
def downgrade() -> None:
"""Downgrade database schema to use old search index."""
# This migration is SQLite-specific (FTS5 virtual tables)
# For Postgres, the search_index table is managed via ORM models
connection = op.get_bind()
if connection.dialect.name != "sqlite":
return
# Drop the updated search_index table
op.execute("DROP TABLE IF EXISTS search_index")
@@ -0,0 +1,37 @@
"""Add scan watermark tracking to Project
Revision ID: e7e1f4367280
Revises: 9d9c1cb7d8f5
Create Date: 2025-10-20 16:42:46.625075
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = "e7e1f4367280"
down_revision: Union[str, None] = "9d9c1cb7d8f5"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table("project", schema=None) as batch_op:
batch_op.add_column(sa.Column("last_scan_timestamp", sa.Float(), nullable=True))
batch_op.add_column(sa.Column("last_file_count", sa.Integer(), nullable=True))
# ### end Alembic commands ###
def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table("project", schema=None) as batch_op:
batch_op.drop_column("last_file_count")
batch_op.drop_column("last_scan_timestamp")
# ### end Alembic commands ###
+22 -3
View File
@@ -20,6 +20,16 @@ from basic_memory.api.routers import (
search,
prompt_router,
)
from basic_memory.api.v2.routers import (
knowledge_router as v2_knowledge,
project_router as v2_project,
memory_router as v2_memory,
search_router as v2_search,
resource_router as v2_resource,
directory_router as v2_directory,
prompt_router as v2_prompt,
importer_router as v2_importer,
)
from basic_memory.config import ConfigManager
from basic_memory.services.initialization import initialize_file_sync, initialize_app
@@ -66,8 +76,7 @@ app = FastAPI(
lifespan=lifespan,
)
# Include routers
# Include v1 routers
app.include_router(knowledge.router, prefix="/{project}")
app.include_router(memory.router, prefix="/{project}")
app.include_router(resource.router, prefix="/{project}")
@@ -77,7 +86,17 @@ app.include_router(directory_router.router, prefix="/{project}")
app.include_router(prompt_router.router, prefix="/{project}")
app.include_router(importer_router.router, prefix="/{project}")
# Project resource router works accross projects
# Include v2 routers (ID-based paths)
app.include_router(v2_knowledge, prefix="/v2/projects/{project_id}")
app.include_router(v2_memory, prefix="/v2/projects/{project_id}")
app.include_router(v2_search, prefix="/v2/projects/{project_id}")
app.include_router(v2_resource, prefix="/v2/projects/{project_id}")
app.include_router(v2_directory, prefix="/v2/projects/{project_id}")
app.include_router(v2_prompt, prefix="/v2/projects/{project_id}")
app.include_router(v2_importer, prefix="/v2/projects/{project_id}")
app.include_router(v2_project, prefix="/v2")
# Project resource router works across projects
app.include_router(project.project_resource_router)
app.include_router(management.router)
@@ -1,4 +1,11 @@
"""Router for knowledge graph operations."""
"""Router for knowledge graph operations.
DEPRECATED: This v1 API is deprecated and will be removed on June 30, 2026.
Please migrate to /v2/{project}/knowledge endpoints which use entity IDs instead
of path-based identifiers for improved performance and stability.
Migration guide: See docs/migration/v1-to-v2.md
"""
from typing import Annotated
@@ -25,7 +32,11 @@ from basic_memory.schemas import (
from basic_memory.schemas.request import EditEntityRequest, MoveEntityRequest
from basic_memory.schemas.base import Permalink, Entity
router = APIRouter(prefix="/knowledge", tags=["knowledge"])
router = APIRouter(
prefix="/knowledge",
tags=["knowledge"],
deprecated=True, # Marks entire router as deprecated in OpenAPI docs
)
async def resolve_relations_background(sync_service, entity_id: int, entity_permalink: str) -> None:
+130 -20
View File
@@ -1,7 +1,7 @@
"""Router for project management."""
import os
from fastapi import APIRouter, HTTPException, Path, Body, BackgroundTasks
from fastapi import APIRouter, HTTPException, Path, Body, BackgroundTasks, Response, Query
from typing import Optional
from loguru import logger
@@ -18,6 +18,7 @@ from basic_memory.schemas.project_info import (
ProjectInfoRequest,
ProjectStatusResponse,
)
from basic_memory.utils import normalize_project_path
# Router for resources in a specific project
# The ProjectPathDep is used in the path as a prefix, so the request path is like /{project}/project/info
@@ -49,8 +50,9 @@ async def get_project(
) # pragma: no cover
return ProjectItem(
id=found_project.id,
name=found_project.name,
path=found_project.path,
path=normalize_project_path(found_project.path),
is_default=found_project.is_default or False,
)
@@ -79,9 +81,17 @@ async def update_project(
raise HTTPException(status_code=400, detail="Path must be absolute")
# Get original project info for the response
old_project = await project_service.get_project(name)
if not old_project:
raise HTTPException(
status_code=400, detail=f"Project '{name}' not found in configuration"
)
old_project_info = ProjectItem(
name=name,
path=project_service.projects.get(name, ""),
id=old_project.id,
name=old_project.name,
path=old_project.path,
is_default=old_project.is_default or False,
)
if path:
@@ -90,14 +100,21 @@ async def update_project(
await project_service.update_project(name, is_active=is_active)
# Get updated project info
updated_path = path if path else project_service.projects.get(name, "")
updated_project = await project_service.get_project(name)
if not updated_project:
raise HTTPException(status_code=404, detail=f"Project '{name}' not found after update")
return ProjectStatusResponse(
message=f"Project '{name}' updated successfully",
status="success",
default=(name == project_service.default_project),
old_project=old_project_info,
new_project=ProjectItem(name=name, path=updated_path),
new_project=ProjectItem(
id=updated_project.id,
name=updated_project.name,
path=updated_project.path,
is_default=updated_project.is_default or False,
),
)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
@@ -109,6 +126,10 @@ async def sync_project(
background_tasks: BackgroundTasks,
sync_service: SyncServiceDep,
project_config: ProjectConfigDep,
force_full: bool = Query(
False, description="Force full scan, bypassing watermark optimization"
),
run_in_background: bool = Query(True, description="Run in background"),
):
"""Force project filesystem sync to database.
@@ -118,17 +139,32 @@ async def sync_project(
background_tasks: FastAPI background tasks
sync_service: Sync service for this project
project_config: Project configuration
force_full: If True, force a full scan even if watermark exists
run_in_background: If True, run sync in background and return immediately
Returns:
Response confirming sync was initiated
Response confirming sync was initiated (background) or SyncReportResponse (foreground)
"""
background_tasks.add_task(sync_service.sync, project_config.home, project_config.name)
logger.info(f"Filesystem sync initiated for project: {project_config.name}")
if run_in_background:
background_tasks.add_task(
sync_service.sync, project_config.home, project_config.name, force_full=force_full
)
logger.info(
f"Filesystem sync initiated for project: {project_config.name} (force_full={force_full})"
)
return {
"status": "sync_started",
"message": f"Filesystem sync initiated for project '{project_config.name}'",
}
return {
"status": "sync_started",
"message": f"Filesystem sync initiated for project '{project_config.name}'",
}
else:
report = await sync_service.sync(
project_config.home, project_config.name, force_full=force_full
)
logger.info(
f"Filesystem sync completed for project: {project_config.name} (force_full={force_full})"
)
return SyncReportResponse.from_sync_report(report)
@project_router.post("/status", response_model=SyncReportResponse)
@@ -166,8 +202,9 @@ async def list_projects(
project_items = [
ProjectItem(
id=project.id,
name=project.name,
path=project.path,
path=normalize_project_path(project.path),
is_default=project.is_default or False,
)
for project in projects
@@ -180,8 +217,9 @@ async def list_projects(
# Add a new project
@project_resource_router.post("/projects", response_model=ProjectStatusResponse)
@project_resource_router.post("/projects", response_model=ProjectStatusResponse, status_code=201)
async def add_project(
response: Response,
project_data: ProjectInfoRequest,
project_service: ProjectServiceDep,
) -> ProjectStatusResponse:
@@ -193,18 +231,57 @@ async def add_project(
Returns:
Response confirming the project was added
"""
# Check if project already exists before attempting to add
existing_project = await project_service.get_project(project_data.name)
if existing_project:
# Project exists - check if paths match for true idempotency
# Normalize paths for comparison (resolve symlinks, etc.)
from pathlib import Path
requested_path = Path(project_data.path).resolve()
existing_path = Path(existing_project.path).resolve()
if requested_path == existing_path:
# Same name, same path - return 200 OK (idempotent)
response.status_code = 200
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message=f"Project '{project_data.name}' already exists",
status="success",
default=existing_project.is_default or False,
new_project=ProjectItem(
id=existing_project.id,
name=existing_project.name,
path=existing_project.path,
is_default=existing_project.is_default or False,
),
)
else:
# Same name, different path - this is an error
raise HTTPException(
status_code=400,
detail=f"Project '{project_data.name}' already exists with different path. Existing: {existing_project.path}, Requested: {project_data.path}",
)
try: # pragma: no cover
# The service layer now handles cloud mode validation and path sanitization
await project_service.add_project(
project_data.name, project_data.path, set_default=project_data.set_default
)
# Fetch the newly created project to get its ID
new_project = await project_service.get_project(project_data.name)
if not new_project:
raise HTTPException(status_code=500, detail="Failed to retrieve newly created project")
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message=f"Project '{project_data.name}' added successfully",
status="success",
default=project_data.set_default,
new_project=ProjectItem(
name=project_data.name, path=project_data.path, is_default=project_data.set_default
id=new_project.id,
name=new_project.name,
path=new_project.path,
is_default=new_project.is_default or False,
),
)
except ValueError as e: # pragma: no cover
@@ -216,11 +293,15 @@ async def add_project(
async def remove_project(
project_service: ProjectServiceDep,
name: str = Path(..., description="Name of the project to remove"),
delete_notes: bool = Query(
False, description="If True, delete project directory from filesystem"
),
) -> ProjectStatusResponse:
"""Remove a project from configuration and database.
Args:
name: The name of the project to remove
delete_notes: If True, delete the project directory from the filesystem
Returns:
Response confirming the project was removed
@@ -232,13 +313,31 @@ async def remove_project(
status_code=404, detail=f"Project: '{name}' does not exist"
) # pragma: no cover
await project_service.remove_project(name)
# Check if trying to delete the default project
if name == project_service.default_project:
available_projects = await project_service.list_projects()
other_projects = [p.name for p in available_projects if p.name != name]
detail = f"Cannot delete default project '{name}'. "
if other_projects:
detail += (
f"Set another project as default first. Available: {', '.join(other_projects)}"
)
else:
detail += "This is the only project in your configuration."
raise HTTPException(status_code=400, detail=detail)
await project_service.remove_project(name, delete_notes=delete_notes)
return ProjectStatusResponse(
message=f"Project '{name}' removed successfully",
status="success",
default=False,
old_project=ProjectItem(name=old_project.name, path=old_project.path),
old_project=ProjectItem(
id=old_project.id,
name=old_project.name,
path=old_project.path,
is_default=old_project.is_default or False,
),
new_project=None,
)
except ValueError as e: # pragma: no cover
@@ -281,8 +380,14 @@ async def set_default_project(
message=f"Project '{name}' set as default successfully",
status="success",
default=True,
old_project=ProjectItem(name=default_name, path=default_project.path),
old_project=ProjectItem(
id=default_project.id,
name=default_name,
path=default_project.path,
is_default=False,
),
new_project=ProjectItem(
id=new_default_project.id,
name=name,
path=new_default_project.path,
is_default=True,
@@ -310,7 +415,12 @@ async def get_default_project(
status_code=404, detail=f"Default Project: '{default_name}' does not exist"
)
return ProjectItem(name=default_project.name, path=default_project.path, is_default=True)
return ProjectItem(
id=default_project.id,
name=default_project.name,
path=default_project.path,
is_default=True,
)
# Synchronize projects between config and database
@@ -151,6 +151,20 @@ async def write_resource(
try:
# Get content from request body
# Defensive type checking: ensure content is a string
# FastAPI should validate this, but if a dict somehow gets through
# (e.g., via JSON body parsing), we need to catch it here
if isinstance(content, dict):
logger.error(
f"Error writing resource {file_path}: "
f"content is a dict, expected string. Keys: {list(content.keys())}"
)
raise HTTPException(
status_code=400,
detail="content must be a string, not a dict. "
"Ensure request body is sent as raw string content, not JSON object.",
)
# Ensure it's UTF-8 string content
if isinstance(content, bytes): # pragma: no cover
content_str = content.decode("utf-8")
+25
View File
@@ -29,6 +29,7 @@ async def to_graph_context(
match item.type:
case SearchItemType.ENTITY:
return EntitySummary(
entity_id=item.id,
title=item.title, # pyright: ignore
permalink=item.permalink,
content=item.content,
@@ -37,6 +38,8 @@ async def to_graph_context(
)
case SearchItemType.OBSERVATION:
return ObservationSummary(
observation_id=item.id,
entity_id=item.entity_id, # pyright: ignore
title=item.title, # pyright: ignore
file_path=item.file_path,
category=item.category, # pyright: ignore
@@ -48,12 +51,16 @@ async def to_graph_context(
from_entity = await entity_repository.find_by_id(item.from_id) # pyright: ignore
to_entity = await entity_repository.find_by_id(item.to_id) if item.to_id else None
return RelationSummary(
relation_id=item.id,
entity_id=item.entity_id, # pyright: ignore
title=item.title, # pyright: ignore
file_path=item.file_path,
permalink=item.permalink, # pyright: ignore
relation_type=item.relation_type, # pyright: ignore
from_entity=from_entity.title if from_entity else None,
from_entity_id=item.from_id, # pyright: ignore
to_entity=to_entity.title if to_entity else None,
to_entity_id=item.to_id,
created_at=item.created_at,
)
case _: # pragma: no cover
@@ -111,6 +118,21 @@ async def to_search_results(entity_service: EntityService, results: List[SearchI
search_results = []
for r in results:
entities = await entity_service.get_entities_by_id([r.entity_id, r.from_id, r.to_id]) # pyright: ignore
# Determine which IDs to set based on type
entity_id = None
observation_id = None
relation_id = None
if r.type == SearchItemType.ENTITY:
entity_id = r.id
elif r.type == SearchItemType.OBSERVATION:
observation_id = r.id
entity_id = r.entity_id # Parent entity
elif r.type == SearchItemType.RELATION:
relation_id = r.id
entity_id = r.entity_id # Parent entity
search_results.append(
SearchResult(
title=r.title, # pyright: ignore
@@ -121,6 +143,9 @@ async def to_search_results(entity_service: EntityService, results: List[SearchI
content=r.content,
file_path=r.file_path,
metadata=r.metadata,
entity_id=entity_id,
observation_id=observation_id,
relation_id=relation_id,
category=r.category,
from_entity=entities[0].permalink if entities else None,
to_entity=entities[1].permalink if len(entities) > 1 else None,
+35
View File
@@ -0,0 +1,35 @@
"""API v2 module - ID-based entity references.
Version 2 of the Basic Memory API uses integer entity IDs as the primary
identifier for improved performance and stability.
Key changes from v1:
- Entity lookups use integer IDs instead of paths/permalinks
- Direct database queries instead of cascading resolution
- Stable references that don't change with file moves
- Better caching support
All v2 routers are registered with the /v2 prefix.
"""
from basic_memory.api.v2.routers import (
knowledge_router,
memory_router,
project_router,
resource_router,
search_router,
directory_router,
prompt_router,
importer_router,
)
__all__ = [
"knowledge_router",
"memory_router",
"project_router",
"resource_router",
"search_router",
"directory_router",
"prompt_router",
"importer_router",
]
@@ -0,0 +1,21 @@
"""V2 API routers."""
from basic_memory.api.v2.routers.knowledge_router import router as knowledge_router
from basic_memory.api.v2.routers.project_router import router as project_router
from basic_memory.api.v2.routers.memory_router import router as memory_router
from basic_memory.api.v2.routers.search_router import router as search_router
from basic_memory.api.v2.routers.resource_router import router as resource_router
from basic_memory.api.v2.routers.directory_router import router as directory_router
from basic_memory.api.v2.routers.prompt_router import router as prompt_router
from basic_memory.api.v2.routers.importer_router import router as importer_router
__all__ = [
"knowledge_router",
"project_router",
"memory_router",
"search_router",
"resource_router",
"directory_router",
"prompt_router",
"importer_router",
]
@@ -0,0 +1,93 @@
"""V2 Directory Router - ID-based directory tree operations.
This router provides directory structure browsing for projects using
integer project IDs instead of name-based identifiers.
Key improvements:
- Direct project lookup via integer primary keys
- Consistent with other v2 endpoints
- Better performance through indexed queries
"""
from typing import List, Optional
from fastapi import APIRouter, Query
from basic_memory.deps import DirectoryServiceV2Dep, ProjectIdPathDep
from basic_memory.schemas.directory import DirectoryNode
router = APIRouter(prefix="/directory", tags=["directory-v2"])
@router.get("/tree", response_model=DirectoryNode, response_model_exclude_none=True)
async def get_directory_tree(
directory_service: DirectoryServiceV2Dep,
project_id: ProjectIdPathDep,
):
"""Get hierarchical directory structure from the knowledge base.
Args:
directory_service: Service for directory operations
project_id: Numeric project ID
Returns:
DirectoryNode representing the root of the hierarchical tree structure
"""
# Get a hierarchical directory tree for the specific project
tree = await directory_service.get_directory_tree()
# Return the hierarchical tree
return tree
@router.get("/structure", response_model=DirectoryNode, response_model_exclude_none=True)
async def get_directory_structure(
directory_service: DirectoryServiceV2Dep,
project_id: ProjectIdPathDep,
):
"""Get folder structure for navigation (no files).
Optimized endpoint for folder tree navigation. Returns only directory nodes
without file metadata. For full tree with files, use /directory/tree.
Args:
directory_service: Service for directory operations
project_id: Numeric project ID
Returns:
DirectoryNode tree containing only folders (type="directory")
"""
structure = await directory_service.get_directory_structure()
return structure
@router.get("/list", response_model=List[DirectoryNode], response_model_exclude_none=True)
async def list_directory(
directory_service: DirectoryServiceV2Dep,
project_id: ProjectIdPathDep,
dir_name: str = Query("/", description="Directory path to list"),
depth: int = Query(1, ge=1, le=10, description="Recursion depth (1-10)"),
file_name_glob: Optional[str] = Query(
None, description="Glob pattern for filtering file names"
),
):
"""List directory contents with filtering and depth control.
Args:
directory_service: Service for directory operations
project_id: Numeric project ID
dir_name: Directory path to list (default: root "/")
depth: Recursion depth (1-10, default: 1 for immediate children only)
file_name_glob: Optional glob pattern for filtering file names (e.g., "*.md", "*meeting*")
Returns:
List of DirectoryNode objects matching the criteria
"""
# Get directory listing with filtering
nodes = await directory_service.list_directory(
dir_name=dir_name,
depth=depth,
file_name_glob=file_name_glob,
)
return nodes
@@ -0,0 +1,182 @@
"""V2 Import Router - ID-based data import operations.
This router uses v2 dependencies for consistent project ID handling.
Import endpoints use project_id in the path for consistency with other v2 endpoints.
"""
import json
import logging
from fastapi import APIRouter, Form, HTTPException, UploadFile, status
from basic_memory.deps import (
ChatGPTImporterV2Dep,
ClaudeConversationsImporterV2Dep,
ClaudeProjectsImporterV2Dep,
MemoryJsonImporterV2Dep,
ProjectIdPathDep,
)
from basic_memory.importers import Importer
from basic_memory.schemas.importer import (
ChatImportResult,
EntityImportResult,
ProjectImportResult,
)
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/import", tags=["import-v2"])
@router.post("/chatgpt", response_model=ChatImportResult)
async def import_chatgpt(
project_id: ProjectIdPathDep,
importer: ChatGPTImporterV2Dep,
file: UploadFile,
folder: str = Form("conversations"),
) -> ChatImportResult:
"""Import conversations from ChatGPT JSON export.
Args:
project_id: Validated numeric project ID from URL path
file: The ChatGPT conversations.json file.
folder: The folder to place the files in.
importer: ChatGPT importer instance.
Returns:
ChatImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
logger.info(f"V2 Importing ChatGPT conversations for project {project_id}")
return await import_file(importer, file, folder)
@router.post("/claude/conversations", response_model=ChatImportResult)
async def import_claude_conversations(
project_id: ProjectIdPathDep,
importer: ClaudeConversationsImporterV2Dep,
file: UploadFile,
folder: str = Form("conversations"),
) -> ChatImportResult:
"""Import conversations from Claude conversations.json export.
Args:
project_id: Validated numeric project ID from URL path
file: The Claude conversations.json file.
folder: The folder to place the files in.
importer: Claude conversations importer instance.
Returns:
ChatImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
logger.info(f"V2 Importing Claude conversations for project {project_id}")
return await import_file(importer, file, folder)
@router.post("/claude/projects", response_model=ProjectImportResult)
async def import_claude_projects(
project_id: ProjectIdPathDep,
importer: ClaudeProjectsImporterV2Dep,
file: UploadFile,
folder: str = Form("projects"),
) -> ProjectImportResult:
"""Import projects from Claude projects.json export.
Args:
project_id: Validated numeric project ID from URL path
file: The Claude projects.json file.
folder: The base folder to place the files in.
importer: Claude projects importer instance.
Returns:
ProjectImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
logger.info(f"V2 Importing Claude projects for project {project_id}")
return await import_file(importer, file, folder)
@router.post("/memory-json", response_model=EntityImportResult)
async def import_memory_json(
project_id: ProjectIdPathDep,
importer: MemoryJsonImporterV2Dep,
file: UploadFile,
folder: str = Form("conversations"),
) -> EntityImportResult:
"""Import entities and relations from a memory.json file.
Args:
project_id: Validated numeric project ID from URL path
file: The memory.json file.
folder: Optional destination folder within the project.
importer: Memory JSON importer instance.
Returns:
EntityImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
logger.info(f"V2 Importing memory.json for project {project_id}")
try:
file_data = []
file_bytes = await file.read()
file_str = file_bytes.decode("utf-8")
for line in file_str.splitlines():
json_data = json.loads(line)
file_data.append(json_data)
result = await importer.import_data(file_data, folder)
if not result.success: # pragma: no cover
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=result.error_message or "Import failed",
)
except Exception as e:
logger.exception("V2 Import failed")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Import failed: {str(e)}",
)
return result
async def import_file(importer: Importer, file: UploadFile, destination_folder: str):
"""Helper function to import a file using an importer instance.
Args:
importer: The importer instance to use
file: The file to import
destination_folder: Destination folder for imported content
Returns:
Import result from the importer
Raises:
HTTPException: If import fails
"""
try:
# Process file
json_data = json.load(file.file)
result = await importer.import_data(json_data, destination_folder)
if not result.success: # pragma: no cover
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=result.error_message or "Import failed",
)
return result
except Exception as e:
logger.exception("V2 Import failed")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Import failed: {str(e)}",
)
@@ -0,0 +1,415 @@
"""V2 Knowledge Router - ID-based entity operations.
This router provides ID-based CRUD operations for entities, replacing the
path-based identifiers used in v1 with direct integer ID lookups.
Key improvements:
- Direct database lookups via integer primary keys
- Stable references that don't change with file moves
- Better performance through indexed queries
- Simplified caching strategies
"""
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Response
from loguru import logger
from basic_memory.deps import (
EntityServiceV2Dep,
SearchServiceV2Dep,
LinkResolverV2Dep,
ProjectConfigV2Dep,
AppConfigDep,
SyncServiceV2Dep,
EntityRepositoryV2Dep,
ProjectIdPathDep,
)
from basic_memory.schemas import DeleteEntitiesResponse
from basic_memory.schemas.base import Entity
from basic_memory.schemas.request import EditEntityRequest
from basic_memory.schemas.v2 import (
EntityResolveRequest,
EntityResolveResponse,
EntityResponseV2,
MoveEntityRequestV2,
)
router = APIRouter(prefix="/knowledge", tags=["knowledge-v2"])
async def resolve_relations_background(sync_service, entity_id: int, entity_permalink: str) -> None:
"""Background task to resolve relations for a specific entity.
This runs asynchronously after the API response is sent, preventing
long delays when creating entities with many relations.
"""
try:
# Only resolve relations for the newly created entity
await sync_service.resolve_relations(entity_id=entity_id)
logger.debug(
f"Background: Resolved relations for entity {entity_permalink} (id={entity_id})"
)
except Exception as e:
# Log but don't fail - this is a background task
logger.warning(
f"Background: Failed to resolve relations for entity {entity_permalink}: {e}"
)
## Resolution endpoint
@router.post("/resolve", response_model=EntityResolveResponse)
async def resolve_identifier(
project_id: ProjectIdPathDep,
data: EntityResolveRequest,
link_resolver: LinkResolverV2Dep,
) -> EntityResolveResponse:
"""Resolve a string identifier (permalink, title, or path) to an entity ID.
This endpoint provides a bridge between v1-style identifiers and v2 entity IDs.
Use this to convert existing references to the new ID-based format.
Args:
data: Request containing the identifier to resolve
Returns:
Entity ID and metadata about how it was resolved
Raises:
HTTPException: 404 if identifier cannot be resolved
Example:
POST /v2/{project}/knowledge/resolve
{"identifier": "specs/search"}
Returns:
{
"entity_id": 123,
"permalink": "specs/search",
"file_path": "specs/search.md",
"title": "Search Specification",
"resolution_method": "permalink"
}
"""
logger.info(f"API v2 request: resolve_identifier for '{data.identifier}'")
# Try to resolve the identifier
entity = await link_resolver.resolve_link(data.identifier)
if not entity:
raise HTTPException(
status_code=404, detail=f"Could not resolve identifier: '{data.identifier}'"
)
# Determine resolution method
resolution_method = "search" # default
if data.identifier.isdigit():
resolution_method = "id"
elif entity.permalink == data.identifier:
resolution_method = "permalink"
elif entity.title == data.identifier:
resolution_method = "title"
elif entity.file_path == data.identifier:
resolution_method = "path"
result = EntityResolveResponse(
entity_id=entity.id,
permalink=entity.permalink,
file_path=entity.file_path,
title=entity.title,
resolution_method=resolution_method,
)
logger.info(
f"API v2 response: resolved '{data.identifier}' to entity_id={result.entity_id} via {resolution_method}"
)
return result
## Read endpoints
@router.get("/entities/{entity_id}", response_model=EntityResponseV2)
async def get_entity_by_id(
project_id: ProjectIdPathDep,
entity_id: int,
entity_repository: EntityRepositoryV2Dep,
) -> EntityResponseV2:
"""Get an entity by its numeric ID.
This is the primary entity retrieval method in v2, using direct database
lookups for maximum performance.
Args:
entity_id: Numeric entity ID
Returns:
Complete entity with observations and relations
Raises:
HTTPException: 404 if entity not found
"""
logger.info(f"API v2 request: get_entity_by_id entity_id={entity_id}")
entity = await entity_repository.get_by_id(entity_id)
if not entity:
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
result = EntityResponseV2.model_validate(entity)
logger.info(f"API v2 response: entity_id={entity_id}, title='{result.title}'")
return result
## Create endpoints
@router.post("/entities", response_model=EntityResponseV2)
async def create_entity(
project_id: ProjectIdPathDep,
data: Entity,
background_tasks: BackgroundTasks,
entity_service: EntityServiceV2Dep,
search_service: SearchServiceV2Dep,
) -> EntityResponseV2:
"""Create a new entity.
Args:
data: Entity data to create
Returns:
Created entity with generated ID
"""
logger.info(
"API v2 request", endpoint="create_entity", entity_type=data.entity_type, title=data.title
)
entity = await entity_service.create_entity(data)
# reindex
await search_service.index_entity(entity, background_tasks=background_tasks)
result = EntityResponseV2.model_validate(entity)
logger.info(
f"API v2 response: endpoint='create_entity' id={entity.id}, title={result.title}, permalink={result.permalink}, status_code=201"
)
return result
## Update endpoints
@router.put("/entities/{entity_id}", response_model=EntityResponseV2)
async def update_entity_by_id(
project_id: ProjectIdPathDep,
entity_id: int,
data: Entity,
response: Response,
background_tasks: BackgroundTasks,
entity_service: EntityServiceV2Dep,
search_service: SearchServiceV2Dep,
sync_service: SyncServiceV2Dep,
entity_repository: EntityRepositoryV2Dep,
) -> EntityResponseV2:
"""Update an entity by ID.
If the entity doesn't exist, it will be created (upsert behavior).
Args:
entity_id: Numeric entity ID
data: Updated entity data
Returns:
Updated entity
"""
logger.info(f"API v2 request: update_entity_by_id entity_id={entity_id}")
# Check if entity exists
existing = await entity_repository.get_by_id(entity_id)
created = existing is None
# Perform update or create
entity, _ = await entity_service.create_or_update_entity(data)
response.status_code = 201 if created else 200
# reindex
await search_service.index_entity(entity, background_tasks=background_tasks)
# Schedule relation resolution for new entities
if created:
background_tasks.add_task(
resolve_relations_background, sync_service, entity.id, entity.permalink or ""
)
result = EntityResponseV2.model_validate(entity)
logger.info(
f"API v2 response: entity_id={entity_id}, created={created}, status_code={response.status_code}"
)
return result
@router.patch("/entities/{entity_id}", response_model=EntityResponseV2)
async def edit_entity_by_id(
project_id: ProjectIdPathDep,
entity_id: int,
data: EditEntityRequest,
background_tasks: BackgroundTasks,
entity_service: EntityServiceV2Dep,
search_service: SearchServiceV2Dep,
entity_repository: EntityRepositoryV2Dep,
) -> EntityResponseV2:
"""Edit an existing entity by ID using operations like append, prepend, etc.
Args:
entity_id: Numeric entity ID
data: Edit operation details
Returns:
Updated entity
Raises:
HTTPException: 404 if entity not found, 400 if edit fails
"""
logger.info(
f"API v2 request: edit_entity_by_id entity_id={entity_id}, operation='{data.operation}'"
)
# Verify entity exists
entity = await entity_repository.get_by_id(entity_id)
if not entity:
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
try:
# Edit using the entity's permalink or path
identifier = entity.permalink or entity.file_path
updated_entity = await entity_service.edit_entity(
identifier=identifier,
operation=data.operation,
content=data.content,
section=data.section,
find_text=data.find_text,
expected_replacements=data.expected_replacements,
)
# Reindex
await search_service.index_entity(updated_entity, background_tasks=background_tasks)
result = EntityResponseV2.model_validate(updated_entity)
logger.info(
f"API v2 response: entity_id={entity_id}, operation='{data.operation}', status_code=200"
)
return result
except Exception as e:
logger.error(f"Error editing entity {entity_id}: {e}")
raise HTTPException(status_code=400, detail=str(e))
## Delete endpoints
@router.delete("/entities/{entity_id}", response_model=DeleteEntitiesResponse)
async def delete_entity_by_id(
project_id: ProjectIdPathDep,
entity_id: int,
background_tasks: BackgroundTasks,
entity_service: EntityServiceV2Dep,
entity_repository: EntityRepositoryV2Dep,
search_service=Depends(lambda: None), # Optional for now
) -> DeleteEntitiesResponse:
"""Delete an entity by ID.
Args:
entity_id: Numeric entity ID
Returns:
Deletion status
Note: Returns deleted=False if entity doesn't exist (idempotent)
"""
logger.info(f"API v2 request: delete_entity_by_id entity_id={entity_id}")
entity = await entity_repository.get_by_id(entity_id)
if entity is None:
logger.info(f"API v2 response: entity_id={entity_id} not found, deleted=False")
return DeleteEntitiesResponse(deleted=False)
# Delete the entity
deleted = await entity_service.delete_entity(entity_id)
# Remove from search index if search service available
if search_service:
background_tasks.add_task(search_service.handle_delete, entity)
logger.info(f"API v2 response: entity_id={entity_id}, deleted={deleted}")
return DeleteEntitiesResponse(deleted=deleted)
## Move endpoint
@router.put("/entities/{entity_id}/move", response_model=EntityResponseV2)
async def move_entity(
project_id: ProjectIdPathDep,
entity_id: int,
data: MoveEntityRequestV2,
background_tasks: BackgroundTasks,
entity_service: EntityServiceV2Dep,
entity_repository: EntityRepositoryV2Dep,
project_config: ProjectConfigV2Dep,
app_config: AppConfigDep,
search_service: SearchServiceV2Dep,
) -> EntityResponseV2:
"""Move an entity to a new file location.
V2 API uses entity ID in the URL path for stable references.
The entity ID will remain stable after the move.
Args:
project_id: Project ID from URL path
entity_id: Entity ID from URL path (primary identifier)
data: Move request with destination path only
Returns:
Updated entity with new file path
"""
logger.info(
f"API v2 request: move_entity entity_id={entity_id}, destination='{data.destination_path}'"
)
try:
# First, get the entity by ID to verify it exists
entity = await entity_repository.find_by_id(entity_id)
if not entity:
raise HTTPException(status_code=404, detail=f"Entity not found: {entity_id}")
# Move the entity using its current file path as identifier
moved_entity = await entity_service.move_entity(
identifier=entity.file_path, # Use file path for resolution
destination_path=data.destination_path,
project_config=project_config,
app_config=app_config,
)
# Reindex at new location
reindexed_entity = await entity_service.link_resolver.resolve_link(data.destination_path)
if reindexed_entity:
await search_service.index_entity(reindexed_entity, background_tasks=background_tasks)
result = EntityResponseV2.model_validate(moved_entity)
logger.info(
f"API v2 response: moved entity_id={moved_entity.id} to '{data.destination_path}'"
)
return result
except HTTPException:
raise
except Exception as e:
logger.error(f"Error moving entity: {e}")
raise HTTPException(status_code=400, detail=str(e))
@@ -0,0 +1,130 @@
"""V2 routes for memory:// URI operations.
This router uses integer project IDs for stable, efficient routing.
V1 uses string-based project names which are less efficient and less stable.
"""
from typing import Annotated, Optional
from fastapi import APIRouter, Query
from loguru import logger
from basic_memory.deps import ContextServiceV2Dep, EntityRepositoryV2Dep, ProjectIdPathDep
from basic_memory.schemas.base import TimeFrame, parse_timeframe
from basic_memory.schemas.memory import (
GraphContext,
normalize_memory_url,
)
from basic_memory.schemas.search import SearchItemType
from basic_memory.api.routers.utils import to_graph_context
# Note: No prefix here - it's added during registration as /v2/{project_id}/memory
router = APIRouter(tags=["memory"])
@router.get("/memory/recent", response_model=GraphContext)
async def recent(
project_id: ProjectIdPathDep,
context_service: ContextServiceV2Dep,
entity_repository: EntityRepositoryV2Dep,
type: Annotated[list[SearchItemType] | None, Query()] = None,
depth: int = 1,
timeframe: TimeFrame = "7d",
page: int = 1,
page_size: int = 10,
max_related: int = 10,
) -> GraphContext:
"""Get recent activity context for a project.
Args:
project_id: Validated numeric project ID from URL path
context_service: Context service scoped to project
entity_repository: Entity repository scoped to project
type: Types of items to include (entities, relations, observations)
depth: How many levels of related entities to include
timeframe: Time window for recent activity (e.g., "7d", "1 week")
page: Page number for pagination
page_size: Number of items per page
max_related: Maximum related entities to include per item
Returns:
GraphContext with recent activity and related entities
"""
# return all types by default
types = (
[SearchItemType.ENTITY, SearchItemType.RELATION, SearchItemType.OBSERVATION]
if not type
else type
)
logger.debug(
f"V2 Getting recent context for project {project_id}: `{types}` depth: `{depth}` timeframe: `{timeframe}` page: `{page}` page_size: `{page_size}` max_related: `{max_related}`"
)
# Parse timeframe
since = parse_timeframe(timeframe)
limit = page_size
offset = (page - 1) * page_size
# Build context
context = await context_service.build_context(
types=types, depth=depth, since=since, limit=limit, offset=offset, max_related=max_related
)
recent_context = await to_graph_context(
context, entity_repository=entity_repository, page=page, page_size=page_size
)
logger.debug(f"V2 Recent context: {recent_context.model_dump_json()}")
return recent_context
# get_memory_context needs to be declared last so other paths can match
@router.get("/memory/{uri:path}", response_model=GraphContext)
async def get_memory_context(
project_id: ProjectIdPathDep,
context_service: ContextServiceV2Dep,
entity_repository: EntityRepositoryV2Dep,
uri: str,
depth: int = 1,
timeframe: Optional[TimeFrame] = None,
page: int = 1,
page_size: int = 10,
max_related: int = 10,
) -> GraphContext:
"""Get rich context from memory:// URI.
V2 supports both legacy path-based URIs and new ID-based URIs:
- Legacy: memory://path/to/note
- ID-based: memory://id/123 or memory://123
Args:
project_id: Validated numeric project ID from URL path
context_service: Context service scoped to project
entity_repository: Entity repository scoped to project
uri: Memory URI path (e.g., "id/123", "123", or "path/to/note")
depth: How many levels of related entities to include
timeframe: Optional time window for filtering related content
page: Page number for pagination
page_size: Number of items per page
max_related: Maximum related entities to include
Returns:
GraphContext with the entity and its related context
"""
logger.debug(
f"V2 Getting context for project {project_id}, URI: `{uri}` depth: `{depth}` timeframe: `{timeframe}` page: `{page}` page_size: `{page_size}` max_related: `{max_related}`"
)
memory_url = normalize_memory_url(uri)
# Parse timeframe
since = parse_timeframe(timeframe) if timeframe else None
limit = page_size
offset = (page - 1) * page_size
# Build context
context = await context_service.build_context(
memory_url, depth=depth, since=since, limit=limit, offset=offset, max_related=max_related
)
return await to_graph_context(
context, entity_repository=entity_repository, page=page, page_size=page_size
)
@@ -0,0 +1,264 @@
"""V2 Project Router - ID-based project management operations.
This router provides ID-based CRUD operations for projects, replacing the
name-based identifiers used in v1 with direct integer ID lookups.
Key improvements:
- Direct database lookups via integer primary keys
- Stable references that don't change with project renames
- Better performance through indexed queries
- Consistent with v2 entity operations
"""
import os
from typing import Optional
from fastapi import APIRouter, HTTPException, Body, Query
from loguru import logger
from basic_memory.deps import (
ProjectServiceDep,
ProjectRepositoryDep,
ProjectIdPathDep,
)
from basic_memory.schemas.project_info import (
ProjectItem,
ProjectStatusResponse,
)
from basic_memory.utils import normalize_project_path
router = APIRouter(prefix="/projects", tags=["project_management-v2"])
@router.get("/{project_id}", response_model=ProjectItem)
async def get_project_by_id(
project_id: ProjectIdPathDep,
project_repository: ProjectRepositoryDep,
) -> ProjectItem:
"""Get project by its numeric ID.
This is the primary project retrieval method in v2, using direct database
lookups for maximum performance.
Args:
project_id: Numeric project ID
Returns:
Project information
Raises:
HTTPException: 404 if project not found
Example:
GET /v2/projects/3
"""
logger.info(f"API v2 request: get_project_by_id for project_id={project_id}")
project = await project_repository.get_by_id(project_id)
if not project:
raise HTTPException(status_code=404, detail=f"Project with ID {project_id} not found")
return ProjectItem(
id=project.id,
name=project.name,
path=normalize_project_path(project.path),
is_default=project.is_default or False,
)
@router.patch("/{project_id}", response_model=ProjectStatusResponse)
async def update_project_by_id(
project_id: ProjectIdPathDep,
project_service: ProjectServiceDep,
project_repository: ProjectRepositoryDep,
path: Optional[str] = Body(None, description="New absolute path for the project"),
is_active: Optional[bool] = Body(None, description="Status of the project (active/inactive)"),
) -> ProjectStatusResponse:
"""Update a project's information by ID.
Args:
project_id: Numeric project ID
path: Optional new absolute path for the project
is_active: Optional status update for the project
Returns:
Response confirming the project was updated
Raises:
HTTPException: 400 if validation fails, 404 if project not found
Example:
PATCH /v2/projects/3
{"path": "/new/path"}
"""
logger.info(f"API v2 request: update_project_by_id for project_id={project_id}")
try:
# Validate that path is absolute if provided
if path and not os.path.isabs(path):
raise HTTPException(status_code=400, detail="Path must be absolute")
# Get original project info for the response
old_project = await project_repository.get_by_id(project_id)
if not old_project:
raise HTTPException(status_code=404, detail=f"Project with ID {project_id} not found")
old_project_info = ProjectItem(
id=old_project.id,
name=old_project.name,
path=old_project.path,
is_default=old_project.is_default or False,
)
# Update using project name (service layer still uses names internally)
if path:
await project_service.move_project(old_project.name, path)
elif is_active is not None:
await project_service.update_project(old_project.name, is_active=is_active)
# Get updated project info
updated_project = await project_repository.get_by_id(project_id)
if not updated_project:
raise HTTPException(
status_code=404, detail=f"Project with ID {project_id} not found after update"
)
return ProjectStatusResponse(
message=f"Project '{updated_project.name}' updated successfully",
status="success",
default=(old_project.name == project_service.default_project),
old_project=old_project_info,
new_project=ProjectItem(
id=updated_project.id,
name=updated_project.name,
path=updated_project.path,
is_default=updated_project.is_default or False,
),
)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
@router.delete("/{project_id}", response_model=ProjectStatusResponse)
async def delete_project_by_id(
project_id: ProjectIdPathDep,
project_service: ProjectServiceDep,
project_repository: ProjectRepositoryDep,
delete_notes: bool = Query(
False, description="If True, delete project directory from filesystem"
),
) -> ProjectStatusResponse:
"""Delete a project by ID.
Args:
project_id: Numeric project ID
delete_notes: If True, delete the project directory from the filesystem
Returns:
Response confirming the project was deleted
Raises:
HTTPException: 400 if trying to delete default project, 404 if not found
Example:
DELETE /v2/projects/3?delete_notes=false
"""
logger.info(
f"API v2 request: delete_project_by_id for project_id={project_id}, delete_notes={delete_notes}"
)
try:
old_project = await project_repository.get_by_id(project_id)
if not old_project:
raise HTTPException(status_code=404, detail=f"Project with ID {project_id} not found")
# Check if trying to delete the default project
if old_project.name == project_service.default_project:
available_projects = await project_service.list_projects()
other_projects = [p.name for p in available_projects if p.id != project_id]
detail = f"Cannot delete default project '{old_project.name}'. "
if other_projects:
detail += (
f"Set another project as default first. Available: {', '.join(other_projects)}"
)
else:
detail += "This is the only project in your configuration."
raise HTTPException(status_code=400, detail=detail)
# Delete using project name (service layer still uses names internally)
await project_service.remove_project(old_project.name, delete_notes=delete_notes)
return ProjectStatusResponse(
message=f"Project '{old_project.name}' removed successfully",
status="success",
default=False,
old_project=ProjectItem(
id=old_project.id,
name=old_project.name,
path=old_project.path,
is_default=old_project.is_default or False,
),
new_project=None,
)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
@router.put("/{project_id}/default", response_model=ProjectStatusResponse)
async def set_default_project_by_id(
project_id: ProjectIdPathDep,
project_service: ProjectServiceDep,
project_repository: ProjectRepositoryDep,
) -> ProjectStatusResponse:
"""Set a project as the default project by ID.
Args:
project_id: Numeric project ID to set as default
Returns:
Response confirming the project was set as default
Raises:
HTTPException: 404 if project not found
Example:
PUT /v2/projects/3/default
"""
logger.info(f"API v2 request: set_default_project_by_id for project_id={project_id}")
try:
# Get the old default project
default_name = project_service.default_project
default_project = await project_service.get_project(default_name)
if not default_project:
raise HTTPException(
status_code=404, detail=f"Default Project: '{default_name}' does not exist"
)
# Get the new default project
new_default_project = await project_repository.get_by_id(project_id)
if not new_default_project:
raise HTTPException(status_code=404, detail=f"Project with ID {project_id} not found")
# Set as default using project name (service layer still uses names internally)
await project_service.set_default_project(new_default_project.name)
return ProjectStatusResponse(
message=f"Project '{new_default_project.name}' set as default successfully",
status="success",
default=True,
old_project=ProjectItem(
id=default_project.id,
name=default_name,
path=default_project.path,
is_default=False,
),
new_project=ProjectItem(
id=new_default_project.id,
name=new_default_project.name,
path=new_default_project.path,
is_default=True,
),
)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
@@ -0,0 +1,270 @@
"""V2 Prompt Router - ID-based prompt generation operations.
This router uses v2 dependencies for consistent project ID handling.
Prompt endpoints are action-based (not resource-based), so they don't
have entity IDs in URLs - they generate formatted prompts from queries.
"""
from datetime import datetime, timezone
from fastapi import APIRouter, HTTPException, status
from loguru import logger
from basic_memory.api.routers.utils import to_graph_context, to_search_results
from basic_memory.api.template_loader import template_loader
from basic_memory.schemas.base import parse_timeframe
from basic_memory.deps import (
ContextServiceV2Dep,
EntityRepositoryV2Dep,
SearchServiceV2Dep,
EntityServiceV2Dep,
ProjectIdPathDep,
)
from basic_memory.schemas.prompt import (
ContinueConversationRequest,
SearchPromptRequest,
PromptResponse,
PromptMetadata,
)
from basic_memory.schemas.search import SearchItemType, SearchQuery
router = APIRouter(prefix="/prompt", tags=["prompt-v2"])
@router.post("/continue-conversation", response_model=PromptResponse)
async def continue_conversation(
project_id: ProjectIdPathDep,
search_service: SearchServiceV2Dep,
entity_service: EntityServiceV2Dep,
context_service: ContextServiceV2Dep,
entity_repository: EntityRepositoryV2Dep,
request: ContinueConversationRequest,
) -> PromptResponse:
"""Generate a prompt for continuing a conversation.
This endpoint takes a topic and/or timeframe and generates a prompt with
relevant context from the knowledge base.
Args:
project_id: Validated numeric project ID from URL path
request: The request parameters
Returns:
Formatted continuation prompt with context
"""
logger.info(
f"V2 Generating continue conversation prompt for project {project_id}, "
f"topic: {request.topic}, timeframe: {request.timeframe}"
)
since = parse_timeframe(request.timeframe) if request.timeframe else None
# Initialize search results
search_results = []
# Get data needed for template
if request.topic:
query = SearchQuery(text=request.topic, after_date=request.timeframe)
results = await search_service.search(query, limit=request.search_items_limit)
search_results = await to_search_results(entity_service, results)
# Build context from results
all_hierarchical_results = []
for result in search_results:
if hasattr(result, "permalink") and result.permalink:
# Get hierarchical context using the new dataclass-based approach
context_result = await context_service.build_context(
result.permalink,
depth=request.depth,
since=since,
max_related=request.related_items_limit,
include_observations=True, # Include observations for entities
)
# Process results into the schema format
graph_context = await to_graph_context(
context_result, entity_repository=entity_repository
)
# Add results to our collection (limit to top results for each permalink)
if graph_context.results:
all_hierarchical_results.extend(graph_context.results[:3])
# Limit to a reasonable number of total results
all_hierarchical_results = all_hierarchical_results[:10]
template_context = {
"topic": request.topic,
"timeframe": request.timeframe,
"hierarchical_results": all_hierarchical_results,
"has_results": len(all_hierarchical_results) > 0,
}
else:
# If no topic, get recent activity
context_result = await context_service.build_context(
types=[SearchItemType.ENTITY],
depth=request.depth,
since=since,
max_related=request.related_items_limit,
include_observations=True,
)
recent_context = await to_graph_context(context_result, entity_repository=entity_repository)
hierarchical_results = recent_context.results[:5] # Limit to top 5 recent items
template_context = {
"topic": f"Recent Activity from ({request.timeframe})",
"timeframe": request.timeframe,
"hierarchical_results": hierarchical_results,
"has_results": len(hierarchical_results) > 0,
}
try:
# Render template
rendered_prompt = await template_loader.render(
"prompts/continue_conversation.hbs", template_context
)
# Calculate metadata
# Count items of different types
observation_count = 0
relation_count = 0
entity_count = 0
# Get the hierarchical results from the template context
hierarchical_results_for_count = template_context.get("hierarchical_results", [])
# For topic-based search
if request.topic:
for item in hierarchical_results_for_count:
if hasattr(item, "observations"):
observation_count += len(item.observations) if item.observations else 0
if hasattr(item, "related_results"):
for related in item.related_results or []:
if hasattr(related, "type"):
if related.type == "relation":
relation_count += 1
elif related.type == "entity": # pragma: no cover
entity_count += 1 # pragma: no cover
# For recent activity
else:
for item in hierarchical_results_for_count:
if hasattr(item, "observations"):
observation_count += len(item.observations) if item.observations else 0
if hasattr(item, "related_results"):
for related in item.related_results or []:
if hasattr(related, "type"):
if related.type == "relation":
relation_count += 1
elif related.type == "entity": # pragma: no cover
entity_count += 1 # pragma: no cover
# Build metadata
metadata = {
"query": request.topic,
"timeframe": request.timeframe,
"search_count": len(search_results)
if request.topic
else 0, # Original search results count
"context_count": len(hierarchical_results_for_count),
"observation_count": observation_count,
"relation_count": relation_count,
"total_items": (
len(hierarchical_results_for_count)
+ observation_count
+ relation_count
+ entity_count
),
"search_limit": request.search_items_limit,
"context_depth": request.depth,
"related_limit": request.related_items_limit,
"generated_at": datetime.now(timezone.utc).isoformat(),
}
prompt_metadata = PromptMetadata(**metadata)
return PromptResponse(
prompt=rendered_prompt, context=template_context, metadata=prompt_metadata
)
except Exception as e:
logger.error(f"Error rendering continue conversation template: {e}")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Error rendering prompt template: {str(e)}",
)
@router.post("/search", response_model=PromptResponse)
async def search_prompt(
project_id: ProjectIdPathDep,
search_service: SearchServiceV2Dep,
entity_service: EntityServiceV2Dep,
request: SearchPromptRequest,
page: int = 1,
page_size: int = 10,
) -> PromptResponse:
"""Generate a prompt for search results.
This endpoint takes a search query and formats the results into a helpful
prompt with context and suggestions.
Args:
project_id: Validated numeric project ID from URL path
request: The search parameters
page: The page number for pagination
page_size: The number of results per page, defaults to 10
Returns:
Formatted search results prompt with context
"""
logger.info(
f"V2 Generating search prompt for project {project_id}, "
f"query: {request.query}, timeframe: {request.timeframe}"
)
limit = page_size
offset = (page - 1) * page_size
query = SearchQuery(text=request.query, after_date=request.timeframe)
results = await search_service.search(query, limit=limit, offset=offset)
search_results = await to_search_results(entity_service, results)
template_context = {
"query": request.query,
"timeframe": request.timeframe,
"results": search_results,
"has_results": len(search_results) > 0,
"result_count": len(search_results),
}
try:
# Render template
rendered_prompt = await template_loader.render("prompts/search.hbs", template_context)
# Build metadata
metadata = {
"query": request.query,
"timeframe": request.timeframe,
"search_count": len(search_results),
"context_count": len(search_results),
"observation_count": 0, # Search results don't include observations
"relation_count": 0, # Search results don't include relations
"total_items": len(search_results),
"search_limit": limit,
"context_depth": 0, # No context depth for basic search
"related_limit": 0, # No related items for basic search
"generated_at": datetime.now(timezone.utc).isoformat(),
}
prompt_metadata = PromptMetadata(**metadata)
return PromptResponse(
prompt=rendered_prompt, context=template_context, metadata=prompt_metadata
)
except Exception as e:
logger.error(f"Error rendering search template: {e}")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Error rendering prompt template: {str(e)}",
)
@@ -0,0 +1,292 @@
"""V2 Resource Router - ID-based resource content operations.
This router uses entity IDs for all operations, with file paths in request bodies
when needed. This is consistent with v2's ID-first design.
Key differences from v1:
- Uses integer entity IDs in URL paths instead of file paths
- File paths are in request bodies for create/update operations
- More RESTful: POST for create, PUT for update, GET for read
"""
from pathlib import Path
from fastapi import APIRouter, HTTPException
from fastapi.responses import FileResponse
from loguru import logger
from basic_memory.deps import (
ProjectConfigV2Dep,
EntityServiceV2Dep,
FileServiceV2Dep,
EntityRepositoryV2Dep,
SearchServiceV2Dep,
ProjectIdPathDep,
)
from basic_memory.models.knowledge import Entity as EntityModel
from basic_memory.schemas.v2.resource import (
CreateResourceRequest,
UpdateResourceRequest,
ResourceResponse,
)
from basic_memory.utils import validate_project_path
from datetime import datetime
router = APIRouter(prefix="/resource", tags=["resources-v2"])
@router.get("/{entity_id}")
async def get_resource_content(
project_id: ProjectIdPathDep,
entity_id: int,
config: ProjectConfigV2Dep,
entity_service: EntityServiceV2Dep,
file_service: FileServiceV2Dep,
) -> FileResponse:
"""Get raw resource content by entity ID.
Args:
project_id: Validated numeric project ID from URL path
entity_id: Numeric entity ID
config: Project configuration
entity_service: Entity service for fetching entity data
file_service: File service for reading file content
Returns:
FileResponse with entity content
Raises:
HTTPException: 404 if entity or file not found
"""
logger.debug(f"V2 Getting content for project {project_id}, entity_id: {entity_id}")
# Get entity by ID
entities = await entity_service.get_entities_by_id([entity_id])
if not entities:
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
entity = entities[0]
# Validate entity file path to prevent path traversal
project_path = Path(config.home)
if not validate_project_path(entity.file_path, project_path):
logger.error(f"Invalid file path in entity {entity.id}: {entity.file_path}")
raise HTTPException(
status_code=500,
detail="Entity contains invalid file path",
)
file_path = Path(f"{config.home}/{entity.file_path}")
if not file_path.exists():
raise HTTPException(
status_code=404,
detail=f"File not found: {file_path}",
)
return FileResponse(path=file_path)
@router.post("", response_model=ResourceResponse)
async def create_resource(
project_id: ProjectIdPathDep,
data: CreateResourceRequest,
config: ProjectConfigV2Dep,
file_service: FileServiceV2Dep,
entity_repository: EntityRepositoryV2Dep,
search_service: SearchServiceV2Dep,
) -> ResourceResponse:
"""Create a new resource file.
Args:
project_id: Validated numeric project ID from URL path
data: Create resource request with file_path and content
config: Project configuration
file_service: File service for writing files
entity_repository: Entity repository for creating entities
search_service: Search service for indexing
Returns:
ResourceResponse with file information including entity_id
Raises:
HTTPException: 400 for invalid file paths, 409 if file already exists
"""
try:
# Validate path to prevent path traversal attacks
project_path = Path(config.home)
if not validate_project_path(data.file_path, project_path):
logger.warning(
f"Invalid file path attempted: {data.file_path} in project {config.name}"
)
raise HTTPException(
status_code=400,
detail=f"Invalid file path: {data.file_path}. "
"Path must be relative and stay within project boundaries.",
)
# Check if entity already exists
existing_entity = await entity_repository.get_by_file_path(data.file_path)
if existing_entity:
raise HTTPException(
status_code=409,
detail=f"Resource already exists at {data.file_path} with entity_id {existing_entity.id}. "
f"Use PUT /resource/{existing_entity.id} to update it.",
)
# Get full file path
full_path = Path(f"{config.home}/{data.file_path}")
# Ensure parent directory exists
full_path.parent.mkdir(parents=True, exist_ok=True)
# Write content to file
checksum = await file_service.write_file(full_path, data.content)
# Get file info
file_stats = file_service.file_stats(full_path)
# Determine file details
file_name = Path(data.file_path).name
content_type = file_service.content_type(full_path)
entity_type = "canvas" if data.file_path.endswith(".canvas") else "file"
# Create a new entity model
entity = EntityModel(
title=file_name,
entity_type=entity_type,
content_type=content_type,
file_path=data.file_path,
checksum=checksum,
created_at=datetime.fromtimestamp(file_stats.st_ctime).astimezone(),
updated_at=datetime.fromtimestamp(file_stats.st_mtime).astimezone(),
)
entity = await entity_repository.add(entity)
# Index the file for search
await search_service.index_entity(entity) # pyright: ignore
# Return success response
return ResourceResponse(
entity_id=entity.id,
file_path=data.file_path,
checksum=checksum,
size=file_stats.st_size,
created_at=file_stats.st_ctime,
modified_at=file_stats.st_mtime,
)
except HTTPException:
# Re-raise HTTP exceptions without wrapping
raise
except Exception as e: # pragma: no cover
logger.error(f"Error creating resource {data.file_path}: {e}")
raise HTTPException(status_code=500, detail=f"Failed to create resource: {str(e)}")
@router.put("/{entity_id}", response_model=ResourceResponse)
async def update_resource(
project_id: ProjectIdPathDep,
entity_id: int,
data: UpdateResourceRequest,
config: ProjectConfigV2Dep,
file_service: FileServiceV2Dep,
entity_repository: EntityRepositoryV2Dep,
search_service: SearchServiceV2Dep,
) -> ResourceResponse:
"""Update an existing resource by entity ID.
Can update content and optionally move the file to a new path.
Args:
project_id: Validated numeric project ID from URL path
entity_id: Entity ID of the resource to update
data: Update resource request with content and optional new file_path
config: Project configuration
file_service: File service for writing files
entity_repository: Entity repository for updating entities
search_service: Search service for indexing
Returns:
ResourceResponse with updated file information
Raises:
HTTPException: 404 if entity not found, 400 for invalid paths
"""
try:
# Get existing entity
entity = await entity_repository.get_by_id(entity_id)
if not entity:
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
# Determine target file path
target_file_path = data.file_path if data.file_path else entity.file_path
# Validate path to prevent path traversal attacks
project_path = Path(config.home)
if not validate_project_path(target_file_path, project_path):
logger.warning(
f"Invalid file path attempted: {target_file_path} in project {config.name}"
)
raise HTTPException(
status_code=400,
detail=f"Invalid file path: {target_file_path}. "
"Path must be relative and stay within project boundaries.",
)
# Get full paths
old_full_path = Path(f"{config.home}/{entity.file_path}")
new_full_path = Path(f"{config.home}/{target_file_path}")
# If moving file, handle the move
if data.file_path and data.file_path != entity.file_path:
# Ensure new parent directory exists
new_full_path.parent.mkdir(parents=True, exist_ok=True)
# If old file exists, remove it
if old_full_path.exists():
old_full_path.unlink()
else:
# Ensure directory exists for in-place update
new_full_path.parent.mkdir(parents=True, exist_ok=True)
# Write content to target file
checksum = await file_service.write_file(new_full_path, data.content)
# Get file info
file_stats = file_service.file_stats(new_full_path)
# Determine file details
file_name = Path(target_file_path).name
content_type = file_service.content_type(new_full_path)
entity_type = "canvas" if target_file_path.endswith(".canvas") else "file"
# Update entity
updated_entity = await entity_repository.update(
entity_id,
{
"title": file_name,
"entity_type": entity_type,
"content_type": content_type,
"file_path": target_file_path,
"checksum": checksum,
"updated_at": datetime.fromtimestamp(file_stats.st_mtime).astimezone(),
},
)
# Index the updated file for search
await search_service.index_entity(updated_entity) # pyright: ignore
# Return success response
return ResourceResponse(
entity_id=entity_id,
file_path=target_file_path,
checksum=checksum,
size=file_stats.st_size,
created_at=file_stats.st_ctime,
modified_at=file_stats.st_mtime,
)
except HTTPException:
# Re-raise HTTP exceptions without wrapping
raise
except Exception as e: # pragma: no cover
logger.error(f"Error updating resource {entity_id}: {e}")
raise HTTPException(status_code=500, detail=f"Failed to update resource: {str(e)}")
@@ -0,0 +1,73 @@
"""V2 router for search operations.
This router uses integer project IDs for stable, efficient routing.
V1 uses string-based project names which are less efficient and less stable.
"""
from fastapi import APIRouter, BackgroundTasks
from basic_memory.api.routers.utils import to_search_results
from basic_memory.schemas.search import SearchQuery, SearchResponse
from basic_memory.deps import SearchServiceV2Dep, EntityServiceV2Dep, ProjectIdPathDep
# Note: No prefix here - it's added during registration as /v2/{project_id}/search
router = APIRouter(tags=["search"])
@router.post("/search/", response_model=SearchResponse)
async def search(
project_id: ProjectIdPathDep,
query: SearchQuery,
search_service: SearchServiceV2Dep,
entity_service: EntityServiceV2Dep,
page: int = 1,
page_size: int = 10,
):
"""Search across all knowledge and documents in a project.
V2 uses integer project IDs for improved performance and stability.
Args:
project_id: Validated numeric project ID from URL path
query: Search query parameters (text, filters, etc.)
search_service: Search service scoped to project
entity_service: Entity service scoped to project
page: Page number for pagination
page_size: Number of results per page
Returns:
SearchResponse with paginated search results
"""
limit = page_size
offset = (page - 1) * page_size
results = await search_service.search(query, limit=limit, offset=offset)
search_results = await to_search_results(entity_service, results)
return SearchResponse(
results=search_results,
current_page=page,
page_size=page_size,
)
@router.post("/search/reindex")
async def reindex(
project_id: ProjectIdPathDep,
background_tasks: BackgroundTasks,
search_service: SearchServiceV2Dep,
):
"""Recreate and populate the search index for a project.
This is a background operation that rebuilds the search index
from scratch. Useful after bulk updates or if the index becomes
corrupted.
Args:
project_id: Validated numeric project ID from URL path
background_tasks: FastAPI background tasks handler
search_service: Search service scoped to project
Returns:
Status message indicating reindex has been initiated
"""
await search_service.reindex_all(background_tasks=background_tasks)
return {"status": "ok", "message": "Reindex initiated"}
+5 -5
View File
@@ -77,7 +77,7 @@ class CLIAuth:
verification_uri = device_response["verification_uri"]
verification_uri_complete = device_response.get("verification_uri_complete")
console.print("\n[bold blue]🔐 Authentication Required[/bold blue]")
console.print("\n[bold blue]Authentication Required[/bold blue]")
console.print("\nTo authenticate, please visit:")
console.print(f"[bold cyan]{verification_uri}[/bold cyan]")
console.print(f"\nAnd enter this code: [bold yellow]{user_code}[/bold yellow]")
@@ -171,7 +171,7 @@ class CLIAuth:
# Secure the token file
os.chmod(self.token_file, 0o600)
console.print(f"[green]Tokens saved to {self.token_file}[/green]")
console.print(f"[green]Tokens saved to {self.token_file}[/green]")
def load_tokens(self) -> dict | None:
"""Load tokens from .bm-auth.json file."""
@@ -233,7 +233,7 @@ class CLIAuth:
if new_tokens:
# Save new tokens (may include rotated refresh token)
self.save_tokens(new_tokens)
console.print("[green]Token refreshed successfully[/green]")
console.print("[green]Token refreshed successfully[/green]")
return new_tokens["access_token"]
else:
console.print("[yellow]Token refresh failed. Please run 'login' again.[/yellow]")
@@ -265,13 +265,13 @@ class CLIAuth:
# Step 4: Save tokens
self.save_tokens(tokens)
console.print("\n[green]Successfully authenticated with Basic Memory Cloud![/green]")
console.print("\n[green]Successfully authenticated with Basic Memory Cloud![/green]")
return True
def logout(self) -> None:
"""Remove stored authentication tokens."""
if self.token_file.exists():
self.token_file.unlink()
console.print("[green]Logged out successfully[/green]")
console.print("[green]Logged out successfully[/green]")
else:
console.print("[yellow]No stored authentication found[/yellow]")
+1 -2
View File
@@ -1,11 +1,10 @@
"""CLI commands for basic-memory."""
from . import status, sync, db, import_memory_json, mcp, import_claude_conversations
from . import status, db, import_memory_json, mcp, import_claude_conversations
from . import import_claude_projects, import_chatgpt, tool, project
__all__ = [
"status",
"sync",
"db",
"import_memory_json",
"mcp",
@@ -2,4 +2,5 @@
# Import all commands to register them with typer
from basic_memory.cli.commands.cloud.core_commands import * # noqa: F401,F403
from basic_memory.cli.commands.cloud.api_client import get_authenticated_headers # noqa: F401
from basic_memory.cli.commands.cloud.api_client import get_authenticated_headers, get_cloud_config # noqa: F401
from basic_memory.cli.commands.cloud.upload_command import * # noqa: F401,F403
@@ -1,33 +1,11 @@
"""Cloud bisync commands for Basic Memory CLI."""
"""Cloud bisync utility functions for Basic Memory CLI."""
import asyncio
import subprocess
import time
from datetime import datetime
from pathlib import Path
from typing import Optional
import typer
from rich.console import Console
from rich.table import Table
from basic_memory.cli.commands.cloud.api_client import CloudAPIError, make_api_request
from basic_memory.cli.commands.cloud.rclone_config import (
add_tenant_to_rclone_config,
)
from basic_memory.cli.commands.cloud.rclone_installer import RcloneInstallError, install_rclone
from basic_memory.cli.commands.cloud.api_client import make_api_request
from basic_memory.config import ConfigManager
from basic_memory.ignore_utils import get_bmignore_path, create_default_bmignore
from basic_memory.schemas.cloud import (
TenantMountInfo,
MountCredentials,
CloudProjectList,
CloudProjectCreateRequest,
CloudProjectCreateResponse,
)
from basic_memory.utils import generate_permalink
console = Console()
from basic_memory.ignore_utils import create_default_bmignore, get_bmignore_path
from basic_memory.schemas.cloud import MountCredentials, TenantMountInfo
class BisyncError(Exception):
@@ -36,52 +14,6 @@ class BisyncError(Exception):
pass
class RcloneBisyncProfile:
"""Bisync profile with safety settings."""
def __init__(
self,
name: str,
conflict_resolve: str,
max_delete: int,
check_access: bool,
description: str,
extra_args: Optional[list[str]] = None,
):
self.name = name
self.conflict_resolve = conflict_resolve
self.max_delete = max_delete
self.check_access = check_access
self.description = description
self.extra_args = extra_args or []
# Bisync profiles based on SPEC-9 Phase 2.1
BISYNC_PROFILES = {
"safe": RcloneBisyncProfile(
name="safe",
conflict_resolve="none",
max_delete=10,
check_access=False,
description="Safe mode with conflict preservation (keeps both versions)",
),
"balanced": RcloneBisyncProfile(
name="balanced",
conflict_resolve="newer",
max_delete=25,
check_access=False,
description="Balanced mode - auto-resolve to newer file (recommended)",
),
"fast": RcloneBisyncProfile(
name="fast",
conflict_resolve="newer",
max_delete=50,
check_access=False,
description="Fast mode for rapid iteration (skip verification)",
),
}
async def get_mount_info() -> TenantMountInfo:
"""Get current tenant information from cloud API."""
try:
@@ -110,128 +42,6 @@ async def generate_mount_credentials(tenant_id: str) -> MountCredentials:
raise BisyncError(f"Failed to generate credentials: {e}") from e
async def fetch_cloud_projects() -> CloudProjectList:
"""Fetch list of projects from cloud API.
Returns:
CloudProjectList with projects from cloud
"""
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
response = await make_api_request(method="GET", url=f"{host_url}/proxy/projects/projects")
return CloudProjectList.model_validate(response.json())
except Exception as e:
raise BisyncError(f"Failed to fetch cloud projects: {e}") from e
def scan_local_directories(sync_dir: Path) -> list[str]:
"""Scan local sync directory for project folders.
Args:
sync_dir: Path to bisync directory
Returns:
List of directory names (project names)
"""
if not sync_dir.exists():
return []
directories = []
for item in sync_dir.iterdir():
if item.is_dir() and not item.name.startswith("."):
directories.append(item.name)
return directories
async def create_cloud_project(project_name: str) -> CloudProjectCreateResponse:
"""Create a new project on cloud.
Args:
project_name: Name of project to create
Returns:
CloudProjectCreateResponse with project details from API
"""
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
# Use generate_permalink to ensure consistent naming
project_path = generate_permalink(project_name)
project_data = CloudProjectCreateRequest(
name=project_name,
path=project_path,
set_default=False,
)
response = await make_api_request(
method="POST",
url=f"{host_url}/proxy/projects/projects",
headers={"Content-Type": "application/json"},
json_data=project_data.model_dump(),
)
return CloudProjectCreateResponse.model_validate(response.json())
except Exception as e:
raise BisyncError(f"Failed to create cloud project '{project_name}': {e}") from e
def get_bisync_state_path(tenant_id: str) -> Path:
"""Get path to bisync state directory."""
return Path.home() / ".basic-memory" / "bisync-state" / tenant_id
def get_bisync_directory() -> Path:
"""Get bisync directory from config.
Returns:
Path to bisync directory (default: ~/basic-memory-cloud-sync)
"""
config_manager = ConfigManager()
config = config_manager.config
sync_dir = config.bisync_config.get("sync_dir", str(Path.home() / "basic-memory-cloud-sync"))
return Path(sync_dir).expanduser().resolve()
def validate_bisync_directory(bisync_dir: Path) -> None:
"""Validate bisync directory doesn't conflict with mount.
Raises:
BisyncError: If bisync directory conflicts with mount directory
"""
# Get fixed mount directory
mount_dir = (Path.home() / "basic-memory-cloud").resolve()
# Check if bisync dir is the same as mount dir
if bisync_dir == mount_dir:
raise BisyncError(
f"Cannot use {bisync_dir} for bisync - it's the mount directory!\n"
f"Mount and bisync must use different directories.\n\n"
f"Options:\n"
f" 1. Use default: ~/basic-memory-cloud-sync/\n"
f" 2. Specify different directory: --dir ~/my-sync-folder"
)
# Check if mount is active at this location
result = subprocess.run(["mount"], capture_output=True, text=True)
if str(bisync_dir) in result.stdout and "rclone" in result.stdout:
raise BisyncError(
f"{bisync_dir} is currently mounted via 'bm cloud mount'\n"
f"Cannot use mounted directory for bisync.\n\n"
f"Either:\n"
f" 1. Unmount first: bm cloud unmount\n"
f" 2. Use different directory for bisync"
)
def convert_bmignore_to_rclone_filters() -> Path:
"""Convert .bmignore patterns to rclone filter format.
@@ -298,521 +108,3 @@ def get_bisync_filter_path() -> Path:
Path to rclone filter file
"""
return convert_bmignore_to_rclone_filters()
def bisync_state_exists(tenant_id: str) -> bool:
"""Check if bisync state exists (has been initialized)."""
state_path = get_bisync_state_path(tenant_id)
return state_path.exists() and any(state_path.iterdir())
def build_bisync_command(
tenant_id: str,
bucket_name: str,
local_path: Path,
profile: RcloneBisyncProfile,
dry_run: bool = False,
resync: bool = False,
verbose: bool = False,
) -> list[str]:
"""Build rclone bisync command with profile settings."""
# Sync with the entire bucket root (all projects)
rclone_remote = f"basic-memory-{tenant_id}:{bucket_name}"
filter_path = get_bisync_filter_path()
state_path = get_bisync_state_path(tenant_id)
# Ensure state directory exists
state_path.mkdir(parents=True, exist_ok=True)
cmd = [
"rclone",
"bisync",
str(local_path),
rclone_remote,
"--create-empty-src-dirs",
"--resilient",
f"--conflict-resolve={profile.conflict_resolve}",
f"--max-delete={profile.max_delete}",
"--filters-file",
str(filter_path),
"--workdir",
str(state_path),
]
# Add verbosity flags
if verbose:
cmd.append("--verbose") # Full details with file-by-file output
else:
# Show progress bar during transfers
cmd.append("--progress")
if profile.check_access:
cmd.append("--check-access")
if dry_run:
cmd.append("--dry-run")
if resync:
cmd.append("--resync")
cmd.extend(profile.extra_args)
return cmd
def setup_cloud_bisync(sync_dir: Optional[str] = None) -> None:
"""Set up cloud bisync with rclone installation and configuration.
Args:
sync_dir: Optional custom sync directory path. If not provided, uses config default.
"""
console.print("[bold blue]Basic Memory Cloud Bisync Setup[/bold blue]")
console.print("Setting up bidirectional sync to your cloud tenant...\n")
try:
# Step 1: Install rclone
console.print("[blue]Step 1: Installing rclone...[/blue]")
install_rclone()
# Step 2: Get mount info (for tenant_id, bucket)
console.print("\n[blue]Step 2: Getting tenant information...[/blue]")
tenant_info = asyncio.run(get_mount_info())
tenant_id = tenant_info.tenant_id
bucket_name = tenant_info.bucket_name
console.print(f"[green]✓ Found tenant: {tenant_id}[/green]")
console.print(f"[green]✓ Bucket: {bucket_name}[/green]")
# Step 3: Generate credentials
console.print("\n[blue]Step 3: Generating sync credentials...[/blue]")
creds = asyncio.run(generate_mount_credentials(tenant_id))
access_key = creds.access_key
secret_key = creds.secret_key
console.print("[green]✓ Generated secure credentials[/green]")
# Step 4: Configure rclone
console.print("\n[blue]Step 4: Configuring rclone...[/blue]")
add_tenant_to_rclone_config(
tenant_id=tenant_id,
bucket_name=bucket_name,
access_key=access_key,
secret_key=secret_key,
)
# Step 5: Configure and create local directory
console.print("\n[blue]Step 5: Configuring sync directory...[/blue]")
# If custom sync_dir provided, save to config
if sync_dir:
config_manager = ConfigManager()
config = config_manager.load_config()
config.bisync_config["sync_dir"] = sync_dir
config_manager.save_config(config)
console.print("[green]✓ Saved custom sync directory to config[/green]")
# Get bisync directory (from config or default)
local_path = get_bisync_directory()
# Validate bisync directory
validate_bisync_directory(local_path)
# Create directory
local_path.mkdir(parents=True, exist_ok=True)
console.print(f"[green]✓ Created sync directory: {local_path}[/green]")
# Step 6: Perform initial resync
console.print("\n[blue]Step 6: Performing initial sync...[/blue]")
console.print("[yellow]This will establish the baseline for bidirectional sync.[/yellow]")
run_bisync(
tenant_id=tenant_id,
bucket_name=bucket_name,
local_path=local_path,
profile_name="balanced",
resync=True,
)
console.print("\n[bold green]✓ Bisync setup completed successfully![/bold green]")
console.print("\nYour local files will now sync bidirectionally with the cloud!")
console.print(f"\nLocal directory: {local_path}")
console.print("\nUseful commands:")
console.print(" bm sync # Run sync (recommended)")
console.print(" bm sync --watch # Start watch mode")
console.print(" bm cloud status # Check sync status")
console.print(" bm cloud check # Verify file integrity")
console.print(" bm cloud bisync --dry-run # Preview changes (advanced)")
except (RcloneInstallError, BisyncError, CloudAPIError) as e:
console.print(f"\n[red]Setup failed: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"\n[red]Unexpected error during setup: {e}[/red]")
raise typer.Exit(1)
def run_bisync(
tenant_id: Optional[str] = None,
bucket_name: Optional[str] = None,
local_path: Optional[Path] = None,
profile_name: str = "balanced",
dry_run: bool = False,
resync: bool = False,
verbose: bool = False,
) -> bool:
"""Run rclone bisync with specified profile."""
try:
# Get tenant info if not provided
if not tenant_id or not bucket_name:
tenant_info = asyncio.run(get_mount_info())
tenant_id = tenant_info.tenant_id
bucket_name = tenant_info.bucket_name
# Set default local path if not provided
if not local_path:
local_path = get_bisync_directory()
# Validate bisync directory
validate_bisync_directory(local_path)
# Check if local path exists
if not local_path.exists():
raise BisyncError(
f"Local directory {local_path} does not exist. Run 'basic-memory cloud bisync-setup' first."
)
# Get bisync profile
if profile_name not in BISYNC_PROFILES:
raise BisyncError(
f"Unknown profile: {profile_name}. Available: {list(BISYNC_PROFILES.keys())}"
)
profile = BISYNC_PROFILES[profile_name]
# Auto-register projects before sync (unless dry-run or resync)
if not dry_run and not resync:
try:
console.print("[dim]Checking for new projects...[/dim]")
# Fetch cloud projects and extract directory names from paths
cloud_data = asyncio.run(fetch_cloud_projects())
cloud_projects = cloud_data.projects
# Extract directory names from cloud project paths
# Compare directory names, not project names
# Cloud path /app/data/basic-memory -> directory name "basic-memory"
cloud_dir_names = set()
for p in cloud_projects:
path = p.path
# Strip /app/data/ prefix if present (cloud mode)
if path.startswith("/app/data/"):
path = path[len("/app/data/") :]
# Get the last segment (directory name)
dir_name = Path(path).name
cloud_dir_names.add(dir_name)
# Scan local directories
local_dirs = scan_local_directories(local_path)
# Create missing cloud projects
new_projects = []
for dir_name in local_dirs:
if dir_name not in cloud_dir_names:
new_projects.append(dir_name)
if new_projects:
console.print(
f"[blue]Found {len(new_projects)} new local project(s), creating on cloud...[/blue]"
)
for project_name in new_projects:
try:
asyncio.run(create_cloud_project(project_name))
console.print(f"[green] ✓ Created project: {project_name}[/green]")
except BisyncError as e:
console.print(
f"[yellow] ⚠ Could not create {project_name}: {e}[/yellow]"
)
else:
console.print("[dim]All local projects already registered on cloud[/dim]")
except Exception as e:
console.print(f"[yellow]Warning: Project auto-registration failed: {e}[/yellow]")
console.print("[yellow]Continuing with sync anyway...[/yellow]")
# Check if first run and require resync
if not resync and not bisync_state_exists(tenant_id) and not dry_run:
raise BisyncError(
"First bisync requires --resync to establish baseline. "
"Run: basic-memory cloud bisync --resync"
)
# Build and execute bisync command
bisync_cmd = build_bisync_command(
tenant_id,
bucket_name,
local_path,
profile,
dry_run=dry_run,
resync=resync,
verbose=verbose,
)
if dry_run:
console.print("[yellow]DRY RUN MODE - No changes will be made[/yellow]")
console.print(
f"[blue]Running bisync with profile '{profile_name}' ({profile.description})...[/blue]"
)
console.print(f"[dim]Command: {' '.join(bisync_cmd)}[/dim]")
console.print() # Blank line before output
# Stream output in real-time so user sees progress
result = subprocess.run(bisync_cmd, text=True)
if result.returncode != 0:
raise BisyncError(f"Bisync command failed with code {result.returncode}")
console.print() # Blank line after output
if dry_run:
console.print("[green]✓ Dry run completed successfully[/green]")
elif resync:
console.print("[green]✓ Initial sync baseline established[/green]")
else:
console.print("[green]✓ Sync completed successfully[/green]")
# Notify container to refresh cache (if not dry run)
if not dry_run:
try:
asyncio.run(notify_container_sync(tenant_id))
except Exception as e:
console.print(f"[yellow]Warning: Could not notify container: {e}[/yellow]")
return True
except BisyncError:
raise
except Exception as e:
raise BisyncError(f"Unexpected error during bisync: {e}") from e
async def notify_container_sync(tenant_id: str) -> None:
"""Sync all projects after bisync completes."""
try:
from basic_memory.cli.commands.command_utils import run_sync
# Fetch all projects and sync each one
cloud_data = await fetch_cloud_projects()
projects = cloud_data.projects
if not projects:
console.print("[dim]No projects to sync[/dim]")
return
console.print(f"[blue]Notifying cloud to index {len(projects)} project(s)...[/blue]")
for project in projects:
project_name = project.name
if project_name:
try:
await run_sync(project=project_name)
except Exception as e:
# Non-critical, log and continue
console.print(f"[yellow] ⚠ Sync failed for {project_name}: {e}[/yellow]")
console.print("[dim]Note: Cloud indexing has started and may take a few moments[/dim]")
except Exception as e:
# Non-critical, don't fail the bisync
console.print(f"[yellow]Warning: Post-sync failed: {e}[/yellow]")
def run_bisync_watch(
tenant_id: Optional[str] = None,
bucket_name: Optional[str] = None,
local_path: Optional[Path] = None,
profile_name: str = "balanced",
interval_seconds: int = 60,
) -> None:
"""Run bisync in watch mode with periodic syncs."""
console.print("[bold blue]Starting bisync watch mode[/bold blue]")
console.print(f"Sync interval: {interval_seconds} seconds")
console.print("Press Ctrl+C to stop\n")
try:
while True:
try:
start_time = time.time()
run_bisync(
tenant_id=tenant_id,
bucket_name=bucket_name,
local_path=local_path,
profile_name=profile_name,
)
elapsed = time.time() - start_time
console.print(f"[dim]Sync completed in {elapsed:.1f}s[/dim]")
# Wait for next interval
time.sleep(interval_seconds)
except BisyncError as e:
console.print(f"[red]Sync error: {e}[/red]")
console.print(f"[yellow]Retrying in {interval_seconds} seconds...[/yellow]")
time.sleep(interval_seconds)
except KeyboardInterrupt:
console.print("\n[yellow]Watch mode stopped[/yellow]")
def show_bisync_status() -> None:
"""Show current bisync status and configuration."""
try:
# Get tenant info
tenant_info = asyncio.run(get_mount_info())
tenant_id = tenant_info.tenant_id
local_path = get_bisync_directory()
state_path = get_bisync_state_path(tenant_id)
# Create status table
table = Table(title="Cloud Bisync Status", show_header=True, header_style="bold blue")
table.add_column("Property", style="green", min_width=20)
table.add_column("Value", style="dim", min_width=30)
# Check initialization status
is_initialized = bisync_state_exists(tenant_id)
init_status = (
"[green]✓ Initialized[/green]" if is_initialized else "[red]✗ Not initialized[/red]"
)
table.add_row("Tenant ID", tenant_id)
table.add_row("Local Directory", str(local_path))
table.add_row("Status", init_status)
table.add_row("State Directory", str(state_path))
# Check for last sync info
if is_initialized:
# Look for most recent state file
state_files = list(state_path.glob("*.lst"))
if state_files:
latest = max(state_files, key=lambda p: p.stat().st_mtime)
last_sync = datetime.fromtimestamp(latest.stat().st_mtime)
table.add_row("Last Sync", last_sync.strftime("%Y-%m-%d %H:%M:%S"))
console.print(table)
# Show bisync profiles
console.print("\n[bold]Available bisync profiles:[/bold]")
for name, profile in BISYNC_PROFILES.items():
console.print(f" {name}: {profile.description}")
console.print(f" - Conflict resolution: {profile.conflict_resolve}")
console.print(f" - Max delete: {profile.max_delete} files")
console.print("\n[dim]To use a profile: bm cloud bisync --profile <name>[/dim]")
# Show setup instructions if not initialized
if not is_initialized:
console.print("\n[yellow]To initialize bisync, run:[/yellow]")
console.print(" bm cloud setup")
console.print(" or")
console.print(" bm cloud bisync --resync")
except Exception as e:
console.print(f"[red]Error getting bisync status: {e}[/red]")
raise typer.Exit(1)
def run_check(
tenant_id: Optional[str] = None,
bucket_name: Optional[str] = None,
local_path: Optional[Path] = None,
one_way: bool = False,
) -> bool:
"""Check file integrity between local and cloud using rclone check.
Args:
tenant_id: Cloud tenant ID (auto-detected if not provided)
bucket_name: S3 bucket name (auto-detected if not provided)
local_path: Local bisync directory (uses config default if not provided)
one_way: If True, only check for missing files on destination (faster)
Returns:
True if check passed (files match), False if differences found
"""
try:
# Check if rclone is installed
from basic_memory.cli.commands.cloud.rclone_installer import is_rclone_installed
if not is_rclone_installed():
raise BisyncError(
"rclone is not installed. Run 'bm cloud bisync-setup' first to set up cloud sync."
)
# Get tenant info if not provided
if not tenant_id or not bucket_name:
tenant_info = asyncio.run(get_mount_info())
tenant_id = tenant_id or tenant_info.tenant_id
bucket_name = bucket_name or tenant_info.bucket_name
# Get local path from config
if not local_path:
local_path = get_bisync_directory()
# Check if bisync is initialized
if not bisync_state_exists(tenant_id):
raise BisyncError(
"Bisync not initialized. Run 'bm cloud bisync --resync' to establish baseline."
)
# Build rclone check command
rclone_remote = f"basic-memory-{tenant_id}:{bucket_name}"
filter_path = get_bisync_filter_path()
cmd = [
"rclone",
"check",
str(local_path),
rclone_remote,
"--filter-from",
str(filter_path),
]
if one_way:
cmd.append("--one-way")
console.print("[bold blue]Checking file integrity between local and cloud[/bold blue]")
console.print(f"[dim]Local: {local_path}[/dim]")
console.print(f"[dim]Remote: {rclone_remote}[/dim]")
console.print(f"[dim]Command: {' '.join(cmd)}[/dim]")
console.print()
# Run check command
result = subprocess.run(cmd, capture_output=True, text=True)
# rclone check returns:
# 0 = success (all files match)
# non-zero = differences found or error
if result.returncode == 0:
console.print("[green]✓ All files match between local and cloud[/green]")
return True
else:
console.print("[yellow]⚠ Differences found:[/yellow]")
if result.stderr:
console.print(result.stderr)
if result.stdout:
console.print(result.stdout)
console.print("\n[dim]To sync differences, run: bm sync[/dim]")
return False
except BisyncError:
raise
except Exception as e:
raise BisyncError(f"Check failed: {e}") from e
@@ -0,0 +1,101 @@
"""Shared utilities for cloud operations."""
from basic_memory.cli.commands.cloud.api_client import make_api_request
from basic_memory.config import ConfigManager
from basic_memory.schemas.cloud import (
CloudProjectList,
CloudProjectCreateRequest,
CloudProjectCreateResponse,
)
from basic_memory.utils import generate_permalink
class CloudUtilsError(Exception):
"""Exception raised for cloud utility errors."""
pass
async def fetch_cloud_projects() -> CloudProjectList:
"""Fetch list of projects from cloud API.
Returns:
CloudProjectList with projects from cloud
"""
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
response = await make_api_request(method="GET", url=f"{host_url}/proxy/projects/projects")
return CloudProjectList.model_validate(response.json())
except Exception as e:
raise CloudUtilsError(f"Failed to fetch cloud projects: {e}") from e
async def create_cloud_project(project_name: str) -> CloudProjectCreateResponse:
"""Create a new project on cloud.
Args:
project_name: Name of project to create
Returns:
CloudProjectCreateResponse with project details from API
"""
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
# Use generate_permalink to ensure consistent naming
project_path = generate_permalink(project_name)
project_data = CloudProjectCreateRequest(
name=project_name,
path=project_path,
set_default=False,
)
response = await make_api_request(
method="POST",
url=f"{host_url}/proxy/projects/projects",
headers={"Content-Type": "application/json"},
json_data=project_data.model_dump(),
)
return CloudProjectCreateResponse.model_validate(response.json())
except Exception as e:
raise CloudUtilsError(f"Failed to create cloud project '{project_name}': {e}") from e
async def sync_project(project_name: str, force_full: bool = False) -> None:
"""Trigger sync for a specific project on cloud.
Args:
project_name: Name of project to sync
force_full: If True, force a full scan bypassing watermark optimization
"""
try:
from basic_memory.cli.commands.command_utils import run_sync
await run_sync(project=project_name, force_full=force_full)
except Exception as e:
raise CloudUtilsError(f"Failed to sync project '{project_name}': {e}") from e
async def project_exists(project_name: str) -> bool:
"""Check if a project exists on cloud.
Args:
project_name: Name of project to check
Returns:
True if project exists, False otherwise
"""
try:
projects = await fetch_cloud_projects()
project_names = {p.name for p in projects.projects}
return project_name in project_names
except Exception:
return False
@@ -1,7 +1,6 @@
"""Core cloud commands for Basic Memory CLI."""
import asyncio
from typing import Optional
import typer
from rich.console import Console
@@ -15,21 +14,16 @@ from basic_memory.cli.commands.cloud.api_client import (
get_cloud_config,
make_api_request,
)
from basic_memory.cli.commands.cloud.mount_commands import (
mount_cloud_files,
setup_cloud_mount,
show_mount_status,
unmount_cloud_files,
)
from basic_memory.cli.commands.cloud.bisync_commands import (
run_bisync,
run_bisync_watch,
run_check,
setup_cloud_bisync,
show_bisync_status,
BisyncError,
generate_mount_credentials,
get_mount_info,
)
from basic_memory.cli.commands.cloud.rclone_config import configure_rclone_remote
from basic_memory.cli.commands.cloud.rclone_installer import (
RcloneInstallError,
install_rclone,
)
from basic_memory.cli.commands.cloud.rclone_config import MOUNT_PROFILES
from basic_memory.cli.commands.cloud.bisync_commands import BISYNC_PROFILES
console = Console()
@@ -58,11 +52,11 @@ def login():
config.cloud_mode = True
config_manager.save_config(config)
console.print("[green]Cloud mode enabled[/green]")
console.print("[green]Cloud mode enabled[/green]")
console.print(f"[dim]All CLI commands now work against {host_url}[/dim]")
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
console.print(
@@ -83,23 +77,13 @@ def logout():
config.cloud_mode = False
config_manager.save_config(config)
console.print("[green]Cloud mode disabled[/green]")
console.print("[green]Cloud mode disabled[/green]")
console.print("[dim]All CLI commands now work locally[/dim]")
@cloud_app.command("status")
def status(
bisync: bool = typer.Option(
True,
"--bisync/--mount",
help="Show bisync status (default) or mount status",
),
) -> None:
"""Check cloud mode status and cloud instance health.
Shows cloud mode status, instance health, and sync/mount status.
Use --bisync (default) to show bisync status or --mount for mount status.
"""
def status() -> None:
"""Check cloud mode status and cloud instance health."""
# Check cloud mode
config_manager = ConfigManager()
config = config_manager.load_config()
@@ -142,12 +126,7 @@ def status(
if "timestamp" in health_data:
console.print(f" Timestamp: {health_data['timestamp']}")
# Show sync/mount status based on flag
console.print()
if bisync:
show_bisync_status()
else:
show_mount_status()
console.print("\n[dim]To sync projects, use: bm project bisync --name <project>[/dim]")
except CloudAPIError as e:
console.print(f"[red]Error checking cloud health: {e}[/red]")
@@ -157,132 +136,60 @@ def status(
raise typer.Exit(1)
# Mount commands
@cloud_app.command("setup")
def setup(
bisync: bool = typer.Option(
True,
"--bisync/--mount",
help="Use bidirectional sync (recommended) or mount as network drive",
),
sync_dir: Optional[str] = typer.Option(
None,
"--dir",
help="Custom sync directory for bisync (default: ~/basic-memory-cloud-sync)",
),
) -> None:
"""Set up cloud file access with automatic rclone installation and configuration.
def setup() -> None:
"""Set up cloud sync by installing rclone and configuring credentials.
Default: Sets up bidirectional sync (recommended).\n
Use --mount: Sets up mount as network drive (alternative workflow).\n
Examples:\n
bm cloud setup # Setup bisync (default)\n
bm cloud setup --mount # Setup mount instead\n
bm cloud setup --dir ~/sync # Custom bisync directory\n
SPEC-20: Simplified to project-scoped workflow.
After setup, use project commands for syncing:
bm project add <name> <path> --local-path ~/projects/<name>
bm project bisync --name <name> --resync # First time
bm project bisync --name <name> # Subsequent syncs
"""
if bisync:
setup_cloud_bisync(sync_dir=sync_dir)
else:
setup_cloud_mount()
console.print("[bold blue]Basic Memory Cloud Setup[/bold blue]")
console.print("Setting up cloud sync with rclone...\n")
@cloud_app.command("mount")
def mount(
profile: str = typer.Option(
"balanced", help=f"Mount profile: {', '.join(MOUNT_PROFILES.keys())}"
),
path: Optional[str] = typer.Option(
None, help="Custom mount path (default: ~/basic-memory-{tenant-id})"
),
) -> None:
"""Mount cloud files locally for editing."""
try:
mount_cloud_files(profile_name=profile)
except Exception as e:
console.print(f"[red]Mount failed: {e}[/red]")
# Step 1: Install rclone
console.print("[blue]Step 1: Installing rclone...[/blue]")
install_rclone()
# Step 2: Get tenant info
console.print("\n[blue]Step 2: Getting tenant information...[/blue]")
tenant_info = asyncio.run(get_mount_info())
console.print(f"[green]Found tenant: {tenant_info.tenant_id}[/green]")
# Step 3: Generate credentials
console.print("\n[blue]Step 3: Generating sync credentials...[/blue]")
creds = asyncio.run(generate_mount_credentials(tenant_info.tenant_id))
console.print("[green]Generated secure credentials[/green]")
# Step 4: Configure rclone remote
console.print("\n[blue]Step 4: Configuring rclone remote...[/blue]")
configure_rclone_remote(
access_key=creds.access_key,
secret_key=creds.secret_key,
)
console.print("\n[bold green]Cloud setup completed successfully![/bold green]")
console.print("\n[bold]Next steps:[/bold]")
console.print("1. Add a project with local sync path:")
console.print(" bm project add research --local-path ~/Documents/research")
console.print("\n Or configure sync for an existing project:")
console.print(" bm project sync-setup research ~/Documents/research")
console.print("\n2. Preview the initial sync (recommended):")
console.print(" bm project bisync --name research --resync --dry-run")
console.print("\n3. If all looks good, run the actual sync:")
console.print(" bm project bisync --name research --resync")
console.print("\n4. Subsequent syncs (no --resync needed):")
console.print(" bm project bisync --name research")
console.print(
"\n[dim]Tip: Always use --dry-run first to preview changes before syncing[/dim]"
)
except (RcloneInstallError, BisyncError, CloudAPIError) as e:
console.print(f"\n[red]Setup failed: {e}[/red]")
raise typer.Exit(1)
@cloud_app.command("unmount")
def unmount() -> None:
"""Unmount cloud files."""
try:
unmount_cloud_files()
except Exception as e:
console.print(f"[red]Unmount failed: {e}[/red]")
raise typer.Exit(1)
# Bisync commands
@cloud_app.command("bisync")
def bisync(
profile: str = typer.Option(
"balanced", help=f"Bisync profile: {', '.join(BISYNC_PROFILES.keys())}"
),
dry_run: bool = typer.Option(False, "--dry-run", help="Preview changes without syncing"),
resync: bool = typer.Option(False, "--resync", help="Force resync to establish new baseline"),
watch: bool = typer.Option(False, "--watch", help="Run continuous sync in watch mode"),
interval: int = typer.Option(60, "--interval", help="Sync interval in seconds for watch mode"),
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed sync output"),
) -> None:
"""Run bidirectional sync between local files and cloud storage.
Examples:
basic-memory cloud bisync # Manual sync with balanced profile
basic-memory cloud bisync --dry-run # Preview what would be synced
basic-memory cloud bisync --resync # Establish new baseline
basic-memory cloud bisync --watch # Continuous sync every 60s
basic-memory cloud bisync --watch --interval 30 # Continuous sync every 30s
basic-memory cloud bisync --profile safe # Use safe profile (keep conflicts)
basic-memory cloud bisync --verbose # Show detailed file sync output
"""
try:
if watch:
run_bisync_watch(profile_name=profile, interval_seconds=interval)
else:
run_bisync(profile_name=profile, dry_run=dry_run, resync=resync, verbose=verbose)
except Exception as e:
console.print(f"[red]Bisync failed: {e}[/red]")
raise typer.Exit(1)
@cloud_app.command("bisync-status")
def bisync_status() -> None:
"""Show current bisync status and configuration.
DEPRECATED: Use 'bm cloud status' instead (bisync is now the default).
"""
console.print(
"[yellow]Note: 'bisync-status' is deprecated. Use 'bm cloud status' instead.[/yellow]"
)
console.print("[dim]Showing bisync status...[/dim]\n")
show_bisync_status()
@cloud_app.command("check")
def check(
one_way: bool = typer.Option(
False,
"--one-way",
help="Only check for missing files on destination (faster)",
),
) -> None:
"""Check file integrity between local and cloud storage using rclone check.
Verifies that files match between your local bisync directory and cloud storage
without transferring any data. This is useful for validating sync integrity.
Examples:
bm cloud check # Full integrity check
bm cloud check --one-way # Faster check (missing files only)
"""
try:
run_check(one_way=one_way)
except Exception as e:
console.print(f"[red]Check failed: {e}[/red]")
console.print(f"\n[red]Unexpected error during setup: {e}[/red]")
raise typer.Exit(1)
@@ -1,295 +0,0 @@
"""Cloud mount commands for Basic Memory CLI."""
import asyncio
import subprocess
import time
from pathlib import Path
from typing import Optional
import typer
from rich.console import Console
from rich.table import Table
from basic_memory.cli.commands.cloud.api_client import CloudAPIError, make_api_request
from basic_memory.cli.commands.cloud.rclone_config import (
MOUNT_PROFILES,
add_tenant_to_rclone_config,
build_mount_command,
cleanup_orphaned_rclone_processes,
get_default_mount_path,
get_rclone_processes,
is_path_mounted,
unmount_path,
)
from basic_memory.cli.commands.cloud.rclone_installer import RcloneInstallError, install_rclone
from basic_memory.config import ConfigManager
console = Console()
class MountError(Exception):
"""Exception raised for mount-related errors."""
pass
async def get_tenant_info() -> dict:
"""Get current tenant information from cloud API."""
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
response = await make_api_request(method="GET", url=f"{host_url}/tenant/mount/info")
return response.json()
except Exception as e:
raise MountError(f"Failed to get tenant info: {e}") from e
async def generate_mount_credentials(tenant_id: str) -> dict:
"""Generate scoped credentials for mounting."""
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
response = await make_api_request(method="POST", url=f"{host_url}/tenant/mount/credentials")
return response.json()
except Exception as e:
raise MountError(f"Failed to generate mount credentials: {e}") from e
def setup_cloud_mount() -> None:
"""Set up cloud mount with rclone installation and configuration."""
console.print("[bold blue]Basic Memory Cloud Setup[/bold blue]")
console.print("Setting up local file access to your cloud tenant...\n")
try:
# Step 1: Install rclone
console.print("[blue]Step 1: Installing rclone...[/blue]")
install_rclone()
# Step 2: Get tenant info
console.print("\n[blue]Step 2: Getting tenant information...[/blue]")
tenant_info = asyncio.run(get_tenant_info())
tenant_id = tenant_info.get("tenant_id")
bucket_name = tenant_info.get("bucket_name")
if not tenant_id or not bucket_name:
raise MountError("Invalid tenant information received from cloud API")
console.print(f"[green]✓ Found tenant: {tenant_id}[/green]")
console.print(f"[green]✓ Bucket: {bucket_name}[/green]")
# Step 3: Generate mount credentials
console.print("\n[blue]Step 3: Generating mount credentials...[/blue]")
creds = asyncio.run(generate_mount_credentials(tenant_id))
access_key = creds.get("access_key")
secret_key = creds.get("secret_key")
if not access_key or not secret_key:
raise MountError("Failed to generate mount credentials")
console.print("[green]✓ Generated secure credentials[/green]")
# Step 4: Configure rclone
console.print("\n[blue]Step 4: Configuring rclone...[/blue]")
add_tenant_to_rclone_config(
tenant_id=tenant_id,
bucket_name=bucket_name,
access_key=access_key,
secret_key=secret_key,
)
# Step 5: Perform initial mount
console.print("\n[blue]Step 5: Mounting cloud files...[/blue]")
mount_path = get_default_mount_path()
MOUNT_PROFILES["balanced"]
mount_cloud_files(
tenant_id=tenant_id,
bucket_name=bucket_name,
mount_path=mount_path,
profile_name="balanced",
)
console.print("\n[bold green]✓ Cloud setup completed successfully![/bold green]")
console.print("\nYour cloud files are now accessible at:")
console.print(f" {mount_path}")
console.print("\nYou can now edit files locally and they will sync to the cloud!")
console.print("\nUseful commands:")
console.print(" basic-memory cloud mount-status # Check mount status")
console.print(" basic-memory cloud unmount # Unmount files")
console.print(" basic-memory cloud mount --profile fast # Remount with faster sync")
except (RcloneInstallError, MountError, CloudAPIError) as e:
console.print(f"\n[red]Setup failed: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"\n[red]Unexpected error during setup: {e}[/red]")
raise typer.Exit(1)
def mount_cloud_files(
tenant_id: Optional[str] = None,
bucket_name: Optional[str] = None,
mount_path: Optional[Path] = None,
profile_name: str = "balanced",
) -> None:
"""Mount cloud files with specified profile."""
try:
# Get tenant info if not provided
if not tenant_id or not bucket_name:
tenant_info = asyncio.run(get_tenant_info())
tenant_id = tenant_info.get("tenant_id")
bucket_name = tenant_info.get("bucket_name")
if not tenant_id or not bucket_name:
raise MountError("Could not determine tenant information")
# Set default mount path if not provided
if not mount_path:
mount_path = get_default_mount_path()
# Get mount profile
if profile_name not in MOUNT_PROFILES:
raise MountError(
f"Unknown profile: {profile_name}. Available: {list(MOUNT_PROFILES.keys())}"
)
profile = MOUNT_PROFILES[profile_name]
# Check if already mounted
if is_path_mounted(mount_path):
console.print(f"[yellow]Path {mount_path} is already mounted[/yellow]")
console.print("Use 'basic-memory cloud unmount' first, or mount to a different path")
return
# Create mount directory
mount_path.mkdir(parents=True, exist_ok=True)
# Build and execute mount command
mount_cmd = build_mount_command(tenant_id, bucket_name, mount_path, profile)
console.print(
f"[blue]Mounting with profile '{profile_name}' ({profile.description})...[/blue]"
)
console.print(f"[dim]Command: {' '.join(mount_cmd)}[/dim]")
result = subprocess.run(mount_cmd, capture_output=True, text=True)
if result.returncode != 0:
error_msg = result.stderr or "Unknown error"
raise MountError(f"Mount command failed: {error_msg}")
# Wait a moment for mount to establish
time.sleep(2)
# Verify mount
if is_path_mounted(mount_path):
console.print(f"[green]✓ Successfully mounted to {mount_path}[/green]")
console.print(f"[green]✓ Sync profile: {profile.description}[/green]")
else:
raise MountError("Mount command succeeded but path is not mounted")
except MountError:
raise
except Exception as e:
raise MountError(f"Unexpected error during mount: {e}") from e
def unmount_cloud_files(tenant_id: Optional[str] = None) -> None:
"""Unmount cloud files."""
try:
# Get tenant info if not provided
if not tenant_id:
tenant_info = asyncio.run(get_tenant_info())
tenant_id = tenant_info.get("tenant_id")
if not tenant_id:
raise MountError("Could not determine tenant ID")
mount_path = get_default_mount_path()
if not is_path_mounted(mount_path):
console.print(f"[yellow]Path {mount_path} is not mounted[/yellow]")
return
console.print(f"[blue]Unmounting {mount_path}...[/blue]")
# Unmount the path
if unmount_path(mount_path):
console.print(f"[green]✓ Successfully unmounted {mount_path}[/green]")
# Clean up any orphaned rclone processes
killed_count = cleanup_orphaned_rclone_processes()
if killed_count > 0:
console.print(
f"[green]✓ Cleaned up {killed_count} orphaned rclone process(es)[/green]"
)
else:
console.print(f"[red]✗ Failed to unmount {mount_path}[/red]")
console.print("You may need to manually unmount or restart your system")
except MountError:
raise
except Exception as e:
raise MountError(f"Unexpected error during unmount: {e}") from e
def show_mount_status() -> None:
"""Show current mount status and running processes."""
try:
# Get tenant info
tenant_info = asyncio.run(get_tenant_info())
tenant_id = tenant_info.get("tenant_id")
if not tenant_id:
console.print("[red]Could not determine tenant ID[/red]")
return
mount_path = get_default_mount_path()
# Create status table
table = Table(title="Cloud Mount Status", show_header=True, header_style="bold blue")
table.add_column("Property", style="green", min_width=15)
table.add_column("Value", style="dim", min_width=30)
# Check mount status
is_mounted = is_path_mounted(mount_path)
mount_status = "[green]✓ Mounted[/green]" if is_mounted else "[red]✗ Not mounted[/red]"
table.add_row("Tenant ID", tenant_id)
table.add_row("Mount Path", str(mount_path))
table.add_row("Status", mount_status)
# Get rclone processes
processes = get_rclone_processes()
if processes:
table.add_row("rclone Processes", f"{len(processes)} running")
else:
table.add_row("rclone Processes", "None")
console.print(table)
# Show running processes details
if processes:
console.print("\n[bold]Running rclone processes:[/bold]")
for proc in processes:
console.print(f" PID {proc['pid']}: {proc['command'][:80]}...")
# Show mount profiles
console.print("\n[bold]Available mount profiles:[/bold]")
for name, profile in MOUNT_PROFILES.items():
console.print(f" {name}: {profile.description}")
except Exception as e:
console.print(f"[red]Error getting mount status: {e}[/red]")
raise typer.Exit(1)
@@ -0,0 +1,326 @@
"""Project-scoped rclone sync commands for Basic Memory Cloud.
This module provides simplified, project-scoped rclone operations:
- Each project syncs independently
- Uses single "basic-memory-cloud" remote (not tenant-specific)
- Balanced defaults from SPEC-8 Phase 4 testing
- Per-project bisync state tracking
Replaces tenant-wide sync with project-scoped workflows.
"""
import subprocess
from dataclasses import dataclass
from pathlib import Path
from typing import Optional
from rich.console import Console
from basic_memory.cli.commands.cloud.rclone_installer import is_rclone_installed
from basic_memory.utils import normalize_project_path
console = Console()
class RcloneError(Exception):
"""Exception raised for rclone command errors."""
pass
def check_rclone_installed() -> None:
"""Check if rclone is installed and raise helpful error if not.
Raises:
RcloneError: If rclone is not installed with installation instructions
"""
if not is_rclone_installed():
raise RcloneError(
"rclone is not installed.\n\n"
"Install rclone by running: bm cloud setup\n"
"Or install manually from: https://rclone.org/downloads/\n\n"
"Windows users: Ensure you have a package manager installed (winget, chocolatey, or scoop)"
)
@dataclass
class SyncProject:
"""Project configured for cloud sync.
Attributes:
name: Project name
path: Cloud path (e.g., "app/data/research")
local_sync_path: Local directory for syncing (optional)
"""
name: str
path: str
local_sync_path: Optional[str] = None
def get_bmignore_filter_path() -> Path:
"""Get path to rclone filter file.
Uses ~/.basic-memory/.bmignore converted to rclone format.
File is automatically created with default patterns on first use.
Returns:
Path to rclone filter file
"""
# Import here to avoid circular dependency
from basic_memory.cli.commands.cloud.bisync_commands import (
convert_bmignore_to_rclone_filters,
)
return convert_bmignore_to_rclone_filters()
def get_project_bisync_state(project_name: str) -> Path:
"""Get path to project's bisync state directory.
Args:
project_name: Name of the project
Returns:
Path to bisync state directory for this project
"""
return Path.home() / ".basic-memory" / "bisync-state" / project_name
def bisync_initialized(project_name: str) -> bool:
"""Check if bisync has been initialized for this project.
Args:
project_name: Name of the project
Returns:
True if bisync state exists, False otherwise
"""
state_path = get_project_bisync_state(project_name)
return state_path.exists() and any(state_path.iterdir())
def get_project_remote(project: SyncProject, bucket_name: str) -> str:
"""Build rclone remote path for project.
Args:
project: Project with cloud path
bucket_name: S3 bucket name
Returns:
Remote path like "basic-memory-cloud:bucket-name/basic-memory-llc"
Note:
The API returns paths like "/app/data/basic-memory-llc" because the S3 bucket
is mounted at /app/data on the fly machine. We need to strip the /app/data/
prefix to get the actual S3 path within the bucket.
"""
# Normalize path to strip /app/data/ mount point prefix
cloud_path = normalize_project_path(project.path).lstrip("/")
return f"basic-memory-cloud:{bucket_name}/{cloud_path}"
def project_sync(
project: SyncProject,
bucket_name: str,
dry_run: bool = False,
verbose: bool = False,
) -> bool:
"""One-way sync: local → cloud.
Makes cloud identical to local using rclone sync.
Args:
project: Project to sync
bucket_name: S3 bucket name
dry_run: Preview changes without applying
verbose: Show detailed output
Returns:
True if sync succeeded, False otherwise
Raises:
RcloneError: If project has no local_sync_path configured or rclone not installed
"""
check_rclone_installed()
if not project.local_sync_path:
raise RcloneError(f"Project {project.name} has no local_sync_path configured")
local_path = Path(project.local_sync_path).expanduser()
remote_path = get_project_remote(project, bucket_name)
filter_path = get_bmignore_filter_path()
cmd = [
"rclone",
"sync",
str(local_path),
remote_path,
"--filter-from",
str(filter_path),
]
if verbose:
cmd.append("--verbose")
else:
cmd.append("--progress")
if dry_run:
cmd.append("--dry-run")
result = subprocess.run(cmd, text=True)
return result.returncode == 0
def project_bisync(
project: SyncProject,
bucket_name: str,
dry_run: bool = False,
resync: bool = False,
verbose: bool = False,
) -> bool:
"""Two-way sync: local ↔ cloud.
Uses rclone bisync with balanced defaults:
- conflict_resolve: newer (auto-resolve to most recent)
- max_delete: 25 (safety limit)
- compare: modtime (ignore size differences from line ending conversions)
- check_access: false (skip for performance)
Args:
project: Project to sync
bucket_name: S3 bucket name
dry_run: Preview changes without applying
resync: Force resync to establish new baseline
verbose: Show detailed output
Returns:
True if bisync succeeded, False otherwise
Raises:
RcloneError: If project has no local_sync_path, needs --resync, or rclone not installed
"""
check_rclone_installed()
if not project.local_sync_path:
raise RcloneError(f"Project {project.name} has no local_sync_path configured")
local_path = Path(project.local_sync_path).expanduser()
remote_path = get_project_remote(project, bucket_name)
filter_path = get_bmignore_filter_path()
state_path = get_project_bisync_state(project.name)
# Ensure state directory exists
state_path.mkdir(parents=True, exist_ok=True)
cmd = [
"rclone",
"bisync",
str(local_path),
remote_path,
"--create-empty-src-dirs",
"--resilient",
"--conflict-resolve=newer",
"--max-delete=25",
"--compare=modtime", # Ignore size differences from line ending conversions
"--filter-from",
str(filter_path),
"--workdir",
str(state_path),
]
if verbose:
cmd.append("--verbose")
else:
cmd.append("--progress")
if dry_run:
cmd.append("--dry-run")
if resync:
cmd.append("--resync")
# Check if first run requires resync
if not resync and not bisync_initialized(project.name) and not dry_run:
raise RcloneError(
f"First bisync for {project.name} requires --resync to establish baseline.\n"
f"Run: bm project bisync --name {project.name} --resync"
)
result = subprocess.run(cmd, text=True)
return result.returncode == 0
def project_check(
project: SyncProject,
bucket_name: str,
one_way: bool = False,
) -> bool:
"""Check integrity between local and cloud.
Verifies files match without transferring data.
Args:
project: Project to check
bucket_name: S3 bucket name
one_way: Only check for missing files on destination (faster)
Returns:
True if files match, False if differences found
Raises:
RcloneError: If project has no local_sync_path configured or rclone not installed
"""
check_rclone_installed()
if not project.local_sync_path:
raise RcloneError(f"Project {project.name} has no local_sync_path configured")
local_path = Path(project.local_sync_path).expanduser()
remote_path = get_project_remote(project, bucket_name)
filter_path = get_bmignore_filter_path()
cmd = [
"rclone",
"check",
str(local_path),
remote_path,
"--filter-from",
str(filter_path),
]
if one_way:
cmd.append("--one-way")
result = subprocess.run(cmd, capture_output=True, text=True)
return result.returncode == 0
def project_ls(
project: SyncProject,
bucket_name: str,
path: Optional[str] = None,
) -> list[str]:
"""List files in remote project.
Args:
project: Project to list files from
bucket_name: S3 bucket name
path: Optional subdirectory within project
Returns:
List of file paths
Raises:
subprocess.CalledProcessError: If rclone command fails
RcloneError: If rclone is not installed
"""
check_rclone_installed()
remote_path = get_project_remote(project, bucket_name)
if path:
remote_path = f"{remote_path}/{path}"
cmd = ["rclone", "ls", remote_path]
result = subprocess.run(cmd, capture_output=True, text=True, check=True)
return result.stdout.splitlines()
@@ -1,11 +1,14 @@
"""rclone configuration management for Basic Memory Cloud."""
"""rclone configuration management for Basic Memory Cloud.
This module provides simplified rclone configuration for SPEC-20.
Uses a single "basic-memory-cloud" remote for all operations.
"""
import configparser
import os
import shutil
import subprocess
from pathlib import Path
from typing import Dict, List, Optional
from typing import Optional
from rich.console import Console
@@ -18,64 +21,6 @@ class RcloneConfigError(Exception):
pass
class RcloneMountProfile:
"""Mount profile with optimized settings."""
def __init__(
self,
name: str,
cache_time: str,
poll_interval: str,
attr_timeout: str,
write_back: str,
description: str,
extra_args: Optional[List[str]] = None,
):
self.name = name
self.cache_time = cache_time
self.poll_interval = poll_interval
self.attr_timeout = attr_timeout
self.write_back = write_back
self.description = description
self.extra_args = extra_args or []
# Mount profiles based on SPEC-7 Phase 4 testing
MOUNT_PROFILES = {
"fast": RcloneMountProfile(
name="fast",
cache_time="5s",
poll_interval="3s",
attr_timeout="3s",
write_back="1s",
description="Ultra-fast development (5s sync, higher bandwidth)",
),
"balanced": RcloneMountProfile(
name="balanced",
cache_time="10s",
poll_interval="5s",
attr_timeout="5s",
write_back="2s",
description="Fast development (10-15s sync, recommended)",
),
"safe": RcloneMountProfile(
name="safe",
cache_time="15s",
poll_interval="10s",
attr_timeout="10s",
write_back="5s",
description="Conflict-aware mount with backup",
extra_args=[
"--conflict-suffix",
".conflict-{DateTimeExt}",
"--backup-dir",
"~/.basic-memory/conflicts",
"--track-renames",
],
),
}
def get_rclone_config_path() -> Path:
"""Get the path to rclone configuration file."""
config_dir = Path.home() / ".config" / "rclone"
@@ -116,173 +61,50 @@ def save_rclone_config(config: configparser.ConfigParser) -> None:
console.print(f"[dim]Updated rclone config: {config_path}[/dim]")
def add_tenant_to_rclone_config(
tenant_id: str,
bucket_name: str,
def configure_rclone_remote(
access_key: str,
secret_key: str,
endpoint: str = "https://fly.storage.tigris.dev",
region: str = "auto",
) -> str:
"""Add tenant configuration to rclone config file."""
"""Configure single rclone remote named 'basic-memory-cloud'.
This is the simplified approach from SPEC-20 that uses one remote
for all Basic Memory cloud operations (not tenant-specific).
Args:
access_key: S3 access key ID
secret_key: S3 secret access key
endpoint: S3-compatible endpoint URL
region: S3 region (default: auto)
Returns:
The remote name: "basic-memory-cloud"
"""
# Backup existing config
backup_rclone_config()
# Load existing config
config = load_rclone_config()
# Create section name
section_name = f"basic-memory-{tenant_id}"
# Single remote name (not tenant-specific)
REMOTE_NAME = "basic-memory-cloud"
# Add/update the tenant section
if not config.has_section(section_name):
config.add_section(section_name)
config.set(section_name, "type", "s3")
config.set(section_name, "provider", "Other")
config.set(section_name, "access_key_id", access_key)
config.set(section_name, "secret_access_key", secret_key)
config.set(section_name, "endpoint", endpoint)
config.set(section_name, "region", region)
# Add/update the remote section
if not config.has_section(REMOTE_NAME):
config.add_section(REMOTE_NAME)
config.set(REMOTE_NAME, "type", "s3")
config.set(REMOTE_NAME, "provider", "Other")
config.set(REMOTE_NAME, "access_key_id", access_key)
config.set(REMOTE_NAME, "secret_access_key", secret_key)
config.set(REMOTE_NAME, "endpoint", endpoint)
config.set(REMOTE_NAME, "region", region)
# Prevent unnecessary encoding of filenames (only encode slashes and invalid UTF-8)
# This prevents files with spaces like "Hello World.md" from being quoted
config.set(REMOTE_NAME, "encoding", "Slash,InvalidUtf8")
# Save updated config
save_rclone_config(config)
console.print(f"[green]✓ Added tenant {tenant_id} to rclone config[/green]")
return section_name
def remove_tenant_from_rclone_config(tenant_id: str) -> bool:
"""Remove tenant configuration from rclone config."""
config = load_rclone_config()
section_name = f"basic-memory-{tenant_id}"
if config.has_section(section_name):
backup_rclone_config()
config.remove_section(section_name)
save_rclone_config(config)
console.print(f"[green]✓ Removed tenant {tenant_id} from rclone config[/green]")
return True
return False
def get_default_mount_path() -> Path:
"""Get default mount path (fixed location per SPEC-9).
Returns:
Fixed mount path: ~/basic-memory-cloud/
"""
return Path.home() / "basic-memory-cloud"
def build_mount_command(
tenant_id: str, bucket_name: str, mount_path: Path, profile: RcloneMountProfile
) -> List[str]:
"""Build rclone mount command with optimized settings."""
rclone_remote = f"basic-memory-{tenant_id}:{bucket_name}"
cmd = [
"rclone",
"nfsmount",
rclone_remote,
str(mount_path),
"--vfs-cache-mode",
"writes",
"--dir-cache-time",
profile.cache_time,
"--vfs-cache-poll-interval",
profile.poll_interval,
"--attr-timeout",
profile.attr_timeout,
"--vfs-write-back",
profile.write_back,
"--daemon",
]
# Add profile-specific extra arguments
cmd.extend(profile.extra_args)
return cmd
def is_path_mounted(mount_path: Path) -> bool:
"""Check if a path is currently mounted."""
if not mount_path.exists():
return False
try:
# Check if mount point is actually mounted by looking for mount table entry
result = subprocess.run(["mount"], capture_output=True, text=True, check=False)
if result.returncode == 0:
# Look for our mount path in mount output
mount_str = str(mount_path.resolve())
return mount_str in result.stdout
return False
except Exception:
return False
def get_rclone_processes() -> List[Dict[str, str]]:
"""Get list of running rclone processes."""
try:
# Use ps to find rclone processes
result = subprocess.run(
["ps", "-eo", "pid,args"], capture_output=True, text=True, check=False
)
processes = []
if result.returncode == 0:
for line in result.stdout.split("\n"):
if "rclone" in line and "basic-memory" in line:
parts = line.strip().split(None, 1)
if len(parts) >= 2:
processes.append({"pid": parts[0], "command": parts[1]})
return processes
except Exception:
return []
def kill_rclone_process(pid: str) -> bool:
"""Kill a specific rclone process."""
try:
subprocess.run(["kill", pid], check=True)
console.print(f"[green]✓ Killed rclone process {pid}[/green]")
return True
except subprocess.CalledProcessError:
console.print(f"[red]✗ Failed to kill rclone process {pid}[/red]")
return False
def unmount_path(mount_path: Path) -> bool:
"""Unmount a mounted path."""
if not is_path_mounted(mount_path):
return True
try:
subprocess.run(["umount", str(mount_path)], check=True)
console.print(f"[green]✓ Unmounted {mount_path}[/green]")
return True
except subprocess.CalledProcessError as e:
console.print(f"[red]✗ Failed to unmount {mount_path}: {e}[/red]")
return False
def cleanup_orphaned_rclone_processes() -> int:
"""Clean up orphaned rclone processes for basic-memory."""
processes = get_rclone_processes()
killed_count = 0
for proc in processes:
console.print(
f"[yellow]Found rclone process: {proc['pid']} - {proc['command'][:80]}...[/yellow]"
)
if kill_rclone_process(proc["pid"]):
killed_count += 1
return killed_count
console.print(f"[green]Configured rclone remote: {REMOTE_NAME}[/green]")
return REMOTE_NAME
@@ -1,5 +1,6 @@
"""Cross-platform rclone installation utilities."""
import os
import platform
import shutil
import subprocess
@@ -58,7 +59,7 @@ def install_rclone_macos() -> None:
try:
console.print("[blue]Installing rclone via Homebrew...[/blue]")
run_command(["brew", "install", "rclone"])
console.print("[green]rclone installed via Homebrew[/green]")
console.print("[green]rclone installed via Homebrew[/green]")
return
except RcloneInstallError:
console.print(
@@ -69,7 +70,7 @@ def install_rclone_macos() -> None:
console.print("[blue]Installing rclone via official script...[/blue]")
try:
run_command(["sh", "-c", "curl https://rclone.org/install.sh | sudo bash"])
console.print("[green]rclone installed via official script[/green]")
console.print("[green]rclone installed via official script[/green]")
except RcloneInstallError:
raise RcloneInstallError(
"Failed to install rclone. Please install manually: brew install rclone"
@@ -83,7 +84,7 @@ def install_rclone_linux() -> None:
try:
console.print("[blue]Installing rclone via snap...[/blue]")
run_command(["sudo", "snap", "install", "rclone"])
console.print("[green]rclone installed via snap[/green]")
console.print("[green]rclone installed via snap[/green]")
return
except RcloneInstallError:
console.print("[yellow]Snap installation failed, trying apt...[/yellow]")
@@ -94,7 +95,7 @@ def install_rclone_linux() -> None:
console.print("[blue]Installing rclone via apt...[/blue]")
run_command(["sudo", "apt", "update"])
run_command(["sudo", "apt", "install", "-y", "rclone"])
console.print("[green]rclone installed via apt[/green]")
console.print("[green]rclone installed via apt[/green]")
return
except RcloneInstallError:
console.print("[yellow]apt installation failed, trying official script...[/yellow]")
@@ -103,7 +104,7 @@ def install_rclone_linux() -> None:
console.print("[blue]Installing rclone via official script...[/blue]")
try:
run_command(["sh", "-c", "curl https://rclone.org/install.sh | sudo bash"])
console.print("[green]rclone installed via official script[/green]")
console.print("[green]rclone installed via official script[/green]")
except RcloneInstallError:
raise RcloneInstallError(
"Failed to install rclone. Please install manually: sudo snap install rclone"
@@ -116,8 +117,16 @@ def install_rclone_windows() -> None:
if shutil.which("winget"):
try:
console.print("[blue]Installing rclone via winget...[/blue]")
run_command(["winget", "install", "Rclone.Rclone"])
console.print("[green]✓ rclone installed via winget[/green]")
run_command(
[
"winget",
"install",
"Rclone.Rclone",
"--accept-source-agreements",
"--accept-package-agreements",
]
)
console.print("[green]rclone installed via winget[/green]")
return
except RcloneInstallError:
console.print("[yellow]winget installation failed, trying chocolatey...[/yellow]")
@@ -127,7 +136,7 @@ def install_rclone_windows() -> None:
try:
console.print("[blue]Installing rclone via chocolatey...[/blue]")
run_command(["choco", "install", "rclone", "-y"])
console.print("[green]rclone installed via chocolatey[/green]")
console.print("[green]rclone installed via chocolatey[/green]")
return
except RcloneInstallError:
console.print("[yellow]chocolatey installation failed, trying scoop...[/yellow]")
@@ -137,16 +146,30 @@ def install_rclone_windows() -> None:
try:
console.print("[blue]Installing rclone via scoop...[/blue]")
run_command(["scoop", "install", "rclone"])
console.print("[green]rclone installed via scoop[/green]")
console.print("[green]rclone installed via scoop[/green]")
return
except RcloneInstallError:
console.print("[yellow]scoop installation failed[/yellow]")
# No package manager available
raise RcloneInstallError(
"Could not install rclone automatically. Please install a package manager "
"(winget, chocolatey, or scoop) or install rclone manually from https://rclone.org/downloads/"
# No package manager available - provide detailed instructions
error_msg = (
"Could not install rclone automatically.\n\n"
"Windows requires a package manager to install rclone. Options:\n\n"
"1. Install winget (recommended, built into Windows 11):\n"
" - Windows 11: Already installed\n"
" - Windows 10: Install 'App Installer' from Microsoft Store\n"
" - Then run: bm cloud setup\n\n"
"2. Install chocolatey:\n"
" - Visit: https://chocolatey.org/install\n"
" - Then run: bm cloud setup\n\n"
"3. Install scoop:\n"
" - Visit: https://scoop.sh\n"
" - Then run: bm cloud setup\n\n"
"4. Manual installation:\n"
" - Download from: https://rclone.org/downloads/\n"
" - Extract and add to PATH\n"
)
raise RcloneInstallError(error_msg)
def install_rclone(platform_override: Optional[str] = None) -> None:
@@ -165,6 +188,7 @@ def install_rclone(platform_override: Optional[str] = None) -> None:
install_rclone_linux()
elif platform_name == "windows":
install_rclone_windows()
refresh_windows_path()
else:
raise RcloneInstallError(f"Unsupported platform: {platform_name}")
@@ -172,7 +196,7 @@ def install_rclone(platform_override: Optional[str] = None) -> None:
if not is_rclone_installed():
raise RcloneInstallError("rclone installation completed but command not found in PATH")
console.print("[green]rclone installation completed successfully[/green]")
console.print("[green]rclone installation completed successfully[/green]")
except RcloneInstallError:
raise
@@ -180,6 +204,47 @@ def install_rclone(platform_override: Optional[str] = None) -> None:
raise RcloneInstallError(f"Unexpected error during installation: {e}") from e
def refresh_windows_path() -> None:
"""Refresh the Windows PATH environment variable for the current session."""
if platform.system().lower() != "windows":
return
# Importing here after performing platform detection. Also note that we have to ignore pylance/pyright
# warnings about winreg attributes so that "errors" don't appear on non-Windows platforms.
import winreg
user_key_path = r"Environment"
system_key_path = r"System\CurrentControlSet\Control\Session Manager\Environment"
new_path = ""
# Read user PATH
try:
reg_key = winreg.OpenKey(winreg.HKEY_CURRENT_USER, user_key_path, 0, winreg.KEY_READ) # type: ignore[reportAttributeAccessIssue]
user_path, _ = winreg.QueryValueEx(reg_key, "PATH") # type: ignore[reportAttributeAccessIssue]
winreg.CloseKey(reg_key) # type: ignore[reportAttributeAccessIssue]
except Exception:
user_path = ""
# Read system PATH
try:
reg_key = winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE, system_key_path, 0, winreg.KEY_READ) # type: ignore[reportAttributeAccessIssue]
system_path, _ = winreg.QueryValueEx(reg_key, "PATH") # type: ignore[reportAttributeAccessIssue]
winreg.CloseKey(reg_key) # type: ignore[reportAttributeAccessIssue]
except Exception:
system_path = ""
# Merge user and system PATHs (system first, then user)
if system_path and user_path:
new_path = system_path + ";" + user_path
elif system_path:
new_path = system_path
elif user_path:
new_path = user_path
if new_path:
os.environ["PATH"] = new_path
def get_rclone_version() -> Optional[str]:
"""Get the installed rclone version."""
if not is_rclone_installed():
@@ -0,0 +1,233 @@
"""WebDAV upload functionality for basic-memory projects."""
import os
from pathlib import Path
import aiofiles
import httpx
from basic_memory.ignore_utils import load_gitignore_patterns, should_ignore_path
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.tools.utils import call_put
# Archive file extensions that should be skipped during upload
ARCHIVE_EXTENSIONS = {".zip", ".tar", ".gz", ".bz2", ".xz", ".7z", ".rar", ".tgz", ".tbz2"}
async def upload_path(
local_path: Path,
project_name: str,
verbose: bool = False,
use_gitignore: bool = True,
dry_run: bool = False,
) -> bool:
"""
Upload a file or directory to cloud project via WebDAV.
Args:
local_path: Path to local file or directory
project_name: Name of cloud project (destination)
verbose: Show detailed information about filtering and upload
use_gitignore: If False, skip .gitignore patterns (still use .bmignore)
dry_run: If True, show what would be uploaded without uploading
Returns:
True if upload succeeded, False otherwise
"""
try:
# Resolve path
local_path = local_path.resolve()
# Check if path exists
if not local_path.exists():
print(f"Error: Path does not exist: {local_path}")
return False
# Get files to upload
if local_path.is_file():
files_to_upload = [(local_path, local_path.name)]
if verbose:
print(f"Uploading single file: {local_path.name}")
else:
files_to_upload = _get_files_to_upload(local_path, verbose, use_gitignore)
if not files_to_upload:
print("No files found to upload")
if verbose:
print(
"\nTip: Use --verbose to see which files are being filtered, "
"or --no-gitignore to skip .gitignore patterns"
)
return True
print(f"Found {len(files_to_upload)} file(s) to upload")
# Calculate total size
total_bytes = sum(file_path.stat().st_size for file_path, _ in files_to_upload)
skipped_count = 0
# If dry run, just show what would be uploaded
if dry_run:
print("\nFiles that would be uploaded:")
for file_path, relative_path in files_to_upload:
# Skip archive files
if _is_archive_file(file_path):
print(f" [SKIP] {relative_path} (archive file)")
skipped_count += 1
continue
size = file_path.stat().st_size
if size < 1024:
size_str = f"{size} bytes"
elif size < 1024 * 1024:
size_str = f"{size / 1024:.1f} KB"
else:
size_str = f"{size / (1024 * 1024):.1f} MB"
print(f" {relative_path} ({size_str})")
else:
# Upload files using httpx
async with get_client() as client:
for i, (file_path, relative_path) in enumerate(files_to_upload, 1):
# Skip archive files (zip, tar, gz, etc.)
if _is_archive_file(file_path):
print(
f"Skipping archive file: {relative_path} ({i}/{len(files_to_upload)})"
)
skipped_count += 1
continue
# Build remote path: /webdav/{project_name}/{relative_path}
remote_path = f"/webdav/{project_name}/{relative_path}"
print(f"Uploading {relative_path} ({i}/{len(files_to_upload)})")
# Get file modification time
file_stat = file_path.stat()
mtime = int(file_stat.st_mtime)
# Read file content asynchronously
async with aiofiles.open(file_path, "rb") as f:
content = await f.read()
# Upload via HTTP PUT to WebDAV endpoint with mtime header
# Using X-OC-Mtime (ownCloud/Nextcloud standard)
response = await call_put(
client, remote_path, content=content, headers={"X-OC-Mtime": str(mtime)}
)
response.raise_for_status()
# Format total size based on magnitude
if total_bytes < 1024:
size_str = f"{total_bytes} bytes"
elif total_bytes < 1024 * 1024:
size_str = f"{total_bytes / 1024:.1f} KB"
else:
size_str = f"{total_bytes / (1024 * 1024):.1f} MB"
uploaded_count = len(files_to_upload) - skipped_count
if dry_run:
print(f"\nTotal: {uploaded_count} file(s) ({size_str})")
if skipped_count > 0:
print(f" Would skip {skipped_count} archive file(s)")
else:
print(f"✓ Upload complete: {uploaded_count} file(s) ({size_str})")
if skipped_count > 0:
print(f" Skipped {skipped_count} archive file(s)")
return True
except httpx.HTTPStatusError as e:
print(f"Upload failed: HTTP {e.response.status_code} - {e.response.text}")
return False
except Exception as e:
print(f"Upload failed: {e}")
return False
def _is_archive_file(file_path: Path) -> bool:
"""
Check if a file is an archive file based on its extension.
Args:
file_path: Path to the file to check
Returns:
True if file is an archive, False otherwise
"""
return file_path.suffix.lower() in ARCHIVE_EXTENSIONS
def _get_files_to_upload(
directory: Path, verbose: bool = False, use_gitignore: bool = True
) -> list[tuple[Path, str]]:
"""
Get list of files to upload from directory.
Uses .bmignore and optionally .gitignore patterns for filtering.
Args:
directory: Directory to scan
verbose: Show detailed filtering information
use_gitignore: If False, skip .gitignore patterns (still use .bmignore)
Returns:
List of (absolute_path, relative_path) tuples
"""
files = []
ignored_files = []
# Load ignore patterns from .bmignore and optionally .gitignore
ignore_patterns = load_gitignore_patterns(directory, use_gitignore=use_gitignore)
if verbose:
gitignore_path = directory / ".gitignore"
gitignore_exists = gitignore_path.exists() and use_gitignore
print(f"\nScanning directory: {directory}")
print("Using .bmignore: Yes")
print(f"Using .gitignore: {'Yes' if gitignore_exists else 'No'}")
print(f"Ignore patterns loaded: {len(ignore_patterns)}")
if ignore_patterns and len(ignore_patterns) <= 20:
print(f"Patterns: {', '.join(sorted(ignore_patterns))}")
print()
# Walk through directory
for root, dirs, filenames in os.walk(directory):
root_path = Path(root)
# Filter directories based on ignore patterns
filtered_dirs = []
for d in dirs:
dir_path = root_path / d
if should_ignore_path(dir_path, directory, ignore_patterns):
if verbose:
rel_path = dir_path.relative_to(directory)
print(f" [IGNORED DIR] {rel_path}/")
else:
filtered_dirs.append(d)
dirs[:] = filtered_dirs
# Process files
for filename in filenames:
file_path = root_path / filename
# Calculate relative path for display/remote
rel_path = file_path.relative_to(directory)
remote_path = str(rel_path).replace("\\", "/")
# Check if file should be ignored
if should_ignore_path(file_path, directory, ignore_patterns):
ignored_files.append(remote_path)
if verbose:
print(f" [IGNORED] {remote_path}")
continue
if verbose:
print(f" [INCLUDE] {remote_path}")
files.append((file_path, remote_path))
if verbose:
print("\nSummary:")
print(f" Files to upload: {len(files)}")
print(f" Files ignored: {len(ignored_files)}")
return files
@@ -0,0 +1,124 @@
"""Upload CLI commands for basic-memory projects."""
import asyncio
from pathlib import Path
import typer
from rich.console import Console
from basic_memory.cli.app import cloud_app
from basic_memory.cli.commands.cloud.cloud_utils import (
create_cloud_project,
project_exists,
sync_project,
)
from basic_memory.cli.commands.cloud.upload import upload_path
console = Console()
@cloud_app.command("upload")
def upload(
path: Path = typer.Argument(
...,
help="Path to local file or directory to upload",
exists=True,
readable=True,
resolve_path=True,
),
project: str = typer.Option(
...,
"--project",
"-p",
help="Cloud project name (destination)",
),
create_project: bool = typer.Option(
False,
"--create-project",
"-c",
help="Create project if it doesn't exist",
),
sync: bool = typer.Option(
True,
"--sync/--no-sync",
help="Sync project after upload (default: true)",
),
verbose: bool = typer.Option(
False,
"--verbose",
"-v",
help="Show detailed information about file filtering and upload",
),
no_gitignore: bool = typer.Option(
False,
"--no-gitignore",
help="Skip .gitignore patterns (still respects .bmignore)",
),
dry_run: bool = typer.Option(
False,
"--dry-run",
help="Show what would be uploaded without actually uploading",
),
) -> None:
"""Upload local files or directories to cloud project via WebDAV.
Examples:
bm cloud upload ~/my-notes --project research
bm cloud upload notes.md --project research --create-project
bm cloud upload ~/docs --project work --no-sync
bm cloud upload ./history --project proto --verbose
bm cloud upload ./notes --project work --no-gitignore
bm cloud upload ./files --project test --dry-run
"""
async def _upload():
# Check if project exists
if not await project_exists(project):
if create_project:
console.print(f"[blue]Creating cloud project '{project}'...[/blue]")
try:
await create_cloud_project(project)
console.print(f"[green]Created project '{project}'[/green]")
except Exception as e:
console.print(f"[red]Failed to create project: {e}[/red]")
raise typer.Exit(1)
else:
console.print(
f"[red]Project '{project}' does not exist.[/red]\n"
f"[yellow]Options:[/yellow]\n"
f" 1. Create it first: bm project add {project}\n"
f" 2. Use --create-project flag to create automatically"
)
raise typer.Exit(1)
# Perform upload (or dry run)
if dry_run:
console.print(
f"[yellow]DRY RUN: Showing what would be uploaded to '{project}'[/yellow]"
)
else:
console.print(f"[blue]Uploading {path} to project '{project}'...[/blue]")
success = await upload_path(
path, project, verbose=verbose, use_gitignore=not no_gitignore, dry_run=dry_run
)
if not success:
console.print("[red]Upload failed[/red]")
raise typer.Exit(1)
if dry_run:
console.print("[yellow]DRY RUN complete - no files were uploaded[/yellow]")
else:
console.print(f"[green]Successfully uploaded to '{project}'[/green]")
# Sync project if requested (skip on dry run)
# Force full scan after bisync to ensure database is up-to-date with synced files
if sync and not dry_run:
console.print(f"[blue]Syncing project '{project}'...[/blue]")
try:
await sync_project(project, force_full=True)
except Exception as e:
console.print(f"[yellow]Warning: Sync failed: {e}[/yellow]")
console.print("[dim]Files uploaded but may not be indexed yet[/dim]")
asyncio.run(_upload())
+14 -6
View File
@@ -16,17 +16,25 @@ from basic_memory.schemas import ProjectInfoResponse
console = Console()
async def run_sync(project: Optional[str] = None):
"""Run sync operation via API endpoint."""
async def run_sync(project: Optional[str] = None, force_full: bool = False):
"""Run sync operation via API endpoint.
Args:
project: Optional project name
force_full: If True, force a full scan bypassing watermark optimization
"""
try:
async with get_client() as client:
project_item = await get_active_project(client, project, None)
response = await call_post(client, f"{project_item.project_url}/project/sync")
url = f"{project_item.project_url}/project/sync"
if force_full:
url += "?force_full=true"
response = await call_post(client, url)
data = response.json()
console.print(f"[green]{data['message']}[/green]")
console.print(f"[green]{data['message']}[/green]")
except (ToolError, ValueError) as e:
console.print(f"[red]Sync failed: {e}[/red]")
console.print(f"[red]Sync failed: {e}[/red]")
raise typer.Exit(1)
@@ -39,5 +47,5 @@ async def get_project_info(project: str):
response = await call_get(client, f"{project_item.project_url}/project/info")
return ProjectInfoResponse.model_validate(response.json())
except (ToolError, ValueError) as e:
console.print(f"[red]Sync failed: {e}[/red]")
console.print(f"[red]Sync failed: {e}[/red]")
raise typer.Exit(1)
+3 -3
View File
@@ -37,8 +37,8 @@ def reset(
logger.info("Database reset complete")
if reindex:
# Import and run sync
from basic_memory.cli.commands.sync import sync
# Run database sync directly
from basic_memory.cli.commands.command_utils import run_sync
logger.info("Rebuilding search index from filesystem...")
sync(watch=False) # pyright: ignore
asyncio.run(run_sync(project=None))
+648 -112
View File
@@ -22,17 +22,26 @@ from basic_memory.mcp.tools.utils import call_post
from basic_memory.schemas.project_info import ProjectStatusResponse
from basic_memory.mcp.tools.utils import call_delete
from basic_memory.mcp.tools.utils import call_put
from basic_memory.utils import generate_permalink
from basic_memory.utils import generate_permalink, normalize_project_path
from basic_memory.mcp.tools.utils import call_patch
# Import rclone commands for project sync
from basic_memory.cli.commands.cloud.rclone_commands import (
SyncProject,
RcloneError,
project_sync,
project_bisync,
project_check,
project_ls,
)
from basic_memory.cli.commands.cloud.bisync_commands import get_mount_info
console = Console()
# Create a project subcommand
project_app = typer.Typer(help="Manage multiple Basic Memory projects")
app.add_typer(project_app, name="project")
config = ConfigManager().config
def format_path(path: str) -> str:
"""Format a path for display, using ~ for home directory."""
@@ -53,15 +62,41 @@ def list_projects() -> None:
try:
result = asyncio.run(_list_projects())
config = ConfigManager().config
table = Table(title="Basic Memory Projects")
table.add_column("Name", style="cyan")
table.add_column("Path", style="green")
table.add_column("Default", style="magenta")
# Add Local Path column if in cloud mode
if config.cloud_mode_enabled:
table.add_column("Local Path", style="yellow", no_wrap=True, overflow="fold")
# Show Default column in local mode or if default_project_mode is enabled in cloud mode
show_default_column = not config.cloud_mode_enabled or config.default_project_mode
if show_default_column:
table.add_column("Default", style="magenta")
for project in result.projects:
is_default = "" if project.is_default else ""
table.add_row(project.name, format_path(project.path), is_default)
is_default = "[X]" if project.is_default else ""
normalized_path = normalize_project_path(project.path)
# Build row based on mode
row = [project.name, format_path(normalized_path)]
# Add local path if in cloud mode
if config.cloud_mode_enabled:
local_path = ""
if project.name in config.cloud_projects:
local_path = config.cloud_projects[project.name].local_path or ""
local_path = format_path(local_path)
row.append(local_path)
# Add default indicator if showing default column
if show_default_column:
row.append(is_default)
table.add_row(*row)
console.print(table)
except Exception as e:
@@ -69,40 +104,52 @@ def list_projects() -> None:
raise typer.Exit(1)
if config.cloud_mode_enabled:
@project_app.command("add")
def add_project(
name: str = typer.Argument(..., help="Name of the project"),
path: str = typer.Argument(
None, help="Path to the project directory (required for local mode)"
),
local_path: str = typer.Option(
None, "--local-path", help="Local sync path for cloud mode (optional)"
),
set_default: bool = typer.Option(False, "--default", help="Set as default project"),
) -> None:
"""Add a new project.
@project_app.command("add")
def add_project_cloud(
name: str = typer.Argument(..., help="Name of the project"),
set_default: bool = typer.Option(False, "--default", help="Set as default project"),
) -> None:
"""Add a new project to Basic Memory Cloud"""
Cloud mode examples:\n
bm project add research # No local sync\n
bm project add research --local-path ~/docs # With local sync\n
Local mode example:\n
bm project add research ~/Documents/research
"""
config = ConfigManager().config
# Resolve local sync path early (needed for both cloud and local mode)
local_sync_path: str | None = None
if local_path:
local_sync_path = Path(os.path.abspath(os.path.expanduser(local_path))).as_posix()
if config.cloud_mode_enabled:
# Cloud mode: path auto-generated from name, local sync is optional
async def _add_project():
async with get_client() as client:
data = {"name": name, "path": generate_permalink(name), "set_default": set_default}
data = {
"name": name,
"path": generate_permalink(name),
"local_sync_path": local_sync_path,
"set_default": set_default,
}
response = await call_post(client, "/projects/projects", json=data)
return ProjectStatusResponse.model_validate(response.json())
try:
result = asyncio.run(_add_project())
console.print(f"[green]{result.message}[/green]")
except Exception as e:
console.print(f"[red]Error adding project: {str(e)}[/red]")
else:
# Local mode: path is required
if path is None:
console.print("[red]Error: path argument is required in local mode[/red]")
raise typer.Exit(1)
# Display usage hint
console.print("\nTo use this project:")
console.print(f" basic-memory --project={name} <command>")
else:
@project_app.command("add")
def add_project(
name: str = typer.Argument(..., help="Name of the project"),
path: str = typer.Argument(..., help="Path to the project directory"),
set_default: bool = typer.Option(False, "--default", help="Set as default project"),
) -> None:
"""Add a new project."""
# Resolve to absolute path
resolved_path = Path(os.path.abspath(os.path.expanduser(path))).as_posix()
@@ -112,119 +159,608 @@ else:
response = await call_post(client, "/projects/projects", json=data)
return ProjectStatusResponse.model_validate(response.json())
try:
result = asyncio.run(_add_project())
console.print(f"[green]{result.message}[/green]")
except Exception as e:
console.print(f"[red]Error adding project: {str(e)}[/red]")
raise typer.Exit(1)
try:
result = asyncio.run(_add_project())
console.print(f"[green]{result.message}[/green]")
# Display usage hint
console.print("\nTo use this project:")
console.print(f" basic-memory --project={name} <command>")
# Save local sync path to config if in cloud mode
if config.cloud_mode_enabled and local_sync_path:
from basic_memory.config import CloudProjectConfig
# Create local directory if it doesn't exist
local_dir = Path(local_sync_path)
local_dir.mkdir(parents=True, exist_ok=True)
# Update config with sync path
config.cloud_projects[name] = CloudProjectConfig(
local_path=local_sync_path,
last_sync=None,
bisync_initialized=False,
)
ConfigManager().save_config(config)
console.print(f"\n[green]Local sync path configured: {local_sync_path}[/green]")
console.print("\nNext steps:")
console.print(f" 1. Preview: bm project bisync --name {name} --resync --dry-run")
console.print(f" 2. Sync: bm project bisync --name {name} --resync")
except Exception as e:
console.print(f"[red]Error adding project: {str(e)}[/red]")
raise typer.Exit(1)
@project_app.command("sync-setup")
def setup_project_sync(
name: str = typer.Argument(..., help="Project name"),
local_path: str = typer.Argument(..., help="Local sync directory"),
) -> None:
"""Configure local sync for an existing cloud project.
Example:
bm project sync-setup research ~/Documents/research
"""
config_manager = ConfigManager()
config = config_manager.config
if not config.cloud_mode_enabled:
console.print("[red]Error: sync-setup only available in cloud mode[/red]")
raise typer.Exit(1)
async def _verify_project_exists():
"""Verify the project exists on cloud by listing all projects."""
async with get_client() as client:
response = await call_get(client, "/projects/projects")
project_list = response.json()
project_names = [p["name"] for p in project_list["projects"]]
if name not in project_names:
raise ValueError(f"Project '{name}' not found on cloud")
return True
try:
# Verify project exists on cloud
asyncio.run(_verify_project_exists())
# Resolve and create local path
resolved_path = Path(os.path.abspath(os.path.expanduser(local_path)))
resolved_path.mkdir(parents=True, exist_ok=True)
# Update local config with sync path
from basic_memory.config import CloudProjectConfig
config.cloud_projects[name] = CloudProjectConfig(
local_path=resolved_path.as_posix(),
last_sync=None,
bisync_initialized=False,
)
config_manager.save_config(config)
console.print(f"[green]Sync configured for project '{name}'[/green]")
console.print(f"\nLocal sync path: {resolved_path}")
console.print("\nNext steps:")
console.print(f" 1. Preview: bm project bisync --name {name} --resync --dry-run")
console.print(f" 2. Sync: bm project bisync --name {name} --resync")
except Exception as e:
console.print(f"[red]Error configuring sync: {str(e)}[/red]")
raise typer.Exit(1)
@project_app.command("remove")
def remove_project(
name: str = typer.Argument(..., help="Name of the project to remove"),
delete_notes: bool = typer.Option(
False, "--delete-notes", help="Delete project files from disk"
),
) -> None:
"""Remove a project."""
async def _remove_project():
async with get_client() as client:
project_permalink = generate_permalink(name)
response = await call_delete(client, f"/projects/{project_permalink}")
response = await call_delete(
client, f"/projects/{project_permalink}?delete_notes={delete_notes}"
)
return ProjectStatusResponse.model_validate(response.json())
try:
# Get config to check for local sync path and bisync state
config = ConfigManager().config
local_path = None
has_bisync_state = False
if config.cloud_mode_enabled and name in config.cloud_projects:
local_path = config.cloud_projects[name].local_path
# Check for bisync state
from basic_memory.cli.commands.cloud.rclone_commands import get_project_bisync_state
bisync_state_path = get_project_bisync_state(name)
has_bisync_state = bisync_state_path.exists()
# Remove project from cloud/API
result = asyncio.run(_remove_project())
console.print(f"[green]{result.message}[/green]")
# Clean up local sync directory if it exists and delete_notes is True
if delete_notes and local_path:
local_dir = Path(local_path)
if local_dir.exists():
import shutil
shutil.rmtree(local_dir)
console.print(f"[green]Removed local sync directory: {local_path}[/green]")
# Clean up bisync state if it exists
if has_bisync_state:
from basic_memory.cli.commands.cloud.rclone_commands import get_project_bisync_state
import shutil
bisync_state_path = get_project_bisync_state(name)
if bisync_state_path.exists():
shutil.rmtree(bisync_state_path)
console.print("[green]Removed bisync state[/green]")
# Clean up cloud_projects config entry
if config.cloud_mode_enabled and name in config.cloud_projects:
del config.cloud_projects[name]
ConfigManager().save_config(config)
# Show informative message if files were not deleted
if not delete_notes:
if local_path:
console.print(f"[yellow]Note: Local files remain at {local_path}[/yellow]")
except Exception as e:
console.print(f"[red]Error removing project: {str(e)}[/red]")
raise typer.Exit(1)
# Show this message regardless of method used
console.print("[yellow]Note: The project files have not been deleted from disk.[/yellow]")
@project_app.command("default")
def set_default_project(
name: str = typer.Argument(..., help="Name of the project to set as CLI default"),
) -> None:
"""Set the default project when 'config.default_project_mode' is set.
Note: This command is only available in local mode.
"""
config = ConfigManager().config
if config.cloud_mode_enabled:
console.print("[red]Error: 'default' command is not available in cloud mode[/red]")
raise typer.Exit(1)
async def _set_default():
async with get_client() as client:
project_permalink = generate_permalink(name)
response = await call_put(client, f"/projects/{project_permalink}/default")
return ProjectStatusResponse.model_validate(response.json())
try:
result = asyncio.run(_set_default())
console.print(f"[green]{result.message}[/green]")
except Exception as e:
console.print(f"[red]Error setting default project: {str(e)}[/red]")
raise typer.Exit(1)
if not config.cloud_mode_enabled:
@project_app.command("sync-config")
def synchronize_projects() -> None:
"""Synchronize project config between configuration file and database.
@project_app.command("default")
def set_default_project(
name: str = typer.Argument(..., help="Name of the project to set as CLI default"),
) -> None:
"""Set the default project when 'config.default_project_mode' is set."""
Note: This command is only available in local mode.
"""
config = ConfigManager().config
async def _set_default():
if config.cloud_mode_enabled:
console.print("[red]Error: 'sync-config' command is not available in cloud mode[/red]")
raise typer.Exit(1)
async def _sync_config():
async with get_client() as client:
response = await call_post(client, "/projects/config/sync")
return ProjectStatusResponse.model_validate(response.json())
try:
result = asyncio.run(_sync_config())
console.print(f"[green]{result.message}[/green]")
except Exception as e: # pragma: no cover
console.print(f"[red]Error synchronizing projects: {str(e)}[/red]")
raise typer.Exit(1)
@project_app.command("move")
def move_project(
name: str = typer.Argument(..., help="Name of the project to move"),
new_path: str = typer.Argument(..., help="New absolute path for the project"),
) -> None:
"""Move a project to a new location.
Note: This command is only available in local mode.
"""
config = ConfigManager().config
if config.cloud_mode_enabled:
console.print("[red]Error: 'move' command is not available in cloud mode[/red]")
raise typer.Exit(1)
# Resolve to absolute path
resolved_path = Path(os.path.abspath(os.path.expanduser(new_path))).as_posix()
async def _move_project():
async with get_client() as client:
data = {"path": resolved_path}
project_permalink = generate_permalink(name)
# TODO fix route to use ProjectPathDep
response = await call_patch(client, f"/{name}/project/{project_permalink}", json=data)
return ProjectStatusResponse.model_validate(response.json())
try:
result = asyncio.run(_move_project())
console.print(f"[green]{result.message}[/green]")
# Show important file movement reminder
console.print() # Empty line for spacing
console.print(
Panel(
"[bold red]IMPORTANT:[/bold red] Project configuration updated successfully.\n\n"
"[yellow]You must manually move your project files from the old location to:[/yellow]\n"
f"[cyan]{resolved_path}[/cyan]\n\n"
"[dim]Basic Memory has only updated the configuration - your files remain in their original location.[/dim]",
title="Manual File Movement Required",
border_style="yellow",
expand=False,
)
)
except Exception as e:
console.print(f"[red]Error moving project: {str(e)}[/red]")
raise typer.Exit(1)
@project_app.command("sync")
def sync_project_command(
name: str = typer.Option(..., "--name", help="Project name to sync"),
dry_run: bool = typer.Option(False, "--dry-run", help="Preview changes without syncing"),
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed output"),
) -> None:
"""One-way sync: local -> cloud (make cloud identical to local).
Example:
bm project sync --name research
bm project sync --name research --dry-run
"""
config = ConfigManager().config
if not config.cloud_mode_enabled:
console.print("[red]Error: sync only available in cloud mode[/red]")
raise typer.Exit(1)
try:
# Get tenant info for bucket name
tenant_info = asyncio.run(get_mount_info())
bucket_name = tenant_info.bucket_name
# Get project info
async def _get_project():
async with get_client() as client:
project_permalink = generate_permalink(name)
response = await call_put(client, f"/projects/{project_permalink}/default")
return ProjectStatusResponse.model_validate(response.json())
response = await call_get(client, "/projects/projects")
projects_list = ProjectList.model_validate(response.json())
for proj in projects_list.projects:
if generate_permalink(proj.name) == generate_permalink(name):
return proj
return None
try:
result = asyncio.run(_set_default())
console.print(f"[green]{result.message}[/green]")
except Exception as e:
console.print(f"[red]Error setting default project: {str(e)}[/red]")
project_data = asyncio.run(_get_project())
if not project_data:
console.print(f"[red]Error: Project '{name}' not found[/red]")
raise typer.Exit(1)
@project_app.command("sync-config")
def synchronize_projects() -> None:
"""Synchronize project config between configuration file and database."""
# Get local_sync_path from cloud_projects config
local_sync_path = None
if name in config.cloud_projects:
local_sync_path = config.cloud_projects[name].local_path
async def _sync_config():
async with get_client() as client:
response = await call_post(client, "/projects/config/sync")
return ProjectStatusResponse.model_validate(response.json())
try:
result = asyncio.run(_sync_config())
console.print(f"[green]{result.message}[/green]")
except Exception as e: # pragma: no cover
console.print(f"[red]Error synchronizing projects: {str(e)}[/red]")
if not local_sync_path:
console.print(f"[red]Error: Project '{name}' has no local_sync_path configured[/red]")
console.print(f"\nConfigure sync with: bm project sync-setup {name} ~/path/to/local")
raise typer.Exit(1)
@project_app.command("move")
def move_project(
name: str = typer.Argument(..., help="Name of the project to move"),
new_path: str = typer.Argument(..., help="New absolute path for the project"),
) -> None:
"""Move a project to a new location."""
# Resolve to absolute path
resolved_path = Path(os.path.abspath(os.path.expanduser(new_path))).as_posix()
# Create SyncProject
sync_project = SyncProject(
name=project_data.name,
path=normalize_project_path(project_data.path),
local_sync_path=local_sync_path,
)
async def _move_project():
# Run sync
console.print(f"[blue]Syncing {name} (local -> cloud)...[/blue]")
success = project_sync(sync_project, bucket_name, dry_run=dry_run, verbose=verbose)
if success:
console.print(f"[green]{name} synced successfully[/green]")
# Trigger database sync if not a dry run
if not dry_run:
async def _trigger_db_sync():
async with get_client() as client:
permalink = generate_permalink(name)
response = await call_post(
client, f"/{permalink}/project/sync?force_full=true", json={}
)
return response.json()
try:
result = asyncio.run(_trigger_db_sync())
console.print(f"[dim]Database sync initiated: {result.get('message')}[/dim]")
except Exception as e:
console.print(f"[yellow]Warning: Could not trigger database sync: {e}[/yellow]")
else:
console.print(f"[red]{name} sync failed[/red]")
raise typer.Exit(1)
except RcloneError as e:
console.print(f"[red]Sync error: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
@project_app.command("bisync")
def bisync_project_command(
name: str = typer.Option(..., "--name", help="Project name to bisync"),
dry_run: bool = typer.Option(False, "--dry-run", help="Preview changes without syncing"),
resync: bool = typer.Option(False, "--resync", help="Force new baseline"),
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed output"),
) -> None:
"""Two-way sync: local <-> cloud (bidirectional sync).
Examples:
bm project bisync --name research --resync # First time
bm project bisync --name research # Subsequent syncs
bm project bisync --name research --dry-run # Preview changes
"""
config = ConfigManager().config
if not config.cloud_mode_enabled:
console.print("[red]Error: bisync only available in cloud mode[/red]")
raise typer.Exit(1)
try:
# Get tenant info for bucket name
tenant_info = asyncio.run(get_mount_info())
bucket_name = tenant_info.bucket_name
# Get project info
async def _get_project():
async with get_client() as client:
data = {"path": resolved_path}
project_permalink = generate_permalink(name)
response = await call_get(client, "/projects/projects")
projects_list = ProjectList.model_validate(response.json())
for proj in projects_list.projects:
if generate_permalink(proj.name) == generate_permalink(name):
return proj
return None
# TODO fix route to use ProjectPathDep
response = await call_patch(
client, f"/{name}/project/{project_permalink}", json=data
)
return ProjectStatusResponse.model_validate(response.json())
project_data = asyncio.run(_get_project())
if not project_data:
console.print(f"[red]Error: Project '{name}' not found[/red]")
raise typer.Exit(1)
try:
result = asyncio.run(_move_project())
console.print(f"[green]{result.message}[/green]")
# Get local_sync_path from cloud_projects config
local_sync_path = None
if name in config.cloud_projects:
local_sync_path = config.cloud_projects[name].local_path
# Show important file movement reminder
console.print() # Empty line for spacing
if not local_sync_path:
console.print(f"[red]Error: Project '{name}' has no local_sync_path configured[/red]")
console.print(f"\nConfigure sync with: bm project sync-setup {name} ~/path/to/local")
raise typer.Exit(1)
# Create SyncProject
sync_project = SyncProject(
name=project_data.name,
path=normalize_project_path(project_data.path),
local_sync_path=local_sync_path,
)
# Run bisync
console.print(f"[blue]Bisync {name} (local <-> cloud)...[/blue]")
success = project_bisync(
sync_project, bucket_name, dry_run=dry_run, resync=resync, verbose=verbose
)
if success:
console.print(f"[green]{name} bisync completed successfully[/green]")
# Update config
config.cloud_projects[name].last_sync = datetime.now()
config.cloud_projects[name].bisync_initialized = True
ConfigManager().save_config(config)
# Trigger database sync if not a dry run
if not dry_run:
async def _trigger_db_sync():
async with get_client() as client:
permalink = generate_permalink(name)
response = await call_post(
client, f"/{permalink}/project/sync?force_full=true", json={}
)
return response.json()
try:
result = asyncio.run(_trigger_db_sync())
console.print(f"[dim]Database sync initiated: {result.get('message')}[/dim]")
except Exception as e:
console.print(f"[yellow]Warning: Could not trigger database sync: {e}[/yellow]")
else:
console.print(f"[red]{name} bisync failed[/red]")
raise typer.Exit(1)
except RcloneError as e:
console.print(f"[red]Bisync error: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
@project_app.command("check")
def check_project_command(
name: str = typer.Option(..., "--name", help="Project name to check"),
one_way: bool = typer.Option(False, "--one-way", help="Check one direction only (faster)"),
) -> None:
"""Verify file integrity between local and cloud.
Example:
bm project check --name research
"""
config = ConfigManager().config
if not config.cloud_mode_enabled:
console.print("[red]Error: check only available in cloud mode[/red]")
raise typer.Exit(1)
try:
# Get tenant info for bucket name
tenant_info = asyncio.run(get_mount_info())
bucket_name = tenant_info.bucket_name
# Get project info
async def _get_project():
async with get_client() as client:
response = await call_get(client, "/projects/projects")
projects_list = ProjectList.model_validate(response.json())
for proj in projects_list.projects:
if generate_permalink(proj.name) == generate_permalink(name):
return proj
return None
project_data = asyncio.run(_get_project())
if not project_data:
console.print(f"[red]Error: Project '{name}' not found[/red]")
raise typer.Exit(1)
# Get local_sync_path from cloud_projects config
local_sync_path = None
if name in config.cloud_projects:
local_sync_path = config.cloud_projects[name].local_path
if not local_sync_path:
console.print(f"[red]Error: Project '{name}' has no local_sync_path configured[/red]")
console.print(f"\nConfigure sync with: bm project sync-setup {name} ~/path/to/local")
raise typer.Exit(1)
# Create SyncProject
sync_project = SyncProject(
name=project_data.name,
path=normalize_project_path(project_data.path),
local_sync_path=local_sync_path,
)
# Run check
console.print(f"[blue]Checking {name} integrity...[/blue]")
match = project_check(sync_project, bucket_name, one_way=one_way)
if match:
console.print(f"[green]{name} files match[/green]")
else:
console.print(f"[yellow]!{name} has differences[/yellow]")
except RcloneError as e:
console.print(f"[red]Check error: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
@project_app.command("bisync-reset")
def bisync_reset(
name: str = typer.Argument(..., help="Project name to reset bisync state for"),
) -> None:
"""Clear bisync state for a project.
This removes the bisync metadata files, forcing a fresh --resync on next bisync.
Useful when bisync gets into an inconsistent state or when remote path changes.
"""
from basic_memory.cli.commands.cloud.rclone_commands import get_project_bisync_state
import shutil
try:
state_path = get_project_bisync_state(name)
if not state_path.exists():
console.print(f"[yellow]No bisync state found for project '{name}'[/yellow]")
return
# Remove the entire state directory
shutil.rmtree(state_path)
console.print(f"[green]Cleared bisync state for project '{name}'[/green]")
console.print("\nNext steps:")
console.print(f" 1. Preview: bm project bisync --name {name} --resync --dry-run")
console.print(f" 2. Sync: bm project bisync --name {name} --resync")
except Exception as e:
console.print(f"[red]Error clearing bisync state: {str(e)}[/red]")
raise typer.Exit(1)
@project_app.command("ls")
def ls_project_command(
name: str = typer.Option(..., "--name", help="Project name to list files from"),
path: str = typer.Argument(None, help="Path within project (optional)"),
) -> None:
"""List files in remote project.
Examples:
bm project ls --name research
bm project ls --name research subfolder
"""
config = ConfigManager().config
if not config.cloud_mode_enabled:
console.print("[red]Error: ls only available in cloud mode[/red]")
raise typer.Exit(1)
try:
# Get tenant info for bucket name
tenant_info = asyncio.run(get_mount_info())
bucket_name = tenant_info.bucket_name
# Get project info
async def _get_project():
async with get_client() as client:
response = await call_get(client, "/projects/projects")
projects_list = ProjectList.model_validate(response.json())
for proj in projects_list.projects:
if generate_permalink(proj.name) == generate_permalink(name):
return proj
return None
project_data = asyncio.run(_get_project())
if not project_data:
console.print(f"[red]Error: Project '{name}' not found[/red]")
raise typer.Exit(1)
# Create SyncProject (local_sync_path not needed for ls)
sync_project = SyncProject(
name=project_data.name,
path=normalize_project_path(project_data.path),
)
# List files
files = project_ls(sync_project, bucket_name, path=path)
if files:
console.print(f"\n[bold]Files in {name}" + (f"/{path}" if path else "") + ":[/bold]")
for file in files:
console.print(f" {file}")
console.print(f"\n[dim]Total: {len(files)} files[/dim]")
else:
console.print(
Panel(
"[bold red]IMPORTANT:[/bold red] Project configuration updated successfully.\n\n"
"[yellow]You must manually move your project files from the old location to:[/yellow]\n"
f"[cyan]{resolved_path}[/cyan]\n\n"
"[dim]Basic Memory has only updated the configuration - your files remain in their original location.[/dim]",
title="⚠️ Manual File Movement Required",
border_style="yellow",
expand=False,
)
f"[yellow]No files found in {name}" + (f"/{path}" if path else "") + "[/yellow]"
)
except Exception as e:
console.print(f"[red]Error moving project: {str(e)}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
@project_app.command("info")
@@ -248,13 +784,13 @@ def display_project_info(
f"[bold]Project:[/bold] {info.project_name}\n"
f"[bold]Path:[/bold] {info.project_path}\n"
f"[bold]Default Project:[/bold] {info.default_project}\n",
title="📊 Basic Memory Project Info",
title="Basic Memory Project Info",
expand=False,
)
)
# Statistics section
stats_table = Table(title="📈 Statistics")
stats_table = Table(title="Statistics")
stats_table.add_column("Metric", style="cyan")
stats_table.add_column("Count", style="green")
@@ -270,7 +806,7 @@ def display_project_info(
# Entity types
if info.statistics.entity_types:
entity_types_table = Table(title="📑 Entity Types")
entity_types_table = Table(title="Entity Types")
entity_types_table.add_column("Type", style="blue")
entity_types_table.add_column("Count", style="green")
@@ -281,7 +817,7 @@ def display_project_info(
# Most connected entities
if info.statistics.most_connected_entities: # pragma: no cover
connected_table = Table(title="🔗 Most Connected Entities")
connected_table = Table(title="Most Connected Entities")
connected_table.add_column("Title", style="blue")
connected_table.add_column("Permalink", style="cyan")
connected_table.add_column("Relations", style="green")
@@ -295,7 +831,7 @@ def display_project_info(
# Recent activity
if info.activity.recently_updated: # pragma: no cover
recent_table = Table(title="🕒 Recent Activity")
recent_table = Table(title="Recent Activity")
recent_table.add_column("Title", style="blue")
recent_table.add_column("Type", style="cyan")
recent_table.add_column("Last Updated", style="green")
@@ -315,7 +851,7 @@ def display_project_info(
console.print(recent_table)
# Available projects
projects_table = Table(title="📁 Available Projects")
projects_table = Table(title="Available Projects")
projects_table.add_column("Name", style="blue")
projects_table.add_column("Path", style="cyan")
projects_table.add_column("Default", style="green")
@@ -323,7 +859,7 @@ def display_project_info(
for name, proj_info in info.available_projects.items():
is_default = name == info.default_project
project_path = proj_info["path"]
projects_table.add_row(name, project_path, "" if is_default else "")
projects_table.add_row(name, project_path, "[X]" if is_default else "")
console.print(projects_table)
+17 -2
View File
@@ -94,7 +94,7 @@ def display_changes(
"""Display changes using Rich for better visualization."""
tree = Tree(f"{project_name}: {title}")
if changes.total == 0:
if changes.total == 0 and not changes.skipped_files:
tree.add("No changes")
console.print(Panel(tree, expand=False))
return
@@ -114,6 +114,13 @@ def display_changes(
if changes.deleted:
del_branch = tree.add("[red]Deleted[/red]")
add_files_to_tree(del_branch, changes.deleted, "red")
if changes.skipped_files:
skip_branch = tree.add("[red]! Skipped (Circuit Breaker)[/red]")
for skipped in sorted(changes.skipped_files, key=lambda x: x.path):
skip_branch.add(
f"[red]{skipped.path}[/red] "
f"(failures: {skipped.failure_count}, reason: {skipped.reason})"
)
else:
# Show directory summaries
by_dir = group_changes_by_directory(changes)
@@ -122,6 +129,14 @@ def display_changes(
if summary: # Only show directories with changes
tree.add(f"[bold]{dir_name}/[/bold] {summary}")
# Show skipped files summary in non-verbose mode
if changes.skipped_files:
skip_count = len(changes.skipped_files)
tree.add(
f"[red]! {skip_count} file{'s' if skip_count != 1 else ''} "
f"skipped due to repeated failures[/red]"
)
console.print(Panel(tree, expand=False))
@@ -137,7 +152,7 @@ async def run_status(project: Optional[str] = None, verbose: bool = False): # p
display_changes(project_item.name, "Status", sync_report, verbose)
except (ValueError, ToolError) as e:
console.print(f"[red]Error: {e}[/red]")
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
-59
View File
@@ -1,59 +0,0 @@
"""Command module for basic-memory sync operations."""
import asyncio
from typing import Annotated, Optional
import typer
from basic_memory.cli.app import app
from basic_memory.cli.commands.command_utils import run_sync
from basic_memory.config import ConfigManager
@app.command()
def sync(
project: Annotated[
Optional[str],
typer.Option(help="The project name."),
] = None,
watch: Annotated[
bool,
typer.Option("--watch", help="Run continuous sync (cloud mode only)"),
] = False,
interval: Annotated[
int,
typer.Option("--interval", help="Sync interval in seconds for watch mode (default: 60)"),
] = 60,
) -> None:
"""Sync knowledge files with the database.
In local mode: Scans filesystem and updates database.
In cloud mode: Runs bidirectional file sync (bisync) then updates database.
Examples:
bm sync # One-time sync
bm sync --watch # Continuous sync every 60s
bm sync --watch --interval 30 # Continuous sync every 30s
"""
config = ConfigManager().config
if config.cloud_mode_enabled:
# Cloud mode: run bisync which includes database sync
from basic_memory.cli.commands.cloud.bisync_commands import run_bisync, run_bisync_watch
try:
if watch:
run_bisync_watch(interval_seconds=interval)
else:
run_bisync()
except Exception:
raise typer.Exit(1)
else:
# Local mode: just database sync
if watch:
typer.echo(
"Error: --watch is only available in cloud mode. Run 'bm cloud login' first."
)
raise typer.Exit(1)
asyncio.run(run_sync(project))
-1
View File
@@ -13,7 +13,6 @@ from basic_memory.cli.commands import ( # noqa: F401 # pragma: no cover
mcp,
project,
status,
sync,
tool,
)
+64 -20
View File
@@ -3,11 +3,13 @@
import json
import os
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from typing import Any, Dict, Literal, Optional, List, Tuple
from enum import Enum
from loguru import logger
from pydantic import Field, field_validator
from pydantic import BaseModel, Field, field_validator
from pydantic_settings import BaseSettings, SettingsConfigDict
import basic_memory
@@ -23,6 +25,13 @@ WATCH_STATUS_JSON = "watch-status.json"
Environment = Literal["test", "dev", "user"]
class DatabaseBackend(str, Enum):
"""Supported database backends."""
SQLITE = "sqlite"
POSTGRES = "postgres"
@dataclass
class ProjectConfig:
"""Configuration for a specific basic-memory project."""
@@ -39,6 +48,22 @@ class ProjectConfig:
return f"/{generate_permalink(self.name)}"
class CloudProjectConfig(BaseModel):
"""Sync configuration for a cloud project.
This tracks the local working directory and sync state for a project
that is synced with Basic Memory Cloud.
"""
local_path: str = Field(description="Local working directory path for this cloud project")
last_sync: Optional[datetime] = Field(
default=None, description="Timestamp of last successful sync operation"
)
bisync_initialized: bool = Field(
default=False, description="Whether rclone bisync baseline has been established"
)
class BasicMemoryConfig(BaseSettings):
"""Pydantic model for Basic Memory global configuration."""
@@ -46,8 +71,10 @@ class BasicMemoryConfig(BaseSettings):
projects: Dict[str, str] = Field(
default_factory=lambda: {
"main": Path(os.getenv("BASIC_MEMORY_HOME", Path.home() / "basic-memory")).as_posix()
},
"main": str(Path(os.getenv("BASIC_MEMORY_HOME", Path.home() / "basic-memory")))
}
if os.getenv("BASIC_MEMORY_HOME")
else {},
description="Mapping of project names to their filesystem paths",
)
default_project: str = Field(
@@ -62,6 +89,17 @@ class BasicMemoryConfig(BaseSettings):
# overridden by ~/.basic-memory/config.json
log_level: str = "INFO"
# Database configuration
database_backend: DatabaseBackend = Field(
default=DatabaseBackend.SQLITE,
description="Database backend to use (sqlite or postgres)",
)
database_url: Optional[str] = Field(
default=None,
description="Database connection URL. For Postgres, use postgresql+asyncpg://user:pass@host:port/db. If not set, SQLite will use default path.",
)
# Watch service configuration
sync_delay: int = Field(
default=1000, description="Milliseconds to wait after changes before syncing", gt=0
@@ -84,7 +122,13 @@ class BasicMemoryConfig(BaseSettings):
sync_thread_pool_size: int = Field(
default=4,
description="Size of thread pool for file I/O operations in sync service",
description="Size of thread pool for file I/O operations in sync service. Default of 4 is optimized for cloud deployments with 1-2GB RAM.",
gt=0,
)
sync_max_concurrent_files: int = Field(
default=10,
description="Maximum number of files to process concurrently during sync. Limits memory usage on large projects (2000+ files). Lower values reduce memory consumption.",
gt=0,
)
@@ -132,6 +176,11 @@ class BasicMemoryConfig(BaseSettings):
description="Enable cloud mode - all requests go to cloud instead of local (config file value)",
)
cloud_projects: Dict[str, CloudProjectConfig] = Field(
default_factory=dict,
description="Cloud project sync configuration mapping project names to their local paths and sync state",
)
@property
def cloud_mode_enabled(self) -> bool:
"""Check if cloud mode is enabled.
@@ -148,14 +197,6 @@ class BasicMemoryConfig(BaseSettings):
# Fall back to config file value
return self.cloud_mode
bisync_config: Dict[str, Any] = Field(
default_factory=lambda: {
"profile": "balanced",
"sync_dir": str(Path.home() / "basic-memory-cloud-sync"),
},
description="Bisync configuration for cloud sync",
)
model_config = SettingsConfigDict(
env_prefix="BASIC_MEMORY_",
extra="ignore",
@@ -172,15 +213,16 @@ class BasicMemoryConfig(BaseSettings):
def model_post_init(self, __context: Any) -> None:
"""Ensure configuration is valid after initialization."""
# Ensure main project exists
if "main" not in self.projects: # pragma: no cover
self.projects["main"] = (
# Ensure at least one project exists; if none exist then create main
if not self.projects: # pragma: no cover
self.projects["main"] = str(
Path(os.getenv("BASIC_MEMORY_HOME", Path.home() / "basic-memory"))
).as_posix()
)
# Ensure default project is valid
# Ensure default project is valid (i.e. points to an existing project)
if self.default_project not in self.projects: # pragma: no cover
self.default_project = "main"
# Set default to first available project
self.default_project = next(iter(self.projects.keys()))
@property
def app_database_path(self) -> Path:
@@ -338,7 +380,7 @@ class ConfigManager:
# Load config, modify it, and save it
config = self.load_config()
config.projects[name] = project_path.as_posix()
config.projects[name] = str(project_path)
self.save_config(config)
return ProjectConfig(name=name, home=project_path)
@@ -421,7 +463,9 @@ def get_project_config(project_name: Optional[str] = None) -> ProjectConfig:
def save_basic_memory_config(file_path: Path, config: BasicMemoryConfig) -> None:
"""Save configuration to file."""
try:
file_path.write_text(json.dumps(config.model_dump(), indent=2))
# Use model_dump with mode='json' to serialize datetime objects properly
config_dict = config.model_dump(mode="json")
file_path.write_text(json.dumps(config_dict, indent=2))
except Exception as e: # pragma: no cover
logger.error(f"Failed to save config: {e}")
+123 -73
View File
@@ -5,7 +5,7 @@ from enum import Enum, auto
from pathlib import Path
from typing import AsyncGenerator, Optional
from basic_memory.config import BasicMemoryConfig, ConfigManager
from basic_memory.config import BasicMemoryConfig, ConfigManager, DatabaseBackend
from alembic import command
from alembic.config import Config
@@ -20,12 +20,12 @@ from sqlalchemy.ext.asyncio import (
)
from sqlalchemy.pool import NullPool
from basic_memory.repository.search_repository import SearchRepository
from basic_memory.repository.postgres_search_repository import PostgresSearchRepository
from basic_memory.repository.sqlite_search_repository import SQLiteSearchRepository
# Module level state
_engine: Optional[AsyncEngine] = None
_session_maker: Optional[async_sessionmaker[AsyncSession]] = None
_migrations_completed: bool = False
class DatabaseType(Enum):
@@ -33,10 +33,41 @@ class DatabaseType(Enum):
MEMORY = auto()
FILESYSTEM = auto()
POSTGRES = auto()
@classmethod
def get_db_url(cls, db_path: Path, db_type: "DatabaseType") -> str:
"""Get SQLAlchemy URL for database path."""
def get_db_url(
cls, db_path: Path, db_type: "DatabaseType", config: Optional[BasicMemoryConfig] = None
) -> str:
"""Get SQLAlchemy URL for database path.
Args:
db_path: Path to SQLite database file (ignored for Postgres)
db_type: Type of database (MEMORY, FILESYSTEM, or POSTGRES)
config: Optional config to check for database backend and URL
Returns:
SQLAlchemy connection URL
"""
# Load config if not provided
if config is None:
config = ConfigManager().config
# Handle explicit Postgres type
if db_type == cls.POSTGRES:
if not config.database_url:
raise ValueError("DATABASE_URL must be set when using Postgres backend")
logger.info(f"Using Postgres database: {config.database_url}")
return config.database_url
# Check if Postgres backend is configured (for backward compatibility)
if config.database_backend == DatabaseBackend.POSTGRES:
if not config.database_url:
raise ValueError("DATABASE_URL must be set when using Postgres backend")
logger.info(f"Using Postgres database: {config.database_url}")
return config.database_url
# SQLite databases
if db_type == cls.MEMORY:
logger.info("Using in-memory SQLite database")
return "sqlite+aiosqlite://"
@@ -64,7 +95,14 @@ async def scoped_session(
factory = get_scoped_session_factory(session_maker)
session = factory()
try:
await session.execute(text("PRAGMA foreign_keys=ON"))
# Only enable foreign keys for SQLite (Postgres has them enabled by default)
# Detect database type from session's bind (engine) dialect
engine = session.get_bind()
dialect_name = engine.dialect.name
if dialect_name == "sqlite":
await session.execute(text("PRAGMA foreign_keys=ON"))
yield session
await session.commit()
except Exception:
@@ -103,13 +141,16 @@ def _configure_sqlite_connection(dbapi_conn, enable_wal: bool = True) -> None:
cursor.close()
def _create_engine_and_session(
db_path: Path, db_type: DatabaseType = DatabaseType.FILESYSTEM
) -> tuple[AsyncEngine, async_sessionmaker[AsyncSession]]:
"""Internal helper to create engine and session maker."""
db_url = DatabaseType.get_db_url(db_path, db_type)
logger.debug(f"Creating engine for db_url: {db_url}")
def _create_sqlite_engine(db_url: str, db_type: DatabaseType) -> AsyncEngine:
"""Create SQLite async engine with appropriate configuration.
Args:
db_url: SQLite connection URL
db_type: Database type (MEMORY or FILESYSTEM)
Returns:
Configured async engine for SQLite
"""
# Configure connection args with Windows-specific settings
connect_args: dict[str, bool | float | None] = {"check_same_thread": False}
@@ -146,6 +187,51 @@ def _create_engine_and_session(
"""Enable WAL mode on each connection."""
_configure_sqlite_connection(dbapi_conn, enable_wal=enable_wal)
return engine
def _create_postgres_engine(db_url: str) -> AsyncEngine:
"""Create Postgres async engine with appropriate configuration.
Args:
db_url: Postgres connection URL (postgresql+asyncpg://...)
Returns:
Configured async engine for Postgres
"""
# Postgres with asyncpg - use standard async connection
engine = create_async_engine(
db_url,
echo=False,
pool_pre_ping=True, # Verify connections before using them
)
return engine
def _create_engine_and_session(
db_path: Path, db_type: DatabaseType = DatabaseType.FILESYSTEM
) -> tuple[AsyncEngine, async_sessionmaker[AsyncSession]]:
"""Internal helper to create engine and session maker.
Args:
db_path: Path to database file (used for SQLite, ignored for Postgres)
db_type: Type of database (MEMORY, FILESYSTEM, or POSTGRES)
Returns:
Tuple of (engine, session_maker)
"""
config = ConfigManager().config
db_url = DatabaseType.get_db_url(db_path, db_type, config)
logger.debug(f"Creating engine for db_url: {db_url}")
# Delegate to backend-specific engine creation
# Check explicit POSTGRES type first, then config setting
if db_type == DatabaseType.POSTGRES or config.database_backend == DatabaseBackend.POSTGRES:
engine = _create_postgres_engine(db_url)
else:
engine = _create_sqlite_engine(db_url, db_type)
session_maker = async_sessionmaker(engine, expire_on_commit=False)
return engine, session_maker
@@ -181,13 +267,12 @@ async def get_or_create_db(
async def shutdown_db() -> None: # pragma: no cover
"""Clean up database connections."""
global _engine, _session_maker, _migrations_completed
global _engine, _session_maker
if _engine:
await _engine.dispose()
_engine = None
_session_maker = None
_migrations_completed = False
@asynccontextmanager
@@ -201,50 +286,12 @@ async def engine_session_factory(
for each test. For production use, use get_or_create_db() instead.
"""
global _engine, _session_maker, _migrations_completed
global _engine, _session_maker
db_url = DatabaseType.get_db_url(db_path, db_type)
logger.debug(f"Creating engine for db_url: {db_url}")
# Configure connection args with Windows-specific settings
connect_args: dict[str, bool | float | None] = {"check_same_thread": False}
# Add Windows-specific parameters to improve reliability
if os.name == "nt": # Windows
connect_args.update(
{
"timeout": 30.0, # Increase timeout to 30 seconds for Windows
"isolation_level": None, # Use autocommit mode
}
)
# Use NullPool for Windows filesystem databases to avoid connection pooling issues
# Important: Do NOT use NullPool for in-memory databases as it will destroy the database
# between connections
if db_type == DatabaseType.FILESYSTEM:
_engine = create_async_engine(
db_url,
connect_args=connect_args,
poolclass=NullPool, # Disable connection pooling on Windows
echo=False,
)
else:
# In-memory databases need connection pooling to maintain state
_engine = create_async_engine(db_url, connect_args=connect_args)
else:
_engine = create_async_engine(db_url, connect_args=connect_args)
# Enable WAL mode for better concurrency and reliability
# Note: WAL mode is not supported for in-memory databases
enable_wal = db_type != DatabaseType.MEMORY
@event.listens_for(_engine.sync_engine, "connect")
def enable_wal_mode(dbapi_conn, connection_record):
"""Enable WAL mode on each connection."""
_configure_sqlite_connection(dbapi_conn, enable_wal=enable_wal)
# Use the same helper function as production code
_engine, _session_maker = _create_engine_and_session(db_path, db_type)
try:
_session_maker = async_sessionmaker(_engine, expire_on_commit=False)
# Verify that engine and session maker are initialized
if _engine is None: # pragma: no cover
logger.error("Database engine is None in engine_session_factory")
@@ -260,20 +307,16 @@ async def engine_session_factory(
await _engine.dispose()
_engine = None
_session_maker = None
_migrations_completed = False
async def run_migrations(
app_config: BasicMemoryConfig, database_type=DatabaseType.FILESYSTEM, force: bool = False
app_config: BasicMemoryConfig, database_type=DatabaseType.FILESYSTEM
): # pragma: no cover
"""Run any pending alembic migrations."""
global _migrations_completed
# Skip if migrations already completed unless forced
if _migrations_completed and not force:
logger.debug("Migrations already completed in this session, skipping")
return
"""Run any pending alembic migrations.
Note: Alembic tracks which migrations have been applied via the alembic_version table,
so it's safe to call this multiple times - it will only run pending migrations.
"""
logger.info("Running database migrations...")
try:
# Get the absolute path to the alembic directory relative to this file
@@ -288,9 +331,11 @@ async def run_migrations(
)
config.set_main_option("timezone", "UTC")
config.set_main_option("revision_environment", "false")
config.set_main_option(
"sqlalchemy.url", DatabaseType.get_db_url(app_config.database_path, database_type)
)
# Get the correct database URL based on backend configuration
# No URL conversion needed - env.py now handles both async and sync engines
db_url = DatabaseType.get_db_url(app_config.database_path, database_type, app_config)
config.set_main_option("sqlalchemy.url", db_url)
command.upgrade(config, "head")
logger.info("Migrations completed successfully")
@@ -301,12 +346,17 @@ async def run_migrations(
else:
session_maker = _session_maker
# initialize the search Index schema
# the project_id is not used for init_search_index, so we pass a dummy value
await SearchRepository(session_maker, 1).init_search_index()
# Mark migrations as completed
_migrations_completed = True
# Initialize the search index schema
# For SQLite: Create FTS5 virtual table
# For Postgres: No-op (tsvector column added by migrations)
# The project_id is not used for init_search_index, so we pass a dummy value
if (
database_type == DatabaseType.POSTGRES
or app_config.database_backend == DatabaseBackend.POSTGRES
):
await PostgresSearchRepository(session_maker, 1).init_search_index()
else:
await SQLiteSearchRepository(session_maker, 1).init_search_index()
except Exception as e: # pragma: no cover
logger.error(f"Error running migrations: {e}")
raise
+289 -3
View File
@@ -25,7 +25,7 @@ from basic_memory.repository.entity_repository import EntityRepository
from basic_memory.repository.observation_repository import ObservationRepository
from basic_memory.repository.project_repository import ProjectRepository
from basic_memory.repository.relation_repository import RelationRepository
from basic_memory.repository.search_repository import SearchRepository
from basic_memory.repository.search_repository import SearchRepository, create_search_repository
from basic_memory.services import EntityService, ProjectService
from basic_memory.services.context_service import ContextService
from basic_memory.services.directory_service import DirectoryService
@@ -76,6 +76,34 @@ async def get_project_config(
ProjectConfigDep = Annotated[ProjectConfig, Depends(get_project_config)] # pragma: no cover
async def get_project_config_v2(
project_id: "ProjectIdPathDep", project_repository: "ProjectRepositoryDep"
) -> ProjectConfig: # pragma: no cover
"""Get the project config for v2 API (uses integer project_id from path).
Args:
project_id: The validated numeric project ID from the URL path
project_repository: Repository for project operations
Returns:
The resolved project config
Raises:
HTTPException: If project is not found
"""
project_obj = await project_repository.get_by_id(project_id)
if project_obj:
return ProjectConfig(name=project_obj.name, home=pathlib.Path(project_obj.path))
# Not found (this should not happen since ProjectIdPathDep already validates existence)
raise HTTPException( # pragma: no cover
status_code=status.HTTP_404_NOT_FOUND, detail=f"Project with ID {project_id} not found."
)
ProjectConfigV2Dep = Annotated[ProjectConfig, Depends(get_project_config_v2)] # pragma: no cover
## sqlalchemy
@@ -130,6 +158,38 @@ ProjectRepositoryDep = Annotated[ProjectRepository, Depends(get_project_reposito
ProjectPathDep = Annotated[str, Path()] # Use Path dependency to extract from URL
async def validate_project_id(
project_id: int,
project_repository: ProjectRepositoryDep,
) -> int:
"""Validate that a numeric project ID exists in the database.
This is used for v2 API endpoints that take project IDs as integers in the path.
The project_id parameter will be automatically extracted from the URL path by FastAPI.
Args:
project_id: The numeric project ID from the URL path
project_repository: Repository for project operations
Returns:
The validated project ID
Raises:
HTTPException: If project with that ID is not found
"""
project_obj = await project_repository.get_by_id(project_id)
if not project_obj:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail=f"Project with ID {project_id} not found.",
)
return project_id
# V2 API: Validated integer project ID from path
ProjectIdPathDep = Annotated[int, Depends(validate_project_id)]
async def get_project_id(
project_repository: ProjectRepositoryDep,
project: ProjectPathDep,
@@ -188,6 +248,17 @@ async def get_entity_repository(
EntityRepositoryDep = Annotated[EntityRepository, Depends(get_entity_repository)]
async def get_entity_repository_v2(
session_maker: SessionMakerDep,
project_id: ProjectIdPathDep,
) -> EntityRepository:
"""Create an EntityRepository instance for v2 API (uses integer project_id from path)."""
return EntityRepository(session_maker, project_id=project_id)
EntityRepositoryV2Dep = Annotated[EntityRepository, Depends(get_entity_repository_v2)]
async def get_observation_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
@@ -199,6 +270,19 @@ async def get_observation_repository(
ObservationRepositoryDep = Annotated[ObservationRepository, Depends(get_observation_repository)]
async def get_observation_repository_v2(
session_maker: SessionMakerDep,
project_id: ProjectIdPathDep,
) -> ObservationRepository:
"""Create an ObservationRepository instance for v2 API."""
return ObservationRepository(session_maker, project_id=project_id)
ObservationRepositoryV2Dep = Annotated[
ObservationRepository, Depends(get_observation_repository_v2)
]
async def get_relation_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
@@ -210,17 +294,43 @@ async def get_relation_repository(
RelationRepositoryDep = Annotated[RelationRepository, Depends(get_relation_repository)]
async def get_relation_repository_v2(
session_maker: SessionMakerDep,
project_id: ProjectIdPathDep,
) -> RelationRepository:
"""Create a RelationRepository instance for v2 API."""
return RelationRepository(session_maker, project_id=project_id)
RelationRepositoryV2Dep = Annotated[RelationRepository, Depends(get_relation_repository_v2)]
async def get_search_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
) -> SearchRepository:
"""Create a SearchRepository instance for the current project."""
return SearchRepository(session_maker, project_id=project_id)
"""Create a backend-specific SearchRepository instance for the current project.
Uses factory function to return SQLiteSearchRepository or PostgresSearchRepository
based on database backend configuration.
"""
return create_search_repository(session_maker, project_id=project_id)
SearchRepositoryDep = Annotated[SearchRepository, Depends(get_search_repository)]
async def get_search_repository_v2(
session_maker: SessionMakerDep,
project_id: ProjectIdPathDep,
) -> SearchRepository:
"""Create a SearchRepository instance for v2 API."""
return create_search_repository(session_maker, project_id=project_id)
SearchRepositoryV2Dep = Annotated[SearchRepository, Depends(get_search_repository_v2)]
# ProjectInfoRepository is deprecated and will be removed in a future version.
# Use ProjectRepository instead, which has the same functionality plus more project-specific operations.
@@ -234,6 +344,13 @@ async def get_entity_parser(project_config: ProjectConfigDep) -> EntityParser:
EntityParserDep = Annotated["EntityParser", Depends(get_entity_parser)]
async def get_entity_parser_v2(project_config: ProjectConfigV2Dep) -> EntityParser:
return EntityParser(project_config.home)
EntityParserV2Dep = Annotated["EntityParser", Depends(get_entity_parser_v2)]
async def get_markdown_processor(entity_parser: EntityParserDep) -> MarkdownProcessor:
return MarkdownProcessor(entity_parser)
@@ -241,6 +358,13 @@ async def get_markdown_processor(entity_parser: EntityParserDep) -> MarkdownProc
MarkdownProcessorDep = Annotated[MarkdownProcessor, Depends(get_markdown_processor)]
async def get_markdown_processor_v2(entity_parser: EntityParserV2Dep) -> MarkdownProcessor:
return MarkdownProcessor(entity_parser)
MarkdownProcessorV2Dep = Annotated[MarkdownProcessor, Depends(get_markdown_processor_v2)]
async def get_file_service(
project_config: ProjectConfigDep, markdown_processor: MarkdownProcessorDep
) -> FileService:
@@ -255,6 +379,20 @@ async def get_file_service(
FileServiceDep = Annotated[FileService, Depends(get_file_service)]
async def get_file_service_v2(
project_config: ProjectConfigV2Dep, markdown_processor: MarkdownProcessorV2Dep
) -> FileService:
logger.debug(
f"Creating FileService for project: {project_config.name}, base_path: {project_config.home}"
)
file_service = FileService(project_config.home, markdown_processor)
logger.debug(f"Created FileService for project: {file_service} ")
return file_service
FileServiceV2Dep = Annotated[FileService, Depends(get_file_service_v2)]
async def get_entity_service(
entity_repository: EntityRepositoryDep,
observation_repository: ObservationRepositoryDep,
@@ -279,6 +417,30 @@ async def get_entity_service(
EntityServiceDep = Annotated[EntityService, Depends(get_entity_service)]
async def get_entity_service_v2(
entity_repository: EntityRepositoryV2Dep,
observation_repository: ObservationRepositoryV2Dep,
relation_repository: RelationRepositoryV2Dep,
entity_parser: EntityParserV2Dep,
file_service: FileServiceV2Dep,
link_resolver: "LinkResolverV2Dep",
app_config: AppConfigDep,
) -> EntityService:
"""Create EntityService for v2 API."""
return EntityService(
entity_repository=entity_repository,
observation_repository=observation_repository,
relation_repository=relation_repository,
entity_parser=entity_parser,
file_service=file_service,
link_resolver=link_resolver,
app_config=app_config,
)
EntityServiceV2Dep = Annotated[EntityService, Depends(get_entity_service_v2)]
async def get_search_service(
search_repository: SearchRepositoryDep,
entity_repository: EntityRepositoryDep,
@@ -291,6 +453,18 @@ async def get_search_service(
SearchServiceDep = Annotated[SearchService, Depends(get_search_service)]
async def get_search_service_v2(
search_repository: SearchRepositoryV2Dep,
entity_repository: EntityRepositoryV2Dep,
file_service: FileServiceV2Dep,
) -> SearchService:
"""Create SearchService for v2 API."""
return SearchService(search_repository, entity_repository, file_service)
SearchServiceV2Dep = Annotated[SearchService, Depends(get_search_service_v2)]
async def get_link_resolver(
entity_repository: EntityRepositoryDep, search_service: SearchServiceDep
) -> LinkResolver:
@@ -300,6 +474,15 @@ async def get_link_resolver(
LinkResolverDep = Annotated[LinkResolver, Depends(get_link_resolver)]
async def get_link_resolver_v2(
entity_repository: EntityRepositoryV2Dep, search_service: SearchServiceV2Dep
) -> LinkResolver:
return LinkResolver(entity_repository=entity_repository, search_service=search_service)
LinkResolverV2Dep = Annotated[LinkResolver, Depends(get_link_resolver_v2)]
async def get_context_service(
search_repository: SearchRepositoryDep,
entity_repository: EntityRepositoryDep,
@@ -315,12 +498,29 @@ async def get_context_service(
ContextServiceDep = Annotated[ContextService, Depends(get_context_service)]
async def get_context_service_v2(
search_repository: SearchRepositoryV2Dep,
entity_repository: EntityRepositoryV2Dep,
observation_repository: ObservationRepositoryV2Dep,
) -> ContextService:
"""Create ContextService for v2 API."""
return ContextService(
search_repository=search_repository,
entity_repository=entity_repository,
observation_repository=observation_repository,
)
ContextServiceV2Dep = Annotated[ContextService, Depends(get_context_service_v2)]
async def get_sync_service(
app_config: AppConfigDep,
entity_service: EntityServiceDep,
entity_parser: EntityParserDep,
entity_repository: EntityRepositoryDep,
relation_repository: RelationRepositoryDep,
project_repository: ProjectRepositoryDep,
search_service: SearchServiceDep,
file_service: FileServiceDep,
) -> SyncService: # pragma: no cover
@@ -334,6 +534,7 @@ async def get_sync_service(
entity_parser=entity_parser,
entity_repository=entity_repository,
relation_repository=relation_repository,
project_repository=project_repository,
search_service=search_service,
file_service=file_service,
)
@@ -342,6 +543,32 @@ async def get_sync_service(
SyncServiceDep = Annotated[SyncService, Depends(get_sync_service)]
async def get_sync_service_v2(
app_config: AppConfigDep,
entity_service: EntityServiceV2Dep,
entity_parser: EntityParserV2Dep,
entity_repository: EntityRepositoryV2Dep,
relation_repository: RelationRepositoryV2Dep,
project_repository: ProjectRepositoryDep,
search_service: SearchServiceV2Dep,
file_service: FileServiceV2Dep,
) -> SyncService: # pragma: no cover
"""Create SyncService for v2 API."""
return SyncService(
app_config=app_config,
entity_service=entity_service,
entity_parser=entity_parser,
entity_repository=entity_repository,
relation_repository=relation_repository,
project_repository=project_repository,
search_service=search_service,
file_service=file_service,
)
SyncServiceV2Dep = Annotated[SyncService, Depends(get_sync_service_v2)]
async def get_project_service(
project_repository: ProjectRepositoryDep,
) -> ProjectService:
@@ -364,6 +591,18 @@ async def get_directory_service(
DirectoryServiceDep = Annotated[DirectoryService, Depends(get_directory_service)]
async def get_directory_service_v2(
entity_repository: EntityRepositoryV2Dep,
) -> DirectoryService:
"""Create DirectoryService for v2 API (uses integer project_id from path)."""
return DirectoryService(
entity_repository=entity_repository,
)
DirectoryServiceV2Dep = Annotated[DirectoryService, Depends(get_directory_service_v2)]
# Import
@@ -407,3 +646,50 @@ async def get_memory_json_importer(
MemoryJsonImporterDep = Annotated[MemoryJsonImporter, Depends(get_memory_json_importer)]
# V2 Import dependencies
async def get_chatgpt_importer_v2(
project_config: ProjectConfigV2Dep, markdown_processor: MarkdownProcessorV2Dep
) -> ChatGPTImporter:
"""Create ChatGPTImporter with v2 dependencies."""
return ChatGPTImporter(project_config.home, markdown_processor)
ChatGPTImporterV2Dep = Annotated[ChatGPTImporter, Depends(get_chatgpt_importer_v2)]
async def get_claude_conversations_importer_v2(
project_config: ProjectConfigV2Dep, markdown_processor: MarkdownProcessorV2Dep
) -> ClaudeConversationsImporter:
"""Create ClaudeConversationsImporter with v2 dependencies."""
return ClaudeConversationsImporter(project_config.home, markdown_processor)
ClaudeConversationsImporterV2Dep = Annotated[
ClaudeConversationsImporter, Depends(get_claude_conversations_importer_v2)
]
async def get_claude_projects_importer_v2(
project_config: ProjectConfigV2Dep, markdown_processor: MarkdownProcessorV2Dep
) -> ClaudeProjectsImporter:
"""Create ClaudeProjectsImporter with v2 dependencies."""
return ClaudeProjectsImporter(project_config.home, markdown_processor)
ClaudeProjectsImporterV2Dep = Annotated[
ClaudeProjectsImporter, Depends(get_claude_projects_importer_v2)
]
async def get_memory_json_importer_v2(
project_config: ProjectConfigV2Dep, markdown_processor: MarkdownProcessorV2Dep
) -> MemoryJsonImporter:
"""Create MemoryJsonImporter with v2 dependencies."""
return MemoryJsonImporter(project_config.home, markdown_processor)
MemoryJsonImporterV2Dep = Annotated[MemoryJsonImporter, Depends(get_memory_json_importer_v2)]
+20 -75
View File
@@ -5,6 +5,7 @@ from pathlib import Path
import re
from typing import Any, Dict, Union
import aiofiles
import yaml
import frontmatter
from loguru import logger
@@ -52,29 +53,12 @@ async def compute_checksum(content: Union[str, bytes]) -> str:
raise FileError(f"Failed to compute checksum: {e}")
async def ensure_directory(path: FilePath) -> None:
"""
Ensure directory exists, creating if necessary.
Args:
path: Directory path to ensure (Path or string)
Raises:
FileWriteError: If directory creation fails
"""
try:
# Convert string to Path if needed
path_obj = Path(path) if isinstance(path, str) else path
path_obj.mkdir(parents=True, exist_ok=True)
except Exception as e: # pragma: no cover
logger.error("Failed to create directory", path=str(path), error=str(e))
raise FileWriteError(f"Failed to create directory {path}: {e}")
async def write_file_atomic(path: FilePath, content: str) -> None:
"""
Write file with atomic operation using temporary file.
Uses aiofiles for true async I/O (non-blocking).
Args:
path: Target file path (Path or string)
content: Content to write
@@ -87,7 +71,11 @@ async def write_file_atomic(path: FilePath, content: str) -> None:
temp_path = path_obj.with_suffix(".tmp")
try:
temp_path.write_text(content, encoding="utf-8")
# Use aiofiles for non-blocking write
async with aiofiles.open(temp_path, mode="w", encoding="utf-8") as f:
await f.write(content)
# Atomic rename (this is fast, doesn't need async)
temp_path.replace(path_obj)
logger.debug("Wrote file atomically", path=str(path_obj), content_length=len(content))
except Exception as e: # pragma: no cover
@@ -185,74 +173,26 @@ def remove_frontmatter(content: str) -> str:
return parts[2].strip()
async def update_frontmatter(path: FilePath, updates: Dict[str, Any]) -> str:
"""Update frontmatter fields in a file while preserving all content.
Only modifies the frontmatter section, leaving all content untouched.
Creates frontmatter section if none exists.
Returns checksum of updated file.
Args:
path: Path to markdown file (Path or string)
updates: Dict of frontmatter fields to update
Returns:
Checksum of updated file
Raises:
FileError: If file operations fail
ParseError: If frontmatter parsing fails
"""
try:
# Convert string to Path if needed
path_obj = Path(path) if isinstance(path, str) else path
# Read current content
content = path_obj.read_text(encoding="utf-8")
# Parse current frontmatter
current_fm = {}
if has_frontmatter(content):
current_fm = parse_frontmatter(content)
content = remove_frontmatter(content)
# Update frontmatter
new_fm = {**current_fm, **updates}
# Write new file with updated frontmatter
yaml_fm = yaml.dump(new_fm, sort_keys=False, allow_unicode=True)
final_content = f"---\n{yaml_fm}---\n\n{content.strip()}"
logger.debug("Updating frontmatter", path=str(path_obj), update_keys=list(updates.keys()))
await write_file_atomic(path_obj, final_content)
return await compute_checksum(final_content)
except Exception as e: # pragma: no cover
logger.error(
"Failed to update frontmatter",
path=str(path) if isinstance(path, (str, Path)) else "<unknown>",
error=str(e),
)
raise FileError(f"Failed to update frontmatter: {e}")
def dump_frontmatter(post: frontmatter.Post) -> str:
"""
Serialize frontmatter.Post to markdown with Obsidian-compatible YAML format.
This function ensures that tags are formatted as YAML lists instead of JSON arrays:
This function ensures that:
1. Tags are formatted as YAML lists instead of JSON arrays
2. String values are properly quoted to handle special characters (colons, etc.)
Good (Obsidian compatible):
---
title: "L2 Governance Core (Split: Core)"
tags:
- system
- overview
- reference
---
Bad (current behavior):
Bad (causes parsing errors):
---
title: L2 Governance Core (Split: Core) # Unquoted colon breaks YAML
tags: ["system", "overview", "reference"]
---
@@ -267,8 +207,13 @@ def dump_frontmatter(post: frontmatter.Post) -> str:
return post.content
# Serialize YAML with block style for lists
# SafeDumper automatically quotes values with special characters (colons, etc.)
yaml_str = yaml.dump(
post.metadata, sort_keys=False, allow_unicode=True, default_flow_style=False
post.metadata,
sort_keys=False,
allow_unicode=True,
default_flow_style=False,
Dumper=yaml.SafeDumper,
)
# Construct the final markdown with frontmatter
+16 -14
View File
@@ -172,15 +172,16 @@ def load_bmignore_patterns() -> Set[str]:
return patterns
def load_gitignore_patterns(base_path: Path) -> Set[str]:
def load_gitignore_patterns(base_path: Path, use_gitignore: bool = True) -> Set[str]:
"""Load gitignore patterns from .gitignore file and .bmignore.
Combines patterns from:
1. ~/.basic-memory/.bmignore (user's global ignore patterns)
2. {base_path}/.gitignore (project-specific patterns)
2. {base_path}/.gitignore (project-specific patterns, if use_gitignore=True)
Args:
base_path: The base directory to search for .gitignore file
use_gitignore: If False, only load patterns from .bmignore (default: True)
Returns:
Set of patterns to ignore
@@ -188,18 +189,19 @@ def load_gitignore_patterns(base_path: Path) -> Set[str]:
# Start with patterns from .bmignore
patterns = load_bmignore_patterns()
gitignore_file = base_path / ".gitignore"
if gitignore_file.exists():
try:
with gitignore_file.open("r", encoding="utf-8") as f:
for line in f:
line = line.strip()
# Skip empty lines and comments
if line and not line.startswith("#"):
patterns.add(line)
except Exception:
# If we can't read .gitignore, just use default patterns
pass
if use_gitignore:
gitignore_file = base_path / ".gitignore"
if gitignore_file.exists():
try:
with gitignore_file.open("r", encoding="utf-8") as f:
for line in f:
line = line.strip()
# Skip empty lines and comments
if line and not line.startswith("#"):
patterns.add(line)
except Exception:
# If we can't read .gitignore, just use default patterns
pass
return patterns
@@ -40,10 +40,13 @@ class ClaudeConversationsImporter(Importer[ChatImportResult]):
chats_imported = 0
for chat in conversations:
# Get name, providing default for unnamed conversations
chat_name = chat.get("name") or f"Conversation {chat.get('uuid', 'untitled')}"
# Convert to entity
entity = self._format_chat_content(
base_path=folder_path,
name=chat["name"],
name=chat_name,
messages=chat["chat_messages"],
created_at=chat["created_at"],
modified_at=chat["updated_at"],
@@ -153,7 +156,12 @@ class ClaudeConversationsImporter(Importer[ChatImportResult]):
# Handle message content
content = msg.get("text", "")
if msg.get("content"):
content = " ".join(c.get("text", "") for c in msg["content"])
# Filter out None values before joining
content = " ".join(
str(c.get("text", ""))
for c in msg["content"]
if c and c.get("text") is not None
)
lines.append(content)
# Handle attachments
+150 -13
View File
@@ -4,12 +4,14 @@ Uses markdown-it with plugins to parse structured data from markdown content.
"""
from dataclasses import dataclass, field
from datetime import datetime
from datetime import date, datetime
from pathlib import Path
from typing import Any, Optional
import dateparser
import frontmatter
import yaml
from loguru import logger
from markdown_it import MarkdownIt
from basic_memory.markdown.plugins import observation_plugin, relation_plugin
@@ -24,6 +26,82 @@ from basic_memory.utils import parse_tags
md = MarkdownIt().use(observation_plugin).use(relation_plugin)
def normalize_frontmatter_value(value: Any) -> Any:
"""Normalize frontmatter values to safe types for processing.
PyYAML automatically converts various string-like values into native Python types:
- Date strings ("2025-10-24") datetime.date objects
- Numbers ("1.0") int or float
- Booleans ("true") bool
- Lists list objects
This can cause AttributeError when code expects strings and calls string methods
like .strip() on these values (see GitHub issue #236).
This function normalizes all frontmatter values to safe types:
- Dates/datetimes ISO format strings
- Numbers (int/float) strings
- Booleans strings ("True"/"False")
- Lists preserved as lists, but items are recursively normalized
- Dicts preserved as dicts, but values are recursively normalized
- Strings kept as-is
- None kept as None
Args:
value: The frontmatter value to normalize
Returns:
The normalized value safe for string operations
Example:
>>> normalize_frontmatter_value(datetime.date(2025, 10, 24))
'2025-10-24'
>>> normalize_frontmatter_value([datetime.date(2025, 10, 24), "tag", 123])
['2025-10-24', 'tag', '123']
>>> normalize_frontmatter_value(True)
'True'
"""
# Convert date/datetime objects to ISO format strings
if isinstance(value, datetime):
return value.isoformat()
if isinstance(value, date):
return value.isoformat()
# Convert boolean to string (must come before int check since bool is subclass of int)
if isinstance(value, bool):
return str(value)
# Convert numbers to strings
if isinstance(value, (int, float)):
return str(value)
# Recursively process lists (preserve as list, normalize items)
if isinstance(value, list):
return [normalize_frontmatter_value(item) for item in value]
# Recursively process dicts (preserve as dict, normalize values)
if isinstance(value, dict):
return {key: normalize_frontmatter_value(val) for key, val in value.items()}
# Keep strings and None as-is
return value
def normalize_frontmatter_metadata(metadata: dict) -> dict:
"""Normalize all values in frontmatter metadata dict.
Converts date/datetime objects to ISO format strings to prevent
AttributeError when code expects strings (GitHub issue #236).
Args:
metadata: The frontmatter metadata dictionary
Returns:
A new dictionary with all values normalized
"""
return {key: normalize_frontmatter_value(value) for key, value in metadata.items()}
@dataclass
class EntityContent:
content: str
@@ -111,25 +189,84 @@ class EntityParser:
return self.base_path / path
async def parse_file_content(self, absolute_path, file_content):
post = frontmatter.loads(file_content)
# Extract file stat info
"""Parse markdown content from file stats.
Delegates to parse_markdown_content() for actual parsing logic.
Exists for backwards compatibility with code that passes file paths.
"""
# Extract file stat info for timestamps
file_stats = absolute_path.stat()
metadata = post.metadata
metadata["title"] = post.metadata.get("title", absolute_path.stem)
metadata["type"] = post.metadata.get("type", "note")
tags = parse_tags(post.metadata.get("tags", [])) # pyright: ignore
# Delegate to parse_markdown_content with timestamps from file stats
return await self.parse_markdown_content(
file_path=absolute_path,
content=file_content,
mtime=file_stats.st_mtime,
ctime=file_stats.st_ctime,
)
async def parse_markdown_content(
self,
file_path: Path,
content: str,
mtime: Optional[float] = None,
ctime: Optional[float] = None,
) -> EntityMarkdown:
"""Parse markdown content without requiring file to exist on disk.
Useful for parsing content from S3 or other remote sources where the file
is not available locally.
Args:
file_path: Path for metadata (doesn't need to exist on disk)
content: Markdown content as string
mtime: Optional modification time (Unix timestamp)
ctime: Optional creation time (Unix timestamp)
Returns:
EntityMarkdown with parsed content
"""
# Parse frontmatter with proper error handling for malformed YAML
try:
post = frontmatter.loads(content)
except yaml.YAMLError as e:
logger.warning(
f"Failed to parse YAML frontmatter in {file_path}: {e}. "
f"Treating file as plain markdown without frontmatter."
)
post = frontmatter.Post(content, metadata={})
# Normalize frontmatter values
metadata = normalize_frontmatter_metadata(post.metadata)
# Ensure required fields have defaults
title = metadata.get("title")
if not title or title == "None":
metadata["title"] = file_path.stem
else:
metadata["title"] = title
entity_type = metadata.get("type")
metadata["type"] = entity_type if entity_type is not None else "note"
tags = parse_tags(metadata.get("tags", [])) # pyright: ignore
if tags:
metadata["tags"] = tags
# frontmatter
entity_frontmatter = EntityFrontmatter(
metadata=post.metadata,
)
# Parse content for observations and relations
entity_frontmatter = EntityFrontmatter(metadata=metadata)
entity_content = parse(post.content)
# Use provided timestamps or current time as fallback
now = datetime.now().astimezone()
created = datetime.fromtimestamp(ctime).astimezone() if ctime else now
modified = datetime.fromtimestamp(mtime).astimezone() if mtime else now
return EntityMarkdown(
frontmatter=entity_frontmatter,
content=post.content,
observations=entity_content.observations,
relations=entity_content.relations,
created=datetime.fromtimestamp(file_stats.st_ctime).astimezone(),
modified=datetime.fromtimestamp(file_stats.st_mtime).astimezone(),
created=created,
modified=modified,
)
-2
View File
@@ -18,7 +18,6 @@ from basic_memory.mcp.tools.canvas import canvas
from basic_memory.mcp.tools.list_directory import list_directory
from basic_memory.mcp.tools.edit_note import edit_note
from basic_memory.mcp.tools.move_note import move_note
from basic_memory.mcp.tools.sync_status import sync_status
from basic_memory.mcp.tools.project_management import (
list_memory_projects,
create_memory_project,
@@ -44,7 +43,6 @@ __all__ = [
"recent_activity",
"search",
"search_notes",
"sync_status",
"view_note",
"write_note",
]
+1 -25
View File
@@ -16,8 +16,6 @@ from basic_memory.schemas.memory import (
memory_url_path,
)
type StringOrInt = str | int
@mcp.tool(
description="""Build context from a memory:// URI to continue conversations naturally.
@@ -39,7 +37,7 @@ type StringOrInt = str | int
async def build_context(
url: MemoryUrl,
project: Optional[str] = None,
depth: Optional[StringOrInt] = 1,
depth: str | int | None = 1,
timeframe: Optional[TimeFrame] = "7d",
page: int = 1,
page_size: int = 10,
@@ -106,28 +104,6 @@ async def build_context(
# Get the active project using the new stateless approach
active_project = await get_active_project(client, project, context)
# Check migration status and wait briefly if needed
from basic_memory.mcp.tools.utils import wait_for_migration_or_return_status
migration_status = await wait_for_migration_or_return_status(
timeout=5.0, project_name=active_project.name
)
if migration_status: # pragma: no cover
# Return a proper GraphContext with status message
from basic_memory.schemas.memory import MemoryMetadata
from datetime import datetime
return GraphContext(
results=[],
metadata=MemoryMetadata(
depth=depth or 1,
timeframe=timeframe,
generated_at=datetime.now().astimezone(),
primary_count=0,
related_count=0,
uri=migration_status, # Include status in metadata
),
)
project_url = active_project.project_url
response = await call_get(
+8 -1
View File
@@ -110,7 +110,14 @@ async def canvas(
# Write the file using the resource API
logger.info(f"Creating canvas file: {file_path} in project {project}")
response = await call_put(client, f"{project_url}/resource/{file_path}", json=canvas_json)
# Send canvas_json as content string, not as json parameter
# The resource endpoint expects Body() string content, not JSON-encoded data
response = await call_put(
client,
f"{project_url}/resource/{file_path}",
content=canvas_json,
headers={"Content-Type": "text/plain"},
)
# Parse response
result = response.json()
-8
View File
@@ -97,14 +97,6 @@ async def read_note(
)
return f"# Error\n\nIdentifier '{identifier}' is not allowed - paths must stay within project boundaries"
# Check migration status and wait briefly if needed
from basic_memory.mcp.tools.utils import wait_for_migration_or_return_status
migration_status = await wait_for_migration_or_return_status(
timeout=5.0, project_name=active_project.name
)
if migration_status: # pragma: no cover
return f"# System Status\n\n{migration_status}\n\nPlease wait for migration to complete before reading notes."
project_url = active_project.project_url
# Get the file via REST API - first try direct permalink lookup
+3 -3
View File
@@ -205,8 +205,8 @@ async def search_notes(
page: int = 1,
page_size: int = 10,
search_type: str = "text",
types: Optional[List[str]] = None,
entity_types: Optional[List[str]] = None,
types: List[str] = [],
entity_types: List[str] = [],
after_date: Optional[str] = None,
context: Context | None = None,
) -> SearchResponse | str:
@@ -345,7 +345,7 @@ async def search_notes(
else:
search_query.text = query # Default to text search
# Add optional filters if provided
# Add optional filters if provided (empty lists are treated as no filter)
if entity_types:
search_query.entity_types = [SearchItemType(t) for t in entity_types]
if types:
-261
View File
@@ -1,261 +0,0 @@
"""Sync status tool for Basic Memory MCP server."""
from typing import Optional
from loguru import logger
from fastmcp import Context
from basic_memory.config import ConfigManager
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.server import mcp
from basic_memory.mcp.project_context import get_active_project
from basic_memory.services.sync_status_service import sync_status_tracker
def _get_all_projects_status() -> list[str]:
"""Get status lines for all configured projects."""
status_lines = []
try:
app_config = ConfigManager().config
if app_config.projects:
status_lines.extend(["", "---", "", "**All Projects Status:**"])
for project_name, project_path in app_config.projects.items():
# Check if this project has sync status
project_sync_status = sync_status_tracker.get_project_status(project_name)
if project_sync_status:
# Project has tracked sync activity
if project_sync_status.status.value == "watching":
# Project is actively watching for changes (steady state)
status_icon = "👁️"
status_text = "Watching for changes"
elif project_sync_status.status.value == "completed":
# Sync completed but not yet watching - transitional state
status_icon = ""
status_text = "Sync completed"
elif project_sync_status.status.value in ["scanning", "syncing"]:
status_icon = "🔄"
status_text = "Sync in progress"
if project_sync_status.files_total > 0:
progress_pct = (
project_sync_status.files_processed
/ project_sync_status.files_total
) * 100
status_text += f" ({project_sync_status.files_processed}/{project_sync_status.files_total}, {progress_pct:.0f}%)"
elif project_sync_status.status.value == "failed":
status_icon = ""
status_text = f"Sync error: {project_sync_status.error or 'Unknown error'}"
else:
status_icon = "⏸️"
status_text = project_sync_status.status.value.title()
else:
# Project has no tracked sync activity - will be synced automatically
status_icon = ""
status_text = "Pending sync"
status_lines.append(f"- {status_icon} **{project_name}**: {status_text}")
except Exception as e:
logger.debug(f"Could not get project config for comprehensive status: {e}")
return status_lines
@mcp.tool(
description="""Check the status of file synchronization and background operations.
Use this tool to:
- Check if file sync is in progress or completed
- Get detailed sync progress information
- Understand if your files are fully indexed
- Get specific error details if sync operations failed
- Monitor initial project setup and legacy migration
This covers all sync operations including:
- Initial project setup and file indexing
- Legacy project migration to unified database
- Ongoing file monitoring and updates
- Background processing of knowledge graphs
""",
)
async def sync_status(project: Optional[str] = None, context: Context | None = None) -> str:
"""Get current sync status and system readiness information.
This tool provides detailed information about any ongoing or completed
sync operations, helping users understand when their files are ready.
Args:
project: Optional project name to get project-specific context
Returns:
Formatted sync status with progress, readiness, and guidance
"""
logger.info("MCP tool call tool=sync_status")
async with get_client() as client:
status_lines = []
try:
from basic_memory.services.sync_status_service import sync_status_tracker
# Get overall summary
summary = sync_status_tracker.get_summary()
is_ready = sync_status_tracker.is_ready
# Header
status_lines.extend(
[
"# Basic Memory Sync Status",
"",
f"**Current Status**: {summary}",
f"**System Ready**: {'✅ Yes' if is_ready else '🔄 Processing'}",
"",
]
)
if is_ready:
status_lines.extend(
[
"✅ **All sync operations completed**",
"",
"- File indexing is complete",
"- Knowledge graphs are up to date",
"- All Basic Memory tools are fully operational",
"",
"Your knowledge base is ready for use!",
]
)
# Show all projects status even when ready
status_lines.extend(_get_all_projects_status())
else:
# System is still processing - show both active and all projects
all_sync_projects = sync_status_tracker.get_all_projects()
active_projects = [
p
for p in all_sync_projects.values()
if p.status.value in ["scanning", "syncing"]
]
failed_projects = [
p for p in all_sync_projects.values() if p.status.value == "failed"
]
if active_projects:
status_lines.extend(
[
"🔄 **File synchronization in progress**",
"",
"Basic Memory is automatically processing all configured projects and building knowledge graphs.",
"This typically takes 1-3 minutes depending on the amount of content.",
"",
"**Currently Processing:**",
]
)
for project_status in active_projects:
progress = ""
if project_status.files_total > 0:
progress_pct = (
project_status.files_processed / project_status.files_total
) * 100
progress = f" ({project_status.files_processed}/{project_status.files_total}, {progress_pct:.0f}%)"
status_lines.append(
f"- **{project_status.project_name}**: {project_status.message}{progress}"
)
status_lines.extend(
[
"",
"**What's happening:**",
"- Scanning and indexing markdown files",
"- Building entity and relationship graphs",
"- Settings up full-text search indexes",
"- Processing file changes and updates",
"",
"**What you can do:**",
"- Wait for automatic processing to complete - no action needed",
"- Use this tool again to check progress",
"- Simple operations may work already",
"- All projects will be available once sync finishes",
]
)
# Handle failed projects (independent of active projects)
if failed_projects:
status_lines.extend(["", "❌ **Some projects failed to sync:**", ""])
for project_status in failed_projects:
status_lines.append(
f"- **{project_status.project_name}**: {project_status.error or 'Unknown error'}"
)
status_lines.extend(
[
"",
"**Next steps:**",
"1. Check the logs for detailed error information",
"2. Ensure file permissions allow read/write access",
"3. Try restarting the MCP server",
"4. If issues persist, consider filing a support issue",
]
)
elif not active_projects:
# No active or failed projects - must be pending
status_lines.extend(
[
"⏳ **Sync operations pending**",
"",
"File synchronization has been queued but hasn't started yet.",
"This usually resolves automatically within a few seconds.",
]
)
# Add comprehensive project status for all configured projects
all_projects_status = _get_all_projects_status()
if all_projects_status:
status_lines.extend(all_projects_status)
# Add explanation about automatic syncing if there are unsynced projects
unsynced_count = sum(1 for line in all_projects_status if "" in line)
if unsynced_count > 0 and not is_ready:
status_lines.extend(
[
"",
"**Note**: All configured projects will be automatically synced during startup.",
]
)
# Add project context if provided
if project:
try:
active_project = await get_active_project(client, project, context)
status_lines.extend(
[
"",
"---",
"",
f"**Active Project**: {active_project.name}",
f"**Project Path**: {active_project.home}",
]
)
except Exception as e:
logger.debug(f"Could not get project info: {e}")
return "\n".join(status_lines)
except Exception as e:
return f"""# Sync Status - Error
**Unable to check sync status**: {str(e)}
**Troubleshooting:**
- The system may still be starting up
- Try waiting a few seconds and checking again
- Check logs for detailed error information
- Consider restarting if the issue persists
"""
-70
View File
@@ -510,73 +510,3 @@ async def call_delete(
except HTTPStatusError as e:
raise ToolError(error_message) from e
def check_migration_status() -> Optional[str]:
"""Check if sync/migration is in progress and return status message if so.
Returns:
Status message if sync is in progress, None if system is ready
"""
try:
from basic_memory.services.sync_status_service import sync_status_tracker
if not sync_status_tracker.is_ready:
return sync_status_tracker.get_summary()
return None
except Exception:
# If there's any error checking sync status, assume ready
return None
async def wait_for_migration_or_return_status(
timeout: float = 5.0, project_name: Optional[str] = None
) -> Optional[str]:
"""Wait briefly for sync/migration to complete, or return status message.
Args:
timeout: Maximum time to wait for sync completion
project_name: Optional project name to check specific project status.
If provided, only checks that project's readiness.
If None, uses global status check (legacy behavior).
Returns:
Status message if sync is still in progress, None if ready
"""
try:
from basic_memory.services.sync_status_service import sync_status_tracker
import asyncio
# Check if we should use project-specific or global status
def is_ready() -> bool:
if project_name:
return sync_status_tracker.is_project_ready(project_name)
return sync_status_tracker.is_ready
if is_ready():
return None
# Wait briefly for sync to complete
start_time = asyncio.get_event_loop().time()
while (asyncio.get_event_loop().time() - start_time) < timeout:
if is_ready():
return None
await asyncio.sleep(0.1) # Check every 100ms
# Still not ready after timeout
if project_name:
# For project-specific checks, get project status details
project_status = sync_status_tracker.get_project_status(project_name)
if project_status and project_status.status.value == "failed":
error_msg = project_status.error or "Unknown sync error"
return f"❌ Sync failed for project '{project_name}': {error_msg}"
elif project_status:
return f"🔄 Project '{project_name}' is still syncing: {project_status.message}"
else:
return f"⚠️ Project '{project_name}' status unknown"
else:
# Fall back to global summary for legacy calls
return sync_status_tracker.get_summary()
except Exception: # pragma: no cover
# If there's any error, assume ready
return None

Some files were not shown because too many files have changed in this diff Show More