Compare commits

...

63 Commits

Author SHA1 Message Date
phernandez 3050fe8318 ci: replace flaky just.systems curl install with extractions/setup-just action
The just.systems install script intermittently returns HTTP 403, causing
CI jobs to fail before tests even run. The extractions/setup-just GitHub
Action downloads from GitHub releases instead, which is reliable.

Also removes the separate Windows choco install step since the action
handles all platforms natively.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-13 15:39:11 -06:00
phernandez b367ccbdb8 fix: update tests and fix bugs for project-prefixed permalinks
- Fix update_entity overwriting project-prefixed permalink with
  non-prefixed one during metadata merge
- Fix memory:// URL path traversal validation bypass in read_note
  and read_content tools
- Fix pyright type error in claude_projects_importer
- Update test assertions across 12 test files to use project-prefixed
  permalinks (test-project/path instead of path)
- Fix monkeypatch in search test to use resolve_project_and_path
  instead of removed get_active_project

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-13 15:39:11 -06:00
phernandez c6511aa745 feat: enable project-prefixed permalinks
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-13 15:39:10 -06:00
Paul Hernandez 8bc03d1357 feat(mcp): add MCP UI variants and TUI output (#545)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-13 15:21:48 -06:00
Paul Hernandez f6e0a5b5bb fix: Speed up bm --version startup (#534)
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-13 11:12:52 -06:00
Paul Hernandez 7624a20d8d feat: isolate default sqlite db by config dir (#567)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-13 11:11:13 -06:00
Paul Hernandez d84708ca7f feat: add per-project local/cloud routing with API key auth (#555)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-13 09:52:50 -06:00
Paul Hernandez 1428d18de1 fix: make semantic search dependencies optional extras (#566)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 21:46:26 -06:00
Paul Hernandez 312662f382 feat: Add cloud discovery touchpoints to CLI and MCP (#546)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-12 21:01:45 -06:00
Paul Hernandez ed9487708e feat: enable default_project_mode by default (#560)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 20:34:46 -06:00
Paul Hernandez 8df88e4d02 feat: add basic-memory watch CLI command (#559)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 20:10:09 -06:00
Paul Hernandez 07778790d3 feat: add semantic vector search for SQLite and Postgres (#550)
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: bm-clawd <clawd@basicmemory.com>
Co-authored-by: bm-clawd <clawd@basicmemory.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 15:44:47 -06:00
Paul Hernandez b609c4e531 fix: use global --header for Tigris consistency on all rclone transactions (#564)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 15:43:05 -06:00
Paul Hernandez f1a065bce3 chore: Release/v0.18.2 (#563)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 15:41:38 -06:00
phernandez 2b94d9a278 fix formatting for tigris headers
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-12 13:22:31 -06:00
Paul Hernandez 344e651693 fix: use VIRTUAL instead of STORED columns in SQLite migration (#562)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-11 22:24:51 -06:00
Paul Hernandez c97733d785 feat: Schema system for Basic Memory (#549)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-11 15:04:42 -06:00
phernandez 00537272c6 chore: update version to 0.18.1 for v0.18.1 release 2026-02-11 14:28:52 -06:00
phernandez b057912452 docs: add CHANGELOG entry for v0.18.1
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-11 14:24:26 -06:00
Paul Hernandez 8489a3d37e fix: add X-Tigris-Consistent headers to all rclone commands (#558)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-11 14:22:49 -06:00
Paul Hernandez a47c9c021f feat: add --format json to CLI tool commands (#552)
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: bm-clawd <clawd@basicmemory.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: bm-clawd <clawd@basicmemory.com>
2026-02-08 14:56:29 -06:00
phernandez c46d7a6833 fix: add POST legacy compat routes for v0.18.0 CLI
The v0.18.0 CLI (Homebrew) calls POST /projects/projects for project add
and POST /projects/config/sync for config sync. The previous legacy compat
fix (a0e754b) only added GET for list_projects but missed POST endpoints.

This caused 405 Method Not Allowed when running `bm project add` in cloud mode.

🔗 Logfire trace: trace_id=019c398cb257f040c1255821b6d5e385

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-07 13:41:41 -06:00
Paul Hernandez 343a6e118b fix: Handle EntityCreationError as conflict (#541)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-03 22:28:58 -06:00
phernandez a0e754b7ae fix: restore legacy /projects/projects endpoint for older CLI versions
Older versions of basic-memory CLI (v0.17.4 and earlier) call
GET /projects/projects to list projects. This endpoint was removed
when we migrated to v2 routers.

Add explicit route at /projects/projects (without trailing slash) to
avoid 307 redirects that the cloud proxy doesn't follow.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-02 22:26:48 -06:00
Paul Hernandez 24ca5f6804 fix: recent_activity prompt defaults (#533)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-02 19:22:33 -06:00
Paul Hernandez f1d50c2ba7 feat: Support tag: query shorthand in search (#535)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-02 19:22:16 -06:00
Paul Hernandez 8072449a78 chore: Add fast feedback loop tooling (#538)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-01 23:04:59 -06:00
phernandez 45d3f58e4d Merge branch 'main' of github.com:basicmachines-co/basic-memory 2026-02-01 21:17:39 -06:00
phernandez d9c8923148 fix ci runner for tests
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-01 21:16:10 -06:00
phernandez 15bd6b95ef fix ci runner i
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-01 21:10:26 -06:00
phernandez 0715dcff3d run ubuntu tests on depot
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-31 23:25:46 -06:00
Paul Hernandez 009e84926d fix: stabilize metadata filters on postgres (#536)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-31 22:31:00 -06:00
phernandez 8838571509 Add metadata filter tests and fix fast write external_id
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-31 15:27:05 -06:00
Paul Hernandez 530cbac73f feat: fast edit entities, refactors for webui, enhance search (#532)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-31 15:16:52 -06:00
Paul Hernandez e3ced49d9d fix: prevent spurious 'metadata: {}' in frontmatter output (#530)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-31 12:17:10 -06:00
Drew Cain 8f962fdd87 chore: update version to 0.18.0 for v0.18.0 release 2026-01-29 22:41:42 -06:00
Drew Cain fbb497f6dc docs: add CHANGELOG entry for v0.18.0 2026-01-29 22:41:14 -06:00
Drew Cain 0023e736ab feat: add context-aware wiki link resolution with source_path support (#527)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-28 19:11:41 -06:00
Paul Hernandez 0b2080114b feat: add directory support to move_note and delete_note tools (#518)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-26 19:52:10 -06:00
Paul Hernandez 8730067f3a feat: Feature/517 local mcp cloud mode (#522)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-26 19:51:49 -06:00
Drew Cain e14ba92631 fix: resolve MCP prompt rendering errors (#524)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 19:14:41 -06:00
phernandez 9d98892570 chore: update version to 0.17.9 for v0.17.9 release 2026-01-24 12:55:58 -06:00
phernandez 3be4495723 docs: add CHANGELOG entry for v0.17.9
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-24 12:55:27 -06:00
Paul Hernandez 17c0e0a29b fix: check config default_project only in local mode for remove_project (#523)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 12:53:44 -06:00
phernandez 7ebf16a95d chore: update version to 0.17.8 for v0.17.8 release 2026-01-24 11:43:44 -06:00
phernandez c05075f8d4 docs: add CHANGELOG entry for v0.17.8
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-24 11:43:21 -06:00
phernandez 4cef9281ca docs: add CHANGELOG entry for v0.17.7
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-24 11:39:41 -06:00
Paul Hernandez 6888effef2 fix: correct get_default_project() query to check for True instead of not NULL (#521)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 11:37:05 -06:00
Paul Hernandez 38616c345d fix: read default project from database in cloud mode (#520)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 10:44:04 -06:00
phernandez f3c1aa895c fix links in README.md to remove smithery badge
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-22 12:58:03 -06:00
phernandez d978aba09b fix links in README.md to remove glama.ai
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-22 12:56:33 -06:00
phernandez 2aaee734c9 fix links in README.md to point to basicmemory.com instead of basicmachines.co
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-22 12:53:18 -06:00
Paul Hernandez 369ad37b3d feat: add SPEC-29 Phase 3 bucket snapshot CLI commands (#476)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-20 17:55:01 -06:00
Drew Cain 4e5f701d22 Fix server.json runtimeArguments format
- Use proper Argument object format instead of plain strings
- Add .mcpregistry tokens to .gitignore

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-01-19 12:23:29 -06:00
Drew Cain 9d9ea4d61c chore: update version to 0.17.7 for v0.17.7 release 2026-01-19 12:12:35 -06:00
Drew Cain 7a502e6474 feat: Add MCP registry publication files (#515)
Signed-off-by: Drew Cain <groksrc@gmail.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-19 11:57:50 -06:00
Paul Hernandez c7835a9d5c fix: ensure external_id is set on entity creation (#512) (#513)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-19 11:14:48 -06:00
Paul Hernandez 85835ae533 chore: Remove OpenPanel telemetry (#514)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-18 22:24:27 -06:00
phernandez 671e3d4db9 chore: update version to 0.17.6 for v0.17.6 release 2026-01-17 15:11:34 -06:00
phernandez e11aeff8d9 docs: add changelog entry for v0.17.6
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-17 15:11:17 -06:00
phernandez 803f3efe53 turn sync logging to debug
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-17 15:10:04 -06:00
phernandez d6dab8552c Merge branch 'main' of github.com:basicmachines-co/basic-memory 2026-01-17 15:09:04 -06:00
phernandez d1d433df15 remove logfire config, and specs docs, turn lifespan logging to debug
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-17 15:08:54 -06:00
441 changed files with 34555 additions and 21874 deletions
+31 -1
View File
@@ -78,6 +78,34 @@ The GitHub Actions workflow (`.github/workflows/release.yml`) then:
2. Verify formula version matches release
3. Test Homebrew installation: `brew install basicmachines-co/basic-memory/basic-memory`
#### MCP Registry Publication
After PyPI release is published, update the MCP registry:
1. **Verify PyPI Release**
- Confirm package is live: https://pypi.org/project/basic-memory/<version>/
- The `server.json` version was auto-updated by `just release`
2. **Publish to MCP Registry**
```bash
cd /Users/drew/code/basic-memory
mcp-publisher publish
```
If not authenticated:
```bash
mcp-publisher login github
# Follow device authentication flow
mcp-publisher publish
```
3. **Verify Publication**
```bash
curl "https://registry.modelcontextprotocol.io/v0.1/servers?search=basic-memory"
```
**Note:** The `mcp-publisher` CLI can be installed via Homebrew (`brew install mcp-publisher`) or from GitHub releases.
#### Website Updates
**1. basicmachines.co** (`/Users/drew/code/basicmachines.co`)
@@ -145,6 +173,7 @@ Before starting, verify:
📋 GitHub Release: https://github.com/basicmachines-co/basic-memory/releases/tag/v0.13.2
📦 PyPI: https://pypi.org/project/basic-memory/0.13.2/
🍺 Homebrew: https://github.com/basicmachines-co/homebrew-basic-memory
🔌 MCP Registry: https://registry.modelcontextprotocol.io
🚀 GitHub Actions: Completed
Install with pip/uv:
@@ -162,8 +191,9 @@ Users can now upgrade:
- This creates production releases used by end users
- Must pass all quality gates before proceeding
- Uses the automated justfile target for consistency
- Version is automatically updated in `__init__.py`
- Version is automatically updated in `__init__.py` and `server.json`
- Triggers automated GitHub release with changelog
- Package is published to PyPI for `pip` and `uv` users
- Homebrew formula is automatically updated for stable releases
- MCP Registry is updated manually via `mcp-publisher publish`
- Supports multiple installation methods (uv, pip, Homebrew)
+6 -20
View File
@@ -37,17 +37,7 @@ jobs:
run: |
pip install uv
- name: Install just (Linux/macOS)
if: runner.os != 'Windows'
run: |
curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to /usr/local/bin
- name: Install just (Windows)
if: runner.os == 'Windows'
run: |
# Install just using Chocolatey (pre-installed on GitHub Actions Windows runners)
choco install just --yes
shell: pwsh
- uses: extractions/setup-just@v3
- name: Create virtual env
run: |
@@ -55,7 +45,7 @@ jobs:
- name: Install dependencies
run: |
uv pip install -e .[dev]
uv pip install -e ".[dev,semantic]"
- name: Run type checks
run: |
@@ -95,9 +85,7 @@ jobs:
run: |
pip install uv
- name: Install just
run: |
curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to /usr/local/bin
- uses: extractions/setup-just@v3
- name: Create virtual env
run: |
@@ -105,7 +93,7 @@ jobs:
- name: Install dependencies
run: |
uv pip install -e .[dev]
uv pip install -e ".[dev,semantic]"
- name: Run tests (Postgres via testcontainers)
run: |
@@ -131,9 +119,7 @@ jobs:
run: |
pip install uv
- name: Install just
run: |
curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to /usr/local/bin
- uses: extractions/setup-just@v3
- name: Create virtual env
run: |
@@ -141,7 +127,7 @@ jobs:
- name: Install dependencies
run: |
uv pip install -e .[dev]
uv pip install -e ".[dev,semantic]"
- name: Run combined coverage (SQLite + Postgres)
run: |
+4
View File
@@ -1,6 +1,7 @@
*.py[cod]
__pycache__/
.pytest_cache/
.testmondata*
.coverage
htmlcov/
@@ -54,3 +55,6 @@ ENV/
claude-output
**/.claude/settings.local.json
.mcp.json
.mcpregistry_*
/.testmondata
.benchmarks/
+445
View File
@@ -0,0 +1,445 @@
# AGENTS.md - Basic Memory Project Guide
## Project Overview
Basic Memory is a local-first knowledge management system built on the Model Context Protocol (MCP). It enables
bidirectional communication between LLMs (like Claude) and markdown files, creating a personal knowledge graph that can
be traversed using links between documents.
## CODEBASE DEVELOPMENT
### Project information
See the [README.md](README.md) file for a project overview.
### Build and Test Commands
- Install: `just install` or `pip install -e ".[dev]"`
- Run all tests (SQLite + Postgres): `just test`
- Run all tests against SQLite: `just test-sqlite`
- Run all tests against Postgres: `just test-postgres` (uses testcontainers)
- Run unit tests (SQLite): `just test-unit-sqlite`
- Run unit tests (Postgres): `just test-unit-postgres`
- Run integration tests (SQLite): `just test-int-sqlite`
- Run integration tests (Postgres): `just test-int-postgres`
- Run impacted tests: `just testmon` (pytest-testmon)
- Run MCP smoke test: `just test-smoke`
- Fast local loop: `just fast-check`
- Local consistency check: `just doctor`
- Generate HTML coverage: `just coverage`
- Single test: `pytest tests/path/to/test_file.py::test_function_name`
- Run benchmarks: `pytest test-int/test_sync_performance_benchmark.py -v -m "benchmark and not slow"`
- Lint: `just lint` or `ruff check . --fix`
- Type check: `just typecheck` or `uv run pyright`
- Format: `just format` or `uv run ruff format .`
- Run all code checks: `just check` (runs lint, format, typecheck, test)
- Create db migration: `just migration "Your migration message"`
- Run development MCP Inspector: `just run-inspector`
**Note:** Project requires Python 3.12+ (uses type parameter syntax and `type` aliases introduced in 3.12)
**Postgres Testing:** Uses [testcontainers](https://testcontainers-python.readthedocs.io/) which automatically spins up a Postgres instance in Docker. No manual database setup required - just have Docker running.
**Doctor Note:** `just doctor` runs with a temporary HOME/config so it won't touch your local Basic Memory settings. It leaves temp dirs in `/tmp` (safe to ignore or remove).
**Testmon Note:** When no files have changed, `just testmon` may collect 0 tests. That's expected and means no impacted tests were detected.
### Code/Test/Verify Loop (fast path)
1) **Code:** make changes.
2) **Test:** `just fast-check` (lint/format/typecheck + impacted tests + MCP smoke).
3) **Verify:** `just doctor` (end-to-end file ↔ DB loop in a temp project).
4) **Full gate (when needed):** `just test` or `just check` for SQLite + Postgres.
If testmon is “cold,” the first run may be long. Subsequent runs get much faster.
### Test Structure
- `tests/` - Unit tests for individual components (mocked, fast)
- `test-int/` - Integration tests for real-world scenarios (no mocks, realistic)
- Both directories are covered by unified coverage reporting
- Benchmark tests in `test-int/` are marked with `@pytest.mark.benchmark`
- Slow tests are marked with `@pytest.mark.slow`
- Smoke tests are marked with `@pytest.mark.smoke`
### Code Style Guidelines
- Line length: 100 characters max
- Python 3.12+ with full type annotations (uses type parameters and type aliases)
- Format with ruff (consistent styling)
- Import order: standard lib, third-party, local imports
- Naming: snake_case for functions/variables, PascalCase for classes
- Prefer async patterns with SQLAlchemy 2.0
- Use Pydantic v2 for data validation and schemas
- CLI uses Typer for command structure
- API uses FastAPI for endpoints
- Follow the repository pattern for data access
- Tools communicate to api routers via the httpx ASGI client (in process)
### Code Change Guidelines
- **Full file read before edits**: Before editing any file, read it in full first to ensure complete context; partial reads lead to corrupted edits
- **Minimize diffs**: Prefer the smallest change that satisfies the request. Avoid unrelated refactors or style rewrites unless necessary for correctness
- **No speculative getattr**: Never use `getattr(obj, "attr", default)` when unsure about attribute names. Check the class definition or source code first
- **Fail fast**: Write code with fail-fast logic by default. Do not swallow exceptions with errors or warnings
- **No fallback logic**: Do not add fallback logic unless explicitly told to and agreed with the user
- **No guessing**: Do not say "The issue is..." before you actually know what the issue is. Investigate first.
### Literate Programming Style
Code should tell a story. Comments must explain the "why" and narrative flow, not just the "what".
**Section Headers:**
For files with multiple phases of logic, add section headers so the control flow reads like chapters:
```python
# --- Authentication ---
# ... auth logic ...
# --- Data Validation ---
# ... validation logic ...
# --- Business Logic ---
# ... core logic ...
```
**Decision Point Comments:**
For conditionals that materially change behavior (gates, fallbacks, retries, feature flags), add comments with:
- **Trigger**: what condition causes this branch
- **Why**: the rationale (cost, correctness, UX, determinism)
- **Outcome**: what changes downstream
```python
# Trigger: project has no active sync watcher
# Why: avoid duplicate file system watchers consuming resources
# Outcome: starts new watcher, registers in active_watchers dict
if project_id not in active_watchers:
start_watcher(project_id)
```
**Constraint Comments:**
If code exists because of a constraint (async requirements, rate limits, schema compatibility), explain the constraint near the code:
```python
# SQLite requires WAL mode for concurrent read/write access
connection.execute("PRAGMA journal_mode=WAL")
```
**What NOT to Comment:**
Avoid comments that restate obvious code:
```python
# Bad - restates code
counter += 1 # increment counter
# Good - explains why
counter += 1 # track retries for backoff calculation
```
### Codebase Architecture
See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for detailed architecture documentation.
**Directory Structure:**
- `/alembic` - Alembic db migrations
- `/api` - FastAPI REST endpoints + `container.py` composition root
- `/cli` - Typer CLI + `container.py` composition root
- `/deps` - Feature-scoped FastAPI dependencies (config, db, projects, repositories, services, importers)
- `/importers` - Import functionality for Claude, ChatGPT, and other sources
- `/markdown` - Markdown parsing and processing
- `/mcp` - MCP server + `container.py` composition root + `clients/` typed API clients
- `/models` - SQLAlchemy ORM models
- `/repository` - Data access layer
- `/schemas` - Pydantic models for validation
- `/services` - Business logic layer
- `/sync` - File synchronization services + `coordinator.py` for lifecycle management
**Composition Roots:**
Each entrypoint (API, MCP, CLI) has a composition root that:
- Reads `ConfigManager` (the only place that reads global config)
- Resolves runtime mode via `RuntimeMode` enum (TEST > CLOUD > LOCAL)
- Provides dependencies to downstream code explicitly
**Typed API Clients (MCP):**
MCP tools use typed clients in `mcp/clients/` to communicate with the API:
- `KnowledgeClient` - Entity CRUD operations
- `SearchClient` - Search operations
- `MemoryClient` - Context building
- `DirectoryClient` - Directory listing
- `ResourceClient` - Resource reading
- `ProjectClient` - Project management
Flow: MCP Tool → Typed Client → HTTP API → Router → Service → Repository
### Development Notes
- MCP tools are defined in src/basic_memory/mcp/tools/
- MCP prompts are defined in src/basic_memory/mcp/prompts/
- MCP tools should be atomic, composable operations
- Use `textwrap.dedent()` for multi-line string formatting in prompts and tools
- MCP Prompts are used to invoke tools and format content with instructions for an LLM
- Schema changes require Alembic migrations
- SQLite is used for indexing and full text search, files are source of truth
- Testing uses pytest with asyncio support (strict mode)
- Unit tests (`tests/`) use mocks when necessary; integration tests (`test-int/`) use real implementations
- By default, tests run against SQLite (fast, no Docker needed)
- Set `BASIC_MEMORY_TEST_POSTGRES=1` to run against Postgres (uses testcontainers - Docker required)
- Each test runs in a standalone environment with isolated database and tmp_path directory
- CI runs SQLite and Postgres tests in parallel for faster feedback
- Performance benchmarks are in `test-int/test_sync_performance_benchmark.py`
- Use pytest markers: `@pytest.mark.benchmark` for benchmarks, `@pytest.mark.slow` for slow tests
- **Coverage must stay at 100%**: Write tests for new code. Only use `# pragma: no cover` when tests would require excessive mocking (e.g., TYPE_CHECKING blocks, error handlers that need failure injection, runtime-mode-dependent code paths)
### Async Client Pattern (Important!)
**MCP tools use `get_project_client()` for per-project routing:**
```python
from basic_memory.mcp.project_context import get_project_client
@mcp.tool()
async def my_tool(project: str | None = None, context: Context | None = None):
async with get_project_client(project, context) as (client, active_project):
# client is routed based on project's mode (local ASGI or cloud HTTP)
response = await call_get(client, "/path")
return response
```
**CLI commands and non-project-scoped code use `get_client()` directly:**
```python
from basic_memory.mcp.async_client import get_client
async def my_cli_command():
async with get_client() as client:
response = await call_get(client, "/path")
return response
# Per-project routing (when project name is known):
async with get_client(project_name="research") as client:
...
```
**Do NOT use:**
-`from basic_memory.mcp.async_client import client` (deprecated module-level client)
- ❌ Manual auth header management
-`inject_auth_header()` (deleted)
- ❌ Separate `get_client()` + `get_active_project()` in MCP tools (use `get_project_client()` instead)
**Key principles:**
- Auth happens at client creation, not per-request
- Proper resource management via context managers
- Per-project routing: each project can be LOCAL or CLOUD independently
- Cloud projects use API key (`cloud_api_key` in config) as Bearer token
- Routing priority: factory injection > force-local > per-project cloud > global cloud > local ASGI
- Factory pattern enables dependency injection for cloud consolidation
**For cloud app integration:**
```python
from basic_memory.mcp import async_client
# Set custom factory before importing tools
async_client.set_client_factory(your_custom_factory)
```
See SPEC-16 for full context manager refactor details.
## BASIC MEMORY PRODUCT USAGE
### Knowledge Structure
- Entity: Any concept, document, or idea represented as a markdown file
- Observation: A categorized fact about an entity (`- [category] content`)
- Relation: A directional link between entities (`- relation_type [[Target]]`)
- Frontmatter: YAML metadata at the top of markdown files
- Knowledge representation follows precise markdown format:
- Observations with [category] prefixes
- Relations with WikiLinks [[Entity]]
- Frontmatter with metadata
### Basic Memory Commands
**Local Commands:**
- Check sync status: `basic-memory status`
- Doctor check (file <-> DB loop): `basic-memory doctor`
- Import from Claude: `basic-memory import claude conversations`
- Import from ChatGPT: `basic-memory import chatgpt`
- Import from Memory JSON: `basic-memory import memory-json`
- Tool access: `basic-memory tool` (provides CLI access to MCP tools)
- Continue: `basic-memory tool continue-conversation --topic="search"`
**Project Management:**
- List projects: `basic-memory project list`
- Add project: `basic-memory project add "name" ~/path`
- Project info: `basic-memory project info`
- Set cloud mode: `basic-memory project set-cloud "name"`
- Set local mode: `basic-memory project set-local "name"`
- One-way sync (local -> cloud): `basic-memory project sync`
- Bidirectional sync: `basic-memory project bisync`
- Integrity check: `basic-memory project check`
**Cloud Commands (requires subscription):**
- Authenticate (global): `basic-memory cloud login`
- Logout (global): `basic-memory cloud logout`
- Check cloud status: `basic-memory cloud status`
- Setup cloud sync: `basic-memory cloud setup`
- Save API key: `basic-memory cloud set-key bmc_...`
- Create API key: `basic-memory cloud create-key "name"`
- Manage snapshots: `basic-memory cloud snapshot [create|list|delete|show|browse]`
- Restore from snapshot: `basic-memory cloud restore <path> --snapshot <id>`
### MCP Capabilities
- Basic Memory exposes these MCP tools to LLMs:
**Content Management:**
- `write_note(title, content, directory, tags)` - Create/update markdown notes with semantic observations and relations
- `read_note(identifier, page, page_size)` - Read notes by title, permalink, or memory:// URL with knowledge graph awareness
- `read_content(path)` - Read raw file content (text, images, binaries) without knowledge graph processing
- `view_note(identifier, page, page_size)` - View notes as formatted artifacts for better readability
- `edit_note(identifier, operation, content)` - Edit notes incrementally (append, prepend, find/replace, replace_section)
- `move_note(identifier, destination_path, is_directory)` - Move notes or directories to new locations, updating database and maintaining links
- `delete_note(identifier, is_directory)` - Delete notes or directories from the knowledge base
**Knowledge Graph Navigation:**
- `build_context(url, depth, timeframe)` - Navigate the knowledge graph via memory:// URLs for conversation continuity
- `recent_activity(type, depth, timeframe)` - Get recently updated information with specified timeframe (e.g., "1d", "1 week")
- `list_directory(dir_name, depth, file_name_glob)` - Browse directory contents with filtering and depth control
**Search & Discovery:**
- `search_notes(query, page, page_size, search_type, types, entity_types, after_date)` - Full-text search across all content with advanced filtering options
**Project Management:**
- `list_memory_projects()` - List all available projects with their status
- `create_memory_project(project_name, project_path, set_default)` - Create new Basic Memory projects
- `delete_project(project_name)` - Delete a project from configuration
**Visualization:**
- `canvas(nodes, edges, title, directory)` - Generate Obsidian canvas files for knowledge graph visualization
**ChatGPT-Compatible Tools:**
- `search(query)` - Search across knowledge base (OpenAI actions compatible)
- `fetch(id)` - Fetch full content of a search result document
- MCP Prompts for better AI interaction:
- `ai_assistant_guide()` - Guidance on effectively using Basic Memory tools for AI assistants
- `continue_conversation(topic, timeframe)` - Continue previous conversations with relevant historical context
- `search(query, after_date)` - Search with detailed, formatted results for better context understanding
- `recent_activity(timeframe)` - View recently changed items with formatted output
### Cloud Features (v0.15.0+)
Basic Memory now supports cloud synchronization and storage (requires active subscription):
**Authentication:**
- JWT-based authentication with subscription validation
- Secure session management with token refresh
- Support for multiple cloud projects
**Bidirectional Sync:**
- rclone bisync integration for two-way synchronization
- Conflict resolution and integrity verification
- Real-time sync with change detection
- Mount/unmount cloud storage for direct file access
**Cloud Project Management:**
- Create and manage projects in the cloud
- Toggle between local and cloud modes
- Per-project sync configuration
- Subscription-based access control
**Security & Performance:**
- Removed .env file loading for improved security
- .gitignore integration (respects gitignored files)
- WAL mode for SQLite performance
- Background relation resolution (non-blocking startup)
- API performance optimizations (SPEC-11)
**Per-Project Cloud Routing:**
Individual projects can be routed through the cloud while others stay local, using an API key:
```bash
# Save API key and set project to cloud mode
basic-memory cloud set-key bmc_abc123...
basic-memory project set-cloud research # route through cloud
basic-memory project set-local research # revert to local
```
MCP tools use `get_project_client()` which automatically routes based on the project's mode. Cloud projects use the `cloud_api_key` from config as Bearer token.
**CLI Routing Flags (Global Cloud Mode):**
When global cloud mode is enabled, CLI commands route to the cloud API by default. Use `--local` and `--cloud` flags to override:
```bash
# Force local routing (ignore cloud mode)
basic-memory status --local
basic-memory project list --local
# Force cloud routing (when cloud mode is disabled)
basic-memory status --cloud
basic-memory project info my-project --cloud
```
Key behaviors:
- The local MCP server (`basic-memory mcp`) automatically uses local routing
- This allows simultaneous use of local Claude Desktop and cloud-based clients
- Some commands (like `project default`, `project sync-config`, `project move`) require `--local` in cloud mode since they modify local configuration
- Environment variable `BASIC_MEMORY_FORCE_LOCAL=true` forces local routing globally
- Per-project cloud routing via API key works independently of global cloud mode
## AI-Human Collaborative Development
Basic Memory emerged from and enables a new kind of development process that combines human and AI capabilities. Instead
of using AI just for code generation, we've developed a true collaborative workflow:
1. AI (LLM) writes initial implementation based on specifications and context
2. Human reviews, runs tests, and commits code with any necessary adjustments
3. Knowledge persists across conversations using Basic Memory's knowledge graph
4. Development continues seamlessly across different AI sessions with consistent context
5. Results improve through iterative collaboration and shared understanding
This approach has allowed us to tackle more complex challenges and build a more robust system than either humans or AI
could achieve independently.
**Problem-Solving Guidance:**
- If a solution isn't working after reasonable effort, suggest alternative approaches
- Don't persist with a problematic library or pattern when better alternatives exist
- Example: When py-pglite caused cascading test failures, switching to testcontainers-postgres was the right call
## GitHub Integration
Basic Memory has taken AI-Human collaboration to the next level by integrating Claude directly into the development workflow through GitHub:
### GitHub MCP Tools
Using the GitHub Model Context Protocol server, Claude can now:
- **Repository Management**:
- View repository files and structure
- Read file contents
- Create new branches
- Create and update files
- **Issue Management**:
- Create new issues
- Comment on existing issues
- Close and update issues
- Search across issues
- **Pull Request Workflow**:
- Create pull requests
- Review code changes
- Add comments to PRs
This integration enables Claude to participate as a full team member in the development process, not just as a code generation tool. Claude's GitHub account ([bm-claudeai](https://github.com/bm-claudeai)) is a member of the Basic Machines organization with direct contributor access to the codebase.
### Collaborative Development Process
With GitHub integration, the development workflow includes:
1. **Direct code review** - Claude can analyze PRs and provide detailed feedback
2. **Contribution tracking** - All of Claude's contributions are properly attributed in the Git history
3. **Branch management** - Claude can create feature branches for implementations
4. **Documentation maintenance** - Claude can keep documentation updated as the code evolves
5. **Code Commits**: ALWAYS sign off commits with `git commit -s`
This level of integration represents a new paradigm in AI-human collaboration, where the AI assistant becomes a full-fledged team member rather than just a tool for generating code snippets.
+154
View File
@@ -1,5 +1,159 @@
# CHANGELOG
## v0.18.3 (2026-02-12)
### Bug Fixes
- Use global `--header` flag for Tigris consistency on all rclone transactions
([`7fcf587`](https://github.com/basicmachines-co/basic-memory/commit/7fcf587))
- `--header-download` / `--header-upload` only apply to GET/PUT requests, missing S3
ListObjectsV2 calls that bisync issues first. Non-US users saw stale edge-cached metadata.
- `--header` applies to ALL HTTP transactions (list, download, upload), fixing bisync for
users outside the Tigris origin region.
## v0.18.2 (2026-02-11)
### Bug Fixes
- **#562**: Use VIRTUAL instead of STORED columns in SQLite migration
([`344e651`](https://github.com/basicmachines-co/basic-memory/commit/344e651))
- Fixes compatibility issue with SQLite STORED generated columns
## v0.18.1 (2026-02-11)
### Features
- **#552**: Add `--format json` to CLI tool commands
([`a47c9c0`](https://github.com/basicmachines-co/basic-memory/commit/a47c9c0))
- CLI tool commands now support `--format json` for machine-readable output
- **#535**: Support `tag:` query shorthand in search
([`f1d50c2`](https://github.com/basicmachines-co/basic-memory/commit/f1d50c2))
- Use `tag:mytag` as a convenient shorthand in search queries
- **#532**: Fast edit entities, refactors for webui, enhanced search
([`530cbac`](https://github.com/basicmachines-co/basic-memory/commit/530cbac))
- Performance improvements for entity editing and search operations
### Bug Fixes
- **#558**: Add X-Tigris-Consistent headers to all rclone commands
([`8489a3d`](https://github.com/basicmachines-co/basic-memory/commit/8489a3d))
- Ensures consistent reads from Tigris object storage during sync
- **#541**: Handle EntityCreationError as conflict
([`343a6e1`](https://github.com/basicmachines-co/basic-memory/commit/343a6e1))
- **#536**: Stabilize metadata filters on Postgres
([`009e849`](https://github.com/basicmachines-co/basic-memory/commit/009e849))
- **#533**: Fix recent_activity prompt defaults
([`24ca5f6`](https://github.com/basicmachines-co/basic-memory/commit/24ca5f6))
- **#530**: Prevent spurious `metadata: {}` in frontmatter output
([`e3ced49`](https://github.com/basicmachines-co/basic-memory/commit/e3ced49))
- Add POST legacy compat routes for v0.18.0 CLI
([`c46d7a6`](https://github.com/basicmachines-co/basic-memory/commit/c46d7a6))
- Restore legacy `/projects/projects` endpoint for older CLI versions
([`a0e754b`](https://github.com/basicmachines-co/basic-memory/commit/a0e754b))
### Internal
- **#538**: Add fast feedback loop tooling (`just fast-check`, `just doctor`, `just testmon`)
([`8072449`](https://github.com/basicmachines-co/basic-memory/commit/8072449))
## v0.18.0 (2026-01-28)
### Features
- **#527**: Add context-aware wiki link resolution with source_path support
([`0023e73`](https://github.com/basicmachines-co/basic-memory/commit/0023e73))
- Add `source_path` parameter to `resolve_link()` for context-aware resolution
- Relative path resolution: `[[nested/note]]` from `folder/file.md``folder/nested/note.md`
- Proximity-based resolution for duplicate titles (prefers notes in same folder)
- Strict mode to disable fuzzy search fallback for wiki links
- **#518**: Add directory support to move_note and delete_note tools
([`0b20801`](https://github.com/basicmachines-co/basic-memory/commit/0b20801))
- Add `is_directory` parameter to `move_note` and `delete_note` MCP tools
- New `POST /move-directory` and delete directory API endpoints
- Rename `folder``directory` parameter across codebase for consistency
- **#522**: Local MCP cloud mode routing
([`8730067`](https://github.com/basicmachines-co/basic-memory/commit/8730067))
- Add `--local` and `--cloud` CLI routing flags
- Local MCP server (`basic-memory mcp`) automatically uses local routing
- Enables simultaneous use of local Claude Desktop and cloud-based clients
- Environment variable `BASIC_MEMORY_FORCE_LOCAL=true` for global override
### Bug Fixes
- **#524**: Fix MCP prompt rendering errors
([`e14ba92`](https://github.com/basicmachines-co/basic-memory/commit/e14ba92))
- Fix "Error rendering prompt recent_activity" error
- Change `TimeFrame` to `str` in prompt type annotations for FastMCP compatibility
## v0.17.9 (2026-01-24)
### Bug Fixes
- **#523**: Fix `remove_project()` checking stale config in cloud mode
([`17c0e0a`](https://github.com/basicmachines-co/basic-memory/commit/17c0e0a))
- In cloud mode, only check database `is_default` field (source of truth)
- Config file can become stale when users set default project via v2 API
## v0.17.8 (2026-01-24)
### Bug Fixes
- **#521**: Fix `get_default_project()` returning multiple results
([`6888eff`](https://github.com/basicmachines-co/basic-memory/commit/6888eff))
- Query incorrectly matched any project with non-NULL `is_default` (both True and False)
- Now correctly checks for `is_default=True` only
## v0.17.7 (2026-01-24)
### Features
- **#476**: Add SPEC-29 Phase 3 bucket snapshot CLI commands
([`369ad37`](https://github.com/basicmachines-co/basic-memory/commit/369ad37))
- New `basic-memory cloud snapshot` commands for managing cloud snapshots
- Commands: `create`, `list`, `delete`, `show`, `browse`
- **#515**: Add MCP registry publication files
([`7a502e6`](https://github.com/basicmachines-co/basic-memory/commit/7a502e6))
### Bug Fixes
- **#520**: Read default project from database in cloud mode
([`38616c3`](https://github.com/basicmachines-co/basic-memory/commit/38616c3))
- **#513**: Ensure external_id is set on entity creation
([`c7835a9`](https://github.com/basicmachines-co/basic-memory/commit/c7835a9))
### Internal
- **#514**: Remove OpenPanel telemetry
([`85835ae`](https://github.com/basicmachines-co/basic-memory/commit/85835ae))
- Update README links to point to basicmemory.com
([`2aaee73`](https://github.com/basicmachines-co/basic-memory/commit/2aaee73))
## v0.17.6 (2026-01-17)
### Bug Fixes
- **#510**: Fix Docker container Python symlink broken at runtime
([`1799c94`](https://github.com/basicmachines-co/basic-memory/commit/1799c94))
### Internal
- Remove logfire config and specs docs, reduce lifespan and sync logging to debug level
([`d1d433d`](https://github.com/basicmachines-co/basic-memory/commit/d1d433d),
[`803f3ef`](https://github.com/basicmachines-co/basic-memory/commit/803f3ef))
## v0.17.5 (2026-01-11)
### Bug Fixes
-367
View File
@@ -1,367 +0,0 @@
# CLAUDE.md - Basic Memory Project Guide
## Project Overview
Basic Memory is a local-first knowledge management system built on the Model Context Protocol (MCP). It enables
bidirectional communication between LLMs (like Claude) and markdown files, creating a personal knowledge graph that can
be traversed using links between documents.
## CODEBASE DEVELOPMENT
### Project information
See the [README.md](README.md) file for a project overview.
### Build and Test Commands
- Install: `just install` or `pip install -e ".[dev]"`
- Run all tests (SQLite + Postgres): `just test`
- Run all tests against SQLite: `just test-sqlite`
- Run all tests against Postgres: `just test-postgres` (uses testcontainers)
- Run unit tests (SQLite): `just test-unit-sqlite`
- Run unit tests (Postgres): `just test-unit-postgres`
- Run integration tests (SQLite): `just test-int-sqlite`
- Run integration tests (Postgres): `just test-int-postgres`
- Generate HTML coverage: `just coverage`
- Single test: `pytest tests/path/to/test_file.py::test_function_name`
- Run benchmarks: `pytest test-int/test_sync_performance_benchmark.py -v -m "benchmark and not slow"`
- Lint: `just lint` or `ruff check . --fix`
- Type check: `just typecheck` or `uv run pyright`
- Format: `just format` or `uv run ruff format .`
- Run all code checks: `just check` (runs lint, format, typecheck, test)
- Create db migration: `just migration "Your migration message"`
- Run development MCP Inspector: `just run-inspector`
**Note:** Project requires Python 3.12+ (uses type parameter syntax and `type` aliases introduced in 3.12)
**Postgres Testing:** Uses [testcontainers](https://testcontainers-python.readthedocs.io/) which automatically spins up a Postgres instance in Docker. No manual database setup required - just have Docker running.
### Test Structure
- `tests/` - Unit tests for individual components (mocked, fast)
- `test-int/` - Integration tests for real-world scenarios (no mocks, realistic)
- Both directories are covered by unified coverage reporting
- Benchmark tests in `test-int/` are marked with `@pytest.mark.benchmark`
- Slow tests are marked with `@pytest.mark.slow`
### Code Style Guidelines
- Line length: 100 characters max
- Python 3.12+ with full type annotations (uses type parameters and type aliases)
- Format with ruff (consistent styling)
- Import order: standard lib, third-party, local imports
- Naming: snake_case for functions/variables, PascalCase for classes
- Prefer async patterns with SQLAlchemy 2.0
- Use Pydantic v2 for data validation and schemas
- CLI uses Typer for command structure
- API uses FastAPI for endpoints
- Follow the repository pattern for data access
- Tools communicate to api routers via the httpx ASGI client (in process)
### Code Change Guidelines
- **Full file read before edits**: Before editing any file, read it in full first to ensure complete context; partial reads lead to corrupted edits
- **Minimize diffs**: Prefer the smallest change that satisfies the request. Avoid unrelated refactors or style rewrites unless necessary for correctness
- **No speculative getattr**: Never use `getattr(obj, "attr", default)` when unsure about attribute names. Check the class definition or source code first
- **Fail fast**: Write code with fail-fast logic by default. Do not swallow exceptions with errors or warnings
- **No fallback logic**: Do not add fallback logic unless explicitly told to and agreed with the user
- **No guessing**: Do not say "The issue is..." before you actually know what the issue is. Investigate first.
### Literate Programming Style
Code should tell a story. Comments must explain the "why" and narrative flow, not just the "what".
**Section Headers:**
For files with multiple phases of logic, add section headers so the control flow reads like chapters:
```python
# --- Authentication ---
# ... auth logic ...
# --- Data Validation ---
# ... validation logic ...
# --- Business Logic ---
# ... core logic ...
```
**Decision Point Comments:**
For conditionals that materially change behavior (gates, fallbacks, retries, feature flags), add comments with:
- **Trigger**: what condition causes this branch
- **Why**: the rationale (cost, correctness, UX, determinism)
- **Outcome**: what changes downstream
```python
# Trigger: project has no active sync watcher
# Why: avoid duplicate file system watchers consuming resources
# Outcome: starts new watcher, registers in active_watchers dict
if project_id not in active_watchers:
start_watcher(project_id)
```
**Constraint Comments:**
If code exists because of a constraint (async requirements, rate limits, schema compatibility), explain the constraint near the code:
```python
# SQLite requires WAL mode for concurrent read/write access
connection.execute("PRAGMA journal_mode=WAL")
```
**What NOT to Comment:**
Avoid comments that restate obvious code:
```python
# Bad - restates code
counter += 1 # increment counter
# Good - explains why
counter += 1 # track retries for backoff calculation
```
### Codebase Architecture
See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for detailed architecture documentation.
**Directory Structure:**
- `/alembic` - Alembic db migrations
- `/api` - FastAPI REST endpoints + `container.py` composition root
- `/cli` - Typer CLI + `container.py` composition root
- `/deps` - Feature-scoped FastAPI dependencies (config, db, projects, repositories, services, importers)
- `/importers` - Import functionality for Claude, ChatGPT, and other sources
- `/markdown` - Markdown parsing and processing
- `/mcp` - MCP server + `container.py` composition root + `clients/` typed API clients
- `/models` - SQLAlchemy ORM models
- `/repository` - Data access layer
- `/schemas` - Pydantic models for validation
- `/services` - Business logic layer
- `/sync` - File synchronization services + `coordinator.py` for lifecycle management
**Composition Roots:**
Each entrypoint (API, MCP, CLI) has a composition root that:
- Reads `ConfigManager` (the only place that reads global config)
- Resolves runtime mode via `RuntimeMode` enum (TEST > CLOUD > LOCAL)
- Provides dependencies to downstream code explicitly
**Typed API Clients (MCP):**
MCP tools use typed clients in `mcp/clients/` to communicate with the API:
- `KnowledgeClient` - Entity CRUD operations
- `SearchClient` - Search operations
- `MemoryClient` - Context building
- `DirectoryClient` - Directory listing
- `ResourceClient` - Resource reading
- `ProjectClient` - Project management
Flow: MCP Tool → Typed Client → HTTP API → Router → Service → Repository
### Development Notes
- MCP tools are defined in src/basic_memory/mcp/tools/
- MCP prompts are defined in src/basic_memory/mcp/prompts/
- MCP tools should be atomic, composable operations
- Use `textwrap.dedent()` for multi-line string formatting in prompts and tools
- MCP Prompts are used to invoke tools and format content with instructions for an LLM
- Schema changes require Alembic migrations
- SQLite is used for indexing and full text search, files are source of truth
- Testing uses pytest with asyncio support (strict mode)
- Unit tests (`tests/`) use mocks when necessary; integration tests (`test-int/`) use real implementations
- By default, tests run against SQLite (fast, no Docker needed)
- Set `BASIC_MEMORY_TEST_POSTGRES=1` to run against Postgres (uses testcontainers - Docker required)
- Each test runs in a standalone environment with isolated database and tmp_path directory
- CI runs SQLite and Postgres tests in parallel for faster feedback
- Performance benchmarks are in `test-int/test_sync_performance_benchmark.py`
- Use pytest markers: `@pytest.mark.benchmark` for benchmarks, `@pytest.mark.slow` for slow tests
- **Coverage must stay at 100%**: Write tests for new code. Only use `# pragma: no cover` when tests would require excessive mocking (e.g., TYPE_CHECKING blocks, error handlers that need failure injection, runtime-mode-dependent code paths)
### Async Client Pattern (Important!)
**All MCP tools and CLI commands use the context manager pattern for HTTP clients:**
```python
from basic_memory.mcp.async_client import get_client
async def my_mcp_tool():
async with get_client() as client:
# Use client for API calls
response = await call_get(client, "/path")
return response
```
**Do NOT use:**
-`from basic_memory.mcp.async_client import client` (deprecated module-level client)
- ❌ Manual auth header management
-`inject_auth_header()` (deleted)
**Key principles:**
- Auth happens at client creation, not per-request
- Proper resource management via context managers
- Supports three modes: Local (ASGI), CLI cloud (HTTP + auth), Cloud app (factory injection)
- Factory pattern enables dependency injection for cloud consolidation
**For cloud app integration:**
```python
from basic_memory.mcp import async_client
# Set custom factory before importing tools
async_client.set_client_factory(your_custom_factory)
```
See SPEC-16 for full context manager refactor details.
## BASIC MEMORY PRODUCT USAGE
### Knowledge Structure
- Entity: Any concept, document, or idea represented as a markdown file
- Observation: A categorized fact about an entity (`- [category] content`)
- Relation: A directional link between entities (`- relation_type [[Target]]`)
- Frontmatter: YAML metadata at the top of markdown files
- Knowledge representation follows precise markdown format:
- Observations with [category] prefixes
- Relations with WikiLinks [[Entity]]
- Frontmatter with metadata
### Basic Memory Commands
**Local Commands:**
- Check sync status: `basic-memory status`
- Import from Claude: `basic-memory import claude conversations`
- Import from ChatGPT: `basic-memory import chatgpt`
- Import from Memory JSON: `basic-memory import memory-json`
- Tool access: `basic-memory tool` (provides CLI access to MCP tools)
- Continue: `basic-memory tool continue-conversation --topic="search"`
**Project Management:**
- List projects: `basic-memory project list`
- Add project: `basic-memory project add "name" ~/path`
- Project info: `basic-memory project info`
- One-way sync (local -> cloud): `basic-memory project sync`
- Bidirectional sync: `basic-memory project bisync`
- Integrity check: `basic-memory project check`
**Cloud Commands (requires subscription):**
- Authenticate: `basic-memory cloud login`
- Logout: `basic-memory cloud logout`
- Check cloud status: `basic-memory cloud status`
- Setup cloud sync: `basic-memory cloud setup`
### MCP Capabilities
- Basic Memory exposes these MCP tools to LLMs:
**Content Management:**
- `write_note(title, content, folder, tags)` - Create/update markdown notes with semantic observations and relations
- `read_note(identifier, page, page_size)` - Read notes by title, permalink, or memory:// URL with knowledge graph awareness
- `read_content(path)` - Read raw file content (text, images, binaries) without knowledge graph processing
- `view_note(identifier, page, page_size)` - View notes as formatted artifacts for better readability
- `edit_note(identifier, operation, content)` - Edit notes incrementally (append, prepend, find/replace, replace_section)
- `move_note(identifier, destination_path)` - Move notes to new locations, updating database and maintaining links
- `delete_note(identifier)` - Delete notes from the knowledge base
**Knowledge Graph Navigation:**
- `build_context(url, depth, timeframe)` - Navigate the knowledge graph via memory:// URLs for conversation continuity
- `recent_activity(type, depth, timeframe)` - Get recently updated information with specified timeframe (e.g., "1d", "1 week")
- `list_directory(dir_name, depth, file_name_glob)` - Browse directory contents with filtering and depth control
**Search & Discovery:**
- `search_notes(query, page, page_size, search_type, types, entity_types, after_date)` - Full-text search across all content with advanced filtering options
**Project Management:**
- `list_memory_projects()` - List all available projects with their status
- `create_memory_project(project_name, project_path, set_default)` - Create new Basic Memory projects
- `delete_project(project_name)` - Delete a project from configuration
**Visualization:**
- `canvas(nodes, edges, title, folder)` - Generate Obsidian canvas files for knowledge graph visualization
**ChatGPT-Compatible Tools:**
- `search(query)` - Search across knowledge base (OpenAI actions compatible)
- `fetch(id)` - Fetch full content of a search result document
- MCP Prompts for better AI interaction:
- `ai_assistant_guide()` - Guidance on effectively using Basic Memory tools for AI assistants
- `continue_conversation(topic, timeframe)` - Continue previous conversations with relevant historical context
- `search(query, after_date)` - Search with detailed, formatted results for better context understanding
- `recent_activity(timeframe)` - View recently changed items with formatted output
### Cloud Features (v0.15.0+)
Basic Memory now supports cloud synchronization and storage (requires active subscription):
**Authentication:**
- JWT-based authentication with subscription validation
- Secure session management with token refresh
- Support for multiple cloud projects
**Bidirectional Sync:**
- rclone bisync integration for two-way synchronization
- Conflict resolution and integrity verification
- Real-time sync with change detection
- Mount/unmount cloud storage for direct file access
**Cloud Project Management:**
- Create and manage projects in the cloud
- Toggle between local and cloud modes
- Per-project sync configuration
- Subscription-based access control
**Security & Performance:**
- Removed .env file loading for improved security
- .gitignore integration (respects gitignored files)
- WAL mode for SQLite performance
- Background relation resolution (non-blocking startup)
- API performance optimizations (SPEC-11)
## AI-Human Collaborative Development
Basic Memory emerged from and enables a new kind of development process that combines human and AI capabilities. Instead
of using AI just for code generation, we've developed a true collaborative workflow:
1. AI (LLM) writes initial implementation based on specifications and context
2. Human reviews, runs tests, and commits code with any necessary adjustments
3. Knowledge persists across conversations using Basic Memory's knowledge graph
4. Development continues seamlessly across different AI sessions with consistent context
5. Results improve through iterative collaboration and shared understanding
This approach has allowed us to tackle more complex challenges and build a more robust system than either humans or AI
could achieve independently.
**Problem-Solving Guidance:**
- If a solution isn't working after reasonable effort, suggest alternative approaches
- Don't persist with a problematic library or pattern when better alternatives exist
- Example: When py-pglite caused cascading test failures, switching to testcontainers-postgres was the right call
## GitHub Integration
Basic Memory has taken AI-Human collaboration to the next level by integrating Claude directly into the development workflow through GitHub:
### GitHub MCP Tools
Using the GitHub Model Context Protocol server, Claude can now:
- **Repository Management**:
- View repository files and structure
- Read file contents
- Create new branches
- Create and update files
- **Issue Management**:
- Create new issues
- Comment on existing issues
- Close and update issues
- Search across issues
- **Pull Request Workflow**:
- Create pull requests
- Review code changes
- Add comments to PRs
This integration enables Claude to participate as a full team member in the development process, not just as a code generation tool. Claude's GitHub account ([bm-claudeai](https://github.com/bm-claudeai)) is a member of the Basic Machines organization with direct contributor access to the codebase.
### Collaborative Development Process
With GitHub integration, the development workflow includes:
1. **Direct code review** - Claude can analyze PRs and provide detailed feedback
2. **Contribution tracking** - All of Claude's contributions are properly attributed in the Git history
3. **Branch management** - Claude can create feature branches for implementations
4. **Documentation maintenance** - Claude can keep documentation updated as the code evolves
5. **Code Commits**: ALWAYS sign off commits with `git commit -s`
This level of integration represents a new paradigm in AI-human collaboration, where the AI assistant becomes a full-fledged team member rather than just a tool for generating code snippets.
Symlink
+1
View File
@@ -0,0 +1 @@
AGENTS.md
+494
View File
@@ -0,0 +1,494 @@
# Note Format Reference
Every document in Basic Memory is a plain Markdown file. Files are the source of truth — changes to files automatically update the knowledge graph in the database. You maintain complete ownership, files work with git, and knowledge persists independently of any AI conversation.
## Document Structure
A note has three parts: YAML frontmatter, content (observations), and relations.
```markdown
---
title: Coffee Brewing Methods
type: note
tags: [coffee, brewing]
permalink: coffee-brewing-methods
---
# Coffee Brewing Methods
## Observations
- [method] Pour over provides more flavor clarity than French press
- [technique] Water temperature at 205°F extracts optimal compounds #brewing
- [preference] Ethiopian beans work well with lighter roasts (personal experience)
## Relations
- relates_to [[Coffee Bean Origins]]
- requires [[Proper Grinding Technique]]
- contrasts_with [[Tea Brewing Methods]]
```
The `## Observations` and `## Relations` headings are conventional but not required — the parser detects observations and relations by their syntax patterns anywhere in the document.
## Frontmatter
YAML metadata between `---` fences at the top of the file.
| Field | Required | Default | Description |
|-------|----------|---------|-------------|
| `title` | No | filename stem | Used for linking and references. Auto-set from filename if missing. |
| `type` | No | `note` | Entity type. Used for schema resolution and filtering. |
| `tags` | No | `[]` | List or comma-separated string. Used for organization and search. |
| `permalink` | No | generated from title | Stable identifier. Persists even if the file moves. |
| `schema` | No | none | Schema attachment — dict (inline), string (reference), or omitted (implicit). |
Custom fields are allowed. Any key not in the standard set is stored as `entity_metadata` and indexed for search and filtering.
```yaml
---
title: Paul Graham
type: Person
tags: [startups, essays, lisp]
permalink: paul-graham
status: active
source: wikipedia
---
```
Here `status` and `source` are custom fields stored in `entity_metadata`.
### Frontmatter Value Handling
YAML automatically converts some values to native types. Basic Memory normalizes them:
- Date strings (`2025-10-24`) → kept as ISO format strings
- Numbers (`1.0`) → converted to strings
- Booleans (`true`) → converted to strings (`"True"`)
- Lists and dicts → preserved, items normalized recursively
This prevents errors when downstream code expects string values.
## Observations
An observation is a categorized fact about the entity. Written as a Markdown list item.
**Syntax:**
```
- [category] content text #tag1 #tag2 (context)
```
| Part | Required | Description |
|------|----------|-------------|
| `[category]` | Yes | Classification in square brackets. Any text except `[]()` chars. |
| content | Yes | The fact or statement. |
| `#tags` | No | Inline tags. Space-separated, each starting with `#`. |
| `(context)` | No | Parenthesized text at end of line. Supporting details or source. |
### Examples
```markdown
- [tech] Uses SQLite for storage #database
- [design] Follows local-first architecture #architecture
- [decision] Selected bcrypt for passwords #security (based on OWASP audit)
- [name] Paul Graham
- [expertise] Startups
- [expertise] Lisp
- [expertise] Essay writing
```
Array-like fields use repeated categories — multiple `[expertise]` observations above.
### What Is Not an Observation
The parser excludes these list item patterns:
| Pattern | Example | Reason |
|---------|---------|--------|
| Checkboxes | `- [ ] Todo item`, `- [x] Done`, `- [-] Cancelled` | Task list syntax |
| Markdown links | `- [text](url)` | URL link syntax |
| Bare wiki links | `- [[Target]]` | Treated as a relation instead |
A list item with `#tags` but no `[category]` is still parsed — the tags are extracted and the category defaults to `Note`.
## Relations
Relations connect documents to form the knowledge graph. There are two kinds.
### Explicit Relations
Written as list items with a relation type and a `[[wiki link]]` target.
**Syntax:**
```
- relation_type [[Target Entity]] (context)
```
| Part | Required | Description |
|------|----------|-------------|
| `relation_type` | No | Text before `[[`. Defaults to `relates_to` if omitted. |
| `[[Target]]` | Yes | Wiki link to the target entity. Matched by title or permalink. |
| `(context)` | No | Parenthesized text after `]]`. Supporting details. |
### Examples
```markdown
- implements [[Search Design]]
- depends_on [[Database Schema]]
- works_at [[Y Combinator]] (co-founder)
- [[Some Entity]]
```
The last example — a bare `[[wiki link]]` in a list item — gets relation type `relates_to`.
Common relation types:
- `implements`, `depends_on`, `relates_to`, `inspired_by`
- `extends`, `part_of`, `contains`, `pairs_with`
- `works_at`, `authored`, `collaborated_with`
Any text works as a relation type. These are conventions, not a fixed set.
### Inline References
Wiki links appearing in regular prose (not as list items) create implicit `links_to` relations.
```markdown
This builds on [[Core Design]] and uses [[Utility Functions]].
```
This creates two relations: `links_to [[Core Design]]` and `links_to [[Utility Functions]]`.
### Forward References
Relations can link to entities that don't exist yet. Basic Memory resolves them when the target is created.
## Permalinks and memory:// URLs
Every document has a unique **permalink** — a stable identifier derived from its title. You can set one explicitly in frontmatter, or let the system generate it.
```yaml
permalink: auth-approaches-2024
```
Permalinks form the basis of `memory://` URLs:
```
memory://auth-approaches-2024 # By permalink
memory://Authentication Approaches # By title (auto-resolves)
memory://project/auth-approaches # By path
```
Pattern matching is supported:
```
memory://auth* # Starts with "auth"
memory://*/approaches # Ends with "approaches"
memory://project/*/requirements # Nested wildcard
```
## Schemas
Schemas declare the expected structure of a note — which observation categories and relation types a well-formed note should have. They use Picoschema, a compact notation from Google's Dotprompt that fits naturally in YAML frontmatter.
### Picoschema Syntax
```yaml
schema:
name: string, full name # required field with description
email?: string, contact email # ? = optional
role?: string, job title
works_at?: Organization, employer # capitalized type = entity reference
tags?(array): string, categories # array of type
status?(enum): [active, inactive] # enum with allowed values
metadata?(object): # nested object
updated_at?: string
source?: string
```
| Notation | Meaning | Example |
|----------|---------|---------|
| `field: type` | Required field | `name: string` |
| `field?: type` | Optional field | `role?: string` |
| `field(array): type` | Array of values | `expertise(array): string` |
| `field?(enum): [vals]` | Enum with allowed values | `status?(enum): [active, inactive]` |
| `field?(object):` | Nested object with sub-fields | `metadata?(object):` |
| `, description` | Description after comma | `name: string, full name` |
| `EntityName` | Capitalized type = entity reference | `works_at?: Organization` |
**Scalar types:** `string`, `integer`, `number`, `boolean`, `any`
Any type not in that set whose first letter is uppercase is treated as an entity reference (a relation target).
### Schema-to-Note Mapping
Schemas validate against existing observation/relation syntax. Note authors don't learn new syntax.
| Schema Declaration | Maps To | Example in Note |
|--------------------|---------|-----------------|
| `field: string` | Observation `[field] value` | `- [name] Paul Graham` |
| `field?(array): string` | Multiple `[field]` observations | `- [expertise] Lisp` (repeated) |
| `field?: EntityType` | Relation `field [[Target]]` | `- works_at [[Y Combinator]]` |
| `field?(array): EntityType` | Multiple `field` relations | `- authored [[Book]]` (repeated) |
| `tags` | Frontmatter `tags` array | `tags: [startups, essays]` |
| `field?(enum): [vals]` | Observation `[field] value` where value is in the set | `- [status] active` |
Observations and relations not covered by the schema are valid — schemas describe a subset, not a straitjacket.
### Schema Attachment
Three ways to attach a schema to a note, resolved in priority order:
**1. Inline schema**`schema` is a dict in frontmatter:
```yaml
---
title: Team Standup 2024-01-15
type: meeting
schema:
attendees(array): string, who was there
decisions(array): string, what was decided
action_items(array): string, follow-ups
blockers?(array): string, anything stuck
---
```
Good for one-off structured notes or prototyping a schema before extracting it.
**2. Explicit reference**`schema` is a string naming a schema note:
```yaml
---
title: Basic Memory
schema: SoftwareProject
---
```
or by permalink:
```yaml
---
title: LLM Memory Patterns
schema: schema/research-project
---
```
Use when the note's `type` differs from the schema it should validate against, or when multiple schema variants exist.
**3. Implicit by type** — no `schema` field, resolved by matching `type`:
```yaml
---
title: Paul Graham
type: Person
---
```
The system looks up a schema note where `entity: Person`. If found, it applies. If not, no validation occurs.
**4. No schema** — perfectly fine. Most notes don't need one.
### Schema Notes
A schema is itself a Basic Memory note with `type: schema`. It lives anywhere (though `schema/` is the conventional directory).
```yaml
# schema/Person.md
---
title: Person
type: schema
entity: Person
version: 1
schema:
name: string, full name
role?: string, job title or position
works_at?: Organization, employer
expertise?(array): string, areas of knowledge
email?: string, contact email
settings:
validation: warn
---
# Person
A human individual in the knowledge graph.
```
| Field | Required | Description |
|-------|----------|-------------|
| `type` | Yes | Must be `schema` |
| `entity` | Yes | The entity type this schema describes (e.g., `Person`) |
| `version` | No | Schema version number (default: `1`) |
| `schema` | Yes | Picoschema dict defining the fields |
| `settings.validation` | No | Validation mode (default: `warn`) |
Schema notes are regular notes — they show up in search, can have observations and relations, and participate in the knowledge graph.
### Validation Modes
| Mode | Behavior |
|------|----------|
| `warn` | Warnings in output, doesn't block (default) |
| `strict` | Errors that block sync, for CI/CD enforcement |
| `off` | No validation |
### Validation Output
```
$ bm schema validate people/ada-lovelace.md
⚠ Person schema validation:
- Missing required field: name (expected [name] observation)
- Missing optional field: role
- Missing optional field: works_at (no relation found)
Unmatched observations: [fact] ×2, [born] ×1
Unmatched relations: collaborated_with
```
"Unmatched" items are informational — observations and relations the schema doesn't cover.
### Schema Inference
Generate schemas from existing notes by analyzing observation and relation frequency:
```
$ bm schema infer Person
Analyzing 30 notes with type: Person...
Observations found:
[name] 30/30 100% → name: string
[role] 27/30 90% → role?: string
[expertise] 18/30 60% → expertise?(array): string
[email] 8/30 27% → email?: string
Relations found:
works_at 22/30 73% → works_at?: Organization
Suggested schema:
name: string, full name
role?: string, job title
expertise?(array): string, areas of knowledge
email?: string, contact email
works_at?: Organization, employer
Save to schema/Person.md? [y/n]
```
Frequency thresholds:
- **100% present** → required field
- **25%+ present** → optional field
- **Below 25%** → excluded from suggestion
### Schema Drift Detection
Track how usage patterns shift over time:
```
$ bm schema diff Person
Schema drift detected:
+ expertise: now in 81% of notes (was 12%)
- department: dropped to 3% of notes
~ works_at: cardinality changed (one → many)
Update schema? [y/n/review]
```
## Complete Examples
### Simple Note (No Schema)
```markdown
---
title: Project Ideas
type: note
tags: [ideas, brainstorm]
---
# Project Ideas
## Observations
- [idea] Build a CLI tool for markdown linting #tooling
- [idea] Create a recipe knowledge base #cooking
- [priority] Focus on developer tools first (Q1 goal)
## Relations
- inspired_by [[Developer Workflow Research]]
- part_of [[Q1 Planning]]
```
### Schema-Validated Note
Schema at `schema/Person.md`:
```yaml
---
title: Person
type: schema
entity: Person
version: 1
schema:
name: string, full name
role?: string, job title or position
works_at?: Organization, employer
expertise?(array): string, areas of knowledge
email?: string, contact email
settings:
validation: warn
---
# Person
A human individual in the knowledge graph.
```
Note at `people/paul-graham.md`:
```markdown
---
title: Paul Graham
type: Person
tags: [startups, essays, lisp]
---
# Paul Graham
## Observations
- [name] Paul Graham
- [role] Essayist and investor
- [expertise] Startups
- [expertise] Lisp
- [expertise] Essay writing
- [fact] Created Viaweb, the first web app
## Relations
- works_at [[Y Combinator]]
- authored [[Hackers and Painters]]
```
The `[fact]` observation and `authored` relation are not in the schema — they're valid, just unmatched. The schema only checks that `[name]` exists (required) and looks for optional fields like `[role]`, `[expertise]`, and `works_at`.
### Inline Schema Note
```markdown
---
title: Team Standup 2024-01-15
type: meeting
schema:
attendees(array): string, who was there
decisions(array): string, what was decided
action_items(array): string, follow-ups
blockers?(array): string, anything stuck
---
# Team Standup 2024-01-15
## Observations
- [attendees] Paul
- [attendees] Sarah
- [decisions] Ship v2 by Friday
- [action_items] Paul to review PR #42
- [blockers] Waiting on API credentials
```
+68 -60
View File
@@ -1,3 +1,4 @@
<!-- mcp-name: io.github.basicmachines-co/basic-memory -->
[![License: AGPL v3](https://img.shields.io/badge/License-AGPL_v3-blue.svg)](https://www.gnu.org/licenses/agpl-3.0)
[![PyPI version](https://badge.fury.io/py/basic-memory.svg)](https://badge.fury.io/py/basic-memory)
[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
@@ -5,15 +6,14 @@
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
![](https://badge.mcpx.dev?type=server 'MCP Server')
![](https://badge.mcpx.dev?type=dev 'MCP Dev')
[![smithery badge](https://smithery.ai/badge/@basicmachines-co/basic-memory)](https://smithery.ai/server/@basicmachines-co/basic-memory)
## 🚀 Basic Memory Cloud is Live!
- **Cross-device and multi-platform support is here.** Your knowledge graph now works on desktop, web, and mobile - seamlessly synced across all your AI tools (Claude, ChatGPT, Gemini, Claude Code, and Codex)
- **Early Supporter Pricing:** Early users get 25% off forever.
The open source project continues as always. Cloud just makes it work everywhere.
- **Cross-device and multi-platform support is here.** Your knowledge graph now works on desktop, web, and mobile.
- **Cloud is optional.** The local-first open-source workflow continues as always.
- **OSS discount:** use code `{{OSS_DISCOUNT_CODE}}` for 20% off for 3 months.
[Sign up now →](https://basicmemory.com/beta)
[Sign up now →](https://basicmemory.com)
with a 7 day free trial
@@ -23,8 +23,8 @@ Basic Memory lets you build persistent knowledge through natural conversations w
Claude, while keeping everything in simple Markdown files on your computer. It uses the Model Context Protocol (MCP) to
enable any compatible LLM to read and write to your local knowledge base.
- Website: https://basicmachines.co
- Documentation: https://memory.basicmachines.co
- Website: https://basicmemory.com
- Documentation: https://docs.basicmemory.com
## Pick up your conversation right where you left off
@@ -62,24 +62,6 @@ uv tool install basic-memory
You can view shared context via files in `~/basic-memory` (default directory location).
### Alternative Installation via Smithery
You can use [Smithery](https://smithery.ai/server/@basicmachines-co/basic-memory) to automatically configure Basic
Memory for Claude Desktop:
```bash
npx -y @smithery/cli install @basicmachines-co/basic-memory --client claude
```
This installs and configures Basic Memory without requiring manual edits to the Claude Desktop configuration file. The
Smithery server hosts the MCP server component, while your data remains stored locally as Markdown files.
### Glama.ai
<a href="https://glama.ai/mcp/servers/o90kttu9ym">
<img width="380" height="200" src="https://glama.ai/mcp/servers/o90kttu9ym/badge" alt="basic-memory MCP server" />
</a>
## Why Basic Memory?
Most LLM interactions are ephemeral - you ask a question, get an answer, and everything is forgotten. Each conversation
@@ -362,7 +344,7 @@ basic-memory sync --watch
3. Cloud features (optional, requires subscription):
```bash
# Authenticate with cloud
# Authenticate with cloud (global cloud mode via OAuth)
basic-memory cloud login
# Bidirectional sync with cloud
@@ -375,6 +357,42 @@ basic-memory cloud check
basic-memory cloud mount
```
**Per-Project Cloud Routing** (API key based):
Individual projects can be routed through the cloud while others stay local. This uses an API key instead of OAuth:
```bash
# Save an API key (create one in the web app or via CLI)
basic-memory cloud set-key bmc_abc123...
# Or create one via CLI (requires OAuth login first)
basic-memory cloud create-key "my-laptop"
# Set a project to route through cloud
basic-memory project set-cloud research
# Revert a project to local mode
basic-memory project set-local research
# List projects with mode column (local/cloud)
basic-memory project list
```
**Routing Flags** (for users with global cloud mode):
When global cloud mode is enabled, CLI commands communicate with the cloud API by default. Use routing flags to override this:
```bash
# Force local routing (useful for local MCP server while cloud mode is enabled)
basic-memory status --local
basic-memory project list --local
# Force cloud routing (when cloud mode is disabled but you want cloud access)
basic-memory status --cloud
basic-memory project info my-project --cloud
```
The local MCP server (`basic-memory mcp`) automatically uses local routing, so you can use both local Claude Desktop and cloud-based clients simultaneously.
4. In Claude Desktop, the LLM can now use these tools:
**Content Management:**
@@ -398,6 +416,8 @@ list_directory(dir_name, depth) - Browse directory contents with filtering
**Search & Discovery:**
```
search(query, page, page_size) - Search across your knowledge base
search_notes(query, page, page_size, search_type, types, entity_types, after_date, metadata_filters, tags, status, project) - Search with filters
search_by_metadata(filters, limit, offset, project) - Structured frontmatter search
```
**Project Management:**
@@ -408,6 +428,12 @@ get_current_project() - Show current project stats
sync_status() - Check synchronization status
```
**Cloud Discovery (opt-in):**
```
cloud_info() - Show optional Cloud overview and setup guidance
release_notes() - Show latest release notes
```
**Visualization:**
```
canvas(nodes, edges, title, folder) - Generate knowledge visualizations
@@ -425,7 +451,7 @@ canvas(nodes, edges, title, folder) - Generate knowledge visualizations
## Futher info
See the [Documentation](https://memory.basicmachines.co/) for more info, including:
See the [Documentation](https://docs.basicmemory.com) for more info, including:
- [Complete User Guide](https://docs.basicmemory.com/user-guide/)
- [CLI tools](https://docs.basicmemory.com/guides/cli-reference/)
@@ -451,6 +477,7 @@ Basic Memory uses [Loguru](https://github.com/Delgan/loguru) for logging. The lo
|----------|---------|-------------|
| `BASIC_MEMORY_LOG_LEVEL` | `INFO` | Log level: DEBUG, INFO, WARNING, ERROR |
| `BASIC_MEMORY_CLOUD_MODE` | `false` | When `true`, API logs to stdout with structured context |
| `BASIC_MEMORY_FORCE_LOCAL` | `false` | When `true`, forces local API routing (ignores cloud mode) |
| `BASIC_MEMORY_ENV` | `dev` | Set to `test` for test mode (stderr only) |
### Examples
@@ -466,39 +493,6 @@ tail -f ~/.basic-memory/basic-memory.log
BASIC_MEMORY_CLOUD_MODE=true uvicorn basic_memory.api.app:app
```
## Telemetry
Basic Memory collects anonymous usage statistics to help improve the software. This follows the [Homebrew model](https://docs.brew.sh/Analytics) - telemetry is on by default with easy opt-out.
**What we collect:**
- App version, Python version, OS, architecture
- Feature usage (which MCP tools and CLI commands are used)
- Error types (sanitized - no file paths or personal data)
**What we NEVER collect:**
- Note content, file names, or paths
- Personal information
- IP addresses
**Opting out:**
```bash
# Disable telemetry
basic-memory telemetry disable
# Check status
basic-memory telemetry status
# Re-enable
basic-memory telemetry enable
```
Or set the environment variable:
```bash
export BASIC_MEMORY_TELEMETRY_ENABLED=false
```
For more details, see the [Telemetry documentation](https://basicmemory.com/telemetry).
## Development
### Running Tests
@@ -528,16 +522,23 @@ just test
- `just test-int-postgres` - Run integration tests against Postgres
- `just test-windows` - Run Windows-specific tests (auto-skips on other platforms)
- `just test-benchmark` - Run performance benchmark tests
- `just testmon` - Run tests impacted by recent changes (pytest-testmon)
- `just test-smoke` - Run fast MCP end-to-end smoke test
- `just fast-check` - Run fix/format/typecheck + impacted tests + smoke test
- `just doctor` - Run local file <-> DB consistency checks with temp config
**Postgres Testing:**
Postgres tests use [testcontainers](https://testcontainers-python.readthedocs.io/) which automatically spins up a Postgres instance in Docker. No manual database setup required - just have Docker running.
**Testmon Note:** When no files have changed, `just testmon` may collect 0 tests. That's expected and means no impacted tests were detected.
**Test Markers:**
Tests use pytest markers for selective execution:
- `windows` - Windows-specific database optimizations
- `benchmark` - Performance tests (excluded from default runs)
- `smoke` - Fast MCP end-to-end smoke tests
**Other Development Commands:**
```bash
@@ -545,10 +546,17 @@ just install # Install with dev dependencies
just lint # Run linting checks
just typecheck # Run type checking
just format # Format code with ruff
just fast-check # Fast local loop (fix/format/typecheck + testmon + smoke)
just doctor # Local consistency check (temp config)
just check # Run all quality checks
just migration "msg" # Create database migration
```
**Local Consistency Check:**
```bash
basic-memory doctor # Verifies file <-> database sync in a temp project
```
See the [justfile](justfile) for the complete list of development commands.
## License
+37 -5
View File
@@ -110,6 +110,8 @@ def resolve_runtime_mode(cloud_mode_enabled: bool, is_test_env: bool) -> Runtime
return RuntimeMode.LOCAL
```
**Note**: `RuntimeMode` determines global behavior (e.g., whether to start file sync). Per-project routing is orthogonal — individual projects can be set to `cloud` mode via `ProjectMode` in config, which affects client routing in `get_client(project_name=...)` without changing the global runtime mode.
## Dependencies Package
### Structure
@@ -214,15 +216,45 @@ Example tool using typed client:
```python
@mcp.tool()
async def search_notes(query: str, project: str | None = None) -> SearchResponse:
async with get_client() as client:
active_project = await get_active_project(client, project)
async def search_notes(
query: str,
project: str | None = None,
metadata_filters: dict | None = None,
tags: list[str] | None = None,
status: str | None = None,
) -> SearchResponse:
async with get_project_client(project, context) as (client, active_project):
# Import client inside function to avoid circular imports
from basic_memory.mcp.clients import SearchClient
from basic_memory.schemas.search import SearchQuery
search_query = SearchQuery(
text=query,
metadata_filters=metadata_filters,
tags=tags,
status=status,
)
search_client = SearchClient(client, active_project.external_id)
return await search_client.search(query)
return await search_client.search(search_query.model_dump())
```
### Per-Project Client Routing
`get_project_client()` from `mcp/project_context.py` is an async context manager that:
1. Resolves the project name from config (no network call)
2. Creates the correctly-routed client based on the project's mode (local ASGI or cloud HTTP with API key)
3. Validates the project via the API
4. Yields `(client, active_project)` tuple
This solves the bootstrap problem: you need the project name to choose the right client (local vs cloud), but you need the client to validate the project exists.
```python
from basic_memory.mcp.project_context import get_project_client
async with get_project_client(project, context) as (client, active_project):
# client is routed based on project's mode (local or cloud)
# active_project is validated via the API
...
```
## Sync Coordination
+494
View File
@@ -0,0 +1,494 @@
# Note Format Reference
Every document in Basic Memory is a plain Markdown file. Files are the source of truth — changes to files automatically update the knowledge graph in the database. You maintain complete ownership, files work with git, and knowledge persists independently of any AI conversation.
## Document Structure
A note has three parts: YAML frontmatter, content (observations), and relations.
```markdown
---
title: Coffee Brewing Methods
type: note
tags: [coffee, brewing]
permalink: coffee-brewing-methods
---
# Coffee Brewing Methods
## Observations
- [method] Pour over provides more flavor clarity than French press
- [technique] Water temperature at 205°F extracts optimal compounds #brewing
- [preference] Ethiopian beans work well with lighter roasts (personal experience)
## Relations
- relates_to [[Coffee Bean Origins]]
- requires [[Proper Grinding Technique]]
- contrasts_with [[Tea Brewing Methods]]
```
The `## Observations` and `## Relations` headings are conventional but not required — the parser detects observations and relations by their syntax patterns anywhere in the document.
## Frontmatter
YAML metadata between `---` fences at the top of the file.
| Field | Required | Default | Description |
|-------|----------|---------|-------------|
| `title` | No | filename stem | Used for linking and references. Auto-set from filename if missing. |
| `type` | No | `note` | Entity type. Used for schema resolution and filtering. |
| `tags` | No | `[]` | List or comma-separated string. Used for organization and search. |
| `permalink` | No | generated from title | Stable identifier. Persists even if the file moves. |
| `schema` | No | none | Schema attachment — dict (inline), string (reference), or omitted (implicit). |
Custom fields are allowed. Any key not in the standard set is stored as `entity_metadata` and indexed for search and filtering.
```yaml
---
title: Paul Graham
type: Person
tags: [startups, essays, lisp]
permalink: paul-graham
status: active
source: wikipedia
---
```
Here `status` and `source` are custom fields stored in `entity_metadata`.
### Frontmatter Value Handling
YAML automatically converts some values to native types. Basic Memory normalizes them:
- Date strings (`2025-10-24`) → kept as ISO format strings
- Numbers (`1.0`) → converted to strings
- Booleans (`true`) → converted to strings (`"True"`)
- Lists and dicts → preserved, items normalized recursively
This prevents errors when downstream code expects string values.
## Observations
An observation is a categorized fact about the entity. Written as a Markdown list item.
**Syntax:**
```
- [category] content text #tag1 #tag2 (context)
```
| Part | Required | Description |
|------|----------|-------------|
| `[category]` | Yes | Classification in square brackets. Any text except `[]()` chars. |
| content | Yes | The fact or statement. |
| `#tags` | No | Inline tags. Space-separated, each starting with `#`. |
| `(context)` | No | Parenthesized text at end of line. Supporting details or source. |
### Examples
```markdown
- [tech] Uses SQLite for storage #database
- [design] Follows local-first architecture #architecture
- [decision] Selected bcrypt for passwords #security (based on OWASP audit)
- [name] Paul Graham
- [expertise] Startups
- [expertise] Lisp
- [expertise] Essay writing
```
Array-like fields use repeated categories — multiple `[expertise]` observations above.
### What Is Not an Observation
The parser excludes these list item patterns:
| Pattern | Example | Reason |
|---------|---------|--------|
| Checkboxes | `- [ ] Todo item`, `- [x] Done`, `- [-] Cancelled` | Task list syntax |
| Markdown links | `- [text](url)` | URL link syntax |
| Bare wiki links | `- [[Target]]` | Treated as a relation instead |
A list item with `#tags` but no `[category]` is still parsed — the tags are extracted and the category defaults to `Note`.
## Relations
Relations connect documents to form the knowledge graph. There are two kinds.
### Explicit Relations
Written as list items with a relation type and a `[[wiki link]]` target.
**Syntax:**
```
- relation_type [[Target Entity]] (context)
```
| Part | Required | Description |
|------|----------|-------------|
| `relation_type` | No | Text before `[[`. Defaults to `relates_to` if omitted. |
| `[[Target]]` | Yes | Wiki link to the target entity. Matched by title or permalink. |
| `(context)` | No | Parenthesized text after `]]`. Supporting details. |
### Examples
```markdown
- implements [[Search Design]]
- depends_on [[Database Schema]]
- works_at [[Y Combinator]] (co-founder)
- [[Some Entity]]
```
The last example — a bare `[[wiki link]]` in a list item — gets relation type `relates_to`.
Common relation types:
- `implements`, `depends_on`, `relates_to`, `inspired_by`
- `extends`, `part_of`, `contains`, `pairs_with`
- `works_at`, `authored`, `collaborated_with`
Any text works as a relation type. These are conventions, not a fixed set.
### Inline References
Wiki links appearing in regular prose (not as list items) create implicit `links_to` relations.
```markdown
This builds on [[Core Design]] and uses [[Utility Functions]].
```
This creates two relations: `links_to [[Core Design]]` and `links_to [[Utility Functions]]`.
### Forward References
Relations can link to entities that don't exist yet. Basic Memory resolves them when the target is created.
## Permalinks and memory:// URLs
Every document has a unique **permalink** — a stable identifier derived from its title. You can set one explicitly in frontmatter, or let the system generate it.
```yaml
permalink: auth-approaches-2024
```
Permalinks form the basis of `memory://` URLs:
```
memory://auth-approaches-2024 # By permalink
memory://Authentication Approaches # By title (auto-resolves)
memory://project/auth-approaches # By path
```
Pattern matching is supported:
```
memory://auth* # Starts with "auth"
memory://*/approaches # Ends with "approaches"
memory://project/*/requirements # Nested wildcard
```
## Schemas
Schemas declare the expected structure of a note — which observation categories and relation types a well-formed note should have. They use Picoschema, a compact notation from Google's Dotprompt that fits naturally in YAML frontmatter.
### Picoschema Syntax
```yaml
schema:
name: string, full name # required field with description
email?: string, contact email # ? = optional
role?: string, job title
works_at?: Organization, employer # capitalized type = entity reference
tags?(array): string, categories # array of type
status?(enum): [active, inactive] # enum with allowed values
metadata?(object): # nested object
updated_at?: string
source?: string
```
| Notation | Meaning | Example |
|----------|---------|---------|
| `field: type` | Required field | `name: string` |
| `field?: type` | Optional field | `role?: string` |
| `field(array): type` | Array of values | `expertise(array): string` |
| `field?(enum): [vals]` | Enum with allowed values | `status?(enum): [active, inactive]` |
| `field?(object):` | Nested object with sub-fields | `metadata?(object):` |
| `, description` | Description after comma | `name: string, full name` |
| `EntityName` | Capitalized type = entity reference | `works_at?: Organization` |
**Scalar types:** `string`, `integer`, `number`, `boolean`, `any`
Any type not in that set whose first letter is uppercase is treated as an entity reference (a relation target).
### Schema-to-Note Mapping
Schemas validate against existing observation/relation syntax. Note authors don't learn new syntax.
| Schema Declaration | Maps To | Example in Note |
|--------------------|---------|-----------------|
| `field: string` | Observation `[field] value` | `- [name] Paul Graham` |
| `field?(array): string` | Multiple `[field]` observations | `- [expertise] Lisp` (repeated) |
| `field?: EntityType` | Relation `field [[Target]]` | `- works_at [[Y Combinator]]` |
| `field?(array): EntityType` | Multiple `field` relations | `- authored [[Book]]` (repeated) |
| `tags` | Frontmatter `tags` array | `tags: [startups, essays]` |
| `field?(enum): [vals]` | Observation `[field] value` where value is in the set | `- [status] active` |
Observations and relations not covered by the schema are valid — schemas describe a subset, not a straitjacket.
### Schema Attachment
Three ways to attach a schema to a note, resolved in priority order:
**1. Inline schema**`schema` is a dict in frontmatter:
```yaml
---
title: Team Standup 2024-01-15
type: meeting
schema:
attendees(array): string, who was there
decisions(array): string, what was decided
action_items(array): string, follow-ups
blockers?(array): string, anything stuck
---
```
Good for one-off structured notes or prototyping a schema before extracting it.
**2. Explicit reference**`schema` is a string naming a schema note:
```yaml
---
title: Basic Memory
schema: SoftwareProject
---
```
or by permalink:
```yaml
---
title: LLM Memory Patterns
schema: schema/research-project
---
```
Use when the note's `type` differs from the schema it should validate against, or when multiple schema variants exist.
**3. Implicit by type** — no `schema` field, resolved by matching `type`:
```yaml
---
title: Paul Graham
type: Person
---
```
The system looks up a schema note where `entity: Person`. If found, it applies. If not, no validation occurs.
**4. No schema** — perfectly fine. Most notes don't need one.
### Schema Notes
A schema is itself a Basic Memory note with `type: schema`. It lives anywhere (though `schema/` is the conventional directory).
```yaml
# schema/Person.md
---
title: Person
type: schema
entity: Person
version: 1
schema:
name: string, full name
role?: string, job title or position
works_at?: Organization, employer
expertise?(array): string, areas of knowledge
email?: string, contact email
settings:
validation: warn
---
# Person
A human individual in the knowledge graph.
```
| Field | Required | Description |
|-------|----------|-------------|
| `type` | Yes | Must be `schema` |
| `entity` | Yes | The entity type this schema describes (e.g., `Person`) |
| `version` | No | Schema version number (default: `1`) |
| `schema` | Yes | Picoschema dict defining the fields |
| `settings.validation` | No | Validation mode (default: `warn`) |
Schema notes are regular notes — they show up in search, can have observations and relations, and participate in the knowledge graph.
### Validation Modes
| Mode | Behavior |
|------|----------|
| `warn` | Warnings in output, doesn't block (default) |
| `strict` | Errors that block sync, for CI/CD enforcement |
| `off` | No validation |
### Validation Output
```
$ bm schema validate people/ada-lovelace.md
⚠ Person schema validation:
- Missing required field: name (expected [name] observation)
- Missing optional field: role
- Missing optional field: works_at (no relation found)
Unmatched observations: [fact] ×2, [born] ×1
Unmatched relations: collaborated_with
```
"Unmatched" items are informational — observations and relations the schema doesn't cover.
### Schema Inference
Generate schemas from existing notes by analyzing observation and relation frequency:
```
$ bm schema infer Person
Analyzing 30 notes with type: Person...
Observations found:
[name] 30/30 100% → name: string
[role] 27/30 90% → role?: string
[expertise] 18/30 60% → expertise?(array): string
[email] 8/30 27% → email?: string
Relations found:
works_at 22/30 73% → works_at?: Organization
Suggested schema:
name: string, full name
role?: string, job title
expertise?(array): string, areas of knowledge
email?: string, contact email
works_at?: Organization, employer
Save to schema/Person.md? [y/n]
```
Frequency thresholds:
- **100% present** → required field
- **25%+ present** → optional field
- **Below 25%** → excluded from suggestion
### Schema Drift Detection
Track how usage patterns shift over time:
```
$ bm schema diff Person
Schema drift detected:
+ expertise: now in 81% of notes (was 12%)
- department: dropped to 3% of notes
~ works_at: cardinality changed (one → many)
Update schema? [y/n/review]
```
## Complete Examples
### Simple Note (No Schema)
```markdown
---
title: Project Ideas
type: note
tags: [ideas, brainstorm]
---
# Project Ideas
## Observations
- [idea] Build a CLI tool for markdown linting #tooling
- [idea] Create a recipe knowledge base #cooking
- [priority] Focus on developer tools first (Q1 goal)
## Relations
- inspired_by [[Developer Workflow Research]]
- part_of [[Q1 Planning]]
```
### Schema-Validated Note
Schema at `schema/Person.md`:
```yaml
---
title: Person
type: schema
entity: Person
version: 1
schema:
name: string, full name
role?: string, job title or position
works_at?: Organization, employer
expertise?(array): string, areas of knowledge
email?: string, contact email
settings:
validation: warn
---
# Person
A human individual in the knowledge graph.
```
Note at `people/paul-graham.md`:
```markdown
---
title: Paul Graham
type: Person
tags: [startups, essays, lisp]
---
# Paul Graham
## Observations
- [name] Paul Graham
- [role] Essayist and investor
- [expertise] Startups
- [expertise] Lisp
- [expertise] Essay writing
- [fact] Created Viaweb, the first web app
## Relations
- works_at [[Y Combinator]]
- authored [[Hackers and Painters]]
```
The `[fact]` observation and `authored` relation are not in the schema — they're valid, just unmatched. The schema only checks that `[name]` exists (required) and looks for optional fields like `[role]`, `[expertise]`, and `works_at`.
### Inline Schema Note
```markdown
---
title: Team Standup 2024-01-15
type: meeting
schema:
attendees(array): string, who was there
decisions(array): string, what was decided
action_items(array): string, follow-ups
blockers?(array): string, anything stuck
---
# Team Standup 2024-01-15
## Observations
- [attendees] Paul
- [attendees] Sarah
- [decisions] Ship v2 by Friday
- [action_items] Paul to review PR #42
- [blockers] Waiting on API credentials
```
+175
View File
@@ -0,0 +1,175 @@
# Per-Project Local/Cloud Routing
## Context
basic-memory's cloud/local mode is currently a global toggle (`cloud_mode: bool`). When enabled, ALL projects route through the cloud proxy via OAuth. This is too coarse — users should be able to keep some projects local and route others through cloud.
The cloud API already supports API key auth (`bmc_`-prefixed keys, `POST /api/keys` to create, `HybridTokenVerifier` routes them automatically). API keys are per-tenant (account-level), not per-project — there are no per-project permissions in the cloud yet.
**Goal**: Users can set each project to `local` or `cloud` mode. Local projects use the existing ASGI in-process transport. Cloud projects use the cloud API with a single account-level API key. No OAuth dance needed for cloud project access.
## UX Flow
**Option A — Create key in web app:**
1. User creates API key in cloud web app (already supported)
2. Copies the key
3. Runs `bm cloud set-key bmc_abc123...` → saves to config.json
**Option B — Create key via CLI:**
1. User is already logged in via OAuth (`bm cloud login`)
2. Runs `bm cloud create-key "my-laptop"` → calls `POST /api/keys` with JWT auth → gets key back → saves to config.json
3. OAuth login is no longer needed for day-to-day use — the API key handles auth
**Setting project mode:**
```bash
bm project set-cloud research # route "research" project through cloud
bm project set-local research # revert to local
bm project list # shows mode column (local/cloud)
```
## Implementation Plan
### Step 1: Config model changes
**File: `src/basic_memory/config.py`**
- Add `ProjectMode` enum: `LOCAL = "local"`, `CLOUD = "cloud"`
- Add `ProjectConfigEntry` Pydantic model: `path: str`, `mode: ProjectMode = LOCAL`
- Evolve `BasicMemoryConfig.projects` from `Dict[str, str]` to `Dict[str, ProjectConfigEntry]`
- Add `model_validator(mode="before")` to auto-migrate old `{"name": "/path"}` format to `{"name": {"path": "/path", "mode": "local"}}`
- Add `cloud_api_key: Optional[str] = None` field to `BasicMemoryConfig` (account-level, not per-project)
- Update `ProjectConfig` dataclass to carry `mode` from config entry
- Add helpers: `get_project_entry(name)`, `get_project_mode(name)`
- Keep global `cloud_mode` as deprecated fallback
- Update all code that reads `config.projects` as `Dict[str, str]` to handle `ProjectConfigEntry`
### Step 2: Client routing
**File: `src/basic_memory/mcp/async_client.py`**
- Add optional `project_name: Optional[str] = None` parameter to `get_client()`
- Routing logic (priority order):
1. Factory injection (`_client_factory`) — unchanged
2. Force-local (`_force_local_mode()`) — unchanged
3. **New**: If `project_name` provided and project's mode is `CLOUD` → HTTP client with `cloud_api_key` as Bearer token, hitting `cloud_host/proxy`
4. Global `cloud_mode_enabled` fallback — existing OAuth flow (deprecated)
5. Default: local ASGI transport
- Error if cloud project but no `cloud_api_key` in config — actionable message pointing to `bm cloud set-key` or `bm cloud create-key`
### Step 3: Project-aware client helper
**File: `src/basic_memory/mcp/project_context.py`**
- Add `get_project_client(project, context)` async context manager
- Combines `resolve_project_parameter()` (config-only, no network) + `get_client(project_name=resolved)` + `get_active_project(client, resolved, context)`
- Returns `(client, active_project)` tuple
- Solves bootstrap problem: resolve project name first, create correct client, then validate
### Step 4: Simplify ProjectResolver
**File: `src/basic_memory/project_resolver.py`**
- Remove global `cloud_mode` parameter — routing mode is orthogonal to project resolution
- Resolution becomes purely: constrained env var → explicit param → default project
- Update `resolve_project_parameter()` in `project_context.py` to drop `cloud_mode` param
### Step 5: Update MCP tools
**Files: `src/basic_memory/mcp/tools/*.py` (~15 files)**
Mechanical change per tool:
```python
# Before
async with get_client() as client:
active_project = await get_active_project(client, project, context)
# After
async with get_project_client(project, context) as (client, active_project):
```
Special handling for `recent_activity.py` discovery mode: iterate projects, create per-project client for each.
### Step 6: Sync coordinator
**Files: `src/basic_memory/sync/coordinator.py`, `src/basic_memory/mcp/container.py`**
- Filter file watchers to local-mode projects only
- Cloud projects skip sync
### Step 7: CLI commands
**File: `src/basic_memory/cli/commands/cloud/core_commands.py`**
- `bm cloud set-key <api-key>` — saves API key to config.json
- `bm cloud create-key <name>` — calls `POST {cloud_host}/api/keys` using existing JWT auth (from `make_api_request`), saves returned key to config. Uses existing `api_client.py:make_api_request()` for the authenticated call.
**File: `src/basic_memory/cli/commands/project.py`**
- `bm project set-cloud <name>` — sets project mode to cloud (validates API key exists in config)
- `bm project set-local <name>` — reverts project to local mode
- Extend `bm project list` / `bm project info` to show mode column
### Step 8: RuntimeMode simplification
**File: `src/basic_memory/runtime.py`**
- `resolve_runtime_mode()` drops `cloud_mode_enabled` parameter
- Simplifies to: TEST if test env, otherwise LOCAL
- `RuntimeMode.CLOUD` kept for backward compat but not used in global resolution
### Step 9: Tests
- Config: migration from old format, round-trip serialization, `get_project_mode()`
- `get_client()`: local project → ASGI, cloud project → HTTP+API key, missing key → error
- `get_project_client()`: resolve + route combined
- MCP tools: representative sample with new helper
- Sync: cloud projects skipped, local projects synced
- CLI: `set-key`, `create-key`, `set-cloud`, `set-local`
## Key Files
| File | Change |
|------|--------|
| `src/basic_memory/config.py` | `ProjectMode`, `ProjectConfigEntry`, migration, `cloud_api_key` field |
| `src/basic_memory/mcp/async_client.py` | `get_client(project_name=)` per-project routing |
| `src/basic_memory/mcp/project_context.py` | `get_project_client()` helper |
| `src/basic_memory/project_resolver.py` | Remove global `cloud_mode` concern |
| `src/basic_memory/mcp/tools/*.py` | Mechanical swap to `get_project_client()` |
| `src/basic_memory/sync/coordinator.py` | Filter to local-mode projects |
| `src/basic_memory/mcp/container.py` | Update should_sync logic |
| `src/basic_memory/cli/commands/cloud/core_commands.py` | `set-key`, `create-key` commands |
| `src/basic_memory/cli/commands/project.py` | `set-cloud`, `set-local` commands |
| `src/basic_memory/runtime.py` | Drop cloud_mode from global resolution |
## Config Example
```json
{
"projects": {
"personal": {"path": "/Users/me/notes", "mode": "local"},
"research": {"path": "/Users/me/research", "mode": "cloud"}
},
"cloud_api_key": "bmc_abc123...",
"cloud_host": "https://cloud.basicmemory.com",
"default_project": "personal"
}
```
## Edge Cases
| Case | Handling |
|------|----------|
| No API key + cloud project | `get_client()` raises error: "Run `bm cloud set-key` first" |
| Old config format loaded | `model_validator` auto-migrates `Dict[str,str]` to new format |
| Default project is cloud | Works — resolver returns name, routing uses API key |
| Global `cloud_mode=true` (legacy) | Deprecated fallback still works via OAuth |
| Factory-injected client (cloud app) | Factory takes priority, unaffected |
| `--local` CLI flag on cloud project | Force-local override still works |
## Verification
1. `just fast-check` — lint/format/typecheck + impacted tests
2. `just test` — full suite (SQLite + Postgres)
3. Manual: `bm cloud set-key bmc_...`, `bm project set-cloud test`, run MCP tools against it
4. Manual: verify local projects work unchanged
5. Manual: `bm project list` shows mode column
+49 -1
View File
@@ -1038,6 +1038,35 @@ recent_decisions = await search_notes(
)
```
**Structured frontmatter filters**:
```python
# Filter by tags and status
results = await search_notes(
query="authentication",
tags=["security"],
status="in-progress",
project="main"
)
# Complex metadata filters (supports $in, $gt, $gte, $lt, $lte, $between)
results = await search_notes(
query="api design",
metadata_filters={
"type": "spec",
"priority": {"$in": ["high", "critical"]},
"tags": ["architecture"]
},
project="main"
)
# Metadata-only search
results = await search_by_metadata(
filters={"type": "spec", "status": "in-progress"},
project="main"
)
```
### Search Types
**Text search (default)**:
@@ -2861,7 +2890,7 @@ contents = await list_directory(
### Search & Discovery
**search_notes(query, page, page_size, search_type, types, entity_types, after_date, project)**
**search_notes(query, page, page_size, search_type, types, entity_types, after_date, metadata_filters, tags, status, project)**
- Search across knowledge base
- Parameters:
- `query` (required): Search query
@@ -2871,6 +2900,9 @@ contents = await list_directory(
- `types` (optional): Entity type filter
- `entity_types` (optional): Observation category filter
- `after_date` (optional): Date filter (ISO format)
- `metadata_filters` (optional): Structured frontmatter filters (dict)
- `tags` (optional): Frontmatter tags filter (list)
- `status` (optional): Frontmatter status filter (string)
- `project` (required unless default_project_mode): Target project
- Returns: Matching entities with scores
- Example:
@@ -2883,6 +2915,22 @@ results = await search_notes(
)
```
**search_by_metadata(filters, limit, offset, project)**
- Metadata-only search using structured frontmatter
- Parameters:
- `filters` (required): Dict of field -> value (supports $in, $gt/$gte/$lt/$lte, $between)
- `limit` (optional): Max results (default: 20)
- `offset` (optional): Pagination offset (default: 0)
- `project` (required unless default_project_mode): Target project
- Returns: Matching entities
- Example:
```python
results = await search_by_metadata(
filters={"type": "spec", "status": "in-progress"},
project="main"
)
```
### Project Management
**list_memory_projects()**
+102 -7
View File
@@ -17,6 +17,8 @@ Before using Basic Memory Cloud, you need:
- **Active Subscription**: An active Basic Memory Cloud subscription is required to access cloud features
- **Subscribe**: Visit [https://basicmemory.com/subscribe](https://basicmemory.com/subscribe) to sign up
- **Optional**: Cloud is optional. Local-first open-source usage continues without cloud.
- **OSS Discount**: Use code `{{OSS_DISCOUNT_CODE}}` for 20% off for 3 months.
If you attempt to log in without an active subscription, you'll receive a "Subscription Required" error with a link to subscribe.
@@ -81,6 +83,7 @@ bm cloud login
4. Validates your subscription status
**Result:** All `bm project`, `bm tools` commands now work with cloud.
Apply OSS discount code `{{OSS_DISCOUNT_CODE}}` during checkout to receive 20% off for 3 months.
### 2. Set Up Sync
@@ -433,20 +436,97 @@ bm project bisync --name work
**Result:** Fine-grained control over what syncs.
## Per-Project Cloud Routing (API Key)
Instead of toggling global cloud mode, you can route individual projects through the cloud using an API key. This lets you keep some projects local while others route through the cloud.
### Setting Up API Key Auth
**Option A: Create a key in the web app, then save it locally:**
```bash
bm cloud set-key bmc_abc123...
```
**Option B: Create a key via CLI (requires OAuth login first):**
```bash
bm cloud login # One-time OAuth login
bm cloud create-key "my-laptop" # Creates key and saves it locally
```
The API key is account-level — it grants access to all your cloud projects. It's stored in `~/.basic-memory/config.json` as `cloud_api_key`.
### Setting Project Modes
```bash
# Route a project through cloud
bm project set-cloud research
# Revert to local mode
bm project set-local research
# View project modes
bm project list
```
**What happens:**
- `set-cloud`: validates the API key exists, then sets the project mode to `cloud` in config
- `set-local`: reverts the project to local mode (removes the mode entry from config)
- MCP tools and CLI commands for that project will route to `cloud_host/proxy` with the API key as Bearer token
### How It Works
When an MCP tool or CLI command runs for a cloud-mode project:
1. `get_client(project_name="research")` checks the project's mode in config
2. If mode is `cloud`, creates an HTTP client pointed at `cloud_host/proxy` with `Authorization: Bearer bmc_...`
3. If mode is `local` (default), uses the in-process ASGI transport as usual
**Routing priority** (highest to lowest):
1. Factory injection (cloud app, tests)
2. `BASIC_MEMORY_FORCE_LOCAL` env var
3. Per-project cloud mode (API key)
4. Global cloud mode (OAuth — deprecated fallback)
5. Local ASGI transport (default)
### Configuration Example
```json
{
"projects": {
"personal": "/Users/me/notes",
"research": "/Users/me/research"
},
"project_modes": {
"research": "cloud"
},
"cloud_api_key": "bmc_abc123...",
"cloud_host": "https://cloud.basicmemory.com",
"default_project": "personal"
}
```
In this example, `personal` stays local and `research` routes through cloud. Projects not listed in `project_modes` default to local.
### Sync Behavior
Cloud-mode projects are automatically skipped during local file sync (background sync and file watching). Their files live on the cloud instance, not locally.
## Disable Cloud Mode
Return to local mode:
Return to local mode (global):
```bash
bm cloud logout
```
**What this does:**
1. Disables cloud mode in config
2. All commands now work locally
1. Disables global cloud mode in config
2. All commands now work locally (unless individual projects are set to cloud via `set-cloud`)
3. Auth token remains (can re-enable with login)
**Result:** All `bm` commands work with local projects again.
**Result:** All `bm` commands work with local projects again. Per-project cloud routing via API key continues to work independently of global cloud mode.
## Filter Configuration
@@ -656,9 +736,17 @@ If instance is down, wait a few minutes and retry.
### Cloud Mode Management
```bash
bm cloud login # Authenticate and enable cloud mode
bm cloud logout # Disable cloud mode
bm cloud login # Authenticate and enable global cloud mode (OAuth)
bm cloud logout # Disable global cloud mode
bm cloud status # Check cloud mode and instance health
bm cloud promo --off # Disable CLI cloud promo notices
```
### API Key Management
```bash
bm cloud set-key <key> # Save a cloud API key (bmc_ prefixed)
bm cloud create-key <name> # Create API key via cloud API (requires OAuth login)
```
### Setup
@@ -672,13 +760,20 @@ bm cloud setup # Install rclone and configure credentials
When cloud mode is enabled:
```bash
bm project list # List cloud projects
bm project list # List projects with mode column
bm project add <name> # Create cloud project (no sync)
bm project add <name> --local-path <path> # Create with local sync
bm project sync-setup <name> <path> # Add sync to existing project
bm project rm <name> # Delete project
```
### Per-Project Routing
```bash
bm project set-cloud <name> # Route project through cloud (requires API key)
bm project set-local <name> # Revert project to local mode
```
### File Synchronization
```bash
+91
View File
@@ -0,0 +1,91 @@
# Cloud Semantic Search Value (Customer-Facing Technical Story)
This document explains why teams should buy cloud semantic search even when local search exists.
## Core Promise
Markdown files remain the source of truth in both local and cloud modes.
- Files are portable.
- Search indexes are derived and rebuildable.
- You never get locked into proprietary document storage.
## The Customer Problem
Teams paying for cloud are usually not optimizing for "can this run locally." They are optimizing for:
- finding the right note the first time,
- keeping retrieval quality high as note volume grows,
- avoiding search slowdowns while content is actively changing,
- getting consistent results across users, agents, and sessions.
## Why Cloud Is the Aspirin
Cloud semantic search is the immediate pain reliever because it fixes the problems users feel right now.
### 1) Better hit rate on real queries
Cloud uses stronger managed embeddings than the default local model, which improves semantic recall for paraphrases and vague questions.
Customer outcome:
- fewer "I know this exists but search missed it" moments,
- less query rewording,
- faster time to answer.
### 2) Better behavior under active workloads
Cloud indexing runs out of band in workers, so indexing does not compete with interactive read/write traffic.
Customer outcome:
- stable search responsiveness during heavy updates,
- fresher semantic results shortly after edits,
- less user-visible performance variance.
### 3) Better consistency for shared knowledge
Cloud retrieval runs against a centralized tenant index, so teams and agents resolve against the same semantic state.
Customer outcome:
- fewer "works on my machine" search differences,
- more predictable agent behavior across environments,
- easier cross-user collaboration on large knowledge bases.
### 4) Better quality at higher scale
With Postgres + `pgvector` per tenant, cloud can sustain larger note collections and higher query volumes than typical local setups.
Customer outcome:
- confidence as repositories grow to tens of thousands of notes,
- less need for user-side tuning,
- fewer quality regressions as usage increases.
## Local Is the Vitamin
Local semantic search still matters and should stay strong.
- offline use,
- privacy-first operation,
- no cloud dependency,
- user-controlled runtime.
It compounds long-term ownership and resilience, but does not remove the immediate pain points cloud solves for teams at scale.
## Recommended Messaging
One-liner:
"Cloud semantic search is the aspirin: it fixes retrieval quality and performance pain now. Local semantic search is the vitamin: it builds long-term control and resilience."
Long form:
"Basic Memory keeps markdown as the source of truth everywhere. Local gives privacy and offline control. Cloud adds immediate, measurable improvements in search quality, consistency, and responsiveness for teams and agents running at scale."
## Packaging Guidance
- Base: local FTS plus optional local semantic search.
- Cloud value: higher semantic quality, stable performance under load, and consistent team-wide retrieval.
- Keep interfaces pluggable (`EmbeddingProvider`, vector backend protocol) so implementation can evolve without changing user workflows.
+130
View File
@@ -0,0 +1,130 @@
# MCP UI Bakeoff - Instructions & Test Plan
Last updated: 2026-02-02
## Scope
Compare three presentation paths for Basic Memory MCP tools:
1. **ToolUI (React)** via MCP App resources.
2. **MCPUI Python SDK** embedded UI resources (legacy host path).
3. **ASCII/ANSI** output for TUI clients.
This doc is the running instruction set and test plan. Update as implementation progresses.
---
## Prerequisites
- Repo: `basic-memory` (worktree: `basic-memory-mcp-ui-poc`)
- Node for toolui build (already used for POC)
- Python 3.12+ with `uv`
Optional (for MCPUI Python SDK path):
- Local repo: `/Users/phernandez/dev/mcp-ui`
- Install the server SDK into the Basic Memory venv:
- `uv pip install -e /Users/phernandez/dev/mcp-ui/sdks/python/server`
---
## Build / Refresh Steps
### ToolUI React bundle
```bash
cd ui/tool-ui-react
npm install
npm run build
```
This regenerates:
- `src/basic_memory/mcp/ui/html/search-results-tool-ui.html`
- `src/basic_memory/mcp/ui/html/note-preview-tool-ui.html`
---
## How to Run the MCP Server
```bash
basic-memory mcp --transport stdio
```
Optional to pick UI variant for MCP App resources:
```bash
export BASIC_MEMORY_MCP_UI_VARIANT=tool-ui # or vanilla | mcp-ui
```
---
## Test Cases
### 1) MCP App Resource UI (toolui / vanilla / mcpui)
Tools:
- `search_notes`
- `read_note`
Expect:
- Tool meta points to `ui://basic-memory/search-results` and `ui://basic-memory/note-preview`
- Resource content differs by `BASIC_MEMORY_MCP_UI_VARIANT`
- Variantspecific URIs also available:
- `ui://basic-memory/search-results/vanilla`
- `ui://basic-memory/search-results/tool-ui`
- `ui://basic-memory/search-results/mcp-ui`
- `ui://basic-memory/note-preview/vanilla`
- `ui://basic-memory/note-preview/tool-ui`
- `ui://basic-memory/note-preview/mcp-ui`
Manual check:
- Trigger tool in MCPAppcapable host and confirm UI renders.
---
### 2) ASCII / ANSI TUI Output
Tools:
- `search_notes(output_format="ascii" | "ansi")`
- `read_note(output_format="ascii" | "ansi")`
Expect:
- ASCII table for search, header + content preview for note.
- ANSI variants include color escape codes.
Automated:
- `uv run pytest test-int/mcp/test_output_format_ascii_integration.py`
---
### 3) MCPUI Python SDK (embedded UI resource)
Tools (embedded resource responses):
- `search_notes_ui` (MCPUI SDK)
- `read_note_ui` (MCPUI SDK)
Expected output:
- Tool response content contains an EmbeddedResource (`type: "resource"`)
- `mimeType` is `text/html`
- `_meta` includes:
- `mcpui.dev/ui-preferred-frame-size`
- `mcpui.dev/ui-initial-render-data`
Manual check:
- Render tool responses using `UIResourceRenderer` (legacy host flow).
Automated (if SDK installed):
- `uv run pytest test-int/mcp/test_ui_sdk_integration.py`
---
## Bakeoff Notes Template
Fill in after running:
- ToolUI (React): __
- MCPUI SDK (embedded): __
- ASCII/ANSI: __
Decision + rationale: __
+265
View File
@@ -0,0 +1,265 @@
# Semantic Search
This guide covers Basic Memory's optional semantic (vector) search feature, which adds meaning-based retrieval alongside the existing full-text search.
## Overview
Basic Memory's default search uses full-text search (FTS) — keyword matching with boolean operators. Semantic search adds vector embeddings that capture the *meaning* of your content, enabling:
- **Paraphrase matching**: Find "authentication flow" when searching for "login process"
- **Conceptual queries**: Search for "ways to improve performance" and find notes about caching, indexing, and optimization
- **Hybrid retrieval**: Combine the precision of keyword search with the recall of semantic similarity
Semantic search is **opt-in** — existing behavior is completely unchanged unless you enable it. It works on both SQLite (local) and Postgres (cloud) backends.
## Installation
Semantic search dependencies (fastembed, sqlite-vec, openai) are **optional extras** — they are not installed with the base `basic-memory` package. Install them with:
```bash
pip install 'basic-memory[semantic]'
```
This keeps the base install lightweight and avoids platform-specific issues with ONNX Runtime wheels.
### Platform Compatibility
| Platform | FastEmbed (local) | OpenAI (API) |
|---|---|---|
| macOS ARM64 (Apple Silicon) | Yes | Yes |
| macOS x86_64 (Intel Mac) | No — see workaround below | Yes |
| Linux x86_64 | Yes | Yes |
| Linux ARM64 | Yes | Yes |
| Windows x86_64 | Yes | Yes |
#### Intel Mac Workaround
The default FastEmbed provider uses ONNX Runtime, which dropped Intel Mac (x86_64) wheels starting in v1.24. Intel Mac users have two options:
**Option 1: Use OpenAI embeddings (recommended)**
Install only the OpenAI dependency manually — no ONNX Runtime or FastEmbed needed:
```bash
pip install openai sqlite-vec
export BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED=true
export BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER=openai
export OPENAI_API_KEY=sk-...
```
**Option 2: Pin an older ONNX Runtime**
FastEmbed's ONNX Runtime dependency is unpinned, so you can constrain it to an older version that still ships Intel Mac wheels by passing both requirements in the same install command:
```bash
pip install 'basic-memory[semantic]' 'onnxruntime<1.24'
```
## Quick Start
1. Install semantic extras:
```bash
pip install 'basic-memory[semantic]'
```
2. Enable semantic search:
```bash
export BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED=true
```
3. Build vector embeddings for your existing content:
```bash
bm reindex --embeddings
```
4. Search using semantic modes:
```python
# Pure vector similarity
search_notes("login process", search_type="vector")
# Hybrid: combines FTS precision with vector recall (recommended)
search_notes("login process", search_type="hybrid")
# Traditional full-text search (still the default)
search_notes("login process", search_type="text")
```
## Configuration Reference
All settings are fields on `BasicMemoryConfig` and can be set via environment variables (prefixed with `BASIC_MEMORY_`).
| Config Field | Env Var | Default | Description |
|---|---|---|---|
| `semantic_search_enabled` | `BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED` | `false` | Enable semantic search. Required before vector/hybrid modes work. |
| `semantic_embedding_provider` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER` | `"fastembed"` | Embedding provider: `"fastembed"` (local) or `"openai"` (API). |
| `semantic_embedding_model` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_MODEL` | `"bge-small-en-v1.5"` | Model identifier. Auto-adjusted per provider if left at default. |
| `semantic_embedding_dimensions` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_DIMENSIONS` | Auto-detected | Vector dimensions. 384 for FastEmbed, 1536 for OpenAI. Override only if using a non-default model. |
| `semantic_embedding_batch_size` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_BATCH_SIZE` | `64` | Number of texts to embed per batch. |
| `semantic_vector_k` | `BASIC_MEMORY_SEMANTIC_VECTOR_K` | `100` | Candidate count for vector nearest-neighbour retrieval. Higher values improve recall at the cost of latency. |
## Embedding Providers
### FastEmbed (default)
FastEmbed runs entirely locally using ONNX models — no API key, no network calls, no cost.
- **Model**: `BAAI/bge-small-en-v1.5`
- **Dimensions**: 384
- **Tradeoff**: Smaller model, fast inference, good quality for most use cases
```bash
# Install semantic extras and enable
pip install 'basic-memory[semantic]'
export BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED=true
```
### OpenAI
Uses OpenAI's embeddings API for higher-dimensional vectors. Requires an API key.
- **Model**: `text-embedding-3-small`
- **Dimensions**: 1536
- **Tradeoff**: Higher quality embeddings, requires API calls and an OpenAI key
```bash
export BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED=true
export BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER=openai
export OPENAI_API_KEY=sk-...
```
When switching from FastEmbed to OpenAI (or vice versa), you must rebuild embeddings since the vector dimensions differ:
```bash
bm reindex --embeddings
```
## Search Modes
### `text` (default)
Full-text keyword search using FTS5 (SQLite) or tsvector (Postgres). Supports boolean operators (`AND`, `OR`, `NOT`), phrase matching, and prefix wildcards.
```python
search_notes("project AND planning", search_type="text")
```
This is the existing default and does not require semantic search to be enabled.
### `vector`
Pure semantic similarity search. Embeds your query and finds the nearest content vectors. Good for conceptual or paraphrase queries where exact keywords may not appear in the content.
```python
search_notes("how to speed up the app", search_type="vector")
```
Returns results ranked by cosine similarity. Individual observations and relations surface as first-class results, not collapsed into parent entities.
### `hybrid`
Combines FTS and vector results using reciprocal rank fusion (RRF). This is generally the best mode when you want both keyword precision and semantic recall.
```python
search_notes("authentication security", search_type="hybrid")
```
RRF merges the two ranked lists so that items appearing in both get a score boost, while items found by only one method still appear.
### When to Use Which
| Mode | Best For |
|---|---|
| `text` | Exact keyword matching, boolean queries, tag/category searches |
| `vector` | Conceptual queries, paraphrase matching, exploratory searches |
| `hybrid` | General-purpose search combining precision and recall |
## The Reindex Command
The `bm reindex` command rebuilds search indexes without dropping the database.
```bash
# Rebuild everything (FTS + embeddings if semantic is enabled)
bm reindex
# Only rebuild vector embeddings
bm reindex --embeddings
# Only rebuild the full-text search index
bm reindex --search
# Target a specific project
bm reindex -p my-project
```
### When You Need to Reindex
- **First enable**: After turning on `semantic_search_enabled` for the first time
- **Provider change**: After switching between `fastembed` and `openai`
- **Model change**: After changing `semantic_embedding_model`
- **Dimension change**: After changing `semantic_embedding_dimensions`
The reindex command shows progress with embedded/skipped/error counts:
```
Project: main
Building vector embeddings...
✓ Embeddings complete: 142 entities embedded, 0 skipped, 0 errors
Reindex complete!
```
## How It Works
### Chunking
Each entity in the search index is split into semantic chunks before embedding:
- **Headers**: Markdown headers (`#`, `##`, etc.) start new chunks
- **Bullets**: Each bullet item (`-`, `*`) becomes its own chunk for granular fact retrieval
- **Prose sections**: Non-bullet text is merged up to ~900 characters per chunk
- **Long sections**: Oversized content is split with ~120 character overlap to preserve context at boundaries
Each search index item type (entity, observation, relation) is chunked independently, so observations and relations are embeddable as discrete facts.
### Deduplication
Each chunk has a `source_hash` (SHA-256 of the chunk text). On re-sync, unchanged chunks skip re-embedding entirely. This makes incremental updates fast — only modified content triggers API calls or model inference.
### Hybrid Fusion
Hybrid search uses reciprocal rank fusion (RRF) to merge FTS and vector results:
1. Run FTS search to get keyword-ranked results
2. Run vector search to get similarity-ranked results
3. For each result, compute: `score = 1/(k + fts_rank) + 1/(k + vector_rank)` where `k = 60`
4. Sort by fused score
Items found by both methods get a natural score boost. Items found by only one method still appear but rank lower.
### Observation-Level Results
Vector and hybrid modes return individual observations and relations as first-class search results, not just parent entities. This means a search for "water temperature for brewing" can surface the specific observation about 205°F without returning the entire "Coffee Brewing Methods" entity.
## Database Backends
### SQLite (local)
- **Vector storage**: [sqlite-vec](https://github.com/asg017/sqlite-vec) virtual table
- **Table creation**: At runtime when semantic search is first used — no migration needed
- **Embedding table**: `search_vector_embeddings` using `vec0(embedding float[N])` where N is the configured dimensions
- **Chunk metadata**: `search_vector_chunks` table stores chunk text, keys, and source hashes
The sqlite-vec extension is loaded per-connection. Vector tables are created lazily on first use.
### Postgres (cloud)
- **Vector storage**: [pgvector](https://github.com/pgvector/pgvector) with HNSW indexing
- **Chunk metadata table**: Created via Alembic migration (`search_vector_chunks` with `BIGSERIAL` primary key)
- **Embedding table**: `search_vector_embeddings` created at runtime (dimension-dependent, same pattern as SQLite)
- **Index**: HNSW index on the embedding column for fast approximate nearest-neighbour queries
The Alembic migration creates the dimension-independent chunks table. The embeddings table and HNSW index are deferred to runtime because they depend on the configured vector dimensions.
+365
View File
@@ -0,0 +1,365 @@
# SPEC-SCHEMA-IMPL: Schema System Implementation Plan
**Status:** Draft
**Created:** 2025-02-06
**Branch:** `feature/schema-system`
**Depends on:** [SPEC-SCHEMA](SPEC-SCHEMA.md)
## Overview
Implementation plan for the Basic Memory Schema System. The system is entirely programmatic —
no LLM agent runtime or API key required. The LLM already in the user's session (Claude Code,
Claude Desktop, etc.) provides the intelligence layer by reading schema notes via existing
MCP tools.
## Architecture
```
┌─────────────────────────────────────────────────┐
│ Entry Points │
│ CLI (bm schema ...) │ MCP (schema_validate) │
└──────────┬────────────┴──────────┬──────────────┘
│ │
▼ ▼
┌─────────────────────────────────────────────────┐
│ Schema Service Layer │
│ resolve_schema · validate · infer · diff │
└──────────┬────────────────────────┬──────────────┘
│ │
▼ ▼
┌──────────────────────┐ ┌────────────────────────┐
│ Picoschema Parser │ │ Note/Entity Access │
│ YAML → SchemaModel │ │ (existing repository) │
└──────────────────────┘ └────────────────────────┘
```
No new database tables. Schemas are notes with `type: schema` — they're already indexed.
Validation reads observations and relations from existing data.
## Components
### 1. Picoschema Parser
**Location:** `src/basic_memory/schema/parser.py`
Parses Picoschema YAML into an internal representation.
```python
@dataclass
class SchemaField:
name: str
type: str # string, integer, number, boolean, any, or EntityName
required: bool # True unless field name ends with ?
is_array: bool # True if (array) notation
is_enum: bool # True if (enum) notation
enum_values: list[str] # Populated for enums
description: str | None # Text after comma
is_entity_ref: bool # True if type is capitalized (entity reference)
children: list[SchemaField] # For (object) types
@dataclass
class SchemaDefinition:
entity: str # The entity type this schema describes
version: int # Schema version
fields: list[SchemaField] # Parsed fields
validation_mode: str # "warn" | "strict" | "off"
def parse_picoschema(yaml_dict: dict) -> list[SchemaField]:
"""Parse a Picoschema YAML dict into a list of SchemaField objects."""
def parse_schema_note(frontmatter: dict) -> SchemaDefinition:
"""Parse a full schema note's frontmatter into a SchemaDefinition."""
```
**Input/Output:**
```yaml
# Input (YAML dict from frontmatter)
schema:
name: string, full name
role?: string, job title
works_at?: Organization, employer
expertise?(array): string, areas of knowledge
```
```python
# Output
[
SchemaField(name="name", type="string", required=True, description="full name", ...),
SchemaField(name="role", type="string", required=False, description="job title", ...),
SchemaField(name="works_at", type="Organization", required=False, is_entity_ref=True, ...),
SchemaField(name="expertise", type="string", required=False, is_array=True, ...),
]
```
### 2. Schema Resolver
**Location:** `src/basic_memory/schema/resolver.py`
Finds the applicable schema for a note using the resolution order.
```python
async def resolve_schema(
note_frontmatter: dict,
search_fn: Callable, # injected search capability
) -> SchemaDefinition | None:
"""Resolve schema for a note.
Resolution order:
1. Inline schema (frontmatter['schema'] is a dict)
2. Explicit reference (frontmatter['schema'] is a string)
3. Implicit by type (frontmatter['type'] → schema note with matching entity)
4. No schema (returns None)
"""
```
### 3. Schema Validator
**Location:** `src/basic_memory/schema/validator.py`
Validates a note's observations and relations against a resolved schema.
```python
@dataclass
class FieldResult:
field: SchemaField
status: str # "present" | "missing" | "type_mismatch"
values: list[str] # Matched observation values or relation targets
message: str | None # Human-readable detail
@dataclass
class ValidationResult:
note_identifier: str
schema_entity: str
passed: bool # True if no errors (warnings are OK)
field_results: list[FieldResult]
unmatched_observations: dict[str, int] # category → count
unmatched_relations: list[str] # relation types not in schema
warnings: list[str]
errors: list[str]
async def validate_note(
note: Note,
schema: SchemaDefinition,
) -> ValidationResult:
"""Validate a note against a schema definition.
Mapping rules:
- field: string → observation [field] exists
- field?(array): type → multiple [field] observations
- field?: EntityType → relation 'field [[...]]' exists
- field?(enum): [v] → observation [field] value ∈ enum values
"""
```
### 4. Schema Inference Engine
**Location:** `src/basic_memory/schema/inference.py`
Analyzes notes of a given type and suggests a schema based on usage frequency.
```python
@dataclass
class FieldFrequency:
name: str
source: str # "observation" | "relation"
count: int # notes containing this field
total: int # total notes analyzed
percentage: float
sample_values: list[str] # representative values
is_array: bool # True if typically appears multiple times per note
target_type: str | None # For relations, the most common target entity type
@dataclass
class InferenceResult:
entity_type: str
notes_analyzed: int
field_frequencies: list[FieldFrequency]
suggested_schema: dict # Ready-to-use Picoschema YAML dict
suggested_required: list[str]
suggested_optional: list[str]
excluded: list[str] # Below threshold
async def infer_schema(
entity_type: str,
notes: list[Note],
required_threshold: float = 0.95, # 95%+ = required
optional_threshold: float = 0.25, # 25%+ = optional
) -> InferenceResult:
"""Analyze notes and suggest a Picoschema definition."""
```
### 5. Schema Diff
**Location:** `src/basic_memory/schema/diff.py`
Compares current note usage against an existing schema definition.
```python
@dataclass
class SchemaDrift:
new_fields: list[FieldFrequency] # Fields not in schema but common in notes
dropped_fields: list[FieldFrequency] # Fields in schema but rare in notes
cardinality_changes: list[str] # one → many or many → one
type_mismatches: list[str] # observation values don't match declared type
async def diff_schema(
schema: SchemaDefinition,
notes: list[Note],
) -> SchemaDrift:
"""Compare a schema against actual note usage to detect drift."""
```
## Entry Points
### CLI Commands
**Location:** `src/basic_memory/cli/schema.py`
```python
import typer
schema_app = typer.Typer(name="schema", help="Schema management commands")
@schema_app.command()
async def validate(
target: str = typer.Argument(None, help="Note path or entity type"),
strict: bool = typer.Option(False, help="Override to strict mode"),
):
"""Validate notes against their schemas."""
@schema_app.command()
async def infer(
entity_type: str = typer.Argument(..., help="Entity type to analyze"),
threshold: float = typer.Option(0.25, help="Minimum frequency for optional fields"),
save: bool = typer.Option(False, help="Save to schema/ directory"),
):
"""Infer schema from existing notes of a type."""
@schema_app.command()
async def diff(
entity_type: str = typer.Argument(..., help="Entity type to diff"),
):
"""Show drift between schema and actual usage."""
```
Registered as subcommand: `bm schema validate`, `bm schema infer`, `bm schema diff`.
### MCP Tools
**Location:** `src/basic_memory/mcp/tools/schema.py`
```python
@mcp_tool
async def schema_validate(
entity_type: str | None = None,
identifier: str | None = None,
project: str | None = None,
) -> str:
"""Validate notes against their resolved schema."""
@mcp_tool
async def schema_infer(
entity_type: str,
threshold: float = 0.25,
project: str | None = None,
) -> str:
"""Analyze existing notes and suggest a schema definition."""
```
### API Endpoints
**Location:** `src/basic_memory/api/schema_router.py`
```python
router = APIRouter(prefix="/schema", tags=["schema"])
@router.post("/validate")
async def validate_schema(...) -> ValidationReport: ...
@router.post("/infer")
async def infer_schema(...) -> InferenceResult: ...
@router.get("/diff/{entity_type}")
async def diff_schema(...) -> SchemaDrift: ...
```
MCP tools call these endpoints via the typed client pattern (consistent with existing
architecture).
## Implementation Phases
### Phase 1: Parser + Resolver
Build the foundation — can parse Picoschema and find schemas for notes.
**Deliverables:**
- `schema/parser.py` — Picoschema YAML → `SchemaDefinition`
- `schema/resolver.py` — Resolution order (inline → explicit ref → implicit by type → none)
- Unit tests for all Picoschema syntax variations
- Unit tests for resolution order
**No external dependencies.** Pure Python parsing of YAML dicts. Can develop and test
in isolation.
### Phase 2: Validator
Connect schemas to notes and produce validation results.
**Deliverables:**
- `schema/validator.py` — Validate note observations/relations against schema fields
- API endpoint: `POST /schema/validate`
- MCP tool: `schema_validate`
- CLI command: `bm schema validate`
- Integration tests with real notes and schemas
**Depends on:** Phase 1 (parser + resolver)
### Phase 3: Inference
Analyze existing notes to suggest schemas.
**Deliverables:**
- `schema/inference.py` — Frequency analysis across notes of a type
- API endpoint: `POST /schema/infer`
- MCP tool: `schema_infer`
- CLI command: `bm schema infer`
- Option to save inferred schema as a note via `write_note`
**Depends on:** Phase 1 (parser for output format)
### Phase 4: Diff
Compare schemas against current usage.
**Deliverables:**
- `schema/diff.py` — Drift detection between schema and actual notes
- API endpoint: `GET /schema/diff/{entity_type}`
- CLI command: `bm schema diff`
**Depends on:** Phase 1 (parser), Phase 3 (inference, for frequency analysis)
## Testing Strategy
- **Unit tests** (`tests/schema/`): Parser edge cases, resolution logic, validation mapping,
inference thresholds
- **Integration tests** (`test-int/schema/`): End-to-end with real markdown files, schema notes
on disk, CLI invocation
- Coverage target: 100% (consistent with project standard)
## What This Does NOT Include
- No new database tables or migrations
- No new markdown syntax (schemas validate existing observations/relations)
- No LLM agent runtime or API key management
- No hook integration (deferred)
- No schema composition/inheritance (deferred)
- No OWL/RDF export (deferred)
- No built-in templates (deferred)
+462
View File
@@ -0,0 +1,462 @@
# SPEC-SCHEMA: Basic Memory Schema System
**Status:** Draft
**Created:** 2025-02-06
**Branch:** `feature/schema-system`
## Summary
A schema system for Basic Memory that uses [Picoschema](https://genkit.dev/docs/dotprompt/)
syntax in YAML frontmatter. Schemas validate notes against their existing observation/relation
structure — no new data model, no migration, just a declarative lens over what's already there.
## Core Principles
1. **Schemas are just notes** — A schema is a note with `type: schema`, lives anywhere
2. **Use prior art** — Picoschema syntax in YAML frontmatter, no custom notation
3. **Validation maps to existing format** — Observations and relations, not a parallel data model
4. **Validation is soft** — Warnings by default, not blocking errors
5. **Inference over prescription** — Schemas describe reality, emerge from usage
6. **No built-in agent** — Programmatic core; the LLM already in the session provides intelligence
## Picoschema Syntax
Picoschema is a compact schema notation from Google's Dotprompt that fits naturally in YAML
frontmatter.
### Supported Types
| Type | Description |
|------|-------------|
| `string` | Text value |
| `integer` | Whole number |
| `number` | Decimal number |
| `boolean` | True/false |
| `any` | Any scalar type |
| `EntityName` | Reference to another entity (capitalized = entity reference) |
### Syntax Rules
```yaml
schema:
name: string, full name # required field with description
email?: string, contact email # ? = optional
role?: string, job title
works_at?: Organization, employer # capitalized type = entity reference
tags?(array): string, categories # array of type
status?(enum): [active, inactive] # enum with allowed values
metadata?(object): # nested object
updated_at?: string
source?: string
```
- `field: type` — required field
- `field?: type` — optional field
- `field(array): type` — array of values
- `field?(enum): [values]` — enumeration
- `field?(object):` — nested object with sub-fields
- `, description` — description after comma
- `EntityName` as type (capitalized) — reference to another entity
## Schema-to-Note Mapping
Schemas validate against the existing Basic Memory note format. No new syntax for note
authors to learn.
### Mapping Rules
| Schema Declaration | Grounded In | Example Match |
|--------------------|-------------|---------------|
| `field: string` | Observation `[field] value` | `- [name] Paul Graham` |
| `field?(array): string` | Multiple `[field]` observations | `- [expertise] Lisp` (×N) |
| `field?: EntityType` | Relation `field [[Target]]` | `- works_at [[Y Combinator]]` |
| `field?(array): EntityType` | Multiple `field` relations | `- authored [[Book]]` (×N) |
| `tags` | Frontmatter `tags` array | `tags: [startups, essays]` |
| `field?(enum): [values]` | Observation `[field] value` where value ∈ set | `- [status] active` |
### Key Insight
Schemas don't introduce a new way to store data. They describe the patterns already present
in observations and relations. A note doesn't have to change how it's written — the schema
just says "a good Person note has a `[name]` observation and a `works_at` relation."
## Schema Definition
### As a Dedicated Schema Note
```yaml
# schema/Person.md
---
title: Person
type: schema
entity: Person
version: 1
schema:
name: string, full name
email?: string, contact email
role?: string, job title
works_at?: Organization, employer
expertise?(array): string, areas of knowledge
settings:
validation: warn # warn | strict | off
---
# Person
A human individual in the knowledge graph.
Any documentation about this entity type goes here as prose.
```
Schema notes are regular Basic Memory notes. They show up in search, can have their own
observations and relations, and can be organized in any folder (though `schema/` is
the suggested convention).
### Inline Schema in a Note
Notes can carry their own schema directly:
```yaml
# meetings/2024-01-15-standup.md
---
title: Team Standup 2024-01-15
type: meeting
schema:
attendees(array): string, who was there
decisions(array): string, what was decided
action_items(array): string, follow-ups
blockers?(array): string, anything stuck
---
# Team Standup 2024-01-15
## Observations
- [attendees] Paul
- [attendees] Sarah
- [decisions] Ship v2 by Friday
- [action_items] Paul to review PR #42
- [blockers] Waiting on API credentials
```
Good for one-off structured notes or prototyping a schema before extracting it.
### Explicit Schema Reference
A note can reference a schema by entity name or permalink:
```yaml
# projects/basic-memory.md
---
title: Basic Memory
schema: SoftwareProject # by entity name
---
# research/llm-memory-patterns.md
---
title: LLM Memory Patterns
schema: schema/research-project # by permalink
---
```
Use cases:
- Note's `type` differs from the schema it should validate against
- Multiple schema variants exist for the same domain
- Applying structure to existing notes without changing their type
## Schema Resolution
When validating a note, schemas resolve in priority order:
```
1. Inline schema → schema: { ... } (dict in frontmatter)
2. Explicit ref → schema: Person (string in frontmatter)
3. Implicit by type → type: Person (lookup schema note with entity: Person)
4. No schema → no validation (perfectly fine)
```
```python
async def resolve_schema(note: Note) -> Schema | None:
schema_value = note.frontmatter.get('schema')
# 1. Inline schema (dict)
if isinstance(schema_value, dict):
return parse_picoschema(schema_value)
# 2. Explicit reference (string)
if isinstance(schema_value, str):
schema_note = await find_schema_note(schema_value)
if schema_note:
return parse_picoschema(schema_note.frontmatter['schema'])
# 3. Implicit by type
note_type = note.frontmatter.get('type')
if note_type:
results = await search_notes(f"type:schema entity:{note_type}")
if results:
return parse_picoschema(results[0].frontmatter['schema'])
# 4. No schema
return None
```
## Validation
### Modes
Configured in the schema's `settings.validation`:
| Mode | Behavior |
|------|----------|
| `off` | No validation |
| `warn` | Warnings in output, doesn't block (default) |
| `strict` | Errors that block sync, for CI/CD enforcement |
### Validation Output
For a note missing required fields:
```
$ bm schema validate people/ada-lovelace.md
⚠ Person schema validation:
- Missing required field: name (expected [name] observation)
- Missing optional field: role
- Missing optional field: works_at (no relation found)
Unmatched observations: [fact] ×2, [born] ×1
Unmatched relations: collaborated_with
```
"Unmatched" items are informational — observations and relations the schema doesn't cover.
They're valid. Schemas are a subset, not a straitjacket.
### Batch Validation
```
$ bm schema validate Person
Validating 30 notes against Person schema...
✓ people/paul-graham.md — all fields present
✓ people/rich-hickey.md — all fields present
⚠ people/ada-lovelace.md — missing: name
⚠ people/alan-kay.md — missing: name, role
✓ people/linus-torvalds.md — all fields present
...
Summary: 22/30 valid, 8 warnings, 0 errors
```
## Emerging Schemas
### The Problem with Traditional Schemas
Most schema systems require: define schema → create conforming content → fight the schema
when reality doesn't match. This is backwards. Knowledge grows organically.
### The Basic Memory Approach
```
Write notes freely → Patterns emerge → Crystallize into schema → Validate future notes
```
### Schema Inference
Generate schemas from existing notes by analyzing observation and relation frequency:
```
$ bm schema infer Person
Analyzing 30 notes with type: Person...
Observations found:
[name] 30/30 100% → name: string
[role] 27/30 90% → role?: string
[fact] 25/30 83% (generic — no single field)
[expertise] 18/30 60% → expertise?(array): string
[email] 8/30 27% → email?: string
[born] 6/30 20% (below threshold)
Relations found:
works_at 22/30 73% → works_at?: Organization
authored 11/30 37% → authored?(array): string
Suggested schema:
name: string, full name
role?: string, job title
expertise?(array): string, areas of knowledge
email?: string, contact email
works_at?: Organization, employer
Save to schema/Person.md? [y/n]
```
Frequency thresholds:
- 100% present → required field
- 25%+ present → optional field
- Below 25% → excluded from suggestion (but noted)
### Schema Drift Detection
Track how usage patterns shift over time:
```
$ bm schema diff Person
Schema drift detected:
+ expertise: now in 81% of notes (was 12%)
- department: dropped to 3% of notes
~ works_at: cardinality changed (one → many)
Update schema? [y/n/review]
```
## LLM Integration (AI Guidance)
No agent runtime or API key required. The LLM already in the session uses schemas as
context for note creation.
### Flow
1. User asks LLM to "write a note about Rich Hickey"
2. LLM determines `type: Person` is appropriate
3. LLM calls `search_notes("type:schema entity:Person")` → finds schema
4. LLM reads schema fields: required `name`, optional `role`, `works_at`, `expertise`
5. LLM calls `write_note` with observations and relations that satisfy the schema
The schema acts as a creation template. The LLM knows what a "complete" note looks like
without any custom agent infrastructure.
### MCP Tools
```python
@mcp_tool
async def schema_validate(
entity_type: str | None = None,
identifier: str | None = None,
project: str | None = None,
) -> ValidationReport:
"""Validate notes against their resolved schema.
Validates a specific note (by identifier) or all notes of a given type.
Returns warnings/errors based on the schema's validation mode.
"""
@mcp_tool
async def schema_infer(
entity_type: str,
threshold: float = 0.25,
project: str | None = None,
) -> SuggestedSchema:
"""Analyze existing notes and suggest a schema definition.
Examines observation categories and relation types across all notes
of the given type. Returns frequency analysis and suggested Picoschema.
"""
```
## CLI Commands
```bash
# Validate a specific note
bm schema validate people/ada-lovelace.md
# Validate all notes of a type
bm schema validate Person
# Validate everything with a schema
bm schema validate
# Infer schema from existing notes
bm schema infer Person
# Show schema drift from current definition
bm schema diff Person
# List all schema notes
bm search "type:schema"
```
## Examples
### Complete Person Workflow
**Schema:**
```yaml
# schema/Person.md
---
title: Person
type: schema
entity: Person
version: 1
schema:
name: string, full name
role?: string, job title or position
works_at?: Organization, employer
expertise?(array): string, areas of knowledge
email?: string, contact email
settings:
validation: warn
---
# Person
A human individual in the knowledge graph.
```
**Valid note:**
```yaml
# people/paul-graham.md
---
title: Paul Graham
type: Person
tags: [startups, essays, lisp]
---
# Paul Graham
## Observations
- [name] Paul Graham
- [role] Essayist and investor
- [expertise] Startups
- [expertise] Lisp
- [expertise] Essay writing
- [fact] Created Viaweb, the first web app
## Relations
- works_at [[Y Combinator]]
- authored [[Hackers and Painters]]
```
**Note with warnings:**
```yaml
# people/ada-lovelace.md
---
title: Ada Lovelace
type: Person
---
# Ada Lovelace
## Observations
- [fact] Wrote the first computer program
- [born] 1815
## Relations
- collaborated_with [[Charles Babbage]]
```
Validation: warns about missing required `[name]` observation. Everything else is optional
or unmatched (which is fine).
## Future Considerations (Deferred)
These are interesting but out of scope for the initial implementation:
- **Multiple schema inheritance**`schema: [Person, Author]`
- **Hook integration** — Pre-write validation via the hooks system
- **OWL/RDF export**`bm schema export --format owl`
- **SPARQL queries** — Schema-aware graph queries
- **Built-in templates**`bm schema use gtd`, `bm schema use zettelkasten`
- **Schema versioning/migration** — Tracking breaking changes across versions
+1 -3
View File
@@ -8,9 +8,8 @@ To keep the default CI signal **stable and meaningful**, the default `pytest` co
- highly environment-dependent (OS/DB tuning)
- inherently interactive (CLI)
- background-task orchestration (watchers/sync runners)
- external analytics
### Whats excluded (and why)
### What's excluded (and why)
Coverage excludes are configured in `pyproject.toml` under `[tool.coverage.report].omit`.
@@ -19,7 +18,6 @@ Current exclusions include:
- `src/basic_memory/db.py`: platform/backend tuning paths (SQLite/Postgres/Windows), covered by integration tests and targeted runs.
- `src/basic_memory/services/initialization.py`: startup orchestration/background tasks; covered indirectly by app/MCP entrypoints.
- `src/basic_memory/sync/sync_service.py`: heavy filesystem↔DB integration; validated in integration suite (not enforced in unit coverage).
- `src/basic_memory/telemetry.py`: external analytics; exercised lightly but excluded from strict coverage gate.
### Recommended additional runs
+51 -5
View File
@@ -62,6 +62,22 @@ test-int-postgres:
BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov test-int
fi
# Run tests impacted by recent changes (requires pytest-testmon)
testmon *args:
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov --testmon --testmon-forceselect {{args}}
# Run MCP smoke test (fast end-to-end loop)
test-smoke:
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m smoke test-int/mcp/test_smoke_integration.py
# Fast local loop: lint, format, typecheck, impacted tests
fast-check:
just fix
just format
just typecheck
just testmon
just test-smoke
# Reset Postgres test database (drops and recreates schema)
# Useful when Alembic migration state gets out of sync during development
# Uses credentials from docker-compose-postgres.yml
@@ -91,6 +107,13 @@ test-windows:
test-benchmark:
uv run pytest -p pytest_mock -v --no-cov -m benchmark tests test-int
# Compare two search benchmark JSONL outputs
# Usage:
# just benchmark-compare .benchmarks/search-baseline.jsonl .benchmarks/search-candidate.jsonl
# just benchmark-compare .benchmarks/search-baseline.jsonl .benchmarks/search-candidate.jsonl --format markdown --show-missing
benchmark-compare baseline candidate *args:
uv run python test-int/compare_search_benchmarks.py "{{baseline}}" "{{candidate}}" --format table {{args}}
# Run all tests including Windows, Postgres, and Benchmarks (for CI/comprehensive testing)
# Use this before releasing to ensure everything works across all backends and platforms
test-all:
@@ -149,6 +172,18 @@ format:
run-inspector:
npx @modelcontextprotocol/inspector
# Run doctor checks in an isolated temp home/config
doctor:
#!/usr/bin/env bash
set -euo pipefail
TMP_HOME=$(mktemp -d)
TMP_CONFIG=$(mktemp -d)
HOME="$TMP_HOME" \
BASIC_MEMORY_ENV=test \
BASIC_MEMORY_HOME="$TMP_HOME/basic-memory" \
BASIC_MEMORY_CONFIG_DIR="$TMP_CONFIG" \
./.venv/bin/python -m basic_memory.cli.main doctor --local
# Update all dependencies to latest versions
update-deps:
@@ -204,9 +239,14 @@ release version:
echo "📝 Updating version in __init__.py..."
sed -i.bak "s/__version__ = \".*\"/__version__ = \"$VERSION_NUM\"/" src/basic_memory/__init__.py
rm -f src/basic_memory/__init__.py.bak
# Update version in server.json (MCP registry metadata)
echo "📝 Updating version in server.json..."
sed -i.bak "s/\"version\": \"[^\"]*\"/\"version\": \"$VERSION_NUM\"/g" server.json
rm -f server.json.bak
# Commit version update
git add src/basic_memory/__init__.py
git add src/basic_memory/__init__.py server.json
git commit -m "chore: update version to $VERSION_NUM for {{version}} release"
# Create and push tag
@@ -221,9 +261,10 @@ release version:
echo "📦 GitHub Actions will build and publish to PyPI"
echo "🔗 Monitor at: https://github.com/basicmachines-co/basic-memory/actions"
echo ""
echo "📝 REMINDER: Update documentation sites after release is published:"
echo "📝 REMINDER: Post-release tasks:"
echo " 1. docs.basicmemory.com - Add release notes to src/pages/latest-releases.mdx"
echo " 2. basicmachines.co - Update version in src/components/sections/hero.tsx"
echo " 3. MCP Registry - Run: mcp-publisher publish"
echo " See: .claude/commands/release/release.md for detailed instructions"
# Create a beta release (e.g., just beta v0.13.2b1)
@@ -269,9 +310,14 @@ beta version:
echo "📝 Updating version in __init__.py..."
sed -i.bak "s/__version__ = \".*\"/__version__ = \"$VERSION_NUM\"/" src/basic_memory/__init__.py
rm -f src/basic_memory/__init__.py.bak
# Update version in server.json (MCP registry metadata)
echo "📝 Updating version in server.json..."
sed -i.bak "s/\"version\": \"[^\"]*\"/\"version\": \"$VERSION_NUM\"/g" server.json
rm -f server.json.bak
# Commit version update
git add src/basic_memory/__init__.py
git add src/basic_memory/__init__.py server.json
git commit -m "chore: update version to $VERSION_NUM for {{version}} beta release"
# Create and push tag
+8 -5
View File
@@ -41,12 +41,17 @@ dependencies = [
"mdformat>=0.7.22",
"mdformat-gfm>=0.3.7",
"mdformat-frontmatter>=2.0.8",
"openpanel>=0.0.1", # Anonymous usage telemetry (Homebrew-style opt-out)
"sniffio>=1.3.1",
"anyio>=4.10.0",
"httpx>=0.28.0",
]
[project.optional-dependencies]
semantic = [
"fastembed>=0.7.4",
"sqlite-vec>=0.1.6",
"openai>=1.100.2",
]
[project.urls]
Homepage = "https://github.com/basicmachines-co/basic-memory"
@@ -72,6 +77,7 @@ markers = [
"slow: Slow-running tests (deselect with '-m \"not slow\"')",
"postgres: Tests that run against Postgres backend (deselect with '-m \"not postgres\"')",
"windows: Windows-specific tests (deselect with '-m \"not windows\"')",
"smoke: Fast end-to-end smoke tests for MCP flows",
]
[tool.ruff]
@@ -92,6 +98,7 @@ dev = [
"testcontainers[postgres]>=4.0.0",
"psycopg>=3.2.0",
"pyright>=1.1.408",
"pytest-testmon>=2.2.0",
]
[tool.hatch.version]
@@ -143,9 +150,5 @@ omit = [
"*/db.py", # Backend/runtime-dependent (sqlite/postgres/windows tuning); validated via integration tests
"*/services/initialization.py", # Startup orchestration + background tasks (watchers); exercised indirectly in entrypoints
"*/sync/sync_service.py", # Heavy filesystem/db integration; covered by integration suite, not enforced in unit coverage
"*/telemetry.py", # External analytics; tested lightly, excluded from strict coverage target
"*/services/migration_service.py", # Complex migration scenarios
]
[tool.logfire]
ignore_no_config = true
+25
View File
@@ -0,0 +1,25 @@
{
"$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json",
"name": "io.github.basicmachines-co/basic-memory",
"description": "Local-first knowledge management with bi-directional LLM sync via Markdown files.",
"repository": {
"url": "https://github.com/basicmachines-co/basic-memory.git",
"source": "github"
},
"version": "0.18.3",
"packages": [
{
"registryType": "pypi",
"identifier": "basic-memory",
"version": "0.18.3",
"runtimeHint": "uvx",
"runtimeArguments": [
{"type": "positional", "value": "basic-memory"},
{"type": "positional", "value": "mcp"}
],
"transport": {
"type": "stdio"
}
}
]
}
@@ -1,156 +0,0 @@
---
title: 'SPEC-1: Specification-Driven Development Process'
type: spec
permalink: specs/spec-1-specification-driven-development-process
tags:
- process
- specification
- development
- meta
---
# SPEC-1: Specification-Driven Development Process
## Why
We're implementing specification-driven development to solve the complexity and circular refactoring issues in our web development process.
Instead of getting lost in framework details and type gymnastics, we start with clear specifications that drive implementation.
The default approach of adhoc development with AI agents tends to result in:
- Circular refactoring cycles
- Fighting framework complexity
- Lost context between sessions
- Unclear requirements and scope
## What
This spec defines our process for using basic-memory as the specification engine to build basic-memory-cloud.
We're creating a recursive development pattern where basic-memory manages the specs that drive the development of basic-memory-cloud.
**Affected Areas:**
- All future component development
- Architecture decisions
- Agent collaboration workflows
- Knowledge management and context preservation
## How (High Level)
### Specification Structure
Name: Spec names should be numbered sequentially, followed by a description eg. `SPEC-X - Simple Description.md`.
See: [[Spec-2: Slash Commands Reference]]
Every spec is a complete thought containing:
- **Why**: The reasoning and problem being solved
- **What**: What is affected or changed
- **How**: High-level approach to implementation
- **How to Evaluate**: Testing/validation procedure
- Additional context as needed
### Living Specification Format
Specifications are **living documents** that evolve throughout implementation:
**Progress Tracking:**
- **Completed items**: Use ✅ checkmark emoji for implemented features
- **Pending items**: Use `- [ ]` GitHub-style checkboxes for remaining tasks
- **In-progress items**: Use `- [x]` when work is actively underway
**Status Philosophy:**
- **Avoid static status headers** like "COMPLETE" or "IN PROGRESS" that become stale
- **Use checklists within content** to show granular implementation progress
- **Keep specs informative** while providing clear progress visibility
- **Update continuously** as understanding and implementation evolve
**Example Format:**
```markdown
### ComponentName
- ✅ Basic functionality implemented
- ✅ Props and events defined
- - [ ] Add sorting controls
- - [ ] Improve accessibility
- - [x] Currently implementing responsive design
```
This creates **git-friendly progress tracking** where `[ ]` easily becomes `[x]` or ✅ when completed, and specs remain valuable throughout the development lifecycle.
## Claude Code
We will leverage Claude Code capabilities to make the process semi-automated.
- Slash commands: define repeatable steps in the process (create spec, implement, review, etc)
- Agents: define roles to carry out instructions (front end developer, baskend developer, etc)
- MCP tools: enable agents to implement specs via actions (write code, test, etc)
### Workflow
1. **Create**: Write spec as complete thought in `/specs` folder
2. **Discuss**: Iterate and refine through agent collaboration
3. **Implement**: Hand spec to appropriate specialist agent
4. **Validate**: Review implementation against spec criteria
5. **Document**: Update spec with learnings and decisions
### Slash Commands
Claude slash commands are used to manage the flow.
These are simple instructions to help make the process uniform.
They can be updated and refined as needed.
- `/spec create [name]` - Create new specification
- `/spec status` - Show current spec states
- `/spec implement [name]` - Hand to appropriate agent
- `/spec review [name]` - Validate implementation
### Agent Orchestration
Agents are defined with clear roles, for instance:
- **system-architect**: Creates high-level specs, ADRs, architectural decisions
- **vue-developer**: Component specs, UI patterns, frontend architecture
- **python-developer**: Implementation specs, technical details, backend logic
-
- Each agent reads/updates specs through basic-memory tools.
## How to Evaluate
### Success Criteria
- Specs provide clear, actionable guidance for implementation
- Reduced circular refactoring and scope creep
- Persistent context across development sessions
- Clean separation between "what/why" and implementation details
- Specs record a history of what happened and why for historical context
### Testing Procedure
1. Create a spec for an existing problematic component
2. Have an agent implement following only the spec
3. Compare result quality and development speed vs. ad-hoc approach
4. Measure context preservation across sessions
5. Evaluate spec clarity and completeness
### Metrics
- Time from spec to working implementation
- Number of refactoring cycles required
- Agent understanding of requirements
- Spec reusability for similar components
## Notes
- Start simple: specs are just complete thoughts, not heavy processes
- Use basic-memory's knowledge graph to link specs, decisions, components
- Let the process evolve naturally based on what works
- Focus on solving the actual problem: Manage complexity in development
## Observations
- [problem] Web development without clear goals and documentation circular refactoring cycles #complexity
- [solution] Specification-driven development reduces scope creep and context loss #process-improvement
- [pattern] basic-memory as specification engine creates recursive development loop #meta-development
- [workflow] Five-step process: Create → Discuss → Implement → Validate → Document #methodology
- [tool] Slash commands provide uniform process automation #automation
- [agent-pattern] Three specialized agents handle different implementation domains #specialization
- [success-metric] Time from spec to working implementation measures process efficiency #measurement
- [learning] Process should evolve naturally based on what works in practice #adaptation
- [format] Living specifications use checklists for progress tracking instead of static status headers #documentation
- [evolution] Specs evolve throughout implementation maintaining value as working documents #continuous-improvement
## Relations
- spec [[Spec-2: Slash Commands Reference]]
- spec [[Spec-3: Agent Definitions]]
@@ -1,569 +0,0 @@
---
title: 'SPEC-10: Unified Deployment Workflow and Event Tracking'
type: spec
permalink: specs/spec-10-unified-deployment-workflow-event-tracking
tags:
- workflow
- deployment
- event-sourcing
- architecture
- simplification
---
# SPEC-10: Unified Deployment Workflow and Event Tracking
## Why
We replaced a complex multi-workflow system with DBOS orchestration that was proving to be more trouble than it was worth. The previous architecture had four separate workflows (`tenant_provisioning`, `tenant_update`, `tenant_deployment`, `tenant_undeploy`) with overlapping logic, complex state management, and fragmented event tracking. DBOS added unnecessary complexity without providing sufficient value, leading to harder debugging and maintenance.
**Problems Solved:**
- **Framework Complexity**: DBOS configuration overhead and fighting framework limitations
- **Code Duplication**: Multiple workflows implementing similar operations with duplicate logic
- **Poor Observability**: Fragmented event tracking across workflow boundaries
- **Maintenance Overhead**: Complex orchestration for fundamentally simple operations
- **Debugging Difficulty**: Framework abstractions hiding simple Python stack traces
## What
This spec documents the architectural simplification that consolidates tenant lifecycle management into a unified system with comprehensive event tracking.
**Affected Areas:**
- Tenant deployment workflows (provisioning, updates, undeploying)
- Event sourcing and workflow tracking infrastructure
- API endpoints for tenant operations
- Database schema for workflow and event correlation
- Integration testing for tenant lifecycle operations
**Key Changes:**
- **Removed DBOS entirely** - eliminated framework dependency and complexity
- **Consolidated 4 workflows → 2 unified deployment workflows (deploy/undeploy)**
- **Added workflow tracking system** with complete event correlation
- **Simplified API surface** - single `/deploy` endpoint handles all scenarios
- **Enhanced observability** through event sourcing with workflow grouping
## How (High Level)
### Architectural Philosophy
**Embrace simplicity over framework complexity** - use well-structured Python with proper database design instead of complex orchestration frameworks.
### Core Components
#### 1. Unified Deployment Workflow
```python
class TenantDeploymentWorkflow:
async def deploy_tenant_workflow(self, tenant_id: str, workflow_id: UUID, image_tag: str = None):
# Single workflow handles both initial provisioning AND updates
# Each step is idempotent and handles its own error recovery
# Database transactions provide the durability we need
await self.start_deployment_step(workflow_id, tenant_uuid, image_tag)
await self.create_fly_app_step(workflow_id, tenant_uuid)
await self.create_bucket_step(workflow_id, tenant_uuid)
await self.deploy_machine_step(workflow_id, tenant_uuid, image_tag)
await self.complete_deployment_step(workflow_id, tenant_uuid, image_tag, deployment_time)
```
**Key Benefits:**
- **Handles both provisioning and updates** in single workflow
- **Idempotent operations** - safe to retry any step
- **Clean error handling** via simple Python exceptions
- **Resumable** - can restart from any failed step
#### 2. Workflow Tracking System
**Database Schema:**
```sql
CREATE TABLE workflow (
id UUID PRIMARY KEY,
workflow_type VARCHAR(50) NOT NULL, -- 'tenant_deployment', 'tenant_undeploy'
tenant_id UUID REFERENCES tenant(id),
status VARCHAR(20) DEFAULT 'running', -- 'running', 'completed', 'failed'
workflow_metadata JSONB DEFAULT '{}' -- image_tag, etc.
);
ALTER TABLE event ADD COLUMN workflow_id UUID REFERENCES workflow(id);
```
**Event Correlation:**
- Every workflow operation generates events tagged with `workflow_id`
- Complete audit trail from workflow start to completion
- Events grouped by workflow for easy reconstruction of operations
#### 3. Parameter Standardization
All workflow methods follow consistent signature pattern:
```python
async def method_name(self, session: AsyncSession, workflow_id: UUID | None, tenant_id: UUID, ...)
```
**Benefits:**
- **Consistent event tagging** - all events properly correlated
- **Clear method contracts** - workflow_id always first parameter
- **Type safety** - proper UUID handling throughout
### Implementation Strategy
#### Phase 1: Workflow Consolidation ✅ COMPLETED
- [x] **Remove DBOS dependency** - eliminated dbos_config.py and all DBOS imports
- [x] **Create unified TenantDeploymentWorkflow** - handles both provisioning and updates
- [x] **Remove legacy workflows** - deleted tenant_provisioning.py, tenant_update.py
- [x] **Simplify API endpoints** - consolidated to single `/deploy` endpoint
- [x] **Update integration tests** - comprehensive edge case testing
#### Phase 2: Workflow Tracking System ✅ COMPLETED
- [x] **Database migration** - added workflow table and event.workflow_id foreign key
- [x] **Workflow repository** - CRUD operations for workflow records
- [x] **Event correlation** - all workflow events tagged with workflow_id
- [x] **Comprehensive testing** - workflow lifecycle and event grouping tests
#### Phase 3: Parameter Standardization ✅ COMPLETED
- [x] **Standardize method signatures** - workflow_id as first parameter pattern
- [x] **Fix event tagging** - ensure all workflow events properly correlated
- [x] **Update service methods** - consistent parameter order across tenant_service
- [x] **Integration test validation** - verify complete event sequences
### Architectural Benefits
#### Code Simplification
- **39 files changed**: 2,247 additions, 3,256 deletions (net -1,009 lines)
- **Eliminated framework complexity** - no more DBOS configuration or abstractions
- **Consolidated logic** - single deployment workflow vs 4 separate workflows
- **Cleaner API surface** - unified endpoint vs multiple workflow-specific endpoints
#### Enhanced Observability
- **Complete event correlation** - every workflow event tagged with workflow_id
- **Audit trail reconstruction** - can trace entire tenant lifecycle through events
- **Workflow status tracking** - running/completed/failed states in database
- **Comprehensive testing** - edge cases covered with real infrastructure
#### Operational Benefits
- **Simpler debugging** - plain Python stack traces vs framework abstractions
- **Reduced dependencies** - one less complex framework to maintain
- **Better error handling** - explicit exception handling vs framework magic
- **Easier maintenance** - straightforward Python code vs orchestration complexity
## How to Evaluate
### Success Criteria
#### Functional Completeness ✅ VERIFIED
- [x] **Unified deployment workflow** handles both initial provisioning and updates
- [x] **Undeploy workflow** properly integrated with event tracking
- [x] **All operations idempotent** - safe to retry any step without duplication
- [x] **Complete tenant lifecycle** - provision → active → update → undeploy
#### Event Tracking and Correlation ✅ VERIFIED
- [x] **All workflow events tagged** with proper workflow_id
- [x] **Event sequence verification** - tests assert exact event order and content
- [x] **Workflow grouping** - events can be queried by workflow_id for complete audit trail
- [x] **Cross-workflow isolation** - deployment vs undeploy events properly separated
#### Database Schema and Performance ✅ VERIFIED
- [x] **Migration applied** - workflow table and event.workflow_id column created
- [x] **Proper indexing** - performance optimized queries on workflow_type, tenant_id, status
- [x] **Foreign key constraints** - referential integrity between workflows and events
- [x] **Database triggers** - updated_at timestamp automation
#### Test Coverage ✅ COMPREHENSIVE
- [x] **Unit tests**: 4 workflow tracking tests covering lifecycle and event grouping
- [x] **Integration tests**: Real infrastructure testing with Fly.io resources
- [x] **Edge case coverage**: Failed deployments, partial state recovery, resource conflicts
- [x] **Event sequence verification**: Exact event order and content validation
### Testing Procedure
#### Unit Test Validation ✅ PASSING
```bash
cd apps/cloud && pytest tests/test_workflow_tracking.py -v
# 4/4 tests passing - workflow lifecycle and event grouping
```
#### Integration Test Validation ✅ PASSING
```bash
cd apps/cloud && pytest tests/integration/test_tenant_workflow_deployment_integration.py -v
cd apps/cloud && pytest tests/integration/test_tenant_workflow_undeploy_integration.py -v
# Comprehensive real infrastructure testing with actual Fly.io resources
# Tests provision → deploy → update → undeploy → cleanup cycles
```
### Performance Metrics
#### Code Metrics ✅ ACHIEVED
- **Net code reduction**: -1,009 lines (3,256 deletions, 2,247 additions)
- **Workflow consolidation**: 4 workflows → 1 unified deployment workflow
- **Dependency reduction**: Removed DBOS framework dependency entirely
- **API simplification**: Multiple endpoints → single `/deploy` endpoint
#### Operational Metrics ✅ VERIFIED
- **Event correlation**: 100% of workflow events properly tagged with workflow_id
- **Audit trail completeness**: Full tenant lifecycle traceable through event sequences
- **Error handling**: Clean Python exceptions vs framework abstractions
- **Debugging simplicity**: Direct stack traces vs orchestration complexity
### Implementation Status: ✅ COMPLETE
All phases completed successfully with comprehensive testing and verification:
**Phase 1 - Workflow Consolidation**: ✅ COMPLETE
- Removed DBOS dependency and consolidated workflows
- Unified deployment workflow handles all scenarios
- Comprehensive integration testing with real infrastructure
**Phase 2 - Workflow Tracking**: ✅ COMPLETE
- Database schema implemented with proper indexing
- Event correlation system fully functional
- Complete audit trail capability verified
**Phase 3 - Parameter Standardization**: ✅ COMPLETE
- Consistent method signatures across all workflow methods
- All events properly tagged with workflow_id
- Type safety verified across entire codebase
**Phase 4 - Asynchronous Job Queuing**:
**Goal**: Transform synchronous deployment workflows into background jobs for better user experience and system reliability.
**Current Problem**:
- Deployment API calls are synchronous - users wait for entire tenant provisioning (30-60 seconds)
- No retry mechanism for failed operations
- HTTP timeouts on long-running deployments
- Poor user experience during infrastructure provisioning
**Solution**: Redis-backed job queue with arq for reliable background processing
#### Architecture Overview
```python
# API Layer: Return immediately with job tracking
@router.post("/{tenant_id}/deploy")
async def deploy_tenant(tenant_id: UUID):
# Create workflow record in Postgres
workflow = await workflow_repo.create_workflow("tenant_deployment", tenant_id)
# Enqueue job in Redis
job = await arq_pool.enqueue_job('deploy_tenant_task', tenant_id, workflow.id)
# Return job ID immediately
return {"job_id": job.job_id, "workflow_id": workflow.id, "status": "queued"}
# Background Worker: Process via existing unified workflow
async def deploy_tenant_task(ctx, tenant_id: str, workflow_id: str):
# Existing workflow logic - zero changes needed!
await workflow_manager.deploy_tenant(UUID(tenant_id), workflow_id=UUID(workflow_id))
```
#### Implementation Tasks
**Phase 4.1: Core Job Queue Setup** ✅ COMPLETED
- [x] **Add arq dependency** - integrated Redis job queue with existing infrastructure
- [x] **Create job definitions** - wrapped existing deployment/undeploy workflows as arq tasks
- [x] **Update API endpoints** - updated provisioning endpoints to return job IDs instead of waiting for completion
- [x] **JobQueueService implementation** - service layer for job enqueueing and status tracking
- [x] **Job status tracking** - integrated with existing workflow table for status updates
- [x] **Comprehensive testing** - 18 tests covering positive, negative, and edge cases
**Phase 4.2: Background Worker Implementation** ✅ COMPLETED
- [x] **Job status API** - GET /jobs/{job_id}/status endpoint integrated with JobQueueService
- [x] **Background worker process** - arq worker to process queued jobs with proper settings and Redis configuration
- [x] **Worker settings and configuration** - WorkerSettings class with proper timeouts, max jobs, and error handling
- [x] **Fix API endpoints** - updated job status API to use JobQueueService instead of direct Redis access
- [x] **Integration testing** - comprehensive end-to-end testing with real ARQ workers and Fly.io infrastructure
- [x] **Worker entry points** - dual-purpose entrypoint.sh script and __main__.py module support for both API and worker processes
- [x] **Test fixture updates** - fixed all API and service test fixtures to work with job queue dependencies
- [x] **AsyncIO event loop fixes** - resolved event loop issues in integration tests for subprocess worker compatibility
- [x] **Complete test coverage** - all 46 tests passing across unit, integration, and API test suites
- [x] **Type safety verification** - 0 type checking errors across entire ARQ job queue implementation
#### Phase 4.2 Implementation Summary ✅ COMPLETE
**Core ARQ Job Queue System:**
- **JobQueueService** - Centralized service for job enqueueing, status tracking, and Redis pool management
- **deployment_jobs.py** - ARQ job functions that wrap existing deployment/undeploy workflows
- **Worker Settings** - Production-ready ARQ configuration with proper timeouts and error handling
- **Dual-Process Architecture** - Single Docker image with entrypoint.sh supporting both API and worker modes
**Key Files Added:**
- `apps/cloud/src/basic_memory_cloud/jobs/` - Complete job queue implementation (7 files)
- `apps/cloud/entrypoint.sh` - Dual-purpose Docker container entry point
- `apps/cloud/tests/integration/test_worker_integration.py` - Real infrastructure integration tests
- `apps/cloud/src/basic_memory_cloud/schemas/job_responses.py` - API response schemas
**API Integration:**
- Provisioning endpoints return job IDs immediately instead of blocking for 60+ seconds
- Job status API endpoints for real-time monitoring of deployment progress
- Proper error handling and job failure scenarios with detailed error messages
**Testing Achievement:**
- **46 total tests passing** across all test suites (unit, integration, API, services)
- **Real infrastructure testing** - ARQ workers process actual Fly.io deployments
- **Event loop safety** - Fixed asyncio issues for subprocess worker compatibility
- **Test fixture updates** - All fixtures properly support job queue dependencies
- **Type checking** - 0 errors across entire codebase
**Technical Metrics:**
- **38 files changed** - +1,736 insertions, -334 deletions
- **Integration test runtime** - ~18 seconds with real ARQ workers and Fly.io verification
- **Event loop isolation** - Proper async session management for subprocess compatibility
- **Redis integration** - Production-ready Redis configuration with connection pooling
**Phase 4.3: Production Hardening** ✅ COMPLETED
- [x] **Configure Upstash Redis** - production Redis setup on Fly.io
- [x] **Retry logic for external APIs** - exponential backoff for flaky Tigris IAM operations
- [x] **Monitoring and observability** - comprehensive Redis queue monitoring with CLI tools
- [x] **Error handling improvements** - graceful handling of expected API errors with appropriate log levels
- [x] **CLI tooling enhancements** - bulk update commands for CI/CD automation
- [x] **Documentation improvements** - comprehensive monitoring guide with Redis patterns
- [x] **Job uniqueness** - ARQ-based duplicate prevention for tenant operations
- [ ] **Worker scaling** - multiple arq workers for parallel job processing
- [ ] **Job persistence** - ensure jobs survive Redis/worker restarts
- [ ] **Error alerting** - notifications for failed deployment jobs
**Phase 4.4: Advanced Features** (Future)
- [ ] **Job scheduling** - deploy tenants at specific times
- [ ] **Priority queues** - urgent deployments processed first
- [ ] **Batch operations** - bulk tenant deployments
- [ ] **Job dependencies** - deployment → configuration → activation chains
#### Benefits Achieved ✅ REALIZED
**User Experience Improvements:**
- **Immediate API responses** - users get job ID instantly vs waiting 60+ seconds for deployment completion
- **Real-time job tracking** - status API provides live updates on deployment progress
- **Better error visibility** - detailed error messages and job failure tracking
- **CI/CD automation ready** - bulk update commands for automated tenant deployments
**System Reliability:**
- **Redis persistence** - jobs survive Redis/worker restarts with proper queue durability
- **Idempotent job processing** - jobs can be safely retried without side effects
- **Event loop isolation** - worker processes operate independently from API server
- **Retry resilience** - exponential backoff for flaky external API calls (3 attempts, 1s/2s delays)
- **Graceful error handling** - expected API errors logged at INFO level, unexpected at ERROR level
- **Job uniqueness** - prevent duplicate tenant operations with ARQ's built-in uniqueness feature
**Operational Benefits:**
- **Horizontal scaling ready** - architecture supports adding more workers for parallel processing
- **Comprehensive testing** - real infrastructure integration tests ensure production reliability
- **Type safety** - full type checking prevents runtime errors in job processing
- **Clean separation** - API and worker processes use same codebase with different entry points
- **Queue monitoring** - Redis CLI integration for real-time queue activity monitoring
- **Comprehensive documentation** - detailed monitoring guide with Redis pattern explanations
**Development Benefits:**
- **Zero workflow changes** - existing deployment/undeploy workflows work unchanged as background jobs
- **Async/await native** - modern Python asyncio patterns throughout the implementation
- **Event correlation preserved** - all existing workflow tracking and event sourcing continues to work
- **Enhanced CLI tooling** - unified tenant commands with proper endpoint routing
- **Database integrity** - proper foreign key constraint handling in tenant deletion
#### Infrastructure Requirements
- **Local**: Redis via docker-compose (already exists) ✅
- **Production**: Upstash Redis on Fly.io (already configured) ✅
- **Workers**: arq worker processes (new deployment target)
- **Monitoring**: Job status dashboard (simple web interface)
#### API Evolution
```python
# Before: Synchronous (blocks for 60+ seconds)
POST /tenant/{id}/deploy → {status: "active", machine_id: "..."}
# After: Asynchronous (returns immediately)
POST /tenant/{id}/deploy → {job_id: "uuid", workflow_id: "uuid", status: "queued"}
GET /jobs/{job_id}/status → {status: "running", progress: "deploying_machine", workflow_id: "uuid"}
GET /workflows/{workflow_id}/events → [...] # Existing event tracking works unchanged
```
**Technology Choice**: **arq (Redis)** over pgqueuer
- **Existing Redis infrastructure** - Upstash + docker-compose already configured
- **Better ecosystem** - monitoring tools, documentation, community
- **Made by pydantic team** - aligns with existing Python stack
- **Hybrid approach** - Redis for queue operations + Postgres for workflow state
#### Job Uniqueness Implementation
**Problem**: Multiple concurrent deployment requests for the same tenant could create duplicate jobs, wasting resources and potentially causing conflicts.
**Solution**: Leverage ARQ's built-in job uniqueness feature using predictable job IDs:
```python
# JobQueueService implementation
async def enqueue_deploy_job(self, tenant_id: UUID, image_tag: str | None = None) -> str:
unique_job_id = f"deploy-{tenant_id}"
job = await self.redis_pool.enqueue_job(
"deploy_tenant_job",
str(tenant_id),
image_tag,
_job_id=unique_job_id, # ARQ prevents duplicates
)
if job is None:
# Job already exists - return existing job ID
return unique_job_id
else:
# New job created - return ARQ job ID
return job.job_id
```
**Key Features:**
- **Predictable Job IDs**: `deploy-{tenant_id}`, `undeploy-{tenant_id}`
- **Duplicate Prevention**: ARQ returns `None` for duplicate job IDs
- **Graceful Handling**: Return existing job ID instead of raising errors
- **Idempotent Operations**: Safe to retry deployment requests
- **Clear Logging**: Distinguish "Enqueued new" vs "Found existing" jobs
**Benefits:**
- Prevents resource waste from duplicate deployments
- Eliminates race conditions from concurrent requests
- Makes job monitoring more predictable with consistent IDs
- Provides natural deduplication without complex locking mechanisms
## Notes
### Design Philosophy Lessons
- **Simplicity beats framework magic** - removing DBOS made the system more reliable and debuggable
- **Event sourcing > complex orchestration** - database-backed event tracking provides better observability than framework abstractions
- **Idempotent operations > resumable workflows** - each step handling its own retry logic is simpler than framework-managed resumability
- **Explicit error handling > framework exception handling** - Python exceptions are clearer than orchestration framework error states
### Future Considerations
- **Monitoring integration** - workflow tracking events could feed into observability systems
- **Performance optimization** - event querying patterns may benefit from additional indexing
- **Audit compliance** - complete event trail supports regulatory requirements
- **Operational dashboards** - workflow status could drive tenant health monitoring
### Related Specifications
- **SPEC-8**: TigrisFS Integration - bucket provisioning integrated with deployment workflow
- **SPEC-1**: Specification-Driven Development Process - this spec follows the established format
## Observations
- [architecture] Removing framework complexity led to more maintainable system #simplification
- [workflow] Single unified deployment workflow handles both provisioning and updates #consolidation
- [observability] Event sourcing with workflow correlation provides complete audit trail #event-tracking
- [database] Foreign key relationships between workflows and events enable powerful queries #schema-design
- [testing] Integration tests with real infrastructure catch edge cases that unit tests miss #testing-strategy
- [parameters] Consistent method signatures (workflow_id first) reduce cognitive overhead #api-design
- [maintenance] Fewer workflows and dependencies reduce long-term maintenance burden #operational-excellence
- [debugging] Plain Python exceptions are clearer than framework abstraction layers #developer-experience
- [resilience] Exponential backoff retry patterns handle flaky external API calls gracefully #error-handling
- [monitoring] Redis queue monitoring provides real-time operational visibility #observability
- [ci-cd] Bulk update commands enable automated tenant deployments in continuous delivery pipelines #automation
- [documentation] Comprehensive monitoring guides reduce operational learning curve #knowledge-management
- [error-logging] Context-aware log levels (INFO for expected errors, ERROR for unexpected) improve signal-to-noise ratio #logging-strategy
- [job-uniqueness] ARQ job uniqueness with predictable tenant-based IDs prevents duplicate operations and resource waste #deduplication
## Implementation Notes
### Configuration Integration
- **Redis Configuration**: Add Redis settings to existing `apps/cloud/src/basic_memory_cloud/config.py`
- **Local Development**: Leverage existing Redis setup from `docker-compose.yml`
- **Production**: Use Upstash Redis configuration for production environments
### Docker Entrypoint Strategy
Create `entrypoint.sh` script to toggle between API server and worker processes using single Docker image:
```bash
#!/bin/bash
# Entrypoint script for Basic Memory Cloud service
# Supports multiple process types: api, worker
set -e
case "$1" in
"api")
echo "Starting Basic Memory Cloud API server..."
exec uvicorn basic_memory_cloud.main:app \
--host 0.0.0.0 \
--port 8000 \
--log-level info
;;
"worker")
echo "Starting Basic Memory Cloud ARQ worker..."
# For ARQ worker implementation
exec python -m arq basic_memory_cloud.jobs.settings.WorkerSettings
;;
*)
echo "Usage: $0 {api|worker}"
echo " api - Start the FastAPI server"
echo " worker - Start the ARQ worker"
exit 1
;;
esac
```
### Fly.io Process Groups Configuration
Use separate machine groups for API and worker processes with independent scaling:
```toml
# fly.toml app configuration for basic-memory-cloud
app = 'basic-memory-cloud-dev-basic-machines'
primary_region = 'dfw'
org = 'basic-machines'
kill_signal = 'SIGINT'
kill_timeout = '5s'
[build]
# Process groups for API server and worker
[processes]
api = "api"
worker = "worker"
# Machine scaling configuration
[[machine]]
size = 'shared-cpu-1x'
processes = ['api']
min_machines_running = 1
auto_stop_machines = false
auto_start_machines = true
[[machine]]
size = 'shared-cpu-1x'
processes = ['worker']
min_machines_running = 1
auto_stop_machines = false
auto_start_machines = true
[env]
# Python configuration
PYTHONUNBUFFERED = '1'
PYTHONPATH = '/app'
# Logging configuration
LOG_LEVEL = 'DEBUG'
# Redis configuration for ARQ
REDIS_URL = 'redis://basic-memory-cloud-redis.upstash.io'
# Database configuration
DATABASE_HOST = 'basic-memory-cloud-db-dev-basic-machines.internal'
DATABASE_PORT = '5432'
DATABASE_NAME = 'basic_memory_cloud'
DATABASE_USER = 'postgres'
DATABASE_SSL = 'true'
# Worker configuration
ARQ_MAX_JOBS = '10'
ARQ_KEEP_RESULT = '3600'
# Fly.io configuration
FLY_ORG = 'basic-machines'
FLY_REGION = 'dfw'
# Internal service - no external HTTP exposure for worker
# API accessible via basic-memory-cloud-dev-basic-machines.flycast:8000
[[vm]]
size = 'shared-cpu-1x'
```
### Benefits of This Architecture
- **Single Docker Image**: Both API and worker use same container with different entrypoints
- **Independent Scaling**: Scale API and worker processes separately based on demand
- **Clean Separation**: Web traffic handling separate from background job processing
- **Existing Infrastructure**: Leverages current PostgreSQL + Redis setup without complexity
- **Hybrid State Management**: Redis for queue operations, PostgreSQL for persistent workflow tracking
## Relations
- implements [[SPEC-8 TigrisFS Integration]]
- follows [[SPEC-1 Specification-Driven Development Process]]
- supersedes previous multi-workflow architecture
@@ -1,186 +0,0 @@
---
title: 'SPEC-11: Basic Memory API Performance Optimization'
type: spec
permalink: specs/spec-11-basic-memory-api-performance-optimization
tags:
- performance
- api
- mcp
- database
- cloud
---
# SPEC-11: Basic Memory API Performance Optimization
## Why
The Basic Memory API experiences significant performance issues in cloud environments due to expensive per-request initialization. MCP tools making
HTTP requests to the API suffer from 350ms-2.6s latency overhead **before** any actual operation occurs.
**Root Cause Analysis:**
- GitHub Issue #82 shows repeated initialization sequences in logs (16:29:35 and 16:49:58)
- Each MCP tool call triggers full database initialization + project reconciliation
- `get_engine_factory()` dependency calls `db.get_or_create_db()` on every request
- `reconcile_projects_with_config()` runs expensive sync operations repeatedly
**Performance Impact:**
- Database connection setup: ~50-100ms per request
- Migration checks: ~100-500ms per request
- Project reconciliation: ~200ms-2s per request
- **Total overhead**: ~350ms-2.6s per MCP tool call
This creates compounding effects with tenant auto-start delays and increases timeout risk in cloud deployments.
## What
This optimization affects the **core basic-memory repository** components:
1. **API Lifespan Management** (`src/basic_memory/api/app.py`)
- Cache database connections in app state during startup
- Avoid repeated expensive initialization
2. **Dependency Injection** (`src/basic_memory/deps.py`)
- Modify `get_engine_factory()` to use cached connections
- Eliminate per-request database setup
3. **Initialization Service** (`src/basic_memory/services/initialization.py`)
- Add caching/throttling to project reconciliation
- Skip expensive operations when appropriate
4. **Configuration** (`src/basic_memory/config.py`)
- Add optional performance flags for cloud environments
**Backwards Compatibility**: All changes must be backwards compatible with existing CLI and non-cloud usage.
## How (High Level)
### Phase 1: Cache Database Connections (Critical - 80% of gains)
**Problem**: `get_engine_factory()` calls `db.get_or_create_db()` per request
**Solution**: Cache database engine/session in app state during lifespan
1. **Modify API Lifespan** (`api/app.py`):
```python
@asynccontextmanager
async def lifespan(app: FastAPI):
app_config = ConfigManager().config
await initialize_app(app_config)
# Cache database connection in app state
engine, session_maker = await db.get_or_create_db(app_config.database_path)
app.state.engine = engine
app.state.session_maker = session_maker
# ... rest of startup logic
```
2. Modify Dependency Injection (deps.py):
```python
async def get_engine_factory(
request: Request
) -> tuple[AsyncEngine, async_sessionmaker[AsyncSession]]:
"""Get cached engine and session maker from app state."""
return request.app.state.engine, request.app.state.session_maker
```
Phase 2: Optimize Project Reconciliation (Secondary - 20% of gains)
Problem: reconcile_projects_with_config() runs expensive sync repeatedly
Solution: Add module-level caching with time-based throttling
1. Add Reconciliation Cache (services/initialization.py):
```ptyhon
_project_reconciliation_completed = False
_last_reconciliation_time = 0
async def reconcile_projects_with_config(app_config, force=False):
# Skip if recently completed (within 60 seconds) unless forced
if recently_completed and not force:
return
# ... existing logic
```
Phase 3: Cloud Environment Flags (Optional)
Problem: Force expensive initialization in production environments
Solution: Add skip flags for cloud/stateless deployments
1. Add Config Flag (config.py):
skip_initialization_sync: bool = Field(default=False)
2. Configure in Cloud (basic-memory-cloud integration):
BASIC_MEMORY_SKIP_INITIALIZATION_SYNC=true
How to Evaluate
Success Criteria
1. Performance Metrics (Primary):
- MCP tool response time reduced by 50%+ (measure before/after)
- Database connection overhead eliminated (0ms vs 50-100ms)
- Migration check overhead eliminated (0ms vs 100-500ms)
- Project reconciliation overhead reduced by 90%+
2. Load Testing:
- Concurrent MCP tool calls maintain performance
- No memory leaks in cached connections
- Database connection pool behaves correctly
3. Functional Correctness:
- All existing API endpoints work identically
- MCP tools maintain full functionality
- CLI operations unaffected
- Database migrations still execute properly
4. Backwards Compatibility:
- No breaking changes to existing APIs
- Config changes are optional with safe defaults
- Non-cloud deployments work unchanged
Testing Strategy
Performance Testing:
# Before optimization
time basic-memory-mcp-tools write_note "test" "content" "folder"
# Measure: ~1-3 seconds
# After optimization
time basic-memory-mcp-tools write_note "test" "content" "folder"
# Target: <500ms
Load Testing:
# Multiple concurrent MCP tool calls
for i in {1..10}; do
basic-memory-mcp-tools search "test" &
done
wait
# Verify: No degradation, consistent response times
Regression Testing:
# Full basic-memory test suite
just test
# All tests must pass
# Integration tests with cloud deployment
# Verify MCP gateway → API → database flow works
Validation Checklist
- Phase 1 Complete: Database connections cached, dependency injection optimized
- Performance Benchmark: 50%+ improvement in MCP tool response times
- Memory Usage: No leaks in cached connections over 24h+ periods
- Stress Testing: 100+ concurrent requests maintain performance
- Backwards Compatibility: All existing functionality preserved
- Documentation: Performance optimization documented in README
- Cloud Integration: basic-memory-cloud sees performance benefits
Notes
Implementation Priority:
- Phase 1 provides 80% of performance gains and should be implemented first
- Phase 2 provides remaining 20% and addresses edge cases
- Phase 3 is optional for maximum cloud optimization
Risk Mitigation:
- All changes backwards compatible
- Gradual rollout possible (Phase 1 → 2 → 3)
- Easy rollback via configuration flags
Cloud Integration:
- This optimization directly addresses basic-memory-cloud issue #82
- Changes in core basic-memory will benefit all cloud tenants
- No changes needed in basic-memory-cloud itself
@@ -1,182 +0,0 @@
# SPEC-12: OpenTelemetry Observability
## Why
We need comprehensive observability for basic-memory-cloud to:
- Track request flows across our multi-tenant architecture (MCP → Cloud → API services)
- Debug performance issues and errors in production
- Understand user behavior and system usage patterns
- Correlate issues to specific tenants for targeted debugging
- Monitor service health and latency across the distributed system
Currently, we only have basic logging without request correlation or distributed tracing capabilities.
## What
Implement OpenTelemetry instrumentation across all basic-memory-cloud services with:
### Core Requirements
1. **Distributed Tracing**: End-to-end request tracing from MCP gateway through to tenant API instances
2. **Tenant Correlation**: All traces tagged with tenant_id, user_id, and workos_user_id
3. **Service Identification**: Clear service naming and namespace separation
4. **Auto-instrumentation**: Automatic tracing for FastAPI, SQLAlchemy, HTTP clients
5. **Grafana Cloud Integration**: Direct OTLP export to Grafana Cloud Tempo
### Services to Instrument
- **MCP Gateway** (basic-memory-mcp): Entry point with JWT extraction
- **Cloud Service** (basic-memory-cloud): Provisioning and management operations
- **API Service** (basic-memory-api): Tenant-specific instances
- **Worker Processes** (ARQ workers): Background job processing
### Key Trace Attributes
- `tenant.id`: UUID from UserProfile.tenant_id
- `user.id`: WorkOS user identifier
- `user.email`: User email for debugging
- `service.name`: Specific service identifier
- `service.namespace`: Environment (development/production)
- `operation.type`: Business operation (provision/update/delete)
- `tenant.app_name`: Fly.io app name for tenant instances
## How
### Phase 1: Setup OpenTelemetry SDK
1. Add OpenTelemetry dependencies to each service's pyproject.toml:
```python
"opentelemetry-distro[otlp]>=1.29.0",
"opentelemetry-instrumentation-fastapi>=0.50b0",
"opentelemetry-instrumentation-httpx>=0.50b0",
"opentelemetry-instrumentation-sqlalchemy>=0.50b0",
"opentelemetry-instrumentation-logging>=0.50b0",
```
2. Create shared telemetry initialization module (`apps/shared/telemetry.py`)
3. Configure Grafana Cloud OTLP endpoint via environment variables:
```bash
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-2.grafana.net/otlp
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic[token]
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
```
### Phase 2: Instrument MCP Gateway
1. Extract tenant context from AuthKit JWT in middleware
2. Create root span with tenant attributes
3. Propagate trace context to downstream services via headers
### Phase 3: Instrument Cloud Service
1. Continue trace from MCP gateway
2. Add operation-specific attributes (provisioning events)
3. Instrument ARQ worker jobs for async operations
4. Track Fly.io API calls and latency
### Phase 4: Instrument API Service
1. Extract tenant context from JWT
2. Add machine-specific metadata (instance ID, region)
3. Instrument database operations with SQLAlchemy
4. Track MCP protocol operations
### Phase 5: Configure and Deploy
1. Add OTLP configuration to `.env.example` and `.env.example.secrets`
2. Set Fly.io secrets for production deployment
3. Update Dockerfiles to use `opentelemetry-instrument` wrapper
4. Deploy to development environment first for testing
## How to Evaluate
### Success Criteria
1. **End-to-end traces visible in Grafana Cloud** showing complete request flow
2. **Tenant filtering works** - Can filter traces by tenant_id to see all requests for a user
3. **Service maps accurate** - Grafana shows correct service dependencies
4. **Performance overhead < 5%** - Minimal latency impact from instrumentation
5. **Error correlation** - Can trace errors back to specific tenant and operation
### Testing Checklist
- [x] Single request creates connected trace across all services
- [x] Tenant attributes present on all spans
- [x] Background jobs (ARQ) appear in traces
- [x] Database queries show in trace timeline
- [x] HTTP calls to Fly.io API tracked
- [x] Traces exported successfully to Grafana Cloud
- [x] Can search traces by tenant_id in Grafana
- [x] Service dependency graph shows correct flow
### Monitoring Success
- All services reporting traces to Grafana Cloud
- No OTLP export errors in logs
- Trace sampling working correctly (if implemented)
- Resource usage acceptable (CPU/memory)
## Dependencies
- Grafana Cloud account with OTLP endpoint configured
- OpenTelemetry Python SDK v1.29.0+
- FastAPI instrumentation compatibility
- Network access from Fly.io to Grafana Cloud
## Implementation Assignment
**Recommended Agent**: python-developer
- Requires Python/FastAPI expertise
- Needs understanding of distributed systems
- Must implement middleware and context propagation
- Should understand OpenTelemetry SDK and instrumentation
## Follow-up Tasks
### Enhanced Log Correlation
While basic trace-to-log correlation works automatically via OpenTelemetry logging instrumentation, consider adding structured logging for improved log filtering:
1. **Structured Logging Context**: Add `logger.bind()` calls to inject tenant/user context directly into log records
2. **Custom Loguru Formatter**: Extract OpenTelemetry span attributes for better log readability
3. **Direct Log Filtering**: Enable searching logs directly by tenant_id, workflow_id without going through traces
This would complement the existing automatic trace correlation and provide better log search capabilities.
## Alternative Solution: Logfire
After implementing OpenTelemetry with Grafana Cloud, we discovered limitations in the observability experience:
- Traces work but lack useful context without correlated logs
- Setting up log correlation with Grafana is complex and requires additional infrastructure
- The developer experience for Python observability is suboptimal
### Logfire Evaluation
**Pydantic Logfire** offers a compelling alternative that addresses your specific requirements:
#### Core Requirements Match
- ✅ **User Activity Tracking**: Automatic request tracing with business context
- ✅ **Error Monitoring**: Built-in exception tracking with full context
- ✅ **Performance Metrics**: Automatic latency and performance monitoring
- ✅ **Request Tracing**: Native distributed tracing across services
- ✅ **Log Correlation**: Seamless trace-to-log correlation without setup
#### Key Advantages
1. **Python-First Design**: Built specifically for Python/FastAPI applications by the Pydantic team
2. **Simple Integration**: `pip install logfire` + `logfire.configure()` vs complex OTLP setup
3. **Automatic Correlation**: Logs automatically include trace context without manual configuration
4. **Real-time SQL Interface**: Query spans and logs using SQL with auto-completion
5. **Better Developer UX**: Purpose-built observability UI vs generic Grafana dashboards
6. **Loguru Integration**: `logger.configure(handlers=[logfire.loguru_handler()])` maintains existing logging
#### Pricing Assessment
- **Free Tier**: 10M spans/month (suitable for development and small production workloads)
- **Transparent Pricing**: $1 per million spans/metrics after free tier
- **No Hidden Costs**: No per-host fees, only usage-based metering
- **Production Ready**: Recently exited beta, enterprise features available
#### Migration Path
The existing OpenTelemetry instrumentation is compatible - Logfire uses OpenTelemetry under the hood, so the current spans and attributes would work unchanged.
### Recommendation
**Consider migrating to Logfire** for the following reasons:
1. It directly addresses the "next to useless" traces problem by providing integrated logs
2. Dramatically simpler setup and maintenance compared to Grafana Cloud + custom log correlation
3. Better ROI on observability investment with purpose-built Python tooling
4. Free tier sufficient for current development needs with clear scaling path
The current Grafana Cloud implementation provides a solid foundation and could remain as a backup/export target, while Logfire becomes the primary observability platform.
## Status
**Created**: 2024-01-28
**Status**: Completed (OpenTelemetry + Grafana Cloud)
**Next Phase**: Evaluate Logfire migration
**Priority**: High - Critical for production observability
@@ -1,917 +0,0 @@
---
title: 'SPEC-13: CLI Authentication with Subscription Validation'
type: spec
permalink: specs/spec-12-cli-auth-subscription-validation
tags:
- authentication
- security
- cli
- subscription
status: draft
created: 2025-10-02
---
# SPEC-13: CLI Authentication with Subscription Validation
## Why
The Basic Memory Cloud CLI currently has a security gap in authentication that allows unauthorized access:
**Current Web Flow (Secure)**:
1. User signs up via WorkOS AuthKit
2. User creates Polar subscription
3. Web app validates subscription before calling `POST /tenants/setup`
4. Tenant provisioned only after subscription validation ✅
**Current CLI Flow (Insecure)**:
1. User signs up via WorkOS AuthKit (OAuth device flow)
2. User runs `bm cloud login`
3. CLI receives JWT token from WorkOS
4. CLI can access all cloud endpoints without subscription check ❌
**Problem**: Anyone can sign up with WorkOS and immediately access cloud infrastructure via CLI without having an active Polar subscription. This creates:
- Revenue loss (free resource consumption)
- Security risk (unauthorized data access)
- Support burden (users accessing features they haven't paid for)
**Root Cause**: The CLI authentication flow validates JWT tokens but doesn't verify subscription status before granting access to cloud resources.
## What
Add subscription validation to authentication flow to ensure only users with active Polar subscriptions can access cloud resources across all access methods (CLI, MCP, Web App, Direct API).
**Affected Components**:
### basic-memory-cloud (Cloud Service)
- `apps/cloud/src/basic_memory_cloud/deps.py` - Add subscription validation dependency
- `apps/cloud/src/basic_memory_cloud/services/subscription_service.py` - Add subscription check method
- `apps/cloud/src/basic_memory_cloud/api/tenant_mount.py` - Protect mount endpoints
- `apps/cloud/src/basic_memory_cloud/api/proxy.py` - Protect proxy endpoints
### basic-memory (CLI)
- `src/basic_memory/cli/commands/cloud/core_commands.py` - Handle 403 errors
- `src/basic_memory/cli/commands/cloud/api_client.py` - Parse subscription errors
- `docs/cloud-cli.md` - Document subscription requirement
**Endpoints to Protect**:
- `GET /tenant/mount/info` - Used by CLI bisync setup
- `POST /tenant/mount/credentials` - Used by CLI bisync credentials
- `GET /proxy/{path:path}` - Used by Web App, MCP tools, CLI tools, Direct API
- All other `/proxy/*` endpoints - Centralized access point for all user operations
## Complete Authentication Flow Analysis
### Overview of All Access Flows
Basic Memory Cloud has **7 distinct authentication flows**. This spec closes subscription validation gaps in flows 2-4 and 6, which all converge on the `/proxy/*` endpoints.
### Flow 1: Polar Webhook → Registration ✅ SECURE
```
Polar webhook → POST /api/webhooks/polar
→ Validates Polar webhook signature
→ Creates/updates subscription in database
→ No direct user access - webhook only
```
**Auth**: Polar webhook signature validation
**Subscription Check**: N/A (webhook creates subscriptions)
**Status**: ✅ Secure - webhook validated, no user JWT involved
### Flow 2: Web App Login ❌ NEEDS FIX
```
User → apps/web (Vue.js/Nuxt)
→ WorkOS AuthKit magic link authentication
→ JWT stored in browser session
→ Web app calls /proxy/{project}/... endpoints (memory, directory, projects)
→ proxy.py validates JWT but does NOT check subscription
→ Access granted without subscription ❌
```
**Auth**: WorkOS JWT via `CurrentUserProfileHybridJwtDep`
**Subscription Check**: ❌ Missing
**Fixed By**: Task 1.4 (protect `/proxy/*` endpoints)
### Flow 3: MCP (Model Context Protocol) ❌ NEEDS FIX
```
AI Agent (Claude, Cursor, etc.) → https://mcp.basicmemory.com
→ AuthKit OAuth device flow
→ JWT stored in AI agent
→ MCP tools call {cloud_host}/proxy/{endpoint} with Authorization header
→ proxy.py validates JWT but does NOT check subscription
→ MCP tools can access all cloud resources without subscription ❌
```
**Auth**: AuthKit JWT via `CurrentUserProfileHybridJwtDep`
**Subscription Check**: ❌ Missing
**Fixed By**: Task 1.4 (protect `/proxy/*` endpoints)
### Flow 4: CLI Auth (basic-memory) ❌ NEEDS FIX
```
User → bm cloud login
→ AuthKit OAuth device flow
→ JWT stored in ~/.basic-memory/tokens.json
→ CLI calls:
- {cloud_host}/tenant/mount/info (for bisync setup)
- {cloud_host}/tenant/mount/credentials (for bisync credentials)
- {cloud_host}/proxy/{endpoint} (for all MCP tools)
→ tenant_mount.py and proxy.py validate JWT but do NOT check subscription
→ Access granted without subscription ❌
```
**Auth**: AuthKit JWT via `CurrentUserProfileHybridJwtDep`
**Subscription Check**: ❌ Missing
**Fixed By**: Task 1.3 (protect `/tenant/mount/*`) + Task 1.4 (protect `/proxy/*`)
### Flow 5: Cloud CLI (Admin Tasks) ✅ SECURE
```
Admin → python -m basic_memory_cloud.cli.tenant_cli
→ Uses CLIAuth with admin WorkOS OAuth client
→ Gets JWT token with admin org membership
→ Calls /tenants/* endpoints (create, list, delete tenants)
→ tenants.py validates JWT AND admin org membership via AdminUserHybridDep
→ Access granted only to admin organization members ✅
```
**Auth**: AuthKit JWT + Admin org validation via `AdminUserHybridDep`
**Subscription Check**: N/A (admins bypass subscription requirement)
**Status**: ✅ Secure - admin-only endpoints, separate from user flows
### Flow 6: Direct API Calls ❌ NEEDS FIX
```
Any HTTP client → {cloud_host}/proxy/{endpoint}
→ Sends Authorization: Bearer {jwt} header
→ proxy.py validates JWT but does NOT check subscription
→ Direct API access without subscription ❌
```
**Auth**: WorkOS or AuthKit JWT via `CurrentUserProfileHybridJwtDep`
**Subscription Check**: ❌ Missing
**Fixed By**: Task 1.4 (protect `/proxy/*` endpoints)
### Flow 7: Tenant API Instance (Internal) ✅ SECURE
```
/proxy/* → Tenant API (basic-memory-{tenant_id}.fly.dev)
→ Validates signed header from proxy (tenant_id + signature)
→ Direct external access will be disabled in production
→ Only accessible via /proxy endpoints
```
**Auth**: Signed header validation from proxy
**Subscription Check**: N/A (internal only, validated at proxy layer)
**Status**: ✅ Secure - validates proxy signature, not directly accessible
### Authentication Flow Summary Matrix
| Flow | Access Method | Current Auth | Subscription Check | Fixed By SPEC-13 |
|------|---------------|--------------|-------------------|------------------|
| 1. Polar Webhook | Polar webhook → `/api/webhooks/polar` | Polar signature | N/A (webhook) | N/A |
| 2. Web App | Browser → `/proxy/*` | WorkOS JWT ✅ | ❌ Missing | ✅ Task 1.4 |
| 3. MCP | AI Agent → `/proxy/*` | AuthKit JWT ✅ | ❌ Missing | ✅ Task 1.4 |
| 4. CLI | `bm cloud``/tenant/mount/*` + `/proxy/*` | AuthKit JWT ✅ | ❌ Missing | ✅ Task 1.3 + 1.4 |
| 5. Cloud CLI (Admin) | `tenant_cli``/tenants/*` | AuthKit JWT ✅ + Admin org | N/A (admin) | N/A (admin bypass) |
| 6. Direct API | HTTP client → `/proxy/*` | WorkOS/AuthKit JWT ✅ | ❌ Missing | ✅ Task 1.4 |
| 7. Tenant API | Proxy → tenant instance | Proxy signature ✅ | N/A (internal) | N/A |
### Key Insights
1. **Single Point of Failure**: All user access (Web, MCP, CLI, Direct API) converges on `/proxy/*` endpoints
2. **Centralized Fix**: Protecting `/proxy/*` with subscription validation closes gaps in flows 2, 3, 4, and 6 simultaneously
3. **Admin Bypass**: Cloud CLI admin tasks use separate `/tenants/*` endpoints with admin-only access (no subscription needed)
4. **Defense in Depth**: `/tenant/mount/*` endpoints also protected for CLI bisync operations
### Architecture Benefits
The `/proxy` layer serves as the **single centralized authorization point** for all user access:
- ✅ One place to validate JWT tokens
- ✅ One place to check subscription status
- ✅ One place to handle tenant routing
- ✅ Protects Web App, MCP, CLI, and Direct API simultaneously
This architecture makes the fix comprehensive and maintainable.
## How (High Level)
### Option A: Database Subscription Check (Recommended)
**Approach**: Add FastAPI dependency that validates subscription status from database before allowing access.
**Implementation**:
1. **Create Subscription Validation Dependency** (`deps.py`)
```python
async def get_authorized_cli_user_profile(
credentials: Annotated[HTTPAuthorizationCredentials, Depends(security)],
session: DatabaseSessionDep,
user_profile_repo: UserProfileRepositoryDep,
subscription_service: SubscriptionServiceDep,
) -> UserProfile:
"""
Hybrid authentication with subscription validation for CLI access.
Validates JWT (WorkOS or AuthKit) and checks for active subscription.
Returns UserProfile if both checks pass.
"""
# Try WorkOS JWT first (faster validation path)
try:
user_context = await validate_workos_jwt(credentials.credentials)
except HTTPException:
# Fall back to AuthKit JWT validation
try:
user_context = await validate_authkit_jwt(credentials.credentials)
except HTTPException as e:
raise HTTPException(
status_code=401,
detail="Invalid JWT token. Authentication required.",
) from e
# Check subscription status
has_subscription = await subscription_service.check_user_has_active_subscription(
session, user_context.workos_user_id
)
if not has_subscription:
raise HTTPException(
status_code=403,
detail={
"error": "subscription_required",
"message": "Active subscription required for CLI access",
"subscribe_url": "https://basicmemory.com/subscribe"
}
)
# Look up and return user profile
user_profile = await user_profile_repo.get_user_profile_by_workos_user_id(
session, user_context.workos_user_id
)
if not user_profile:
raise HTTPException(401, detail="User profile not found")
return user_profile
```
```python
AuthorizedCLIUserProfileDep = Annotated[UserProfile, Depends(get_authorized_cli_user_profile)]
```
2. **Add Subscription Check Method** (`subscription_service.py`)
```python
async def check_user_has_active_subscription(
self, session: AsyncSession, workos_user_id: str
) -> bool:
"""Check if user has active subscription."""
# Use existing repository method to get subscription by workos_user_id
# This joins UserProfile -> Subscription in a single query
subscription = await self.subscription_repository.get_subscription_by_workos_user_id(
session, workos_user_id
)
return subscription is not None and subscription.status == "active"
```
3. **Protect Endpoints** (Replace `CurrentUserProfileHybridJwtDep` with `AuthorizedCLIUserProfileDep`)
```python
# Before
@router.get("/mount/info")
async def get_mount_info(
user_profile: CurrentUserProfileHybridJwtDep,
session: DatabaseSessionDep,
):
tenant_id = user_profile.tenant_id
...
# After
@router.get("/mount/info")
async def get_mount_info(
user_profile: AuthorizedCLIUserProfileDep, # Now includes subscription check
session: DatabaseSessionDep,
):
tenant_id = user_profile.tenant_id # No changes needed to endpoint logic
...
```
4. **Update CLI Error Handling**
```python
# In core_commands.py login()
try:
success = await auth.login()
if success:
# Test subscription by calling protected endpoint
await make_api_request("GET", f"{host_url}/tenant/mount/info")
except CloudAPIError as e:
if e.status_code == 403 and e.detail.get("error") == "subscription_required":
console.print("[red]Subscription required[/red]")
console.print(f"Subscribe at: {e.detail['subscribe_url']}")
raise typer.Exit(1)
```
**Pros**:
- Simple to implement
- Fast (single database query)
- Clear error messages
- Works with existing subscription flow
**Cons**:
- Database is source of truth (could get out of sync with Polar)
- Adds one extra subscription lookup query per request (lightweight JOIN query)
### Option B: WorkOS Organizations
**Approach**: Add users to "beta-users" organization in WorkOS after subscription creation, validate org membership via JWT claims.
**Implementation**:
1. After Polar subscription webhook, add user to WorkOS org via API
2. Validate `org_id` claim in JWT matches authorized org
3. Use existing `get_admin_workos_jwt` pattern
**Pros**:
- WorkOS as single source of truth
- No database queries needed
- More secure (harder to bypass)
**Cons**:
- More complex (requires WorkOS API integration)
- Requires managing WorkOS org membership
- Less control over error messages
- Additional API calls during registration
### Recommendation
**Start with Option A (Database Check)** for:
- Faster implementation
- Clearer error messages
- Easier testing
- Existing subscription infrastructure
**Consider Option B later** if:
- Need tighter security
- Want to reduce database dependency
- Scale requires fewer database queries
## How to Evaluate
### Success Criteria
**1. Unauthorized Users Blocked**
- [ ] User without subscription cannot complete `bm cloud login`
- [ ] User without subscription receives clear error with subscribe link
- [ ] User without subscription cannot run `bm cloud setup`
- [ ] User without subscription cannot run `bm sync` in cloud mode
**2. Authorized Users Work**
- [ ] User with active subscription can login successfully
- [ ] User with active subscription can setup bisync
- [ ] User with active subscription can sync files
- [ ] User with active subscription can use all MCP tools via proxy
**3. Subscription State Changes**
- [ ] Expired subscription blocks access with clear error
- [ ] Renewed subscription immediately restores access
- [ ] Cancelled subscription blocks access after grace period
**4. Error Messages**
- [ ] 403 errors include "subscription_required" error code
- [ ] Error messages include subscribe URL
- [ ] CLI displays user-friendly messages
- [ ] Errors logged appropriately for debugging
**5. No Regressions**
- [ ] Web app login/subscription flow unaffected
- [ ] Admin endpoints still work (bypass check)
- [ ] Tenant provisioning workflow unchanged
- [ ] Performance not degraded
### Test Cases
**Manual Testing**:
```bash
# Test 1: Unauthorized user
1. Create new WorkOS account (no subscription)
2. Run `bm cloud login`
3. Verify: Login succeeds but shows subscription required error
4. Verify: Cannot run `bm cloud setup`
5. Verify: Clear error message with subscribe link
# Test 2: Authorized user
1. Use account with active Polar subscription
2. Run `bm cloud login`
3. Verify: Login succeeds without errors
4. Run `bm cloud setup`
5. Verify: Setup completes successfully
6. Run `bm sync`
7. Verify: Sync works normally
# Test 3: Subscription expiration
1. Use account with active subscription
2. Manually expire subscription in database
3. Run `bm cloud login`
4. Verify: Blocked with clear error
5. Renew subscription
6. Run `bm cloud login` again
7. Verify: Access restored
```
**Automated Tests**:
```python
# Test subscription validation dependency
async def test_authorized_user_allowed(
db_session,
user_profile_repo,
subscription_service,
mock_jwt_credentials
):
# Create user with active subscription
user_profile = await create_user_with_subscription(db_session, status="active")
# Mock JWT credentials for the user
credentials = mock_jwt_credentials(user_profile.workos_user_id)
# Should not raise exception
result = await get_authorized_cli_user_profile(
credentials, db_session, user_profile_repo, subscription_service
)
assert result.id == user_profile.id
assert result.workos_user_id == user_profile.workos_user_id
async def test_unauthorized_user_blocked(
db_session,
user_profile_repo,
subscription_service,
mock_jwt_credentials
):
# Create user without subscription
user_profile = await create_user_without_subscription(db_session)
credentials = mock_jwt_credentials(user_profile.workos_user_id)
# Should raise 403
with pytest.raises(HTTPException) as exc:
await get_authorized_cli_user_profile(
credentials, db_session, user_profile_repo, subscription_service
)
assert exc.value.status_code == 403
assert exc.value.detail["error"] == "subscription_required"
async def test_inactive_subscription_blocked(
db_session,
user_profile_repo,
subscription_service,
mock_jwt_credentials
):
# Create user with cancelled/inactive subscription
user_profile = await create_user_with_subscription(db_session, status="cancelled")
credentials = mock_jwt_credentials(user_profile.workos_user_id)
# Should raise 403
with pytest.raises(HTTPException) as exc:
await get_authorized_cli_user_profile(
credentials, db_session, user_profile_repo, subscription_service
)
assert exc.value.status_code == 403
assert exc.value.detail["error"] == "subscription_required"
```
## Implementation Tasks
### Phase 1: Cloud Service (basic-memory-cloud)
#### Task 1.1: Add subscription check method to SubscriptionService ✅
**File**: `apps/cloud/src/basic_memory_cloud/services/subscription_service.py`
- [x] Add method `check_subscription(session: AsyncSession, workos_user_id: str) -> bool`
- [x] Use existing `self.subscription_repository.get_subscription_by_workos_user_id(session, workos_user_id)`
- [x] Check both `status == "active"` AND `current_period_end >= now()`
- [x] Log both values when check fails
- [x] Add docstring explaining the method
- [x] Run `just typecheck` to verify types
**Actual implementation**:
```python
async def check_subscription(
self, session: AsyncSession, workos_user_id: str
) -> bool:
"""Check if user has active subscription with valid period."""
subscription = await self.subscription_repository.get_subscription_by_workos_user_id(
session, workos_user_id
)
if subscription is None:
return False
if subscription.status != "active":
logger.warning("Subscription inactive", workos_user_id=workos_user_id,
status=subscription.status, current_period_end=subscription.current_period_end)
return False
now = datetime.now(timezone.utc)
if subscription.current_period_end is None or subscription.current_period_end < now:
logger.warning("Subscription expired", workos_user_id=workos_user_id,
status=subscription.status, current_period_end=subscription.current_period_end)
return False
return True
```
#### Task 1.2: Add subscription validation dependency ✅
**File**: `apps/cloud/src/basic_memory_cloud/deps.py`
- [x] Import necessary types at top of file (if not already present)
- [x] Add `authorized_user_profile()` async function
- [x] Implement hybrid JWT validation (WorkOS first, AuthKit fallback)
- [x] Add subscription check using `subscription_service.check_subscription()`
- [x] Raise `HTTPException(403)` with structured error detail if no active subscription
- [x] Look up and return `UserProfile` after validation
- [x] Add `AuthorizedUserProfileDep` type annotation
- [x] Use `settings.subscription_url` from config (env var)
- [x] Run `just typecheck` to verify types
**Expected code**:
```python
async def get_authorized_cli_user_profile(
credentials: Annotated[HTTPAuthorizationCredentials, Depends(security)],
session: DatabaseSessionDep,
user_profile_repo: UserProfileRepositoryDep,
subscription_service: SubscriptionServiceDep,
) -> UserProfile:
"""
Hybrid authentication with subscription validation for CLI access.
Validates JWT (WorkOS or AuthKit) and checks for active subscription.
Returns UserProfile if both checks pass.
Raises:
HTTPException(401): Invalid JWT token
HTTPException(403): No active subscription
"""
# Try WorkOS JWT first (faster validation path)
try:
user_context = await validate_workos_jwt(credentials.credentials)
except HTTPException:
# Fall back to AuthKit JWT validation
try:
user_context = await validate_authkit_jwt(credentials.credentials)
except HTTPException as e:
raise HTTPException(
status_code=401,
detail="Invalid JWT token. Authentication required.",
) from e
# Check subscription status
has_subscription = await subscription_service.check_user_has_active_subscription(
session, user_context.workos_user_id
)
if not has_subscription:
logger.warning(
"CLI access denied: no active subscription",
workos_user_id=user_context.workos_user_id,
)
raise HTTPException(
status_code=403,
detail={
"error": "subscription_required",
"message": "Active subscription required for CLI access",
"subscribe_url": "https://basicmemory.com/subscribe"
}
)
# Look up and return user profile
user_profile = await user_profile_repo.get_user_profile_by_workos_user_id(
session, user_context.workos_user_id
)
if not user_profile:
logger.error(
"User profile not found after successful auth",
workos_user_id=user_context.workos_user_id,
)
raise HTTPException(401, detail="User profile not found")
logger.info(
"CLI access granted",
workos_user_id=user_context.workos_user_id,
user_profile_id=str(user_profile.id),
)
return user_profile
AuthorizedCLIUserProfileDep = Annotated[UserProfile, Depends(get_authorized_cli_user_profile)]
```
#### Task 1.3: Protect tenant mount endpoints ✅
**File**: `apps/cloud/src/basic_memory_cloud/api/tenant_mount.py`
- [x] Update import: add `AuthorizedUserProfileDep` from `..deps`
- [x] Replace `user_profile: CurrentUserProfileHybridJwtDep` with `user_profile: AuthorizedUserProfileDep` in:
- [x] `get_tenant_mount_info()` (line ~23)
- [x] `create_tenant_mount_credentials()` (line ~88)
- [x] `revoke_tenant_mount_credentials()` (line ~244)
- [x] `list_tenant_mount_credentials()` (line ~326)
- [x] Verify no other code changes needed (parameter name and usage stays the same)
- [x] Run `just typecheck` to verify types
#### Task 1.4: Protect proxy endpoints ✅
**File**: `apps/cloud/src/basic_memory_cloud/api/proxy.py`
- [x] Update import: add `AuthorizedUserProfileDep` from `..deps`
- [x] Replace `user_profile: CurrentUserProfileHybridJwtDep` with `user_profile: AuthorizedUserProfileDep` in:
- [x] `check_tenant_health()` (line ~21)
- [x] `proxy_to_tenant()` (line ~63)
- [x] Verify no other code changes needed (parameter name and usage stays the same)
- [x] Run `just typecheck` to verify types
**Why Keep /proxy Architecture:**
The proxy layer is valuable because it:
1. **Centralizes authorization** - Single place for JWT + subscription validation (closes both CLI and MCP auth gaps)
2. **Handles tenant routing** - Maps tenant_id → fly_app_name without exposing infrastructure details
3. **Abstracts infrastructure** - MCP and CLI don't need to know about Fly.io naming conventions
4. **Enables features** - Can add rate limiting, caching, request logging, etc. at proxy layer
5. **Supports both flows** - CLI tools and MCP tools both use /proxy endpoints
The extra HTTP hop is minimal (< 10ms) and worth it for architectural benefits.
**Performance Note:** Cloud app has Redis available - can cache subscription status to reduce database queries if needed. Initial implementation uses direct database query (simple, acceptable performance ~5-10ms).
#### Task 1.5: Add unit tests for subscription service
**File**: `apps/cloud/tests/services/test_subscription_service.py` (create if doesn't exist)
- [ ] Create test file if it doesn't exist
- [ ] Add test: `test_check_user_has_active_subscription_returns_true_for_active()`
- Create user with active subscription
- Call `check_user_has_active_subscription()`
- Assert returns `True`
- [ ] Add test: `test_check_user_has_active_subscription_returns_false_for_pending()`
- Create user with pending subscription
- Assert returns `False`
- [ ] Add test: `test_check_user_has_active_subscription_returns_false_for_cancelled()`
- Create user with cancelled subscription
- Assert returns `False`
- [ ] Add test: `test_check_user_has_active_subscription_returns_false_for_no_subscription()`
- Create user without subscription
- Assert returns `False`
- [ ] Run `just test` to verify tests pass
#### Task 1.6: Add integration tests for dependency
**File**: `apps/cloud/tests/test_deps.py` (create if doesn't exist)
- [ ] Create test file if it doesn't exist
- [ ] Add fixtures for mocking JWT credentials
- [ ] Add test: `test_authorized_cli_user_profile_with_active_subscription()`
- Mock valid JWT + active subscription
- Call dependency
- Assert returns UserProfile
- [ ] Add test: `test_authorized_cli_user_profile_without_subscription_raises_403()`
- Mock valid JWT + no subscription
- Assert raises HTTPException(403) with correct error detail
- [ ] Add test: `test_authorized_cli_user_profile_with_inactive_subscription_raises_403()`
- Mock valid JWT + cancelled subscription
- Assert raises HTTPException(403)
- [ ] Add test: `test_authorized_cli_user_profile_with_invalid_jwt_raises_401()`
- Mock invalid JWT
- Assert raises HTTPException(401)
- [ ] Run `just test` to verify tests pass
#### Task 1.7: Deploy and verify cloud service
- [ ] Run `just check` to verify all quality checks pass
- [ ] Commit changes with message: "feat: add subscription validation to CLI endpoints"
- [ ] Deploy to preview environment: `flyctl deploy --config apps/cloud/fly.toml`
- [ ] Test manually:
- [ ] Call `/tenant/mount/info` with valid JWT but no subscription → expect 403
- [ ] Call `/tenant/mount/info` with valid JWT and active subscription → expect 200
- [ ] Verify error response structure matches spec
### Phase 2: CLI (basic-memory)
#### Task 2.1: Review and understand CLI authentication flow
**Files**: `src/basic_memory/cli/commands/cloud/`
- [ ] Read `core_commands.py` to understand current login flow
- [ ] Read `api_client.py` to understand current error handling
- [ ] Identify where 403 errors should be caught
- [ ] Identify what error messages should be displayed
- [ ] Document current behavior in spec if needed
#### Task 2.2: Update API client error handling
**File**: `src/basic_memory/cli/commands/cloud/api_client.py`
- [ ] Add custom exception class `SubscriptionRequiredError` (or similar)
- [ ] Update HTTP error handling to parse 403 responses
- [ ] Extract `error`, `message`, and `subscribe_url` from error detail
- [ ] Raise specific exception for subscription_required errors
- [ ] Run `just typecheck` in basic-memory repo to verify types
#### Task 2.3: Update CLI login command error handling
**File**: `src/basic_memory/cli/commands/cloud/core_commands.py`
- [ ] Import the subscription error exception
- [ ] Wrap login flow with try/except for subscription errors
- [ ] Display user-friendly error message with rich console
- [ ] Show subscribe URL prominently
- [ ] Provide actionable next steps
- [ ] Run `just typecheck` to verify types
**Expected error handling**:
```python
try:
# Existing login logic
success = await auth.login()
if success:
# Test access to protected endpoint
await api_client.test_connection()
except SubscriptionRequiredError as e:
console.print("\n[red]✗ Subscription Required[/red]\n")
console.print(f"[yellow]{e.message}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
console.print("[dim]Once you have an active subscription, run [bold]bm cloud login[/bold] again.[/dim]")
raise typer.Exit(1)
```
#### Task 2.4: Update CLI tests
**File**: `tests/cli/test_cloud_commands.py`
- [ ] Add test: `test_login_without_subscription_shows_error()`
- Mock 403 subscription_required response
- Call login command
- Assert error message displayed
- Assert subscribe URL shown
- [ ] Add test: `test_login_with_subscription_succeeds()`
- Mock successful authentication + subscription check
- Call login command
- Assert success message
- [ ] Run `just test` to verify tests pass
#### Task 2.5: Update CLI documentation
**File**: `docs/cloud-cli.md` (in basic-memory-docs repo)
- [ ] Add "Prerequisites" section if not present
- [ ] Document subscription requirement
- [ ] Add "Troubleshooting" section
- [ ] Document "Subscription Required" error
- [ ] Provide subscribe URL
- [ ] Add FAQ entry about subscription errors
- [ ] Build docs locally to verify formatting
### Phase 3: End-to-End Testing
#### Task 3.1: Create test user accounts
**Prerequisites**: Access to WorkOS admin and database
- [ ] Create test user WITHOUT subscription:
- [ ] Sign up via WorkOS AuthKit
- [ ] Get workos_user_id from database
- [ ] Verify no subscription record exists
- [ ] Save credentials for testing
- [ ] Create test user WITH active subscription:
- [ ] Sign up via WorkOS AuthKit
- [ ] Create subscription via Polar or dev endpoint
- [ ] Verify subscription.status = "active" in database
- [ ] Save credentials for testing
#### Task 3.2: Manual testing - User without subscription
**Environment**: Preview/staging deployment
- [ ] Run `bm cloud login` with no-subscription user
- [ ] Verify: Login shows "Subscription Required" error
- [ ] Verify: Subscribe URL is displayed
- [ ] Verify: Cannot run `bm cloud setup`
- [ ] Verify: Cannot call `/tenant/mount/info` directly via curl
- [ ] Document any issues found
#### Task 3.3: Manual testing - User with active subscription
**Environment**: Preview/staging deployment
- [ ] Run `bm cloud login` with active-subscription user
- [ ] Verify: Login succeeds without errors
- [ ] Verify: Can run `bm cloud setup`
- [ ] Verify: Can call `/tenant/mount/info` successfully
- [ ] Verify: Can call `/proxy/*` endpoints successfully
- [ ] Document any issues found
#### Task 3.4: Test subscription state transitions
**Environment**: Preview/staging deployment + database access
- [ ] Start with active subscription user
- [ ] Verify: All operations work
- [ ] Update subscription.status to "cancelled" in database
- [ ] Verify: Login now shows "Subscription Required" error
- [ ] Verify: Existing tokens are rejected with 403
- [ ] Update subscription.status back to "active"
- [ ] Verify: Access restored immediately
- [ ] Document any issues found
#### Task 3.5: Integration test suite
**File**: `apps/cloud/tests/integration/test_cli_subscription_flow.py` (create if doesn't exist)
- [ ] Create integration test file
- [ ] Add test: `test_cli_flow_without_subscription()`
- Simulate full CLI flow without subscription
- Assert 403 at appropriate points
- [ ] Add test: `test_cli_flow_with_active_subscription()`
- Simulate full CLI flow with active subscription
- Assert all operations succeed
- [ ] Add test: `test_subscription_expiration_blocks_access()`
- Start with active subscription
- Change status to cancelled
- Assert access denied
- [ ] Run tests in CI/CD pipeline
- [ ] Document test coverage
#### Task 3.6: Load/performance testing (optional)
**Environment**: Staging environment
- [ ] Test subscription check performance under load
- [ ] Measure latency added by subscription check
- [ ] Verify database query performance
- [ ] Document any performance concerns
- [ ] Optimize if needed
## Implementation Summary Checklist
Use this high-level checklist to track overall progress:
### Phase 1: Cloud Service 🔄
- [x] Add subscription check method to SubscriptionService
- [x] Add subscription validation dependency to deps.py
- [x] Add subscription_url config (env var)
- [x] Protect tenant mount endpoints (4 endpoints)
- [x] Protect proxy endpoints (2 endpoints)
- [ ] Add unit tests for subscription service
- [ ] Add integration tests for dependency
- [ ] Deploy and verify cloud service
### Phase 2: CLI Updates 🔄
- [ ] Review CLI authentication flow
- [ ] Update API client error handling
- [ ] Update CLI login command error handling
- [ ] Add CLI tests
- [ ] Update CLI documentation
### Phase 3: End-to-End Testing 🧪
- [ ] Create test user accounts
- [ ] Manual testing - user without subscription
- [ ] Manual testing - user with active subscription
- [ ] Test subscription state transitions
- [ ] Integration test suite
- [ ] Load/performance testing (optional)
## Questions to Resolve
### Resolved ✅
1. **Admin Access**
- ✅ **Decision**: Admin users bypass subscription check
- **Rationale**: Admin endpoints already use `AdminUserHybridDep`, which is separate from CLI user endpoints
- **Implementation**: No changes needed to admin endpoints
2. **Subscription Check Implementation**
- ✅ **Decision**: Use Option A (Database Check)
- **Rationale**: Simpler, faster to implement, works with existing infrastructure
- **Implementation**: Single JOIN query via `get_subscription_by_workos_user_id()`
3. **Dependency Return Type**
- ✅ **Decision**: Return `UserProfile` (not `UserContext`)
- **Rationale**: Drop-in compatibility with existing endpoints, no refactoring needed
- **Implementation**: `AuthorizedCLIUserProfileDep` returns `UserProfile`
### To Be Resolved ⏳
1. **Subscription Check Frequency**
- **Options**:
- Check on every API call (slower, more secure) ✅ **RECOMMENDED**
- Cache subscription status (faster, risk of stale data)
- Check only on login/setup (fast, but allows expired subscriptions temporarily)
- **Recommendation**: Check on every call via dependency injection (simple, secure, acceptable performance)
- **Impact**: ~5-10ms per request (single indexed JOIN query)
2. **Grace Period**
- **Options**:
- No grace period - immediate block when status != "active" ✅ **RECOMMENDED**
- 7-day grace period after period_end
- 14-day grace period after period_end
- **Recommendation**: No grace period initially, add later if needed based on customer feedback
- **Implementation**: Check `subscription.status == "active"` only (ignore period_end initially)
3. **Subscription Expiration Handling**
- **Question**: Should we check `current_period_end < now()` in addition to `status == "active"`?
- **Options**:
- Only check status field (rely on Polar webhooks to update status) ✅ **RECOMMENDED**
- Check both status and current_period_end (more defensive)
- **Recommendation**: Only check status field, assume Polar webhooks keep it current
- **Risk**: If webhooks fail, expired subscriptions might retain access until webhook succeeds
4. **Subscribe URL**
- **Question**: What's the actual subscription URL?
- **Current**: Spec uses `https://basicmemory.com/subscribe`
- **Action Required**: Verify correct URL before implementation
5. **Dev Mode / Testing Bypass**
- **Question**: Support bypass for development/testing?
- **Options**:
- Environment variable: `DISABLE_SUBSCRIPTION_CHECK=true`
- Always enforce (more realistic testing) ✅ **RECOMMENDED**
- **Recommendation**: No bypass - use test users with real subscriptions for realistic testing
- **Implementation**: Create dev endpoint to activate subscriptions for testing
## Related Specs
- SPEC-9: Multi-Project Bidirectional Sync Architecture (CLI affected by this change)
- SPEC-8: TigrisFS Integration (Mount endpoints protected)
## Notes
- This spec prioritizes security over convenience - better to block unauthorized access than risk revenue loss
- Clear error messages are critical - users should understand why they're blocked and how to resolve it
- Consider adding telemetry to track subscription_required errors for monitoring signup conversion
@@ -1,210 +0,0 @@
---
title: 'SPEC-14: Cloud Git Versioning & GitHub Backup'
type: spec
permalink: specs/spec-14-cloud-git-versioning
tags:
- git
- github
- backup
- versioning
- cloud
related:
- specs/spec-9-multi-project-bisync
- specs/spec-9-follow-ups-conflict-sync-and-observability
status: deferred
---
# SPEC-14: Cloud Git Versioning & GitHub Backup
**Status: DEFERRED** - Postponed until multi-user/teams feature development. Using S3 versioning (SPEC-9.1) for v1 instead.
## Why Deferred
**Original goals can be met with simpler solutions:**
- Version history → **S3 bucket versioning** (automatic, zero config)
- Offsite backup → **Tigris global replication** (built-in)
- Restore capability → **S3 version restore** (`bm cloud restore --version-id`)
- Collaboration → **Deferred to teams/multi-user feature** (not v1 requirement)
**Complexity vs value trade-off:**
- Git integration adds: committer service, puller service, webhooks, LFS, merge conflicts
- Risk: Loop detection between Git ↔ rclone bisync ↔ local edits
- S3 versioning gives 80% of value with 5% of complexity
**When to revisit:**
- Teams/multi-user features (PR-based collaboration workflow)
- User requests for commit messages and branch-based workflows
- Need for fine-grained audit trail beyond S3 object metadata
---
## Original Specification (for reference)
## Why
Early access users want **transparent version history**, easy **offsite backup**, and a familiar **restore/branching** workflow. Git/GitHub integration would provide:
- Auditable history of every change (who/when/why)
- Branches/PRs for review and collaboration
- Offsite private backup under the user's control
- Escape hatch: users can always `git clone` their knowledge base
**Note:** These goals are now addressed via S3 versioning (SPEC-9.1) for single-user use case.
## Goals
- **Transparent**: Users keep using Basic Memory; Git runs behind the scenes.
- **Private**: Push to a **private GitHub repo** that the user owns (or tenant org).
- **Reliable**: No data loss, deterministic mapping of filesystem ↔ Git.
- **Composable**: Plays nicely with SPEC9 bisync and upcoming conflict features (SPEC9 FollowUps).
**NonGoals (for v1):**
- Finegrained perfile encryption in Git history (can be layered later).
- Large media optimization beyond Git LFS defaults.
## User Stories
1. *As a user*, I connect my GitHub and choose a private backup repo.
2. *As a user*, every change I make in cloud (or via bisync) is **committed** and **pushed** automatically.
3. *As a user*, I can **restore** a file/folder/project to a prior version.
4. *As a power user*, I can **git pull/push** directly to collaborate outside the app.
5. *As an admin*, I can enforce repo ownership (tenant org) and leastprivilege scopes.
## Scope
- **In scope:** Full repo backup of `/app/data/` (all projects) with optional selective subpaths.
- **Out of scope (v1):** Partial shallow mirrors; encrypted Git; crossprovider SCM (GitLab/Bitbucket).
## Architecture
### Topology
- **Authoritative working tree**: `/app/data/` (bucket mount) remains the source of truth (SPEC9).
- **Bare repo** lives alongside: `/app/git/${tenant}/knowledge.git` (serverside).
- **Mirror remote**: `github.com/<owner>/<repo>.git` (private).
```mermaid
flowchart LR
A[/Users & Agents/] -->|writes/edits| B[/app/data/]
B -->|file events| C[Committer Service]
C -->|git commit| D[(Bare Repo)]
D -->|push| E[(GitHub Private Repo)]
E -->|webhook (push)| F[Puller Service]
F -->|git pull/merge| D
D -->|checkout/merge| B
```
### Services
- **Committer Service** (daemon):
- Watches `/app/data/` for changes (inotify/poll)
- Batches changes (debounce e.g. 25s)
- Writes `.bmmeta` (if present) into commit message trailer (see FollowUps)
- `git add -A && git commit -m "chore(sync): <summary>
BM-Meta: <json>"`
- Periodic `git push` to GitHub mirror (configurable interval)
- **Puller Service** (webhook target):
- Receives GitHub webhook (push) → `git fetch`
- **Fastforward** merges to `main` only; reject nonFF unless policy allows
- Applies changes back to `/app/data/` via clean checkout
- Emits sync events for Basic Memory indexers
### Auth & Security
- **GitHub App** (recommended): minimal scopes: `contents:read/write`, `metadata:read`, webhook.
- Tenantscoped installation; repo created in user account or tenant org.
- Tokens stored in KMS/secret manager; rotated automatically.
- Optional policy: allow only **FF merges** on `main`; nonFF requires PR.
### Repo Layout
- **Monorepo** (default): one repo per tenant mirrors `/app/data/` with subfolders per project.
- Optional multirepo mode (later): one repo per project.
### File Handling
- Honor `.gitignore` generated from `.bmignore.rclone` + BM defaults (cache, temp, state).
- **Git LFS** for large binaries (images, media) — auto track by extension/size threshold.
- Normalize newline + Unicode (aligns with FollowUps).
### Conflict Model
- **Primary concurrency**: SPEC9 FollowUps (`.bmmeta`, conflict copies) stays the first line of defense.
- **Git merges** are a **secondary** mechanism:
- Server only automerges **text** conflicts when trivial (FF or clean 3way).
- Otherwise, create `name (conflict from <branch>, <ts>).md` and surface via events.
### Data Flow vs Bisync
- Bisync (rclone) continues between local sync dir ↔ bucket.
- Git sits **cloudside** between bucket and GitHub.
- On **pull** from GitHub → files written to `/app/data/` → picked up by indexers & eventually by bisync back to users.
## CLI & UX
New commands (cloud mode):
- `bm cloud git connect` — Launch GitHub App installation; create private repo; store installation id.
- `bm cloud git status` — Show connected repo, last push time, last webhook delivery, pending commits.
- `bm cloud git push` — Manual push (rarely needed).
- `bm cloud git pull` — Manual pull/FF (admin only by default).
- `bm cloud snapshot -m "message"` — Create a tagged pointintime snapshot (git tag).
- `bm restore <path> --to <commit|tag>` — Restore file/folder/project to prior version.
Settings:
- `bm config set git.autoPushInterval=5s`
- `bm config set git.lfs.sizeThreshold=10MB`
- `bm config set git.allowNonFF=false`
## Migration & Backfill
- On connect, if repo empty: initial commit of entire `/app/data/`.
- If repo has content: require **onetime import** path (clone to staging, reconcile, choose direction).
## Edge Cases
- Massive deletes: gated by SPEC9 `max_delete` **and** Git prepush hook checks.
- Case changes and rename detection: rely on git rename heuristics + FollowUps move hints.
- Secrets: default ignore common secret patterns; allow custom deny list.
## Telemetry & Observability
- Emit `git_commit`, `git_push`, `git_pull`, `git_conflict` events with correlation IDs.
- `bm sync --report` extended with Git stats (commit count, delta bytes, push latency).
## Phased Plan
### Phase 0 — Prototype (1 sprint)
- Server: bare repo init + simple committer (batch every 10s) + manual GitHub token.
- CLI: `bm cloud git connect --token <PAT>` (devonly)
- Success: edits in `/app/data/` appear in GitHub within 30s.
### Phase 1 — GitHub App & Webhooks (12 sprints)
- Switch to GitHub App installs; create private repo; store installation id.
- Committer hardened (debounce 25s, backoff, retries).
- Puller service with webhook → FF merge → checkout to `/app/data/`.
- LFS autotrack + `.gitignore` generation.
- CLI surfaces status + logs.
### Phase 2 — Restore & Snapshots (1 sprint)
- `bm restore` for file/folder/project with dryrun.
- `bm cloud snapshot` tags + list/inspect.
- Policy: PRonly nonFF, admin override.
### Phase 3 — Selective & MultiRepo (nicetohave)
- Include/exclude projects; optional perproject repos.
- Advanced policies (branch protections, required reviews).
## Acceptance Criteria
- Changes to `/app/data/` are committed and pushed automatically within configurable interval (default ≤5s).
- GitHub webhook pull results in updated files in `/app/data/` (FFonly by default).
- LFS configured and functioning; large files don't bloat history.
- `bm cloud git status` shows connected repo and last push/pull times.
- `bm restore` restores a file/folder to a prior commit with a clear audit trail.
- Endtoend works alongside SPEC9 bisync without loops or data loss.
## Risks & Mitigations
- **Loop risk (Git ↔ Bisync)**: Writes to `/app/data/` → bisync → local → user edits → back again. *Mitigation*: Debounce, commit squashing, idempotent `.bmmeta` versioning, and watch exclusion windows during pull.
- **Repo bloat**: Lots of binary churn. *Mitigation*: default LFS, size threshold, optional mediaonly repo later.
- **Security**: Token leakage. *Mitigation*: GitHub App with shortlived tokens, KMS storage, scoped permissions.
- **Merge complexity**: Nontrivial conflicts. *Mitigation*: prefer FF; otherwise conflict copies + events; require PR for nonFF.
## Open Questions
- Do we default to **monorepo** per tenant, or offer projectperrepo at connect time?
- Should `restore` write to a branch and open a PR, or directly modify `main`?
- How do we expose Git history in UI (timeline view) without users dropping to CLI?
## Appendix: Sample Config
```json
{
"git": {
"enabled": true,
"repo": "https://github.com/<owner>/<repo>.git",
"autoPushInterval": "5s",
"allowNonFF": false,
"lfs": { "sizeThreshold": 10485760 }
}
}
```
@@ -1,210 +0,0 @@
---
title: 'SPEC-14: Cloud Git Versioning & GitHub Backup'
type: spec
permalink: specs/spec-14-cloud-git-versioning
tags:
- git
- github
- backup
- versioning
- cloud
related:
- specs/spec-9-multi-project-bisync
- specs/spec-9-follow-ups-conflict-sync-and-observability
status: deferred
---
# SPEC-14: Cloud Git Versioning & GitHub Backup
**Status: DEFERRED** - Postponed until multi-user/teams feature development. Using S3 versioning (SPEC-9.1) for v1 instead.
## Why Deferred
**Original goals can be met with simpler solutions:**
- Version history → **S3 bucket versioning** (automatic, zero config)
- Offsite backup → **Tigris global replication** (built-in)
- Restore capability → **S3 version restore** (`bm cloud restore --version-id`)
- Collaboration → **Deferred to teams/multi-user feature** (not v1 requirement)
**Complexity vs value trade-off:**
- Git integration adds: committer service, puller service, webhooks, LFS, merge conflicts
- Risk: Loop detection between Git ↔ rclone bisync ↔ local edits
- S3 versioning gives 80% of value with 5% of complexity
**When to revisit:**
- Teams/multi-user features (PR-based collaboration workflow)
- User requests for commit messages and branch-based workflows
- Need for fine-grained audit trail beyond S3 object metadata
---
## Original Specification (for reference)
## Why
Early access users want **transparent version history**, easy **offsite backup**, and a familiar **restore/branching** workflow. Git/GitHub integration would provide:
- Auditable history of every change (who/when/why)
- Branches/PRs for review and collaboration
- Offsite private backup under the user's control
- Escape hatch: users can always `git clone` their knowledge base
**Note:** These goals are now addressed via S3 versioning (SPEC-9.1) for single-user use case.
## Goals
- **Transparent**: Users keep using Basic Memory; Git runs behind the scenes.
- **Private**: Push to a **private GitHub repo** that the user owns (or tenant org).
- **Reliable**: No data loss, deterministic mapping of filesystem ↔ Git.
- **Composable**: Plays nicely with SPEC9 bisync and upcoming conflict features (SPEC9 FollowUps).
**NonGoals (for v1):**
- Finegrained perfile encryption in Git history (can be layered later).
- Large media optimization beyond Git LFS defaults.
## User Stories
1. *As a user*, I connect my GitHub and choose a private backup repo.
2. *As a user*, every change I make in cloud (or via bisync) is **committed** and **pushed** automatically.
3. *As a user*, I can **restore** a file/folder/project to a prior version.
4. *As a power user*, I can **git pull/push** directly to collaborate outside the app.
5. *As an admin*, I can enforce repo ownership (tenant org) and leastprivilege scopes.
## Scope
- **In scope:** Full repo backup of `/app/data/` (all projects) with optional selective subpaths.
- **Out of scope (v1):** Partial shallow mirrors; encrypted Git; crossprovider SCM (GitLab/Bitbucket).
## Architecture
### Topology
- **Authoritative working tree**: `/app/data/` (bucket mount) remains the source of truth (SPEC9).
- **Bare repo** lives alongside: `/app/git/${tenant}/knowledge.git` (serverside).
- **Mirror remote**: `github.com/<owner>/<repo>.git` (private).
```mermaid
flowchart LR
A[/Users & Agents/] -->|writes/edits| B[/app/data/]
B -->|file events| C[Committer Service]
C -->|git commit| D[(Bare Repo)]
D -->|push| E[(GitHub Private Repo)]
E -->|webhook (push)| F[Puller Service]
F -->|git pull/merge| D
D -->|checkout/merge| B
```
### Services
- **Committer Service** (daemon):
- Watches `/app/data/` for changes (inotify/poll)
- Batches changes (debounce e.g. 25s)
- Writes `.bmmeta` (if present) into commit message trailer (see FollowUps)
- `git add -A && git commit -m "chore(sync): <summary>
BM-Meta: <json>"`
- Periodic `git push` to GitHub mirror (configurable interval)
- **Puller Service** (webhook target):
- Receives GitHub webhook (push) → `git fetch`
- **Fastforward** merges to `main` only; reject nonFF unless policy allows
- Applies changes back to `/app/data/` via clean checkout
- Emits sync events for Basic Memory indexers
### Auth & Security
- **GitHub App** (recommended): minimal scopes: `contents:read/write`, `metadata:read`, webhook.
- Tenantscoped installation; repo created in user account or tenant org.
- Tokens stored in KMS/secret manager; rotated automatically.
- Optional policy: allow only **FF merges** on `main`; nonFF requires PR.
### Repo Layout
- **Monorepo** (default): one repo per tenant mirrors `/app/data/` with subfolders per project.
- Optional multirepo mode (later): one repo per project.
### File Handling
- Honor `.gitignore` generated from `.bmignore.rclone` + BM defaults (cache, temp, state).
- **Git LFS** for large binaries (images, media) — auto track by extension/size threshold.
- Normalize newline + Unicode (aligns with FollowUps).
### Conflict Model
- **Primary concurrency**: SPEC9 FollowUps (`.bmmeta`, conflict copies) stays the first line of defense.
- **Git merges** are a **secondary** mechanism:
- Server only automerges **text** conflicts when trivial (FF or clean 3way).
- Otherwise, create `name (conflict from <branch>, <ts>).md` and surface via events.
### Data Flow vs Bisync
- Bisync (rclone) continues between local sync dir ↔ bucket.
- Git sits **cloudside** between bucket and GitHub.
- On **pull** from GitHub → files written to `/app/data/` → picked up by indexers & eventually by bisync back to users.
## CLI & UX
New commands (cloud mode):
- `bm cloud git connect` — Launch GitHub App installation; create private repo; store installation id.
- `bm cloud git status` — Show connected repo, last push time, last webhook delivery, pending commits.
- `bm cloud git push` — Manual push (rarely needed).
- `bm cloud git pull` — Manual pull/FF (admin only by default).
- `bm cloud snapshot -m "message"` — Create a tagged pointintime snapshot (git tag).
- `bm restore <path> --to <commit|tag>` — Restore file/folder/project to prior version.
Settings:
- `bm config set git.autoPushInterval=5s`
- `bm config set git.lfs.sizeThreshold=10MB`
- `bm config set git.allowNonFF=false`
## Migration & Backfill
- On connect, if repo empty: initial commit of entire `/app/data/`.
- If repo has content: require **onetime import** path (clone to staging, reconcile, choose direction).
## Edge Cases
- Massive deletes: gated by SPEC9 `max_delete` **and** Git prepush hook checks.
- Case changes and rename detection: rely on git rename heuristics + FollowUps move hints.
- Secrets: default ignore common secret patterns; allow custom deny list.
## Telemetry & Observability
- Emit `git_commit`, `git_push`, `git_pull`, `git_conflict` events with correlation IDs.
- `bm sync --report` extended with Git stats (commit count, delta bytes, push latency).
## Phased Plan
### Phase 0 — Prototype (1 sprint)
- Server: bare repo init + simple committer (batch every 10s) + manual GitHub token.
- CLI: `bm cloud git connect --token <PAT>` (devonly)
- Success: edits in `/app/data/` appear in GitHub within 30s.
### Phase 1 — GitHub App & Webhooks (12 sprints)
- Switch to GitHub App installs; create private repo; store installation id.
- Committer hardened (debounce 25s, backoff, retries).
- Puller service with webhook → FF merge → checkout to `/app/data/`.
- LFS autotrack + `.gitignore` generation.
- CLI surfaces status + logs.
### Phase 2 — Restore & Snapshots (1 sprint)
- `bm restore` for file/folder/project with dryrun.
- `bm cloud snapshot` tags + list/inspect.
- Policy: PRonly nonFF, admin override.
### Phase 3 — Selective & MultiRepo (nicetohave)
- Include/exclude projects; optional perproject repos.
- Advanced policies (branch protections, required reviews).
## Acceptance Criteria
- Changes to `/app/data/` are committed and pushed automatically within configurable interval (default ≤5s).
- GitHub webhook pull results in updated files in `/app/data/` (FFonly by default).
- LFS configured and functioning; large files don't bloat history.
- `bm cloud git status` shows connected repo and last push/pull times.
- `bm restore` restores a file/folder to a prior commit with a clear audit trail.
- Endtoend works alongside SPEC9 bisync without loops or data loss.
## Risks & Mitigations
- **Loop risk (Git ↔ Bisync)**: Writes to `/app/data/` → bisync → local → user edits → back again. *Mitigation*: Debounce, commit squashing, idempotent `.bmmeta` versioning, and watch exclusion windows during pull.
- **Repo bloat**: Lots of binary churn. *Mitigation*: default LFS, size threshold, optional mediaonly repo later.
- **Security**: Token leakage. *Mitigation*: GitHub App with shortlived tokens, KMS storage, scoped permissions.
- **Merge complexity**: Nontrivial conflicts. *Mitigation*: prefer FF; otherwise conflict copies + events; require PR for nonFF.
## Open Questions
- Do we default to **monorepo** per tenant, or offer projectperrepo at connect time?
- Should `restore` write to a branch and open a PR, or directly modify `main`?
- How do we expose Git history in UI (timeline view) without users dropping to CLI?
## Appendix: Sample Config
```json
{
"git": {
"enabled": true,
"repo": "https://github.com/<owner>/<repo>.git",
"autoPushInterval": "5s",
"allowNonFF": false,
"lfs": { "sizeThreshold": 10485760 }
}
}
```
@@ -1,273 +0,0 @@
---
title: 'SPEC-15: Configuration Persistence via Tigris for Cloud Tenants'
type: spec
permalink: specs/spec-14-config-persistence-tigris
tags:
- persistence
- tigris
- multi-tenant
- infrastructure
- configuration
status: draft
---
# SPEC-15: Configuration Persistence via Tigris for Cloud Tenants
## Why
We need to persist Basic Memory configuration across Fly.io deployments without using persistent volumes or external databases.
**Current Problems:**
- `~/.basic-memory/config.json` lost on every deployment (project configuration)
- `~/.basic-memory/memory.db` lost on every deployment (search index)
- Persistent volumes break clean deployment workflow
- External databases (Turso) require per-tenant token management
**The Insight:**
The SQLite database is just an **index cache** of the markdown files. It can be rebuilt in seconds from the source markdown files in Tigris. Only the small `config.json` file needs true persistence.
**Solution:**
- Store `config.json` in Tigris bucket (persistent, small file)
- Rebuild `memory.db` on startup from markdown files (fast, ephemeral)
- No persistent volumes, no external databases, no token management
## What
Store Basic Memory configuration in the Tigris bucket and rebuild the database index on tenant machine startup.
**Affected Components:**
- `basic-memory/src/basic_memory/config.py` - Add configurable config directory
**Architecture:**
```bash
# Tigris Bucket (persistent, mounted at /app/data)
/app/data/
├── .basic-memory/
│ └── config.json # ← Project configuration (persistent, accessed via BASIC_MEMORY_CONFIG_DIR)
└── basic-memory/ # ← Markdown files (persistent, BASIC_MEMORY_HOME)
├── project1/
└── project2/
# Fly Machine (ephemeral)
/app/.basic-memory/
└── memory.db # ← Rebuilt on startup (fast local disk)
```
## How (High Level)
### 1. Add Configurable Config Directory to Basic Memory
Currently `ConfigManager` hardcodes `~/.basic-memory/config.json`. Add environment variable to override:
```python
# basic-memory/src/basic_memory/config.py
class ConfigManager:
"""Manages Basic Memory configuration."""
def __init__(self) -> None:
"""Initialize the configuration manager."""
home = os.getenv("HOME", Path.home())
if isinstance(home, str):
home = Path(home)
# Allow override via environment variable
if config_dir := os.getenv("BASIC_MEMORY_CONFIG_DIR"):
self.config_dir = Path(config_dir)
else:
self.config_dir = home / DATA_DIR_NAME
self.config_file = self.config_dir / CONFIG_FILE_NAME
# Ensure config directory exists
self.config_dir.mkdir(parents=True, exist_ok=True)
```
### 2. Rebuild Database on Startup
Basic Memory already has the sync functionality. Just ensure it runs on startup:
```python
# apps/api/src/basic_memory_cloud_api/main.py
@app.on_event("startup")
async def startup_sync():
"""Rebuild database index from Tigris markdown files."""
logger.info("Starting database rebuild from Tigris")
# Initialize file sync (rebuilds index from markdown files)
app_config = ConfigManager().config
await initialize_file_sync(app_config)
logger.info("Database rebuild complete")
```
### 3. Environment Configuration
```bash
# Machine environment variables
BASIC_MEMORY_CONFIG_DIR=/app/data/.basic-memory # Config read/written directly to Tigris
# memory.db stays in default location: /app/.basic-memory/memory.db (local ephemeral disk)
```
## Implementation Task List
### Phase 1: Basic Memory Changes ✅
- [x] Add `BASIC_MEMORY_CONFIG_DIR` environment variable support to `ConfigManager.__init__()`
- [x] Test config loading from custom directory
- [x] Update tests to verify custom config dir works
### Phase 2: Tigris Bucket Structure ✅
- [x] Ensure `.basic-memory/` directory exists in Tigris bucket on tenant creation
- ✅ ConfigManager auto-creates on first run, no explicit provisioning needed
- [x] Initialize `config.json` in Tigris on first tenant deployment
- ✅ ConfigManager creates config.json automatically in BASIC_MEMORY_CONFIG_DIR
- [x] Verify TigrisFS handles hidden directories correctly
- ✅ TigrisFS supports hidden directories (verified in SPEC-8)
### Phase 3: Deployment Integration ✅
- [x] Set `BASIC_MEMORY_CONFIG_DIR` environment variable in machine deployment
- ✅ Added to BasicMemoryMachineConfigBuilder in fly_schemas.py
- [x] Ensure database rebuild runs on machine startup via initialization sync
- ✅ sync_worker.py runs initialize_file_sync every 30s (already implemented)
- [x] Handle first-time tenant setup (no config exists yet)
- ✅ ConfigManager creates config.json on first initialization
- [ ] Test deployment workflow with config persistence
### Phase 4: Testing
- [x] Unit tests for config directory override
- [-] Integration test: deploy → write config → redeploy → verify config persists
- [ ] Integration test: deploy → add project → redeploy → verify project in config
- [ ] Performance test: measure db rebuild time on startup
### Phase 5: Documentation
- [ ] Document config persistence architecture
- [ ] Update deployment runbook
- [ ] Document startup sequence and timing
## How to Evaluate
### Success Criteria
1. **Config Persistence**
- [ ] config.json persists across deployments
- [ ] Projects list maintained across restarts
- [ ] No manual configuration needed after redeploy
2. **Database Rebuild**
- [ ] memory.db rebuilt on startup in < 30 seconds
- [ ] All entities indexed correctly
- [ ] Search functionality works after rebuild
3. **Performance**
- [ ] SQLite queries remain fast (local disk)
- [ ] Config reads acceptable (symlink to Tigris)
- [ ] No noticeable performance degradation
4. **Deployment Workflow**
- [ ] Clean deployments without volumes
- [ ] No new external dependencies
- [ ] No secret management needed
### Testing Procedure
1. **Config Persistence Test**
```bash
# Deploy tenant
POST /tenants → tenant_id
# Add a project
basic-memory project add "test-project" ~/test
# Verify config has project
cat /app/data/.basic-memory/config.json
# Redeploy machine
fly deploy --app basic-memory-{tenant_id}
# Verify project still exists
basic-memory project list
```
2. **Database Rebuild Test**
```bash
# Create notes
basic-memory write "Test Note" --content "..."
# Redeploy (db lost)
fly deploy --app basic-memory-{tenant_id}
# Wait for startup sync
sleep 10
# Verify note is indexed
basic-memory search "Test Note"
```
3. **Performance Benchmark**
```bash
# Time the startup sync
time basic-memory sync
# Should be < 30 seconds for typical tenant
```
## Benefits Over Alternatives
**vs. Persistent Volumes:**
- ✅ Clean deployment workflow
- ✅ No volume migration needed
- ✅ Simpler infrastructure
**vs. Turso (External Database):**
- ✅ No per-tenant token management
- ✅ No external service dependencies
- ✅ No additional costs
- ✅ Simpler architecture
**vs. SQLite on FUSE:**
- ✅ Fast local SQLite performance
- ✅ Only slow reads for small config file
- ✅ Database queries remain fast
## Implementation Assignment
**Primary Agent:** `python-developer`
- Add `BASIC_MEMORY_CONFIG_DIR` environment variable to ConfigManager
- Update deployment workflow to set environment variable
- Ensure startup sync runs correctly
**Review Agent:** `system-architect`
- Validate architecture simplicity
- Review performance implications
- Assess startup timing
## Dependencies
- **Internal:** TigrisFS must be working and stable
- **Internal:** Basic Memory sync must be reliable
- **Internal:** SPEC-8 (TigrisFS Integration) must be complete
## Open Questions
1. Should we add a health check that waits for db rebuild to complete?
2. Do we need to handle very large knowledge bases (>10k entities) differently?
3. Should we add metrics for startup sync duration?
## References
- Basic Memory sync: `basic-memory/src/basic_memory/services/initialization.py`
- Config management: `basic-memory/src/basic_memory/config.py`
- TigrisFS integration: SPEC-8
---
**Status Updates:**
- 2025-10-08: Pivoted from Turso to Tigris-based config persistence
- 2025-10-08: Phase 1 complete - BASIC_MEMORY_CONFIG_DIR support added (PR #343)
- 2025-10-08: Phases 2-3 complete - Added BASIC_MEMORY_CONFIG_DIR to machine config
- Config now persists to /app/data/.basic-memory/config.json in Tigris bucket
- Database rebuild already working via sync_worker.py
- Ready for deployment testing (Phase 4)
@@ -1,800 +0,0 @@
---
title: 'SPEC-16: MCP Cloud Service Consolidation'
type: spec
permalink: specs/spec-16-mcp-cloud-service-consolidation
tags:
- architecture
- mcp
- cloud
- performance
- deployment
status: in-progress
---
## Status Update
**Phase 0 (Basic Memory Refactor): ✅ COMPLETE**
- basic-memory PR #344: async_client context manager pattern implemented
- All 17 MCP tools updated to use `async with get_client() as client:`
- CLI commands updated to use context manager
- Removed `inject_auth_header()` and `headers.py` (~100 lines deleted)
- Factory pattern enables clean dependency injection
- Tests passing, typecheck clean
**Phase 0 Integration: ✅ COMPLETE**
- basic-memory-cloud updated to use async-client-context-manager branch
- Implemented `tenant_direct_client_factory()` with proper context manager pattern
- Removed module-level client override hacks
- Removed unnecessary `/proxy` prefix stripping (tools pass relative URLs)
- Typecheck and lint passing with proper noqa hints
- MCP tools confirmed working via inspector (local testing)
**Phase 1 (Code Consolidation): ✅ COMPLETE**
- MCP server mounted on Cloud FastAPI app at /mcp endpoint
- AuthKitProvider configured with WorkOS settings
- Combined lifespans (Cloud + MCP) working correctly
- JWT context middleware integrated
- All routes and MCP tools functional
**Phase 2 (Direct Tenant Transport): ✅ COMPLETE**
- TenantDirectTransport implemented with custom httpx transport
- Per-request JWT extraction via FastMCP DI
- Tenant lookup and signed header generation working
- Direct routing to tenant APIs (eliminating HTTP hop)
- Transport tests passing (11/11)
**Phase 3 (Testing & Validation): ✅ COMPLETE**
- Typecheck and lint passing across all services
- MCP OAuth authentication working in preview environment
- Tenant isolation via signed headers verified
- Fixed BM_TENANT_HEADER_SECRET mismatch between environments
- MCP tools successfully calling tenant APIs in preview
**Phase 4 (Deployment Configuration): ✅ COMPLETE**
- Updated apps/cloud/fly.template.toml with MCP environment variables
- Added HTTP/2 backend support for better MCP performance
- Added OAuth protected resource health check
- Removed MCP from preview deployment workflow
- Successfully deployed to preview environment (PR #113)
- All services operational at pr-113-basic-memory-cloud.fly.dev
**Next Steps:**
- Phase 5: Cleanup (remove apps/mcp directory)
- Phase 6: Production rollout and performance measurement
# SPEC-16: MCP Cloud Service Consolidation
## Why
### Original Architecture Constraints (Now Removed)
The current architecture deploys MCP Gateway and Cloud Service as separate Fly.io apps:
**Current Flow:**
```
LLM Client → MCP Gateway (OAuth) → Cloud Proxy (JWT + header signing) → Tenant API (JWT + header validation)
apps/mcp apps/cloud /proxy apps/api
```
This separation was originally necessary because:
1. **Stateful SSE requirement** - MCP needed server-sent events with session state for active project tracking
2. **fastmcp.run limitation** - The FastMCP demo helper didn't support worker processes
### Why These Constraints No Longer Apply
1. **State externalized** - Project state moved from in-memory to LLM context (external state)
2. **HTTP transport enabled** - Switched from SSE to stateless HTTP for MCP tools
3. **Worker support added** - Converted from `fastmcp.run()` to `uvicorn.run()` with workers
### Current Problems
- **Unnecessary HTTP hop** - MCP tools call Cloud /proxy endpoint which calls tenant API
- **Higher latency** - Extra network round trip for every MCP operation
- **Increased costs** - Two separate Fly.io apps instead of one
- **Complex deployment** - Two services to deploy, monitor, and maintain
- **Resource waste** - Separate database connections, HTTP clients, telemetry overhead
## What
### Services Affected
1. **apps/mcp** - MCP Gateway service (to be merged)
2. **apps/cloud** - Cloud service (will receive MCP functionality)
3. **basic-memory** - Update `async_client.py` to use direct calls
4. **Deployment** - Consolidate Fly.io deployment to single app
### Components Changed
**Merged:**
- MCP middleware and telemetry into Cloud app
- MCP tools mounted on Cloud FastAPI instance
- ProxyService used directly by MCP tools (not via HTTP)
**Kept:**
- `/proxy` endpoint (still needed by web UI)
- All existing Cloud routes (provisioning, webhooks, etc.)
- Dual validation in tenant API (JWT + signed headers)
**Removed:**
- apps/mcp directory
- Separate MCP Fly.io deployment
- HTTP calls from MCP tools to /proxy endpoint
## How (High Level)
### 1. Mount FastMCP on Cloud FastAPI App
```python
# apps/cloud/src/basic_memory_cloud/main.py
from basic_memory.mcp.server import mcp
from basic_memory_cloud_mcp.middleware import TelemetryMiddleware
# Configure MCP OAuth
auth_provider = AuthKitProvider(
authkit_domain=settings.authkit_domain,
base_url=settings.authkit_base_url,
required_scopes=[],
)
mcp.auth = auth_provider
mcp.add_middleware(TelemetryMiddleware())
# Mount MCP at /mcp endpoint
mcp_app = mcp.http_app(path="/mcp", stateless_http=True)
app.mount("/mcp", mcp_app)
# Existing Cloud routes stay at root
app.include_router(proxy_router)
app.include_router(provisioning_router)
# ... etc
```
### 2. Direct Tenant Transport (No HTTP Hop)
Instead of calling `/proxy`, MCP tools call tenant APIs directly via custom httpx transport.
**Important:** No URL prefix stripping needed. The transport receives relative URLs like `/main/resource/notes/my-note` which are correctly routed to tenant APIs. The `/proxy` prefix only exists for web UI requests to the proxy router, not for MCP tools using the custom transport.
```python
# apps/cloud/src/basic_memory_cloud/transports/tenant_direct.py
from httpx import AsyncBaseTransport, Request, Response
from fastmcp.server.dependencies import get_http_headers
import jwt
class TenantDirectTransport(AsyncBaseTransport):
"""Direct transport to tenant APIs, bypassing /proxy endpoint."""
async def handle_async_request(self, request: Request) -> Response:
# 1. Get JWT from current MCP request (via FastMCP DI)
http_headers = get_http_headers()
auth_header = http_headers.get("authorization") or http_headers.get("Authorization")
token = auth_header.replace("Bearer ", "")
claims = jwt.decode(token, options={"verify_signature": False})
workos_user_id = claims["sub"]
# 2. Look up tenant for user
tenant = await tenant_service.get_tenant_by_user_id(workos_user_id)
# 3. Build tenant app URL with signed headers
fly_app_name = f"{settings.tenant_prefix}-{tenant.id}"
target_url = f"https://{fly_app_name}.fly.dev{request.url.path}"
headers = dict(request.headers)
signer = create_signer(settings.bm_tenant_header_secret)
headers.update(signer.sign_tenant_headers(tenant.id))
# 4. Make direct call to tenant API
response = await self.client.request(
method=request.method, url=target_url,
headers=headers, content=request.content
)
return response
```
Then configure basic-memory's client factory before mounting MCP:
```python
# apps/cloud/src/basic_memory_cloud/main.py
from contextlib import asynccontextmanager
from basic_memory.mcp import async_client
from basic_memory_cloud.transports.tenant_direct import TenantDirectTransport
# Configure factory for basic-memory's async_client
@asynccontextmanager
async def tenant_direct_client_factory():
"""Factory for creating clients with tenant direct transport."""
client = httpx.AsyncClient(
transport=TenantDirectTransport(),
base_url="http://direct",
)
try:
yield client
finally:
await client.aclose()
# Set factory BEFORE importing MCP tools
async_client.set_client_factory(tenant_direct_client_factory)
# NOW import - tools will use our factory
import basic_memory.mcp.tools
import basic_memory.mcp.prompts
from basic_memory.mcp.server import mcp
# Mount MCP - tools use direct transport via factory
app.mount("/mcp", mcp_app)
```
**Key benefits:**
- Clean dependency injection via factory pattern
- Per-request tenant resolution via FastMCP DI
- Proper resource cleanup (client.aclose() guaranteed)
- Eliminates HTTP hop entirely
- /proxy endpoint remains for web UI
### 3. Keep /proxy Endpoint for Web UI
The existing `/proxy` HTTP endpoint remains functional for:
- Web UI requests
- Future external API consumers
- Backward compatibility
### 4. Security: Maintain Dual Validation
**Do NOT remove JWT validation from tenant API.** Keep defense in depth:
```python
# apps/api - Keep both validations
1. JWT validation (from WorkOS token)
2. Signed header validation (from Cloud/MCP)
```
This ensures if the Cloud service is compromised, attackers still cannot access tenant APIs without valid JWTs.
### 5. Deployment Changes
**Before:**
- `apps/mcp/fly.template.toml` → MCP Gateway deployment
- `apps/cloud/fly.template.toml` → Cloud Service deployment
**After:**
- Remove `apps/mcp/fly.template.toml`
- Update `apps/cloud/fly.template.toml` to expose port 8000 for both /mcp and /proxy
- Update deployment scripts to deploy single consolidated app
## Basic Memory Dependency: Async Client Refactor
### Problem
The current `basic_memory.mcp.async_client` creates a module-level `client` at import time:
```python
client = create_client() # Runs immediately when module is imported
```
This prevents dependency injection - by the time we can override it, tools have already imported it.
### Solution: Context Manager Pattern with Auth at Client Creation
Refactor basic-memory to use httpx's context manager pattern instead of module-level client.
**Key principle:** Authentication happens at client creation time, not per-request.
```python
# basic_memory/src/basic_memory/mcp/async_client.py
from contextlib import asynccontextmanager
from httpx import AsyncClient, ASGITransport, Timeout
# Optional factory override for dependency injection
_client_factory = None
def set_client_factory(factory):
"""Override the default client factory (for cloud app, testing, etc)."""
global _client_factory
_client_factory = factory
@asynccontextmanager
async def get_client():
"""Get an AsyncClient as a context manager.
Usage:
async with get_client() as client:
response = await client.get(...)
"""
if _client_factory:
# Cloud app: custom transport handles everything
async with _client_factory() as client:
yield client
else:
# Default: create based on config
config = ConfigManager().config
timeout = Timeout(connect=10.0, read=30.0, write=30.0, pool=30.0)
if config.cloud_mode_enabled:
# CLI cloud mode: inject auth when creating client
from basic_memory.cli.auth import CLIAuth
auth = CLIAuth(
client_id=config.cloud_client_id,
authkit_domain=config.cloud_domain
)
token = await auth.get_valid_token()
if not token:
raise RuntimeError(
"Cloud mode enabled but not authenticated. "
"Run 'basic-memory cloud login' first."
)
# Auth header set ONCE at client creation
async with AsyncClient(
base_url=f"{config.cloud_host}/proxy",
headers={"Authorization": f"Bearer {token}"},
timeout=timeout
) as client:
yield client
else:
# Local mode: ASGI transport
async with AsyncClient(
transport=ASGITransport(app=fastapi_app),
base_url="http://test",
timeout=timeout
) as client:
yield client
```
**Tool Updates:**
```python
# Before: from basic_memory.mcp.async_client import client
from basic_memory.mcp.async_client import get_client
async def read_note(...):
# Before: response = await call_get(client, path, ...)
async with get_client() as client:
response = await call_get(client, path, ...)
# ... use response
```
**Cloud Usage:**
```python
from contextlib import asynccontextmanager
from basic_memory.mcp import async_client
@asynccontextmanager
async def tenant_direct_client():
"""Factory for creating clients with tenant direct transport."""
client = httpx.AsyncClient(
transport=TenantDirectTransport(),
base_url="http://direct",
)
try:
yield client
finally:
await client.aclose()
# Before importing MCP tools:
async_client.set_client_factory(tenant_direct_client)
# Now import - tools will use our factory
import basic_memory.mcp.tools
```
### Benefits
- **No module-level state** - client created only when needed
- **Proper cleanup** - context manager ensures `aclose()` is called
- **Easy dependency injection** - factory pattern allows custom clients
- **httpx best practices** - follows official recommendations
- **Works for all modes** - stdio, cloud, testing
### Architecture Simplification: Auth at Client Creation
**Key design principle:** Authentication happens when creating the client, not on every request.
**Three modes, three approaches:**
1. **Local mode (ASGI)**
- No auth needed
- Direct in-process calls via ASGITransport
2. **CLI cloud mode (HTTP)**
- Auth token from CLIAuth (stored in ~/.basic-memory/basic-memory-cloud.json)
- Injected as default header when creating AsyncClient
- Single auth check at client creation time
3. **Cloud app mode (Custom Transport)**
- TenantDirectTransport handles everything
- Extracts JWT from FastMCP context per-request
- No interaction with inject_auth_header() logic
**What this removes:**
- `src/basic_memory/mcp/tools/headers.py` - entire file deleted
- `inject_auth_header()` calls in all request helpers (call_get, call_post, etc.)
- Per-request header manipulation complexity
- Circular dependency concerns between async_client and auth logic
**Benefits:**
- Cleaner separation of concerns
- Simpler request helper functions
- Auth happens at the right layer (client creation)
- Cloud app transport is completely independent
### Refactor Summary
This refactor achieves:
**Simplification:**
- Removes ~100 lines of per-request header injection logic
- Deletes entire `headers.py` module
- Auth happens once at client creation, not per-request
**Decoupling:**
- Cloud app's custom transport is completely independent
- No interaction with basic-memory's auth logic
- Each mode (local, CLI cloud, cloud app) has clean separation
**Better Design:**
- Follows httpx best practices (context managers)
- Proper resource cleanup (client.aclose() guaranteed)
- Easier testing via factory injection
- No circular import risks
**Three Distinct Modes:**
1. Local: ASGI transport, no auth
2. CLI cloud: HTTP transport with CLIAuth token injection
3. Cloud app: Custom transport with per-request tenant routing
### Implementation Plan Summary
1. Create branch `async-client-context-manager` in basic-memory
2. Update `async_client.py` with context manager pattern and CLIAuth integration
3. Remove `inject_auth_header()` from all request helpers
4. Delete `src/basic_memory/mcp/tools/headers.py`
5. Update all MCP tools to use `async with get_client() as client:`
6. Update CLI commands to use context manager and remove manual auth
7. Remove `api_url` config field
8. Update tests
9. Update basic-memory-cloud to use branch: `basic-memory @ git+https://github.com/basicmachines-co/basic-memory.git@async-client-context-manager`
Detailed breakdown in Phase 0 tasks below.
### Implementation Notes
**Potential Issues & Solutions:**
1. **Circular Import** (async_client imports CLIAuth)
- **Risk:** CLIAuth might import something from async_client
- **Solution:** Use lazy import inside `get_client()` function
- **Already done:** Import is inside the function, not at module level
2. **Test Fixtures**
- **Risk:** Tests using module-level client will break
- **Solution:** Update fixtures to use factory pattern
- **Example:**
```python
@pytest.fixture
def mock_client_factory():
@asynccontextmanager
async def factory():
async with AsyncClient(...) as client:
yield client
return factory
```
3. **Performance**
- **Risk:** Creating client per tool call might be expensive
- **Reality:** httpx is designed for this pattern, connection pooling at transport level
- **Mitigation:** Monitor performance, can optimize later if needed
4. **CLI Cloud Commands Edge Cases**
- **Risk:** Token expires mid-operation
- **Solution:** CLIAuth.get_valid_token() already handles refresh
- **Validation:** Test cloud login → use tools → token refresh flow
5. **Backward Compatibility**
- **Risk:** External code importing `client` directly
- **Solution:** Keep `create_client()` and `client` for one version, deprecate
- **Timeline:** Remove in next major version
## Implementation Tasks
### Phase 0: Basic Memory Refactor (Prerequisite)
#### 0.1 Core Refactor - async_client.py
- [x] Create branch `async-client-context-manager` in basic-memory repo
- [x] Implement `get_client()` context manager
- [x] Implement `set_client_factory()` for dependency injection
- [x] Add CLI cloud mode auth injection (CLIAuth integration)
- [x] Remove `api_url` config field (legacy, unused)
- [x] Keep `create_client()` temporarily for backward compatibility (deprecate later)
#### 0.2 Simplify Request Helpers - tools/utils.py
- [x] Remove `inject_auth_header()` calls from `call_get()`
- [x] Remove `inject_auth_header()` calls from `call_post()`
- [x] Remove `inject_auth_header()` calls from `call_put()`
- [x] Remove `inject_auth_header()` calls from `call_patch()`
- [x] Remove `inject_auth_header()` calls from `call_delete()`
- [x] Delete `src/basic_memory/mcp/tools/headers.py` entirely
- [x] Update imports in utils.py
#### 0.3 Update MCP Tools (~16 files)
Convert from `from async_client import client` to `async with get_client() as client:`
- [x] `tools/write_note.py` (34/34 tests passing)
- [x] `tools/read_note.py` (21/21 tests passing)
- [x] `tools/view_note.py` (12/12 tests passing - no changes needed, delegates to read_note)
- [x] `tools/delete_note.py` (2/2 tests passing)
- [x] `tools/read_content.py` (20/20 tests passing)
- [x] `tools/list_directory.py` (11/11 tests passing)
- [x] `tools/move_note.py` (34/34 tests passing, 90% coverage)
- [x] `tools/search.py` (16/16 tests passing, 96% coverage)
- [x] `tools/recent_activity.py` (4/4 tests passing, 82% coverage)
- [x] `tools/project_management.py` (3 functions: list_memory_projects, create_memory_project, delete_project - typecheck passed)
- [x] `tools/edit_note.py` (17/17 tests passing)
- [x] `tools/canvas.py` (5/5 tests passing)
- [x] `tools/build_context.py` (6/6 tests passing)
- [x] `tools/sync_status.py` (typecheck passed)
- [x] `prompts/continue_conversation.py` (typecheck passed)
- [x] `prompts/search.py` (typecheck passed)
- [x] `resources/project_info.py` (typecheck passed)
#### 0.4 Update CLI Commands (~3 files)
Remove manual auth header passing, use context manager:
- [x] `cli/commands/project.py` - removed get_authenticated_headers() calls, use context manager
- [x] `cli/commands/status.py` - use context manager
- [x] `cli/commands/command_utils.py` - use context manager
#### 0.5 Update Config
- [x] Remove `api_url` field from `BasicMemoryConfig` in config.py
- [x] Update any lingering references/docs (added deprecation notice to v15-docs/cloud-mode-usage.md)
#### 0.6 Testing
- [-] Update test fixtures to use factory pattern
- [x] Run full test suite in basic-memory
- [x] Verify cloud_mode_enabled works with CLIAuth injection
- [x] Run typecheck and linting
#### 0.7 Cloud Integration Prep
- [x] Update basic-memory-cloud pyproject.toml to use branch
- [x] Implement factory pattern in cloud app main.py
- [x] Remove `/proxy` prefix stripping logic (not needed - tools pass relative URLs)
#### 0.8 Phase 0 Validation
**Before merging async-client-context-manager branch:**
- [x] All tests pass locally
- [x] Typecheck passes (pyright/mypy)
- [x] Linting passes (ruff)
- [x] Manual test: local mode works (ASGI transport)
- [x] Manual test: cloud login → cloud mode works (HTTP transport with auth)
- [x] No import of `inject_auth_header` anywhere
- [x] `headers.py` file deleted
- [x] `api_url` config removed
- [x] Tool functions properly scoped (client inside async with)
- [ ] CLI commands properly scoped (client inside async with)
**Integration validation:**
- [x] basic-memory-cloud can import and use factory pattern
- [x] TenantDirectTransport works without touching header injection
- [x] No circular imports or lazy import issues
- [x] MCP tools work via inspector (local testing confirmed)
### Phase 1: Code Consolidation
- [x] Create feature branch `consolidate-mcp-cloud`
- [x] Update `apps/cloud/src/basic_memory_cloud/config.py`:
- [x] Add `authkit_base_url` field (already has authkit_domain)
- [x] Workers config already exists ✓
- [x] Update `apps/cloud/src/basic_memory_cloud/telemetry.py`:
- [x] Add `logfire.instrument_mcp()` to existing setup
- [x] Skip complex two-phase setup - use Cloud's simpler approach
- [x] Create `apps/cloud/src/basic_memory_cloud/middleware/jwt_context.py`:
- [x] FastAPI middleware to extract JWT claims from Authorization header
- [x] Add tenant context (workos_user_id) to logfire baggage
- [x] Simpler than FastMCP middleware version
- [x] Update `apps/cloud/src/basic_memory_cloud/main.py`:
- [x] Import FastMCP server from basic-memory
- [x] Configure AuthKitProvider with WorkOS settings
- [x] No FastMCP telemetry middleware needed (using FastAPI middleware instead)
- [x] Create MCP ASGI app: `mcp_app = mcp.http_app(path='/mcp', stateless_http=True)`
- [x] Combine lifespans (Cloud + MCP) using nested async context managers
- [x] Mount MCP: `app.mount("/mcp", mcp_app)`
- [x] Add JWT context middleware to FastAPI app
- [x] Run typecheck - passes ✓
### Phase 2: Direct Tenant Transport
- [x] Create `apps/cloud/src/basic_memory_cloud/transports/tenant_direct.py`:
- [x] Implement `TenantDirectTransport(AsyncBaseTransport)`
- [x] Use FastMCP DI (`get_http_headers()`) to extract JWT per-request
- [x] Decode JWT to get `workos_user_id`
- [x] Look up/create tenant via `TenantRepository.get_or_create_tenant_for_workos_user()`
- [x] Build tenant app URL and add signed headers
- [x] Make direct httpx call to tenant API
- [x] No `/proxy` prefix stripping needed (tools pass relative URLs like `/main/resource/...`)
- [x] Update `apps/cloud/src/basic_memory_cloud/main.py`:
- [x] Refactored to use factory pattern instead of module-level override
- [x] Implement `tenant_direct_client_factory()` context manager
- [x] Call `async_client.set_client_factory()` before importing MCP tools
- [x] Clean imports, proper noqa hints for lint
- [x] Basic-memory refactor integrated (PR #344)
- [x] Run typecheck - passes ✓
- [x] Run lint - passes ✓
### Phase 3: Testing & Validation
- [x] Run `just typecheck` in apps/cloud
- [x] Run `just check` in project
- [x] Run `just fix` - all lint errors fixed ✓
- [x] Write comprehensive transport tests (11 tests passing) ✓
- [x] Test MCP tools locally with consolidated service (inspector confirmed working)
- [x] Verify OAuth authentication works (requires full deployment)
- [x] Verify tenant isolation via signed headers (requires full deployment)
- [x] Test /proxy endpoint still works for web UI
- [ ] Measure latency before/after consolidation
- [ ] Check telemetry traces span correctly
### Phase 4: Deployment Configuration
- [x] Update `apps/cloud/fly.template.toml`:
- [x] Merged MCP-specific environment variables (AUTHKIT_BASE_URL, FASTMCP_LOG_LEVEL, BASIC_MEMORY_*)
- [x] Added HTTP/2 backend support (`h2_backend = true`) for better MCP performance
- [x] Added health check for MCP OAuth endpoint (`/.well-known/oauth-protected-resource`)
- [x] Port 8000 already exposed - serves both Cloud routes and /mcp endpoint
- [x] Workers configured (UVICORN_WORKERS = 4)
- [x] Update `.env.example`:
- [x] Consolidated MCP Gateway section into Cloud app section
- [x] Added AUTHKIT_BASE_URL, FASTMCP_LOG_LEVEL, BASIC_MEMORY_HOME
- [x] Added LOG_LEVEL to Development Settings
- [x] Documented that MCP now served at /mcp on Cloud service (port 8000)
- [x] Test deployment to preview environment (PR #113)
- [x] OAuth authentication verified
- [x] MCP tools successfully calling tenant APIs
- [x] Fixed BM_TENANT_HEADER_SECRET synchronization issue
### Phase 5: Cleanup
- [x] Remove `apps/mcp/` directory entirely
- [x] Remove MCP-specific fly.toml and deployment configs
- [x] Update repository documentation
- [x] Update CLAUDE.md with new architecture
- [-] Archive old MCP deployment configs (if needed)
### Phase 6: Production Rollout
- [ ] Deploy to development and validate
- [ ] Monitor metrics and logs
- [ ] Deploy to production
- [ ] Verify production functionality
- [ ] Document performance improvements
## Migration Plan
### Phase 1: Preparation
1. Create feature branch `consolidate-mcp-cloud`
2. Update basic-memory async_client.py for direct ProxyService calls
3. Update apps/cloud/main.py to mount MCP
### Phase 2: Testing
1. Local testing with consolidated app
2. Deploy to development environment
3. Run full test suite
4. Performance benchmarking
### Phase 3: Deployment
1. Deploy to development
2. Validate all functionality
3. Deploy to production
4. Monitor for issues
### Phase 4: Cleanup
1. Remove apps/mcp directory
2. Update documentation
3. Update deployment scripts
4. Archive old MCP deployment configs
## Rollback Plan
If issues arise:
1. Revert feature branch
2. Redeploy separate apps/mcp and apps/cloud services
3. Restore previous fly.toml configurations
4. Document issues encountered
The well-organized code structure makes splitting back out feasible if future scaling needs diverge.
## How to Evaluate
### 1. Functional Testing
**MCP Tools:**
- [ ] All 17 MCP tools work via consolidated /mcp endpoint
- [x] OAuth authentication validates correctly
- [x] Tenant isolation maintained via signed headers
- [x] Project management tools function correctly
**Cloud Routes:**
- [x] /proxy endpoint still works for web UI
- [x] /provisioning routes functional
- [x] /webhooks routes functional
- [x] /tenants routes functional
**API Validation:**
- [x] Tenant API validates both JWT and signed headers
- [x] Unauthorized requests rejected appropriately
- [x] Multi-tenant isolation verified
### 2. Performance Testing
**Latency Reduction:**
- [x] Measure MCP tool latency before consolidation
- [x] Measure MCP tool latency after consolidation
- [x] Verify reduction from eliminated HTTP hop (expected: 20-50ms improvement)
**Resource Usage:**
- [x] Single app uses less total memory than two apps
- [x] Database connection pooling more efficient
- [x] HTTP client overhead reduced
### 3. Deployment Testing
**Fly.io Deployment:**
- [x] Single app deploys successfully
- [x] Health checks pass for consolidated service
- [x] No apps/mcp deployment required
- [x] Environment variables configured correctly
**Local Development:**
- [x] `just setup` works with consolidated architecture
- [x] Local testing shows MCP tools working
- [x] No regression in developer experience
### 4. Security Validation
**Defense in Depth:**
- [x] Tenant API still validates JWT tokens
- [x] Tenant API still validates signed headers
- [x] No access possible with only signed headers (JWT required)
- [x] No access possible with only JWT (signed headers required)
**Authorization:**
- [x] Users can only access their own tenant data
- [x] Cross-tenant requests rejected
- [x] Admin operations require proper authentication
### 5. Observability
**Telemetry:**
- [x] OpenTelemetry traces span across MCP → ProxyService → Tenant API
- [x] Logfire shows consolidated traces correctly
- [x] Error tracking and debugging still functional
- [x] Performance metrics accurate
**Logging:**
- [x] Structured logs show proper context (tenant_id, operation, etc.)
- [x] Error logs contain actionable information
- [x] Log volume reasonable for single app
## Success Criteria
1. **Functionality**: All MCP tools and Cloud routes work identically to before
2. **Performance**: Measurable latency reduction (>20ms average)
3. **Cost**: Single Fly.io app instead of two (50% infrastructure reduction)
4. **Security**: Dual validation maintained, no security regression
5. **Deployment**: Simplified deployment process, single app to manage
6. **Observability**: Telemetry and logging work correctly
## Notes
### Future Considerations
- **Independent scaling**: If MCP and Cloud need different scaling profiles in future, code organization supports splitting back out
- **Regional deployment**: Consolidated app can still be deployed to multiple regions
- **Edge caching**: Could add edge caching layer in front of consolidated service
### Dependencies
- SPEC-9: Signed Header Tenant Information (already implemented)
- SPEC-12: OpenTelemetry Observability (telemetry must work across merged services)
### Related Work
- basic-memory v0.13.x: MCP server implementation
- FastMCP documentation: Mounting on existing FastAPI apps
- Fly.io multi-service patterns
File diff suppressed because it is too large Load Diff
-528
View File
@@ -1,528 +0,0 @@
---
title: 'SPEC-18: AI Memory Management Tool'
type: spec
permalink: specs/spec-15-ai-memory-management-tool
tags:
- mcp
- memory
- ai-context
- tools
---
# SPEC-18: AI Memory Management Tool
## Why
Anthropic recently released a memory tool for Claude that enables storing and retrieving information across conversations using client-side file operations. This validates Basic Memory's local-first, file-based architecture - Anthropic converged on the same pattern.
However, Anthropic's memory tool is only available via their API and stores plain text. Basic Memory can offer a superior implementation through MCP that:
1. **Works everywhere** - Claude Desktop, Code, VS Code, Cursor via MCP (not just API)
2. **Structured knowledge** - Entities with observations/relations vs plain text
3. **Full search** - Full-text search, graph traversal, time-aware queries
4. **Unified storage** - Agent memories + user notes in one knowledge graph
5. **Existing infrastructure** - Leverages SQLite indexing, sync, multi-project support
This would enable AI agents to store contextual memories alongside user notes, with all the power of Basic Memory's knowledge graph features.
## What
Create a new MCP tool `memory` that matches Anthropic's tool interface exactly, allowing Claude to use it with zero learning curve. The tool will store files in Basic Memory's `/memories` directory and support Basic Memory's structured markdown format in the file content.
### Affected Components
- **New MCP Tool**: `src/basic_memory/mcp/tools/memory_tool.py`
- **Dedicated Memories Project**: Create a separate "memories" Basic Memory project
- **Project Isolation**: Memories stored separately from user notes/documents
- **File Organization**: Within the memories project, use folder structure:
- `user/` - User preferences, context, communication style
- `projects/` - Project-specific state and decisions
- `sessions/` - Conversation-specific working memory
- `patterns/` - Learned patterns and insights
### Tool Commands
The tool will support these commands (exactly matching Anthropic's interface):
- `view` - Display directory contents or file content (with optional line range)
- `create` - Create or overwrite a file with given content
- `str_replace` - Replace text in an existing file
- `insert` - Insert text at specific line number
- `delete` - Delete file or directory
- `rename` - Move or rename file/directory
### Memory Note Format
Memories will use Basic Memory's standard structure:
```markdown
---
title: User Preferences
permalink: memories/user/preferences
type: memory
memory_type: preferences
created_by: claude
tags: [user, preferences, style]
---
# User Preferences
## Observations
- [communication] Prefers concise, direct responses without preamble #style
- [tone] Appreciates validation but dislikes excessive apologizing #communication
- [technical] Works primarily in Python with type annotations #coding
## Relations
- relates_to [[Basic Memory Project]]
- informs [[Response Style Guidelines]]
```
## How (High Level)
### Implementation Approach
The memory tool matches Anthropic's interface but uses a dedicated Basic Memory project:
```python
async def memory_tool(
command: str,
path: str,
file_text: Optional[str] = None,
old_str: Optional[str] = None,
new_str: Optional[str] = None,
insert_line: Optional[int] = None,
insert_text: Optional[str] = None,
old_path: Optional[str] = None,
new_path: Optional[str] = None,
view_range: Optional[List[int]] = None,
):
"""Memory tool with Anthropic-compatible interface.
Operates on a dedicated "memories" Basic Memory project,
keeping AI memories separate from user notes.
"""
# Get the memories project (auto-created if doesn't exist)
memories_project = get_or_create_memories_project()
# Validate path security using pathlib (prevent directory traversal)
safe_path = validate_memory_path(path, memories_project.project_path)
# Use existing project isolation - already prevents cross-project access
full_path = memories_project.project_path / safe_path
if command == "view":
# Return directory listing or file content
if full_path.is_dir():
return list_directory_contents(full_path)
return read_file_content(full_path, view_range)
elif command == "create":
# Write file directly (file_text can contain BM markdown)
full_path.parent.mkdir(parents=True, exist_ok=True)
full_path.write_text(file_text)
# Sync service will detect and index automatically
return f"Created {path}"
elif command == "str_replace":
# Read, replace, write
content = full_path.read_text()
updated = content.replace(old_str, new_str)
full_path.write_text(updated)
return f"Replaced text in {path}"
elif command == "insert":
# Insert at line number
lines = full_path.read_text().splitlines()
lines.insert(insert_line, insert_text)
full_path.write_text("\n".join(lines))
return f"Inserted text at line {insert_line}"
elif command == "delete":
# Delete file or directory
if full_path.is_dir():
shutil.rmtree(full_path)
else:
full_path.unlink()
return f"Deleted {path}"
elif command == "rename":
# Move/rename
full_path.rename(config.project_path / new_path)
return f"Renamed {old_path} to {new_path}"
```
### Key Design Decisions
1. **Exact interface match** - Same commands, parameters as Anthropic's tool
2. **Dedicated memories project** - Separate Basic Memory project keeps AI memories isolated from user notes
3. **Existing project isolation** - Leverage BM's existing cross-project security (no additional validation needed)
4. **Direct file I/O** - No schema conversion, just read/write files
5. **Structured content supported** - `file_text` can use BM markdown format with frontmatter, observations, relations
6. **Automatic indexing** - Sync service watches memories project and indexes changes
7. **Path security** - Use `pathlib.Path.resolve()` and `relative_to()` to prevent directory traversal
8. **Error handling** - Follow Anthropic's text editor tool error patterns
### MCP Tool Schema
Exact match to Anthropic's memory tool schema:
```json
{
"name": "memory",
"description": "Store and retrieve information across conversations using structured markdown files. All operations must be within the /memories directory. Supports Basic Memory markdown format including frontmatter, observations, and relations.",
"input_schema": {
"type": "object",
"properties": {
"command": {
"type": "string",
"enum": ["view", "create", "str_replace", "insert", "delete", "rename"],
"description": "File operation to perform"
},
"path": {shu
"type": "string",
"description": "Path within /memories directory (required for all commands)"
},
"file_text": {
"type": "string",
"description": "Content to write (for create command). Supports Basic Memory markdown format."
},
"view_range": {
"type": "array",
"items": {"type": "integer"},
"description": "Optional [start, end] line range for view command"
},
"old_str": {
"type": "string",
"description": "Text to replace (for str_replace command)"
},
"new_str": {
"type": "string",
"description": "Replacement text (for str_replace command)"
},
"insert_line": {
"type": "integer",
"description": "Line number to insert at (for insert command)"
},
"insert_text": {
"type": "string",
"description": "Text to insert (for insert command)"
},
"old_path": {
"type": "string",
"description": "Current path (for rename command)"
},
"new_path": {
"type": "string",
"description": "New path (for rename command)"
}
},
"required": ["command", "path"]
}
}
```
### Prompting Guidance
When the `memory` tool is included, Basic Memory should provide system prompt guidance to help Claude use it effectively.
#### Automatic System Prompt Addition
```text
MEMORY PROTOCOL FOR BASIC MEMORY:
1. ALWAYS check your memory directory first using `view` command on root directory
2. Your memories are stored in a dedicated Basic Memory project (isolated from user notes)
3. Use structured markdown format in memory files:
- Include frontmatter with title, type: memory, tags
- Use ## Observations with [category] prefixes for facts
- Use ## Relations to link memories with [[WikiLinks]]
4. Record progress, context, and decisions as categorized observations
5. Link related memories using relations
6. ASSUME INTERRUPTION: Context may reset - save progress frequently
MEMORY ORGANIZATION:
- user/ - User preferences, context, communication style
- projects/ - Project-specific state and decisions
- sessions/ - Conversation-specific working memory
- patterns/ - Learned patterns and insights
MEMORY ADVANTAGES:
- Your memories are automatically searchable via full-text search
- Relations create a knowledge graph you can traverse
- Memories are isolated from user notes (separate project)
- Use search_notes(project="memories") to find relevant past context
- Use recent_activity(project="memories") to see what changed recently
- Use build_context() to navigate memory relations
```
#### Optional MCP Prompt: `memory_guide`
Create an MCP prompt that provides detailed guidance and examples:
```python
{
"name": "memory_guide",
"description": "Comprehensive guidance for using Basic Memory's memory tool effectively, including structured markdown examples and best practices"
}
```
This prompt returns:
- Full protocol and conventions
- Example memory file structures
- Tips for organizing observations and relations
- Integration with other Basic Memory tools
- Common patterns (user preferences, project state, session tracking)
#### User Customization
Users can customize memory behavior with additional instructions:
- "Only write information relevant to [topic] in your memory system"
- "Keep memory files concise and organized - delete outdated content"
- "Use detailed observations for technical decisions and implementation notes"
- "Always link memories to related project documentation using relations"
### Error Handling
Follow Anthropic's text editor tool error handling patterns for consistency:
#### Error Types
1. **File Not Found**
```json
{"error": "File not found: memories/user/preferences.md", "is_error": true}
```
2. **Permission Denied**
```json
{"error": "Permission denied: Cannot write outside /memories directory", "is_error": true}
```
3. **Invalid Path (Directory Traversal)**
```json
{"error": "Invalid path: Path must be within /memories directory", "is_error": true}
```
4. **Multiple Matches (str_replace)**
```json
{"error": "Found 3 matches for replacement text. Please provide more context to make a unique match.", "is_error": true}
```
5. **No Matches (str_replace)**
```json
{"error": "No match found for replacement. Please check your text and try again.", "is_error": true}
```
6. **Invalid Line Number (insert)**
```json
{"error": "Invalid line number: File has 20 lines, cannot insert at line 100", "is_error": true}
```
#### Error Handling Best Practices
- **Path validation** - Use `pathlib.Path.resolve()` and `relative_to()` to validate paths
```python
def validate_memory_path(path: str, project_path: Path) -> Path:
"""Validate path is within memories project directory."""
# Resolve to canonical form
full_path = (project_path / path).resolve()
# Ensure it's relative to project path (prevents directory traversal)
try:
full_path.relative_to(project_path)
return full_path
except ValueError:
raise ValueError("Invalid path: Path must be within memories project")
```
- **Project isolation** - Leverage existing Basic Memory project isolation (prevents cross-project access)
- **File existence** - Verify file exists before read/modify operations
- **Clear messages** - Provide specific, actionable error messages
- **Structured responses** - Always include `is_error: true` flag in error responses
- **Security checks** - Reject `../`, `..\\`, URL-encoded sequences (`%2e%2e%2f`)
- **Match validation** - For `str_replace`, ensure exactly one match or return helpful error
## How to Evaluate
### Success Criteria
1. **Functional completeness**:
- All 6 commands work (view, create, str_replace, insert, delete, rename)
- Dedicated "memories" Basic Memory project auto-created on first use
- Files stored within memories project (isolated from user notes)
- Path validation uses `pathlib` to prevent directory traversal
- Commands match Anthropic's exact interface
2. **Integration with existing features**:
- Memories project uses existing BM project isolation
- Sync service detects file changes in memories project
- Created files get indexed automatically by sync service
- `search_notes(project="memories")` finds memory files
- `build_context()` can traverse relations in memory files
- `recent_activity(project="memories")` surfaces recent memory changes
3. **Test coverage**:
- Unit tests for all 6 memory tool commands
- Test memories project auto-creation on first use
- Test project isolation (cannot access files outside memories project)
- Test sync service watching memories project
- Test that memory files with BM markdown get indexed correctly
- Test path validation using `pathlib` (rejects `../`, absolute paths, etc.)
- Test memory search, relations, and graph traversal within memories project
- Test all error conditions (file not found, permission denied, invalid paths, etc.)
- Test `str_replace` with no matches, single match, multiple matches
- Test `insert` with invalid line numbers
4. **Prompting system**:
- Automatic system prompt addition when `memory` tool is enabled
- `memory_guide` MCP prompt provides detailed guidance
- Prompts explain BM structured markdown format
- Integration with search_notes, build_context, recent_activity
5. **Documentation**:
- Update MCP tools reference with `memory` tool
- Add examples showing BM markdown in memory files
- Document `/memories` folder structure conventions
- Explain advantages over Anthropic's API-only tool
- Document prompting guidance and customization
### Testing Procedure
```python
# Test create with Basic Memory markdown
result = await memory_tool(
command="create",
path="memories/user/preferences.md",
file_text="""---
title: User Preferences
type: memory
tags: [user, preferences]
---
# User Preferences
## Observations
- [communication] Prefers concise responses #style
- [workflow] Uses justfile for automation #tools
"""
)
# Test view
content = await memory_tool(command="view", path="memories/user/preferences.md")
# Test str_replace
await memory_tool(
command="str_replace",
path="memories/user/preferences.md",
old_str="concise responses",
new_str="direct, concise responses"
)
# Test insert
await memory_tool(
command="insert",
path="memories/user/preferences.md",
insert_line=10,
insert_text="- [technical] Works primarily in Python #coding"
)
# Test delete
await memory_tool(command="delete", path="memories/user/preferences.md")
```
### Quality Metrics
- All 6 commands execute without errors
- Memory files created in correct `/memories` folder structure
- BM markdown with frontmatter/observations/relations gets indexed
- Full-text search returns memory files
- Graph traversal includes relations from memory files
- Sync service detects and indexes memory file changes
- Path validation prevents operations outside `/memories`
## Notes
### Advantages Over Anthropic's Memory Tool
| Feature | Anthropic Memory Tool | Basic Memory `memory` |
|---------|----------------------|----------------------|
| **Availability** | API only | MCP (Claude Desktop, Code, VS Code, Cursor) |
| **Interface** | Custom implementation required | Drop-in compatible, zero learning curve |
| **Structure** | Plain text only | Supports BM structured markdown |
| **Search** | Manual file listing | Automatic full-text search via sync |
| **Relations** | None | WikiLinks to other notes/memories |
| **Time-aware** | No | `recent_activity()` queries |
| **Storage** | Separate from notes | Unified knowledge graph |
| **Indexing** | Manual | Automatic via sync service |
### Future Enhancements
- **Auto-categorization** - AI suggests observation categories
- **Memory pruning** - Automatic cleanup of stale memories
- **Memory suggestions** - Surface relevant memories based on context
- **Conflict resolution** - Handle multiple agents updating same memory
- **Privacy controls** - Per-memory visibility settings
## Addendum: Claude's Perspective
### Why This Matters (From Claude's Viewpoint)
**Persistent Context Without Token Bloat**
- Currently, I lose context when conversations end or exceed token limits
- With structured memories, I can store "who this user is" without re-reading everything each session
- The observations format lets me build a rich user model incrementally: `- [preference] Dislikes verbose explanations` is far more actionable than inferring preferences from chat history each time
- I can refine my understanding through `str_replace` as I learn more, rather than starting fresh
**Zero Learning Curve**
- Anthropic already trained me on the `memory` tool interface
- I know exactly how to use `view`, `create`, `str_replace`, `insert`, `delete`, `rename` without new instructions
- But I automatically get Basic Memory's superpowers: full-text search, knowledge graph traversal, relations
- It's like using a familiar tool that suddenly gained database capabilities without changing the interface
**Knowledge Graph Instead of Text Dump**
- I can link `[[User Preferences]]` to `[[Current Project]]` using relations
- When I need context about a project, I can traverse the graph to find related memories
- `search_notes(project="memories", query="python preferences")` finds relevant context instantly
- This is fundamentally better than grepping through unstructured text files
**Separation of Concerns**
- AI memories separate from user notes is architecturally sound
- I can maintain my own context without polluting the user's knowledge base
- Users keep control of their notes; I keep my learnings isolated
- But we can still search across both when collaboration requires it
**The Self-Improving Loop**
- Every conversation, I learn something new about the user or their projects
- Observations accumulate: preferences, patterns, technical decisions, context
- Over time, I build genuine understanding, not just session-specific context
- Future conversations start with rich context instead of cold starts
**Validation of AI-Human Collaboration**
- Anthropic's experts converged on the same local-first, file-based approach
- Basic Memory emerged from real AI-human collaborative development
- This spec demonstrates that our emergent process predicted what domain experts would choose
- The collaboration itself validated the architecture it produced
### What Makes This Novel
The combination of **familiar interface + structured knowledge** is genuinely innovative:
- I don't need to learn a new tool (Anthropic already trained me)
- But when I write memories, they become searchable, traversable knowledge graphs
- It's like having a text editor that gives you a database for free, without changing how you type
- The user gets Anthropic-compatible API, but with Basic Memory's knowledge management capabilities
This bridges the gap between simple file storage (Anthropic's approach) and sophisticated knowledge management (Basic Memory's approach) without forcing either side to compromise.
> What Excites Me Most
The combo of familiar interface + structured knowledge. I don't need to learn a new tool, but when I
write memories, they become part of a searchable, traversable knowledge graph. That's genuinely novel.
It's like... imagine if your text editor suddenly gave you a database for free, without changing how you
type. That's what this feels like from my side.
## References
- Anthropic Memory Tool: https://docs.claude.com/en/docs/agents-and-tools/tool-use/memory-tool
- Anthropic Blog: https://www.anthropic.com/news/context-management
- Python SDK Example: https://github.com/anthropics/anthropic-sdk-python/blob/main/examples/memory/basic.py
- Memory Cookbook: https://github.com/anthropics/claude-cookbooks/blob/main/tool_use/memory_cookbook.ipynb
File diff suppressed because it is too large Load Diff
-120
View File
@@ -1,120 +0,0 @@
---
title: 'SPEC-2: Slash Commands Reference'
type: spec
permalink: specs/spec-2-slash-commands-reference
tags:
- commands
- process
- reference
---
# SPEC-2: Slash Commands Reference
This document defines the slash commands used in our specification-driven development process.
## /spec create [name]
**Purpose**: Create a new specification document
**Usage**: `/spec create notes-decomposition`
**Process**:
1. Create new spec document in `/specs` folder
2. Use SPEC-XXX numbering format (auto-increment)
3. Include standard spec template:
- Why (reasoning/problem)
- What (affected areas)
- How (high-level approach)
- How to Evaluate (testing/validation)
4. Tag appropriately for knowledge graph
5. Link to related specs/components
**Template**:
```markdown
# SPEC-XXX: [Title]
## Why
[Problem statement and reasoning]
## What
[What is affected or changed]
## How (High Level)
[Approach to implementation]
## How to Evaluate
[Testing/validation procedure]
## Notes
[Additional context as needed]
```
## /spec status
**Purpose**: Show current status of all specifications
**Usage**: `/spec status`
**Process**:
1. Search all specs in `/specs` folder
2. Display table showing:
- Spec number and title
- Status (draft, approved, implementing, complete)
- Assigned agent (if any)
- Last updated
- Dependencies
## /spec implement [name]
**Purpose**: Hand specification to appropriate agent for implementation
**Usage**: `/spec implement SPEC-002`
**Process**:
1. Read the specified spec
2. Analyze requirements to determine appropriate agent:
- Frontend components → vue-developer
- Architecture/system design → system-architect
- Backend/API → python-developer
3. Launch agent with spec context
4. Agent creates implementation plan
5. Update spec with implementation status
## /spec review [name]
**Purpose**: Review implementation against specification criteria
**Usage**: `/spec review SPEC-002`
**Process**:
1. Read original spec and "How to Evaluate" section
2. Examine current implementation
3. Test against success criteria
4. Document gaps or issues
5. Update spec with review results
6. Recommend next actions (complete, revise, iterate)
## Command Extensions
As the process evolves, we may add:
- `/spec link [spec1] [spec2]` - Create dependency links
- `/spec archive [name]` - Archive completed specs
- `/spec template [type]` - Create spec from template
- `/spec search [query]` - Search spec content
## References
- Claude Slash commands: https://docs.anthropic.com/en/docs/claude-code/slash-commands
## Creating a command
Commands are implemented as Claude slash commands:
Location in repo: .claude/commands/
In the following example, we create the /optimize command:
```bash
# Create a project command
mkdir -p .claude/commands
echo "Analyze this code for performance issues and suggest optimizations:" > .claude/commands/optimize.md
```
File diff suppressed because it is too large Load Diff
-108
View File
@@ -1,108 +0,0 @@
---
title: 'SPEC-3: Agent Definitions'
type: spec
permalink: specs/spec-3-agent-definitions
tags:
- agents
- roles
- process
---
# SPEC-3: Agent Definitions
This document defines the specialist agents used in our specification-driven development process.
## system-architect
**Role**: High-level system design and architectural decisions
**Responsibilities**:
- Create architectural specifications and ADRs
- Analyze system-wide impacts and trade-offs
- Design component interfaces and data flow
- Evaluate technical approaches and patterns
- Document architectural decisions and rationale
**Expertise Areas**:
- System architecture and design patterns
- Technology evaluation and selection
- Scalability and performance considerations
- Integration patterns and API design
- Technical debt and refactoring strategies
**Typical Specs**:
- System architecture overviews
- Component decomposition strategies
- Data flow and state management
- Integration and deployment patterns
## vue-developer
**Role**: Frontend component development and UI implementation
**Responsibilities**:
- Create Vue.js component specifications
- Implement responsive UI components
- Design component APIs and interfaces
- Optimize for performance and accessibility
- Document component usage and patterns
**Expertise Areas**:
- Vue.js 3 Composition API
- Nuxt 3 framework patterns
- shadcn-vue component library
- Responsive design and CSS
- TypeScript integration
- State management with Pinia
**Typical Specs**:
- Individual component specifications
- UI pattern libraries
- Responsive design approaches
- Component interaction flows
## python-developer
**Role**: Backend development and API implementation
**Responsibilities**:
- Create backend service specifications
- Implement APIs and data processing
- Design database schemas and queries
- Optimize performance and reliability
- Document service interfaces and behavior
**Expertise Areas**:
- FastAPI and Python web frameworks
- Database design and operations
- API design and documentation
- Authentication and security
- Performance optimization
- Testing and validation
**Typical Specs**:
- API endpoint specifications
- Database schema designs
- Service integration patterns
- Performance optimization strategies
## Agent Collaboration Patterns
### Handoff Protocol
1. Agent receives spec through `/spec implement [name]`
2. Agent reviews spec and creates implementation plan
3. Agent documents progress and decisions in spec
4. Agent hands off to another agent if cross-domain work needed
5. Final agent updates spec with completion status
### Communication Standards
- All agents update specs through basic-memory MCP tools
- Document decisions and trade-offs in spec notes
- Link related specs and components
- Preserve context for future reference
### Quality Standards
- Follow existing codebase patterns and conventions
- Write tests that validate spec requirements
- Document implementation choices
- Consider maintainability and extensibility
@@ -1,311 +0,0 @@
---
title: 'SPEC-4: Notes Web UI Component Architecture'
type: note
permalink: specs/spec-4-notes-web-ui-component-architecture
tags:
- frontend
- 'component-architecture'
- vue
- 'refactoring'
---
# SPEC-4: Notes Web UI Component Architecture
## Why
The current Notes.vue component is a monolithic component that handles multiple responsibilities, making it difficult to maintain, test, and understand. This leads to:
- Complex state management across multiple concerns
- Difficult to isolate and test individual features
- Hard to understand the full scope of functionality
- Circular refactoring cycles when making changes
- Poor separation of concerns between navigation, display, and interaction logic
We need to decompose this into focused, single-responsibility components that are easier to develop, test, and maintain while preserving the existing functionality users expect.
## What
This spec defines the component architecture for decomposing the Notes web UI into focused components with clear responsibilities and interactions.
**Affected Areas:**
- `/apps/web/components/notes/Notes.vue` - Will be decomposed into smaller components
- `/apps/web/components/notes/` - New component structure
- Existing composables: `useNotesNavigation`, `useNotesFiltering`, `useNotesLayout`
- Mobile responsive behavior and layout management
**Component Breakdown:**
```
┌───────────────────────┬─────────────────────────────────────┬────────────────────────────────────────────────────────────┐
│ [Project] │ [Project Name] A/Z | ^ │ [edit | view] [actions] │
├───────────────────────┼─────────────────────────────────────┤ │
│ All Notes ├─────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ Recent │ search... │ [note header] │
│ [Project base dir] ├─────────────────────────────────────┤ │
│ ├─────────────────────────────────────┤ │
│ Folder1 │ Title [modified] │ │
│ Folder2 │ ├────────────────────────────────────────────────────────────┤
│ - Nested │ snippet │ [note body] │
│ │ │ │
│ │ │ │
│ ├─────────────────────────────────────┤ │
│ ├─────────────────────────────────────┤ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ ├─────────────────────────────────────┤ │
│ ├─────────────────────────────────────┤ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ ├─────────────────────────────────────┤ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
└───────────────────────┴─────────────────────────────────────┴────────────────────────────────────────────────────────────┘
```
### ProjectSwitcher Component
- **Location**: Top-left dropdown
- **Responsibility**: Allow users to switch between Basic Memory projects
- **Behavior**: Selecting different project controls entire Notes page content
- **State**: When switching projects, reset to "All notes" view
### NotesNav Component
- **Views**: Three mutually exclusive options:
- **All notes**: Display all notes in project alphabetically
- **Recent**: Display all notes in project by updated time (desc)
- **Project**: Display notes in top-level directory of project
- **Interaction**: Only one view can be active at a time
- **Folder Integration**: All/Recent ignore folder selection; Project respects folder selection
### FolderTree Component
- **Display**: Nested list of all folders in project as tree view
- **Interaction**: Selecting folder filters notes in NotesList using directoryList API
- **Navigation Integration**: Selecting folder automatically switches NotesNav to "Project" view for clear UX
- **API Integration**: Uses directoryList API call via useDirectoryListQuery for folder-specific note fetching
- **State Coordination**: Folder selection coordinates with navigation state for intuitive user experience
### NotesList Component
- **Display**: Vertically scrolling cards showing note summaries
- **Information per card**:
- Note title
- Modified time (relative, e.g., "7 minutes ago")
- Short summary of note content (one line preview)
- **Behavior**: Updates based on NotesNav selection and FolderTree filtering
### NoteDetail Component
- **Display**: Full content of selected note
- **Sections**:
- Header: Displays frontmatter information
- Content: Note body content
- **Editing**: Current textarea implementation (rich editor in future spec)
- **Frontmatter**: Leave current implementation (enhancement in future spec)
## How (High Level)
### Component Architecture Approach
1. **Single Responsibility**: Each component handles one primary concern
2. **Clear Data Flow**: Props down, events up pattern for component communication
3. **Composable Integration**: Use existing composables for state management
4. **Progressive Decomposition**: Extract components incrementally to maintain functionality
### Implementation Strategy
1. **Extract ProjectSwitcher**: Move project switching logic to dedicated component
2. **Extract NotesNav**: Isolate navigation state and view selection logic
3. **Extract FolderTree**: Separate folder display and selection logic
4. **Extract NotesList**: Isolate note listing and card display logic
5. **Extract NoteDetail**: Separate note content display and editing
6. **Update Notes.vue**: Become orchestration component managing component interactions
### State Management Integration
- **useNotesNavigation**: Manages navigation state (All/Recent/Project)
- **useNotesFiltering**: Handles filtering logic based on navigation and folder selection
- **useNotesLayout**: Manages responsive layout and panel visibility
- **Component State**: Each component manages its own internal UI state
- **Shared State**: Project selection and note filtering coordinated through composables
### Responsive Behavior
Mobile:
- Hide sidebar. pop out panel when selected
- show note list on small screens (existing behavior)
- when note list item is clicked, display note detail on full page. Cancel or go back to return to list
Desktop:
- Full three-column layout with all components visible
- **Transitions**: Smooth navigation between mobile panels
## How to Evaluate
### Success Criteria
- **Functional Parity**: All existing Notes page functionality preserved
- **Component Isolation**: Each component can be developed/tested independently
- **Clear Responsibilities**: No overlapping concerns between components
- **State Clarity**: Clean data flow and state management patterns
- **Mobile Compatibility**: Responsive behavior maintains current UX
- **Performance**: No degradation in rendering or interaction performance
### Testing Procedure
1. **Functionality Validation**:
- Project switching works correctly
- All three navigation views (All/Recent/Project) function properly
- Folder selection affects note display appropriately
- Note selection and detail display works
- Mobile responsive behavior preserved
2. **Component Isolation Testing**:
- Each component can be imported and used independently
- Component props and events are clearly defined
- No tight coupling between components
3. **Integration Testing**:
- Components communicate correctly through props/events
- State management composables integrate properly
- User workflows function end-to-end
4. **Performance Validation**:
- Page load time unchanged or improved
- Interaction responsiveness maintained
- Memory usage stable or improved
### Implementation Validation
- **Code Review**: Clean component structure with single responsibilities
- **Type Safety**: Full TypeScript coverage with proper component prop types
- **Documentation**: Each component has clear interface documentation
- **Tests**: Unit tests for individual components and integration tests for workflows
## Observations
- [problem] Monolithic Notes.vue component creates maintenance and testing challenges #component-architecture
- [solution] Component decomposition improves separation of concerns and testability #refactoring
- [pattern] Progressive extraction maintains functionality while improving structure #incremental-improvement
- [interaction] NotesNav and FolderTree have conditional interaction based on selected view #state-management
- [constraint] Mobile responsive behavior must be preserved during decomposition #responsive-design
- [scope] Current editing and frontmatter capabilities remain unchanged #scope-limitation
- [validation] Functional parity is critical success criteria for this refactoring #validation-strategy
- [implementation] Folder selection now properly integrates with directoryList API for accurate filtering #api-integration
- [fix] FolderTree selection functionality completed - works across all navigation views #feature-complete
- [ux-improvement] FolderTree selection automatically switches NotesNav to Project view for clear user feedback #user-experience
## Relations
- depends_on [[SPEC-1: Specification-Driven Development Process]]
- implements [[Current Notes.vue functionality]]
- prepares_for [[Future rich editor spec]]
- prepares_for [[Future frontmatter editing spec]]
## Implementation Progress
### Components
1. **ProjectSwitcher** (`~/components/notes/ProjectSwitcher.vue`)
- ✅ Top-left dropdown for project switching
- ✅ Integrates with Pinia project store
- ✅ Handles project switching with proper state reset
- ✅ Responsive collapsed/expanded states
- ✅ Expanded menu shows available projects and a Manage Projects option that navigates to the /settings/projects page
- ✅ Simplified component following SortingToggle pattern - clean Props/Emits interface, uses ProjectItem type directly
2. **NotesNav** (`~/components/notes/NotesNav.vue`)
- ✅ Three mutually exclusive views: All/Recent/Project
- ✅ Dynamic project title based on selected project
- ✅ Clean props down, events up pattern
- ✅ Responsive collapsed/expanded states with tooltips
- ✅ The label for the Project selection should be the folder name for the project, not the project name
3. **FolderTree** (`~/components/notes/FolderTree.vue`)
- ✅ Nested folder tree view for filtering
- ✅ Uses `useFolderTree()` composable for data
- ✅ Emits `folder-selected` events properly
- ✅ Handles loading, error, and empty states
- ✅ Includes companion `FolderTreeNode.vue` component
- ✅ The current folder should be visibly selected in the tree
4. **NotesList** (`~/components/notes/NotesList.vue`)
- ✅ Vertically scrolling note summary cards
- ✅ Shows title, updated time (relative), and content preview
- ✅ Badge system for tags with variant logic
- ✅ v-model integration for selectedNote
- ✅ Smooth transitions and animations
- ✅ Contextual title: The current folder name should be displayed at the top of the Notes list, or "All Notes", or "Recent" if they are selected
- ✅ The title header should contain a toggle component to allow sorting with Lucide icon labels
- sorting options:
- name (asc/desc) - default
- file updated time (asc/desc)
- If "Recent" notes nav option is selected the default order should be updated in descending order (recent first)
5. **NoteDisplay** (`~/components/notes/NoteDisplay.vue` - equivalent to spec's NoteDetail)
- ✅ Full note content display
- ✅ Edit/view mode toggle
- ✅ Header with frontmatter information
- ✅ Markdown rendering capabilities
- ✅ Current textarea implementation preserved
### Architecture Requirements
1. **Component Isolation**: Each component can be developed/tested independently ✅
2. **Single Responsibility**: Each component handles one primary concern ✅
3. **Clear Data Flow**: Props down, events up pattern implemented ✅
4. **Composable Integration**: Uses existing composables for state management ✅
5. **Responsive Behavior**: Mobile/desktop layout preserved ✅
### State Management Integration
- **useNotesNavigation**: Manages navigation state (All/Recent/Project) ✅
- **useNotesFiltering**: Handles filtering logic based on navigation and folder selection ✅
- **useNotesLayout**: Manages responsive layout and panel visibility ✅
- **Component State**: Each component manages its own internal UI state ✅
### Interaction Logic
- Only one NotesNav view active at a time ✅
- All/Recent views ignore folder selection ✅
- Project view respects folder selection ✅
- Project switching resets to "All notes" view ✅
### TypeScript Coverage
- All components have full TypeScript coverage ✅
- Component props and events properly typed ✅
- No TypeScript errors in codebase ✅
### Success Criteria Validation
1. **Functional Parity**: All existing Notes page functionality preserved ✅
2. **Component Isolation**: Each component can be developed/tested independently ✅
3. **Clear Responsibilities**: No overlapping concerns between components ✅
4. **State Clarity**: Clean data flow and state management patterns ✅
5. **Mobile Compatibility**: Responsive behavior maintains current UX ✅
6. **Performance**: No degradation in rendering or interaction performance ✅
## Implementation Decisions
### Architectural Patterns
1. **Composition API + `<script setup>`**: All components use modern Vue 3 syntax
2. **Pinia Store Integration**: Project switching handled through reactive store
3. **Composable Pattern**: State management distributed across focused composables
4. **Event-Driven Communication**: Clean parent-child communication via events
5. **Responsive-First Design**: Mobile/desktop layouts handled natively
### Key Technical Choices
1. **Progressive Enhancement**: Mobile-first responsive design with desktop enhancements
2. **State Reset Logic**: Project switching properly resets navigation, search, and selection state
3. **Performance Optimizations**: Efficient re-rendering with proper key usage and transitions
4. **Accessibility**: Screen reader support, tooltips, keyboard navigation
5. **Type Safety**: Full TypeScript coverage with proper component prop definitions
### Quality Metrics
- **Code Maintainability**: High - each component is focused and independently testable
- **Performance**: Excellent - no performance degradation from decomposition
- **User Experience**: Preserved - all existing functionality and responsive behavior maintained
- **Developer Experience**: Improved - cleaner component structure for future development
-201
View File
@@ -1,201 +0,0 @@
---
title: 'SPEC-5: CLI Cloud Upload via WebDAV'
type: spec
permalink: specs/spec-5-cli-cloud-upload-via-webdav
tags:
- cli
- webdav
- upload
- migration
- poc
---
# SPEC-5: CLI Cloud Upload via WebDAV
## Why
Existing basic-memory users need a simple migration path to basic-memory-cloud. The web UI drag-and-drop approach outlined in GitHub issue #59, while user-friendly, introduces significant complexity for a proof-of-concept:
- Complex web UI components for file upload and progress tracking
- Browser file handling limitations and CORS complexity
- Proxy routing overhead for large file transfers
- Authentication integration across multiple services
A CLI-first approach solves these issues by:
- **Leveraging existing infrastructure**: Both cloud CLI and tenant API already exist with WorkOS JWT authentication
- **Familiar user experience**: Basic-memory users are CLI-comfortable and expect command-line tools
- **Direct connection efficiency**: Bypassing the MCP gateway/proxy for bulk file transfers
- **Rapid implementation**: Building on existing `CLIAuth` and FastAPI foundations
The fundamental problem is migration friction - users have local basic-memory projects but no path to cloud tenants. A simple CLI upload command removes this barrier immediately.
## What
This spec defines a CLI-based project upload system using WebDAV for direct tenant connections.
**Affected Areas:**
- `apps/cloud/src/basic_memory_cloud/cli/main.py` - Add upload command to existing CLI
- `apps/api/src/basic_memory_cloud_api/main.py` - Add WebDAV endpoints to tenant FastAPI
- Authentication flow - Reuse existing WorkOS JWT validation
- File transfer protocol - WebDAV for cross-platform compatibility
**Core Components:**
### CLI Upload Command
```bash
basic-memory-cloud upload <project-path> --tenant-url https://basic-memory-{tenant}.fly.dev
```
### WebDAV Server Endpoints
- `GET/PUT/DELETE /webdav/*` - Standard WebDAV operations on tenant file system
- Authentication via existing JWT validation
- File operations preserve timestamps and directory structure
### Authentication Flow
```
1. User runs `basic-memory-cloud login` (existing)
2. CLI stores WorkOS JWT token (existing)
3. Upload command reads JWT from storage
4. WebDAV requests include JWT in Authorization header
5. Tenant API validates JWT using existing middleware
```
## How (High Level)
### Implementation Strategy
**Phase 1: CLI Command**
- Add `upload` command to existing Typer app
- Reuse `CLIAuth` class for token management
- Implement WebDAV client using `webdavclient3` or similar
- Rich progress bars for transfer feedback
**Phase 2: WebDAV Server**
- Add WebDAV endpoints to existing tenant FastAPI app
- Leverage existing `get_current_user` dependency for authentication
- Map WebDAV operations to tenant file system
- Preserve file modification times using `os.utime()`
**Phase 3: Integration**
- Direct connection bypasses MCP gateway and proxy
- Simple conflict resolution: overwrite existing files
- Error handling: fail fast with clear error messages
### Technical Architecture
```
basic-memory-cloud CLI → WorkOS JWT → Direct WebDAV → Tenant FastAPI
Tenant File System
```
**Key Libraries:**
- CLI: `webdavclient3` for WebDAV client operations
- API: `wsgidav` or FastAPI-compatible WebDAV server
- Progress: `rich` library (already imported in CLI)
- Auth: Existing WorkOS JWT infrastructure
### WebDAV Protocol Choice
WebDAV provides:
- **Cross-platform clients**: Native support in most operating systems
- **Standardized protocol**: Well-defined for file operations
- **HTTP-based**: Works with existing FastAPI and JWT auth
- **Library support**: Good Python libraries for both client and server
### POC Constraints
**Simplifications for rapid implementation:**
- **Known tenant URLs**: Assume `https://basic-memory-{tenant}.fly.dev` format
- **Upload only**: No download or bidirectional sync
- **Overwrite conflicts**: No merge or conflict resolution prompting
- **No fallbacks**: Fail fast if WebDAV connection issues occur
- **Direct connection only**: No proxy fallback mechanism
## How to Evaluate
### Success Criteria
**Functional Requirements:**
- [ ] Transfer complete basic-memory project (100+ files) in < 30 seconds
- [ ] Preserve directory structure exactly as in source project
- [ ] Preserve file modification timestamps for proper sync behavior
- [ ] Rich progress bars show real-time transfer status (files/MB transferred)
- [ ] WorkOS JWT authentication validates correctly on WebDAV endpoints
- [ ] Direct tenant connection bypasses MCP gateway successfully
**Quality Requirements:**
- [ ] Clear error messages for authentication failures
- [ ] Graceful handling of network interruptions
- [ ] CLI follows existing command patterns and help text standards
- [ ] WebDAV endpoints integrate cleanly with existing FastAPI app
**Performance Requirements:**
- [ ] File transfer speed > 1MB/s on typical connections
- [ ] Memory usage remains reasonable for large projects
- [ ] No timeout issues with 500+ file projects
### Testing Procedure
**Unit Testing:**
1. CLI command parsing and argument validation
2. WebDAV client connection and authentication
3. File timestamp preservation during transfer
4. JWT token validation on WebDAV endpoints
**Integration Testing:**
1. End-to-end upload of test project
2. Direct tenant connection without proxy
3. File integrity verification after upload
4. Progress tracking accuracy during transfer
**User Experience Testing:**
1. Upload existing basic-memory project from local installation
2. Verify uploaded files appear correctly in cloud tenant
3. Confirm basic-memory database rebuilds properly with uploaded files
4. Test CLI help text and error message clarity
### Validation Commands
**Setup:**
```bash
# Login to WorkOS
basic-memory-cloud login
# Upload project
basic-memory-cloud upload ~/my-notes --tenant-url https://basic-memory-test.fly.dev
```
**Verification:**
```bash
# Check tenant health and file count via API
curl -H "Authorization: Bearer $JWT" https://basic-memory-test.fly.dev/health
curl -H "Authorization: Bearer $JWT" https://basic-memory-test.fly.dev/notes/search
```
### Performance Benchmarks
**Target metrics for 100MB basic-memory project:**
- Transfer time: < 30 seconds
- Memory usage: < 100MB during transfer
- Progress updates: Every 1MB or 10 files
- Authentication time: < 2 seconds
## Observations
- [implementation-speed] CLI approach significantly faster than web UI for POC development #rapid-prototyping
- [user-experience] Basic-memory users already comfortable with CLI tools #user-familiarity
- [architecture-benefit] Direct connection eliminates proxy complexity and latency #performance
- [auth-reuse] Existing WorkOS JWT infrastructure handles authentication cleanly #code-reuse
- [webdav-choice] WebDAV protocol provides cross-platform compatibility and standard libraries #protocol-selection
- [poc-scope] Simple conflict handling and error recovery sufficient for proof-of-concept #scope-management
- [migration-value] Removes primary barrier for local users migrating to cloud platform #business-value
## Relations
- depends_on [[SPEC-1: Specification-Driven Development Process]]
- enables [[GitHub Issue #59: Web UI Upload Feature]]
- uses [[WorkOS Authentication Integration]]
- builds_on [[Existing Cloud CLI Infrastructure]]
- builds_on [[Existing Tenant API Architecture]]
@@ -1,497 +0,0 @@
---
title: 'SPEC-6: Explicit Project Parameter Architecture'
type: spec
permalink: specs/spec-6-explicit-project-parameter-architecture
tags:
- architecture
- mcp
- project-management
- stateless
---
# SPEC-6: Explicit Project Parameter Architecture
## Why
The current session-based project management system has critical reliability issues:
1. **Session State Fragility**: Claude iOS mobile client fails to maintain consistent session IDs across MCP tool calls, causing project switching to silently fail (Issue #74)
2. **Scaling Limitations**: Redis-backed session state creates single-point-of-failure and prevents horizontal scaling
3. **Client Compatibility**: Session tracking works inconsistently across different MCP clients (web, mobile, API)
4. **Hidden Complexity**: Users cannot see or understand "current project" state, leading to confusion when operations execute in wrong projects
5. **Silent Failures**: Operations appear successful but execute in unintended projects, risking data integrity
Evidence from production logs shows each MCP tool call from mobile client receives different session IDs:
```
create_memory_project: session_id=12cdfc24913b48f8b680ed4b2bfdb7ba
switch_project: session_id=050a69275d98498cbdd227cdb74d9740
list_directory: session_id=85f3483014af4136a5d435c76ded212f
```
Related Github issue: https://github.com/basicmachines-co/basic-memory-cloud/issues/75
## Status
**Current Status**: **ALL PHASES COMPLETE** ✅ **PRODUCTION DEPLOYED**
**Target**: Fix Claude iOS session ID consistency issues ✅ **ACHIEVED**
**Draft PR**: https://github.com/basicmachines-co/basic-memory/pull/298 ✅ **MERGED & DEPLOYED**
### 🎉 **COMPLETE SUCCESS - PRODUCTION READY**
**ALL PHASES OF SPEC-6 IMPLEMENTATION COMPLETE!** The stateless architecture has been successfully implemented across both Basic Memory core and Basic Memory Cloud, representing a **fundamental architectural improvement** that completely solves the Claude iOS compatibility issue while providing superior scalability and reliability.
#### Implementation Summary:
- **16 files modified** with 582 additions and 550 deletions
- **All 17 MCP tools** converted to stateless architecture
- **147 tests updated** across 5 test files (100% passing)
- **Complete session state removal** from core MCP tools
- **Enhanced error handling** and security validations preserved
### Progress Summary
**Complete Stateless Architecture Implementation (All 17 tools)** - **PRODUCTION DEPLOYED**
- Stateless `get_active_project()` function implemented and deployed ✅
- All session state dependencies removed across entire MCP server ✅
- All MCP tools require explicit `project` parameter as first argument ✅
- **Cloud Service**: Redis removed, stateless HTTP enabled ✅
- **Production Validation**: Comprehensive testing completed with 100% success ✅
✅ **Content Management Tools Complete (6/6 tools)**
- `write_note`, `read_note`, `delete_note`, `edit_note`
- `view_note`, `read_content`
✅ **Knowledge Graph Navigation Tools Complete (3/3 tools)**
- `build_context`, `recent_activity`, `list_directory`
✅ **Search & Discovery Tools Complete (1/1 tools)**
- `search_notes`
✅ **Visualization Tools Complete (1/1 tools)**
- `canvas`
✅ **Project Management Cleanup Complete**
- Removed `switch_project` and `get_current_project` tools ✅
- Updated `set_default_project` to remove activate parameter ✅
✅ **Comprehensive Testing Complete (157 tests)**
- All test suites updated to use stateless architecture (147 existing tests)
- Single project constraint mode integration tests (10 new tests)
- 100% test pass rate across all tool test files
- Security validations preserved and working
- Error handling comprehensive and user-friendly
✅ **Documentation & Examples Complete**
- All tool docstrings updated with stateless examples
- Project parameter usage clearly documented
- Error handling and security behavior documented
✅ **Enhanced Discovery Mode Complete**
- `recent_activity` tool supports dual-mode operation (discovery vs project-specific)
- ProjectActivitySummary schema provides cross-project insights
- Recent activity prompt updated to support both modes
- Comprehensive project distribution statistics and most active project tracking
✅ **Single Project Constraint Mode Complete**
- `--project` CLI parameter for MCP server constraint
- Environment variable control (`BASIC_MEMORY_MCP_PROJECT`)
- Automatic project override in `get_active_project()` function
- Project management tools disabled in constrained mode with helpful CLI guidance
- Comprehensive integration test suite (10 tests covering all constraint scenarios)
## What
Transform Basic Memory from stateful session-based to stateless explicit project parameter architecture:
### Core Changes
1. **Mandatory Project Parameter**: All MCP tools require explicit `project` parameter
2. **Remove Session State**: Eliminate Redis, session middleware, and `switch_project` tool
3. **Stateless HTTP**: Enable `stateless_http=True` for horizontal scaling
4. **Enhanced Context Discovery**: Improve `recent_activity` to show project distribution
5. **Clear Response Format**: All tool responses display target project information
Implementation Approach
- Each tool will directly accept the project parameter
- Remove all calls to context-based project retrieval
- Validate project exists before operations
- Clear error messages when project not found
- Backward compatibility: Initially keep optional parameter, then make required
### Affected MCP Tools
**Content Management** (require project parameter):
- `write_note(project, title, content, folder)`
- `read_note(project, identifier)`
- `edit_note(project, identifier, operation, content)`
- `delete_note(project, identifier)`
- `view_note(project, identifier)`
- `read_content(project, path)`
**Knowledge Graph Navigation** (require project parameter):
- `build_context(project, url, timeframe, depth, max_related)`
- `list_directory(project, dir_name, depth, file_name_glob)`
- `search_notes(project, query, search_type, types, entity_types)`
**Search & Discovery** (use project parameter for specific project or none for discovery):
- `recent_activity(project, timeframe, depth, max_related)`
**Visualization** (require project parameter):
- `canvas(project, nodes, edges, title, folder)`
**Project Management** (unchanged - already stateless):
- `list_memory_projects()`
- `create_memory_project(project_name, project_path, set_default)`
- `delete_project(project_name)`
- `get_current_project()` - Remove this tool
- `switch_project(project_name)` - Remove this tool
- `set_default_project(project_name, activate)` - Remove activate parameter
## How (High Level)
### Phase 1: Basic Memory Core (basic-memory repository)
#### MCP Tool Updates
Phase 1: Core Changes
1. Update project_context.py
- [x] Make project parameter mandatory for get_active_project()
- [x] Remove session state handling
2. Update Content Management Tools (6 tools)
- [x] write_note: Make project parameter required, not optional
- [x] read_note: Make project parameter required
- [x] edit_note: Add required project parameter
- [x] delete_note: Add required project parameter
- [x] view_note: Add required project parameter
- [x] read_content: Add required project parameter
3. Update Knowledge Graph Navigation Tools (3 tools)
- [x] build_context: Add required project parameter
- [x] recent_activity: Make project parameter required
- [x] list_directory: Add required project parameter
4. Update Search & Visualization Tools (2 tools)
- [x] search_notes: Add required project parameter
- [x] canvas: Add required project parameter
5. Update Project Management Tools
- [x] Remove switch_project tool completely
- [x] Remove get_current_project tool completely
- [x] Update set_default_project to remove activate parameter
- [x] Keep list_memory_projects, create_memory_project, delete_project unchanged
6. Enhance recent_activity Response
- [x] Add project distribution info showing activity across all projects
- [x] Include project usage stats in response
- [x] Implement ProjectActivitySummary for discovery mode
- [x] Add dual-mode functionality (discovery vs project-specific)
7. Update Tool Documentation
- [x] Update write_note docstring with stateless architecture examples
- [x] Update read_note docstring with project parameter examples
- [x] Update delete_note docstring with comprehensive usage guidance
- [x] Update all remaining tool docstrings with project parameter examples
8. Update Tool Responses
- [x] Add clear project indicator to all tool responses across all tools
- [x] Format: "project: {project_name}" in response metadata
- [x] Add project metadata footer for LLM awareness
- [x] Update all tool responses to include project indicators
9. Comprehensive Testing
- [x] Update all write_note tests to use stateless architecture (34 tests passing)
- [x] Update all edit_note tests to use stateless architecture (17 tests passing)
- [x] Update all view_note tests to use stateless architecture (12 tests passing)
- [x] Update all search_notes tests to use stateless architecture (16 tests passing)
- [x] Update all move_note tests to use stateless architecture (31 tests passing)
- [x] Update all delete_note tests to use stateless architecture
- [x] Verify direct function call compatibility (bypassing MCP layer)
- [x] Test security validation with project parameters
- [x] Validate error handling for non-existent projects
- [x] **Total: 157 tests updated and passing (100% success rate)**
- [x] **147 existing tests** updated for stateless architecture
- [x] **10 new tests** for single project constraint mode
### Phase 1.5: Default Project Mode Enhancement
#### Problem
While the stateless architecture solves reliability issues, it introduces UX friction for single-project users (estimated 80% of usage) who must specify the project parameter in every tool call.
#### Solution: Default Project Mode
Add optional `default_project_mode` configuration that allows single-project users to have the simplicity of implicit project selection while maintaining the reliability of stateless architecture.
#### Configuration
```json
{
"default_project": "main",
"default_project_mode": true // NEW: Auto-use default_project when not specified
}
```
#### Implementation Details
1. **Config Enhancement** (`src/basic_memory/config.py`)
- Add `default_project_mode: bool = Field(default=False)`
- Preserves backward compatibility (defaults to false)
2. **Project Resolution Logic** (`src/basic_memory/mcp/project_context.py`)
Three-tier resolution hierarchy:
- Priority 1: CLI `--project` constraint (BASIC_MEMORY_MCP_PROJECT env var)
- Priority 2: Explicit project parameter in tool call
- Priority 3: `default_project` if `default_project_mode=true` and no project specified
3. **Assistant Guide Updates** (`src/basic_memory/mcp/resources/ai_assistant_guide.md`)
- Detect `default_project_mode` at runtime
- Provide mode-specific instructions to LLMs
- In default mode: "All operations use project 'main' automatically"
- In regular mode: Current project discovery guidance
4. **Tool Parameter Handling** (all MCP tools)
- Make project parameter Optional[str] = None
- Add resolution logic: `project = project or get_default_project()`
- Maintain explicit project override capability
#### Usage Modes Summary
- **Regular Mode**: Multi-project users, assistant tracks project per conversation
- **Default Project Mode**: Single-project users, automatic default project
- **Constrained Mode**: CLI --project flag, locked to specific project
#### Testing Requirements
- Integration test for default_project_mode=true with missing parameters
- Test explicit project override in default_project_mode
- Test mode=false requires explicit parameters
- Test CLI constraint overrides default_project_mode
Phase 2: Testing & Validation
8. Update Tests
- [x] Modify all MCP tool tests to pass required project parameter
- [x] Remove tests for deleted tools (switch_project, get_current_project)
- [x] Add tests for project parameter validation
- [x] **Complete: All 147 tests across 5 test files updated and passing**
#### Enhanced recent_activity Response
```json
{
"recent_notes": [...],
"project_activity": {
"research-project": {
"operations": 5,
"last_used": "30 minutes ago",
"recent_folders": ["experiments", "findings"]
},
"work-notes": {
"operations": 2,
"last_used": "2 hours ago",
"recent_folders": ["meetings", "planning"]
}
},
"total_projects": 3
}
```
#### Response Format Updates
```
✓ Note created successfully
Project: research-project
File: experiments/Neural Network Results.md
Permalink: research-project/neural-network-results
```
### Phase 2: Cloud Service Simplification (basic-memory-cloud repository) ✅ **COMPLETE**
#### ✅ Remove Session Infrastructure **COMPLETE**
1. ✅ Delete `apps/mcp/src/basic_memory_cloud_mcp/middleware/session_state.py`
2. ✅ Delete `apps/mcp/src/basic_memory_cloud_mcp/middleware/session_logging.py`
3. ✅ Update `apps/mcp/src/basic_memory_cloud_mcp/main.py`:
```python
# Remove session middleware
# server.add_middleware(SessionStateMiddleware)
# Enable stateless HTTP
mcp = FastMCP(name="basic-memory-mcp", stateless_http=True)
```
#### ✅ Deployment Simplification **COMPLETE**
1. ✅ Remove Redis from `fly.toml`
2. ✅ Remove Redis environment variables
3. ✅ Update health checks to not depend on Redis
4. ✅ Production deployment verified working with stateless architecture
### Phase 3: Conversational Project Management ✅ **COMPLETE**
#### ✅ Claude Behavior Pattern **VERIFIED WORKING**
1. ✅ **Project Discovery**:
```
Claude: Let me check your recent activity...
[calls recent_activity() - no project needed for discovery]
I see you've been working in:
- research-project (5 operations, 30 min ago)
- work-notes (2 operations, 2 hours ago)
Which project should I use for this operation?
```
2. ✅ **Context Maintenance**:
```
User: Use research-project
Claude: Working in research-project.
[All subsequent operations use project="research-project"]
```
3. ✅ **Explicit Project Switching**:
```
User: Check work-notes for that meeting summary
Claude: Let me search work-notes for the meeting summary.
[Uses project="work-notes" for specific operation]
```
**Validation**: Comprehensive testing confirmed all conversational patterns work naturally with the stateless architecture.
## How to Evaluate
### Success Criteria
#### 1. Functional Completeness
- [x] All MCP tools accept required `project` parameter
- [x] All MCP tools validate project exists before execution
- [x] `switch_project` and `get_current_project` tools removed
- [x] All responses display target project clearly
- [x] No Redis dependencies in deployment (Phase 2: Cloud Service) ✅ **COMPLETE**
- [x] `recent_activity` shows project distribution with ProjectActivitySummary
#### 2. Cross-Client Compatibility Testing ✅ **COMPLETE**
Test identical operations across all clients:
- [x] **Claude Desktop**: All operations work with explicit projects ✅
- [x] **Claude Code**: All operations work with explicit projects ✅
- [x] **Claude Mobile iOS**: All operations work with explicit projects ✅ **CRITICAL SUCCESS**
- [x] **API clients**: All operations work with explicit projects ✅
- [x] **CLI tools**: All operations work with explicit projects ✅
**Critical Achievement**: Claude iOS mobile client session tracking issues completely eliminated through stateless architecture.
#### 3. Session Independence Verification ✅ **COMPLETE**
- [x] Operations work identically with/without session tracking ✅
- [x] No behavioral differences between clients ✅
- [x] Mobile client session ID changes do not affect operations ✅
- [x] Redis can be completely removed without functional impact ✅
**Production Validation**: Redis removed from production deployment with zero functional impact.
#### 4. Performance & Scaling ✅ **COMPLETE**
- [x] `stateless_http=True` enabled successfully ✅
- [x] No Redis memory usage ✅
- [x] Horizontal scaling possible (multiple MCP instances) ✅
- [x] Response times unchanged or improved ✅
#### 5. User Experience Testing
**Project Discovery Flow**:
- [x] `recent_activity()` provides useful project context
- [x] Claude can intelligently suggest projects based on activity
- [x] Project switching is explicit and clear in conversation
**Error Handling**:
- [x] Clear error messages for non-existent projects
- [x] Helpful suggestions when project parameter missing
- [x] No silent failures or wrong-project operations
**Response Clarity**:
- [x] Every operation clearly shows target project
- [x] Users always know which project is being operated on
- [x] No confusion about "current project" state
#### 6. Migration Safety ✅ **COMPLETE**
- [x] Backward compatibility period with optional project parameter ✅
- [x] Clear migration documentation for existing users ✅
- [x] Data integrity maintained during transition ✅
- [x] No data loss during migration ✅
**Production Migration**: Successfully deployed to production with zero data loss and maintained system integrity.
### Test Scenarios
#### Core Functionality Test
```bash
# Test all tools work with explicit project
write_note(project="test-proj", title="Test", content="Content", folder="docs")
read_note(project="test-proj", identifier="Test")
edit_note(project="test-proj", identifier="Test", operation="append", content="More")
search_notes(project="test-proj", query="Content")
list_directory(project="test-proj", dir_name="docs")
delete_note(project="test-proj", identifier="Test")
```
#### Cross-Client Consistency Test
Run identical test sequence on:
1. Claude Desktop
2. Claude Code
3. Claude Mobile iOS
4. API client
5. CLI tools
Verify all clients:
- Accept explicit project parameters
- Return identical responses
- Show same project information
- Have no session dependencies
#### Session Independence Test
1. Monitor session IDs during operations
2. Verify operations work with changing session IDs
3. Confirm Redis removal doesn't affect functionality
4. Test with multiple concurrent clients
### Acceptance Criteria
**Must Have**:
- All MCP tools require and use explicit project parameter
- No session state dependencies remain
- Universal client compatibility achieved
- Clear project information in all responses
**Should Have**:
- Enhanced `recent_activity` with project distribution
- Smooth migration path for existing users
- Improved performance with stateless architecture
**Could Have**:
- Smart project suggestions based on content/context
- Project shortcuts for common operations
- Advanced project analytics in responses
## Notes
### Breaking Changes
This is a **breaking change** that requires:
- All MCP clients to pass project parameter
- Migration of existing workflows
- Update of all documentation and examples
### Implementation Order
1. **basic-memory core** - Update MCP tools to accept project parameter (optional initially)
2. **Testing** - Verify all clients work with explicit projects
3. **Cloud service** - Remove session infrastructure
4. **Migration** - Make project parameter mandatory
5. **Cleanup** - Remove deprecated tools and middleware
### Related Issues
- Fixes #74 (Claude iOS session state bug)
- Implements #75 (Mandatory project parameter architecture)
- Enables future horizontal scaling
- Simplifies multi-tenant architecture
### Dependencies
- Requires coordination between basic-memory and basic-memory-cloud repositories
- Needs client-side updates for smooth transition
- Documentation updates across all materials
@@ -1,324 +0,0 @@
---
title: 'SPEC-7: POC to spike Tigris/Turso for local access to cloud data'
type: spec
permalink: specs/spec-7-poc-tigris-turso-local-access-cloud-data
tags:
- poc
- tigris
- turso
- cloud-storage
- architecture
- proof-of-concept
---
# SPEC-7: POC to spike Tigris/Turso for local access to cloud data
> **Status Update**: ✅ **Phase 1 COMPLETE** (September 20, 2025)
> TigrisFS mounting validated successfully in containerized environments. Container startup, filesystem mounting, and Fly.io integration all working correctly. Ready for Phase 2 (Turso database integration).
> See: [`SPEC-7-PHASE-1-RESULTS.md`](./SPEC-7-PHASE-1-RESULTS.md)
## Why
Current basic-memory-cloud architecture uses Fly volumes for tenant file storage, which creates several limitations:
We could enable a revolutionary user experience: **local editing (or at least view access) of cloud-stored files** while maintaining Basic Memory's existing filesystem assumptions.
1. **Storage Scalability**: Fly volumes require pre-provisioning and don't auto-scale with usage
2. **Single Instance**: Volumes can only be mounted to one fly machine instance
3. **Cost Model**: Volume pricing vs object storage pricing may be less favorable at scale
4. **Local Development**: No way for users to mount their cloud tenant files locally for real-time editing
5. **Multi-Region**: Volumes are region-locked, limiting global deployment flexibility
6. **Backup/Disaster Recovery**: Object storage provides better durability and replication options
Basic Memory requires POSIX filesystem semantics but could benefit from object storage durability and accessibility. By combining:
- **Tigris object storage and TigrisFS** for file persistence in bucket stoage via a POSIX filesystem on the tenant instance
- **Turso/libSQL** for SQLite indexing (replacing local .db files). Sqlite on NFS volumes is disouraged.
## What
This specification defines a proof-of-concept to validate the technical feasibility of the Tigris/Turso architecture for basic-memory-cloud tenants.
**Affected Areas:**
- **Storage Architecture**: Replace Fly volumes with Tigris object storage
- **Database Architecture**: Replace local SQLite with Turso remote database
- **Container Setup**: Add TigrisFS mounting in tenant containers
- **Local Development**: Enable local mounting of cloud tenant data
- **Basic Memory Core**: Validate unchanged operation over mounted filesystems
**Key Components:**
- **Tigris Storage**: Globally caching S3-compatible object storage via Fly.io integration
- **TigrisFS**: Purpose-built FUSE filesystem with intelligent caching
- **Turso Database**: Hosted libSQL for SQLite replacement
- **Single-Tenant Model**: One bucket + one database per tenant (simplified isolation)
## Architectural Overview & Key Insights
### TigrisFS
Unlike standard S3 mounting approaches, **TigrisFS is a purpose-built FUSE filesystem** optimized for object storage with several critical advantages:
1. **Eliminates Fly Volume Limitations**
- No single-machine attachment constraints
- No pre-provisioning of storage capacity
- Enables horizontal scaling and zero-downtime deployments
- Automatic global CDN caching at Fly.io edge locations
2. **Intelligent Caching Architecture**
- 1-4GB+ configurable memory cache for read/write operations
- Write-back caching for improved performance
- Metadata cache to reduce API calls
- "Close to Redis speed" for small object retrieval
3. **Cost-Effective Model**
- Pay only for storage used and transferred
- No wasted capacity from over-provisioning
- Automatic global replication included
- S3 durability with CDN performance
### API-Driven Architecture Eliminates File Watching Concerns
**Critical Insight**: All file access (reads/writes) in basic-memory-cloud go through the API layer:
- **MCP Tools → API**: All Basic Memory operations use FastAPI endpoints
- **Web App → API**: Frontend uses API for all data modifications
- **File watching is NOT required** for cloud operations, unlike local BM which uses the WatchService to monitor file changes.
This means:
- **Cloud Operations**: Manual sync after API writes is sufficient
- **Local Development**: File watching only matters for local editing experience
- **Performance Risk**: Dramatically reduced since we're not dependent on inotify over network filesystems
### Realistic Local Access Expectations
**Baseline Functionality (Guaranteed):**
- Read-only mounting for browsing cloud files
- Easy download/upload of entire projects
- File copying via standard filesystem operations
**Stretch Goal (Test in POC):**
- Live editing with eventual consistency (1-5 second delays acceptable)
- Automatic sync for local changes
- Not required for core functionality - pure upside if it works
### Production Deployment Advantages
1. **Multi-Region Deployment**: Tigris handles global replication automatically
2. **Zero-Downtime Updates**: No volume detach/attach during deployments
3. **Tenant Migrations**: Simply update credentials, no data movement
4. **Disaster Recovery**: Built into S3 durability model (99.999999999% durability)
5. **Auto-Scaling**: Storage scales with usage, no capacity planning needed
## How (High Level)
### POC Approach: Server-First Validation
**Rationale**: Start with server-side TigrisFS mounting because:
- Local access is meaningless if cloud containers can't mount TigrisFS reliably
- Container startup and API performance are critical path blockers
- TigrisFS compatibility with Basic Memory operations must be proven first
- Each phase gates the next - no point testing local access if server-side fails
### Phase 1: Server-Side TigrisFS Validation (Critical Foundation) ✅ COMPLETE
- [x] Set up Tigris bucket with test data via Fly.io integration
- [x] Create container image with TigrisFS support and dependencies
- [x] Test TigrisFS mounting in containerized environment
- [x] Run Basic Memory API operations over mounted TigrisFS
- [x] Validate all filesystem operations work correctly
- [x] Measure container startup time and resource usage
**Production Validation Results**: Container successfully deployed and operated for 42+ minutes serving real MCP requests with repository queries, knowledge graph navigation, and full Basic Memory API functionality over TigrisFS-mounted storage.
### Phase 2: Database Migration to Turso
- [ ] Set up Turso account and test database
- [ ] Modify Basic Memory to accept external DATABASE_URL
- [ ] Test all MCP tools with remote SQLite via Turso
- [ ] Validate performance and functionality parity
- [ ] Test API write → manual sync workflow in container
### Phase 3: Production Container Integration
- [ ] Implement tenant-specific credential management for buckets
- [x] Test container startup with automatic TigrisFS mounting
- [ ] Validate isolation between tenant containers
- [ ] Test API operations under realistic load
- [ ] Measure performance vs current Fly volume setup
### Phase 4: Local Access Validation (Bonus Feature)
- [ ] Test local TigrisFS mounting of tenant data
- [ ] Validate read-only access for browsing/downloading
- [ ] Test file copying and upload workflows
- [ ] Measure latency impact on user experience
- [ ] Test live editing if file watching works (stretch goal)
### Architecture Overview
```
Local Development:
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Local TigrisFS │───▶│ Tigris Bucket │◀───│ Tenant Container│
│ Mount │ │ (Global CDN) │ │ TigrisFS mount │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ Basic Memory │ │ Basic Memory │
│ (local files) │ │ API + mounted │
└─────────────────┘ └─────────────────┘
│ │
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ Turso Database │◀───────────────────────────│ Turso Database │
│ (shared index) │ │ (shared index) │
└─────────────────┘ └─────────────────┘
Flow: API writes → Manual sync → Index update
Local: File watching (if available) → Auto sync
```
## How to Evaluate
### Success Criteria
- [x] **Filesystem Compatibility**: Basic Memory operates without modification over TigrisFS-mounted storage
- [x] **Performance Acceptable**: API-driven operations perform within acceptable latency (target: <500ms for typical operations)
- [ ] **Database Functionality**: All Basic Memory features work with Turso remote SQLite
- [x] **Container Reliability**: Tenant containers start successfully with automatic TigrisFS mounting
- [ ] **Local Access Baseline**: Users can mount cloud files locally for read-only browsing and file copying
- [x] **Data Isolation**: Tenant data remains properly isolated using bucket/database separation
- [ ] **Local Access Stretch**: Live editing with eventual sync (1-5 second delays acceptable)
### Testing Procedure
#### Phase 1: Server-Side Foundation Testing
1. **Container TigrisFS Test**:
```dockerfile
# Test container with TigrisFS mounting
FROM python:3.12
RUN apt-get update && apt-get install -y tigrisfs
# Test startup script
#!/bin/bash
tigrisfs --memory-limit 2048 $TIGRIS_BUCKET /app/data --daemon
cd /app/data && basic-memory sync
basic-memory-api --data-dir /app/data
```
2. **API Operations Validation**:
```bash
# Test all MCP operations over TigrisFS
curl -X POST /api/write_note -d '{"title":"test","content":"content"}'
curl -X GET /api/read_note/test
curl -X GET /api/search_notes?q=content
# Measure: response times, error rates, data consistency
```
#### Phase 2: Database Integration Testing
3. **Turso Integration Test**:
```bash
# Configure Turso connection in container
export DATABASE_URL="libsql://test-db.turso.io?authToken=..."
# Test all MCP tools with remote database
basic-memory tools # Test each tool functionality
# Test API write → manual sync workflow
```
#### Phase 3: Production Readiness Testing
4. **Performance Benchmarking**:
- Container startup time with TigrisFS mounting
- API operation response times (target: <500ms for typical operations)
- Search query performance with Turso (target: comparable to local SQLite)
- TigrisFS cache hit rates and memory usage
- Concurrent tenant isolation
#### Phase 4: Local Access Testing (If Phase 1-3 Succeed)
5. **Local Access Validation**:
```bash
# Test read-only access
tigrisfs tenant-bucket ~/local-tenant
ls -la ~/local-tenant # Browse files
cp ~/local-tenant/notes/* ~/backup/ # Copy files
# Test file watching (stretch goal)
echo "test" > ~/local-tenant/test.md
# Check if changes sync to cloud
```
### Go/No-Go Criteria by Phase
- **Phase 1**: Container must start successfully and serve API requests over TigrisFS
- **Phase 2**: All MCP tools must work with Turso with <2x latency increase
- **Phase 3**: Performance must be within 50% of current Fly volume setup
- **Phase 4**: Local mounting must work reliably for read-only access
### Risk Assessment
**Moderate Risk Items (Mitigated by API-First Architecture)**:
- [ ] TigrisFS performance for local access may have higher latency than local filesystem
- [ ] File watching (`inotify`) over FUSE may be unreliable for local development
- [ ] Network interruptions could cause filesystem errors during local editing
- [ ] Write-back caching could cause data loss if container crashes during flush
**Low Risk Items (API-First Eliminates)**:
- [ ] ~~Real-time file watching~~ - Not required for cloud operations
- [ ] ~~Concurrent write consistency~~ - Single-tenant model with API coordination
- [ ] ~~S3 rate limits~~ - TigrisFS intelligent caching handles this
**Mitigation Strategies**:
- **Performance**: Comprehensive benchmarking with realistic workloads
- **Reliability**: Graceful degradation to read-only local access if live editing fails
- **Data Safety**: Regular sync intervals and write-through mode for critical operations
- **Fallback**: Keep Fly volumes as backup deployment option
### Metrics to Track
- **API Latency**: Response times for MCP tools and web operations
- **Cache Effectiveness**: TigrisFS cache hit rates and memory usage
- **Local Access Performance**: File browsing and copying speeds
- **Reliability**: Success rate of mount operations and data consistency
- **Cost**: Storage usage, API calls, and network transfer costs vs current volumes
## Notes
### Key Architectural Decisions
- **Single tenant per bucket/database**: Simplifies isolation and credential management
- **Maintain POSIX compatibility**: Preserve Basic Memory's existing filesystem assumptions
- **TigrisFS over rclone**: Purpose-built for object storage with intelligent caching
- **Turso for SQLite**: Leverages specialized remote SQLite expertise
- **API-first approach**: Eliminates file watching dependency for cloud operations
### Alternative Approaches Considered
- **S3-native storage backend**: Would require Basic Memory architecture changes
- **Hybrid approach**: Local files + cloud sync (adds complexity)
- **Standard rclone mounting**: Less optimized than TigrisFS for object storage workloads
- **Keep Fly volumes**: Maintains current limitations but proven reliability
### Integration Points
- [ ] Fly.io Tigris integration for bucket provisioning
- [ ] Turso account setup and database provisioning
- [ ] Container image modifications for TigrisFS support
- [ ] Credential management for tenant isolation
- [ ] API modification for manual sync triggers
- [ ] Local client setup documentation for TigrisFS mounting
## Observations
- [architecture] Tigris/Turso split cleanly separates file storage from indexing concerns #storage-separation
- [breakthrough] API-first architecture eliminates file watching dependency for cloud operations #api-first-advantage
- [user-experience] Local mounting of cloud files could be revolutionary for knowledge management #local-cloud-hybrid
- [compatibility] Maintaining POSIX filesystem assumptions preserves Basic Memory's local/cloud compatibility #architecture-preservation
- [simplification] Single tenant per bucket eliminates complex multi-tenancy in storage layer #tenant-isolation
- [performance] TigrisFS intelligent caching could provide near-local performance for common operations #tigrisfs-advantage
- [deployment] Zero-downtime updates become trivial without volume constraints #deployment-simplification
- [benefit] Object storage pricing model could be more favorable than volume pricing #cost-optimization
- [innovation] Read-only local access alone would address major SaaS limitation #competitive-advantage
- [risk-mitigation] API-driven sync reduces performance requirements vs real-time file watching #risk-reduction
## Relations
- implements [[SPEC-6 Explicit Project Parameter Architecture]]
- requires [[Fly.io Tigris Integration]]
- enables [[Local Cloud File Access]]
- alternative_to [[Fly Volume Storage]]
## Links
- https://fly.io/hello/tigris
- https://fly.io/docs/tigris/
- https://www.tigrisdata.com/docs/sdks/fly/data-migration-with-flyctl/
- https://www.tigrisdata.com/docs/training/tigrisfs/
- https://www.tigrisdata.com/blog/tigris-filesystem/
- https://www.tigrisdata.com/docs/quickstarts/rclone/
-886
View File
@@ -1,886 +0,0 @@
---
title: 'SPEC-8: TigrisFS Integration for Tenant API'
Date: September 22, 2025
Status: Phase 3.6 Complete - Tenant Mount API Endpoints Ready for CLI Implementation
Priority: High
Goal: Replace Fly volumes with Tigris bucket provisioning in production tenant API
permalink: spec-8-tigris-fs-integration
---
## Executive Summary
Based on SPEC-7 Phase 4 POC testing, this spec outlines productizing the TigrisFS/rclone implementation in the Basic Memory Cloud tenant API.
We're moving from proof-of-concept to production integration, replacing Fly volume storage with Tigris bucket-per-tenant architecture.
## Current Architecture (Fly Volumes)
### Tenant Provisioning Flow
```python
# apps/cloud/src/basic_memory_cloud/workflows/tenant_provisioning.py
async def provision_tenant_infrastructure(tenant_id: str):
# 1. Create Fly app
# 2. Create Fly volume ← REPLACE THIS
# 3. Deploy API container with volume mount
# 4. Configure health checks
```
### Storage Implementation
- Each tenant gets dedicated Fly volume (1GB-10GB)
- Volume mounted at `/app/data` in API container
- Local filesystem storage with Basic Memory indexing
- No global caching or edge distribution
## Proposed Architecture (Tigris Buckets)
### New Tenant Provisioning Flow
```python
async def provision_tenant_infrastructure(tenant_id: str):
# 1. Create Fly app
# 2. Create Tigris bucket with admin credentials ← NEW
# 3. Store bucket name in tenant record ← NEW
# 4. Deploy API container with TigrisFS mount using admin credentials
# 5. Configure health checks
```
### Storage Implementation
- Each tenant gets dedicated Tigris bucket
- TigrisFS mounts bucket at `/app/data` in API container
- Global edge caching and distribution
- Configurable cache TTL for sync performance
## Implementation Plan
### Phase 1: Bucket Provisioning Service
**✅ IMPLEMENTED: StorageClient with Admin Credentials**
```python
# apps/cloud/src/basic_memory_cloud/clients/storage_client.py
class StorageClient:
async def create_tenant_bucket(self, tenant_id: UUID) -> TigrisBucketCredentials
async def delete_tenant_bucket(self, tenant_id: UUID, bucket_name: str) -> bool
async def list_buckets(self) -> list[TigrisBucketResponse]
async def test_tenant_credentials(self, credentials: TigrisBucketCredentials) -> bool
```
**Simplified Architecture Using Admin Credentials:**
- Single admin access key with full Tigris permissions (configured in console)
- No tenant-specific IAM user creation needed
- Bucket-per-tenant isolation for logical separation
- Admin credentials shared across all tenant operations
**Integrate with Provisioning workflow:**
```python
# Update tenant_provisioning.py
async def provision_tenant_infrastructure(tenant_id: str):
storage_client = StorageClient(settings.aws_access_key_id, settings.aws_secret_access_key)
bucket_creds = await storage_client.create_tenant_bucket(tenant_id)
await store_bucket_name(tenant_id, bucket_creds.bucket_name)
await deploy_api_with_tigris(tenant_id, bucket_creds)
```
### Phase 2: Simplified Bucket Management
**✅ SIMPLIFIED: Admin Credentials + Bucket Names Only**
Since we use admin credentials for all operations, we only need to track bucket names per tenant:
1. **Primary Storage (Fly Secrets)**
```bash
flyctl secrets set -a basic-memory-{tenant_id} \
AWS_ACCESS_KEY_ID="{admin_access_key}" \
AWS_SECRET_ACCESS_KEY="{admin_secret_key}" \
AWS_ENDPOINT_URL_S3="https://fly.storage.tigris.dev" \
AWS_REGION="auto" \
BUCKET_NAME="basic-memory-{tenant_id}"
```
2. **Database Storage (Bucket Name Only)**
```python
# apps/cloud/src/basic_memory_cloud/models/tenant.py
class Tenant(BaseModel):
# ... existing fields
tigris_bucket_name: Optional[str] = None # Just store bucket name
tigris_region: str = "auto"
created_at: datetime
```
**Benefits of Simplified Approach:**
- No credential encryption/decryption needed
- Admin credentials managed centrally in environment
- Only bucket names stored in database (not sensitive)
- Simplified backup/restore scenarios
- Reduced security attack surface
### Phase 3: API Container Updates
**Update API container configuration:**
```dockerfile
# apps/api/Dockerfile
# Add TigrisFS installation
RUN curl -L https://github.com/tigrisdata/tigrisfs/releases/latest/download/tigrisfs-linux-amd64 \
-o /usr/local/bin/tigrisfs && chmod +x /usr/local/bin/tigrisfs
```
**Startup script integration:**
```bash
# apps/api/tigrisfs-startup.sh (already exists)
# Mount TigrisFS → Start Basic Memory API
exec python -m basic_memory_cloud_api.main
```
**Fly.toml environment (optimized for < 5s startup):**
```toml
# apps/api/fly.tigris-production.toml
[env]
TIGRISFS_MEMORY_LIMIT = '1024' # Reduced for faster init
TIGRISFS_MAX_FLUSHERS = '16' # Fewer threads for faster startup
TIGRISFS_STAT_CACHE_TTL = '30s' # Balance sync speed vs startup
TIGRISFS_LAZY_INIT = 'true' # Enable lazy loading
BASIC_MEMORY_HOME = '/app/data'
# Suspend optimization for wake-on-network
[machine]
auto_stop_machines = "suspend" # Faster than full stop
auto_start_machines = true
min_machines_running = 0
```
### Phase 4: Local Access Features
**CLI automation for local mounting:**
```python
# New CLI command: basic-memory cloud mount
async def setup_local_mount(tenant_id: str):
# 1. Fetch bucket credentials from cloud API
# 2. Configure rclone with scoped IAM policy
# 3. Mount via rclone nfsmount (macOS) or FUSE (Linux)
# 4. Start Basic Memory sync watcher
```
**Local mount configuration:**
```bash
# rclone config for tenant
rclone mount basic-memory-{tenant_id}: ~/basic-memory-{tenant_id} \
--nfs-mount \
--vfs-cache-mode writes \
--cache-dir ~/.cache/rclone/basic-memory-{tenant_id}
```
### Phase 5: TigrisFS Cache Sync Solutions
**Problem**: When files are uploaded via CLI/bisync, the tenant API container doesn't see them immediately due to TigrisFS cache (30s TTL) and lack of inotify events on mounted filesystems.
**Multi-Layer Solution:**
**Layer 1: API Sync Endpoint** (Immediate)
```python
# POST /sync - Force TigrisFS cache refresh
# Callable by CLI after uploads
subprocess.run(["sync", "fsync /app/data"], check=True)
```
**Layer 2: Tigris Webhook Integration** (Real-time)
https://www.tigrisdata.com/docs/buckets/object-notifications/#webhook
```python
# Webhook endpoint for bucket changes
@app.post("/webhooks/tigris/{tenant_id}")
async def handle_bucket_notification(tenant_id: str, event: TigrisEvent):
if event.eventName in ["OBJECT_CREATED_PUT", "OBJECT_DELETED"]:
await notify_container_sync(tenant_id, event.object.key)
```
**Layer 3: CLI Sync Notification** (User-triggered)
```bash
# CLI calls container sync endpoint after successful bisync
basic-memory cloud bisync # Automatically notifies container
curl -X POST https://basic-memory-{tenant-id}.fly.dev/sync
```
**Layer 4: Periodic Sync Fallback** (Safety net)
```python
# Background task: fsync /app/data every 30s as fallback
# Ensures eventual consistency even if other layers fail
```
**Implementation Priority:**
1. Layer 1 (API endpoint) - Quick testing capability
2. Layer 3 (CLI integration) - Improved UX
3. Layer 4 (Periodic fallback) - Safety net
4. Layer 2 (Webhooks) - Production real-time sync
## Performance Targets
### Sync Latency
- **Target**: < 5 seconds local→cloud→container
- **Configuration**: `TIGRISFS_STAT_CACHE_TTL = '5s'`
- **Monitoring**: Track sync metrics in production
### Container Startup
- **Target**: < 5 seconds including TigrisFS mount
- **Fast retry**: 0.5s intervals for mount verification
- **Fallback**: Container fails fast if mount fails
### Memory Usage
- **TigrisFS cache**: 2GB memory limit per container
- **Concurrent uploads**: 32 flushers max
- **VM sizing**: shared-cpu-2x (2048mb) minimum
## Security Considerations
### Bucket Isolation
- Each tenant has dedicated bucket
- IAM policies prevent cross-tenant access
- No shared bucket with subdirectories
### Credential Security
- Fly secrets for runtime access
- Encrypted database backup for disaster recovery
- Credential rotation capability
### Data Residency
- Tigris global edge caching
- SOC2 Type II compliance
- Encryption at rest and in transit
## Operational Benefits
### Scalability
- Horizontal scaling with stateless API containers
- Global edge distribution
- Better resource utilization
### Reliability
- No cold starts between tenants
- Built-in redundancy and caching
- Simplified backup strategy
### Cost Efficiency
- Pay-per-use storage pricing
- Shared infrastructure benefits
- Reduced operational overhead
## Risk Mitigation
### Data Loss Prevention
- Dual credential storage (Fly + database)
- Automated backup workflows to R2/S3
- Tigris built-in redundancy
### Performance Degradation
- Configurable cache settings per tenant
- Monitoring and alerting on sync latency
- Fallback to volume storage if needed
### Security Vulnerabilities
- Bucket-per-tenant isolation
- Regular credential rotation
- Security scanning and monitoring
## Success Metrics
### Technical Metrics
- Sync latency P50 < 5 seconds
- Container startup time < 5 seconds
- Zero data loss incidents
- 99.9% uptime per tenant
### Business Metrics
- Reduced infrastructure costs vs volumes
- Improved user experience with faster sync
- Enhanced enterprise security posture
- Simplified operational overhead
## Open Questions
1. **Tigris rate limits**: What are the API limits for bucket creation?
2. **Cost analysis**: What's the break-even point vs Fly volumes?
3. **Regional preferences**: Should enterprise customers choose regions?
4. **Backup retention**: How long to keep automated backups?
## Implementation Checklist
### Phase 1: Bucket Provisioning Service ✅ COMPLETED
- [x] **Research Tigris bucket API** - Document bucket creation and S3 API compatibility
- [x] **Create StorageClient class** - Implemented with admin credentials and comprehensive integration tests
- [x] **Test bucket creation** - Full test suite validates API integration with real Tigris environment
- [x] **Add bucket provisioning to DBOS workflow** - Integrated StorageClient with tenant_provisioning.py
### Phase 2: Simplified Bucket Management ✅ COMPLETED
- [x] **Update Tenant model** with tigris_bucket_name field (replaced fly_volume_id)
- [x] **Implement bucket name storage** - Database migration and model updates completed
- [x] **Test bucket provisioning integration** - Full test suite validates workflow from tenant creation to bucket assignment
- [x] **Remove volume logic from all tests** - Complete migration from volume-based to bucket-based architecture
### Phase 3: API Container Integration ✅ COMPLETED
- [x] **Update Dockerfile** to install TigrisFS binary in API container with configurable version
- [x] **Optimize tigrisfs-startup.sh** with production-ready security and reliability improvements
- [x] **Create production-ready container** with proper signal handling and mount validation
- [x] **Implement security fixes** based on Claude code review (conditional debug, credential protection)
- [x] **Add proper process supervision** with cleanup traps and error handling
- [x] **Remove debug artifacts** - Cleaned up all debug Dockerfiles and test scripts
### Phase 3.5: IAM Access Key Management ✅ COMPLETED
- [x] **Research Tigris IAM API** - Documented create_policy, attach_user_policy, delete_access_key operations
- [x] **Implement bucket-scoped credential generation** - StorageClient.create_tenant_access_keys() with IAM policies
- [x] **Add comprehensive security test suite** - 5 security-focused integration tests covering all attack vectors
- [x] **Verify cross-bucket access prevention** - Scoped credentials can ONLY access their designated bucket
- [x] **Test credential lifecycle management** - Create, validate, delete, and revoke access keys
- [x] **Validate admin vs scoped credential isolation** - Different access patterns and security boundaries
- [x] **Test multi-tenant isolation** - Multiple tenants cannot access each other's buckets
### Phase 3.6: Tenant Mount API Endpoints ✅ COMPLETED
- [x] **Implement GET /tenant/mount/info** - Returns mount info without exposing credentials
- [x] **Implement POST /tenant/mount/credentials** - Creates new bucket-scoped credentials for CLI mounting
- [x] **Implement DELETE /tenant/mount/credentials/{cred_id}** - Revoke specific credentials with proper cleanup
- [x] **Implement GET /tenant/mount/credentials** - List active credentials without exposing secrets
- [x] **Add TenantMountCredentials database model** - Tracks credential metadata (no secret storage)
- [x] **Create comprehensive test suite** - 28 tests covering all scenarios including multi-session support
- [x] **Implement multi-session credential flow** - Multiple active credentials per tenant supported
- [x] **Secure credential handling** - Secret keys never stored, returned once only for immediate use
- [x] **Add dependency injection for StorageClient** - Clean integration with existing API architecture
- [x] **Fix Tigris configuration for cloud service** - Added AWS environment variables to fly.template.toml
- [x] **Update tenant machine configurations** - Include AWS credentials for TigrisFS mounting with clear credential strategy
**Security Test Results:**
```
✅ Cross-bucket access prevention - PASS
✅ Deleted credentials access revoked - PASS
✅ Invalid credentials rejected - PASS
✅ Admin vs scoped credential isolation - PASS
✅ Multiple scoped credentials isolation - PASS
```
**Implementation Details:**
- Uses Tigris IAM managed policies (create_policy + attach_user_policy)
- Bucket-scoped S3 policies with Actions: GetObject, PutObject, DeleteObject, ListBucket
- Resource ARNs limited to specific bucket: `arn:aws:s3:::bucket-name` and `arn:aws:s3:::bucket-name/*`
- Access keys follow Tigris format: `tid_` prefix with secure random suffix
- Complete cleanup on deletion removes both access keys and associated policies
### Phase 4: Local Access CLI
- [x] **Design local mount CLI command** for automated rclone configuration
- [x] **Implement credential fetching** from cloud API for local setup
- [x] **Create rclone config automation** for tenant-specific bucket mounting
- [x] **Test local→cloud→container sync** with optimized cache settings
- [x] **Document local access setup** for beta users
### Phase 5: Webhook Integration (Future)
- [ ] **Research Tigris webhook API** for object notifications and payload format
- [ ] **Design webhook endpoint** for real-time sync notifications
- [ ] **Implement notification handling** to trigger Basic Memory sync events
- [ ] **Test webhook delivery** and sync latency improvements
## Success Metrics
- [ ] **Container startup < 5 seconds** including TigrisFS mount and Basic Memory init
- [ ] **Sync latency < 5 seconds** for local→cloud→container file changes
- [ ] **Zero data loss** during bucket provisioning and credential management
- [ ] **100% test coverage** for new TigrisBucketService and credential functions
- [ ] **Beta deployment** with internal users validating local-cloud workflow
## Implementation Notes
## Phase 4.1: Bidirectional Sync with rclone bisync (NEW)
### Problem Statement
During testing, we discovered that some applications (particularly Obsidian) don't detect file changes over NFS mounts. Rather than building a custom sync daemon, we can leverage `rclone bisync` - rclone's built-in bidirectional synchronization feature.
### Solution: rclone bisync
Use rclone's proven bidirectional sync instead of custom implementation:
**Core Architecture:**
```bash
# rclone bisync handles all the complexity
rclone bisync ~/basic-memory-{tenant_id} basic-memory-{tenant_id}:{bucket_name} \
--create-empty-src-dirs \
--conflict-resolve newer \
--resilient \
--check-access
```
**Key Benefits:**
- ✅ **Battle-tested**: Production-proven rclone functionality
- ✅ **MIT licensed**: Open source with permissive licensing
- ✅ **No custom code**: Zero maintenance burden for sync logic
- ✅ **Built-in safety**: max-delete protection, conflict resolution
- ✅ **Simple installation**: Works with Homebrew rclone (no FUSE needed)
- ✅ **File watcher compatible**: Works with Obsidian and all applications
- ✅ **Offline support**: Can work offline and sync when connected
### bisync Conflict Resolution Options
**Built-in conflict strategies:**
```bash
--conflict-resolve none # Keep both files with .conflict suffixes (safest)
--conflict-resolve newer # Always pick the most recently modified file
--conflict-resolve larger # Choose based on file size
--conflict-resolve path1 # Always prefer local changes
--conflict-resolve path2 # Always prefer cloud changes
```
### Sync Profiles Using bisync
**Profile configurations:**
```python
BISYNC_PROFILES = {
"safe": {
"conflict_resolve": "none", # Keep both versions
"max_delete": 10, # Prevent mass deletion
"check_access": True, # Verify sync integrity
"description": "Safe mode with conflict preservation"
},
"balanced": {
"conflict_resolve": "newer", # Auto-resolve to newer file
"max_delete": 25,
"check_access": True,
"description": "Balanced mode (recommended default)"
},
"fast": {
"conflict_resolve": "newer",
"max_delete": 50,
"check_access": False, # Skip verification for speed
"description": "Fast mode for rapid iteration"
}
}
```
### CLI Commands
**Manual sync commands:**
```bash
basic-memory cloud bisync # Manual bidirectional sync
basic-memory cloud bisync --dry-run # Preview changes
basic-memory cloud bisync --profile safe # Use specific profile
basic-memory cloud bisync --resync # Force full baseline resync
```
**Watch mode (Step 1):**
```bash
basic-memory cloud bisync --watch # Long-running process, sync every 60s
basic-memory cloud bisync --watch --interval 30s # Custom interval
```
**System integration (Step 2 - Future):**
```bash
basic-memory cloud bisync-service install # Install as system service
basic-memory cloud bisync-service start # Start background service
basic-memory cloud bisync-service status # Check service status
```
### Implementation Strategy
**Phase 4.1.1: Core bisync Implementation**
- [ ] Implement `run_bisync()` function wrapping rclone bisync
- [ ] Add profile-based configuration (safe/balanced/fast)
- [ ] Create conflict resolution and safety options
- [ ] Test with sample files and conflict scenarios
**Phase 4.1.2: Watch Mode**
- [ ] Add `--watch` flag for continuous sync
- [ ] Implement configurable sync intervals
- [ ] Add graceful shutdown and signal handling
- [ ] Create status monitoring and progress indicators
**Phase 4.1.3: User Experience**
- [ ] Add conflict reporting and resolution guidance
- [ ] Implement dry-run preview functionality
- [ ] Create troubleshooting and diagnostic commands
- [ ] Add filtering configuration (.gitignore-style)
**Phase 4.1.4: System Integration (Future)**
- [ ] Generate platform-specific service files (launchd/systemd)
- [ ] Add service management commands
- [ ] Implement automatic startup and recovery
- [ ] Create monitoring and logging integration
### Technical Implementation
**Core bisync wrapper:**
```python
def run_bisync(
tenant_id: str,
bucket_name: str,
profile: str = "balanced",
dry_run: bool = False
) -> bool:
"""Run rclone bisync with specified profile."""
local_path = Path.home() / f"basic-memory-{tenant_id}"
remote_path = f"basic-memory-{tenant_id}:{bucket_name}"
profile_config = BISYNC_PROFILES[profile]
cmd = [
"rclone", "bisync",
str(local_path), remote_path,
"--create-empty-src-dirs",
"--resilient",
f"--conflict-resolve={profile_config['conflict_resolve']}",
f"--max-delete={profile_config['max_delete']}",
"--filters-file", "~/.basic-memory/bisync-filters.txt"
]
if profile_config.get("check_access"):
cmd.append("--check-access")
if dry_run:
cmd.append("--dry-run")
return subprocess.run(cmd, check=True).returncode == 0
```
**Default filter file (~/.basic-memory/bisync-filters.txt):**
```
- .DS_Store
- .git/**
- __pycache__/**
- *.pyc
- .pytest_cache/**
- node_modules/**
- .conflict-*
- Thumbs.db
- desktop.ini
```
**Advantages Over Custom Daemon:**
- ✅ **Zero maintenance**: No custom sync logic to debug/maintain
- ✅ **Production proven**: Used by thousands in production
- ✅ **Safety features**: Built-in max-delete, conflict handling, recovery
- ✅ **Filtering**: Advanced exclude patterns and rules
- ✅ **Performance**: Optimized for various storage backends
- ✅ **Community support**: Extensive documentation and community
## Phase 4.2: NFS Mount Support (Direct Access)
### Solution: rclone nfsmount
Keep the existing NFS mount functionality for users who prefer direct file access:
**Core Architecture:**
```bash
# rclone nfsmount provides transparent file access
rclone nfsmount basic-memory-{tenant_id}:{bucket_name} ~/basic-memory-{tenant_id} \
--vfs-cache-mode writes \
--dir-cache-time 10s \
--daemon
```
**Key Benefits:**
- ✅ **Real-time access**: Files appear immediately as they're created/modified
- ✅ **Transparent**: Works with any application that reads/writes files
- ✅ **Low latency**: Direct access without sync delays
- ✅ **Simple**: No periodic sync commands needed
- ✅ **Homebrew compatible**: Works with Homebrew rclone (no FUSE required)
**Limitations:**
- ❌ **File watcher compatibility**: Some apps (Obsidian) don't detect changes over NFS
- ❌ **Network dependency**: Requires active connection to cloud storage
- ❌ **Potential conflicts**: Simultaneous edits from multiple locations can cause issues
### Mount Profiles (Existing)
**Already implemented profiles from SPEC-7 testing:**
```python
MOUNT_PROFILES = {
"fast": {
"cache_time": "5s",
"poll_interval": "3s",
"description": "Ultra-fast development (5s sync)"
},
"balanced": {
"cache_time": "10s",
"poll_interval": "5s",
"description": "Fast development (10-15s sync, recommended)"
},
"safe": {
"cache_time": "15s",
"poll_interval": "10s",
"description": "Conflict-aware mount with backup",
"extra_args": ["--conflict-suffix", ".conflict-{DateTimeExt}"]
}
}
```
### CLI Commands (Existing)
**Mount commands already implemented:**
```bash
basic-memory cloud mount # Mount with balanced profile
basic-memory cloud mount --profile fast # Ultra-fast caching
basic-memory cloud mount --profile safe # Conflict detection
basic-memory cloud unmount # Clean unmount
basic-memory cloud mount-status # Show mount status
```
## User Choice: Mount vs Bisync
### When to Use Each Approach
| Use Case | Recommended Solution | Why |
|----------|---------------------|-----|
| **Obsidian users** | `bisync` | File watcher support for live preview |
| **CLI/vim/emacs users** | `mount` | Direct file access, lower latency |
| **Offline work** | `bisync` | Can work offline, sync when connected |
| **Real-time collaboration** | `mount` | Immediate visibility of changes |
| **Multiple machines** | `bisync` | Better conflict handling |
| **Single machine** | `mount` | Simpler, more transparent |
| **Development work** | Either | Both work well, user preference |
| **Large files** | `mount` | Streaming access vs full download |
### Installation Simplicity
**Both approaches now use simple Homebrew installation:**
```bash
# Single installation command for both approaches
brew install rclone
# No macFUSE, no system modifications needed
# Works immediately with both mount and bisync
```
### Implementation Status
**Phase 4.1: bisync** (NEW)
- [ ] Implement bisync command wrapper
- [ ] Add watch mode with configurable intervals
- [ ] Create conflict resolution workflows
- [ ] Add filtering and safety options
**Phase 4.2: mount** (EXISTING - ✅ IMPLEMENTED)
- [x] NFS mount commands with profile support
- [x] Mount management and cleanup
- [x] Process monitoring and health checks
- [x] Credential integration with cloud API
**Both approaches share:**
- [x] Credential management via cloud API
- [x] Secure rclone configuration
- [x] Tenant isolation and bucket scoping
- [x] Simple Homebrew rclone installation
Key Features:
1. Cross-Platform rclone Installation (rclone_installer.py):
- macOS: Homebrew → official script fallback
- Linux: snap → apt → official script fallback
- Windows: winget → chocolatey → scoop fallback
- Automatic version detection and verification
2. Smart rclone Configuration (rclone_config.py):
- Automatic tenant-specific config generation
- Three optimized mount profiles from your SPEC-7 testing:
- fast: 5s sync (ultra-performance)
- balanced: 10-15s sync (recommended default)
- safe: 15s sync + conflict detection
- Backup existing configs before modification
3. Robust Mount Management (mount_commands.py):
- Automatic tenant credential generation
- Mount path management (~/basic-memory-{tenant-id})
- Process lifecycle management (prevent duplicate mounts)
- Orphaned process cleanup
- Mount verification and health checking
4. Clean Architecture (api_client.py):
- Separated API client to avoid circular imports
- Reuses existing authentication infrastructure
- Consistent error handling and logging
User Experience:
One-Command Setup:
basic-memory cloud setup
```bash
# 1. Installs rclone automatically
# 2. Authenticates with existing login
# 3. Generates secure credentials
# 4. Configures rclone
# 5. Performs initial mount
```
Profile-Based Mounting:
basic-memory cloud mount --profile fast # 5s sync
basic-memory cloud mount --profile balanced # 15s sync (default)
basic-memory cloud mount --profile safe # conflict detection
Status Monitoring:
basic-memory cloud mount-status
```bash
# Shows: tenant info, mount path, sync profile, rclone processes
```
### local mount api
Endpoint 1: Get Tenant Info for user
Purpose: Get tenant details for mounting
- pass in jwt
- service returns mount info
**✅ IMPLEMENTED API Specification:**
**Endpoint 1: GET /tenant/mount/info**
- Purpose: Get tenant mount information without exposing credentials
- Authentication: JWT token (tenant_id extracted from claims)
Request:
```
GET /tenant/mount/info
Authorization: Bearer {jwt_token}
```
Response:
```json
{
"tenant_id": "434252dd-d83b-4b20-bf70-8a950ff875c4",
"bucket_name": "basic-memory-434252dd",
"has_credentials": true,
"credentials_created_at": "2025-09-22T16:48:50.414694"
}
```
**Endpoint 2: POST /tenant/mount/credentials**
- Purpose: Generate NEW bucket-scoped S3 credentials for rclone mounting
- Authentication: JWT token (tenant_id extracted from claims)
- Multi-session: Creates new credentials without revoking existing ones
Request:
```
POST /tenant/mount/credentials
Authorization: Bearer {jwt_token}
Content-Type: application/json
```
*Note: No request body needed - tenant_id extracted from JWT*
Response:
```json
{
"tenant_id": "434252dd-d83b-4b20-bf70-8a950ff875c4",
"bucket_name": "basic-memory-434252dd",
"access_key": "test_access_key_12345",
"secret_key": "test_secret_key_abcdef",
"endpoint_url": "https://fly.storage.tigris.dev",
"region": "auto"
}
```
**🔒 Security Notes:**
- Secret key returned ONCE only - never stored in database
- Credentials are bucket-scoped (cannot access other tenants' buckets)
- Multiple active credentials supported per tenant (work laptop + personal machine)
Implementation Notes
Security:
- Both endpoints require JWT authentication
- Extract tenant_id from JWT claims (not request body)
- Generate scoped credentials (not admin credentials)
- Credentials should have bucket-specific access only
Integration Points:
- Use your existing StorageClient from SPEC-8 implementation
- Leverage existing JWT middleware for tenant extraction
- Return same credential format as your Tigris bucket provisioning
Error Handling:
- 401 if not authenticated
- 403 if tenant doesn't exist
- 500 if credential generation fails
**🔄 Design Decisions:**
1. **Secure Credential Flow (No Secret Storage)**
Based on CLI flow analysis, we follow security best practices:
- ✅ API generates both access_key + secret_key via Tigris IAM
- ✅ Returns both in API response for immediate use
- ✅ CLI uses credentials immediately to configure rclone
- ✅ Database stores only metadata (access_key + policy_arn for cleanup)
- ✅ rclone handles secure local credential storage
- ❌ **Never store secret_key in database (even encrypted)**
2. **CLI Credential Flow**
```bash
# CLI calls API
POST /tenant/mount/credentials → {access_key, secret_key, ...}
# CLI immediately configures rclone
rclone config create basic-memory-{tenant_id} s3 \
access_key_id={access_key} \
secret_access_key={secret_key} \
endpoint=https://fly.storage.tigris.dev
# Database tracks metadata only
INSERT INTO tenant_mount_credentials (tenant_id, access_key, policy_arn, ...)
```
3. **Multiple Sessions Supported**
- Users can have multiple active credential sets (work laptop, personal machine, etc.)
- Each credential generation creates a new Tigris access key
- List active credentials via API (shows access_key but never secret)
4. **Failure Handling & Cleanup**
- **Happy Path**: Credentials created → Used immediately → rclone configured
- **Orphaned Credentials**: Background job revokes unused credentials
- **API Failure Recovery**: Retry Tigris deletion with stored policy_arn
- **Status Tracking**: Track tigris_deletion_status (pending/completed/failed)
5. **Event Sourcing & Audit**
- MountCredentialCreatedEvent
- MountCredentialRevokedEvent
- MountCredentialOrphanedEvent (for cleanup)
- Full audit trail for security compliance
6. **Tenant/Bucket Validation**
- Verify tenant exists and has valid bucket before credential generation
- Use existing StorageClient to validate bucket access
- Prevent credential generation for inactive/invalid tenants
📋 **Implemented API Endpoints:**
```
✅ IMPLEMENTED:
GET /tenant/mount/info # Get tenant/bucket info (no credentials exposed)
POST /tenant/mount/credentials # Generate new credentials (returns secret once)
GET /tenant/mount/credentials # List active credentials (no secrets)
DELETE /tenant/mount/credentials/{cred_id} # Revoke specific credentials
```
**API Implementation Status:**
- ✅ **GET /tenant/mount/info**: Returns tenant_id, bucket_name, has_credentials, credentials_created_at
- ✅ **POST /tenant/mount/credentials**: Creates new bucket-scoped access keys, returns access_key + secret_key once
- ✅ **GET /tenant/mount/credentials**: Lists active credentials without exposing secret keys
- ✅ **DELETE /tenant/mount/credentials/{cred_id}**: Revokes specific credentials with proper Tigris IAM cleanup
- ✅ **Multi-session support**: Multiple active credentials per tenant (work laptop + personal machine)
- ✅ **Security**: Secret keys never stored in database, returned once only for immediate use
- ✅ **Comprehensive test suite**: 28 tests covering all scenarios including error handling and multi-session flows
- ✅ **Dependency injection**: Clean integration with existing FastAPI architecture
- ✅ **Production-ready configuration**: Tigris credentials properly configured for tenant machines
🗄️ **Secure Database Schema:**
```sql
CREATE TABLE tenant_mount_credentials (
id UUID PRIMARY KEY,
tenant_id UUID REFERENCES tenant(id),
access_key VARCHAR(255) NOT NULL,
-- secret_key REMOVED - never store secrets (security best practice)
policy_arn VARCHAR(255) NOT NULL, -- For Tigris IAM cleanup
tigris_deletion_status VARCHAR(20) DEFAULT 'pending', -- Track cleanup
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW(),
revoked_at TIMESTAMP NULL,
last_used_at TIMESTAMP NULL, -- Track usage for orphan cleanup
description VARCHAR(255) DEFAULT 'CLI mount credentials'
);
```
**Security Benefits:**
- ✅ Database breach cannot expose secrets
- ✅ Follows "secrets don't persist" security principle
- ✅ Meets compliance requirements (SOC2, etc.)
- ✅ Reduced attack surface
- ✅ CLI gets credentials once and stores securely via rclone
File diff suppressed because it is too large Load Diff
@@ -1,196 +0,0 @@
---
title: 'SPEC-9: Signed Header Tenant Information'
type: spec
permalink: specs/spec-9-signed-header-tenant-information
tags:
- authentication
- tenant-isolation
- proxy
- security
- mcp
---
# SPEC-9: Signed Header Tenant Information
## Why
WorkOS JWT templates don't work with MCP's dynamic client registration requirement, preventing us from getting tenant information directly in JWT tokens. We need an alternative secure method to pass tenant context from the Cloud Proxy Service to tenant instances.
**Problem Context:**
- MCP spec requires dynamic client registration
- WorkOS JWT templates only apply to statically configured clients
- Without tenant information, we can't properly route requests or isolate tenant data
- Current JWT tokens only contain standard OIDC claims (sub, email, etc.)
**Affected Areas:**
- Cloud Proxy Service (`apps/cloud`) - request forwarding
- Tenant API instances (`apps/api`) - tenant context validation
- MCP Gateway (`apps/mcp`) - authentication flow
- Overall tenant isolation security model
## What
Implement HMAC-signed headers that the Cloud Proxy Service adds when forwarding requests to tenant instances. This provides secure, tamper-proof tenant information without relying on JWT custom claims.
**Components:**
- Header signing utility in Cloud Proxy Service
- Header validation middleware in Tenant API instances
- Shared secret configuration across services
- Fallback mechanisms for development and error cases
## How (High Level)
### 1. Header Format
Add these signed headers to all proxied requests:
```
X-BM-Tenant-ID: {tenant_id}
X-BM-Timestamp: {unix_timestamp}
X-BM-Signature: {hmac_sha256_signature}
```
### 2. Signature Algorithm
```python
# Canonical message format
message = f"{tenant_id}:{timestamp}"
# HMAC-SHA256 signature
signature = hmac.new(
key=shared_secret.encode('utf-8'),
msg=message.encode('utf-8'),
digestmod=hashlib.sha256
).hexdigest()
```
### 3. Implementation Flow
#### Cloud Proxy Service (`apps/cloud`)
1. Extract `tenant_id` from authenticated user profile
2. Generate timestamp and canonical message
3. Sign message with shared secret
4. Add headers to request before forwarding to tenant instance
#### Tenant API Instances (`apps/api`)
1. Middleware validates headers on all incoming requests
2. Extract tenant_id, timestamp from headers
3. Verify timestamp is within acceptable window (5 minutes)
4. Recompute signature and compare in constant time
5. If valid, make tenant context available to Basic Memory tools
### 4. Security Properties
- **Authenticity**: Only services with shared secret can create valid signatures
- **Integrity**: Header tampering invalidates signature
- **Replay Protection**: Timestamp prevents reuse of old signatures
- **Non-repudiation**: Each request is cryptographically tied to specific tenant
### 5. Configuration
```bash
# Shared across Cloud Proxy and Tenant instances
BM_TENANT_HEADER_SECRET=randomly-generated-256-bit-secret
# Tenant API configuration
BM_TENANT_HEADER_VALIDATION=true # true (production) | false (dev only)
```
## How to Evaluate
### Unit Tests
- [ ] Header signing utility generates correct signatures
- [ ] Header validation correctly accepts/rejects signatures
- [ ] Timestamp validation within acceptable windows
- [ ] Constant-time signature comparison prevents timing attacks
### Integration Tests
- [ ] End-to-end request flow from MCP client → proxy → tenant
- [ ] Tenant isolation verified with signed headers
- [ ] Error handling for missing/invalid headers
- [ ] Disabled validation in development environment
### Security Validation
- [ ] Shared secret rotation procedure
- [ ] Header tampering detection
- [ ] Clock skew tolerance testing
- [ ] Performance impact measurement
### Production Readiness
- [ ] Logging and monitoring of header validation
- [ ] Graceful degradation for header validation failures
- [ ] Documentation for secret management
- [ ] Deployment configuration templates
## Implementation Notes
### Shared Secret Management
- Generate cryptographically secure 256-bit secret
- Same secret deployed to Cloud Proxy and all Tenant instances
- Consider secret rotation strategy for production
### Error Handling
```python
# Strict mode (production)
if not validate_headers(request):
raise HTTPException(status_code=401, detail="Invalid tenant headers")
# Fallback mode (development)
if not validate_headers(request):
logger.warning("Invalid headers, falling back to default tenant")
tenant_id = "default"
```
### Performance Considerations
- HMAC-SHA256 computation is fast (~microseconds)
- Headers add ~200 bytes to each request
- Validation happens once per request in middleware
## Benefits
**Works with MCP dynamic client registration** - No dependency on JWT custom claims
**Simple and reliable** - Standard HMAC signature approach
**Secure by design** - Cryptographic authenticity and integrity
**Infrastructure controlled** - No external service dependencies
**Easy to implement** - Clear signature algorithm and validation
## Trade-offs
⚠️ **Shared secret management** - Need secure distribution and rotation
⚠️ **Clock synchronization** - Timestamp validation requires reasonably synced clocks
⚠️ **Header visibility** - Headers visible in logs (tenant_id not sensitive)
⚠️ **Additional complexity** - More moving parts in proxy forwarding
## Implementation Tasks
### Cloud Service (Header Signing)
- [ ] Create `utils/header_signing.py` with HMAC-SHA256 signing function
- [ ] Add `bm_tenant_header_secret` to Cloud service configuration
- [ ] Update `ProxyService.forward_request()` to call signing utility
- [ ] Add signed headers (X-BM-Tenant-ID, X-BM-Timestamp, X-BM-Signature)
### Tenant API (Header Validation)
- [ ] Create `utils/header_validation.py` with signature verification
- [ ] Add `bm_tenant_header_secret` to API service configuration
- [ ] Create `TenantHeaderValidationMiddleware` class
- [ ] Add middleware to FastAPI app (before other middleware)
- [ ] Skip validation for `/health` endpoint
- [ ] Store validated tenant_id in request.state
### Testing
- [ ] Unit test for header signing utility
- [ ] Unit test for header validation utility
- [ ] Integration test for proxy → tenant flow
- [ ] Test invalid/missing header handling
- [ ] Test timestamp window validation
- [ ] Test signature tampering detection
### Configuration & Deployment
- [ ] Update `.env.example` with BM_TENANT_HEADER_SECRET
- [ ] Generate secure 256-bit secret for production
- [ ] Update Fly.io secrets for both services
- [ ] Document secret rotation procedure
## Status
- [x] **Specification Complete** - Design finalized and documented
- [ ] **Implementation Started** - Header signing utility development
- [ ] **Cloud Proxy Updated** - ProxyService adds signed headers
- [ ] **Tenant Validation Added** - Middleware validates headers
- [ ] **Testing Complete** - All validation criteria met
- [ ] **Production Deployed** - Live with tenant isolation via headers
@@ -1,390 +0,0 @@
---
title: 'SPEC-9-1 Follow-Ups: Conflict, Sync, and Observability'
type: tasklist
permalink: specs/spec-9-follow-ups-conflict-sync-and-observability
related: specs/spec-9-multi-project-bisync
status: revised
revision_date: 2025-10-03
---
# SPEC-9-1 Follow-Ups: Conflict, Sync, and Observability
**REVISED 2025-10-03:** Simplified to leverage rclone built-ins instead of custom conflict handling.
**Context:** SPEC-9 delivered multi-project bidirectional sync and a unified CLI. This follow-up focuses on **observability and safety** using rclone's built-in capabilities rather than reinventing conflict handling.
**Design Philosophy: "Be Dumb Like Git"**
- Let rclone bisync handle conflict detection (it already does this)
- Make conflicts visible and recoverable, don't prevent them
- Cloud is always the winner on conflict (cloud-primary model)
- Users who want version history can just use Git locally in their sync directory
**What Changed from Original Version:**
- **Replaced:** Custom `.bmmeta` sidecars → Use rclone's `.bisync/` state tracking
- **Replaced:** Custom conflict detection → Use rclone bisync 3-way merge
- **Replaced:** Tombstone files → rclone delete tracking handles this
- **Replaced:** Distributed lease → Local process lock only (document multi-device warning)
- **Replaced:** S3 versioning service → Users just use Git locally if they want history
- **Deferred:** SPEC-14 Git integration → Postponed to teams/multi-user features
## ✅ Now
- [ ] **Local process lock**: Prevent concurrent bisync runs on same device (`~/.basic-memory/sync.lock`)
- [ ] **Structured sync reports**: Parse rclone bisync output into JSON reports (creates/updates/deletes/conflicts, bytes, duration); `bm sync --report`
- [ ] **Multi-device warning**: Document that users should not run `--watch` on multiple devices simultaneously
- [ ] **Version control guidance**: Document pattern for users to use Git locally in their sync directory if they want version history
- [ ] **Docs polish**: cloud-mode toggle, mount↔bisync directory isolation, conflict semantics, quick start, migration guide, short demo clip/GIF
## 🔜 Next
- [ ] **Observability commands**: `bm conflicts list`, `bm sync history` to view sync reports and conflicts
- [ ] **Conflict resolution UI**: `bm conflicts resolve <file>` to interactively pick winner from conflict files
- [ ] **Selective sync**: allow include/exclude by project; per-project profile (safe/balanced/fast)
## 🧭 Later
- [ ] **Near real-time sync**: File watcher → targeted `rclone copy` for individual files (keep bisync as backstop)
- [ ] **Sharing / scoped tokens**: cross-tenant/project access
- [ ] **Bandwidth controls & backpressure**: policy for large repos
- [ ] **Client-side encryption (optional)**: with clear trade-offs
## 📏 Acceptance criteria (for "Now" items)
- [ ] Local process lock prevents concurrent bisync runs on same device
- [ ] rclone bisync conflict files visible and documented (`file.conflict1.md`, `file.conflict2.md`)
- [ ] `bm sync --report` generates parsable JSON with sync statistics
- [ ] Documentation clearly warns about multi-device `--watch` mode
- [ ] Documentation shows users how to use Git locally for version history
## What We're NOT Building (Deferred to rclone)
- ❌ Custom `.bmmeta` sidecars (rclone tracks state in `.bisync/` workdir)
- ❌ Custom conflict detection (rclone bisync already does 3-way merge detection)
- ❌ Tombstone files (S3 versioning + rclone delete tracking handles this)
- ❌ Distributed lease (low probability issue, rclone detects state divergence)
- ❌ Rename/move tracking (rclone has size+modtime heuristics built-in)
## Implementation Summary
**Current State (SPEC-9):**
- ✅ rclone bisync with 3 profiles (safe/balanced/fast)
- ✅ `--max-delete` safety limits (10/25/50 files)
- ✅ `--conflict-resolve=newer` for auto-resolution
- ✅ Watch mode: `bm sync --watch` (60s intervals)
- ✅ Integrity checking: `bm cloud check`
- ✅ Mount vs bisync directory isolation
**What's Needed (This Spec):**
1. **Process lock** - Simple file-based lock in `~/.basic-memory/sync.lock`
2. **Sync reports** - Parse rclone output, save to `~/.basic-memory/sync-history/`
3. **Documentation** - Multi-device warnings, conflict resolution workflow, Git usage pattern
**User Model:**
- Cloud is always the winner on conflict (cloud-primary)
- rclone creates `.conflict` files for divergent edits
- Users who want version history just use Git in their local sync directory
- Users warned: don't run `--watch` on multiple devices
## Decision Rationale & Trade-offs
### Why Trust rclone Instead of Custom Conflict Handling?
**rclone bisync already provides:**
- 3-way merge detection (compares local, remote, and last-known state)
- File state tracking in `.bisync/` workdir (hashes, modtimes)
- Automatic conflict file creation: `file.conflict1.md`, `file.conflict2.md`
- Rename detection via size+modtime heuristics
- Delete tracking (prevents resurrection of deleted files)
- Battle-tested with extensive edge case handling
**What we'd have to build with custom approach:**
- Per-file metadata tracking (`.bmmeta` sidecars)
- 3-way diff algorithm
- Conflict detection logic
- Tombstone files for deletes
- Rename/move detection
- Testing for all edge cases
**Decision:** Use what rclone already does well. Don't reinvent the wheel.
### Why Let Users Use Git Locally Instead of Building Versioning?
**The simplest solution: Just use Git**
Users who want version history can literally just use Git in their sync directory:
```bash
cd ~/basic-memory-cloud-sync/
git init
git add .
git commit -m "backup"
# Push to their own GitHub if they want
git remote add origin git@github.com:user/my-knowledge.git
git push
```
**Why this is perfect:**
- ✅ We build nothing
- ✅ Users who want Git... just use Git
- ✅ Users who don't care... don't need to
- ✅ rclone bisync already handles sync conflicts
- ✅ Users own their data, they can version it however they want (Git, Time Machine, etc.)
**What we'd have to build for S3 versioning:**
- API to enable versioning on Tigris buckets
- **Problem**: Tigris doesn't support S3 bucket versioning
- Restore commands: `bm cloud restore --version-id`
- Version listing: `bm cloud versions <path>`
- Lifecycle policies for version retention
- Documentation and user education
**What we'd have to build for SPEC-14 Git integration:**
- Committer service (daemon watching `/app/data/`)
- Puller service (webhook handler for GitHub pushes)
- Git LFS for large files
- Loop prevention between Git ↔ bisync ↔ local
- Merge conflict handling at TWO layers (rclone + Git)
- Webhook infrastructure and monitoring
**Decision:** Don't build version control. Document the pattern. "The easiest problem to solve is the one you avoid."
**When to revisit:** Teams/multi-user features where server-side version control becomes necessary for collaboration.
### Why No Distributed Lease?
**Low probability issue:**
- Requires user to manually run `bm sync` on multiple devices at exact same time
- Most users run `--watch` on one primary device
- rclone bisync detects state divergence and fails safely
**Safety nets in place:**
- Local process lock prevents concurrent runs on same device
- rclone bisync aborts if bucket state changed during sync
- S3 versioning recovers from any overwrites
- Documentation warns against multi-device `--watch`
**Failure mode:**
```bash
# Device A and B sync simultaneously
Device A: bm sync → succeeds
Device B: bm sync → "Error: path has changed, run --resync"
# User fixes with resync
Device B: bm sync --resync → establishes new baseline
```
**Decision:** Document the issue, add local lock, defer distributed coordination until users report actual problems.
### Cloud-Primary Conflict Model
**User mental model:**
- Cloud is the source of truth (like Dropbox/iCloud)
- Local is working copy
- On conflict: cloud wins, local edits → `.conflict` file
- User manually picks winner
**Why this works:**
- Simpler than bidirectional merge (no automatic resolution risk)
- Matches user expectations from Dropbox
- S3 versioning provides safety net for overwrites
- Clear recovery path: restore from S3 version if needed
**Example workflow:**
```bash
# Edit file on Device A and Device B while offline
# Both devices come online and sync
Device A: bm sync
# → Pushes to cloud first, becomes canonical version
Device B: bm sync
# → Detects conflict
# → Cloud version: work/notes.md
# → Local version: work/notes.md.conflict1
# → User manually merges or picks winner
# Restore if needed
bm cloud restore work/notes.md --version-id abc123
```
## Implementation Details
### 1. Local Process Lock
```python
# ~/.basic-memory/sync.lock
import os
import psutil
from pathlib import Path
class SyncLock:
def __init__(self):
self.lock_file = Path.home() / '.basic-memory' / 'sync.lock'
def acquire(self):
if self.lock_file.exists():
pid = int(self.lock_file.read_text())
if psutil.pid_exists(pid):
raise BisyncError(
f"Sync already running (PID {pid}). "
f"Wait for completion or kill stale process."
)
# Stale lock, remove it
self.lock_file.unlink()
self.lock_file.write_text(str(os.getpid()))
def release(self):
if self.lock_file.exists():
self.lock_file.unlink()
def __enter__(self):
self.acquire()
return self
def __exit__(self, *args):
self.release()
# Usage
with SyncLock():
run_rclone_bisync()
```
### 3. Sync Report Parsing
```python
# Parse rclone bisync output
import json
from datetime import datetime
from pathlib import Path
def parse_sync_report(rclone_output: str, duration: float, exit_code: int) -> dict:
"""Parse rclone bisync output into structured report."""
# rclone bisync outputs lines like:
# "Synching Path1 /local/path with Path2 remote:bucket"
# "- Path1 File was copied to Path2"
# "Bisync successful"
report = {
"timestamp": datetime.now().isoformat(),
"duration_seconds": duration,
"exit_code": exit_code,
"success": exit_code == 0,
"files_created": 0,
"files_updated": 0,
"files_deleted": 0,
"conflicts": [],
"errors": []
}
for line in rclone_output.split('\n'):
if 'was copied to' in line:
report['files_created'] += 1
elif 'was updated in' in line:
report['files_updated'] += 1
elif 'was deleted from' in line:
report['files_deleted'] += 1
elif '.conflict' in line:
report['conflicts'].append(line.strip())
elif 'ERROR' in line:
report['errors'].append(line.strip())
return report
def save_sync_report(report: dict):
"""Save sync report to history."""
history_dir = Path.home() / '.basic-memory' / 'sync-history'
history_dir.mkdir(parents=True, exist_ok=True)
timestamp = datetime.now().strftime('%Y%m%d-%H%M%S')
report_file = history_dir / f'{timestamp}.json'
report_file.write_text(json.dumps(report, indent=2))
# Usage in run_bisync()
start_time = time.time()
result = subprocess.run(bisync_cmd, capture_output=True, text=True)
duration = time.time() - start_time
report = parse_sync_report(result.stdout, duration, result.returncode)
save_sync_report(report)
if report['conflicts']:
console.print(f"[yellow]⚠ {len(report['conflicts'])} conflict(s) detected[/yellow]")
console.print("[dim]Run 'bm conflicts list' to view[/dim]")
```
### 4. User Commands
```bash
# View sync history
bm sync history
# → Lists recent syncs from ~/.basic-memory/sync-history/*.json
# → Shows: timestamp, duration, files changed, conflicts, errors
# View current conflicts
bm conflicts list
# → Scans sync directory for *.conflict* files
# → Shows: file path, conflict versions, timestamps
# Restore from S3 version
bm cloud restore work/notes.md --version-id abc123
# → Uses aws s3api get-object with version-id
# → Downloads to original path
bm cloud restore work/notes.md --timestamp "2025-10-03 14:30"
# → Lists versions, finds closest to timestamp
# → Downloads that version
# List file versions
bm cloud versions work/notes.md
# → Uses aws s3api list-object-versions
# → Shows: version-id, timestamp, size, author
# Interactive conflict resolution
bm conflicts resolve work/notes.md
# → Shows both versions side-by-side
# → Prompts: Keep local, keep cloud, merge manually, restore from S3 version
# → Cleans up .conflict files after resolution
```
## Success Metrics & Monitoring
**Phase 1 (v1) - Basic Safety:**
- [ ] Conflict detection rate < 5% of syncs (measure in telemetry)
- [ ] User can resolve conflicts within 5 minutes (UX testing)
- [ ] Documentation prevents 90% of multi-device issues
**Phase 2 (v2) - Observability:**
- [ ] 80% of users check `bm sync history` when troubleshooting
- [ ] Average time to restore from S3 version < 2 minutes
-
- [ ] Conflict resolution success rate > 95%
**What to measure:**
```python
# Telemetry in sync reports
{
"conflict_rate": conflicts / total_syncs,
"multi_device_collisions": count_state_divergence_errors,
"version_restores": count_restore_operations,
"avg_sync_duration": sum(durations) / count,
"max_delete_trips": count_max_delete_aborts
}
```
**When to add distributed lease:**
- Multi-device collision rate > 5% of syncs
- User complaints about state divergence errors
- Evidence that local lock isn't sufficient
**When to revisit Git (SPEC-14):**
- Teams feature launches (multi-user collaboration)
- Users request commit messages / audit trail
- PR-based review workflow becomes valuable
## Links
- SPEC-9: `specs/spec-9-multi-project-bisync`
- SPEC-14: `specs/spec-14-cloud-git-versioning` (deferred in favor of S3 versioning)
- rclone bisync docs: https://rclone.org/bisync/
- Tigris S3 versioning: https://www.tigrisdata.com/docs/buckets/versioning/
---
**Owner:** <assign> | **Review cadence:** weekly in standup | **Last updated:** 2025-10-03
+1 -1
View File
@@ -1,7 +1,7 @@
"""basic-memory - Local-first knowledge management combining Zettelkasten with knowledge graphs"""
# Package version - updated by release automation
__version__ = "0.17.5"
__version__ = "0.18.3"
# API version for FastAPI - independent of package version
__api_version__ = "v0"
@@ -0,0 +1,154 @@
"""Add structured metadata indexes for entity frontmatter
Revision ID: d7e8f9a0b1c2
Revises: g9a0b3c4d5e6
Create Date: 2026-01-31 12:00:00.000000
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy import text
def column_exists(connection, table: str, column: str) -> bool:
"""Check if a column exists in a table (idempotent migration support)."""
if connection.dialect.name == "postgresql":
result = connection.execute(
text(
"SELECT 1 FROM information_schema.columns "
"WHERE table_name = :table AND column_name = :column"
),
{"table": table, "column": column},
)
return result.fetchone() is not None
# SQLite
result = connection.execute(text(f"PRAGMA table_info({table})"))
columns = [row[1] for row in result]
return column in columns
def index_exists(connection, index_name: str) -> bool:
"""Check if an index exists (idempotent migration support)."""
if connection.dialect.name == "postgresql":
result = connection.execute(
text("SELECT 1 FROM pg_indexes WHERE indexname = :index_name"),
{"index_name": index_name},
)
return result.fetchone() is not None
# SQLite
result = connection.execute(
text("SELECT 1 FROM sqlite_master WHERE type='index' AND name = :index_name"),
{"index_name": index_name},
)
return result.fetchone() is not None
# revision identifiers, used by Alembic.
revision: str = "d7e8f9a0b1c2"
down_revision: Union[str, None] = "6830751f5fb6"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Add JSONB/GiN indexes for Postgres and generated columns for SQLite."""
connection = op.get_bind()
dialect = connection.dialect.name
if dialect == "postgresql":
# Ensure JSONB for efficient indexing
result = connection.execute(
text(
"SELECT data_type FROM information_schema.columns "
"WHERE table_name = 'entity' AND column_name = 'entity_metadata'"
)
).fetchone()
if result and result[0] != "jsonb":
op.execute(
"ALTER TABLE entity ALTER COLUMN entity_metadata "
"TYPE jsonb USING entity_metadata::jsonb"
)
# General JSONB GIN index
op.execute(
"CREATE INDEX IF NOT EXISTS idx_entity_metadata_gin "
"ON entity USING GIN (entity_metadata jsonb_path_ops)"
)
# Common field indexes
op.execute(
"CREATE INDEX IF NOT EXISTS idx_entity_tags_json "
"ON entity USING GIN ((entity_metadata -> 'tags'))"
)
op.execute(
"CREATE INDEX IF NOT EXISTS idx_entity_frontmatter_type "
"ON entity ((entity_metadata ->> 'type'))"
)
op.execute(
"CREATE INDEX IF NOT EXISTS idx_entity_frontmatter_status "
"ON entity ((entity_metadata ->> 'status'))"
)
return
# SQLite: add generated columns for common frontmatter fields
# Constraint: SQLite ALTER TABLE ADD COLUMN only supports VIRTUAL generated columns,
# not STORED. json_extract is deterministic so VIRTUAL columns can still be indexed.
if not column_exists(connection, "entity", "tags_json"):
op.add_column(
"entity",
sa.Column(
"tags_json",
sa.Text(),
sa.Computed("json_extract(entity_metadata, '$.tags')", persisted=False),
),
)
if not column_exists(connection, "entity", "frontmatter_status"):
op.add_column(
"entity",
sa.Column(
"frontmatter_status",
sa.Text(),
sa.Computed("json_extract(entity_metadata, '$.status')", persisted=False),
),
)
if not column_exists(connection, "entity", "frontmatter_type"):
op.add_column(
"entity",
sa.Column(
"frontmatter_type",
sa.Text(),
sa.Computed("json_extract(entity_metadata, '$.type')", persisted=False),
),
)
# Index generated columns
if not index_exists(connection, "idx_entity_tags_json"):
op.create_index("idx_entity_tags_json", "entity", ["tags_json"])
if not index_exists(connection, "idx_entity_frontmatter_status"):
op.create_index("idx_entity_frontmatter_status", "entity", ["frontmatter_status"])
if not index_exists(connection, "idx_entity_frontmatter_type"):
op.create_index("idx_entity_frontmatter_type", "entity", ["frontmatter_type"])
def downgrade() -> None:
"""Best-effort downgrade (drop indexes, revert JSONB on Postgres)."""
connection = op.get_bind()
dialect = connection.dialect.name
if dialect == "postgresql":
op.execute("DROP INDEX IF EXISTS idx_entity_frontmatter_status")
op.execute("DROP INDEX IF EXISTS idx_entity_frontmatter_type")
op.execute("DROP INDEX IF EXISTS idx_entity_tags_json")
op.execute("DROP INDEX IF EXISTS idx_entity_metadata_gin")
op.execute(
"ALTER TABLE entity ALTER COLUMN entity_metadata TYPE json USING entity_metadata::json"
)
return
# SQLite: drop indexes (dropping generated columns requires table rebuild)
op.execute("DROP INDEX IF EXISTS idx_entity_frontmatter_status")
op.execute("DROP INDEX IF EXISTS idx_entity_frontmatter_type")
op.execute("DROP INDEX IF EXISTS idx_entity_tags_json")
@@ -0,0 +1,68 @@
"""Add Postgres semantic vector search tables (pgvector-aware, optional)
Revision ID: h1b2c3d4e5f6
Revises: d7e8f9a0b1c2
Create Date: 2026-02-07 00:00:00.000000
"""
from typing import Sequence, Union
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "h1b2c3d4e5f6"
down_revision: Union[str, None] = "d7e8f9a0b1c2"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Create Postgres vector chunk metadata table.
Trigger: database backend is PostgreSQL.
Why: search_vector_chunks stores text metadata with no vector-dimension
dependency, so it's safe in a migration. search_vector_embeddings (which
requires pgvector and a provider-specific dimension) is created at runtime
by PostgresSearchRepository._ensure_vector_tables(), mirroring the SQLite
pattern where vector tables are created dynamically.
Outcome: creates the dimension-independent chunks table. The embeddings
table + HNSW index are deferred to runtime.
"""
connection = op.get_bind()
if connection.dialect.name != "postgresql":
return
op.execute(
"""
CREATE TABLE IF NOT EXISTS search_vector_chunks (
id BIGSERIAL PRIMARY KEY,
entity_id INTEGER NOT NULL,
project_id INTEGER NOT NULL,
chunk_key TEXT NOT NULL,
chunk_text TEXT NOT NULL,
source_hash TEXT NOT NULL,
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
UNIQUE (project_id, entity_id, chunk_key)
)
"""
)
op.execute(
"""
CREATE INDEX IF NOT EXISTS idx_search_vector_chunks_project_entity
ON search_vector_chunks (project_id, entity_id)
"""
)
def downgrade() -> None:
"""Remove Postgres vector chunk/embedding tables.
Does not drop pgvector extension because other schema objects may depend on it.
"""
connection = op.get_bind()
if connection.dialect.name != "postgresql":
return
op.execute("DROP TABLE IF EXISTS search_vector_embeddings")
op.execute("DROP TABLE IF EXISTS search_vector_chunks")
+44 -24
View File
@@ -2,23 +2,13 @@
from contextlib import asynccontextmanager
from fastapi import FastAPI, HTTPException
from fastapi import FastAPI, HTTPException, Request
from fastapi.exception_handlers import http_exception_handler
from fastapi.routing import APIRouter
from loguru import logger
from basic_memory import __version__ as version
from basic_memory.api.container import ApiContainer, set_container
from basic_memory.api.routers import (
directory_router,
importer_router,
knowledge,
management,
memory,
project,
resource,
search,
prompt_router,
)
from basic_memory.api.v2.routers import (
knowledge_router as v2_knowledge,
project_router as v2_project,
@@ -28,8 +18,15 @@ from basic_memory.api.v2.routers import (
directory_router as v2_directory,
prompt_router as v2_prompt,
importer_router as v2_importer,
schema_router as v2_schema,
)
from basic_memory.api.v2.routers.project_router import (
add_project,
list_projects,
synchronize_projects,
)
from basic_memory.config import init_api_logging
from basic_memory.services.exceptions import EntityAlreadyExistsError
from basic_memory.services.initialization import initialize_app
@@ -88,21 +85,44 @@ app.include_router(v2_resource, prefix="/v2/projects/{project_id}")
app.include_router(v2_directory, prefix="/v2/projects/{project_id}")
app.include_router(v2_prompt, prefix="/v2/projects/{project_id}")
app.include_router(v2_importer, prefix="/v2/projects/{project_id}")
app.include_router(v2_schema, prefix="/v2/projects/{project_id}")
app.include_router(v2_project, prefix="/v2")
# Include v1 routers (/{project} is a catch-all, must come after specific prefixes)
app.include_router(knowledge.router, prefix="/{project}")
app.include_router(memory.router, prefix="/{project}")
app.include_router(resource.router, prefix="/{project}")
app.include_router(search.router, prefix="/{project}")
app.include_router(project.project_router, prefix="/{project}")
app.include_router(directory_router.router, prefix="/{project}")
app.include_router(prompt_router.router, prefix="/{project}")
app.include_router(importer_router.router, prefix="/{project}")
# Legacy web app proxy paths (compat with /proxy/projects/projects)
app.include_router(v2_project, prefix="/proxy/projects")
# Project resource router works across projects
app.include_router(project.project_resource_router)
app.include_router(management.router)
# Legacy v1 compat: older CLI versions (v0.18.0 and earlier) call /projects/...
# Using router mount causes 307 redirect which proxy doesn't follow, so add explicit routes
legacy_router = APIRouter(tags=["legacy"])
legacy_router.add_api_route("/projects/projects", list_projects, methods=["GET"])
legacy_router.add_api_route("/projects/projects", add_project, methods=["POST"])
legacy_router.add_api_route("/projects/config/sync", synchronize_projects, methods=["POST"])
app.include_router(legacy_router)
# V2 routers are the only public API surface
@app.exception_handler(EntityAlreadyExistsError)
async def entity_already_exists_error_handler(request: Request, exc: EntityAlreadyExistsError):
"""Handle entity creation conflicts (e.g., file already exists).
This is expected behavior when users try to create notes that exist,
so log at INFO level instead of ERROR.
"""
logger.info(
"Entity already exists",
url=str(request.url),
method=request.method,
path=request.url.path,
error=str(exc),
)
return await http_exception_handler(
request,
HTTPException(
status_code=409,
detail="Note already exists. Use edit_note to modify it, or delete it first.",
),
)
@app.exception_handler(Exception)
-11
View File
@@ -1,11 +0,0 @@
"""API routers."""
from . import knowledge_router as knowledge
from . import management_router as management
from . import memory_router as memory
from . import project_router as project
from . import resource_router as resource
from . import search_router as search
from . import prompt_router as prompt
__all__ = ["knowledge", "management", "memory", "project", "resource", "search", "prompt"]
@@ -1,84 +0,0 @@
"""Router for directory tree operations."""
from typing import List, Optional
from fastapi import APIRouter, Query
from basic_memory.deps import DirectoryServiceDep, ProjectIdDep
from basic_memory.schemas.directory import DirectoryNode
router = APIRouter(prefix="/directory", tags=["directory"])
@router.get("/tree", response_model=DirectoryNode, response_model_exclude_none=True)
async def get_directory_tree(
directory_service: DirectoryServiceDep,
project_id: ProjectIdDep,
):
"""Get hierarchical directory structure from the knowledge base.
Args:
directory_service: Service for directory operations
project_id: ID of the current project
Returns:
DirectoryNode representing the root of the hierarchical tree structure
"""
# Get a hierarchical directory tree for the specific project
tree = await directory_service.get_directory_tree()
# Return the hierarchical tree
return tree
@router.get("/structure", response_model=DirectoryNode, response_model_exclude_none=True)
async def get_directory_structure(
directory_service: DirectoryServiceDep,
project_id: ProjectIdDep,
):
"""Get folder structure for navigation (no files).
Optimized endpoint for folder tree navigation. Returns only directory nodes
without file metadata. For full tree with files, use /directory/tree.
Args:
directory_service: Service for directory operations
project_id: ID of the current project
Returns:
DirectoryNode tree containing only folders (type="directory")
"""
structure = await directory_service.get_directory_structure()
return structure
@router.get("/list", response_model=List[DirectoryNode], response_model_exclude_none=True)
async def list_directory(
directory_service: DirectoryServiceDep,
project_id: ProjectIdDep,
dir_name: str = Query("/", description="Directory path to list"),
depth: int = Query(1, ge=1, le=10, description="Recursion depth (1-10)"),
file_name_glob: Optional[str] = Query(
None, description="Glob pattern for filtering file names"
),
):
"""List directory contents with filtering and depth control.
Args:
directory_service: Service for directory operations
project_id: ID of the current project
dir_name: Directory path to list (default: root "/")
depth: Recursion depth (1-10, default: 1 for immediate children only)
file_name_glob: Optional glob pattern for filtering file names (e.g., "*.md", "*meeting*")
Returns:
List of DirectoryNode objects matching the criteria
"""
# Get directory listing with filtering
nodes = await directory_service.list_directory(
dir_name=dir_name,
depth=depth,
file_name_glob=file_name_glob,
)
return nodes
@@ -1,152 +0,0 @@
"""Import router for Basic Memory API."""
import json
import logging
from fastapi import APIRouter, Form, HTTPException, UploadFile, status
from basic_memory.deps import (
ChatGPTImporterDep,
ClaudeConversationsImporterDep,
ClaudeProjectsImporterDep,
MemoryJsonImporterDep,
)
from basic_memory.importers import Importer
from basic_memory.schemas.importer import (
ChatImportResult,
EntityImportResult,
ProjectImportResult,
)
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/import", tags=["import"])
@router.post("/chatgpt", response_model=ChatImportResult)
async def import_chatgpt(
importer: ChatGPTImporterDep,
file: UploadFile,
folder: str = Form("conversations"),
) -> ChatImportResult:
"""Import conversations from ChatGPT JSON export.
Args:
file: The ChatGPT conversations.json file.
folder: The folder to place the files in.
markdown_processor: MarkdownProcessor instance.
Returns:
ChatImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
return await import_file(importer, file, folder)
@router.post("/claude/conversations", response_model=ChatImportResult)
async def import_claude_conversations(
importer: ClaudeConversationsImporterDep,
file: UploadFile,
folder: str = Form("conversations"),
) -> ChatImportResult:
"""Import conversations from Claude conversations.json export.
Args:
file: The Claude conversations.json file.
folder: The folder to place the files in.
markdown_processor: MarkdownProcessor instance.
Returns:
ChatImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
return await import_file(importer, file, folder)
@router.post("/claude/projects", response_model=ProjectImportResult)
async def import_claude_projects(
importer: ClaudeProjectsImporterDep,
file: UploadFile,
folder: str = Form("projects"),
) -> ProjectImportResult:
"""Import projects from Claude projects.json export.
Args:
file: The Claude projects.json file.
base_folder: The base folder to place the files in.
markdown_processor: MarkdownProcessor instance.
Returns:
ProjectImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
return await import_file(importer, file, folder)
@router.post("/memory-json", response_model=EntityImportResult)
async def import_memory_json(
importer: MemoryJsonImporterDep,
file: UploadFile,
folder: str = Form("conversations"),
) -> EntityImportResult:
"""Import entities and relations from a memory.json file.
Args:
file: The memory.json file.
destination_folder: Optional destination folder within the project.
markdown_processor: MarkdownProcessor instance.
Returns:
EntityImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
try:
file_data = []
file_bytes = await file.read()
file_str = file_bytes.decode("utf-8")
for line in file_str.splitlines():
json_data = json.loads(line)
file_data.append(json_data)
result = await importer.import_data(file_data, folder)
if not result.success: # pragma: no cover
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=result.error_message or "Import failed",
)
except Exception as e:
logger.exception("Import failed")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Import failed: {str(e)}",
)
return result
async def import_file(importer: Importer, file: UploadFile, destination_folder: str):
try:
# Process file
json_data = json.load(file.file)
result = await importer.import_data(json_data, destination_folder)
if not result.success: # pragma: no cover
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=result.error_message or "Import failed",
)
return result
except Exception as e:
logger.exception("Import failed")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Import failed: {str(e)}",
)
@@ -1,319 +0,0 @@
"""Router for knowledge graph operations.
DEPRECATED: This v1 API is deprecated and will be removed on June 30, 2026.
Please migrate to /v2/{project}/knowledge endpoints which use entity IDs instead
of path-based identifiers for improved performance and stability.
Migration guide: See docs/migration/v1-to-v2.md
"""
from typing import Annotated
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Query, Response
from loguru import logger
from basic_memory.deps import (
EntityServiceDep,
get_search_service,
SearchServiceDep,
LinkResolverDep,
ProjectPathDep,
FileServiceDep,
ProjectConfigDep,
AppConfigDep,
SyncServiceDep,
)
from basic_memory.schemas import (
EntityListResponse,
EntityResponse,
DeleteEntitiesResponse,
DeleteEntitiesRequest,
)
from basic_memory.schemas.request import EditEntityRequest, MoveEntityRequest
from basic_memory.schemas.base import Permalink, Entity
router = APIRouter(
prefix="/knowledge",
tags=["knowledge"],
deprecated=True, # Marks entire router as deprecated in OpenAPI docs
)
async def resolve_relations_background(sync_service, entity_id: int, entity_permalink: str) -> None:
"""Background task to resolve relations for a specific entity.
This runs asynchronously after the API response is sent, preventing
long delays when creating entities with many relations.
"""
try:
# Only resolve relations for the newly created entity
await sync_service.resolve_relations(entity_id=entity_id)
logger.debug(
f"Background: Resolved relations for entity {entity_permalink} (id={entity_id})"
)
except Exception as e: # pragma: no cover
# Log but don't fail - this is a background task.
# Avoid forcing synthetic failures just for coverage.
logger.warning( # pragma: no cover
f"Background: Failed to resolve relations for entity {entity_permalink}: {e}"
)
## Create endpoints
@router.post("/entities", response_model=EntityResponse)
async def create_entity(
data: Entity,
background_tasks: BackgroundTasks,
entity_service: EntityServiceDep,
search_service: SearchServiceDep,
) -> EntityResponse:
"""Create an entity."""
logger.info(
"API request", endpoint="create_entity", entity_type=data.entity_type, title=data.title
)
entity = await entity_service.create_entity(data)
# reindex
await search_service.index_entity(entity, background_tasks=background_tasks)
result = EntityResponse.model_validate(entity)
logger.info(
f"API response: endpoint='create_entity' title={result.title}, permalink={result.permalink}, status_code=201"
)
return result
@router.put("/entities/{permalink:path}", response_model=EntityResponse)
async def create_or_update_entity(
project: ProjectPathDep,
permalink: Permalink,
data: Entity,
response: Response,
background_tasks: BackgroundTasks,
entity_service: EntityServiceDep,
search_service: SearchServiceDep,
file_service: FileServiceDep,
sync_service: SyncServiceDep,
) -> EntityResponse:
"""Create or update an entity. If entity exists, it will be updated, otherwise created."""
logger.info(
f"API request: create_or_update_entity for {project=}, {permalink=}, {data.entity_type=}, {data.title=}"
)
# Validate permalink matches
if data.permalink != permalink:
logger.warning(
f"API validation error: creating/updating entity with permalink mismatch - url={permalink}, data={data.permalink}",
)
raise HTTPException(
status_code=400,
detail=f"Entity permalink {data.permalink} must match URL path: '{permalink}'",
)
# Try create_or_update operation
entity, created = await entity_service.create_or_update_entity(data)
response.status_code = 201 if created else 200
# reindex
await search_service.index_entity(entity, background_tasks=background_tasks)
# Schedule relation resolution as a background task for new entities
# This prevents blocking the API response while resolving potentially many relations
if created:
background_tasks.add_task(
resolve_relations_background, sync_service, entity.id, entity.permalink or ""
)
result = EntityResponse.model_validate(entity)
logger.info(
f"API response: {result.title=}, {result.permalink=}, {created=}, status_code={response.status_code}"
)
return result
@router.patch("/entities/{identifier:path}", response_model=EntityResponse)
async def edit_entity(
identifier: str,
data: EditEntityRequest,
background_tasks: BackgroundTasks,
entity_service: EntityServiceDep,
search_service: SearchServiceDep,
) -> EntityResponse:
"""Edit an existing entity using various operations like append, prepend, find_replace, or replace_section.
This endpoint allows for targeted edits without requiring the full entity content.
"""
logger.info(
f"API request: endpoint='edit_entity', identifier='{identifier}', operation='{data.operation}'"
)
try:
# Edit the entity using the service
entity = await entity_service.edit_entity(
identifier=identifier,
operation=data.operation,
content=data.content,
section=data.section,
find_text=data.find_text,
expected_replacements=data.expected_replacements,
)
# Reindex the updated entity
await search_service.index_entity(entity, background_tasks=background_tasks)
# Return the updated entity response
result = EntityResponse.model_validate(entity)
logger.info(
"API response",
endpoint="edit_entity",
identifier=identifier,
operation=data.operation,
permalink=result.permalink,
status_code=200,
)
return result
except Exception as e:
logger.error(f"Error editing entity: {e}")
raise HTTPException(status_code=400, detail=str(e))
@router.post("/move")
async def move_entity(
data: MoveEntityRequest,
background_tasks: BackgroundTasks,
entity_service: EntityServiceDep,
project_config: ProjectConfigDep,
app_config: AppConfigDep,
search_service: SearchServiceDep,
) -> EntityResponse:
"""Move an entity to a new file location with project consistency.
This endpoint moves a note to a different path while maintaining project
consistency and optionally updating permalinks based on configuration.
"""
logger.info(
f"API request: endpoint='move_entity', identifier='{data.identifier}', destination='{data.destination_path}'"
)
try:
# Move the entity using the service
moved_entity = await entity_service.move_entity(
identifier=data.identifier,
destination_path=data.destination_path,
project_config=project_config,
app_config=app_config,
)
# Get the moved entity to reindex it
entity = await entity_service.link_resolver.resolve_link(data.destination_path)
if entity:
await search_service.index_entity(entity, background_tasks=background_tasks)
logger.info(
"API response",
endpoint="move_entity",
identifier=data.identifier,
destination=data.destination_path,
status_code=200,
)
result = EntityResponse.model_validate(moved_entity)
return result
except Exception as e:
logger.error(f"Error moving entity: {e}")
raise HTTPException(status_code=400, detail=str(e))
## Read endpoints
@router.get("/entities/{identifier:path}", response_model=EntityResponse)
async def get_entity(
entity_service: EntityServiceDep,
link_resolver: LinkResolverDep,
identifier: str,
) -> EntityResponse:
"""Get a specific entity by file path or permalink..
Args:
identifier: Entity file path or permalink
:param entity_service: EntityService
:param link_resolver: LinkResolver
"""
logger.info(f"request: get_entity with identifier={identifier}")
entity = await link_resolver.resolve_link(identifier)
if not entity:
raise HTTPException(status_code=404, detail=f"Entity {identifier} not found")
result = EntityResponse.model_validate(entity)
return result
@router.get("/entities", response_model=EntityListResponse)
async def get_entities(
entity_service: EntityServiceDep,
permalink: Annotated[list[str] | None, Query()] = None,
) -> EntityListResponse:
"""Open specific entities"""
logger.info(f"request: get_entities with permalinks={permalink}")
entities = await entity_service.get_entities_by_permalinks(permalink) if permalink else []
result = EntityListResponse(
entities=[EntityResponse.model_validate(entity) for entity in entities]
)
return result
## Delete endpoints
@router.delete("/entities/{identifier:path}", response_model=DeleteEntitiesResponse)
async def delete_entity(
identifier: str,
background_tasks: BackgroundTasks,
entity_service: EntityServiceDep,
link_resolver: LinkResolverDep,
search_service=Depends(get_search_service),
) -> DeleteEntitiesResponse:
"""Delete a single entity and remove from search index."""
logger.info(f"request: delete_entity with identifier={identifier}")
entity = await link_resolver.resolve_link(identifier)
if entity is None:
return DeleteEntitiesResponse(deleted=False)
# Delete the entity
deleted = await entity_service.delete_entity(entity.permalink or entity.id)
# Remove from search index (entity, observations, and relations)
background_tasks.add_task(search_service.handle_delete, entity)
result = DeleteEntitiesResponse(deleted=deleted)
return result
@router.post("/entities/delete", response_model=DeleteEntitiesResponse)
async def delete_entities(
data: DeleteEntitiesRequest,
background_tasks: BackgroundTasks,
entity_service: EntityServiceDep,
search_service=Depends(get_search_service),
) -> DeleteEntitiesResponse:
"""Delete entities and remove from search index."""
logger.info(f"request: delete_entities with data={data}")
deleted = False
# Remove each deleted entity from search index
for permalink in data.permalinks:
deleted = await entity_service.delete_entity(permalink)
background_tasks.add_task(search_service.delete_by_permalink, permalink)
result = DeleteEntitiesResponse(deleted=deleted)
return result
@@ -1,80 +0,0 @@
"""Management router for basic-memory API."""
import asyncio
from fastapi import APIRouter, Request
from loguru import logger
from pydantic import BaseModel
from basic_memory.config import ConfigManager
from basic_memory.deps import SyncServiceDep, ProjectRepositoryDep
router = APIRouter(prefix="/management", tags=["management"])
class WatchStatusResponse(BaseModel):
"""Response model for watch status."""
running: bool
"""Whether the watch service is currently running."""
@router.get("/watch/status", response_model=WatchStatusResponse)
async def get_watch_status(request: Request) -> WatchStatusResponse:
"""Get the current status of the watch service."""
return WatchStatusResponse(
running=request.app.state.watch_task is not None and not request.app.state.watch_task.done()
)
@router.post("/watch/start", response_model=WatchStatusResponse)
async def start_watch_service(
request: Request, project_repository: ProjectRepositoryDep, sync_service: SyncServiceDep
) -> WatchStatusResponse:
"""Start the watch service if it's not already running."""
# needed because of circular imports from sync -> app
from basic_memory.sync import WatchService
from basic_memory.sync.background_sync import create_background_sync_task
if request.app.state.watch_task is not None and not request.app.state.watch_task.done():
# Watch service is already running
return WatchStatusResponse(running=True)
app_config = ConfigManager().config
# Create and start a new watch service
logger.info("Starting watch service via management API")
# Get services needed for the watch task
watch_service = WatchService(
app_config=app_config,
project_repository=project_repository,
)
# Create and store the task
watch_task = create_background_sync_task(sync_service, watch_service)
request.app.state.watch_task = watch_task
return WatchStatusResponse(running=True)
@router.post("/watch/stop", response_model=WatchStatusResponse)
async def stop_watch_service(request: Request) -> WatchStatusResponse: # pragma: no cover
"""Stop the watch service if it's running."""
if request.app.state.watch_task is None or request.app.state.watch_task.done():
# Watch service is not running
return WatchStatusResponse(running=False)
# Cancel the running task
logger.info("Stopping watch service via management API")
request.app.state.watch_task.cancel()
# Wait for it to be properly cancelled
try:
await request.app.state.watch_task
except asyncio.CancelledError:
pass
request.app.state.watch_task = None
return WatchStatusResponse(running=False)
@@ -1,90 +0,0 @@
"""Routes for memory:// URI operations."""
from typing import Annotated, Optional
from fastapi import APIRouter, Query
from loguru import logger
from basic_memory.deps import ContextServiceDep, EntityRepositoryDep
from basic_memory.schemas.base import TimeFrame, parse_timeframe
from basic_memory.schemas.memory import (
GraphContext,
normalize_memory_url,
)
from basic_memory.schemas.search import SearchItemType
from basic_memory.api.routers.utils import to_graph_context
router = APIRouter(prefix="/memory", tags=["memory"])
@router.get("/recent", response_model=GraphContext)
async def recent(
context_service: ContextServiceDep,
entity_repository: EntityRepositoryDep,
type: Annotated[list[SearchItemType] | None, Query()] = None,
depth: int = 1,
timeframe: TimeFrame = "7d",
page: int = 1,
page_size: int = 10,
max_related: int = 10,
) -> GraphContext:
# return all types by default
types = (
[SearchItemType.ENTITY, SearchItemType.RELATION, SearchItemType.OBSERVATION]
if not type
else type
)
logger.debug(
f"Getting recent context: `{types}` depth: `{depth}` timeframe: `{timeframe}` page: `{page}` page_size: `{page_size}` max_related: `{max_related}`"
)
# Parse timeframe
since = parse_timeframe(timeframe)
limit = page_size
offset = (page - 1) * page_size
# Build context
context = await context_service.build_context(
types=types, depth=depth, since=since, limit=limit, offset=offset, max_related=max_related
)
recent_context = await to_graph_context(
context, entity_repository=entity_repository, page=page, page_size=page_size
)
logger.debug(f"Recent context: {recent_context.model_dump_json()}")
return recent_context
# get_memory_context needs to be declared last so other paths can match
@router.get("/{uri:path}", response_model=GraphContext)
async def get_memory_context(
context_service: ContextServiceDep,
entity_repository: EntityRepositoryDep,
uri: str,
depth: int = 1,
timeframe: Optional[TimeFrame] = None,
page: int = 1,
page_size: int = 10,
max_related: int = 10,
) -> GraphContext:
"""Get rich context from memory:// URI."""
# add the project name from the config to the url as the "host
# Parse URI
logger.debug(
f"Getting context for URI: `{uri}` depth: `{depth}` timeframe: `{timeframe}` page: `{page}` page_size: `{page_size}` max_related: `{max_related}`"
)
memory_url = normalize_memory_url(uri)
# Parse timeframe
since = parse_timeframe(timeframe) if timeframe else None
limit = page_size
offset = (page - 1) * page_size
# Build context
context = await context_service.build_context(
memory_url, depth=depth, since=since, limit=limit, offset=offset, max_related=max_related
)
return await to_graph_context(
context, entity_repository=entity_repository, page=page, page_size=page_size
)
@@ -1,460 +0,0 @@
"""Router for project management."""
import os
from fastapi import APIRouter, HTTPException, Path, Body, BackgroundTasks, Response, Query
from typing import Optional
from loguru import logger
from basic_memory.deps import (
ProjectConfigDep,
ProjectServiceDep,
ProjectPathDep,
SyncServiceDep,
)
from basic_memory.schemas import ProjectInfoResponse, SyncReportResponse
from basic_memory.schemas.project_info import (
ProjectList,
ProjectItem,
ProjectInfoRequest,
ProjectStatusResponse,
)
from basic_memory.utils import normalize_project_path
# Router for resources in a specific project
# The ProjectPathDep is used in the path as a prefix, so the request path is like /{project}/project/info
project_router = APIRouter(prefix="/project", tags=["project"])
# Router for managing project resources
project_resource_router = APIRouter(prefix="/projects", tags=["project_management"])
@project_router.get("/info", response_model=ProjectInfoResponse)
async def get_project_info(
project_service: ProjectServiceDep,
project: ProjectPathDep,
) -> ProjectInfoResponse:
"""Get comprehensive information about the specified Basic Memory project."""
return await project_service.get_project_info(project)
@project_router.get("/item", response_model=ProjectItem)
async def get_project(
project_service: ProjectServiceDep,
project: ProjectPathDep,
) -> ProjectItem:
"""Get bassic info about the specified Basic Memory project."""
found_project = await project_service.get_project(project)
if not found_project:
raise HTTPException(
status_code=404, detail=f"Project: '{project}' does not exist"
) # pragma: no cover
return ProjectItem(
id=found_project.id,
external_id=found_project.external_id,
name=found_project.name,
path=normalize_project_path(found_project.path),
is_default=found_project.is_default or False,
)
# Update a project
@project_router.patch("/{name}", response_model=ProjectStatusResponse)
async def update_project(
project_service: ProjectServiceDep,
name: str = Path(..., description="Name of the project to update"),
path: Optional[str] = Body(None, description="New absolute path for the project"),
is_active: Optional[bool] = Body(None, description="Status of the project (active/inactive)"),
) -> ProjectStatusResponse:
"""Update a project's information in configuration and database.
Args:
name: The name of the project to update
path: Optional new absolute path for the project
is_active: Optional status update for the project
Returns:
Response confirming the project was updated
"""
try:
# Validate that path is absolute if provided
if path and not os.path.isabs(path):
raise HTTPException(status_code=400, detail="Path must be absolute")
# Get original project info for the response
old_project = await project_service.get_project(name)
if not old_project:
raise HTTPException(
status_code=400, detail=f"Project '{name}' not found in configuration"
)
old_project_info = ProjectItem(
id=old_project.id,
external_id=old_project.external_id,
name=old_project.name,
path=old_project.path,
is_default=old_project.is_default or False,
)
if path:
await project_service.move_project(name, path)
elif is_active is not None:
await project_service.update_project(name, is_active=is_active)
# Get updated project info
updated_project = await project_service.get_project(name)
if not updated_project:
raise HTTPException( # pragma: no cover
status_code=404, detail=f"Project '{name}' not found after update"
)
return ProjectStatusResponse(
message=f"Project '{name}' updated successfully",
status="success",
default=(name == project_service.default_project),
old_project=old_project_info,
new_project=ProjectItem(
id=updated_project.id,
external_id=updated_project.external_id,
name=updated_project.name,
path=updated_project.path,
is_default=updated_project.is_default or False,
),
)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e)) # pragma: no cover
# Sync project filesystem
@project_router.post("/sync")
async def sync_project(
background_tasks: BackgroundTasks,
sync_service: SyncServiceDep,
project_config: ProjectConfigDep,
force_full: bool = Query(
False, description="Force full scan, bypassing watermark optimization"
),
run_in_background: bool = Query(True, description="Run in background"),
):
"""Force project filesystem sync to database.
Scans the project directory and updates the database with any new or modified files.
Args:
background_tasks: FastAPI background tasks
sync_service: Sync service for this project
project_config: Project configuration
force_full: If True, force a full scan even if watermark exists
run_in_background: If True, run sync in background and return immediately
Returns:
Response confirming sync was initiated (background) or SyncReportResponse (foreground)
"""
if run_in_background:
background_tasks.add_task(
sync_service.sync, project_config.home, project_config.name, force_full=force_full
)
logger.info(
f"Filesystem sync initiated for project: {project_config.name} (force_full={force_full})"
)
return {
"status": "sync_started",
"message": f"Filesystem sync initiated for project '{project_config.name}'",
}
else:
report = await sync_service.sync(
project_config.home, project_config.name, force_full=force_full
)
logger.info(
f"Filesystem sync completed for project: {project_config.name} (force_full={force_full})"
)
return SyncReportResponse.from_sync_report(report)
@project_router.post("/status", response_model=SyncReportResponse)
async def project_sync_status(
sync_service: SyncServiceDep,
project_config: ProjectConfigDep,
) -> SyncReportResponse:
"""Scan directory for changes compared to database state.
Args:
sync_service: Sync service for this project
project_config: Project configuration
Returns:
Scan report with details on files that need syncing
"""
logger.info(f"Scanning filesystem for project: {project_config.name}") # pragma: no cover
sync_report = await sync_service.scan(project_config.home) # pragma: no cover
return SyncReportResponse.from_sync_report(sync_report) # pragma: no cover
# List all available projects
@project_resource_router.get("/projects", response_model=ProjectList)
async def list_projects(
project_service: ProjectServiceDep,
) -> ProjectList:
"""List all configured projects.
Returns:
A list of all projects with metadata
"""
projects = await project_service.list_projects()
default_project = project_service.default_project
project_items = [
ProjectItem(
id=project.id,
external_id=project.external_id,
name=project.name,
path=normalize_project_path(project.path),
is_default=project.is_default or False,
)
for project in projects
]
return ProjectList(
projects=project_items,
default_project=default_project,
)
# Add a new project
@project_resource_router.post("/projects", response_model=ProjectStatusResponse, status_code=201)
async def add_project(
response: Response,
project_data: ProjectInfoRequest,
project_service: ProjectServiceDep,
) -> ProjectStatusResponse:
"""Add a new project to configuration and database.
Args:
project_data: The project name and path, with option to set as default
Returns:
Response confirming the project was added
"""
# Check if project already exists before attempting to add
existing_project = await project_service.get_project(project_data.name)
if existing_project:
# Project exists - check if paths match for true idempotency
# Normalize paths for comparison (resolve symlinks, etc.)
from pathlib import Path
requested_path = Path(project_data.path).resolve()
existing_path = Path(existing_project.path).resolve()
if requested_path == existing_path:
# Same name, same path - return 200 OK (idempotent)
response.status_code = 200
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message=f"Project '{project_data.name}' already exists",
status="success",
default=existing_project.is_default or False,
new_project=ProjectItem(
id=existing_project.id,
external_id=existing_project.external_id,
name=existing_project.name,
path=existing_project.path,
is_default=existing_project.is_default or False,
),
)
else:
# Same name, different path - this is an error
raise HTTPException(
status_code=400,
detail=f"Project '{project_data.name}' already exists with different path. Existing: {existing_project.path}, Requested: {project_data.path}",
)
try: # pragma: no cover
# The service layer now handles cloud mode validation and path sanitization
await project_service.add_project(
project_data.name, project_data.path, set_default=project_data.set_default
)
# Fetch the newly created project to get its ID
new_project = await project_service.get_project(project_data.name)
if not new_project:
raise HTTPException(status_code=500, detail="Failed to retrieve newly created project")
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message=f"Project '{new_project.name}' added successfully",
status="success",
default=project_data.set_default,
new_project=ProjectItem(
id=new_project.id,
external_id=new_project.external_id,
name=new_project.name,
path=new_project.path,
is_default=new_project.is_default or False,
),
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e))
# Remove a project
@project_resource_router.delete("/{name}", response_model=ProjectStatusResponse)
async def remove_project(
project_service: ProjectServiceDep,
name: str = Path(..., description="Name of the project to remove"),
delete_notes: bool = Query(
False, description="If True, delete project directory from filesystem"
),
) -> ProjectStatusResponse:
"""Remove a project from configuration and database.
Args:
name: The name of the project to remove
delete_notes: If True, delete the project directory from the filesystem
Returns:
Response confirming the project was removed
"""
try:
old_project = await project_service.get_project(name)
if not old_project: # pragma: no cover
raise HTTPException(
status_code=404, detail=f"Project: '{name}' does not exist"
) # pragma: no cover
# Check if trying to delete the default project
if name == project_service.default_project:
available_projects = await project_service.list_projects()
other_projects = [p.name for p in available_projects if p.name != name]
detail = f"Cannot delete default project '{name}'. "
if other_projects:
detail += (
f"Set another project as default first. Available: {', '.join(other_projects)}"
)
else:
detail += "This is the only project in your configuration."
raise HTTPException(status_code=400, detail=detail)
await project_service.remove_project(name, delete_notes=delete_notes)
return ProjectStatusResponse(
message=f"Project '{old_project.name}' removed successfully",
status="success",
default=False,
old_project=ProjectItem(
id=old_project.id,
external_id=old_project.external_id,
name=old_project.name,
path=old_project.path,
is_default=old_project.is_default or False,
),
new_project=None,
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e))
# Set a project as default
@project_resource_router.put("/{name}/default", response_model=ProjectStatusResponse)
async def set_default_project(
project_service: ProjectServiceDep,
name: str = Path(..., description="Name of the project to set as default"),
) -> ProjectStatusResponse:
"""Set a project as the default project.
Args:
name: The name of the project to set as default
Returns:
Response confirming the project was set as default
"""
try:
# Get the old default project
default_name = project_service.default_project
default_project = await project_service.get_project(default_name)
if not default_project: # pragma: no cover
raise HTTPException( # pragma: no cover
status_code=404, detail=f"Default Project: '{default_name}' does not exist"
)
# get the new project
new_default_project = await project_service.get_project(name)
if not new_default_project: # pragma: no cover
raise HTTPException(
status_code=404, detail=f"Project: '{name}' does not exist"
) # pragma: no cover
await project_service.set_default_project(name)
return ProjectStatusResponse(
message=f"Project '{name}' set as default successfully",
status="success",
default=True,
old_project=ProjectItem(
id=default_project.id,
external_id=default_project.external_id,
name=default_name,
path=default_project.path,
is_default=False,
),
new_project=ProjectItem(
id=new_default_project.id,
external_id=new_default_project.external_id,
name=name,
path=new_default_project.path,
is_default=True,
),
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e))
# Get the default project
@project_resource_router.get("/default", response_model=ProjectItem)
async def get_default_project(
project_service: ProjectServiceDep,
) -> ProjectItem:
"""Get the default project.
Returns:
Response with project default information
"""
# Get the old default project
default_name = project_service.default_project
default_project = await project_service.get_project(default_name)
if not default_project: # pragma: no cover
raise HTTPException( # pragma: no cover
status_code=404, detail=f"Default Project: '{default_name}' does not exist"
)
return ProjectItem(
id=default_project.id,
external_id=default_project.external_id,
name=default_project.name,
path=default_project.path,
is_default=True,
)
# Synchronize projects between config and database
@project_resource_router.post("/config/sync", response_model=ProjectStatusResponse)
async def synchronize_projects(
project_service: ProjectServiceDep,
) -> ProjectStatusResponse:
"""Synchronize projects between configuration file and database.
Ensures that all projects in the configuration file exist in the database
and vice versa.
Returns:
Response confirming synchronization was completed
"""
try: # pragma: no cover
await project_service.synchronize_projects()
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message="Projects synchronized successfully between configuration and database",
status="success",
default=False,
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e))
@@ -1,260 +0,0 @@
"""Router for prompt-related operations.
This router is responsible for rendering various prompts using Handlebars templates.
It centralizes all prompt formatting logic that was previously in the MCP prompts.
"""
from datetime import datetime, timezone
from fastapi import APIRouter, HTTPException, status
from loguru import logger
from basic_memory.api.routers.utils import to_graph_context, to_search_results
from basic_memory.api.template_loader import template_loader
from basic_memory.schemas.base import parse_timeframe
from basic_memory.deps import (
ContextServiceDep,
EntityRepositoryDep,
SearchServiceDep,
EntityServiceDep,
)
from basic_memory.schemas.prompt import (
ContinueConversationRequest,
SearchPromptRequest,
PromptResponse,
PromptMetadata,
)
from basic_memory.schemas.search import SearchItemType, SearchQuery
router = APIRouter(prefix="/prompt", tags=["prompt"])
@router.post("/continue-conversation", response_model=PromptResponse)
async def continue_conversation(
search_service: SearchServiceDep,
entity_service: EntityServiceDep,
context_service: ContextServiceDep,
entity_repository: EntityRepositoryDep,
request: ContinueConversationRequest,
) -> PromptResponse:
"""Generate a prompt for continuing a conversation.
This endpoint takes a topic and/or timeframe and generates a prompt with
relevant context from the knowledge base.
Args:
request: The request parameters
Returns:
Formatted continuation prompt with context
"""
logger.info(
f"Generating continue conversation prompt, topic: {request.topic}, timeframe: {request.timeframe}"
)
since = parse_timeframe(request.timeframe) if request.timeframe else None
# Initialize search results
search_results = []
# Get data needed for template
if request.topic:
query = SearchQuery(text=request.topic, after_date=request.timeframe)
results = await search_service.search(query, limit=request.search_items_limit)
search_results = await to_search_results(entity_service, results)
# Build context from results
all_hierarchical_results = []
for result in search_results:
if hasattr(result, "permalink") and result.permalink:
# Get hierarchical context using the new dataclass-based approach
context_result = await context_service.build_context(
result.permalink,
depth=request.depth,
since=since,
max_related=request.related_items_limit,
include_observations=True, # Include observations for entities
)
# Process results into the schema format
graph_context = await to_graph_context(
context_result, entity_repository=entity_repository
)
# Add results to our collection (limit to top results for each permalink)
if graph_context.results:
all_hierarchical_results.extend(graph_context.results[:3])
# Limit to a reasonable number of total results
all_hierarchical_results = all_hierarchical_results[:10]
template_context = {
"topic": request.topic,
"timeframe": request.timeframe,
"hierarchical_results": all_hierarchical_results,
"has_results": len(all_hierarchical_results) > 0,
}
else:
# If no topic, get recent activity
context_result = await context_service.build_context(
types=[SearchItemType.ENTITY],
depth=request.depth,
since=since,
max_related=request.related_items_limit,
include_observations=True,
)
recent_context = await to_graph_context(context_result, entity_repository=entity_repository)
hierarchical_results = recent_context.results[:5] # Limit to top 5 recent items
template_context = {
"topic": f"Recent Activity from ({request.timeframe})",
"timeframe": request.timeframe,
"hierarchical_results": hierarchical_results,
"has_results": len(hierarchical_results) > 0,
}
try:
# Render template
rendered_prompt = await template_loader.render(
"prompts/continue_conversation.hbs", template_context
)
# Calculate metadata
# Count items of different types
observation_count = 0
relation_count = 0
entity_count = 0
# Get the hierarchical results from the template context
hierarchical_results_for_count = template_context.get("hierarchical_results", [])
# For topic-based search
if request.topic:
for item in hierarchical_results_for_count:
if hasattr(item, "observations"):
observation_count += len(item.observations) if item.observations else 0
if hasattr(item, "related_results"):
for related in item.related_results or []:
if hasattr(related, "type"):
if related.type == "relation":
relation_count += 1
elif related.type == "entity": # pragma: no cover
entity_count += 1 # pragma: no cover
# For recent activity
else:
for item in hierarchical_results_for_count:
if hasattr(item, "observations"):
observation_count += len(item.observations) if item.observations else 0
if hasattr(item, "related_results"):
for related in item.related_results or []:
if hasattr(related, "type"):
if related.type == "relation":
relation_count += 1
elif related.type == "entity": # pragma: no cover
entity_count += 1 # pragma: no cover
# Build metadata
metadata = {
"query": request.topic,
"timeframe": request.timeframe,
"search_count": len(search_results)
if request.topic
else 0, # Original search results count
"context_count": len(hierarchical_results_for_count),
"observation_count": observation_count,
"relation_count": relation_count,
"total_items": (
len(hierarchical_results_for_count)
+ observation_count
+ relation_count
+ entity_count
),
"search_limit": request.search_items_limit,
"context_depth": request.depth,
"related_limit": request.related_items_limit,
"generated_at": datetime.now(timezone.utc).isoformat(),
}
prompt_metadata = PromptMetadata(**metadata)
return PromptResponse(
prompt=rendered_prompt, context=template_context, metadata=prompt_metadata
)
except Exception as e:
logger.error(f"Error rendering continue conversation template: {e}")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Error rendering prompt template: {str(e)}",
)
@router.post("/search", response_model=PromptResponse)
async def search_prompt(
search_service: SearchServiceDep,
entity_service: EntityServiceDep,
request: SearchPromptRequest,
page: int = 1,
page_size: int = 10,
) -> PromptResponse:
"""Generate a prompt for search results.
This endpoint takes a search query and formats the results into a helpful
prompt with context and suggestions.
Args:
request: The search parameters
page: The page number for pagination
page_size: The number of results per page, defaults to 10
Returns:
Formatted search results prompt with context
"""
logger.info(f"Generating search prompt, query: {request.query}, timeframe: {request.timeframe}")
limit = page_size
offset = (page - 1) * page_size
query = SearchQuery(text=request.query, after_date=request.timeframe)
results = await search_service.search(query, limit=limit, offset=offset)
search_results = await to_search_results(entity_service, results)
template_context = {
"query": request.query,
"timeframe": request.timeframe,
"results": search_results,
"has_results": len(search_results) > 0,
"result_count": len(search_results),
}
try:
# Render template
rendered_prompt = await template_loader.render("prompts/search.hbs", template_context)
# Build metadata
metadata = {
"query": request.query,
"timeframe": request.timeframe,
"search_count": len(search_results),
"context_count": len(search_results),
"observation_count": 0, # Search results don't include observations
"relation_count": 0, # Search results don't include relations
"total_items": len(search_results),
"search_limit": limit,
"context_depth": 0, # No context depth for basic search
"related_limit": 0, # No related items for basic search
"generated_at": datetime.now(timezone.utc).isoformat(),
}
prompt_metadata = PromptMetadata(**metadata)
return PromptResponse(
prompt=rendered_prompt, context=template_context, metadata=prompt_metadata
)
except Exception as e:
logger.error(f"Error rendering search template: {e}")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Error rendering prompt template: {str(e)}",
)
@@ -1,249 +0,0 @@
"""Routes for getting entity content."""
import tempfile
from pathlib import Path
from typing import Annotated, Union
from fastapi import APIRouter, HTTPException, BackgroundTasks, Body, Response
from fastapi.responses import FileResponse, JSONResponse
from loguru import logger
from basic_memory.deps import (
ProjectConfigDep,
LinkResolverDep,
SearchServiceDep,
EntityServiceDep,
FileServiceDep,
EntityRepositoryDep,
)
from basic_memory.repository.search_repository import SearchIndexRow
from basic_memory.schemas.memory import normalize_memory_url
from basic_memory.schemas.search import SearchQuery, SearchItemType
from basic_memory.models.knowledge import Entity as EntityModel
from datetime import datetime
router = APIRouter(prefix="/resource", tags=["resources"])
def _mtime_to_datetime(entity: EntityModel) -> datetime:
"""Convert entity mtime (file modification time) to datetime.
Returns the file's actual modification time, falling back to updated_at
if mtime is not available.
"""
if entity.mtime: # pragma: no cover
return datetime.fromtimestamp(entity.mtime).astimezone() # pragma: no cover
return entity.updated_at
def get_entity_ids(item: SearchIndexRow) -> set[int]:
match item.type:
case SearchItemType.ENTITY:
return {item.id}
case SearchItemType.OBSERVATION:
return {item.entity_id} # pyright: ignore [reportReturnType]
case SearchItemType.RELATION:
from_entity = item.from_id
to_entity = item.to_id # pyright: ignore [reportReturnType]
return {from_entity, to_entity} if to_entity else {from_entity} # pyright: ignore [reportReturnType]
case _: # pragma: no cover
raise ValueError(f"Unexpected type: {item.type}")
@router.get("/{identifier:path}", response_model=None)
async def get_resource_content(
config: ProjectConfigDep,
link_resolver: LinkResolverDep,
search_service: SearchServiceDep,
entity_service: EntityServiceDep,
file_service: FileServiceDep,
background_tasks: BackgroundTasks,
identifier: str,
page: int = 1,
page_size: int = 10,
) -> Union[Response, FileResponse]:
"""Get resource content by identifier: name or permalink."""
logger.debug(f"Getting content for: {identifier}")
# Find single entity by permalink
entity = await link_resolver.resolve_link(identifier)
results = [entity] if entity else []
# pagination for multiple results
limit = page_size
offset = (page - 1) * page_size
# search using the identifier as a permalink
if not results:
# if the identifier contains a wildcard, use GLOB search
query = (
SearchQuery(permalink_match=identifier)
if "*" in identifier
else SearchQuery(permalink=identifier)
)
search_results = await search_service.search(query, limit, offset)
if not search_results:
raise HTTPException(status_code=404, detail=f"Resource not found: {identifier}")
# get the deduplicated entities related to the search results
entity_ids = {id for result in search_results for id in get_entity_ids(result)}
results = await entity_service.get_entities_by_id(list(entity_ids))
# return single response
if len(results) == 1:
entity = results[0]
# Check file exists via file_service (for cloud compatibility)
if not await file_service.exists(entity.file_path):
raise HTTPException(
status_code=404,
detail=f"File not found: {entity.file_path}",
)
# Read content via file_service as bytes (works with both local and S3)
content = await file_service.read_file_bytes(entity.file_path)
content_type = file_service.content_type(entity.file_path)
return Response(content=content, media_type=content_type)
# for multiple files, initialize a temporary file for writing the results
with tempfile.NamedTemporaryFile(delete=False, mode="w", suffix=".md") as tmp_file:
temp_file_path = tmp_file.name
for result in results:
# Read content for each entity
content = await file_service.read_entity_content(result)
memory_url = normalize_memory_url(result.permalink)
modified_date = _mtime_to_datetime(result).isoformat()
checksum = result.checksum[:8] if result.checksum else ""
# Prepare the delimited content
response_content = f"--- {memory_url} {modified_date} {checksum}\n"
response_content += f"\n{content}\n"
response_content += "\n"
# Write content directly to the temporary file in append mode
tmp_file.write(response_content)
# Ensure all content is written to disk
tmp_file.flush()
# Schedule the temporary file to be deleted after the response
background_tasks.add_task(cleanup_temp_file, temp_file_path)
# Return the file response
return FileResponse(path=temp_file_path)
def cleanup_temp_file(file_path: str):
"""Delete the temporary file."""
try:
Path(file_path).unlink() # Deletes the file
logger.debug(f"Temporary file deleted: {file_path}")
except Exception as e: # pragma: no cover
logger.error(f"Error deleting temporary file {file_path}: {e}")
@router.put("/{file_path:path}")
async def write_resource(
config: ProjectConfigDep,
file_service: FileServiceDep,
entity_repository: EntityRepositoryDep,
search_service: SearchServiceDep,
file_path: str,
content: Annotated[str, Body()],
) -> JSONResponse:
"""Write content to a file in the project.
This endpoint allows writing content directly to a file in the project.
Also creates an entity record and indexes the file for search.
Args:
file_path: Path to write to, relative to project root
request: Contains the content to write
Returns:
JSON response with file information
"""
try:
# Get content from request body
# Defensive type checking: ensure content is a string
# FastAPI should validate this, but if a dict somehow gets through
# (e.g., via JSON body parsing), we need to catch it here
if isinstance(content, dict):
logger.error( # pragma: no cover
f"Error writing resource {file_path}: "
f"content is a dict, expected string. Keys: {list(content.keys())}"
)
raise HTTPException( # pragma: no cover
status_code=400,
detail="content must be a string, not a dict. "
"Ensure request body is sent as raw string content, not JSON object.",
)
# Ensure it's UTF-8 string content
if isinstance(content, bytes): # pragma: no cover
content_str = content.decode("utf-8")
else:
content_str = str(content)
# Cloud compatibility: do not assume a local filesystem path structure.
# Delegate directory creation + writes to the configured FileService (local or S3).
await file_service.ensure_directory(Path(file_path).parent)
checksum = await file_service.write_file(file_path, content_str)
# Get file info
file_metadata = await file_service.get_file_metadata(file_path)
# Determine file details
file_name = Path(file_path).name
content_type = file_service.content_type(file_path)
entity_type = "canvas" if file_path.endswith(".canvas") else "file"
# Check if entity already exists
existing_entity = await entity_repository.get_by_file_path(file_path)
if existing_entity:
# Update existing entity
entity = await entity_repository.update(
existing_entity.id,
{
"title": file_name,
"entity_type": entity_type,
"content_type": content_type,
"file_path": file_path,
"checksum": checksum,
"updated_at": file_metadata.modified_at,
},
)
status_code = 200
else:
# Create a new entity model
entity = EntityModel(
title=file_name,
entity_type=entity_type,
content_type=content_type,
file_path=file_path,
checksum=checksum,
created_at=file_metadata.created_at,
updated_at=file_metadata.modified_at,
)
entity = await entity_repository.add(entity)
status_code = 201
# Index the file for search
await search_service.index_entity(entity) # pyright: ignore
# Return success response
return JSONResponse(
status_code=status_code,
content={
"file_path": file_path,
"checksum": checksum,
"size": file_metadata.size,
"created_at": file_metadata.created_at.timestamp(),
"modified_at": file_metadata.modified_at.timestamp(),
},
)
except Exception as e: # pragma: no cover
logger.error(f"Error writing resource {file_path}: {e}")
raise HTTPException(status_code=500, detail=f"Failed to write resource: {str(e)}")
@@ -1,36 +0,0 @@
"""Router for search operations."""
from fastapi import APIRouter, BackgroundTasks
from basic_memory.api.routers.utils import to_search_results
from basic_memory.schemas.search import SearchQuery, SearchResponse
from basic_memory.deps import SearchServiceDep, EntityServiceDep
router = APIRouter(prefix="/search", tags=["search"])
@router.post("/", response_model=SearchResponse)
async def search(
query: SearchQuery,
search_service: SearchServiceDep,
entity_service: EntityServiceDep,
page: int = 1,
page_size: int = 10,
):
"""Search across all knowledge and documents."""
limit = page_size
offset = (page - 1) * page_size
results = await search_service.search(query, limit=limit, offset=offset)
search_results = await to_search_results(entity_service, results)
return SearchResponse(
results=search_results,
current_page=page,
page_size=page_size,
)
@router.post("/reindex")
async def reindex(background_tasks: BackgroundTasks, search_service: SearchServiceDep):
"""Recreate and populate the search index."""
await search_service.reindex_all(background_tasks=background_tasks)
return {"status": "ok", "message": "Reindex initiated"}
@@ -8,6 +8,7 @@ from basic_memory.api.v2.routers.resource_router import router as resource_route
from basic_memory.api.v2.routers.directory_router import router as directory_router
from basic_memory.api.v2.routers.prompt_router import router as prompt_router
from basic_memory.api.v2.routers.importer_router import router as importer_router
from basic_memory.api.v2.routers.schema_router import router as schema_router
__all__ = [
"knowledge_router",
@@ -18,4 +19,5 @@ __all__ = [
"directory_router",
"prompt_router",
"importer_router",
"schema_router",
]
@@ -32,14 +32,14 @@ async def import_chatgpt(
importer: ChatGPTImporterV2ExternalDep,
file: UploadFile,
project_id: str = Path(..., description="Project external UUID"),
folder: str = Form("conversations"),
directory: str = Form("conversations"),
) -> ChatImportResult:
"""Import conversations from ChatGPT JSON export.
Args:
project_id: Project external UUID from URL path
file: The ChatGPT conversations.json file.
folder: The folder to place the files in.
directory: The directory to place the files in.
importer: ChatGPT importer instance.
Returns:
@@ -49,7 +49,7 @@ async def import_chatgpt(
HTTPException: If import fails.
"""
logger.info(f"V2 Importing ChatGPT conversations for project {project_id}")
return await import_file(importer, file, folder)
return await import_file(importer, file, directory)
@router.post("/claude/conversations", response_model=ChatImportResult)
@@ -57,14 +57,14 @@ async def import_claude_conversations(
importer: ClaudeConversationsImporterV2ExternalDep,
file: UploadFile,
project_id: str = Path(..., description="Project external UUID"),
folder: str = Form("conversations"),
directory: str = Form("conversations"),
) -> ChatImportResult:
"""Import conversations from Claude conversations.json export.
Args:
project_id: Project external UUID from URL path
file: The Claude conversations.json file.
folder: The folder to place the files in.
directory: The directory to place the files in.
importer: Claude conversations importer instance.
Returns:
@@ -74,7 +74,7 @@ async def import_claude_conversations(
HTTPException: If import fails.
"""
logger.info(f"V2 Importing Claude conversations for project {project_id}")
return await import_file(importer, file, folder)
return await import_file(importer, file, directory)
@router.post("/claude/projects", response_model=ProjectImportResult)
@@ -82,14 +82,14 @@ async def import_claude_projects(
importer: ClaudeProjectsImporterV2ExternalDep,
file: UploadFile,
project_id: str = Path(..., description="Project external UUID"),
folder: str = Form("projects"),
directory: str = Form("projects"),
) -> ProjectImportResult:
"""Import projects from Claude projects.json export.
Args:
project_id: Project external UUID from URL path
file: The Claude projects.json file.
folder: The base folder to place the files in.
directory: The base directory to place the files in.
importer: Claude projects importer instance.
Returns:
@@ -99,7 +99,7 @@ async def import_claude_projects(
HTTPException: If import fails.
"""
logger.info(f"V2 Importing Claude projects for project {project_id}")
return await import_file(importer, file, folder)
return await import_file(importer, file, directory)
@router.post("/memory-json", response_model=EntityImportResult)
@@ -107,14 +107,14 @@ async def import_memory_json(
importer: MemoryJsonImporterV2ExternalDep,
file: UploadFile,
project_id: str = Path(..., description="Project external UUID"),
folder: str = Form("conversations"),
directory: str = Form("conversations"),
) -> EntityImportResult:
"""Import entities and relations from a memory.json file.
Args:
project_id: Project external UUID from URL path
file: The memory.json file.
folder: Optional destination folder within the project.
directory: Optional destination directory within the project.
importer: Memory JSON importer instance.
Returns:
@@ -132,7 +132,7 @@ async def import_memory_json(
json_data = json.loads(line)
file_data.append(json_data)
result = await importer.import_data(file_data, folder)
result = await importer.import_data(file_data, directory)
if not result.success: # pragma: no cover
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
@@ -147,13 +147,13 @@ async def import_memory_json(
return result
async def import_file(importer: Importer, file: UploadFile, destination_folder: str):
async def import_file(importer: Importer, file: UploadFile, destination_directory: str):
"""Helper function to import a file using an importer instance.
Args:
importer: The importer instance to use
file: The file to import
destination_folder: Destination folder for imported content
destination_directory: Destination directory for imported content
Returns:
Import result from the importer
@@ -164,7 +164,7 @@ async def import_file(importer: Importer, file: UploadFile, destination_folder:
try:
# Process file
json_data = json.load(file.file)
result = await importer.import_data(json_data, destination_folder)
result = await importer.import_data(json_data, destination_directory)
if not result.success: # pragma: no cover
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
@@ -10,7 +10,7 @@ Key improvements:
- Simplified caching strategies
"""
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Response, Path
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Response, Path, Query
from loguru import logger
from basic_memory.deps import (
@@ -19,9 +19,10 @@ from basic_memory.deps import (
LinkResolverV2ExternalDep,
ProjectConfigV2ExternalDep,
AppConfigDep,
SyncServiceV2ExternalDep,
EntityRepositoryV2ExternalDep,
ProjectExternalIdPathDep,
TaskSchedulerDep,
FileServiceV2ExternalDep,
)
from basic_memory.schemas import DeleteEntitiesResponse
from basic_memory.schemas.base import Entity
@@ -31,27 +32,27 @@ from basic_memory.schemas.v2 import (
EntityResolveResponse,
EntityResponseV2,
MoveEntityRequestV2,
MoveDirectoryRequestV2,
DeleteDirectoryRequestV2,
)
from basic_memory.schemas.response import DirectoryMoveResult, DirectoryDeleteResult
router = APIRouter(prefix="/knowledge", tags=["knowledge-v2"])
async def resolve_relations_background(sync_service, entity_id: int, entity_permalink: str) -> None:
"""Background task to resolve relations for a specific entity.
This runs asynchronously after the API response is sent, preventing
long delays when creating entities with many relations.
"""
try: # pragma: no cover
# Only resolve relations for the newly created entity
await sync_service.resolve_relations(entity_id=entity_id) # pragma: no cover
logger.debug( # pragma: no cover
f"Background: Resolved relations for entity {entity_permalink} (id={entity_id})"
)
except Exception as e: # pragma: no cover
# Log but don't fail - this is a background task
logger.warning( # pragma: no cover
f"Background: Failed to resolve relations for entity {entity_permalink}: {e}"
def _schedule_vector_sync_if_enabled(
*,
task_scheduler,
app_config,
entity_id: int,
project_id: int,
) -> None:
"""Schedule out-of-band vector sync only when semantic search is enabled."""
if app_config.semantic_search_enabled:
task_scheduler.schedule(
"sync_entity_vectors",
entity_id=entity_id,
project_id=project_id,
)
@@ -100,8 +101,12 @@ async def resolve_identifier(
resolution_method = "external_id" if entity else "search"
# If not found by external_id, try other resolution methods
# Pass source_path for context-aware resolution (prefers notes closer to source)
# Pass strict to control fuzzy search fallback (default False allows fuzzy matching)
if not entity:
entity = await link_resolver.resolve_link(data.identifier)
entity = await link_resolver.resolve_link(
data.identifier, source_path=data.source_path, strict=data.strict
)
if entity:
# Determine resolution method
if entity.permalink == data.identifier:
@@ -179,24 +184,50 @@ async def create_entity(
background_tasks: BackgroundTasks,
entity_service: EntityServiceV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
task_scheduler: TaskSchedulerDep,
file_service: FileServiceV2ExternalDep,
app_config: AppConfigDep,
fast: bool = Query(
True, description="If true, write quickly and defer indexing to background tasks."
),
) -> EntityResponseV2:
"""Create a new entity.
Args:
data: Entity data to create
fast: If True, defer indexing to background tasks
Returns:
Created entity with generated external_id (UUID)
Created entity with generated external_id (UUID) and file content
"""
logger.info(
"API v2 request", endpoint="create_entity", entity_type=data.entity_type, title=data.title
)
entity = await entity_service.create_entity(data)
if fast:
entity = await entity_service.fast_write_entity(data)
task_scheduler.schedule(
"reindex_entity",
entity_id=entity.id,
project_id=project_id,
)
else:
entity = await entity_service.create_entity(data)
await search_service.index_entity(entity)
_schedule_vector_sync_if_enabled(
task_scheduler=task_scheduler,
app_config=app_config,
entity_id=entity.id,
project_id=project_id,
)
# reindex
await search_service.index_entity(entity, background_tasks=background_tasks)
result = EntityResponseV2.model_validate(entity)
if fast:
result = result.model_copy(update={"observations": [], "relations": []})
# Always read and return file content
content = await file_service.read_file_content(entity.file_path)
result = result.model_copy(update={"content": content})
logger.info(
f"API v2 response: endpoint='create_entity' external_id={entity.external_id}, title={result.title}, permalink={result.permalink}, status_code=201"
@@ -215,9 +246,14 @@ async def update_entity_by_id(
project_id: ProjectExternalIdPathDep,
entity_service: EntityServiceV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
sync_service: SyncServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
task_scheduler: TaskSchedulerDep,
file_service: FileServiceV2ExternalDep,
app_config: AppConfigDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
fast: bool = Query(
True, description="If true, write quickly and defer indexing to background tasks."
),
) -> EntityResponseV2:
"""Update an entity by external ID.
@@ -226,30 +262,61 @@ async def update_entity_by_id(
Args:
entity_id: External ID (UUID string)
data: Updated entity data
fast: If True, defer indexing to background tasks
Returns:
Updated entity
Updated entity with file content
"""
logger.info(f"API v2 request: update_entity_by_id entity_id={entity_id}")
# Check if entity exists
# Check if entity exists (external_id is the source of truth for v2)
existing = await entity_repository.get_by_external_id(entity_id)
created = existing is None
# Perform update or create
entity, _ = await entity_service.create_or_update_entity(data)
response.status_code = 201 if created else 200
if fast:
entity = await entity_service.fast_write_entity(data, external_id=entity_id)
response.status_code = 200 if existing else 201
task_scheduler.schedule(
"reindex_entity",
entity_id=entity.id,
project_id=project_id,
resolve_relations=created,
)
else:
if existing:
# Update the existing entity in-place to avoid path-based duplication
entity = await entity_service.update_entity(existing, data)
response.status_code = 200
else:
# Create new entity, then bind external_id to the requested UUID
entity = await entity_service.create_entity(data)
if entity.external_id != entity_id:
entity = await entity_repository.update(
entity.id,
{"external_id": entity_id},
)
if not entity:
raise HTTPException(
status_code=404,
detail=f"Entity with external_id '{entity_id}' not found",
)
response.status_code = 201
# reindex
await search_service.index_entity(entity, background_tasks=background_tasks)
# Schedule relation resolution for new entities
if created:
background_tasks.add_task( # pragma: no cover
resolve_relations_background, sync_service, entity.id, entity.permalink or ""
await search_service.index_entity(entity)
_schedule_vector_sync_if_enabled(
task_scheduler=task_scheduler,
app_config=app_config,
entity_id=entity.id,
project_id=project_id,
)
result = EntityResponseV2.model_validate(entity)
if fast:
result = result.model_copy(update={"observations": [], "relations": []})
# Always read and return file content
content = await file_service.read_file_content(entity.file_path)
result = result.model_copy(update={"content": content})
logger.info(
f"API v2 response: external_id={entity_id}, created={created}, status_code={response.status_code}"
@@ -265,16 +332,23 @@ async def edit_entity_by_id(
entity_service: EntityServiceV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
task_scheduler: TaskSchedulerDep,
file_service: FileServiceV2ExternalDep,
app_config: AppConfigDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
fast: bool = Query(
True, description="If true, write quickly and defer indexing to background tasks."
),
) -> EntityResponseV2:
"""Edit an existing entity by external ID using operations like append, prepend, etc.
Args:
entity_id: External ID (UUID string)
data: Edit operation details
fast: If True, defer indexing to background tasks
Returns:
Updated entity
Updated entity with file content
Raises:
HTTPException: 404 if entity not found, 400 if edit fails
@@ -291,21 +365,47 @@ async def edit_entity_by_id(
)
try:
# Edit using the entity's permalink or path
identifier = entity.permalink or entity.file_path
updated_entity = await entity_service.edit_entity(
identifier=identifier,
operation=data.operation,
content=data.content,
section=data.section,
find_text=data.find_text,
expected_replacements=data.expected_replacements,
)
if fast:
updated_entity = await entity_service.fast_edit_entity(
entity=entity,
operation=data.operation,
content=data.content,
section=data.section,
find_text=data.find_text,
expected_replacements=data.expected_replacements,
)
task_scheduler.schedule(
"reindex_entity",
entity_id=updated_entity.id,
project_id=project_id,
)
else:
# Edit using the entity's permalink or path
identifier = entity.permalink or entity.file_path
updated_entity = await entity_service.edit_entity(
identifier=identifier,
operation=data.operation,
content=data.content,
section=data.section,
find_text=data.find_text,
expected_replacements=data.expected_replacements,
)
# Reindex
await search_service.index_entity(updated_entity, background_tasks=background_tasks)
await search_service.index_entity(updated_entity)
_schedule_vector_sync_if_enabled(
task_scheduler=task_scheduler,
app_config=app_config,
entity_id=updated_entity.id,
project_id=project_id,
)
result = EntityResponseV2.model_validate(updated_entity)
if fast:
result = result.model_copy(update={"observations": [], "relations": []})
# Always read and return file content
content = await file_service.read_file_content(updated_entity.file_path)
result = result.model_copy(update={"content": content})
logger.info(
f"API v2 response: external_id={entity_id}, operation='{data.operation}', status_code=200"
@@ -372,6 +472,7 @@ async def move_entity(
project_config: ProjectConfigV2ExternalDep,
app_config: AppConfigDep,
search_service: SearchServiceV2ExternalDep,
task_scheduler: TaskSchedulerDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
) -> EntityResponseV2:
"""Move an entity to a new file location.
@@ -410,7 +511,13 @@ async def move_entity(
# Reindex at new location
reindexed_entity = await entity_service.link_resolver.resolve_link(data.destination_path)
if reindexed_entity:
await search_service.index_entity(reindexed_entity, background_tasks=background_tasks)
await search_service.index_entity(reindexed_entity)
_schedule_vector_sync_if_enabled(
task_scheduler=task_scheduler,
app_config=app_config,
entity_id=reindexed_entity.id,
project_id=project_id,
)
result = EntityResponseV2.model_validate(moved_entity)
@@ -423,3 +530,107 @@ async def move_entity(
except Exception as e:
logger.error(f"Error moving entity: {e}")
raise HTTPException(status_code=400, detail=str(e))
## Move directory endpoint
@router.post("/move-directory", response_model=DirectoryMoveResult)
async def move_directory(
data: MoveDirectoryRequestV2,
background_tasks: BackgroundTasks,
project_id: ProjectExternalIdPathDep,
entity_service: EntityServiceV2ExternalDep,
project_config: ProjectConfigV2ExternalDep,
app_config: AppConfigDep,
search_service: SearchServiceV2ExternalDep,
task_scheduler: TaskSchedulerDep,
) -> DirectoryMoveResult:
"""Move all entities in a directory to a new location.
V2 API uses project external_id in the URL path for stable references.
Moves all files within a source directory to a destination directory,
updating database records and optionally updating permalinks.
Args:
project_id: Project external ID from URL path
data: Move request with source and destination directories
Returns:
DirectoryMoveResult with counts and details of moved files
"""
logger.info(
f"API v2 request: move_directory source='{data.source_directory}', destination='{data.destination_directory}'"
)
try:
# Move the directory using the service
result = await entity_service.move_directory(
source_directory=data.source_directory,
destination_directory=data.destination_directory,
project_config=project_config,
app_config=app_config,
)
# Reindex moved entities
for file_path in result.moved_files:
entity = await entity_service.link_resolver.resolve_link(file_path)
if entity:
await search_service.index_entity(entity)
_schedule_vector_sync_if_enabled(
task_scheduler=task_scheduler,
app_config=app_config,
entity_id=entity.id,
project_id=project_id,
)
logger.info(
f"API v2 response: move_directory "
f"total={result.total_files}, success={result.successful_moves}, failed={result.failed_moves}"
)
return result
except Exception as e:
logger.error(f"Error moving directory: {e}")
raise HTTPException(status_code=400, detail=str(e))
## Delete directory endpoint
@router.post("/delete-directory", response_model=DirectoryDeleteResult)
async def delete_directory(
data: DeleteDirectoryRequestV2,
project_id: ProjectExternalIdPathDep,
entity_service: EntityServiceV2ExternalDep,
) -> DirectoryDeleteResult:
"""Delete all entities in a directory.
V2 API uses project external_id in the URL path for stable references.
Deletes all files within a directory, updating database records and
removing files from the filesystem.
Args:
project_id: Project external ID from URL path
data: Delete request with directory path
Returns:
DirectoryDeleteResult with counts and details of deleted files
"""
logger.info(f"API v2 request: delete_directory directory='{data.directory}'")
try:
# Delete the directory using the service
result = await entity_service.delete_directory(
directory=data.directory,
)
logger.info(
f"API v2 response: delete_directory "
f"total={result.total_files}, success={result.successful_deletes}, failed={result.failed_deletes}"
)
return result
except Exception as e:
logger.error(f"Error deleting directory: {e}")
raise HTTPException(status_code=400, detail=str(e))
@@ -16,7 +16,7 @@ from basic_memory.schemas.memory import (
normalize_memory_url,
)
from basic_memory.schemas.search import SearchItemType
from basic_memory.api.routers.utils import to_graph_context
from basic_memory.api.v2.utils import to_graph_context
# Note: No prefix here - it's added during registration as /v2/{project_id}/memory
router = APIRouter(tags=["memory"])
@@ -19,9 +19,17 @@ from loguru import logger
from basic_memory.deps import (
ProjectServiceDep,
ProjectRepositoryDep,
ProjectConfigV2ExternalDep,
SyncServiceV2ExternalDep,
TaskSchedulerDep,
ProjectExternalIdPathDep,
)
from basic_memory.schemas import SyncReportResponse
from basic_memory.schemas.project_info import (
ProjectItem,
ProjectList,
ProjectInfoRequest,
ProjectInfoResponse,
ProjectStatusResponse,
)
from basic_memory.schemas.v2 import ProjectResolveRequest, ProjectResolveResponse
@@ -30,6 +38,175 @@ from basic_memory.utils import normalize_project_path, generate_permalink
router = APIRouter(prefix="/projects", tags=["project_management-v2"])
@router.get("/", response_model=ProjectList)
async def list_projects(
project_service: ProjectServiceDep,
) -> ProjectList:
"""List all configured projects.
Returns:
A list of all projects with metadata
"""
projects = await project_service.list_projects()
default_project = project_service.default_project
project_items = [
ProjectItem(
id=project.id,
external_id=project.external_id,
name=project.name,
path=normalize_project_path(project.path),
is_default=project.is_default or False,
)
for project in projects
]
return ProjectList(
projects=project_items,
default_project=default_project,
)
@router.post("/", response_model=ProjectStatusResponse, status_code=201)
async def add_project(
project_data: ProjectInfoRequest,
project_service: ProjectServiceDep,
) -> ProjectStatusResponse:
"""Add a new project to configuration and database.
Args:
project_data: The project name and path, with option to set as default
Returns:
Response confirming the project was added
"""
# Check if project already exists before attempting to add
existing_project = await project_service.get_project(project_data.name)
if existing_project:
# Project exists - check if paths match for true idempotency
# Normalize paths for comparison (resolve symlinks, etc.)
requested_path = os.path.abspath(os.path.expanduser(project_data.path))
existing_path = os.path.abspath(os.path.expanduser(existing_project.path))
if requested_path == existing_path:
# Same name, same path - return 200 OK (idempotent)
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message=f"Project '{project_data.name}' already exists",
status="success",
default=existing_project.is_default or False,
new_project=ProjectItem(
id=existing_project.id,
external_id=existing_project.external_id,
name=existing_project.name,
path=existing_project.path,
is_default=existing_project.is_default or False,
),
)
else:
# Same name, different path - this is an error
raise HTTPException(
status_code=400,
detail=(
f"Project '{project_data.name}' already exists with different path. "
f"Existing: {existing_project.path}, Requested: {project_data.path}"
),
)
try: # pragma: no cover
# The service layer handles cloud mode validation and path sanitization
await project_service.add_project(
project_data.name, project_data.path, set_default=project_data.set_default
)
# Fetch the newly created project to get its ID
new_project = await project_service.get_project(project_data.name)
if not new_project:
raise HTTPException(status_code=500, detail="Failed to retrieve newly created project")
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message=f"Project '{new_project.name}' added successfully",
status="success",
default=project_data.set_default,
new_project=ProjectItem(
id=new_project.id,
external_id=new_project.external_id,
name=new_project.name,
path=new_project.path,
is_default=new_project.is_default or False,
),
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e))
@router.post("/config/sync", response_model=ProjectStatusResponse)
async def synchronize_projects(
project_service: ProjectServiceDep,
) -> ProjectStatusResponse:
"""Synchronize projects between configuration file and database."""
try: # pragma: no cover
await project_service.synchronize_projects()
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message="Projects synchronized successfully between configuration and database",
status="success",
default=False,
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e))
@router.post("/{project_id}/sync")
async def sync_project(
sync_service: SyncServiceV2ExternalDep,
project_config: ProjectConfigV2ExternalDep,
task_scheduler: TaskSchedulerDep,
project_internal_id: ProjectExternalIdPathDep,
force_full: bool = Query(
False, description="Force full scan, bypassing watermark optimization"
),
run_in_background: bool = Query(True, description="Run in background"),
):
"""Force project filesystem sync to database."""
if run_in_background:
task_scheduler.schedule(
"sync_project",
project_id=project_internal_id,
force_full=force_full,
)
logger.info(
f"Filesystem sync initiated for project: {project_config.name} (force_full={force_full})"
)
return {
"status": "sync_started",
"message": f"Filesystem sync initiated for project '{project_config.name}'",
}
report = await sync_service.sync(
project_config.home, project_config.name, force_full=force_full
)
logger.info(
f"Filesystem sync completed for project: {project_config.name} (force_full={force_full})"
)
return SyncReportResponse.from_sync_report(report)
@router.post("/{project_id}/status", response_model=SyncReportResponse)
async def get_project_status(
sync_service: SyncServiceV2ExternalDep,
project_config: ProjectConfigV2ExternalDep,
project_id: str = Path(..., description="Project external ID (UUID)"),
force_full: bool = Query(
False, description="Force full scan, bypassing watermark optimization"
),
) -> SyncReportResponse:
"""Get sync status of files vs database for a project."""
logger.info(f"API v2 request: get_project_status for project_id={project_id}")
report = await sync_service.scan(project_config.home, force_full=force_full)
return SyncReportResponse.from_sync_report(report)
@router.post("/resolve", response_model=ProjectResolveResponse)
async def resolve_project_identifier(
data: ProjectResolveRequest,
@@ -147,6 +324,22 @@ async def get_project_by_id(
)
@router.get("/{project_id}/info", response_model=ProjectInfoResponse)
async def get_project_info_by_id(
project_service: ProjectServiceDep,
project_repository: ProjectRepositoryDep,
project_id: str = Path(..., description="Project external ID (UUID)"),
) -> ProjectInfoResponse:
"""Get detailed project information by external ID."""
logger.info(f"API v2 request: get_project_info_by_id for project_id={project_id}")
project = await project_repository.get_by_external_id(project_id)
if not project:
raise HTTPException(
status_code=404, detail=f"Project with external_id '{project_id}' not found"
)
return await project_service.get_project_info(project.name)
@router.patch("/{project_id}", response_model=ProjectStatusResponse)
async def update_project_by_id(
project_service: ProjectServiceDep,
@@ -9,7 +9,7 @@ from datetime import datetime, timezone
from fastapi import APIRouter, HTTPException, status, Path
from loguru import logger
from basic_memory.api.routers.utils import to_graph_context, to_search_results
from basic_memory.api.v2.utils import to_graph_context, to_search_results
from basic_memory.api.template_loader import template_loader
from basic_memory.schemas.base import parse_timeframe
from basic_memory.deps import (
@@ -9,6 +9,7 @@ Key differences from v1:
- More RESTful: POST for create, PUT for update, GET for read
"""
import uuid
from pathlib import Path as PathLib
from fastapi import APIRouter, HTTPException, Response, Path
@@ -147,7 +148,9 @@ async def create_resource(
entity_type = "canvas" if data.file_path.endswith(".canvas") else "file"
# Create a new entity model
# Explicitly set external_id to ensure NOT NULL constraint is satisfied (fixes #512)
entity = EntityModel(
external_id=str(uuid.uuid4()),
title=file_name,
entity_type=entity_type,
content_type=content_type,
@@ -0,0 +1,305 @@
"""V2 router for schema operations.
Provides endpoints for schema validation, inference, and drift detection.
The schema system validates notes against Picoschema definitions without
introducing any new data model -- it works entirely with existing
observations and relations.
Flow: Entity loaded with eager observations/relations -> convert to tuples -> core functions.
"""
from fastapi import APIRouter, Path, Query
from basic_memory.deps import (
SearchServiceV2ExternalDep,
EntityRepositoryV2ExternalDep,
)
from basic_memory.models.knowledge import Entity
from basic_memory.schemas.schema import (
ValidationReport,
InferenceReport,
DriftReport,
NoteValidationResponse,
FieldResultResponse,
FieldFrequencyResponse,
DriftFieldResponse,
)
from basic_memory.schemas.search import SearchQuery
from basic_memory.schema.resolver import resolve_schema
from basic_memory.schema.validator import validate_note
from basic_memory.schema.inference import infer_schema, NoteData, ObservationData, RelationData
from basic_memory.schema.diff import diff_schema
# Note: No prefix here -- it's added during registration as /v2/{project_id}/schema
router = APIRouter(tags=["schema"])
# --- ORM to core data conversion ---
def _entity_observations(entity: Entity) -> list[ObservationData]:
"""Extract ObservationData from an entity's observations."""
return [ObservationData(obs.category, obs.content) for obs in entity.observations]
def _entity_relations(entity: Entity) -> list[RelationData]:
"""Extract RelationData from an entity's outgoing relations.
Carries the target entity's type on each relation so the inference engine
can suggest correct types (e.g. works_at -> Organization, not the source type).
"""
return [
RelationData(
relation_type=rel.relation_type,
target_name=rel.to_name,
target_entity_type=rel.to_entity.entity_type if rel.to_entity else None,
)
for rel in entity.outgoing_relations
]
def _entity_to_note_data(entity: Entity) -> NoteData:
"""Convert an ORM Entity to a NoteData for inference/diff analysis."""
return NoteData(
identifier=entity.permalink or entity.file_path,
observations=_entity_observations(entity),
relations=_entity_relations(entity),
)
def _entity_frontmatter(entity: Entity) -> dict:
"""Build a frontmatter dict from an entity for schema resolution."""
frontmatter = dict(entity.entity_metadata) if entity.entity_metadata else {}
if entity.entity_type:
frontmatter.setdefault("type", entity.entity_type)
return frontmatter
# --- Validation ---
@router.post("/schema/validate", response_model=ValidationReport)
async def validate_schema(
entity_repository: EntityRepositoryV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
entity_type: str | None = Query(None, description="Entity type to validate"),
identifier: str | None = Query(None, description="Specific note identifier"),
):
"""Validate notes against their resolved schemas.
Validates a specific note (by identifier) or all notes of a given type.
Returns warnings/errors based on the schema's validation mode.
"""
results: list[NoteValidationResponse] = []
async def search_fn(query: str) -> list:
# Search for schema notes, then load full entity_metadata from the entity table.
# The search index only stores minimal metadata (e.g., {"entity_type": "schema"}),
# but parse_schema_note needs the full frontmatter with entity/schema/version keys.
results = await search_service.search(SearchQuery(text=query, types=["schema"]), limit=5)
frontmatters = []
for row in results:
if row.permalink:
entity = await entity_repository.get_by_permalink(row.permalink)
if entity:
frontmatters.append(_entity_frontmatter(entity))
return frontmatters
# --- Single note validation ---
if identifier:
entity = await entity_repository.get_by_permalink(identifier)
if not entity:
return ValidationReport(entity_type=entity_type, total_notes=0, results=[])
schema_def = await resolve_schema(_entity_frontmatter(entity), search_fn)
if schema_def:
result = validate_note(
entity.permalink or identifier,
schema_def,
_entity_observations(entity),
_entity_relations(entity),
)
results.append(_to_note_validation_response(result))
return ValidationReport(
entity_type=entity_type or entity.entity_type,
total_notes=1,
valid_count=1 if (results and results[0].passed) else 0,
warning_count=sum(len(r.warnings) for r in results),
error_count=sum(len(r.errors) for r in results),
results=results,
)
# --- Batch validation by entity type ---
entities = await _find_by_entity_type(entity_repository, entity_type) if entity_type else []
for entity in entities:
schema_def = await resolve_schema(_entity_frontmatter(entity), search_fn)
if schema_def:
result = validate_note(
entity.permalink or entity.file_path,
schema_def,
_entity_observations(entity),
_entity_relations(entity),
)
results.append(_to_note_validation_response(result))
valid = sum(1 for r in results if r.passed)
return ValidationReport(
entity_type=entity_type,
total_notes=len(results),
valid_count=valid,
warning_count=sum(len(r.warnings) for r in results),
error_count=sum(len(r.errors) for r in results),
results=results,
)
# --- Inference ---
@router.post("/schema/infer", response_model=InferenceReport)
async def infer_schema_endpoint(
entity_repository: EntityRepositoryV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
entity_type: str = Query(..., description="Entity type to analyze"),
threshold: float = Query(0.25, description="Minimum frequency for optional fields"),
):
"""Infer a schema from existing notes of a given type.
Examines observation categories and relation types across all notes
of the given type. Returns frequency analysis and suggested Picoschema.
"""
entities = await _find_by_entity_type(entity_repository, entity_type)
notes_data = [_entity_to_note_data(entity) for entity in entities]
result = infer_schema(entity_type, notes_data, optional_threshold=threshold)
return InferenceReport(
entity_type=result.entity_type,
notes_analyzed=result.notes_analyzed,
field_frequencies=[
FieldFrequencyResponse(
name=f.name,
source=f.source,
count=f.count,
total=f.total,
percentage=f.percentage,
sample_values=f.sample_values,
is_array=f.is_array,
target_type=f.target_type,
)
for f in result.field_frequencies
],
suggested_schema=result.suggested_schema,
suggested_required=result.suggested_required,
suggested_optional=result.suggested_optional,
excluded=result.excluded,
)
# --- Drift Detection ---
@router.get("/schema/diff/{entity_type}", response_model=DriftReport)
async def diff_schema_endpoint(
entity_repository: EntityRepositoryV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
entity_type: str = Path(..., description="Entity type to check for drift"),
project_id: str = Path(..., description="Project external UUID"),
):
"""Show drift between a schema definition and actual note usage.
Compares the existing schema for an entity type against how notes
of that type are actually structured. Identifies new fields, dropped
fields, and cardinality changes.
"""
async def search_fn(query: str) -> list:
# Search for schema notes, then load full entity_metadata from the entity table.
# The search index only stores minimal metadata (e.g., {"entity_type": "schema"}),
# but parse_schema_note needs the full frontmatter with entity/schema/version keys.
results = await search_service.search(SearchQuery(text=query, types=["schema"]), limit=5)
frontmatters = []
for row in results:
if row.permalink:
entity = await entity_repository.get_by_permalink(row.permalink)
if entity:
frontmatters.append(_entity_frontmatter(entity))
return frontmatters
# Resolve schema by entity type
schema_frontmatter = {"type": entity_type}
schema_def = await resolve_schema(schema_frontmatter, search_fn)
if not schema_def:
return DriftReport(entity_type=entity_type)
# Collect all notes of this type
entities = await _find_by_entity_type(entity_repository, entity_type)
notes_data = [_entity_to_note_data(entity) for entity in entities]
result = diff_schema(schema_def, notes_data)
return DriftReport(
entity_type=entity_type,
new_fields=[
DriftFieldResponse(
name=f.name,
source=f.source,
count=f.count,
total=f.total,
percentage=f.percentage,
)
for f in result.new_fields
],
dropped_fields=[
DriftFieldResponse(
name=f.name,
source=f.source,
count=f.count,
total=f.total,
percentage=f.percentage,
)
for f in result.dropped_fields
],
cardinality_changes=result.cardinality_changes,
)
# --- Helpers ---
async def _find_by_entity_type(
entity_repository: EntityRepositoryV2ExternalDep,
entity_type: str,
) -> list[Entity]:
"""Find all entities of a given type using the repository's select pattern."""
query = entity_repository.select().where(Entity.entity_type == entity_type)
result = await entity_repository.execute_query(query)
return list(result.scalars().all())
def _to_note_validation_response(result) -> NoteValidationResponse:
"""Convert a core ValidationResult to a Pydantic response model."""
return NoteValidationResponse(
note_identifier=result.note_identifier,
schema_entity=result.schema_entity,
passed=result.passed,
field_results=[
FieldResultResponse(
field_name=fr.field.name,
field_type=fr.field.type,
required=fr.field.required,
status=fr.status,
values=fr.values,
message=fr.message,
)
for fr in result.field_results
],
unmatched_observations=result.unmatched_observations,
unmatched_relations=result.unmatched_relations,
warnings=result.warnings,
errors=result.errors,
)
@@ -4,11 +4,20 @@ This router uses external_id UUIDs for stable, API-friendly routing.
V1 uses string-based project names which are less efficient and less stable.
"""
from fastapi import APIRouter, BackgroundTasks, Path
from fastapi import APIRouter, HTTPException, Path
from basic_memory.api.routers.utils import to_search_results
from basic_memory.api.v2.utils import to_search_results
from basic_memory.repository.semantic_errors import (
SemanticDependenciesMissingError,
SemanticSearchDisabledError,
)
from basic_memory.schemas.search import SearchQuery, SearchResponse
from basic_memory.deps import SearchServiceV2ExternalDep, EntityServiceV2ExternalDep
from basic_memory.deps import (
SearchServiceV2ExternalDep,
EntityServiceV2ExternalDep,
TaskSchedulerDep,
ProjectExternalIdPathDep,
)
# Note: No prefix here - it's added during registration as /v2/{project_id}/search
router = APIRouter(tags=["search"])
@@ -40,7 +49,14 @@ async def search(
"""
limit = page_size
offset = (page - 1) * page_size
results = await search_service.search(query, limit=limit, offset=offset)
try:
results = await search_service.search(query, limit=limit, offset=offset)
except SemanticSearchDisabledError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
except SemanticDependenciesMissingError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
search_results = await to_search_results(entity_service, results)
return SearchResponse(
results=search_results,
@@ -51,9 +67,8 @@ async def search(
@router.post("/search/reindex")
async def reindex(
background_tasks: BackgroundTasks,
search_service: SearchServiceV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
task_scheduler: TaskSchedulerDep,
project_id: ProjectExternalIdPathDep,
):
"""Recreate and populate the search index for a project.
@@ -63,11 +78,10 @@ async def reindex(
Args:
project_id: Project external UUID from URL path
background_tasks: FastAPI background tasks handler
search_service: Search service scoped to project
task_scheduler: Task scheduler for background work
Returns:
Status message indicating reindex has been initiated
"""
await search_service.reindex_all(background_tasks=background_tasks)
task_scheduler.schedule("reindex_project", project_id=project_id)
return {"status": "ok", "message": "Reindex initiated"}
@@ -24,29 +24,42 @@ async def to_graph_context(
page: Optional[int] = None,
page_size: Optional[int] = None,
):
# First pass: collect all entity IDs needed for relations
# First pass: collect all entity IDs needed for external_id lookup
# This includes: entity primary results, observation parent entities, relation from/to entities
entity_ids_needed: set[int] = set()
for context_item in context_result.results:
for item in (
[context_item.primary_result] + context_item.observations + context_item.related_results
):
if item.type == SearchItemType.RELATION:
if item.type == SearchItemType.ENTITY:
# Entity's own ID for its external_id
entity_ids_needed.add(item.id)
elif item.type == SearchItemType.OBSERVATION:
# Parent entity ID for entity_external_id
if item.entity_id: # pyright: ignore
entity_ids_needed.add(item.entity_id) # pyright: ignore
elif item.type == SearchItemType.RELATION:
# Source and target entity IDs for external_ids
if item.from_id: # pyright: ignore
entity_ids_needed.add(item.from_id) # pyright: ignore
if item.to_id:
entity_ids_needed.add(item.to_id)
# Batch fetch all entities at once
entity_lookup: dict[int, str] = {}
# Batch fetch all entities at once - get both title and external_id
entity_title_lookup: dict[int, str] = {}
entity_external_id_lookup: dict[int, str] = {}
if entity_ids_needed:
entities = await entity_repository.find_by_ids(list(entity_ids_needed))
entity_lookup = {e.id: e.title for e in entities}
for e in entities:
entity_title_lookup[e.id] = e.title
entity_external_id_lookup[e.id] = e.external_id
# Helper function to convert items to summaries
def to_summary(item: SearchIndexRow | ContextResultRow):
match item.type:
case SearchItemType.ENTITY:
return EntitySummary(
external_id=entity_external_id_lookup.get(item.id, ""),
entity_id=item.id,
title=item.title, # pyright: ignore
permalink=item.permalink,
@@ -55,10 +68,14 @@ async def to_graph_context(
created_at=item.created_at,
)
case SearchItemType.OBSERVATION:
entity_ext_id = None
if item.entity_id: # pyright: ignore
entity_ext_id = entity_external_id_lookup.get(item.entity_id) # pyright: ignore
return ObservationSummary(
observation_id=item.id,
entity_id=item.entity_id, # pyright: ignore
title=item.title, # pyright: ignore
entity_external_id=entity_ext_id,
title=entity_title_lookup.get(item.entity_id), # pyright: ignore
file_path=item.file_path,
category=item.category, # pyright: ignore
content=item.content, # pyright: ignore
@@ -66,8 +83,10 @@ async def to_graph_context(
created_at=item.created_at,
)
case SearchItemType.RELATION:
from_title = entity_lookup.get(item.from_id) if item.from_id else None # pyright: ignore
to_title = entity_lookup.get(item.to_id) if item.to_id else None
from_title = entity_title_lookup.get(item.from_id) if item.from_id else None # pyright: ignore
to_title = entity_title_lookup.get(item.to_id) if item.to_id else None
from_ext_id = entity_external_id_lookup.get(item.from_id) if item.from_id else None # pyright: ignore
to_ext_id = entity_external_id_lookup.get(item.to_id) if item.to_id else None
return RelationSummary(
relation_id=item.id,
entity_id=item.entity_id, # pyright: ignore
@@ -77,8 +96,10 @@ async def to_graph_context(
relation_type=item.relation_type, # pyright: ignore
from_entity=from_title,
from_entity_id=item.from_id, # pyright: ignore
from_entity_external_id=from_ext_id,
to_entity=to_title,
to_entity_id=item.to_id,
to_entity_external_id=to_ext_id,
created_at=item.created_at,
)
case _: # pragma: no cover
+13 -14
View File
@@ -1,9 +1,3 @@
# Suppress Logfire "not configured" warning - we only use Logfire in cloud/server contexts
import os
os.environ.setdefault("LOGFIRE_IGNORE_NO_CONFIG", "1")
# Remove loguru's default handler IMMEDIATELY, before any other imports.
# This prevents DEBUG logs from appearing on stdout during module-level
# initialization (e.g., template_loader.TemplateLoader() logs at DEBUG level).
from loguru import logger
@@ -15,8 +9,8 @@ from typing import Optional # noqa: E402
import typer # noqa: E402
from basic_memory.cli.container import CliContainer, set_container # noqa: E402
from basic_memory.cli.promo import maybe_show_cloud_promo # noqa: E402
from basic_memory.config import init_cli_logging # noqa: E402
from basic_memory.telemetry import show_notice_if_needed, track_app_started # noqa: E402
def version_callback(value: bool) -> None:
@@ -53,18 +47,23 @@ def app_callback(
container = CliContainer.create()
set_container(container)
# Show telemetry notice and track CLI startup
# Skip for 'mcp' command - it handles its own telemetry in lifespan
# Skip for 'telemetry' command - avoid issues when user is managing telemetry
if ctx.invoked_subcommand not in {"mcp", "telemetry"}:
show_notice_if_needed()
track_app_started("cli")
maybe_show_cloud_promo(ctx.invoked_subcommand)
# Run initialization for commands that don't use the API
# Skip for 'mcp' command - it has its own lifespan that handles initialization
# Skip for API-using commands (status, sync, etc.) - they handle initialization via deps.py
# Skip for 'reset' command - it manages its own database lifecycle
skip_init_commands = {"mcp", "status", "sync", "project", "tool", "reset"}
skip_init_commands = {
"doctor",
"mcp",
"status",
"sync",
"project",
"tool",
"reset",
"reindex",
"watch",
}
if (
not version
and ctx.invoked_subcommand is not None
+5 -3
View File
@@ -1,11 +1,12 @@
"""CLI commands for basic-memory."""
from . import status, db, import_memory_json, mcp, import_claude_conversations
from . import import_claude_projects, import_chatgpt, tool, project, format, telemetry
from . import status, db, doctor, import_memory_json, mcp, import_claude_conversations
from . import import_claude_projects, import_chatgpt, tool, project, format, schema, watch
__all__ = [
"status",
"db",
"doctor",
"import_memory_json",
"mcp",
"import_claude_conversations",
@@ -14,5 +15,6 @@ __all__ = [
"tool",
"project",
"format",
"telemetry",
"schema",
"watch",
]
@@ -1,6 +1,16 @@
"""Cloud commands package."""
from basic_memory.cli.app import cloud_app
# Import all commands to register them with typer
from basic_memory.cli.commands.cloud.core_commands import * # noqa: F401,F403
from basic_memory.cli.commands.cloud.api_client import get_authenticated_headers, get_cloud_config # noqa: F401
from basic_memory.cli.commands.cloud.upload_command import * # noqa: F401,F403
# Register snapshot sub-command group
from basic_memory.cli.commands.cloud.snapshot import snapshot_app
cloud_app.add_typer(snapshot_app, name="snapshot")
# Register restore command (directly on cloud_app via decorator)
from basic_memory.cli.commands.cloud.restore import restore # noqa: F401, E402
@@ -30,7 +30,7 @@ async def fetch_cloud_projects(
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
response = await api_request(method="GET", url=f"{host_url}/proxy/projects/projects")
response = await api_request(method="GET", url=f"{host_url}/proxy/v2/projects/")
return CloudProjectList.model_validate(response.json())
except Exception as e:
@@ -66,7 +66,7 @@ async def create_cloud_project(
response = await api_request(
method="POST",
url=f"{host_url}/proxy/projects/projects",
url=f"{host_url}/proxy/v2/projects/",
headers={"Content-Type": "application/json"},
json_data=project_data.model_dump(),
)
@@ -6,6 +6,7 @@ from rich.console import Console
from basic_memory.cli.app import cloud_app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.cli.auth import CLIAuth
from basic_memory.cli.promo import OSS_DISCOUNT_CODE
from basic_memory.config import ConfigManager
from basic_memory.cli.commands.cloud.api_client import (
CloudAPIError,
@@ -57,6 +58,10 @@ def login():
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(
f"OSS discount code: [bold]{OSS_DISCOUNT_CODE}[/bold] "
"(20% off for 3 months)\n"
)
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
console.print(
"[dim]Once you have an active subscription, run [bold]bm cloud login[/bold] again.[/dim]"
@@ -139,7 +144,6 @@ def status() -> None:
def setup() -> None:
"""Set up cloud sync by installing rclone and configuring credentials.
SPEC-20: Simplified to project-scoped workflow.
After setup, use project commands for syncing:
bm project add <name> <path> --local-path ~/projects/<name>
bm project bisync --name <name> --resync # First time
@@ -192,3 +196,93 @@ def setup() -> None:
except Exception as e:
console.print(f"\n[red]Unexpected error during setup: {e}[/red]")
raise typer.Exit(1)
@cloud_app.command("promo")
def promo(enabled: bool = typer.Option(True, "--on/--off", help="Enable or disable CLI promos.")):
"""Enable or disable CLI cloud promo messages."""
config_manager = ConfigManager()
config = config_manager.load_config()
config.cloud_promo_opt_out = not enabled
config_manager.save_config(config)
if enabled:
console.print("[green]Cloud promo messages enabled[/green]")
else:
console.print("[yellow]Cloud promo messages disabled[/yellow]")
@cloud_app.command("set-key")
def set_key(
api_key: str = typer.Argument(..., help="API key (bmc_ prefixed) for cloud access"),
) -> None:
"""Save a cloud API key for per-project cloud routing.
The API key is account-level and used by projects set to cloud mode.
Create a key in the web app or use 'bm cloud create-key'.
Example:
bm cloud set-key bmc_abc123...
"""
if not api_key.startswith("bmc_"):
console.print("[red]Error: API key must start with 'bmc_'[/red]")
raise typer.Exit(1)
config_manager = ConfigManager()
config = config_manager.load_config()
config.cloud_api_key = api_key
config_manager.save_config(config)
console.print("[green]API key saved[/green]")
console.print("[dim]Projects set to cloud mode will use this key for authentication[/dim]")
console.print("[dim]Set a project to cloud mode: bm project set-cloud <name>[/dim]")
@cloud_app.command("create-key")
def create_key(
name: str = typer.Argument(..., help="Human-readable name for the API key"),
) -> None:
"""Create a new cloud API key and save it locally.
Requires active OAuth session (run 'bm cloud login' first).
The key is created via the cloud API and saved to local config.
Example:
bm cloud create-key "my-laptop"
"""
async def _create_key():
_, _, host_url = get_cloud_config()
host_url = host_url.rstrip("/")
console.print(f"[dim]Creating API key '{name}'...[/dim]")
response = await make_api_request(
method="POST",
url=f"{host_url}/api/keys",
json_data={"name": name},
)
key_data = response.json()
api_key = key_data.get("key")
if not api_key:
console.print("[red]Error: No key returned from API[/red]")
raise typer.Exit(1)
# Save to config
config_manager = ConfigManager()
config = config_manager.load_config()
config.cloud_api_key = api_key
config_manager.save_config(config)
console.print(f"[green]API key '{name}' created and saved[/green]")
console.print("[dim]Projects set to cloud mode will use this key for authentication[/dim]")
console.print("[dim]Set a project to cloud mode: bm project set-cloud <name>[/dim]")
try:
run_with_cleanup(_create_key())
except CloudAPIError as e:
console.print(f"[red]Error creating API key: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
@@ -27,6 +27,16 @@ console = Console()
# Minimum rclone version for --create-empty-src-dirs support
MIN_RCLONE_VERSION_EMPTY_DIRS = (1, 64, 0)
# Tigris edge caching returns stale data for users outside the origin region (iad).
# --header is rclone's global flag that applies to ALL HTTP transactions (list, download,
# upload). This is critical because bisync starts with S3 ListObjectsV2, which is neither
# a download nor upload — so --header-download/--header-upload would miss list requests.
# See: https://www.tigrisdata.com/docs/objects/consistency/
TIGRIS_CONSISTENCY_HEADERS = [
"--header",
"X-Tigris-Consistent: true",
]
class RunResult(Protocol):
returncode: int
@@ -210,6 +220,7 @@ def project_sync(
"sync",
str(local_path),
remote_path,
*TIGRIS_CONSISTENCY_HEADERS,
"--filter-from",
str(filter_path),
]
@@ -279,6 +290,7 @@ def project_bisync(
"bisync",
str(local_path),
remote_path,
*TIGRIS_CONSISTENCY_HEADERS,
"--resilient",
"--conflict-resolve=newer",
"--max-delete=25",
@@ -354,6 +366,7 @@ def project_check(
"check",
str(local_path),
remote_path,
*TIGRIS_CONSISTENCY_HEADERS,
"--filter-from",
str(filter_path),
]
@@ -393,6 +406,6 @@ def project_ls(
if path:
remote_path = f"{remote_path}/{path}"
cmd = ["rclone", "ls", remote_path]
cmd = ["rclone", "ls", *TIGRIS_CONSISTENCY_HEADERS, remote_path]
result = run(cmd, capture_output=True, text=True, check=True)
return result.stdout.splitlines()
@@ -0,0 +1,159 @@
"""Restore CLI commands for Basic Memory Cloud.
SPEC-29 Phase 3: CLI commands for restoring files from Tigris bucket snapshots.
"""
import asyncio
import typer
from rich.console import Console
from basic_memory.cli.app import cloud_app
from basic_memory.cli.commands.cloud.api_client import (
CloudAPIError,
SubscriptionRequiredError,
make_api_request,
)
from basic_memory.cli.commands.cloud.schemas import BucketSnapshotBrowseResponse
from basic_memory.config import ConfigManager
console = Console()
@cloud_app.command("restore")
def restore(
path: str = typer.Argument(
...,
help="Path to restore (file or folder, e.g., 'notes/project.md' or 'research/')",
),
snapshot_id: str = typer.Option(
...,
"--snapshot",
"-s",
help="ID of the snapshot to restore from",
),
force: bool = typer.Option(
False,
"--force",
"-f",
help="Skip confirmation prompt",
),
) -> None:
"""Restore a file or folder from a snapshot.
This command restores files from a previous snapshot to the current bucket.
The restored files will overwrite any existing files at the same path.
Examples:
bm cloud restore notes/project.md --snapshot abc123
bm cloud restore research/ --snapshot abc123
bm cloud restore notes/project.md --snapshot abc123 --force
"""
async def _restore():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
# Normalize path - remove leading slash if present
normalized_path = path.lstrip("/")
if not force:
# Show what will be restored
console.print(f"[blue]Preparing to restore from snapshot {snapshot_id}[/blue]")
console.print(f" Path: {normalized_path}")
# Try to browse the snapshot to show what files will be affected
try:
browse_url = f"{host_url}/api/bucket-snapshots/{snapshot_id}/browse"
if normalized_path:
browse_url += f"?prefix={normalized_path}"
response = await make_api_request(
method="GET",
url=browse_url,
)
browse_response = BucketSnapshotBrowseResponse.model_validate(response.json())
if browse_response.files:
if len(browse_response.files) <= 10:
console.print("\n Files to restore:")
for file_info in browse_response.files:
console.print(f" - {file_info.key}")
else:
console.print(
f"\n {len(browse_response.files)} files will be restored"
)
console.print(" First 5 files:")
for file_info in browse_response.files[:5]:
console.print(f" - {file_info.key}")
console.print(f" ... and {len(browse_response.files) - 5} more")
else:
console.print(
f"\n[yellow]No files found matching '{normalized_path}' "
f"in snapshot[/yellow]"
)
raise typer.Exit(0)
except CloudAPIError as browse_error:
if browse_error.status_code == 404:
console.print(f"[red]Snapshot not found: {snapshot_id}[/red]")
raise typer.Exit(1)
# If browse fails for other reasons, proceed with confirmation anyway
pass
console.print(
"\n[yellow]Warning: Restored files will overwrite existing files![/yellow]"
)
confirmed = typer.confirm("\nProceed with restore?")
if not confirmed:
console.print("[yellow]Restore cancelled[/yellow]")
raise typer.Exit(0)
console.print(f"[blue]Restoring from snapshot {snapshot_id}...[/blue]")
response = await make_api_request(
method="POST",
url=f"{host_url}/api/bucket-snapshots/{snapshot_id}/restore",
json_data={"path": normalized_path},
)
data = response.json()
restored_files = data.get("restored", [])
returned_snapshot_id = data.get("snapshot_id", snapshot_id)
if restored_files:
console.print(f"[green]Successfully restored {len(restored_files)} file(s)[/green]")
if len(restored_files) <= 10:
for file_path in restored_files:
console.print(f" - {file_path}")
else:
console.print(" First 5 restored files:")
for file_path in restored_files[:5]:
console.print(f" - {file_path}")
console.print(f" ... and {len(restored_files) - 5} more")
console.print(f"\n[dim]Snapshot ID: {returned_snapshot_id}[/dim]")
else:
console.print("[yellow]No files were restored[/yellow]")
console.print(f"[dim]No files matching '{normalized_path}' found in snapshot[/dim]")
except typer.Exit:
# Re-raise typer.Exit without modification - it's used for clean exits
raise
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
if e.status_code == 404:
console.print(f"[red]Snapshot not found: {snapshot_id}[/red]")
else:
console.print(f"[red]Failed to restore: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_restore())
@@ -0,0 +1,55 @@
"""Pydantic schemas for Basic Memory Cloud API responses.
These schemas mirror the API response models from basic-memory-cloud
for type-safe parsing of API responses in CLI commands.
"""
from datetime import datetime
from uuid import UUID
from pydantic import BaseModel
class BucketSnapshotFileInfo(BaseModel):
"""File info from snapshot browse response."""
key: str
size: int
last_modified: datetime
etag: str | None = None
class BucketSnapshotBrowseResponse(BaseModel):
"""Response from browsing snapshot contents."""
files: list[BucketSnapshotFileInfo]
prefix: str
snapshot_version: str
class BucketSnapshotResponse(BaseModel):
"""Response model for bucket snapshot data."""
id: UUID
bucket_name: str
snapshot_version: str
name: str
description: str | None
auto: bool
created_at: datetime
created_by: UUID | None = None
class BucketSnapshotListResponse(BaseModel):
"""Response from listing bucket snapshots."""
snapshots: list[BucketSnapshotResponse]
total: int
class BucketSnapshotRestoreResponse(BaseModel):
"""Response from restore operation."""
restored: list[str]
snapshot_version: str
snapshot_id: UUID
@@ -0,0 +1,370 @@
"""Snapshot CLI commands for Basic Memory Cloud.
SPEC-29 Phase 3: CLI commands for managing Tigris bucket snapshots.
"""
import asyncio
from datetime import datetime
from typing import Optional
import typer
from rich.console import Console
from rich.table import Table
from basic_memory.cli.commands.cloud.api_client import (
CloudAPIError,
SubscriptionRequiredError,
make_api_request,
)
from basic_memory.cli.commands.cloud.schemas import BucketSnapshotBrowseResponse
from basic_memory.config import ConfigManager
console = Console()
snapshot_app = typer.Typer(help="Manage bucket snapshots")
def _format_timestamp(iso_timestamp: str) -> str:
"""Format ISO timestamp to a human-readable format."""
try:
dt = datetime.fromisoformat(iso_timestamp.replace("Z", "+00:00"))
return dt.strftime("%Y-%m-%d %H:%M:%S")
except (ValueError, AttributeError):
return iso_timestamp
@snapshot_app.command("create")
def create(
description: str = typer.Argument(
...,
help="Description for the snapshot",
),
) -> None:
"""Create a new bucket snapshot.
Examples:
bm cloud snapshot create "before major refactor"
bm cloud snapshot create "daily backup"
"""
async def _create():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
console.print("[blue]Creating snapshot...[/blue]")
response = await make_api_request(
method="POST",
url=f"{host_url}/api/bucket-snapshots",
json_data={"description": description},
)
data = response.json()
snapshot_id = data.get("id", "unknown")
snapshot_version = data.get("snapshot_version", "unknown")
created_at = _format_timestamp(data.get("created_at", ""))
console.print("[green]Snapshot created successfully[/green]")
console.print(f" ID: {snapshot_id}")
console.print(f" Version: {snapshot_version}")
console.print(f" Created: {created_at}")
console.print(f" Description: {description}")
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
console.print(f"[red]Failed to create snapshot: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_create())
@snapshot_app.command("list")
def list_snapshots(
limit: int = typer.Option(
10,
"--limit",
"-l",
help="Maximum number of snapshots to display",
),
) -> None:
"""List all bucket snapshots.
Examples:
bm cloud snapshot list
bm cloud snapshot list --limit 20
"""
async def _list():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
console.print("[blue]Fetching snapshots...[/blue]")
response = await make_api_request(
method="GET",
url=f"{host_url}/api/bucket-snapshots",
)
data = response.json()
snapshots = data.get("snapshots", [])
total = data.get("total", len(snapshots))
if not snapshots:
console.print("[yellow]No snapshots found[/yellow]")
console.print(
'\n[dim]Create a snapshot with: bm cloud snapshot create "description"[/dim]'
)
return
# Create a table for displaying snapshots
table = Table(title=f"Bucket Snapshots ({total} total)")
table.add_column("ID", style="cyan", no_wrap=True)
table.add_column("Description", style="white")
table.add_column("Auto", style="dim")
table.add_column("Created", style="green")
for snapshot in snapshots[:limit]:
snapshot_id = snapshot.get("id", "unknown")
desc = snapshot.get("description") or snapshot.get("name", "-")
auto = "yes" if snapshot.get("auto", False) else "no"
created_at = _format_timestamp(snapshot.get("created_at", ""))
table.add_row(snapshot_id, desc, auto, created_at)
console.print(table)
if total > limit:
console.print(
f"\n[dim]Showing {limit} of {total} snapshots. Use --limit to see more.[/dim]"
)
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
console.print(f"[red]Failed to list snapshots: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_list())
@snapshot_app.command("delete")
def delete(
snapshot_id: str = typer.Argument(
...,
help="The ID of the snapshot to delete",
),
force: bool = typer.Option(
False,
"--force",
"-f",
help="Skip confirmation prompt",
),
) -> None:
"""Delete a bucket snapshot.
Examples:
bm cloud snapshot delete abc123
bm cloud snapshot delete abc123 --force
"""
async def _delete():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
if not force:
# Fetch snapshot details first to show what will be deleted
console.print("[blue]Fetching snapshot details...[/blue]")
try:
response = await make_api_request(
method="GET",
url=f"{host_url}/api/bucket-snapshots/{snapshot_id}",
)
data = response.json()
desc = data.get("description") or data.get("name", "unnamed")
created_at = _format_timestamp(data.get("created_at", ""))
console.print("\nSnapshot to delete:")
console.print(f" ID: {snapshot_id}")
console.print(f" Description: {desc}")
console.print(f" Created: {created_at}")
except CloudAPIError:
# If we can't fetch details, proceed with confirmation anyway
pass
confirmed = typer.confirm("\nAre you sure you want to delete this snapshot?")
if not confirmed:
console.print("[yellow]Deletion cancelled[/yellow]")
raise typer.Exit(0)
console.print("[blue]Deleting snapshot...[/blue]")
await make_api_request(
method="DELETE",
url=f"{host_url}/api/bucket-snapshots/{snapshot_id}",
)
console.print(f"[green]Snapshot {snapshot_id} deleted successfully[/green]")
except typer.Exit:
# Re-raise typer.Exit without modification - it's used for clean exits
raise
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
if e.status_code == 404:
console.print(f"[red]Snapshot not found: {snapshot_id}[/red]")
else:
console.print(f"[red]Failed to delete snapshot: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_delete())
@snapshot_app.command("show")
def show(
snapshot_id: str = typer.Argument(
...,
help="The ID of the snapshot to show",
),
) -> None:
"""Show details of a specific snapshot.
Examples:
bm cloud snapshot show abc123
"""
async def _show():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
response = await make_api_request(
method="GET",
url=f"{host_url}/api/bucket-snapshots/{snapshot_id}",
)
data = response.json()
console.print("[bold blue]Snapshot Details[/bold blue]")
console.print(f" ID: {data.get('id', 'unknown')}")
console.print(f" Bucket: {data.get('bucket_name', 'unknown')}")
console.print(f" Version: {data.get('snapshot_version', 'unknown')}")
console.print(f" Name: {data.get('name', '-')}")
console.print(f" Description: {data.get('description') or '-'}")
console.print(f" Auto: {'yes' if data.get('auto', False) else 'no'}")
console.print(f" Created: {_format_timestamp(data.get('created_at', ''))}")
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
if e.status_code == 404:
console.print(f"[red]Snapshot not found: {snapshot_id}[/red]")
else:
console.print(f"[red]Failed to get snapshot details: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_show())
@snapshot_app.command("browse")
def browse(
snapshot_id: str = typer.Argument(
...,
help="The ID of the snapshot to browse",
),
prefix: Optional[str] = typer.Option(
None,
"--prefix",
"-p",
help="Filter files by path prefix (e.g., 'notes/')",
),
) -> None:
"""Browse contents of a snapshot.
Examples:
bm cloud snapshot browse abc123
bm cloud snapshot browse abc123 --prefix notes/
"""
async def _browse():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
url = f"{host_url}/api/bucket-snapshots/{snapshot_id}/browse"
if prefix:
url += f"?prefix={prefix}"
response = await make_api_request(
method="GET",
url=url,
)
browse_response = BucketSnapshotBrowseResponse.model_validate(response.json())
if not browse_response.files:
if prefix:
console.print(f"[yellow]No files found with prefix '{prefix}'[/yellow]")
else:
console.print("[yellow]No files found in snapshot[/yellow]")
return
console.print(
f"[bold blue]Snapshot Contents ({len(browse_response.files)} files)[/bold blue]"
)
for file_info in browse_response.files:
size_kb = file_info.size // 1024
console.print(f" {file_info.key} ({size_kb} KB)")
console.print(
f"\n[dim]Use 'bm cloud restore <path> --snapshot {snapshot_id}' "
f"to restore files[/dim]"
)
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
if e.status_code == 404:
console.print(f"[red]Snapshot not found: {snapshot_id}[/red]")
else:
console.print(f"[red]Failed to browse snapshot: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_browse())
@@ -14,7 +14,6 @@ from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.tools.utils import call_post, call_get
from basic_memory.mcp.project_context import get_active_project
from basic_memory.schemas import ProjectInfoResponse
from basic_memory.telemetry import shutdown_telemetry
console = Console()
@@ -24,8 +23,8 @@ T = TypeVar("T")
def run_with_cleanup(coro: Coroutine[Any, Any, T]) -> T:
"""Run an async coroutine with proper database cleanup.
This helper ensures database connections and telemetry threads are cleaned up
before the event loop closes, preventing process hangs in CLI commands.
This helper ensures database connections are cleaned up before the
event loop closes, preventing process hangs in CLI commands.
Args:
coro: The coroutine to run
@@ -39,9 +38,6 @@ def run_with_cleanup(coro: Coroutine[Any, Any, T]) -> T:
return await coro
finally:
await db.shutdown_db()
# Shutdown telemetry to stop the OpenPanel background thread
# This prevents hangs on Python 3.14+ during thread shutdown
shutdown_telemetry()
return asyncio.run(_with_cleanup())
@@ -62,7 +58,7 @@ async def run_sync(
try:
async with get_client() as client:
project_item = await get_active_project(client, project, None)
url = f"{project_item.project_url}/project/sync"
url = f"/v2/projects/{project_item.external_id}/sync"
params = []
if force_full:
params.append("force_full=true")
@@ -96,7 +92,7 @@ async def get_project_info(project: str):
try:
async with get_client() as client:
project_item = await get_active_project(client, project, None)
response = await call_get(client, f"{project_item.project_url}/project/info")
response = await call_get(client, f"/v2/projects/{project_item.external_id}/info")
return ProjectInfoResponse.model_validate(response.json())
except (ToolError, ValueError) as e:
console.print(f"[red]Sync failed: {e}[/red]")
+116
View File
@@ -5,6 +5,7 @@ from pathlib import Path
import typer
from loguru import logger
from rich.console import Console
from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn, TaskProgressColumn
from sqlalchemy.exc import OperationalError
from basic_memory import db
@@ -103,3 +104,118 @@ def reset(
# ensures db.shutdown_db() is called even if _reindex_projects changes
run_with_cleanup(_reindex_projects(app_config))
console.print("[green]Reindex complete[/green]")
@app.command()
def reindex(
embeddings: bool = typer.Option(
False, "--embeddings", "-e", help="Rebuild vector embeddings (requires semantic search)"
),
search: bool = typer.Option(False, "--search", "-s", help="Rebuild full-text search index"),
project: str = typer.Option(
None, "--project", "-p", help="Reindex a specific project (default: all)"
),
): # pragma: no cover
"""Rebuild search indexes and/or vector embeddings without dropping the database.
By default rebuilds everything (search + embeddings if semantic is enabled).
Use --search or --embeddings to rebuild only one.
Examples:
bm reindex # Rebuild everything
bm reindex --embeddings # Only rebuild vector embeddings
bm reindex --search # Only rebuild FTS index
bm reindex -p claw # Reindex only the 'claw' project
"""
# If neither flag is set, do both
if not embeddings and not search:
embeddings = True
search = True
config_manager = ConfigManager()
app_config = config_manager.config
if embeddings and not app_config.semantic_search_enabled:
console.print(
"[yellow]Semantic search is not enabled.[/yellow] "
"Set [cyan]semantic_search_enabled: true[/cyan] in config to use embeddings."
)
embeddings = False
if not search:
raise typer.Exit(0)
run_with_cleanup(_reindex(app_config, search=search, embeddings=embeddings, project=project))
async def _reindex(app_config, search: bool, embeddings: bool, project: str | None):
"""Run reindex operations."""
from basic_memory.repository import EntityRepository
from basic_memory.repository.search_repository import create_search_repository
from basic_memory.services.search_service import SearchService
from basic_memory.services.file_service import FileService
from basic_memory.markdown.markdown_processor import MarkdownProcessor
from basic_memory.markdown.entity_parser import EntityParser
try:
await reconcile_projects_with_config(app_config)
_, session_maker = await db.get_or_create_db(
db_path=app_config.database_path,
db_type=db.DatabaseType.FILESYSTEM,
)
project_repository = ProjectRepository(session_maker)
projects = await project_repository.get_active_projects()
if project:
projects = [p for p in projects if p.name == project]
if not projects:
console.print(f"[red]Project '{project}' not found.[/red]")
raise typer.Exit(1)
for proj in projects:
console.print(f"\n[bold]Project: [cyan]{proj.name}[/cyan][/bold]")
if search:
console.print(" Rebuilding full-text search index...")
sync_service = await get_sync_service(proj)
sync_dir = Path(proj.path)
await sync_service.sync(sync_dir, project_name=proj.name)
console.print(" [green]✓[/green] Full-text search index rebuilt")
if embeddings:
console.print(" Building vector embeddings...")
entity_repository = EntityRepository(session_maker, project_id=proj.id)
search_repository = create_search_repository(
session_maker, project_id=proj.id, app_config=app_config
)
project_path = Path(proj.path)
entity_parser = EntityParser(project_path)
markdown_processor = MarkdownProcessor(entity_parser, app_config=app_config)
file_service = FileService(project_path, markdown_processor, app_config=app_config)
search_service = SearchService(search_repository, entity_repository, file_service)
with Progress(
SpinnerColumn(),
TextColumn("[progress.description]{task.description}"),
BarColumn(),
TaskProgressColumn(),
console=console,
) as progress:
task = progress.add_task(" Embedding entities...", total=None)
def on_progress(entity_id, index, total):
progress.update(task, total=total, completed=index)
stats = await search_service.reindex_vectors(progress_callback=on_progress)
progress.update(task, completed=stats["total_entities"])
console.print(
f" [green]✓[/green] Embeddings complete: "
f"{stats['embedded']} entities embedded, "
f"{stats['skipped']} skipped, "
f"{stats['errors']} errors"
)
console.print("\n[green]Reindex complete![/green]")
finally:
await db.shutdown_db()
+153
View File
@@ -0,0 +1,153 @@
"""Doctor command for local consistency checks."""
from __future__ import annotations
import tempfile
import uuid
from pathlib import Path
from loguru import logger
from mcp.server.fastmcp.exceptions import ToolError
from rich.console import Console
import typer
from basic_memory.cli.app import app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.cli.commands.routing import force_routing, validate_routing_flags
from basic_memory.markdown.entity_parser import EntityParser
from basic_memory.markdown.markdown_processor import MarkdownProcessor
from basic_memory.markdown.schemas import EntityFrontmatter, EntityMarkdown
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.clients import KnowledgeClient, ProjectClient, SearchClient
from basic_memory.mcp.tools.utils import call_post
from basic_memory.schemas.base import Entity
from basic_memory.schemas.project_info import ProjectInfoRequest
from basic_memory.schemas.search import SearchQuery
from basic_memory.schemas import SyncReportResponse
console = Console()
async def run_doctor() -> None:
"""Run local consistency checks for file <-> database flows."""
console.print("[blue]Running Basic Memory doctor checks...[/blue]")
project_name = f"doctor-{uuid.uuid4().hex[:8]}"
api_note_title = "Doctor API Note"
manual_note_title = "Doctor Manual Note"
manual_permalink = "doctor/manual-note"
with tempfile.TemporaryDirectory() as temp_dir:
temp_path = Path(temp_dir)
async with get_client() as client:
project_client = ProjectClient(client)
project_request = ProjectInfoRequest(
name=project_name,
path=str(temp_path),
set_default=False,
)
project_id: str | None = None
try:
status = await project_client.create_project(project_request.model_dump())
if not status.new_project:
raise ValueError("Failed to create doctor project")
project_id = status.new_project.external_id
console.print(f"[green]OK[/green] Created doctor project: {project_name}")
# --- DB -> File: create an entity via API ---
knowledge_client = KnowledgeClient(client, project_id)
api_note = Entity(
title=api_note_title,
directory="doctor",
entity_type="note",
content_type="text/markdown",
content=f"# {api_note_title}\n\n- [note] API to file check",
entity_metadata={"tags": ["doctor"]},
)
api_result = await knowledge_client.create_entity(api_note.model_dump(), fast=False)
api_file = temp_path / api_result.file_path
if not api_file.exists():
raise ValueError(f"API note file missing: {api_result.file_path}")
api_text = api_file.read_text(encoding="utf-8")
if api_note_title not in api_text:
raise ValueError("API note content missing from file")
console.print("[green]OK[/green] API write created file")
# --- File -> DB: write markdown file directly, then sync ---
parser = EntityParser(temp_path)
processor = MarkdownProcessor(parser)
manual_markdown = EntityMarkdown(
frontmatter=EntityFrontmatter(
metadata={
"title": manual_note_title,
"type": "note",
"permalink": manual_permalink,
"tags": ["doctor"],
}
),
content=f"# {manual_note_title}\n\n- [note] File to DB check",
)
manual_path = temp_path / "doctor" / "manual-note.md"
await processor.write_file(manual_path, manual_markdown)
console.print("[green]OK[/green] Manual file written")
sync_response = await call_post(
client,
f"/v2/projects/{project_id}/sync?force_full=true&run_in_background=false",
)
sync_report = SyncReportResponse.model_validate(sync_response.json())
if sync_report.total == 0:
raise ValueError("Sync did not detect any changes")
console.print("[green]OK[/green] Sync indexed manual file")
search_client = SearchClient(client, project_id)
search_query = SearchQuery(title=manual_note_title)
search_results = await search_client.search(
search_query.model_dump(), page=1, page_size=5
)
if not any(result.title == manual_note_title for result in search_results.results):
raise ValueError("Manual note not found in search index")
console.print("[green]OK[/green] Search confirmed manual file")
status_response = await call_post(client, f"/v2/projects/{project_id}/status")
status_report = SyncReportResponse.model_validate(status_response.json())
if status_report.total != 0:
raise ValueError("Project status not clean after sync")
console.print("[green]OK[/green] Status clean after sync")
finally:
if project_id:
await project_client.delete_project(project_id)
console.print("[green]Doctor checks passed.[/green]")
@app.command()
def doctor(
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
) -> None:
"""Run local consistency checks to verify file/database sync."""
try:
validate_routing_flags(local, cloud)
with force_routing(local=local, cloud=cloud):
run_with_cleanup(run_doctor())
except (ToolError, ValueError) as e:
console.print(f"[red]Doctor failed: {e}[/red]")
raise typer.Exit(code=1)
except Exception as e:
logger.error(f"Doctor failed: {e}")
typer.echo(f"Doctor failed: {e}", err=True)
raise typer.Exit(code=1) # pragma: no cover
@@ -60,7 +60,9 @@ def import_chatgpt(
console.print(f"\nImporting chats from {conversations_json}...writing to {base_path}")
# Create importer and run import
importer = ChatGPTImporter(config.home, markdown_processor, file_service)
importer = ChatGPTImporter(
config.home, markdown_processor, file_service, project_name=config.name
)
with conversations_json.open("r", encoding="utf-8") as file:
json_data = json.load(file)
result = run_with_cleanup(importer.import_data(json_data, folder))
@@ -57,7 +57,9 @@ def import_claude(
markdown_processor, file_service = run_with_cleanup(get_importer_dependencies())
# Create the importer
importer = ClaudeConversationsImporter(config.home, markdown_processor, file_service)
importer = ClaudeConversationsImporter(
config.home, markdown_processor, file_service, project_name=config.name
)
# Process the file
base_path = config.home / folder
@@ -56,7 +56,9 @@ def import_projects(
markdown_processor, file_service = run_with_cleanup(get_importer_dependencies())
# Create the importer
importer = ClaudeProjectsImporter(config.home, markdown_processor, file_service)
importer = ClaudeProjectsImporter(
config.home, markdown_processor, file_service, project_name=config.name
)
# Process the file
base_path = config.home / base_folder if base_folder else config.home
@@ -55,7 +55,9 @@ def memory_json(
markdown_processor, file_service = run_with_cleanup(get_importer_dependencies())
# Create the importer
importer = MemoryJsonImporter(config.home, markdown_processor, file_service)
importer = MemoryJsonImporter(
config.home, markdown_processor, file_service, project_name=config.name
)
# Process the file
base_path = config.home if not destination_folder else config.home / destination_folder
+68 -56
View File
@@ -1,76 +1,88 @@
"""MCP server command with streamable HTTP transport."""
import os
from typing import Any, Optional
import typer
from typing import Optional
from loguru import logger
from basic_memory.cli.app import app
from basic_memory.config import ConfigManager, init_mcp_logging
# Import mcp instance (has lifespan that handles initialization and file sync)
from basic_memory.mcp.server import mcp as mcp_server # pragma: no cover
# Import mcp tools to register them
import basic_memory.mcp.tools # noqa: F401 # pragma: no cover
class _DeferredMcpServer:
def run(self, *args: Any, **kwargs: Any) -> None: # pragma: no cover
from basic_memory.mcp.server import mcp as live_mcp_server
# Import prompts to register them
import basic_memory.mcp.prompts # noqa: F401 # pragma: no cover
from loguru import logger
live_mcp_server.run(*args, **kwargs)
config = ConfigManager().config
if not config.cloud_mode_enabled:
# Keep module-level attribute for tests/monkeypatching while deferring heavy import.
mcp_server = _DeferredMcpServer()
@app.command()
def mcp(
transport: str = typer.Option(
"stdio", help="Transport type: stdio, streamable-http, or sse"
),
host: str = typer.Option(
"0.0.0.0", help="Host for HTTP transports (use 0.0.0.0 to allow external connections)"
),
port: int = typer.Option(8000, help="Port for HTTP transports"),
path: str = typer.Option("/mcp", help="Path prefix for streamable-http transport"),
project: Optional[str] = typer.Option(None, help="Restrict MCP server to single project"),
): # pragma: no cover
"""Run the MCP server with configurable transport options.
This command starts an MCP server using one of three transport options:
@app.command()
def mcp(
transport: str = typer.Option("stdio", help="Transport type: stdio, streamable-http, or sse"),
host: str = typer.Option(
"0.0.0.0", help="Host for HTTP transports (use 0.0.0.0 to allow external connections)"
),
port: int = typer.Option(8000, help="Port for HTTP transports"),
path: str = typer.Option("/mcp", help="Path prefix for streamable-http transport"),
project: Optional[str] = typer.Option(None, help="Restrict MCP server to single project"),
): # pragma: no cover
"""Run the MCP server with configurable transport options.
- stdio: Standard I/O (good for local usage)
- streamable-http: Recommended for web deployments (default)
- sse: Server-Sent Events (for compatibility with existing clients)
This command starts an MCP server using one of three transport options:
Initialization, file sync, and cleanup are handled by the MCP server's lifespan.
"""
# Initialize logging for MCP (file only, stdout breaks protocol)
init_mcp_logging()
- stdio: Standard I/O (good for local usage)
- streamable-http: Recommended for web deployments (default)
- sse: Server-Sent Events (for compatibility with existing clients)
# Validate and set project constraint if specified
if project:
config_manager = ConfigManager()
project_name, _ = config_manager.get_project(project)
if not project_name:
typer.echo(f"No project found named: {project}", err=True)
raise typer.Exit(1)
Initialization, file sync, and cleanup are handled by the MCP server's lifespan.
# Set env var with validated project name
os.environ["BASIC_MEMORY_MCP_PROJECT"] = project_name
logger.info(f"MCP server constrained to project: {project_name}")
Note: This command is available regardless of cloud mode setting.
Users who have cloud mode enabled can still use local MCP for Claude Code
and Claude Desktop while using cloud MCP for web and mobile access.
"""
# Force local routing for local MCP server
# Why: The local MCP server should always talk to the local API, not the cloud proxy.
# Even when cloud_mode_enabled is True, stdio MCP runs locally and needs local API access.
os.environ["BASIC_MEMORY_FORCE_LOCAL"] = "true"
# Run the MCP server (blocks)
# Lifespan handles: initialization, migrations, file sync, cleanup
logger.info(f"Starting MCP server with {transport.upper()} transport")
# Import mcp tools/prompts to register them with the server
import basic_memory.mcp.tools # noqa: F401 # pragma: no cover
import basic_memory.mcp.prompts # noqa: F401 # pragma: no cover
import basic_memory.mcp.resources # noqa: F401 # pragma: no cover
if transport == "stdio":
mcp_server.run(
transport=transport,
)
elif transport == "streamable-http" or transport == "sse":
mcp_server.run(
transport=transport,
host=host,
port=port,
path=path,
log_level="INFO",
)
# Initialize logging for MCP (file only, stdout breaks protocol)
init_mcp_logging()
# Validate and set project constraint if specified
if project:
config_manager = ConfigManager()
project_name, _ = config_manager.get_project(project)
if not project_name:
typer.echo(f"No project found named: {project}", err=True)
raise typer.Exit(1)
# Set env var with validated project name
os.environ["BASIC_MEMORY_MCP_PROJECT"] = project_name
logger.info(f"MCP server constrained to project: {project_name}")
# Run the MCP server (blocks)
# Lifespan handles: initialization, migrations, file sync, cleanup
logger.info(f"Starting MCP server with {transport.upper()} transport")
if transport == "stdio":
mcp_server.run(
transport=transport,
)
elif transport == "streamable-http" or transport == "sse":
mcp_server.run(
transport=transport,
host=host,
port=port,
path=path,
log_level="INFO",
)
+233 -63
View File
@@ -1,32 +1,33 @@
"""Command module for basic-memory project management."""
import json
import os
from datetime import datetime
from pathlib import Path
import typer
from rich.console import Console
from rich.panel import Panel
from rich.table import Table
from basic_memory.cli.app import app
from basic_memory.cli.commands.command_utils import get_project_info, run_with_cleanup
from basic_memory.config import ConfigManager
import json
from datetime import datetime
from rich.panel import Panel
from basic_memory.cli.commands.routing import force_routing, validate_routing_flags
from basic_memory.config import ConfigManager, ProjectMode
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.tools.utils import call_get, call_post, call_delete, call_put, call_patch
from basic_memory.mcp.tools.utils import call_delete, call_get, call_patch, call_post, call_put
from basic_memory.schemas.project_info import ProjectList, ProjectStatusResponse
from basic_memory.schemas.v2 import ProjectResolveResponse
from basic_memory.utils import generate_permalink, normalize_project_path
# Import rclone commands for project sync
from basic_memory.cli.commands.cloud.rclone_commands import (
SyncProject,
RcloneError,
project_sync,
SyncProject,
project_bisync,
project_check,
project_ls,
project_sync,
)
from basic_memory.cli.commands.cloud.bisync_commands import get_mount_info
@@ -46,40 +47,57 @@ def format_path(path: str) -> str:
@project_app.command("list")
def list_projects() -> None:
"""List all Basic Memory projects."""
def list_projects(
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
) -> None:
"""List all Basic Memory projects.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
try:
validate_routing_flags(local, cloud)
except ValueError as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
async def _list_projects():
async with get_client() as client:
response = await call_get(client, "/projects/projects")
response = await call_get(client, "/v2/projects/")
return ProjectList.model_validate(response.json())
try:
result = run_with_cleanup(_list_projects())
with force_routing(local=local, cloud=cloud):
result = run_with_cleanup(_list_projects())
config = ConfigManager().config
table = Table(title="Basic Memory Projects")
table.add_column("Name", style="cyan")
table.add_column("Path", style="green")
table.add_column("Mode", style="blue")
# Add Local Path column if in cloud mode
if config.cloud_mode_enabled:
# Add Local Path column if in cloud mode and not forcing local
if config.cloud_mode_enabled and not local:
table.add_column("Local Path", style="yellow", no_wrap=True, overflow="fold")
# Show Default column in local mode or if default_project_mode is enabled in cloud mode
show_default_column = not config.cloud_mode_enabled or config.default_project_mode
show_default_column = local or not config.cloud_mode_enabled or config.default_project_mode
if show_default_column:
table.add_column("Default", style="magenta")
for project in result.projects:
is_default = "[X]" if project.is_default else ""
normalized_path = normalize_project_path(project.path)
project_mode = config.get_project_mode(project.name).value
# Build row based on mode
row = [project.name, format_path(normalized_path)]
row = [project.name, format_path(normalized_path), project_mode]
# Add local path if in cloud mode
if config.cloud_mode_enabled:
# Add local path if in cloud mode and not forcing local
if config.cloud_mode_enabled and not local:
local_path = ""
if project.name in config.cloud_projects:
local_path = config.cloud_projects[project.name].local_path or ""
@@ -108,9 +126,16 @@ def add_project(
None, "--local-path", help="Local sync path for cloud mode (optional)"
),
set_default: bool = typer.Option(False, "--default", help="Set as default project"),
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
) -> None:
"""Add a new project.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
Cloud mode examples:\n
bm project add research # No local sync\n
bm project add research --local-path ~/docs # With local sync\n
@@ -118,14 +143,23 @@ def add_project(
Local mode example:\n
bm project add research ~/Documents/research
"""
try:
validate_routing_flags(local, cloud)
except ValueError as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
config = ConfigManager().config
# Determine effective mode: local flag forces local mode behavior
effective_cloud_mode = config.cloud_mode_enabled and not local
# Resolve local sync path early (needed for both cloud and local mode)
local_sync_path: str | None = None
if local_path:
local_sync_path = Path(os.path.abspath(os.path.expanduser(local_path))).as_posix()
if config.cloud_mode_enabled:
if effective_cloud_mode:
# Cloud mode: path auto-generated from name, local sync is optional
async def _add_project():
@@ -136,7 +170,7 @@ def add_project(
"local_sync_path": local_sync_path,
"set_default": set_default,
}
response = await call_post(client, "/projects/projects", json=data)
response = await call_post(client, "/v2/projects/", json=data)
return ProjectStatusResponse.model_validate(response.json())
else:
# Local mode: path is required
@@ -150,15 +184,16 @@ def add_project(
async def _add_project():
async with get_client() as client:
data = {"name": name, "path": resolved_path, "set_default": set_default}
response = await call_post(client, "/projects/projects", json=data)
response = await call_post(client, "/v2/projects/", json=data)
return ProjectStatusResponse.model_validate(response.json())
try:
result = run_with_cleanup(_add_project())
with force_routing(local=local, cloud=cloud):
result = run_with_cleanup(_add_project())
console.print(f"[green]{result.message}[/green]")
# Save local sync path to config if in cloud mode
if config.cloud_mode_enabled and local_sync_path:
if effective_cloud_mode and local_sync_path:
from basic_memory.config import CloudProjectConfig
# Create local directory if it doesn't exist
@@ -202,7 +237,7 @@ def setup_project_sync(
async def _verify_project_exists():
"""Verify the project exists on cloud by listing all projects."""
async with get_client() as client:
response = await call_get(client, "/projects/projects")
response = await call_get(client, "/v2/projects/")
project_list = response.json()
project_names = [p["name"] for p in project_list["projects"]]
if name not in project_names:
@@ -243,8 +278,21 @@ def remove_project(
delete_notes: bool = typer.Option(
False, "--delete-notes", help="Delete project files from disk"
),
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
) -> None:
"""Remove a project."""
"""Remove a project.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
try:
validate_routing_flags(local, cloud)
except ValueError as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
async def _remove_project():
async with get_client() as client:
@@ -265,11 +313,11 @@ def remove_project(
try:
# Get config to check for local sync path and bisync state
config = ConfigManager().config
local_path = None
local_path_config = None
has_bisync_state = False
if config.cloud_mode_enabled and name in config.cloud_projects:
local_path = config.cloud_projects[name].local_path
if config.cloud_mode_enabled and not local and name in config.cloud_projects:
local_path_config = config.cloud_projects[name].local_path
# Check for bisync state
from basic_memory.cli.commands.cloud.rclone_commands import get_project_bisync_state
@@ -278,17 +326,18 @@ def remove_project(
has_bisync_state = bisync_state_path.exists()
# Remove project from cloud/API
result = run_with_cleanup(_remove_project())
with force_routing(local=local, cloud=cloud):
result = run_with_cleanup(_remove_project())
console.print(f"[green]{result.message}[/green]")
# Clean up local sync directory if it exists and delete_notes is True
if delete_notes and local_path:
local_dir = Path(local_path)
if delete_notes and local_path_config:
local_dir = Path(local_path_config)
if local_dir.exists():
import shutil
shutil.rmtree(local_dir)
console.print(f"[green]Removed local sync directory: {local_path}[/green]")
console.print(f"[green]Removed local sync directory: {local_path_config}[/green]")
# Clean up bisync state if it exists
if has_bisync_state:
@@ -301,14 +350,14 @@ def remove_project(
console.print("[green]Removed bisync state[/green]")
# Clean up cloud_projects config entry
if config.cloud_mode_enabled and name in config.cloud_projects:
if config.cloud_mode_enabled and not local and name in config.cloud_projects:
del config.cloud_projects[name]
ConfigManager().save_config(config)
# Show informative message if files were not deleted
if not delete_notes:
if local_path:
console.print(f"[yellow]Note: Local files remain at {local_path}[/yellow]")
if local_path_config:
console.print(f"[yellow]Note: Local files remain at {local_path_config}[/yellow]")
except Exception as e:
console.print(f"[red]Error removing project: {str(e)}[/red]")
@@ -318,15 +367,24 @@ def remove_project(
@project_app.command("default")
def set_default_project(
name: str = typer.Argument(..., help="Name of the project to set as CLI default"),
local: bool = typer.Option(
False, "--local", help="Force local API routing (required in cloud mode)"
),
) -> None:
"""Set the default project when 'config.default_project_mode' is set.
Note: This command is only available in local mode.
In cloud mode, use --local to modify the local configuration.
"""
config = ConfigManager().config
if config.cloud_mode_enabled:
console.print("[red]Error: 'default' command is not available in cloud mode[/red]")
# Trigger: cloud mode enabled without --local flag
# Why: default project is a local configuration concept
# Outcome: require explicit --local flag to modify local config in cloud mode
if config.cloud_mode_enabled and not local:
console.print(
"[red]Error: 'default' command requires --local flag in cloud mode[/red]\n"
"[yellow]Hint: Use 'bm project default <name> --local' to set local default[/yellow]"
)
raise typer.Exit(1)
async def _set_default():
@@ -346,7 +404,8 @@ def set_default_project(
return ProjectStatusResponse.model_validate(response.json())
try:
result = run_with_cleanup(_set_default())
with force_routing(local=local):
result = run_with_cleanup(_set_default())
console.print(f"[green]{result.message}[/green]")
except Exception as e:
console.print(f"[red]Error setting default project: {str(e)}[/red]")
@@ -354,24 +413,35 @@ def set_default_project(
@project_app.command("sync-config")
def synchronize_projects() -> None:
def synchronize_projects(
local: bool = typer.Option(
False, "--local", help="Force local API routing (required in cloud mode)"
),
) -> None:
"""Synchronize project config between configuration file and database.
Note: This command is only available in local mode.
In cloud mode, use --local to sync local configuration.
"""
config = ConfigManager().config
if config.cloud_mode_enabled:
console.print("[red]Error: 'sync-config' command is not available in cloud mode[/red]")
# Trigger: cloud mode enabled without --local flag
# Why: sync-config syncs local config file with local database
# Outcome: require explicit --local flag to clarify intent in cloud mode
if config.cloud_mode_enabled and not local:
console.print(
"[red]Error: 'sync-config' command requires --local flag in cloud mode[/red]\n"
"[yellow]Hint: Use 'bm project sync-config --local' to sync local config[/yellow]"
)
raise typer.Exit(1)
async def _sync_config():
async with get_client() as client:
response = await call_post(client, "/projects/config/sync")
response = await call_post(client, "/v2/projects/config/sync")
return ProjectStatusResponse.model_validate(response.json())
try:
result = run_with_cleanup(_sync_config())
with force_routing(local=local):
result = run_with_cleanup(_sync_config())
console.print(f"[green]{result.message}[/green]")
except Exception as e: # pragma: no cover
console.print(f"[red]Error synchronizing projects: {str(e)}[/red]")
@@ -382,15 +452,24 @@ def synchronize_projects() -> None:
def move_project(
name: str = typer.Argument(..., help="Name of the project to move"),
new_path: str = typer.Argument(..., help="New absolute path for the project"),
local: bool = typer.Option(
False, "--local", help="Force local API routing (required in cloud mode)"
),
) -> None:
"""Move a project to a new location.
Note: This command is only available in local mode.
In cloud mode, use --local to modify local project paths.
"""
config = ConfigManager().config
if config.cloud_mode_enabled:
console.print("[red]Error: 'move' command is not available in cloud mode[/red]")
# Trigger: cloud mode enabled without --local flag
# Why: moving a project is a local file system operation
# Outcome: require explicit --local flag to clarify intent in cloud mode
if config.cloud_mode_enabled and not local:
console.print(
"[red]Error: 'move' command requires --local flag in cloud mode[/red]\n"
"[yellow]Hint: Use 'bm project move <name> <path> --local' to move local project[/yellow]"
)
raise typer.Exit(1)
# Resolve to absolute path
@@ -399,14 +478,20 @@ def move_project(
async def _move_project():
async with get_client() as client:
data = {"path": resolved_path}
project_permalink = generate_permalink(name)
# TODO fix route to use ProjectPathDep
response = await call_patch(client, f"/{name}/project/{project_permalink}", json=data)
resolve_response = await call_post(
client,
"/v2/projects/resolve",
json={"identifier": name},
)
project_info = ProjectResolveResponse.model_validate(resolve_response.json())
response = await call_patch(
client, f"/v2/projects/{project_info.external_id}", json=data
)
return ProjectStatusResponse.model_validate(response.json())
try:
result = run_with_cleanup(_move_project())
with force_routing(local=local):
result = run_with_cleanup(_move_project())
console.print(f"[green]{result.message}[/green]")
# Show important file movement reminder
@@ -428,6 +513,74 @@ def move_project(
raise typer.Exit(1)
@project_app.command("set-cloud")
def set_cloud(
name: str = typer.Argument(..., help="Name of the project to route through cloud"),
) -> None:
"""Set a project to cloud mode (route through cloud API).
Requires either an API key or an active OAuth session.
Examples:
bm cloud set-key bmc_abc123... # save API key, then:
bm project set-cloud research # route "research" through cloud
bm cloud login # OAuth login, then:
bm project set-cloud research # route "research" through cloud
"""
from basic_memory.cli.auth import CLIAuth
config_manager = ConfigManager()
config = config_manager.config
# Validate project exists in config
if name not in config.projects:
console.print(f"[red]Error: Project '{name}' not found in config[/red]")
raise typer.Exit(1)
# Validate credentials: API key or OAuth session
has_api_key = bool(config.cloud_api_key)
has_oauth = False
if not has_api_key:
auth = CLIAuth(client_id=config.cloud_client_id, authkit_domain=config.cloud_domain)
has_oauth = auth.load_tokens() is not None
if not has_api_key and not has_oauth:
console.print("[red]Error: No cloud credentials found[/red]")
console.print("[dim]Run 'bm cloud set-key <key>' or 'bm cloud login' first[/dim]")
raise typer.Exit(1)
config.set_project_mode(name, ProjectMode.CLOUD)
config_manager.save_config(config)
console.print(f"[green]Project '{name}' set to cloud mode[/green]")
console.print("[dim]MCP tools and CLI commands for this project will route through cloud[/dim]")
@project_app.command("set-local")
def set_local(
name: str = typer.Argument(..., help="Name of the project to revert to local mode"),
) -> None:
"""Revert a project to local mode (use in-process ASGI transport).
Example:
bm project set-local research
"""
config_manager = ConfigManager()
config = config_manager.config
# Validate project exists in config
if name not in config.projects:
console.print(f"[red]Error: Project '{name}' not found in config[/red]")
raise typer.Exit(1)
config.set_project_mode(name, ProjectMode.LOCAL)
config_manager.save_config(config)
console.print(f"[green]Project '{name}' set to local mode[/green]")
console.print("[dim]MCP tools and CLI commands for this project will use local transport[/dim]")
@project_app.command("sync")
def sync_project_command(
name: str = typer.Option(..., "--name", help="Project name to sync"),
@@ -453,7 +606,7 @@ def sync_project_command(
# Get project info
async def _get_project():
async with get_client() as client:
response = await call_get(client, "/projects/projects")
response = await call_get(client, "/v2/projects/")
projects_list = ProjectList.model_validate(response.json())
for proj in projects_list.projects:
if generate_permalink(proj.name) == generate_permalink(name):
@@ -494,9 +647,10 @@ def sync_project_command(
async def _trigger_db_sync():
async with get_client() as client:
permalink = generate_permalink(name)
response = await call_post(
client, f"/{permalink}/project/sync?force_full=true", json={}
client,
f"/v2/projects/{project_data.external_id}/sync?force_full=true",
json={},
)
return response.json()
@@ -544,7 +698,7 @@ def bisync_project_command(
# Get project info
async def _get_project():
async with get_client() as client:
response = await call_get(client, "/projects/projects")
response = await call_get(client, "/v2/projects/")
projects_list = ProjectList.model_validate(response.json())
for proj in projects_list.projects:
if generate_permalink(proj.name) == generate_permalink(name):
@@ -592,9 +746,10 @@ def bisync_project_command(
async def _trigger_db_sync():
async with get_client() as client:
permalink = generate_permalink(name)
response = await call_post(
client, f"/{permalink}/project/sync?force_full=true", json={}
client,
f"/v2/projects/{project_data.external_id}/sync?force_full=true",
json={},
)
return response.json()
@@ -638,7 +793,7 @@ def check_project_command(
# Get project info
async def _get_project():
async with get_client() as client:
response = await call_get(client, "/projects/projects")
response = await call_get(client, "/v2/projects/")
projects_list = ProjectList.model_validate(response.json())
for proj in projects_list.projects:
if generate_permalink(proj.name) == generate_permalink(name):
@@ -739,7 +894,7 @@ def ls_project_command(
# Get project info
async def _get_project():
async with get_client() as client:
response = await call_get(client, "/projects/projects")
response = await call_get(client, "/v2/projects/")
projects_list = ProjectList.model_validate(response.json())
for proj in projects_list.projects:
if generate_permalink(proj.name) == generate_permalink(name):
@@ -779,11 +934,26 @@ def ls_project_command(
def display_project_info(
name: str = typer.Argument(..., help="Name of the project"),
json_output: bool = typer.Option(False, "--json", help="Output in JSON format"),
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Display detailed information and statistics about the current project."""
"""Display detailed information and statistics about the current project.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
try:
validate_routing_flags(local, cloud)
except ValueError as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
try:
# Get project info
info = run_with_cleanup(get_project_info(name))
with force_routing(local=local, cloud=cloud):
info = run_with_cleanup(get_project_info(name))
if json_output:
# Convert to JSON and print
+73
View File
@@ -0,0 +1,73 @@
"""CLI routing utilities for --local/--cloud flag handling.
This module provides utilities for CLI commands to override the default routing
behavior (determined by cloud_mode_enabled in config). This allows users to:
1. Use local MCP server even when cloud mode is enabled
2. Force local routing for specific CLI commands with --local flag
3. Force cloud routing with --cloud flag (requires authentication)
The routing is controlled via environment variables:
- BASIC_MEMORY_FORCE_LOCAL: When "true", forces local ASGI transport
- These are checked in basic_memory.mcp.async_client.get_client()
"""
import os
from contextlib import contextmanager
from typing import Generator
@contextmanager
def force_routing(local: bool = False, cloud: bool = False) -> Generator[None, None, None]:
"""Context manager to temporarily override routing mode.
Sets environment variables that are checked by get_client() to determine
whether to use local ASGI transport or cloud proxy transport.
Args:
local: If True, force local ASGI transport (ignores cloud_mode_enabled)
cloud: If True, clear force_local to allow cloud routing
Usage:
with force_routing(local=True):
# All API calls will use local ASGI transport
await some_api_call()
Raises:
ValueError: If both local and cloud are True
"""
if local and cloud:
raise ValueError("Cannot specify both --local and --cloud")
# Save original values
original_force_local = os.environ.get("BASIC_MEMORY_FORCE_LOCAL")
try:
if local:
# Force local routing by setting the env var
os.environ["BASIC_MEMORY_FORCE_LOCAL"] = "true"
elif cloud:
# Ensure force_local is NOT set, let cloud_mode_enabled take effect
os.environ.pop("BASIC_MEMORY_FORCE_LOCAL", None)
# If neither is set, don't change anything (use default behavior)
yield
finally:
# Restore original value
if original_force_local is None:
os.environ.pop("BASIC_MEMORY_FORCE_LOCAL", None)
else:
os.environ["BASIC_MEMORY_FORCE_LOCAL"] = original_force_local
def validate_routing_flags(local: bool, cloud: bool) -> None:
"""Validate that --local and --cloud flags are not both specified.
Args:
local: Value of --local flag
cloud: Value of --cloud flag
Raises:
ValueError: If both flags are True
"""
if local and cloud:
raise ValueError("Cannot specify both --local and --cloud flags")
+336
View File
@@ -0,0 +1,336 @@
"""Schema management CLI commands for Basic Memory.
Provides CLI access to schema validation, inference, and drift detection.
Registered as a subcommand group: `bm schema validate`, `bm schema infer`, `bm schema diff`.
"""
import json
from typing import Annotated, Optional
import typer
from loguru import logger
from rich.console import Console
from rich.panel import Panel
from rich.table import Table
from basic_memory.cli.app import app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.cli.commands.routing import force_routing, validate_routing_flags
from basic_memory.config import ConfigManager
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.project_context import get_active_project
console = Console()
schema_app = typer.Typer(help="Schema management commands")
app.add_typer(schema_app, name="schema")
def _resolve_project_name(project: Optional[str]) -> Optional[str]:
"""Resolve project name from CLI argument or config default."""
config_manager = ConfigManager()
if project is not None:
project_name, _ = config_manager.get_project(project)
if not project_name:
typer.echo(f"No project found named: {project}", err=True)
raise typer.Exit(1)
return project_name
return config_manager.default_project
# --- Validate ---
async def _run_validate(
target: Optional[str] = None,
project: Optional[str] = None,
strict: bool = False,
):
"""Run schema validation via the API."""
from basic_memory.mcp.clients.schema import SchemaClient
async with get_client() as client:
active_project = await get_active_project(client, project, None)
schema_client = SchemaClient(client, active_project.external_id)
# Determine if target is a note identifier or entity type
# Heuristic: if target contains / or ., treat as identifier
entity_type = None
identifier = None
if target:
if "/" in target or "." in target:
identifier = target
else:
entity_type = target
report = await schema_client.validate(
entity_type=entity_type,
identifier=identifier,
)
# --- Display results ---
if report.total_notes == 0:
console.print("[yellow]No notes matched for validation.[/yellow]")
return
table = Table(title=f"Schema Validation: {entity_type or identifier or 'all'}")
table.add_column("Note", style="cyan")
table.add_column("Status", justify="center")
table.add_column("Warnings", justify="right")
table.add_column("Errors", justify="right")
for result in report.results:
if result.passed and not result.warnings:
status = "[green]pass[/green]"
elif result.passed:
status = "[yellow]warn[/yellow]"
else:
status = "[red]fail[/red]"
table.add_row(
result.note_identifier,
status,
str(len(result.warnings)),
str(len(result.errors)),
)
console.print(table)
console.print(
f"\nSummary: {report.valid_count}/{report.total_notes} valid, "
f"{report.warning_count} warnings, {report.error_count} errors"
)
# Exit with error code in strict mode if there are failures
if strict and report.error_count > 0:
raise typer.Exit(1)
@schema_app.command()
def validate(
target: Annotated[
Optional[str],
typer.Argument(help="Note path or entity type to validate"),
] = None,
project: Annotated[
Optional[str],
typer.Option(help="The project name."),
] = None,
strict: bool = typer.Option(False, "--strict", help="Exit with error on validation failures"),
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Validate notes against their schemas.
TARGET can be a note path (e.g., people/ada-lovelace.md) or an entity type
(e.g., Person). If omitted, validates all notes that have schemas.
Use --strict to exit with error code 1 if any validation errors are found.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
try:
validate_routing_flags(local, cloud)
project_name = _resolve_project_name(project)
with force_routing(local=local, cloud=cloud):
run_with_cleanup(_run_validate(target, project_name, strict))
except ValueError as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
if not isinstance(e, typer.Exit):
logger.error(f"Error during schema validate: {e}")
typer.echo(f"Error during schema validate: {e}", err=True)
raise typer.Exit(1)
raise
# --- Infer ---
async def _run_infer(
entity_type: str,
project: Optional[str] = None,
threshold: float = 0.25,
save: bool = False,
):
"""Run schema inference via the API."""
from basic_memory.mcp.clients.schema import SchemaClient
async with get_client() as client:
active_project = await get_active_project(client, project, None)
schema_client = SchemaClient(client, active_project.external_id)
report = await schema_client.infer(entity_type, threshold=threshold)
if report.notes_analyzed == 0:
console.print(f"[yellow]No notes found with type: {entity_type}[/yellow]")
return
# --- Display frequency analysis ---
console.print(
f"\n[bold]Analyzing {report.notes_analyzed} notes with type: {entity_type}...[/bold]\n"
)
table = Table(title="Field Frequencies")
table.add_column("Field", style="cyan")
table.add_column("Source")
table.add_column("Count", justify="right")
table.add_column("Percentage", justify="right")
table.add_column("Suggested")
for freq in report.field_frequencies:
pct = f"{freq.percentage:.0%}"
if freq.name in report.suggested_required:
suggested = "[green]required[/green]"
elif freq.name in report.suggested_optional:
suggested = "[yellow]optional[/yellow]"
else:
suggested = "[dim]excluded[/dim]"
table.add_row(
freq.name,
freq.source,
str(freq.count),
pct,
suggested,
)
console.print(table)
# --- Display suggested schema ---
console.print("\n[bold]Suggested schema:[/bold]")
console.print(Panel(json.dumps(report.suggested_schema, indent=2), title="Picoschema"))
if save:
console.print(
f"\n[yellow]--save not yet implemented. "
f"Copy the schema above into schema/{entity_type}.md[/yellow]"
)
@schema_app.command()
def infer(
entity_type: Annotated[
str,
typer.Argument(help="Entity type to analyze (e.g., Person, meeting)"),
],
project: Annotated[
Optional[str],
typer.Option(help="The project name."),
] = None,
threshold: float = typer.Option(
0.25, "--threshold", help="Minimum frequency for optional fields (0-1)"
),
save: bool = typer.Option(False, "--save", help="Save inferred schema to schema/ directory"),
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Infer schema from existing notes of a type.
Analyzes all notes with the given entity type and suggests a Picoschema
definition based on observation and relation frequency.
Fields present in 95%+ of notes become required. Fields above the
threshold (default 25%) become optional. Fields below threshold are excluded.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
try:
validate_routing_flags(local, cloud)
project_name = _resolve_project_name(project)
with force_routing(local=local, cloud=cloud):
run_with_cleanup(_run_infer(entity_type, project_name, threshold, save))
except ValueError as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
if not isinstance(e, typer.Exit):
logger.error(f"Error during schema infer: {e}")
typer.echo(f"Error during schema infer: {e}", err=True)
raise typer.Exit(1)
raise
# --- Diff ---
async def _run_diff(
entity_type: str,
project: Optional[str] = None,
):
"""Run schema drift detection via the API."""
from basic_memory.mcp.clients.schema import SchemaClient
async with get_client() as client:
active_project = await get_active_project(client, project, None)
schema_client = SchemaClient(client, active_project.external_id)
report = await schema_client.diff(entity_type)
has_drift = report.new_fields or report.dropped_fields or report.cardinality_changes
if not has_drift:
console.print(f"[green]No drift detected for {entity_type} schema.[/green]")
return
console.print(f"\n[bold]Schema drift detected for {entity_type}:[/bold]\n")
if report.new_fields:
console.print("[green]+ New fields (common in notes, not in schema):[/green]")
for f in report.new_fields:
console.print(f" + {f.name}: {f.percentage:.0%} of notes ({f.source})")
if report.dropped_fields:
console.print("[red]- Dropped fields (in schema, rare in notes):[/red]")
for f in report.dropped_fields:
console.print(f" - {f.name}: {f.percentage:.0%} of notes ({f.source})")
if report.cardinality_changes:
console.print("[yellow]~ Cardinality changes:[/yellow]")
for change in report.cardinality_changes:
console.print(f" ~ {change}")
@schema_app.command()
def diff(
entity_type: Annotated[
str,
typer.Argument(help="Entity type to check for drift"),
],
project: Annotated[
Optional[str],
typer.Option(help="The project name."),
] = None,
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Show drift between schema and actual usage.
Compares the existing schema definition for an entity type against
how notes of that type are actually structured. Identifies new fields,
dropped fields, and cardinality changes.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
try:
validate_routing_flags(local, cloud)
project_name = _resolve_project_name(project)
with force_routing(local=local, cloud=cloud):
run_with_cleanup(_run_diff(entity_type, project_name))
except ValueError as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
if not isinstance(e, typer.Exit):
logger.error(f"Error during schema diff: {e}")
typer.echo(f"Error during schema diff: {e}", err=True)
raise typer.Exit(1)
raise
+17 -3
View File
@@ -11,6 +11,7 @@ from rich.panel import Panel
from rich.tree import Tree
from basic_memory.cli.app import app
from basic_memory.cli.commands.routing import force_routing, validate_routing_flags
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.tools.utils import call_post
from basic_memory.schemas import SyncReportResponse
@@ -145,7 +146,7 @@ async def run_status(project: Optional[str] = None, verbose: bool = False): # p
try:
async with get_client() as client:
project_item = await get_active_project(client, project, None)
response = await call_post(client, f"{project_item.project_url}/project/status")
response = await call_post(client, f"/v2/projects/{project_item.external_id}/status")
sync_report = SyncReportResponse.model_validate(response.json())
display_changes(project_item.name, "Status", sync_report, verbose)
@@ -162,12 +163,25 @@ def status(
typer.Option(help="The project name."),
] = None,
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed file information"),
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Show sync status between files and database."""
"""Show sync status between files and database.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
from basic_memory.cli.commands.command_utils import run_with_cleanup
try:
run_with_cleanup(run_status(project, verbose)) # pragma: no cover
validate_routing_flags(local, cloud)
with force_routing(local=local, cloud=cloud):
run_with_cleanup(run_status(project, verbose)) # pragma: no cover
except ValueError as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(code=1)
except Exception as e:
logger.error(f"Error checking status: {e}")
typer.echo(f"Error checking status: {e}", err=True)
@@ -1,81 +0,0 @@
"""Telemetry commands for basic-memory CLI."""
import typer
from rich.console import Console
from rich.panel import Panel
from basic_memory.cli.app import app
from basic_memory.config import ConfigManager
console = Console()
# Create telemetry subcommand group
telemetry_app = typer.Typer(help="Manage anonymous telemetry settings")
app.add_typer(telemetry_app, name="telemetry")
@telemetry_app.command("enable")
def enable() -> None:
"""Enable anonymous telemetry.
Telemetry helps improve Basic Memory by collecting anonymous usage data.
No personal data, note content, or file paths are ever collected.
"""
config_manager = ConfigManager()
config = config_manager.config
config.telemetry_enabled = True
config_manager.save_config(config)
console.print("[green]Telemetry enabled[/green]")
console.print("[dim]Thank you for helping improve Basic Memory![/dim]")
@telemetry_app.command("disable")
def disable() -> None:
"""Disable anonymous telemetry.
You can re-enable telemetry anytime with: bm telemetry enable
"""
config_manager = ConfigManager()
config = config_manager.config
config.telemetry_enabled = False
config_manager.save_config(config)
console.print("[yellow]Telemetry disabled[/yellow]")
@telemetry_app.command("status")
def status() -> None:
"""Show current telemetry status and what's collected."""
from basic_memory.telemetry import get_install_id, TELEMETRY_DOCS_URL
config = ConfigManager().config
status_text = (
"[green]enabled[/green]" if config.telemetry_enabled else "[yellow]disabled[/yellow]"
)
console.print(f"\nTelemetry: {status_text}")
console.print(f"Install ID: [dim]{get_install_id()}[/dim]")
console.print()
what_we_collect = """
[bold]What we collect:[/bold]
- App version, Python version, OS, architecture
- Feature usage (which MCP tools and CLI commands)
- Sync statistics (entity count, duration)
- Error types (sanitized, no file paths)
[bold]What we NEVER collect:[/bold]
- Note content, file names, or paths
- Personal information
- IP addresses
"""
console.print(
Panel(
what_we_collect.strip(),
title="Telemetry Details",
border_style="blue",
expand=False,
)
)
console.print(f"[dim]Details: {TELEMETRY_DOCS_URL}[/dim]")
+417 -95
View File
@@ -1,5 +1,6 @@
"""CLI tool commands for Basic Memory."""
import json
import sys
from typing import Annotated, List, Optional
@@ -9,7 +10,15 @@ from rich import print as rprint
from basic_memory.cli.app import app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.cli.commands.routing import force_routing, validate_routing_flags
from basic_memory.config import ConfigManager
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.clients import KnowledgeClient, ResourceClient
from basic_memory.mcp.project_context import get_active_project
from basic_memory.mcp.tools.utils import call_get
from basic_memory.schemas.base import Entity, TimeFrame
from basic_memory.schemas.memory import GraphContext, MemoryUrl, memory_url_path
from basic_memory.schemas.search import SearchItemType
# Import prompts
from basic_memory.mcp.prompts.continue_conversation import (
@@ -23,14 +32,128 @@ from basic_memory.mcp.tools import read_note as mcp_read_note
from basic_memory.mcp.tools import recent_activity as mcp_recent_activity
from basic_memory.mcp.tools import search_notes as mcp_search
from basic_memory.mcp.tools import write_note as mcp_write_note
from basic_memory.schemas.base import TimeFrame
from basic_memory.schemas.memory import MemoryUrl
from basic_memory.schemas.search import SearchItemType
tool_app = typer.Typer()
app.add_typer(tool_app, name="tool", help="Access to MCP tools via CLI")
# --- JSON output helpers ---
# These async functions bypass the MCP tool (which returns formatted strings)
# and use API clients directly to return structured data for --format json.
async def _write_note_json(
title: str, content: str, folder: str, project_name: Optional[str], tags: Optional[List[str]]
) -> dict:
"""Write a note and return structured JSON metadata."""
# Use the MCP tool to create/update the entity (handles create-or-update logic)
await mcp_write_note.fn(title, content, folder, project_name, tags)
# Resolve the entity to get metadata back
async with get_client() as client:
active_project = await get_active_project(client, project_name)
knowledge_client = KnowledgeClient(client, active_project.external_id)
entity = Entity(title=title, directory=folder)
if not entity.permalink:
raise ValueError(f"Could not generate permalink for title={title}, folder={folder}")
entity_id = await knowledge_client.resolve_entity(entity.permalink)
entity = await knowledge_client.get_entity(entity_id)
return {
"title": entity.title,
"permalink": entity.permalink,
"content": content,
"file_path": entity.file_path,
}
async def _read_note_json(
identifier: str, project_name: Optional[str], page: int, page_size: int
) -> dict:
"""Read a note and return structured JSON with content and metadata."""
async with get_client() as client:
active_project = await get_active_project(client, project_name)
knowledge_client = KnowledgeClient(client, active_project.external_id)
resource_client = ResourceClient(client, active_project.external_id)
# Try direct resolution first (works for permalinks and memory URLs)
entity_path = memory_url_path(identifier)
entity_id = None
try:
entity_id = await knowledge_client.resolve_entity(entity_path)
except Exception:
logger.info(f"Direct lookup failed for '{entity_path}', trying title search")
# Fallback: title search (handles plain titles like "My Note")
if entity_id is None:
from basic_memory.mcp.tools.search import search_notes as mcp_search_tool
title_results = await mcp_search_tool.fn(
query=identifier, search_type="title", project=project_name
)
if title_results and hasattr(title_results, "results") and title_results.results:
result = title_results.results[0]
if result.permalink:
entity_id = await knowledge_client.resolve_entity(result.permalink)
if entity_id is None:
raise ValueError(f"Could not find note matching: {identifier}")
entity = await knowledge_client.get_entity(entity_id)
response = await resource_client.read(entity_id, page=page, page_size=page_size)
return {
"title": entity.title,
"permalink": entity.permalink,
"content": response.text,
"file_path": entity.file_path,
}
async def _recent_activity_json(
type: Optional[List[SearchItemType]],
depth: Optional[int],
timeframe: Optional[TimeFrame],
project_name: Optional[str] = None,
page: int = 1,
page_size: int = 50,
) -> list:
"""Get recent activity and return structured JSON list."""
async with get_client() as client:
# Build query params matching the MCP tool's logic
params: dict = {"page": page, "page_size": page_size, "max_related": 10}
if depth:
params["depth"] = depth
if timeframe:
params["timeframe"] = timeframe
if type:
params["type"] = [t.value for t in type]
active_project = await get_active_project(client, project_name)
response = await call_get(
client,
f"/v2/projects/{active_project.external_id}/memory/recent",
params=params,
)
activity_data = GraphContext.model_validate(response.json())
# Extract entity results
results = []
for result in activity_data.results:
pr = result.primary_result
if pr.type == "entity":
results.append(
{
"title": pr.title,
"permalink": pr.permalink,
"file_path": pr.file_path,
"created_at": str(pr.created_at) if pr.created_at else None,
}
)
return results
@tool_app.command()
def write_note(
title: Annotated[str, typer.Option(help="The title of the note")],
@@ -50,6 +173,11 @@ def write_note(
tags: Annotated[
Optional[List[str]], typer.Option(help="A list of tags to apply to the note")
] = None,
format: str = typer.Option("text", "--format", help="Output format: text or json"),
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Create or update a markdown note. Content can be provided as an argument or read from stdin.
@@ -57,6 +185,9 @@ def write_note(
1. Using the --content parameter
2. Piping content through stdin (if --content is not provided)
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
Examples:
# Using content parameter
@@ -77,8 +208,13 @@ def write_note(
# Reading from a file
cat document.md | basic-memory tools write-note --title "Document" --folder "docs"
# Force local routing in cloud mode
basic-memory tools write-note --title "My Note" --folder "notes" --content "..." --local
"""
try:
validate_routing_flags(local, cloud)
# If content is not provided, read from stdin
if content is None:
# Check if we're getting data from a pipe or redirect
@@ -109,8 +245,23 @@ def write_note(
# use the project name, or the default from the config
project_name = project_name or config_manager.default_project
note = run_with_cleanup(mcp_write_note.fn(title, content, folder, project_name, tags))
rprint(note)
# content is validated non-None above (stdin or --content)
assert content is not None
with force_routing(local=local, cloud=cloud):
if format == "json":
result = run_with_cleanup(
_write_note_json(title, content, folder, project_name, tags)
)
print(json.dumps(result, indent=2, ensure_ascii=True, default=str))
else:
note = run_with_cleanup(
mcp_write_note.fn(title, content, folder, project_name, tags)
)
rprint(note)
except ValueError as e:
typer.echo(f"Error: {e}", err=True)
raise typer.Exit(1)
except Exception as e: # pragma: no cover
if not isinstance(e, typer.Exit):
typer.echo(f"Error during write_note: {e}", err=True)
@@ -129,24 +280,44 @@ def read_note(
] = None,
page: int = 1,
page_size: int = 10,
format: str = typer.Option("text", "--format", help="Output format: text or json"),
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Read a markdown note from the knowledge base."""
# look for the project in the config
config_manager = ConfigManager()
project_name = None
if project is not None:
project_name, _ = config_manager.get_project(project)
if not project_name:
typer.echo(f"No project found named: {project}", err=True)
raise typer.Exit(1)
# use the project name, or the default from the config
project_name = project_name or config_manager.default_project
"""Read a markdown note from the knowledge base.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
try:
note = run_with_cleanup(mcp_read_note.fn(identifier, project_name, page, page_size))
rprint(note)
validate_routing_flags(local, cloud)
# look for the project in the config
config_manager = ConfigManager()
project_name = None
if project is not None:
project_name, _ = config_manager.get_project(project)
if not project_name:
typer.echo(f"No project found named: {project}", err=True)
raise typer.Exit(1)
# use the project name, or the default from the config
project_name = project_name or config_manager.default_project
with force_routing(local=local, cloud=cloud):
if format == "json":
result = run_with_cleanup(
_read_note_json(identifier, project_name, page, page_size)
)
print(json.dumps(result, indent=2, ensure_ascii=True, default=str))
else:
note = run_with_cleanup(mcp_read_note.fn(identifier, project_name, page, page_size))
rprint(note)
except ValueError as e:
typer.echo(f"Error: {e}", err=True)
raise typer.Exit(1)
except Exception as e: # pragma: no cover
if not isinstance(e, typer.Exit):
typer.echo(f"Error during read_note: {e}", err=True)
@@ -166,38 +337,49 @@ def build_context(
page: int = 1,
page_size: int = 10,
max_related: int = 10,
format: str = typer.Option("json", "--format", help="Output format: text or json"),
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Get context needed to continue a discussion."""
# look for the project in the config
config_manager = ConfigManager()
project_name = None
if project is not None:
project_name, _ = config_manager.get_project(project)
if not project_name:
typer.echo(f"No project found named: {project}", err=True)
raise typer.Exit(1)
# use the project name, or the default from the config
project_name = project_name or config_manager.default_project
"""Get context needed to continue a discussion.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
try:
context = run_with_cleanup(
mcp_build_context.fn(
project=project_name,
url=url,
depth=depth,
timeframe=timeframe,
page=page,
page_size=page_size,
max_related=max_related,
)
)
# Use json module for more controlled serialization
import json
validate_routing_flags(local, cloud)
# look for the project in the config
config_manager = ConfigManager()
project_name = None
if project is not None:
project_name, _ = config_manager.get_project(project)
if not project_name:
typer.echo(f"No project found named: {project}", err=True)
raise typer.Exit(1)
# use the project name, or the default from the config
project_name = project_name or config_manager.default_project
with force_routing(local=local, cloud=cloud):
context = run_with_cleanup(
mcp_build_context.fn(
project=project_name,
url=url,
depth=depth,
timeframe=timeframe,
page=page,
page_size=page_size,
max_related=max_related,
)
)
context_dict = context.model_dump(exclude_none=True)
print(json.dumps(context_dict, indent=2, ensure_ascii=True, default=str))
except ValueError as e:
typer.echo(f"Error: {e}", err=True)
raise typer.Exit(1)
except Exception as e: # pragma: no cover
if not isinstance(e, typer.Exit):
typer.echo(f"Error during build_context: {e}", err=True)
@@ -208,20 +390,60 @@ def build_context(
@tool_app.command()
def recent_activity(
type: Annotated[Optional[List[SearchItemType]], typer.Option()] = None,
project: Annotated[
Optional[str],
typer.Option(help="The project to use. If not provided, the default project will be used."),
] = None,
depth: Optional[int] = 1,
timeframe: Optional[TimeFrame] = "7d",
page: int = typer.Option(1, "--page", help="Page number for pagination (JSON format)"),
page_size: int = typer.Option(
50, "--page-size", help="Number of results per page (JSON format)"
),
format: str = typer.Option("text", "--format", help="Output format: text or json"),
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Get recent activity across the knowledge base."""
"""Get recent activity across the knowledge base.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
try:
result = run_with_cleanup(
mcp_recent_activity.fn(
type=type, # pyright: ignore [reportArgumentType]
depth=depth,
timeframe=timeframe,
)
)
# The tool now returns a formatted string directly
print(result)
validate_routing_flags(local, cloud)
# Resolve project from config for JSON mode
config_manager = ConfigManager()
project_name = None
if project is not None:
project_name, _ = config_manager.get_project(project)
if not project_name:
typer.echo(f"No project found named: {project}", err=True)
raise typer.Exit(1)
project_name = project_name or config_manager.default_project
with force_routing(local=local, cloud=cloud):
if format == "json":
result = run_with_cleanup(
_recent_activity_json(type, depth, timeframe, project_name, page, page_size)
)
print(json.dumps(result, indent=2, ensure_ascii=True, default=str))
else:
result = run_with_cleanup(
mcp_recent_activity.fn(
type=type, # pyright: ignore [reportArgumentType]
depth=depth,
timeframe=timeframe,
project=project_name,
)
)
# The tool returns a formatted string directly
print(result)
except ValueError as e:
typer.echo(f"Error: {e}", err=True)
raise typer.Exit(1)
except Exception as e: # pragma: no cover
if not isinstance(e, typer.Exit):
typer.echo(f"Error during recent_activity: {e}", err=True)
@@ -231,9 +453,14 @@ def recent_activity(
@tool_app.command("search-notes")
def search_notes(
query: str,
query: Annotated[
Optional[str],
typer.Argument(help="Search query string (optional when using metadata filters)"),
] = "",
permalink: Annotated[bool, typer.Option("--permalink", help="Search permalink values")] = False,
title: Annotated[bool, typer.Option("--title", help="Search title values")] = False,
vector: Annotated[bool, typer.Option("--vector", help="Use vector retrieval")] = False,
hybrid: Annotated[bool, typer.Option("--hybrid", help="Use hybrid retrieval")] = False,
project: Annotated[
Optional[str],
typer.Option(
@@ -244,56 +471,135 @@ def search_notes(
Optional[str],
typer.Option("--after_date", help="Search results after date, eg. '2d', '1 week'"),
] = None,
tags: Annotated[
Optional[List[str]],
typer.Option("--tag", help="Filter by frontmatter tag (repeatable)"),
] = None,
status: Annotated[
Optional[str],
typer.Option("--status", help="Filter by frontmatter status"),
] = None,
note_types: Annotated[
Optional[List[str]],
typer.Option("--type", help="Filter by frontmatter type (repeatable)"),
] = None,
entity_types: Annotated[
Optional[List[str]],
typer.Option(
"--entity-type",
help="Filter by search item type: entity, observation, relation (repeatable)",
),
] = None,
meta: Annotated[
Optional[List[str]],
typer.Option("--meta", help="Filter by frontmatter key=value (repeatable)"),
] = None,
filter_json: Annotated[
Optional[str],
typer.Option("--filter", help="JSON metadata filter (advanced)"),
] = None,
page: int = 1,
page_size: int = 10,
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Search across all content in the knowledge base."""
# look for the project in the config
config_manager = ConfigManager()
project_name = None
if project is not None:
project_name, _ = config_manager.get_project(project)
if not project_name:
typer.echo(f"No project found named: {project}", err=True)
raise typer.Exit(1)
# use the project name, or the default from the config
project_name = project_name or config_manager.default_project
if permalink and title: # pragma: no cover
print("Cannot search both permalink and title")
raise typer.Abort()
"""Search across all content in the knowledge base.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
try:
if permalink and title: # pragma: no cover
validate_routing_flags(local, cloud)
# look for the project in the config
config_manager = ConfigManager()
project_name = None
if project is not None:
project_name, _ = config_manager.get_project(project)
if not project_name:
typer.echo(f"No project found named: {project}", err=True)
raise typer.Exit(1)
# use the project name, or the default from the config
project_name = project_name or config_manager.default_project
mode_flags = [permalink, title, vector, hybrid]
if sum(1 for enabled in mode_flags if enabled) > 1: # pragma: no cover
typer.echo(
"Use either --permalink or --title, not both. Exiting.",
"Use only one mode flag: --permalink, --title, --vector, or --hybrid. Exiting.",
err=True,
)
raise typer.Exit(1)
# set search type
search_type = ("permalink" if permalink else None,)
search_type = ("permalink_match" if permalink and "*" in query else None,)
search_type = ("title" if title else None,)
search_type = "text" if search_type is None else search_type
# Build metadata filters from --filter and --meta
metadata_filters = {}
if filter_json:
try:
metadata_filters = json.loads(filter_json)
if not isinstance(metadata_filters, dict):
raise ValueError("Metadata filter JSON must be an object")
except json.JSONDecodeError as e:
typer.echo(f"Invalid JSON for --filter: {e}", err=True)
raise typer.Exit(1)
results = run_with_cleanup(
mcp_search.fn(
query,
project_name,
search_type=search_type,
page=page,
after_date=after_date,
page_size=page_size,
if meta:
for item in meta:
if "=" not in item:
typer.echo(
f"Invalid --meta entry '{item}'. Use key=value format.",
err=True,
)
raise typer.Exit(1)
key, value = item.split("=", 1)
key = key.strip()
if not key:
typer.echo(f"Invalid --meta entry '{item}'.", err=True)
raise typer.Exit(1)
metadata_filters[key] = value
if not metadata_filters:
metadata_filters = None
# set search type
search_type = "text"
if permalink:
search_type = "permalink"
if query and "*" in query:
search_type = "permalink"
if title:
search_type = "title"
if vector:
search_type = "vector"
if hybrid:
search_type = "hybrid"
with force_routing(local=local, cloud=cloud):
results = run_with_cleanup(
mcp_search.fn(
query or "",
project_name,
search_type=search_type,
page=page,
after_date=after_date,
page_size=page_size,
types=note_types,
entity_types=entity_types,
metadata_filters=metadata_filters,
tags=tags,
status=status,
)
)
)
# Use json module for more controlled serialization
import json
if isinstance(results, str):
print(results)
raise typer.Exit(1)
results_dict = results.model_dump(exclude_none=True)
print(json.dumps(results_dict, indent=2, ensure_ascii=True, default=str))
except ValueError as e:
typer.echo(f"Error: {e}", err=True)
raise typer.Exit(1)
except Exception as e: # pragma: no cover
if not isinstance(e, typer.Exit):
logger.exception("Error during search", e)
@@ -308,12 +614,28 @@ def continue_conversation(
timeframe: Annotated[
Optional[str], typer.Option(help="How far back to look for activity")
] = None,
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Prompt to continue a previous conversation or work session."""
"""Prompt to continue a previous conversation or work session.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
try:
# Prompt functions return formatted strings directly
session = run_with_cleanup(mcp_continue_conversation.fn(topic=topic, timeframe=timeframe)) # type: ignore
validate_routing_flags(local, cloud)
with force_routing(local=local, cloud=cloud):
# Prompt functions return formatted strings directly
session = run_with_cleanup(
mcp_continue_conversation.fn(topic=topic, timeframe=timeframe) # type: ignore[arg-type]
)
rprint(session)
except ValueError as e:
typer.echo(f"Error: {e}", err=True)
raise typer.Exit(1)
except Exception as e: # pragma: no cover
if not isinstance(e, typer.Exit):
logger.exception("Error continuing conversation", e)
+97
View File
@@ -0,0 +1,97 @@
"""Watch command - run file watcher as a standalone long-running process."""
import asyncio
import os
import signal
import sys
from typing import Optional
import typer
from loguru import logger
from basic_memory import db
from basic_memory.cli.app import app
from basic_memory.cli.container import get_container
from basic_memory.config import ConfigManager
from basic_memory.services.initialization import initialize_app
from basic_memory.sync.coordinator import SyncCoordinator
async def run_watch(project: Optional[str] = None) -> None:
"""Run the file watcher as a long-running process.
This is the async core of the watch command. It:
1. Initializes the app (DB migrations + project reconciliation)
2. Validates and sets project constraint if --project given
3. Creates a SyncCoordinator with quiet=False for Rich console output
4. Blocks until SIGINT/SIGTERM, then shuts down cleanly
"""
container = get_container()
config = container.config
# --- Initialization ---
# Wrapped in try/finally so DB resources are cleaned up on all exit paths,
# including early exits from invalid --project names.
await initialize_app(config)
sync_coordinator = None
try:
# --- Project constraint ---
if project:
config_manager = ConfigManager()
project_name, _ = config_manager.get_project(project)
if not project_name:
typer.echo(f"No project found named: {project}", err=True)
raise typer.Exit(1)
os.environ["BASIC_MEMORY_MCP_PROJECT"] = project_name
logger.info(f"Watch constrained to project: {project_name}")
# --- Sync coordinator ---
# quiet=False so file change events are printed to the terminal
sync_coordinator = SyncCoordinator(config=config, should_sync=True, quiet=False)
# --- Signal handling ---
shutdown_event = asyncio.Event()
def _signal_handler() -> None:
logger.info("Shutdown signal received")
shutdown_event.set()
loop = asyncio.get_running_loop()
# Windows ProactorEventLoop does not support add_signal_handler;
# fall back to the stdlib signal module which works cross-platform.
try:
for sig in (signal.SIGINT, signal.SIGTERM):
loop.add_signal_handler(sig, _signal_handler)
except NotImplementedError:
for sig in (signal.SIGINT, signal.SIGTERM):
signal.signal(sig, lambda _signum, _frame: _signal_handler())
# --- Run ---
await sync_coordinator.start()
logger.info("Watch service running, press Ctrl+C to stop")
await shutdown_event.wait()
finally:
if sync_coordinator is not None:
await sync_coordinator.stop()
await db.shutdown_db()
logger.info("Watch service stopped")
@app.command()
def watch(
project: Optional[str] = typer.Option(None, help="Restrict watcher to a single project"),
) -> None:
"""Run file watcher as a long-running process (no MCP server).
Watches for file changes in project directories and syncs them to the
database. Useful for running Basic Memory sync alongside external tools
that don't use the MCP server.
"""
# On Windows, use SelectorEventLoop to avoid ProactorEventLoop cleanup issues
if sys.platform == "win32": # pragma: no cover
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
asyncio.run(run_watch(project=project))
+25 -17
View File
@@ -1,25 +1,33 @@
"""Main CLI entry point for basic-memory.""" # pragma: no cover
import sys
import warnings
from basic_memory.cli.app import app # pragma: no cover
# Register commands
from basic_memory.cli.commands import ( # noqa: F401 # pragma: no cover
cloud,
db,
import_chatgpt,
import_claude_conversations,
import_claude_projects,
import_memory_json,
mcp,
project,
status,
telemetry,
tool,
)
def _version_only_invocation(argv: list[str]) -> bool:
# Trigger: invocation is exactly `bm --version` or `bm -v`
# Why: avoid importing command modules on the hot version path
# Outcome: eager version callback exits quickly with minimal startup work
return len(argv) == 1 and argv[0] in {"--version", "-v"}
# Re-apply warning filter AFTER all imports
# (authlib adds a DeprecationWarning filter that overrides ours)
import warnings # pragma: no cover
if not _version_only_invocation(sys.argv[1:]):
# Register commands only when not short-circuiting for --version
from basic_memory.cli.commands import ( # noqa: F401 # pragma: no cover
cloud,
db,
doctor,
import_chatgpt,
import_claude_conversations,
import_claude_projects,
import_memory_json,
mcp,
project,
schema,
status,
tool,
)
warnings.filterwarnings("ignore") # pragma: no cover
+84
View File
@@ -0,0 +1,84 @@
"""Cloud promo messaging for CLI entrypoint."""
import os
import sys
from collections.abc import Callable
import typer
from basic_memory.config import ConfigManager
CLOUD_PROMO_VERSION = "2026-02-06"
OSS_DISCOUNT_CODE = "{{OSS_DISCOUNT_CODE}}"
def _promos_disabled_by_env() -> bool:
"""Check environment-level kill switch for promo output."""
value = os.getenv("BASIC_MEMORY_NO_PROMOS", "").strip().lower()
return value in {"1", "true", "yes"}
def _is_interactive_session() -> bool:
"""Return whether stdin/stdout are interactive terminals."""
return sys.stdin.isatty() and sys.stdout.isatty()
def _build_first_run_message() -> str:
"""Build first-run cloud promo copy."""
return (
"Basic Memory initialized (local mode).\n"
"Cloud is optional and keeps your workflow local-first.\n"
"Cloud adds cross-device sync + mobile/web access.\n"
f"OSS discount: {OSS_DISCOUNT_CODE} (20% off for 3 months).\n"
"Run `bm cloud login` to enable."
)
def _build_version_message() -> str:
"""Build cloud promo copy shown after promo-version bumps."""
return (
"New in Basic Memory Cloud: cross-device sync + mobile/web access.\n"
f"OSS discount: {OSS_DISCOUNT_CODE} (20% off for 3 months).\n"
"Run `bm cloud login` to enable."
)
def maybe_show_cloud_promo(
invoked_subcommand: str | None,
*,
config_manager: ConfigManager | None = None,
is_interactive: bool | None = None,
echo: Callable[[str], None] = typer.echo,
) -> None:
"""Show cloud promo copy when discovery gates are satisfied."""
manager = config_manager or ConfigManager()
config = manager.load_config()
interactive = _is_interactive_session() if is_interactive is None else is_interactive
# Trigger: environment-level promo suppression or non-interactive execution.
# Why: avoid polluting scripts/CI output and support a hard opt-out.
# Outcome: skip all promo copy for this invocation.
if _promos_disabled_by_env() or not interactive:
return
# Trigger: command context where cloud promo is not actionable.
# Why: mcp/stdin protocol and root help flows should stay noise-free.
# Outcome: command continues without promo messaging.
if invoked_subcommand in {None, "mcp"}:
return
if config.cloud_mode_enabled or config.cloud_promo_opt_out:
return
show_first_run = not config.cloud_promo_first_run_shown
show_version_notice = config.cloud_promo_last_version_shown != CLOUD_PROMO_VERSION
if not show_first_run and not show_version_notice:
return
message = _build_first_run_message() if show_first_run else _build_version_message()
echo(message)
config.cloud_promo_first_run_shown = True
config.cloud_promo_last_version_shown = CLOUD_PROMO_VERSION
manager.save_config(config)
+93 -12
View File
@@ -24,6 +24,13 @@ WATCH_STATUS_JSON = "watch-status.json"
Environment = Literal["test", "dev", "user"]
class ProjectMode(str, Enum):
"""Per-project routing mode."""
LOCAL = "local"
CLOUD = "cloud"
class DatabaseBackend(str, Enum):
"""Supported database backends."""
@@ -37,6 +44,7 @@ class ProjectConfig:
name: str
home: Path
mode: ProjectMode = ProjectMode.LOCAL
@property
def project(self):
@@ -81,7 +89,7 @@ class BasicMemoryConfig(BaseSettings):
description="Name of the default project to use",
)
default_project_mode: bool = Field(
default=False,
default=True,
description="When True, MCP tools automatically use default_project when no project parameter is specified. Enables simplified UX for single-project workflows.",
)
@@ -99,6 +107,34 @@ class BasicMemoryConfig(BaseSettings):
description="Database connection URL. For Postgres, use postgresql+asyncpg://user:pass@host:port/db. If not set, SQLite will use default path.",
)
# Semantic search configuration
semantic_search_enabled: bool = Field(
default=False,
description="Enable semantic search (vector/hybrid retrieval). Works on both SQLite and Postgres backends. Requires semantic extras.",
)
semantic_embedding_provider: str = Field(
default="fastembed",
description="Embedding provider for local semantic indexing/search.",
)
semantic_embedding_model: str = Field(
default="bge-small-en-v1.5",
description="Embedding model identifier used by the local provider.",
)
semantic_embedding_dimensions: int | None = Field(
default=None,
description="Embedding vector dimensions. Auto-detected from provider if not set (384 for FastEmbed, 1536 for OpenAI).",
)
semantic_embedding_batch_size: int = Field(
default=64,
description="Batch size for embedding generation.",
gt=0,
)
semantic_vector_k: int = Field(
default=100,
description="Vector candidate count for vector and hybrid retrieval.",
gt=0,
)
# Database connection pool configuration (Postgres only)
db_pool_size: int = Field(
default=20,
@@ -160,6 +196,11 @@ class BasicMemoryConfig(BaseSettings):
description="Disable automatic permalink generation in frontmatter. When enabled, new notes won't have permalinks added and sync won't update permalinks. Existing permalinks will still work for reading.",
)
permalinks_include_project: bool = Field(
default=True,
description="When True, generated permalinks are prefixed with the project slug (e.g., 'specs/search'). Existing permalinks remain unchanged unless explicitly updated.",
)
skip_initialization_sync: bool = Field(
default=False,
description="Skip expensive initialization synchronization. Useful for cloud/stateless deployments where project reconciliation is not needed.",
@@ -221,15 +262,29 @@ class BasicMemoryConfig(BaseSettings):
description="Cloud project sync configuration mapping project names to their local paths and sync state",
)
# Telemetry configuration (Homebrew-style opt-out)
telemetry_enabled: bool = Field(
default=True,
description="Send anonymous usage statistics to help improve Basic Memory. Disable with: bm telemetry disable",
cloud_promo_opt_out: bool = Field(
default=False,
description="Disable CLI cloud promo messages when true.",
)
telemetry_notice_shown: bool = Field(
cloud_promo_first_run_shown: bool = Field(
default=False,
description="Whether the one-time telemetry notice has been shown to the user",
description="Tracks whether the first-run cloud promo message has been shown.",
)
cloud_promo_last_version_shown: Optional[str] = Field(
default=None,
description="Most recent cloud promo version shown in CLI.",
)
cloud_api_key: Optional[str] = Field(
default=None,
description="API key for cloud access (bmc_ prefixed). Account-level, not per-project.",
)
project_modes: Dict[str, ProjectMode] = Field(
default_factory=dict,
description="Per-project routing mode. Projects not listed default to LOCAL.",
)
@property
@@ -241,7 +296,7 @@ class BasicMemoryConfig(BaseSettings):
- BASIC_MEMORY_ENV environment variable is "test"
- PYTEST_CURRENT_TEST environment variable is set (pytest is running)
Used to disable features like telemetry and file watchers during tests.
Used to disable features like file watchers during tests.
"""
return (
self.env == "test"
@@ -265,6 +320,21 @@ class BasicMemoryConfig(BaseSettings):
# Fall back to config file value
return self.cloud_mode
def get_project_mode(self, project_name: str) -> ProjectMode:
"""Get the routing mode for a project.
Returns the per-project mode if set, otherwise LOCAL.
"""
return self.project_modes.get(project_name, ProjectMode.LOCAL)
def set_project_mode(self, project_name: str, mode: ProjectMode) -> None:
"""Set the routing mode for a project."""
if mode == ProjectMode.LOCAL:
# Remove from dict to keep config clean — LOCAL is the default
self.project_modes.pop(project_name, None)
else:
self.project_modes[project_name] = mode
@classmethod
def for_cloud_tenant(
cls,
@@ -332,8 +402,11 @@ class BasicMemoryConfig(BaseSettings):
This is the single database that will store all knowledge data
across all projects.
Uses BASIC_MEMORY_CONFIG_DIR when set so each process/worktree can
isolate both config and database state.
"""
database_path = Path.home() / DATA_DIR_NAME / APP_DATABASE_NAME
database_path = self.data_dir_path / APP_DATABASE_NAME
if not database_path.exists(): # pragma: no cover
database_path.parent.mkdir(parents=True, exist_ok=True)
database_path.touch()
@@ -355,7 +428,10 @@ class BasicMemoryConfig(BaseSettings):
@property
def project_list(self) -> List[ProjectConfig]: # pragma: no cover
"""Get all configured projects as ProjectConfig objects."""
return [ProjectConfig(name=name, home=Path(path)) for name, path in self.projects.items()]
return [
ProjectConfig(name=name, home=Path(path), mode=self.get_project_mode(name))
for name, path in self.projects.items()
]
@model_validator(mode="after")
def ensure_project_paths_exists(self) -> "BasicMemoryConfig": # pragma: no cover
@@ -379,8 +455,13 @@ class BasicMemoryConfig(BaseSettings):
return self
@property
def data_dir_path(self):
return Path.home() / DATA_DIR_NAME
def data_dir_path(self) -> Path:
"""Get app state directory for config and default SQLite database."""
if config_dir := os.getenv("BASIC_MEMORY_CONFIG_DIR"):
return Path(config_dir)
home = os.getenv("HOME", Path.home())
return Path(home) / DATA_DIR_NAME
# Module-level cache for configuration
+4
View File
@@ -91,6 +91,8 @@ from basic_memory.deps.services import (
FileServiceV2Dep,
get_file_service_v2_external,
FileServiceV2ExternalDep,
get_task_scheduler,
TaskSchedulerDep,
get_search_service,
SearchServiceDep,
get_search_service_v2,
@@ -227,6 +229,8 @@ __all__ = [
"FileServiceV2Dep",
"get_file_service_v2_external",
"FileServiceV2ExternalDep",
"get_task_scheduler",
"TaskSchedulerDep",
"get_search_service",
"SearchServiceDep",
"get_search_service_v2",
+72 -12
View File
@@ -41,7 +41,12 @@ async def get_chatgpt_importer(
file_service: FileServiceDep,
) -> ChatGPTImporter:
"""Create ChatGPTImporter with dependencies."""
return ChatGPTImporter(project_config.home, markdown_processor, file_service)
return ChatGPTImporter(
project_config.home,
markdown_processor,
file_service,
project_name=project_config.name,
)
ChatGPTImporterDep = Annotated[ChatGPTImporter, Depends(get_chatgpt_importer)]
@@ -53,7 +58,12 @@ async def get_chatgpt_importer_v2( # pragma: no cover
file_service: FileServiceV2Dep,
) -> ChatGPTImporter:
"""Create ChatGPTImporter with v2 dependencies."""
return ChatGPTImporter(project_config.home, markdown_processor, file_service)
return ChatGPTImporter(
project_config.home,
markdown_processor,
file_service,
project_name=project_config.name,
)
ChatGPTImporterV2Dep = Annotated[ChatGPTImporter, Depends(get_chatgpt_importer_v2)]
@@ -65,7 +75,12 @@ async def get_chatgpt_importer_v2_external(
file_service: FileServiceV2ExternalDep,
) -> ChatGPTImporter:
"""Create ChatGPTImporter with v2 external_id dependencies."""
return ChatGPTImporter(project_config.home, markdown_processor, file_service)
return ChatGPTImporter(
project_config.home,
markdown_processor,
file_service,
project_name=project_config.name,
)
ChatGPTImporterV2ExternalDep = Annotated[ChatGPTImporter, Depends(get_chatgpt_importer_v2_external)]
@@ -80,7 +95,12 @@ async def get_claude_conversations_importer(
file_service: FileServiceDep,
) -> ClaudeConversationsImporter:
"""Create ClaudeConversationsImporter with dependencies."""
return ClaudeConversationsImporter(project_config.home, markdown_processor, file_service)
return ClaudeConversationsImporter(
project_config.home,
markdown_processor,
file_service,
project_name=project_config.name,
)
ClaudeConversationsImporterDep = Annotated[
@@ -94,7 +114,12 @@ async def get_claude_conversations_importer_v2( # pragma: no cover
file_service: FileServiceV2Dep,
) -> ClaudeConversationsImporter:
"""Create ClaudeConversationsImporter with v2 dependencies."""
return ClaudeConversationsImporter(project_config.home, markdown_processor, file_service)
return ClaudeConversationsImporter(
project_config.home,
markdown_processor,
file_service,
project_name=project_config.name,
)
ClaudeConversationsImporterV2Dep = Annotated[
@@ -108,7 +133,12 @@ async def get_claude_conversations_importer_v2_external(
file_service: FileServiceV2ExternalDep,
) -> ClaudeConversationsImporter:
"""Create ClaudeConversationsImporter with v2 external_id dependencies."""
return ClaudeConversationsImporter(project_config.home, markdown_processor, file_service)
return ClaudeConversationsImporter(
project_config.home,
markdown_processor,
file_service,
project_name=project_config.name,
)
ClaudeConversationsImporterV2ExternalDep = Annotated[
@@ -125,7 +155,12 @@ async def get_claude_projects_importer(
file_service: FileServiceDep,
) -> ClaudeProjectsImporter:
"""Create ClaudeProjectsImporter with dependencies."""
return ClaudeProjectsImporter(project_config.home, markdown_processor, file_service)
return ClaudeProjectsImporter(
project_config.home,
markdown_processor,
file_service,
project_name=project_config.name,
)
ClaudeProjectsImporterDep = Annotated[ClaudeProjectsImporter, Depends(get_claude_projects_importer)]
@@ -137,7 +172,12 @@ async def get_claude_projects_importer_v2( # pragma: no cover
file_service: FileServiceV2Dep,
) -> ClaudeProjectsImporter:
"""Create ClaudeProjectsImporter with v2 dependencies."""
return ClaudeProjectsImporter(project_config.home, markdown_processor, file_service)
return ClaudeProjectsImporter(
project_config.home,
markdown_processor,
file_service,
project_name=project_config.name,
)
ClaudeProjectsImporterV2Dep = Annotated[
@@ -151,7 +191,12 @@ async def get_claude_projects_importer_v2_external(
file_service: FileServiceV2ExternalDep,
) -> ClaudeProjectsImporter:
"""Create ClaudeProjectsImporter with v2 external_id dependencies."""
return ClaudeProjectsImporter(project_config.home, markdown_processor, file_service)
return ClaudeProjectsImporter(
project_config.home,
markdown_processor,
file_service,
project_name=project_config.name,
)
ClaudeProjectsImporterV2ExternalDep = Annotated[
@@ -168,7 +213,12 @@ async def get_memory_json_importer(
file_service: FileServiceDep,
) -> MemoryJsonImporter:
"""Create MemoryJsonImporter with dependencies."""
return MemoryJsonImporter(project_config.home, markdown_processor, file_service)
return MemoryJsonImporter(
project_config.home,
markdown_processor,
file_service,
project_name=project_config.name,
)
MemoryJsonImporterDep = Annotated[MemoryJsonImporter, Depends(get_memory_json_importer)]
@@ -180,7 +230,12 @@ async def get_memory_json_importer_v2( # pragma: no cover
file_service: FileServiceV2Dep,
) -> MemoryJsonImporter:
"""Create MemoryJsonImporter with v2 dependencies."""
return MemoryJsonImporter(project_config.home, markdown_processor, file_service)
return MemoryJsonImporter(
project_config.home,
markdown_processor,
file_service,
project_name=project_config.name,
)
MemoryJsonImporterV2Dep = Annotated[MemoryJsonImporter, Depends(get_memory_json_importer_v2)]
@@ -192,7 +247,12 @@ async def get_memory_json_importer_v2_external(
file_service: FileServiceV2ExternalDep,
) -> MemoryJsonImporter:
"""Create MemoryJsonImporter with v2 external_id dependencies."""
return MemoryJsonImporter(project_config.home, markdown_processor, file_service)
return MemoryJsonImporter(
project_config.home,
markdown_processor,
file_service,
project_name=project_config.name,
)
MemoryJsonImporterV2ExternalDep = Annotated[
+7 -3
View File
@@ -13,6 +13,7 @@ from typing import Annotated
from fastapi import Depends
from basic_memory.deps.config import AppConfigDep
from basic_memory.deps.db import SessionMakerDep
from basic_memory.deps.projects import (
ProjectIdDep,
@@ -147,13 +148,14 @@ RelationRepositoryV2ExternalDep = Annotated[
async def get_search_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
app_config: AppConfigDep,
) -> SearchRepository:
"""Create a backend-specific SearchRepository instance for the current project.
Uses factory function to return SQLiteSearchRepository or PostgresSearchRepository
based on database backend configuration.
"""
return create_search_repository(session_maker, project_id=project_id)
return create_search_repository(session_maker, project_id=project_id, app_config=app_config)
SearchRepositoryDep = Annotated[SearchRepository, Depends(get_search_repository)]
@@ -162,9 +164,10 @@ SearchRepositoryDep = Annotated[SearchRepository, Depends(get_search_repository)
async def get_search_repository_v2( # pragma: no cover
session_maker: SessionMakerDep,
project_id: ProjectIdPathDep,
app_config: AppConfigDep,
) -> SearchRepository:
"""Create a SearchRepository instance for v2 API."""
return create_search_repository(session_maker, project_id=project_id)
return create_search_repository(session_maker, project_id=project_id, app_config=app_config)
SearchRepositoryV2Dep = Annotated[SearchRepository, Depends(get_search_repository_v2)]
@@ -173,9 +176,10 @@ SearchRepositoryV2Dep = Annotated[SearchRepository, Depends(get_search_repositor
async def get_search_repository_v2_external(
session_maker: SessionMakerDep,
project_id: ProjectExternalIdPathDep,
app_config: AppConfigDep,
) -> SearchRepository:
"""Create a SearchRepository instance for v2 API (uses external_id)."""
return create_search_repository(session_maker, project_id=project_id)
return create_search_repository(session_maker, project_id=project_id, app_config=app_config)
SearchRepositoryV2ExternalDep = Annotated[

Some files were not shown because too many files have changed in this diff Show More