Compare commits

...

63 Commits

Author SHA1 Message Date
phernandez 46b372c3e1 Add MCP output_format json mode across memory tools
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-18 19:30:49 -06:00
phernandez 0a36256f8a Merge branch 'main' of github.com:basicmachines-co/basic-memory 2026-02-18 10:50:58 -06:00
phernandez 36e67e6eec ci: split and speed up PR test matrix
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-18 10:41:02 -06:00
Paul Hernandez f2683291e4 feat: add workspace selection flow for MCP and CLI (#576)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-17 22:48:06 -06:00
phernandez 8c05a9ec80 fix: stabilize semantic search defaults, FTS fallback, and postgres project sync
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-17 15:36:46 -06:00
phernandez a6d8d4c0f6 Add ensure_frontmatter_on_sync with precedence warning
Add a new config option to enforce frontmatter on markdown sync when missing, writing derived title/type/permalink and updating in-memory metadata before upsert. Add startup warning when this option is combined with disable_permalinks to make precedence explicit. Add config/sync/initialization tests for the new behavior, and stabilize project list CLI integration assertions by forcing a wide terminal in tests to avoid Rich truncation.

Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-16 22:08:09 -06:00
phernandez 0239f4abb4 Simplify local/cloud routing and clarify project targeting
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-16 21:42:25 -06:00
Paul Hernandez 9259a7eb59 feat: min-similarity override, edit-note CLI, and strip-frontmatter (#571)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 17:56:00 -06:00
Paul Hernandez 55d675e278 feat: min_similarity override, cloud promo improvements (#570)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 16:44:02 -06:00
Drew Cain 6afe4fd0cc feat: expose external_id in EntityResponse and link resolver (#569)
Signed-off-by: Drew Cain <groksrc@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 20:33:37 -06:00
phernandez 113d1b6f1b formatting fix
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-14 09:56:37 -06:00
Paul Hernandez 545804f194 feat: Add project-prefixed permalinks and memory URL routing (#544)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-13 17:45:50 -06:00
Paul Hernandez 8bc03d1357 feat(mcp): add MCP UI variants and TUI output (#545)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-13 15:21:48 -06:00
Paul Hernandez f6e0a5b5bb fix: Speed up bm --version startup (#534)
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-13 11:12:52 -06:00
Paul Hernandez 7624a20d8d feat: isolate default sqlite db by config dir (#567)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-13 11:11:13 -06:00
Paul Hernandez d84708ca7f feat: add per-project local/cloud routing with API key auth (#555)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-13 09:52:50 -06:00
Paul Hernandez 1428d18de1 fix: make semantic search dependencies optional extras (#566)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 21:46:26 -06:00
Paul Hernandez 312662f382 feat: Add cloud discovery touchpoints to CLI and MCP (#546)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-12 21:01:45 -06:00
Paul Hernandez ed9487708e feat: enable default_project_mode by default (#560)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 20:34:46 -06:00
Paul Hernandez 8df88e4d02 feat: add basic-memory watch CLI command (#559)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 20:10:09 -06:00
Paul Hernandez 07778790d3 feat: add semantic vector search for SQLite and Postgres (#550)
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: bm-clawd <clawd@basicmemory.com>
Co-authored-by: bm-clawd <clawd@basicmemory.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 15:44:47 -06:00
Paul Hernandez b609c4e531 fix: use global --header for Tigris consistency on all rclone transactions (#564)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 15:43:05 -06:00
Paul Hernandez f1a065bce3 chore: Release/v0.18.2 (#563)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 15:41:38 -06:00
phernandez 2b94d9a278 fix formatting for tigris headers
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-12 13:22:31 -06:00
Paul Hernandez 344e651693 fix: use VIRTUAL instead of STORED columns in SQLite migration (#562)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-11 22:24:51 -06:00
Paul Hernandez c97733d785 feat: Schema system for Basic Memory (#549)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-11 15:04:42 -06:00
phernandez 00537272c6 chore: update version to 0.18.1 for v0.18.1 release 2026-02-11 14:28:52 -06:00
phernandez b057912452 docs: add CHANGELOG entry for v0.18.1
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-11 14:24:26 -06:00
Paul Hernandez 8489a3d37e fix: add X-Tigris-Consistent headers to all rclone commands (#558)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-11 14:22:49 -06:00
Paul Hernandez a47c9c021f feat: add --format json to CLI tool commands (#552)
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: bm-clawd <clawd@basicmemory.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: bm-clawd <clawd@basicmemory.com>
2026-02-08 14:56:29 -06:00
phernandez c46d7a6833 fix: add POST legacy compat routes for v0.18.0 CLI
The v0.18.0 CLI (Homebrew) calls POST /projects/projects for project add
and POST /projects/config/sync for config sync. The previous legacy compat
fix (a0e754b) only added GET for list_projects but missed POST endpoints.

This caused 405 Method Not Allowed when running `bm project add` in cloud mode.

🔗 Logfire trace: trace_id=019c398cb257f040c1255821b6d5e385

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-07 13:41:41 -06:00
Paul Hernandez 343a6e118b fix: Handle EntityCreationError as conflict (#541)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-03 22:28:58 -06:00
phernandez a0e754b7ae fix: restore legacy /projects/projects endpoint for older CLI versions
Older versions of basic-memory CLI (v0.17.4 and earlier) call
GET /projects/projects to list projects. This endpoint was removed
when we migrated to v2 routers.

Add explicit route at /projects/projects (without trailing slash) to
avoid 307 redirects that the cloud proxy doesn't follow.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-02 22:26:48 -06:00
Paul Hernandez 24ca5f6804 fix: recent_activity prompt defaults (#533)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-02 19:22:33 -06:00
Paul Hernandez f1d50c2ba7 feat: Support tag: query shorthand in search (#535)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-02 19:22:16 -06:00
Paul Hernandez 8072449a78 chore: Add fast feedback loop tooling (#538)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-01 23:04:59 -06:00
phernandez 45d3f58e4d Merge branch 'main' of github.com:basicmachines-co/basic-memory 2026-02-01 21:17:39 -06:00
phernandez d9c8923148 fix ci runner for tests
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-01 21:16:10 -06:00
phernandez 15bd6b95ef fix ci runner i
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-01 21:10:26 -06:00
phernandez 0715dcff3d run ubuntu tests on depot
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-31 23:25:46 -06:00
Paul Hernandez 009e84926d fix: stabilize metadata filters on postgres (#536)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-31 22:31:00 -06:00
phernandez 8838571509 Add metadata filter tests and fix fast write external_id
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-31 15:27:05 -06:00
Paul Hernandez 530cbac73f feat: fast edit entities, refactors for webui, enhance search (#532)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-31 15:16:52 -06:00
Paul Hernandez e3ced49d9d fix: prevent spurious 'metadata: {}' in frontmatter output (#530)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-31 12:17:10 -06:00
Drew Cain 8f962fdd87 chore: update version to 0.18.0 for v0.18.0 release 2026-01-29 22:41:42 -06:00
Drew Cain fbb497f6dc docs: add CHANGELOG entry for v0.18.0 2026-01-29 22:41:14 -06:00
Drew Cain 0023e736ab feat: add context-aware wiki link resolution with source_path support (#527)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-28 19:11:41 -06:00
Paul Hernandez 0b2080114b feat: add directory support to move_note and delete_note tools (#518)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-26 19:52:10 -06:00
Paul Hernandez 8730067f3a feat: Feature/517 local mcp cloud mode (#522)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-26 19:51:49 -06:00
Drew Cain e14ba92631 fix: resolve MCP prompt rendering errors (#524)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 19:14:41 -06:00
phernandez 9d98892570 chore: update version to 0.17.9 for v0.17.9 release 2026-01-24 12:55:58 -06:00
phernandez 3be4495723 docs: add CHANGELOG entry for v0.17.9
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-24 12:55:27 -06:00
Paul Hernandez 17c0e0a29b fix: check config default_project only in local mode for remove_project (#523)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 12:53:44 -06:00
phernandez 7ebf16a95d chore: update version to 0.17.8 for v0.17.8 release 2026-01-24 11:43:44 -06:00
phernandez c05075f8d4 docs: add CHANGELOG entry for v0.17.8
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-24 11:43:21 -06:00
phernandez 4cef9281ca docs: add CHANGELOG entry for v0.17.7
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-24 11:39:41 -06:00
Paul Hernandez 6888effef2 fix: correct get_default_project() query to check for True instead of not NULL (#521)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 11:37:05 -06:00
Paul Hernandez 38616c345d fix: read default project from database in cloud mode (#520)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 10:44:04 -06:00
phernandez f3c1aa895c fix links in README.md to remove smithery badge
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-22 12:58:03 -06:00
phernandez d978aba09b fix links in README.md to remove glama.ai
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-22 12:56:33 -06:00
phernandez 2aaee734c9 fix links in README.md to point to basicmemory.com instead of basicmachines.co
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-22 12:53:18 -06:00
Paul Hernandez 369ad37b3d feat: add SPEC-29 Phase 3 bucket snapshot CLI commands (#476)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-20 17:55:01 -06:00
Drew Cain 4e5f701d22 Fix server.json runtimeArguments format
- Use proper Argument object format instead of plain strings
- Add .mcpregistry tokens to .gitignore

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-01-19 12:23:29 -06:00
461 changed files with 45590 additions and 10311 deletions
+1 -3
View File
@@ -1,5 +1,3 @@
{
"enabledPlugins": {
"basic-memory@basicmachines": true
}
"enabledPlugins": {}
}
+199 -45
View File
@@ -1,25 +1,70 @@
name: Tests
concurrency:
group: bm-ci-${{ github.workflow }}-${{ github.repository }}-${{ github.head_ref || github.ref }}
cancel-in-progress: true
on:
push:
branches: [ "main" ]
pull_request:
branches: [ "main" ]
# pull_request_target runs on the BASE of the PR, not the merge result.
# It has write permissions and access to secrets.
# It's useful for PRs from forks or automated PRs but requires careful use for security reasons.
# See: https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request_target
pull_request_target:
branches: [ "main" ]
jobs:
test-sqlite:
name: Test SQLite (${{ matrix.os }}, Python ${{ matrix.python-version }})
static-checks:
name: Static Checks (Python 3.12)
timeout-minutes: 20
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python 3.12
uses: actions/setup-python@v4
with:
python-version: "3.12"
cache: "pip"
- name: Install uv
run: |
pip install uv
- uses: extractions/setup-just@v3
- name: Create virtual env
run: |
uv venv
- name: Install dependencies
run: |
uv pip install -e ".[dev]"
- name: Run type checks
run: |
just typecheck
- name: Run linting
run: |
just lint
test-sqlite-unit:
name: Test SQLite Unit (${{ matrix.os }}, Python ${{ matrix.python-version }})
timeout-minutes: 30
needs: [static-checks]
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest]
python-version: [ "3.12", "3.13", "3.14" ]
include:
- os: ubuntu-latest
python-version: "3.12"
- os: ubuntu-latest
python-version: "3.13"
- os: ubuntu-latest
python-version: "3.14"
- os: windows-latest
python-version: "3.12"
runs-on: ${{ matrix.os }}
steps:
@@ -37,17 +82,7 @@ jobs:
run: |
pip install uv
- name: Install just (Linux/macOS)
if: runner.os != 'Windows'
run: |
curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to /usr/local/bin
- name: Install just (Windows)
if: runner.os == 'Windows'
run: |
# Install just using Chocolatey (pre-installed on GitHub Actions Windows runners)
choco install just --yes
shell: pwsh
- uses: extractions/setup-just@v3
- name: Create virtual env
run: |
@@ -55,23 +90,63 @@ jobs:
- name: Install dependencies
run: |
uv pip install -e .[dev]
uv pip install -e ".[dev]"
- name: Run type checks
- name: Run tests (SQLite Unit)
run: |
just typecheck
just test-unit-sqlite
- name: Run linting
test-sqlite-integration:
name: Test SQLite Integration (${{ matrix.os }}, Python ${{ matrix.python-version }})
timeout-minutes: 45
needs: [static-checks]
strategy:
fail-fast: false
matrix:
include:
- os: ubuntu-latest
python-version: "3.12"
- os: ubuntu-latest
python-version: "3.13"
- os: ubuntu-latest
python-version: "3.14"
- os: windows-latest
python-version: "3.12"
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
- name: Install uv
run: |
just lint
pip install uv
- name: Run tests (SQLite)
- uses: extractions/setup-just@v3
- name: Create virtual env
run: |
uv pip install pytest pytest-cov
just test-sqlite
uv venv
test-postgres:
name: Test Postgres (Python ${{ matrix.python-version }})
- name: Install dependencies
run: |
uv pip install -e ".[dev]"
- name: Run tests (SQLite Integration)
run: |
just test-int-sqlite
test-postgres-unit:
name: Test Postgres Unit (Python ${{ matrix.python-version }})
timeout-minutes: 30
needs: [static-checks]
strategy:
fail-fast: false
matrix:
@@ -95,9 +170,7 @@ jobs:
run: |
pip install uv
- name: Install just
run: |
curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to /usr/local/bin
- uses: extractions/setup-just@v3
- name: Create virtual env
run: |
@@ -105,15 +178,57 @@ jobs:
- name: Install dependencies
run: |
uv pip install -e .[dev]
uv pip install -e ".[dev]"
- name: Run tests (Postgres via testcontainers)
- name: Run tests (Postgres Unit)
run: |
uv pip install pytest pytest-cov
just test-postgres
just test-unit-postgres
coverage:
name: Coverage Summary (combined, Python 3.12)
test-postgres-integration:
name: Test Postgres Integration (Python ${{ matrix.python-version }})
timeout-minutes: 45
needs: [static-checks]
strategy:
fail-fast: false
matrix:
python-version: [ "3.12", "3.13", "3.14" ]
runs-on: ubuntu-latest
# Note: No services section needed - testcontainers handles Postgres in Docker
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
- name: Install uv
run: |
pip install uv
- uses: extractions/setup-just@v3
- name: Create virtual env
run: |
uv venv
- name: Install dependencies
run: |
uv pip install -e ".[dev]"
- name: Run tests (Postgres Integration)
run: |
just test-int-postgres
test-semantic:
name: Test Semantic (Python 3.12)
timeout-minutes: 45
needs: [static-checks]
runs-on: ubuntu-latest
steps:
@@ -131,9 +246,7 @@ jobs:
run: |
pip install uv
- name: Install just
run: |
curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to /usr/local/bin
- uses: extractions/setup-just@v3
- name: Create virtual env
run: |
@@ -141,11 +254,52 @@ jobs:
- name: Install dependencies
run: |
uv pip install -e .[dev]
uv pip install -e ".[dev,semantic]"
- name: Run tests (Semantic)
run: |
just test-semantic
coverage:
name: Coverage Summary (combined, Python 3.12)
timeout-minutes: 60
needs:
- static-checks
- test-sqlite-unit
- test-sqlite-integration
- test-postgres-unit
- test-postgres-integration
- test-semantic
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python 3.12
uses: actions/setup-python@v4
with:
python-version: "3.12"
cache: "pip"
- name: Install uv
run: |
pip install uv
- uses: extractions/setup-just@v3
- name: Create virtual env
run: |
uv venv
- name: Install dependencies
run: |
uv pip install -e ".[dev,semantic]"
- name: Run combined coverage (SQLite + Postgres)
run: |
uv pip install pytest pytest-cov
just coverage
- name: Add coverage report to job summary
+4
View File
@@ -1,6 +1,7 @@
*.py[cod]
__pycache__/
.pytest_cache/
.testmondata*
.coverage
htmlcov/
@@ -54,3 +55,6 @@ ENV/
claude-output
**/.claude/settings.local.json
.mcp.json
.mcpregistry_*
/.testmondata
.benchmarks/
+445
View File
@@ -0,0 +1,445 @@
# AGENTS.md - Basic Memory Project Guide
## Project Overview
Basic Memory is a local-first knowledge management system built on the Model Context Protocol (MCP). It enables
bidirectional communication between LLMs (like Claude) and markdown files, creating a personal knowledge graph that can
be traversed using links between documents.
## CODEBASE DEVELOPMENT
### Project information
See the [README.md](README.md) file for a project overview.
### Build and Test Commands
- Install: `just install` or `pip install -e ".[dev]"`
- Run all tests (SQLite + Postgres): `just test`
- Run all tests against SQLite: `just test-sqlite`
- Run all tests against Postgres: `just test-postgres` (uses testcontainers)
- Run unit tests (SQLite): `just test-unit-sqlite`
- Run unit tests (Postgres): `just test-unit-postgres`
- Run integration tests (SQLite): `just test-int-sqlite`
- Run integration tests (Postgres): `just test-int-postgres`
- Run impacted tests: `just testmon` (pytest-testmon)
- Run MCP smoke test: `just test-smoke`
- Fast local loop: `just fast-check`
- Local consistency check: `just doctor`
- Generate HTML coverage: `just coverage`
- Single test: `pytest tests/path/to/test_file.py::test_function_name`
- Run benchmarks: `pytest test-int/test_sync_performance_benchmark.py -v -m "benchmark and not slow"`
- Lint: `just lint` or `ruff check . --fix`
- Type check: `just typecheck` or `uv run pyright`
- Format: `just format` or `uv run ruff format .`
- Run all code checks: `just check` (runs lint, format, typecheck, test)
- Create db migration: `just migration "Your migration message"`
- Run development MCP Inspector: `just run-inspector`
**Note:** Project requires Python 3.12+ (uses type parameter syntax and `type` aliases introduced in 3.12)
**Postgres Testing:** Uses [testcontainers](https://testcontainers-python.readthedocs.io/) which automatically spins up a Postgres instance in Docker. No manual database setup required - just have Docker running.
**Doctor Note:** `just doctor` runs with a temporary HOME/config so it won't touch your local Basic Memory settings. It leaves temp dirs in `/tmp` (safe to ignore or remove).
**Testmon Note:** When no files have changed, `just testmon` may collect 0 tests. That's expected and means no impacted tests were detected.
### Code/Test/Verify Loop (fast path)
1) **Code:** make changes.
2) **Test:** `just fast-check` (lint/format/typecheck + impacted tests + MCP smoke).
3) **Verify:** `just doctor` (end-to-end file ↔ DB loop in a temp project).
4) **Full gate (when needed):** `just test` or `just check` for SQLite + Postgres.
If testmon is “cold,” the first run may be long. Subsequent runs get much faster.
### Test Structure
- `tests/` - Unit tests for individual components (mocked, fast)
- `test-int/` - Integration tests for real-world scenarios (no mocks, realistic)
- Both directories are covered by unified coverage reporting
- Benchmark tests in `test-int/` are marked with `@pytest.mark.benchmark`
- Slow tests are marked with `@pytest.mark.slow`
- Smoke tests are marked with `@pytest.mark.smoke`
### Code Style Guidelines
- Line length: 100 characters max
- Python 3.12+ with full type annotations (uses type parameters and type aliases)
- Format with ruff (consistent styling)
- Import order: standard lib, third-party, local imports
- Naming: snake_case for functions/variables, PascalCase for classes
- Prefer async patterns with SQLAlchemy 2.0
- Use Pydantic v2 for data validation and schemas
- CLI uses Typer for command structure
- API uses FastAPI for endpoints
- Follow the repository pattern for data access
- Tools communicate to api routers via the httpx ASGI client (in process)
### Code Change Guidelines
- **Full file read before edits**: Before editing any file, read it in full first to ensure complete context; partial reads lead to corrupted edits
- **Minimize diffs**: Prefer the smallest change that satisfies the request. Avoid unrelated refactors or style rewrites unless necessary for correctness
- **No speculative getattr**: Never use `getattr(obj, "attr", default)` when unsure about attribute names. Check the class definition or source code first
- **Fail fast**: Write code with fail-fast logic by default. Do not swallow exceptions with errors or warnings
- **No fallback logic**: Do not add fallback logic unless explicitly told to and agreed with the user
- **No guessing**: Do not say "The issue is..." before you actually know what the issue is. Investigate first.
### Literate Programming Style
Code should tell a story. Comments must explain the "why" and narrative flow, not just the "what".
**Section Headers:**
For files with multiple phases of logic, add section headers so the control flow reads like chapters:
```python
# --- Authentication ---
# ... auth logic ...
# --- Data Validation ---
# ... validation logic ...
# --- Business Logic ---
# ... core logic ...
```
**Decision Point Comments:**
For conditionals that materially change behavior (gates, fallbacks, retries, feature flags), add comments with:
- **Trigger**: what condition causes this branch
- **Why**: the rationale (cost, correctness, UX, determinism)
- **Outcome**: what changes downstream
```python
# Trigger: project has no active sync watcher
# Why: avoid duplicate file system watchers consuming resources
# Outcome: starts new watcher, registers in active_watchers dict
if project_id not in active_watchers:
start_watcher(project_id)
```
**Constraint Comments:**
If code exists because of a constraint (async requirements, rate limits, schema compatibility), explain the constraint near the code:
```python
# SQLite requires WAL mode for concurrent read/write access
connection.execute("PRAGMA journal_mode=WAL")
```
**What NOT to Comment:**
Avoid comments that restate obvious code:
```python
# Bad - restates code
counter += 1 # increment counter
# Good - explains why
counter += 1 # track retries for backoff calculation
```
### Codebase Architecture
See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for detailed architecture documentation.
**Directory Structure:**
- `/alembic` - Alembic db migrations
- `/api` - FastAPI REST endpoints + `container.py` composition root
- `/cli` - Typer CLI + `container.py` composition root
- `/deps` - Feature-scoped FastAPI dependencies (config, db, projects, repositories, services, importers)
- `/importers` - Import functionality for Claude, ChatGPT, and other sources
- `/markdown` - Markdown parsing and processing
- `/mcp` - MCP server + `container.py` composition root + `clients/` typed API clients
- `/models` - SQLAlchemy ORM models
- `/repository` - Data access layer
- `/schemas` - Pydantic models for validation
- `/services` - Business logic layer
- `/sync` - File synchronization services + `coordinator.py` for lifecycle management
**Composition Roots:**
Each entrypoint (API, MCP, CLI) has a composition root that:
- Reads `ConfigManager` (the only place that reads global config)
- Resolves runtime mode via `RuntimeMode` enum (TEST > CLOUD > LOCAL)
- Provides dependencies to downstream code explicitly
**Typed API Clients (MCP):**
MCP tools use typed clients in `mcp/clients/` to communicate with the API:
- `KnowledgeClient` - Entity CRUD operations
- `SearchClient` - Search operations
- `MemoryClient` - Context building
- `DirectoryClient` - Directory listing
- `ResourceClient` - Resource reading
- `ProjectClient` - Project management
Flow: MCP Tool → Typed Client → HTTP API → Router → Service → Repository
### Development Notes
- MCP tools are defined in src/basic_memory/mcp/tools/
- MCP prompts are defined in src/basic_memory/mcp/prompts/
- MCP tools should be atomic, composable operations
- Use `textwrap.dedent()` for multi-line string formatting in prompts and tools
- MCP Prompts are used to invoke tools and format content with instructions for an LLM
- Schema changes require Alembic migrations
- SQLite is used for indexing and full text search, files are source of truth
- Testing uses pytest with asyncio support (strict mode)
- Unit tests (`tests/`) use mocks when necessary; integration tests (`test-int/`) use real implementations
- By default, tests run against SQLite (fast, no Docker needed)
- Set `BASIC_MEMORY_TEST_POSTGRES=1` to run against Postgres (uses testcontainers - Docker required)
- Each test runs in a standalone environment with isolated database and tmp_path directory
- CI runs SQLite and Postgres tests in parallel for faster feedback
- Performance benchmarks are in `test-int/test_sync_performance_benchmark.py`
- Use pytest markers: `@pytest.mark.benchmark` for benchmarks, `@pytest.mark.slow` for slow tests
- **Coverage must stay at 100%**: Write tests for new code. Only use `# pragma: no cover` when tests would require excessive mocking (e.g., TYPE_CHECKING blocks, error handlers that need failure injection, runtime-mode-dependent code paths)
### Async Client Pattern (Important!)
**MCP tools use `get_project_client()` for per-project routing:**
```python
from basic_memory.mcp.project_context import get_project_client
@mcp.tool()
async def my_tool(project: str | None = None, context: Context | None = None):
async with get_project_client(project, context) as (client, active_project):
# client is routed based on project's mode (local ASGI or cloud HTTP)
response = await call_get(client, "/path")
return response
```
**CLI commands and non-project-scoped code use `get_client()` directly:**
```python
from basic_memory.mcp.async_client import get_client
async def my_cli_command():
async with get_client() as client:
response = await call_get(client, "/path")
return response
# Per-project routing (when project name is known):
async with get_client(project_name="research") as client:
...
```
**Do NOT use:**
-`from basic_memory.mcp.async_client import client` (deprecated module-level client)
- ❌ Manual auth header management
-`inject_auth_header()` (deleted)
- ❌ Separate `get_client()` + `get_active_project()` in MCP tools (use `get_project_client()` instead)
**Key principles:**
- Auth happens at client creation, not per-request
- Proper resource management via context managers
- Per-project routing: each project can be LOCAL or CLOUD independently
- Cloud projects use API key (`cloud_api_key` in config) as Bearer token
- Routing priority: factory injection > force-local > per-project cloud > global cloud > local ASGI
- Factory pattern enables dependency injection for cloud consolidation
**For cloud app integration:**
```python
from basic_memory.mcp import async_client
# Set custom factory before importing tools
async_client.set_client_factory(your_custom_factory)
```
See SPEC-16 for full context manager refactor details.
## BASIC MEMORY PRODUCT USAGE
### Knowledge Structure
- Entity: Any concept, document, or idea represented as a markdown file
- Observation: A categorized fact about an entity (`- [category] content`)
- Relation: A directional link between entities (`- relation_type [[Target]]`)
- Frontmatter: YAML metadata at the top of markdown files
- Knowledge representation follows precise markdown format:
- Observations with [category] prefixes
- Relations with WikiLinks [[Entity]]
- Frontmatter with metadata
### Basic Memory Commands
**Local Commands:**
- Check sync status: `basic-memory status`
- Doctor check (file <-> DB loop): `basic-memory doctor`
- Import from Claude: `basic-memory import claude conversations`
- Import from ChatGPT: `basic-memory import chatgpt`
- Import from Memory JSON: `basic-memory import memory-json`
- Tool access: `basic-memory tool` (provides CLI access to MCP tools)
- Continue: `basic-memory tool continue-conversation --topic="search"`
**Project Management:**
- List projects: `basic-memory project list`
- Add project: `basic-memory project add "name" ~/path`
- Project info: `basic-memory project info`
- Set cloud mode: `basic-memory project set-cloud "name"`
- Set local mode: `basic-memory project set-local "name"`
- One-way sync (local -> cloud): `basic-memory project sync`
- Bidirectional sync: `basic-memory project bisync`
- Integrity check: `basic-memory project check`
**Cloud Commands (requires subscription):**
- Authenticate (global): `basic-memory cloud login`
- Logout (global): `basic-memory cloud logout`
- Check cloud status: `basic-memory cloud status`
- Setup cloud sync: `basic-memory cloud setup`
- Save API key: `basic-memory cloud set-key bmc_...`
- Create API key: `basic-memory cloud create-key "name"`
- Manage snapshots: `basic-memory cloud snapshot [create|list|delete|show|browse]`
- Restore from snapshot: `basic-memory cloud restore <path> --snapshot <id>`
### MCP Capabilities
- Basic Memory exposes these MCP tools to LLMs:
**Content Management:**
- `write_note(title, content, directory, tags)` - Create/update markdown notes with semantic observations and relations
- `read_note(identifier, page, page_size)` - Read notes by title, permalink, or memory:// URL with knowledge graph awareness
- `read_content(path)` - Read raw file content (text, images, binaries) without knowledge graph processing
- `view_note(identifier, page, page_size)` - View notes as formatted artifacts for better readability
- `edit_note(identifier, operation, content)` - Edit notes incrementally (append, prepend, find/replace, replace_section)
- `move_note(identifier, destination_path, is_directory)` - Move notes or directories to new locations, updating database and maintaining links
- `delete_note(identifier, is_directory)` - Delete notes or directories from the knowledge base
**Knowledge Graph Navigation:**
- `build_context(url, depth, timeframe)` - Navigate the knowledge graph via memory:// URLs for conversation continuity
- `recent_activity(type, depth, timeframe)` - Get recently updated information with specified timeframe (e.g., "1d", "1 week")
- `list_directory(dir_name, depth, file_name_glob)` - Browse directory contents with filtering and depth control
**Search & Discovery:**
- `search_notes(query, page, page_size, search_type, types, entity_types, after_date)` - Full-text search across all content with advanced filtering options
**Project Management:**
- `list_memory_projects()` - List all available projects with their status
- `create_memory_project(project_name, project_path, set_default)` - Create new Basic Memory projects
- `delete_project(project_name)` - Delete a project from configuration
**Visualization:**
- `canvas(nodes, edges, title, directory)` - Generate Obsidian canvas files for knowledge graph visualization
**ChatGPT-Compatible Tools:**
- `search(query)` - Search across knowledge base (OpenAI actions compatible)
- `fetch(id)` - Fetch full content of a search result document
- MCP Prompts for better AI interaction:
- `ai_assistant_guide()` - Guidance on effectively using Basic Memory tools for AI assistants
- `continue_conversation(topic, timeframe)` - Continue previous conversations with relevant historical context
- `search(query, after_date)` - Search with detailed, formatted results for better context understanding
- `recent_activity(timeframe)` - View recently changed items with formatted output
### Cloud Features (v0.15.0+)
Basic Memory now supports cloud synchronization and storage (requires active subscription):
**Authentication:**
- JWT-based authentication with subscription validation
- Secure session management with token refresh
- Support for multiple cloud projects
**Bidirectional Sync:**
- rclone bisync integration for two-way synchronization
- Conflict resolution and integrity verification
- Real-time sync with change detection
- Mount/unmount cloud storage for direct file access
**Cloud Project Management:**
- Create and manage projects in the cloud
- Toggle between local and cloud modes
- Per-project sync configuration
- Subscription-based access control
**Security & Performance:**
- Removed .env file loading for improved security
- .gitignore integration (respects gitignored files)
- WAL mode for SQLite performance
- Background relation resolution (non-blocking startup)
- API performance optimizations (SPEC-11)
**Per-Project Cloud Routing:**
Individual projects can be routed through the cloud while others stay local, using an API key:
```bash
# Save API key and set project to cloud mode
basic-memory cloud set-key bmc_abc123...
basic-memory project set-cloud research # route through cloud
basic-memory project set-local research # revert to local
```
MCP tools use `get_project_client()` which automatically routes based on the project's mode. Cloud projects use the `cloud_api_key` from config as Bearer token.
**CLI Routing Flags (Global Cloud Mode):**
When global cloud mode is enabled, CLI commands route to the cloud API by default. Use `--local` and `--cloud` flags to override:
```bash
# Force local routing (ignore cloud mode)
basic-memory status --local
basic-memory project list --local
# Force cloud routing (when cloud mode is disabled)
basic-memory status --cloud
basic-memory project info my-project --cloud
```
Key behaviors:
- The local MCP server (`basic-memory mcp`) automatically uses local routing
- This allows simultaneous use of local Claude Desktop and cloud-based clients
- Some commands (like `project default`, `project sync-config`, `project move`) require `--local` in cloud mode since they modify local configuration
- Environment variable `BASIC_MEMORY_FORCE_LOCAL=true` forces local routing globally
- Per-project cloud routing via API key works independently of global cloud mode
## AI-Human Collaborative Development
Basic Memory emerged from and enables a new kind of development process that combines human and AI capabilities. Instead
of using AI just for code generation, we've developed a true collaborative workflow:
1. AI (LLM) writes initial implementation based on specifications and context
2. Human reviews, runs tests, and commits code with any necessary adjustments
3. Knowledge persists across conversations using Basic Memory's knowledge graph
4. Development continues seamlessly across different AI sessions with consistent context
5. Results improve through iterative collaboration and shared understanding
This approach has allowed us to tackle more complex challenges and build a more robust system than either humans or AI
could achieve independently.
**Problem-Solving Guidance:**
- If a solution isn't working after reasonable effort, suggest alternative approaches
- Don't persist with a problematic library or pattern when better alternatives exist
- Example: When py-pglite caused cascading test failures, switching to testcontainers-postgres was the right call
## GitHub Integration
Basic Memory has taken AI-Human collaboration to the next level by integrating Claude directly into the development workflow through GitHub:
### GitHub MCP Tools
Using the GitHub Model Context Protocol server, Claude can now:
- **Repository Management**:
- View repository files and structure
- Read file contents
- Create new branches
- Create and update files
- **Issue Management**:
- Create new issues
- Comment on existing issues
- Close and update issues
- Search across issues
- **Pull Request Workflow**:
- Create pull requests
- Review code changes
- Add comments to PRs
This integration enables Claude to participate as a full team member in the development process, not just as a code generation tool. Claude's GitHub account ([bm-claudeai](https://github.com/bm-claudeai)) is a member of the Basic Machines organization with direct contributor access to the codebase.
### Collaborative Development Process
With GitHub integration, the development workflow includes:
1. **Direct code review** - Claude can analyze PRs and provide detailed feedback
2. **Contribution tracking** - All of Claude's contributions are properly attributed in the Git history
3. **Branch management** - Claude can create feature branches for implementations
4. **Documentation maintenance** - Claude can keep documentation updated as the code evolves
5. **Code Commits**: ALWAYS sign off commits with `git commit -s`
This level of integration represents a new paradigm in AI-human collaboration, where the AI assistant becomes a full-fledged team member rather than just a tool for generating code snippets.
+151
View File
@@ -1,5 +1,156 @@
# CHANGELOG
## Unreleased
### Features
- Add `--strip-frontmatter` to `basic-memory tool read-note`
- Default behavior is unchanged: `content` still includes raw markdown with frontmatter.
- With `--strip-frontmatter`, both text and JSON modes return body-only markdown content.
- JSON output now includes an additive `frontmatter` field with parsed YAML metadata (or `null`
when no valid opening frontmatter block exists).
## v0.18.3 (2026-02-12)
### Bug Fixes
- Use global `--header` flag for Tigris consistency on all rclone transactions
([`7fcf587`](https://github.com/basicmachines-co/basic-memory/commit/7fcf587))
- `--header-download` / `--header-upload` only apply to GET/PUT requests, missing S3
ListObjectsV2 calls that bisync issues first. Non-US users saw stale edge-cached metadata.
- `--header` applies to ALL HTTP transactions (list, download, upload), fixing bisync for
users outside the Tigris origin region.
## v0.18.2 (2026-02-11)
### Bug Fixes
- **#562**: Use VIRTUAL instead of STORED columns in SQLite migration
([`344e651`](https://github.com/basicmachines-co/basic-memory/commit/344e651))
- Fixes compatibility issue with SQLite STORED generated columns
## v0.18.1 (2026-02-11)
### Features
- **#552**: Add `--format json` to CLI tool commands
([`a47c9c0`](https://github.com/basicmachines-co/basic-memory/commit/a47c9c0))
- CLI tool commands now support `--format json` for machine-readable output
- **#535**: Support `tag:` query shorthand in search
([`f1d50c2`](https://github.com/basicmachines-co/basic-memory/commit/f1d50c2))
- Use `tag:mytag` as a convenient shorthand in search queries
- **#532**: Fast edit entities, refactors for webui, enhanced search
([`530cbac`](https://github.com/basicmachines-co/basic-memory/commit/530cbac))
- Performance improvements for entity editing and search operations
### Bug Fixes
- **#558**: Add X-Tigris-Consistent headers to all rclone commands
([`8489a3d`](https://github.com/basicmachines-co/basic-memory/commit/8489a3d))
- Ensures consistent reads from Tigris object storage during sync
- **#541**: Handle EntityCreationError as conflict
([`343a6e1`](https://github.com/basicmachines-co/basic-memory/commit/343a6e1))
- **#536**: Stabilize metadata filters on Postgres
([`009e849`](https://github.com/basicmachines-co/basic-memory/commit/009e849))
- **#533**: Fix recent_activity prompt defaults
([`24ca5f6`](https://github.com/basicmachines-co/basic-memory/commit/24ca5f6))
- **#530**: Prevent spurious `metadata: {}` in frontmatter output
([`e3ced49`](https://github.com/basicmachines-co/basic-memory/commit/e3ced49))
- Add POST legacy compat routes for v0.18.0 CLI
([`c46d7a6`](https://github.com/basicmachines-co/basic-memory/commit/c46d7a6))
- Restore legacy `/projects/projects` endpoint for older CLI versions
([`a0e754b`](https://github.com/basicmachines-co/basic-memory/commit/a0e754b))
### Internal
- **#538**: Add fast feedback loop tooling (`just fast-check`, `just doctor`, `just testmon`)
([`8072449`](https://github.com/basicmachines-co/basic-memory/commit/8072449))
## v0.18.0 (2026-01-28)
### Features
- **#527**: Add context-aware wiki link resolution with source_path support
([`0023e73`](https://github.com/basicmachines-co/basic-memory/commit/0023e73))
- Add `source_path` parameter to `resolve_link()` for context-aware resolution
- Relative path resolution: `[[nested/note]]` from `folder/file.md``folder/nested/note.md`
- Proximity-based resolution for duplicate titles (prefers notes in same folder)
- Strict mode to disable fuzzy search fallback for wiki links
- **#518**: Add directory support to move_note and delete_note tools
([`0b20801`](https://github.com/basicmachines-co/basic-memory/commit/0b20801))
- Add `is_directory` parameter to `move_note` and `delete_note` MCP tools
- New `POST /move-directory` and delete directory API endpoints
- Rename `folder``directory` parameter across codebase for consistency
- **#522**: Local MCP cloud mode routing
([`8730067`](https://github.com/basicmachines-co/basic-memory/commit/8730067))
- Add `--local` and `--cloud` CLI routing flags
- Local MCP server (`basic-memory mcp`) automatically uses local routing
- Enables simultaneous use of local Claude Desktop and cloud-based clients
- Environment variable `BASIC_MEMORY_FORCE_LOCAL=true` for global override
### Bug Fixes
- **#524**: Fix MCP prompt rendering errors
([`e14ba92`](https://github.com/basicmachines-co/basic-memory/commit/e14ba92))
- Fix "Error rendering prompt recent_activity" error
- Change `TimeFrame` to `str` in prompt type annotations for FastMCP compatibility
## v0.17.9 (2026-01-24)
### Bug Fixes
- **#523**: Fix `remove_project()` checking stale config in cloud mode
([`17c0e0a`](https://github.com/basicmachines-co/basic-memory/commit/17c0e0a))
- In cloud mode, only check database `is_default` field (source of truth)
- Config file can become stale when users set default project via v2 API
## v0.17.8 (2026-01-24)
### Bug Fixes
- **#521**: Fix `get_default_project()` returning multiple results
([`6888eff`](https://github.com/basicmachines-co/basic-memory/commit/6888eff))
- Query incorrectly matched any project with non-NULL `is_default` (both True and False)
- Now correctly checks for `is_default=True` only
## v0.17.7 (2026-01-24)
### Features
- **#476**: Add SPEC-29 Phase 3 bucket snapshot CLI commands
([`369ad37`](https://github.com/basicmachines-co/basic-memory/commit/369ad37))
- New `basic-memory cloud snapshot` commands for managing cloud snapshots
- Commands: `create`, `list`, `delete`, `show`, `browse`
- **#515**: Add MCP registry publication files
([`7a502e6`](https://github.com/basicmachines-co/basic-memory/commit/7a502e6))
### Bug Fixes
- **#520**: Read default project from database in cloud mode
([`38616c3`](https://github.com/basicmachines-co/basic-memory/commit/38616c3))
- **#513**: Ensure external_id is set on entity creation
([`c7835a9`](https://github.com/basicmachines-co/basic-memory/commit/c7835a9))
### Internal
- **#514**: Remove OpenPanel telemetry
([`85835ae`](https://github.com/basicmachines-co/basic-memory/commit/85835ae))
- Update README links to point to basicmemory.com
([`2aaee73`](https://github.com/basicmachines-co/basic-memory/commit/2aaee73))
## v0.17.6 (2026-01-17)
### Bug Fixes
-367
View File
@@ -1,367 +0,0 @@
# CLAUDE.md - Basic Memory Project Guide
## Project Overview
Basic Memory is a local-first knowledge management system built on the Model Context Protocol (MCP). It enables
bidirectional communication between LLMs (like Claude) and markdown files, creating a personal knowledge graph that can
be traversed using links between documents.
## CODEBASE DEVELOPMENT
### Project information
See the [README.md](README.md) file for a project overview.
### Build and Test Commands
- Install: `just install` or `pip install -e ".[dev]"`
- Run all tests (SQLite + Postgres): `just test`
- Run all tests against SQLite: `just test-sqlite`
- Run all tests against Postgres: `just test-postgres` (uses testcontainers)
- Run unit tests (SQLite): `just test-unit-sqlite`
- Run unit tests (Postgres): `just test-unit-postgres`
- Run integration tests (SQLite): `just test-int-sqlite`
- Run integration tests (Postgres): `just test-int-postgres`
- Generate HTML coverage: `just coverage`
- Single test: `pytest tests/path/to/test_file.py::test_function_name`
- Run benchmarks: `pytest test-int/test_sync_performance_benchmark.py -v -m "benchmark and not slow"`
- Lint: `just lint` or `ruff check . --fix`
- Type check: `just typecheck` or `uv run pyright`
- Format: `just format` or `uv run ruff format .`
- Run all code checks: `just check` (runs lint, format, typecheck, test)
- Create db migration: `just migration "Your migration message"`
- Run development MCP Inspector: `just run-inspector`
**Note:** Project requires Python 3.12+ (uses type parameter syntax and `type` aliases introduced in 3.12)
**Postgres Testing:** Uses [testcontainers](https://testcontainers-python.readthedocs.io/) which automatically spins up a Postgres instance in Docker. No manual database setup required - just have Docker running.
### Test Structure
- `tests/` - Unit tests for individual components (mocked, fast)
- `test-int/` - Integration tests for real-world scenarios (no mocks, realistic)
- Both directories are covered by unified coverage reporting
- Benchmark tests in `test-int/` are marked with `@pytest.mark.benchmark`
- Slow tests are marked with `@pytest.mark.slow`
### Code Style Guidelines
- Line length: 100 characters max
- Python 3.12+ with full type annotations (uses type parameters and type aliases)
- Format with ruff (consistent styling)
- Import order: standard lib, third-party, local imports
- Naming: snake_case for functions/variables, PascalCase for classes
- Prefer async patterns with SQLAlchemy 2.0
- Use Pydantic v2 for data validation and schemas
- CLI uses Typer for command structure
- API uses FastAPI for endpoints
- Follow the repository pattern for data access
- Tools communicate to api routers via the httpx ASGI client (in process)
### Code Change Guidelines
- **Full file read before edits**: Before editing any file, read it in full first to ensure complete context; partial reads lead to corrupted edits
- **Minimize diffs**: Prefer the smallest change that satisfies the request. Avoid unrelated refactors or style rewrites unless necessary for correctness
- **No speculative getattr**: Never use `getattr(obj, "attr", default)` when unsure about attribute names. Check the class definition or source code first
- **Fail fast**: Write code with fail-fast logic by default. Do not swallow exceptions with errors or warnings
- **No fallback logic**: Do not add fallback logic unless explicitly told to and agreed with the user
- **No guessing**: Do not say "The issue is..." before you actually know what the issue is. Investigate first.
### Literate Programming Style
Code should tell a story. Comments must explain the "why" and narrative flow, not just the "what".
**Section Headers:**
For files with multiple phases of logic, add section headers so the control flow reads like chapters:
```python
# --- Authentication ---
# ... auth logic ...
# --- Data Validation ---
# ... validation logic ...
# --- Business Logic ---
# ... core logic ...
```
**Decision Point Comments:**
For conditionals that materially change behavior (gates, fallbacks, retries, feature flags), add comments with:
- **Trigger**: what condition causes this branch
- **Why**: the rationale (cost, correctness, UX, determinism)
- **Outcome**: what changes downstream
```python
# Trigger: project has no active sync watcher
# Why: avoid duplicate file system watchers consuming resources
# Outcome: starts new watcher, registers in active_watchers dict
if project_id not in active_watchers:
start_watcher(project_id)
```
**Constraint Comments:**
If code exists because of a constraint (async requirements, rate limits, schema compatibility), explain the constraint near the code:
```python
# SQLite requires WAL mode for concurrent read/write access
connection.execute("PRAGMA journal_mode=WAL")
```
**What NOT to Comment:**
Avoid comments that restate obvious code:
```python
# Bad - restates code
counter += 1 # increment counter
# Good - explains why
counter += 1 # track retries for backoff calculation
```
### Codebase Architecture
See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for detailed architecture documentation.
**Directory Structure:**
- `/alembic` - Alembic db migrations
- `/api` - FastAPI REST endpoints + `container.py` composition root
- `/cli` - Typer CLI + `container.py` composition root
- `/deps` - Feature-scoped FastAPI dependencies (config, db, projects, repositories, services, importers)
- `/importers` - Import functionality for Claude, ChatGPT, and other sources
- `/markdown` - Markdown parsing and processing
- `/mcp` - MCP server + `container.py` composition root + `clients/` typed API clients
- `/models` - SQLAlchemy ORM models
- `/repository` - Data access layer
- `/schemas` - Pydantic models for validation
- `/services` - Business logic layer
- `/sync` - File synchronization services + `coordinator.py` for lifecycle management
**Composition Roots:**
Each entrypoint (API, MCP, CLI) has a composition root that:
- Reads `ConfigManager` (the only place that reads global config)
- Resolves runtime mode via `RuntimeMode` enum (TEST > CLOUD > LOCAL)
- Provides dependencies to downstream code explicitly
**Typed API Clients (MCP):**
MCP tools use typed clients in `mcp/clients/` to communicate with the API:
- `KnowledgeClient` - Entity CRUD operations
- `SearchClient` - Search operations
- `MemoryClient` - Context building
- `DirectoryClient` - Directory listing
- `ResourceClient` - Resource reading
- `ProjectClient` - Project management
Flow: MCP Tool → Typed Client → HTTP API → Router → Service → Repository
### Development Notes
- MCP tools are defined in src/basic_memory/mcp/tools/
- MCP prompts are defined in src/basic_memory/mcp/prompts/
- MCP tools should be atomic, composable operations
- Use `textwrap.dedent()` for multi-line string formatting in prompts and tools
- MCP Prompts are used to invoke tools and format content with instructions for an LLM
- Schema changes require Alembic migrations
- SQLite is used for indexing and full text search, files are source of truth
- Testing uses pytest with asyncio support (strict mode)
- Unit tests (`tests/`) use mocks when necessary; integration tests (`test-int/`) use real implementations
- By default, tests run against SQLite (fast, no Docker needed)
- Set `BASIC_MEMORY_TEST_POSTGRES=1` to run against Postgres (uses testcontainers - Docker required)
- Each test runs in a standalone environment with isolated database and tmp_path directory
- CI runs SQLite and Postgres tests in parallel for faster feedback
- Performance benchmarks are in `test-int/test_sync_performance_benchmark.py`
- Use pytest markers: `@pytest.mark.benchmark` for benchmarks, `@pytest.mark.slow` for slow tests
- **Coverage must stay at 100%**: Write tests for new code. Only use `# pragma: no cover` when tests would require excessive mocking (e.g., TYPE_CHECKING blocks, error handlers that need failure injection, runtime-mode-dependent code paths)
### Async Client Pattern (Important!)
**All MCP tools and CLI commands use the context manager pattern for HTTP clients:**
```python
from basic_memory.mcp.async_client import get_client
async def my_mcp_tool():
async with get_client() as client:
# Use client for API calls
response = await call_get(client, "/path")
return response
```
**Do NOT use:**
-`from basic_memory.mcp.async_client import client` (deprecated module-level client)
- ❌ Manual auth header management
-`inject_auth_header()` (deleted)
**Key principles:**
- Auth happens at client creation, not per-request
- Proper resource management via context managers
- Supports three modes: Local (ASGI), CLI cloud (HTTP + auth), Cloud app (factory injection)
- Factory pattern enables dependency injection for cloud consolidation
**For cloud app integration:**
```python
from basic_memory.mcp import async_client
# Set custom factory before importing tools
async_client.set_client_factory(your_custom_factory)
```
See SPEC-16 for full context manager refactor details.
## BASIC MEMORY PRODUCT USAGE
### Knowledge Structure
- Entity: Any concept, document, or idea represented as a markdown file
- Observation: A categorized fact about an entity (`- [category] content`)
- Relation: A directional link between entities (`- relation_type [[Target]]`)
- Frontmatter: YAML metadata at the top of markdown files
- Knowledge representation follows precise markdown format:
- Observations with [category] prefixes
- Relations with WikiLinks [[Entity]]
- Frontmatter with metadata
### Basic Memory Commands
**Local Commands:**
- Check sync status: `basic-memory status`
- Import from Claude: `basic-memory import claude conversations`
- Import from ChatGPT: `basic-memory import chatgpt`
- Import from Memory JSON: `basic-memory import memory-json`
- Tool access: `basic-memory tool` (provides CLI access to MCP tools)
- Continue: `basic-memory tool continue-conversation --topic="search"`
**Project Management:**
- List projects: `basic-memory project list`
- Add project: `basic-memory project add "name" ~/path`
- Project info: `basic-memory project info`
- One-way sync (local -> cloud): `basic-memory project sync`
- Bidirectional sync: `basic-memory project bisync`
- Integrity check: `basic-memory project check`
**Cloud Commands (requires subscription):**
- Authenticate: `basic-memory cloud login`
- Logout: `basic-memory cloud logout`
- Check cloud status: `basic-memory cloud status`
- Setup cloud sync: `basic-memory cloud setup`
### MCP Capabilities
- Basic Memory exposes these MCP tools to LLMs:
**Content Management:**
- `write_note(title, content, folder, tags)` - Create/update markdown notes with semantic observations and relations
- `read_note(identifier, page, page_size)` - Read notes by title, permalink, or memory:// URL with knowledge graph awareness
- `read_content(path)` - Read raw file content (text, images, binaries) without knowledge graph processing
- `view_note(identifier, page, page_size)` - View notes as formatted artifacts for better readability
- `edit_note(identifier, operation, content)` - Edit notes incrementally (append, prepend, find/replace, replace_section)
- `move_note(identifier, destination_path)` - Move notes to new locations, updating database and maintaining links
- `delete_note(identifier)` - Delete notes from the knowledge base
**Knowledge Graph Navigation:**
- `build_context(url, depth, timeframe)` - Navigate the knowledge graph via memory:// URLs for conversation continuity
- `recent_activity(type, depth, timeframe)` - Get recently updated information with specified timeframe (e.g., "1d", "1 week")
- `list_directory(dir_name, depth, file_name_glob)` - Browse directory contents with filtering and depth control
**Search & Discovery:**
- `search_notes(query, page, page_size, search_type, types, entity_types, after_date)` - Full-text search across all content with advanced filtering options
**Project Management:**
- `list_memory_projects()` - List all available projects with their status
- `create_memory_project(project_name, project_path, set_default)` - Create new Basic Memory projects
- `delete_project(project_name)` - Delete a project from configuration
**Visualization:**
- `canvas(nodes, edges, title, folder)` - Generate Obsidian canvas files for knowledge graph visualization
**ChatGPT-Compatible Tools:**
- `search(query)` - Search across knowledge base (OpenAI actions compatible)
- `fetch(id)` - Fetch full content of a search result document
- MCP Prompts for better AI interaction:
- `ai_assistant_guide()` - Guidance on effectively using Basic Memory tools for AI assistants
- `continue_conversation(topic, timeframe)` - Continue previous conversations with relevant historical context
- `search(query, after_date)` - Search with detailed, formatted results for better context understanding
- `recent_activity(timeframe)` - View recently changed items with formatted output
### Cloud Features (v0.15.0+)
Basic Memory now supports cloud synchronization and storage (requires active subscription):
**Authentication:**
- JWT-based authentication with subscription validation
- Secure session management with token refresh
- Support for multiple cloud projects
**Bidirectional Sync:**
- rclone bisync integration for two-way synchronization
- Conflict resolution and integrity verification
- Real-time sync with change detection
- Mount/unmount cloud storage for direct file access
**Cloud Project Management:**
- Create and manage projects in the cloud
- Toggle between local and cloud modes
- Per-project sync configuration
- Subscription-based access control
**Security & Performance:**
- Removed .env file loading for improved security
- .gitignore integration (respects gitignored files)
- WAL mode for SQLite performance
- Background relation resolution (non-blocking startup)
- API performance optimizations (SPEC-11)
## AI-Human Collaborative Development
Basic Memory emerged from and enables a new kind of development process that combines human and AI capabilities. Instead
of using AI just for code generation, we've developed a true collaborative workflow:
1. AI (LLM) writes initial implementation based on specifications and context
2. Human reviews, runs tests, and commits code with any necessary adjustments
3. Knowledge persists across conversations using Basic Memory's knowledge graph
4. Development continues seamlessly across different AI sessions with consistent context
5. Results improve through iterative collaboration and shared understanding
This approach has allowed us to tackle more complex challenges and build a more robust system than either humans or AI
could achieve independently.
**Problem-Solving Guidance:**
- If a solution isn't working after reasonable effort, suggest alternative approaches
- Don't persist with a problematic library or pattern when better alternatives exist
- Example: When py-pglite caused cascading test failures, switching to testcontainers-postgres was the right call
## GitHub Integration
Basic Memory has taken AI-Human collaboration to the next level by integrating Claude directly into the development workflow through GitHub:
### GitHub MCP Tools
Using the GitHub Model Context Protocol server, Claude can now:
- **Repository Management**:
- View repository files and structure
- Read file contents
- Create new branches
- Create and update files
- **Issue Management**:
- Create new issues
- Comment on existing issues
- Close and update issues
- Search across issues
- **Pull Request Workflow**:
- Create pull requests
- Review code changes
- Add comments to PRs
This integration enables Claude to participate as a full team member in the development process, not just as a code generation tool. Claude's GitHub account ([bm-claudeai](https://github.com/bm-claudeai)) is a member of the Basic Machines organization with direct contributor access to the codebase.
### Collaborative Development Process
With GitHub integration, the development workflow includes:
1. **Direct code review** - Claude can analyze PRs and provide detailed feedback
2. **Contribution tracking** - All of Claude's contributions are properly attributed in the Git history
3. **Branch management** - Claude can create feature branches for implementations
4. **Documentation maintenance** - Claude can keep documentation updated as the code evolves
5. **Code Commits**: ALWAYS sign off commits with `git commit -s`
This level of integration represents a new paradigm in AI-human collaboration, where the AI assistant becomes a full-fledged team member rather than just a tool for generating code snippets.
Symlink
+1
View File
@@ -0,0 +1 @@
AGENTS.md
+494
View File
@@ -0,0 +1,494 @@
# Note Format Reference
Every document in Basic Memory is a plain Markdown file. Files are the source of truth — changes to files automatically update the knowledge graph in the database. You maintain complete ownership, files work with git, and knowledge persists independently of any AI conversation.
## Document Structure
A note has three parts: YAML frontmatter, content (observations), and relations.
```markdown
---
title: Coffee Brewing Methods
type: note
tags: [coffee, brewing]
permalink: coffee-brewing-methods
---
# Coffee Brewing Methods
## Observations
- [method] Pour over provides more flavor clarity than French press
- [technique] Water temperature at 205°F extracts optimal compounds #brewing
- [preference] Ethiopian beans work well with lighter roasts (personal experience)
## Relations
- relates_to [[Coffee Bean Origins]]
- requires [[Proper Grinding Technique]]
- contrasts_with [[Tea Brewing Methods]]
```
The `## Observations` and `## Relations` headings are conventional but not required — the parser detects observations and relations by their syntax patterns anywhere in the document.
## Frontmatter
YAML metadata between `---` fences at the top of the file.
| Field | Required | Default | Description |
|-------|----------|---------|-------------|
| `title` | No | filename stem | Used for linking and references. Auto-set from filename if missing. |
| `type` | No | `note` | Entity type. Used for schema resolution and filtering. |
| `tags` | No | `[]` | List or comma-separated string. Used for organization and search. |
| `permalink` | No | generated from title | Stable identifier. Persists even if the file moves. |
| `schema` | No | none | Schema attachment — dict (inline), string (reference), or omitted (implicit). |
Custom fields are allowed. Any key not in the standard set is stored as `entity_metadata` and indexed for search and filtering.
```yaml
---
title: Paul Graham
type: Person
tags: [startups, essays, lisp]
permalink: paul-graham
status: active
source: wikipedia
---
```
Here `status` and `source` are custom fields stored in `entity_metadata`.
### Frontmatter Value Handling
YAML automatically converts some values to native types. Basic Memory normalizes them:
- Date strings (`2025-10-24`) → kept as ISO format strings
- Numbers (`1.0`) → converted to strings
- Booleans (`true`) → converted to strings (`"True"`)
- Lists and dicts → preserved, items normalized recursively
This prevents errors when downstream code expects string values.
## Observations
An observation is a categorized fact about the entity. Written as a Markdown list item.
**Syntax:**
```
- [category] content text #tag1 #tag2 (context)
```
| Part | Required | Description |
|------|----------|-------------|
| `[category]` | Yes | Classification in square brackets. Any text except `[]()` chars. |
| content | Yes | The fact or statement. |
| `#tags` | No | Inline tags. Space-separated, each starting with `#`. |
| `(context)` | No | Parenthesized text at end of line. Supporting details or source. |
### Examples
```markdown
- [tech] Uses SQLite for storage #database
- [design] Follows local-first architecture #architecture
- [decision] Selected bcrypt for passwords #security (based on OWASP audit)
- [name] Paul Graham
- [expertise] Startups
- [expertise] Lisp
- [expertise] Essay writing
```
Array-like fields use repeated categories — multiple `[expertise]` observations above.
### What Is Not an Observation
The parser excludes these list item patterns:
| Pattern | Example | Reason |
|---------|---------|--------|
| Checkboxes | `- [ ] Todo item`, `- [x] Done`, `- [-] Cancelled` | Task list syntax |
| Markdown links | `- [text](url)` | URL link syntax |
| Bare wiki links | `- [[Target]]` | Treated as a relation instead |
A list item with `#tags` but no `[category]` is still parsed — the tags are extracted and the category defaults to `Note`.
## Relations
Relations connect documents to form the knowledge graph. There are two kinds.
### Explicit Relations
Written as list items with a relation type and a `[[wiki link]]` target.
**Syntax:**
```
- relation_type [[Target Entity]] (context)
```
| Part | Required | Description |
|------|----------|-------------|
| `relation_type` | No | Text before `[[`. Defaults to `relates_to` if omitted. |
| `[[Target]]` | Yes | Wiki link to the target entity. Matched by title or permalink. |
| `(context)` | No | Parenthesized text after `]]`. Supporting details. |
### Examples
```markdown
- implements [[Search Design]]
- depends_on [[Database Schema]]
- works_at [[Y Combinator]] (co-founder)
- [[Some Entity]]
```
The last example — a bare `[[wiki link]]` in a list item — gets relation type `relates_to`.
Common relation types:
- `implements`, `depends_on`, `relates_to`, `inspired_by`
- `extends`, `part_of`, `contains`, `pairs_with`
- `works_at`, `authored`, `collaborated_with`
Any text works as a relation type. These are conventions, not a fixed set.
### Inline References
Wiki links appearing in regular prose (not as list items) create implicit `links_to` relations.
```markdown
This builds on [[Core Design]] and uses [[Utility Functions]].
```
This creates two relations: `links_to [[Core Design]]` and `links_to [[Utility Functions]]`.
### Forward References
Relations can link to entities that don't exist yet. Basic Memory resolves them when the target is created.
## Permalinks and memory:// URLs
Every document has a unique **permalink** — a stable identifier derived from its title. You can set one explicitly in frontmatter, or let the system generate it.
```yaml
permalink: auth-approaches-2024
```
Permalinks form the basis of `memory://` URLs:
```
memory://auth-approaches-2024 # By permalink
memory://Authentication Approaches # By title (auto-resolves)
memory://project/auth-approaches # By path
```
Pattern matching is supported:
```
memory://auth* # Starts with "auth"
memory://*/approaches # Ends with "approaches"
memory://project/*/requirements # Nested wildcard
```
## Schemas
Schemas declare the expected structure of a note — which observation categories and relation types a well-formed note should have. They use Picoschema, a compact notation from Google's Dotprompt that fits naturally in YAML frontmatter.
### Picoschema Syntax
```yaml
schema:
name: string, full name # required field with description
email?: string, contact email # ? = optional
role?: string, job title
works_at?: Organization, employer # capitalized type = entity reference
tags?(array): string, categories # array of type
status?(enum): [active, inactive] # enum with allowed values
metadata?(object): # nested object
updated_at?: string
source?: string
```
| Notation | Meaning | Example |
|----------|---------|---------|
| `field: type` | Required field | `name: string` |
| `field?: type` | Optional field | `role?: string` |
| `field(array): type` | Array of values | `expertise(array): string` |
| `field?(enum): [vals]` | Enum with allowed values | `status?(enum): [active, inactive]` |
| `field?(object):` | Nested object with sub-fields | `metadata?(object):` |
| `, description` | Description after comma | `name: string, full name` |
| `EntityName` | Capitalized type = entity reference | `works_at?: Organization` |
**Scalar types:** `string`, `integer`, `number`, `boolean`, `any`
Any type not in that set whose first letter is uppercase is treated as an entity reference (a relation target).
### Schema-to-Note Mapping
Schemas validate against existing observation/relation syntax. Note authors don't learn new syntax.
| Schema Declaration | Maps To | Example in Note |
|--------------------|---------|-----------------|
| `field: string` | Observation `[field] value` | `- [name] Paul Graham` |
| `field?(array): string` | Multiple `[field]` observations | `- [expertise] Lisp` (repeated) |
| `field?: EntityType` | Relation `field [[Target]]` | `- works_at [[Y Combinator]]` |
| `field?(array): EntityType` | Multiple `field` relations | `- authored [[Book]]` (repeated) |
| `tags` | Frontmatter `tags` array | `tags: [startups, essays]` |
| `field?(enum): [vals]` | Observation `[field] value` where value is in the set | `- [status] active` |
Observations and relations not covered by the schema are valid — schemas describe a subset, not a straitjacket.
### Schema Attachment
Three ways to attach a schema to a note, resolved in priority order:
**1. Inline schema**`schema` is a dict in frontmatter:
```yaml
---
title: Team Standup 2024-01-15
type: meeting
schema:
attendees(array): string, who was there
decisions(array): string, what was decided
action_items(array): string, follow-ups
blockers?(array): string, anything stuck
---
```
Good for one-off structured notes or prototyping a schema before extracting it.
**2. Explicit reference**`schema` is a string naming a schema note:
```yaml
---
title: Basic Memory
schema: SoftwareProject
---
```
or by permalink:
```yaml
---
title: LLM Memory Patterns
schema: schema/research-project
---
```
Use when the note's `type` differs from the schema it should validate against, or when multiple schema variants exist.
**3. Implicit by type** — no `schema` field, resolved by matching `type`:
```yaml
---
title: Paul Graham
type: Person
---
```
The system looks up a schema note where `entity: Person`. If found, it applies. If not, no validation occurs.
**4. No schema** — perfectly fine. Most notes don't need one.
### Schema Notes
A schema is itself a Basic Memory note with `type: schema`. It lives anywhere (though `schema/` is the conventional directory).
```yaml
# schema/Person.md
---
title: Person
type: schema
entity: Person
version: 1
schema:
name: string, full name
role?: string, job title or position
works_at?: Organization, employer
expertise?(array): string, areas of knowledge
email?: string, contact email
settings:
validation: warn
---
# Person
A human individual in the knowledge graph.
```
| Field | Required | Description |
|-------|----------|-------------|
| `type` | Yes | Must be `schema` |
| `entity` | Yes | The entity type this schema describes (e.g., `Person`) |
| `version` | No | Schema version number (default: `1`) |
| `schema` | Yes | Picoschema dict defining the fields |
| `settings.validation` | No | Validation mode (default: `warn`) |
Schema notes are regular notes — they show up in search, can have observations and relations, and participate in the knowledge graph.
### Validation Modes
| Mode | Behavior |
|------|----------|
| `warn` | Warnings in output, doesn't block (default) |
| `strict` | Errors that block sync, for CI/CD enforcement |
| `off` | No validation |
### Validation Output
```
$ bm schema validate people/ada-lovelace.md
⚠ Person schema validation:
- Missing required field: name (expected [name] observation)
- Missing optional field: role
- Missing optional field: works_at (no relation found)
Unmatched observations: [fact] ×2, [born] ×1
Unmatched relations: collaborated_with
```
"Unmatched" items are informational — observations and relations the schema doesn't cover.
### Schema Inference
Generate schemas from existing notes by analyzing observation and relation frequency:
```
$ bm schema infer Person
Analyzing 30 notes with type: Person...
Observations found:
[name] 30/30 100% → name: string
[role] 27/30 90% → role?: string
[expertise] 18/30 60% → expertise?(array): string
[email] 8/30 27% → email?: string
Relations found:
works_at 22/30 73% → works_at?: Organization
Suggested schema:
name: string, full name
role?: string, job title
expertise?(array): string, areas of knowledge
email?: string, contact email
works_at?: Organization, employer
Save to schema/Person.md? [y/n]
```
Frequency thresholds:
- **100% present** → required field
- **25%+ present** → optional field
- **Below 25%** → excluded from suggestion
### Schema Drift Detection
Track how usage patterns shift over time:
```
$ bm schema diff Person
Schema drift detected:
+ expertise: now in 81% of notes (was 12%)
- department: dropped to 3% of notes
~ works_at: cardinality changed (one → many)
Update schema? [y/n/review]
```
## Complete Examples
### Simple Note (No Schema)
```markdown
---
title: Project Ideas
type: note
tags: [ideas, brainstorm]
---
# Project Ideas
## Observations
- [idea] Build a CLI tool for markdown linting #tooling
- [idea] Create a recipe knowledge base #cooking
- [priority] Focus on developer tools first (Q1 goal)
## Relations
- inspired_by [[Developer Workflow Research]]
- part_of [[Q1 Planning]]
```
### Schema-Validated Note
Schema at `schema/Person.md`:
```yaml
---
title: Person
type: schema
entity: Person
version: 1
schema:
name: string, full name
role?: string, job title or position
works_at?: Organization, employer
expertise?(array): string, areas of knowledge
email?: string, contact email
settings:
validation: warn
---
# Person
A human individual in the knowledge graph.
```
Note at `people/paul-graham.md`:
```markdown
---
title: Paul Graham
type: Person
tags: [startups, essays, lisp]
---
# Paul Graham
## Observations
- [name] Paul Graham
- [role] Essayist and investor
- [expertise] Startups
- [expertise] Lisp
- [expertise] Essay writing
- [fact] Created Viaweb, the first web app
## Relations
- works_at [[Y Combinator]]
- authored [[Hackers and Painters]]
```
The `[fact]` observation and `authored` relation are not in the schema — they're valid, just unmatched. The schema only checks that `[name]` exists (required) and looks for optional fields like `[role]`, `[expertise]`, and `works_at`.
### Inline Schema Note
```markdown
---
title: Team Standup 2024-01-15
type: meeting
schema:
attendees(array): string, who was there
decisions(array): string, what was decided
action_items(array): string, follow-ups
blockers?(array): string, anything stuck
---
# Team Standup 2024-01-15
## Observations
- [attendees] Paul
- [attendees] Sarah
- [decisions] Ship v2 by Friday
- [action_items] Paul to review PR #42
- [blockers] Waiting on API credentials
```
+108 -42
View File
@@ -6,15 +6,14 @@
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
![](https://badge.mcpx.dev?type=server 'MCP Server')
![](https://badge.mcpx.dev?type=dev 'MCP Dev')
[![smithery badge](https://smithery.ai/badge/@basicmachines-co/basic-memory)](https://smithery.ai/server/@basicmachines-co/basic-memory)
## 🚀 Basic Memory Cloud is Live!
- **Cross-device and multi-platform support is here.** Your knowledge graph now works on desktop, web, and mobile - seamlessly synced across all your AI tools (Claude, ChatGPT, Gemini, Claude Code, and Codex)
- **Early Supporter Pricing:** Early users get 25% off forever.
The open source project continues as always. Cloud just makes it work everywhere.
- **Cross-device and multi-platform support is here.** Your knowledge graph now works on desktop, web, and mobile.
- **Cloud is optional.** The local-first open-source workflow continues as always.
- **OSS discount:** use code `{{OSS_DISCOUNT_CODE}}` for 20% off for 3 months.
[Sign up now →](https://basicmemory.com/beta)
[Sign up now →](https://basicmemory.com)
with a 7 day free trial
@@ -24,8 +23,8 @@ Basic Memory lets you build persistent knowledge through natural conversations w
Claude, while keeping everything in simple Markdown files on your computer. It uses the Model Context Protocol (MCP) to
enable any compatible LLM to read and write to your local knowledge base.
- Website: https://basicmachines.co
- Documentation: https://memory.basicmachines.co
- Website: https://basicmemory.com
- Documentation: https://docs.basicmemory.com
## Pick up your conversation right where you left off
@@ -63,24 +62,6 @@ uv tool install basic-memory
You can view shared context via files in `~/basic-memory` (default directory location).
### Alternative Installation via Smithery
You can use [Smithery](https://smithery.ai/server/@basicmachines-co/basic-memory) to automatically configure Basic
Memory for Claude Desktop:
```bash
npx -y @smithery/cli install @basicmachines-co/basic-memory --client claude
```
This installs and configures Basic Memory without requiring manual edits to the Claude Desktop configuration file. The
Smithery server hosts the MCP server component, while your data remains stored locally as Markdown files.
### Glama.ai
<a href="https://glama.ai/mcp/servers/o90kttu9ym">
<img width="380" height="200" src="https://glama.ai/mcp/servers/o90kttu9ym/badge" alt="basic-memory MCP server" />
</a>
## Why Basic Memory?
Most LLM interactions are ephemeral - you ask a question, get an answer, and everything is forgotten. Each conversation
@@ -363,52 +344,120 @@ basic-memory sync --watch
3. Cloud features (optional, requires subscription):
```bash
# Authenticate with cloud
# Authenticate with cloud (stores OAuth token locally)
basic-memory cloud login
# Bidirectional sync with cloud
basic-memory cloud sync
# (Optional) install/configure rclone for file sync commands
basic-memory cloud setup
# Verify cloud integrity
basic-memory cloud check
# Check cloud auth + health
basic-memory cloud status
```
# Mount cloud storage
basic-memory cloud mount
**Per-Project Cloud Routing** (API key based):
Individual projects can be routed through the cloud while others stay local. This uses an API key for routed
project calls:
```bash
# Save an API key (create one in the web app or via CLI)
basic-memory cloud set-key bmc_abc123...
# Or create one via CLI (requires OAuth login first)
basic-memory cloud create-key "my-laptop"
# Set a project to route through cloud
basic-memory project set-cloud research
# Revert a project to local mode
basic-memory project set-local research
# List projects and route metadata
basic-memory project list
```
`basic-memory cloud login` / `basic-memory cloud logout` are authentication commands. They do not change default CLI
routing behavior.
**Routing Flags**:
Use routing flags to disambiguate command targets:
```bash
# Force local routing for this command
basic-memory status --local
basic-memory project list --local
basic-memory project ls --name main --local
# Force cloud routing for this command
basic-memory status --cloud
basic-memory project info my-project --cloud
basic-memory project ls --name main --cloud
```
No-flag behavior defaults to local when no project context is present.
The local MCP server (`basic-memory mcp`) always uses local routing (including `--transport stdio`).
**CLI Note Editing (`tool edit-note`):**
```bash
# Append content
basic-memory tool edit-note project-plan --operation append --content $'\n## Next Steps\n- Finalize rollout'
# Find/replace with replacement count validation
basic-memory tool edit-note docs/api --operation find_replace --find-text "v0.14.0" --content "v0.15.0" --expected-replacements 2
# Replace a section body
basic-memory tool edit-note docs/setup --operation replace_section --section "## Installation" --content $'Updated install steps\n- Run just install'
# JSON metadata output for integrations
basic-memory tool edit-note docs/setup --operation append --content $'\n- Added note' --format json
```
4. In Claude Desktop, the LLM can now use these tools:
**Content Management:**
```
write_note(title, content, folder, tags) - Create or update notes
read_note(identifier, page, page_size) - Read notes by title or permalink
write_note(title, content, folder, tags, output_format="text"|"json") - Create or update notes
read_note(identifier, page, page_size, output_format="text"|"json") - Read notes by title or permalink
read_content(path) - Read raw file content (text, images, binaries)
view_note(identifier) - View notes as formatted artifacts
edit_note(identifier, operation, content) - Edit notes incrementally
move_note(identifier, destination_path) - Move notes with database consistency
delete_note(identifier) - Delete notes from knowledge base
edit_note(identifier, operation, content, output_format="text"|"json") - Edit notes incrementally
move_note(identifier, destination_path, output_format="text"|"json") - Move notes with database consistency
delete_note(identifier, output_format="text"|"json") - Delete notes from knowledge base
```
**Knowledge Graph Navigation:**
```
build_context(url, depth, timeframe) - Navigate knowledge graph via memory:// URLs
recent_activity(type, depth, timeframe) - Find recently updated information
build_context(url, depth, timeframe, output_format="json"|"text") - Navigate knowledge graph via memory:// URLs
recent_activity(type, depth, timeframe, output_format="text"|"json") - Find recently updated information
list_directory(dir_name, depth) - Browse directory contents with filtering
```
**Search & Discovery:**
```
search(query, page, page_size) - Search across your knowledge base
search_notes(query, page, page_size, search_type, types, entity_types, after_date, metadata_filters, tags, status, project) - Search with filters
search_by_metadata(filters, limit, offset, project) - Structured frontmatter search
```
**Project Management:**
```
list_memory_projects() - List all available projects
create_memory_project(project_name, project_path) - Create new projects
list_memory_projects(output_format="text"|"json") - List all available projects
create_memory_project(project_name, project_path, output_format="text"|"json") - Create new projects
get_current_project() - Show current project stats
sync_status() - Check synchronization status
```
`output_format` defaults to `"text"` for these tools, preserving current human-readable responses.
`build_context` defaults to `"json"` and can be switched to `"text"` when compact markdown output is preferred.
**Cloud Discovery (opt-in):**
```
cloud_info() - Show optional Cloud overview and setup guidance
release_notes() - Show latest release notes
```
**Visualization:**
```
canvas(nodes, edges, title, folder) - Generate knowledge visualizations
@@ -426,7 +475,7 @@ canvas(nodes, edges, title, folder) - Generate knowledge visualizations
## Futher info
See the [Documentation](https://memory.basicmachines.co/) for more info, including:
See the [Documentation](https://docs.basicmemory.com) for more info, including:
- [Complete User Guide](https://docs.basicmemory.com/user-guide/)
- [CLI tools](https://docs.basicmemory.com/guides/cli-reference/)
@@ -452,6 +501,9 @@ Basic Memory uses [Loguru](https://github.com/Delgan/loguru) for logging. The lo
|----------|---------|-------------|
| `BASIC_MEMORY_LOG_LEVEL` | `INFO` | Log level: DEBUG, INFO, WARNING, ERROR |
| `BASIC_MEMORY_CLOUD_MODE` | `false` | When `true`, API logs to stdout with structured context |
| `BASIC_MEMORY_FORCE_LOCAL` | `false` | When `true`, forces local API routing |
| `BASIC_MEMORY_FORCE_CLOUD` | `false` | When `true`, forces cloud API routing |
| `BASIC_MEMORY_EXPLICIT_ROUTING` | `false` | When `true`, marks route selection as explicit (`--local`/`--cloud`) |
| `BASIC_MEMORY_ENV` | `dev` | Set to `test` for test mode (stderr only) |
### Examples
@@ -496,16 +548,23 @@ just test
- `just test-int-postgres` - Run integration tests against Postgres
- `just test-windows` - Run Windows-specific tests (auto-skips on other platforms)
- `just test-benchmark` - Run performance benchmark tests
- `just testmon` - Run tests impacted by recent changes (pytest-testmon)
- `just test-smoke` - Run fast MCP end-to-end smoke test
- `just fast-check` - Run fix/format/typecheck + impacted tests + smoke test
- `just doctor` - Run local file <-> DB consistency checks with temp config
**Postgres Testing:**
Postgres tests use [testcontainers](https://testcontainers-python.readthedocs.io/) which automatically spins up a Postgres instance in Docker. No manual database setup required - just have Docker running.
**Testmon Note:** When no files have changed, `just testmon` may collect 0 tests. That's expected and means no impacted tests were detected.
**Test Markers:**
Tests use pytest markers for selective execution:
- `windows` - Windows-specific database optimizations
- `benchmark` - Performance tests (excluded from default runs)
- `smoke` - Fast MCP end-to-end smoke tests
**Other Development Commands:**
```bash
@@ -513,10 +572,17 @@ just install # Install with dev dependencies
just lint # Run linting checks
just typecheck # Run type checking
just format # Format code with ruff
just fast-check # Fast local loop (fix/format/typecheck + testmon + smoke)
just doctor # Local consistency check (temp config)
just check # Run all quality checks
just migration "msg" # Create database migration
```
**Local Consistency Check:**
```bash
basic-memory doctor # Verifies file <-> database sync in a temp project
```
See the [justfile](justfile) for the complete list of development commands.
## License
+44 -14
View File
@@ -18,7 +18,7 @@ Each entrypoint uses a **composition root** pattern to manage configuration and
A composition root is the single place in an application where dependencies are wired together. In Basic Memory, each entrypoint has its own composition root that:
1. Reads configuration from `ConfigManager`
2. Resolves runtime mode (cloud/local/test)
2. Resolves runtime mode (local/test)
3. Creates and provides dependencies to downstream code
**Key principle**: Only composition roots read global configuration. All other modules receive configuration explicitly.
@@ -52,10 +52,7 @@ class Container:
def create(cls) -> "Container":
"""Create container by reading ConfigManager."""
config = ConfigManager().config
mode = resolve_runtime_mode(
cloud_mode_enabled=config.cloud_mode_enabled,
is_test_env=config.is_test_env,
)
mode = resolve_runtime_mode(is_test_env=config.is_test_env)
return cls(config=config, mode=mode)
@property
@@ -99,17 +96,20 @@ class RuntimeMode(Enum):
return self == RuntimeMode.TEST
```
Resolution follows this precedence: **TEST > CLOUD > LOCAL**
Resolution follows this precedence in local app flows: **TEST > LOCAL**
```python
def resolve_runtime_mode(cloud_mode_enabled: bool, is_test_env: bool) -> RuntimeMode:
def resolve_runtime_mode(is_test_env: bool) -> RuntimeMode:
if is_test_env:
return RuntimeMode.TEST
if cloud_mode_enabled:
return RuntimeMode.CLOUD
return RuntimeMode.LOCAL
```
**Note**: `RuntimeMode` determines global behavior (e.g., whether to start file sync).
Per-project routing is orthogonal: individual projects can be set to `cloud` mode via `ProjectMode`,
which affects client routing in `get_client(project_name=...)` without changing global runtime mode.
`RuntimeMode.CLOUD` may remain for compatibility, but standard local runtime resolution does not select it.
## Dependencies Package
### Structure
@@ -214,15 +214,45 @@ Example tool using typed client:
```python
@mcp.tool()
async def search_notes(query: str, project: str | None = None) -> SearchResponse:
async with get_client() as client:
active_project = await get_active_project(client, project)
async def search_notes(
query: str,
project: str | None = None,
metadata_filters: dict | None = None,
tags: list[str] | None = None,
status: str | None = None,
) -> SearchResponse:
async with get_project_client(project, context) as (client, active_project):
# Import client inside function to avoid circular imports
from basic_memory.mcp.clients import SearchClient
from basic_memory.schemas.search import SearchQuery
search_query = SearchQuery(
text=query,
metadata_filters=metadata_filters,
tags=tags,
status=status,
)
search_client = SearchClient(client, active_project.external_id)
return await search_client.search(query)
return await search_client.search(search_query.model_dump())
```
### Per-Project Client Routing
`get_project_client()` from `mcp/project_context.py` is an async context manager that:
1. Resolves the project name from config (no network call)
2. Creates the correctly-routed client based on the project's mode (local ASGI or cloud HTTP with API key)
3. Validates the project via the API
4. Yields `(client, active_project)` tuple
This solves the bootstrap problem: you need the project name to choose the right client (local vs cloud), but you need the client to validate the project exists.
```python
from basic_memory.mcp.project_context import get_project_client
async with get_project_client(project, context) as (client, active_project):
# client is routed based on project's mode (local or cloud)
# active_project is validated via the API
...
```
## Sync Coordination
+494
View File
@@ -0,0 +1,494 @@
# Note Format Reference
Every document in Basic Memory is a plain Markdown file. Files are the source of truth — changes to files automatically update the knowledge graph in the database. You maintain complete ownership, files work with git, and knowledge persists independently of any AI conversation.
## Document Structure
A note has three parts: YAML frontmatter, content (observations), and relations.
```markdown
---
title: Coffee Brewing Methods
type: note
tags: [coffee, brewing]
permalink: coffee-brewing-methods
---
# Coffee Brewing Methods
## Observations
- [method] Pour over provides more flavor clarity than French press
- [technique] Water temperature at 205°F extracts optimal compounds #brewing
- [preference] Ethiopian beans work well with lighter roasts (personal experience)
## Relations
- relates_to [[Coffee Bean Origins]]
- requires [[Proper Grinding Technique]]
- contrasts_with [[Tea Brewing Methods]]
```
The `## Observations` and `## Relations` headings are conventional but not required — the parser detects observations and relations by their syntax patterns anywhere in the document.
## Frontmatter
YAML metadata between `---` fences at the top of the file.
| Field | Required | Default | Description |
|-------|----------|---------|-------------|
| `title` | No | filename stem | Used for linking and references. Auto-set from filename if missing. |
| `type` | No | `note` | Entity type. Used for schema resolution and filtering. |
| `tags` | No | `[]` | List or comma-separated string. Used for organization and search. |
| `permalink` | No | generated from title | Stable identifier. Persists even if the file moves. |
| `schema` | No | none | Schema attachment — dict (inline), string (reference), or omitted (implicit). |
Custom fields are allowed. Any key not in the standard set is stored as `entity_metadata` and indexed for search and filtering.
```yaml
---
title: Paul Graham
type: Person
tags: [startups, essays, lisp]
permalink: paul-graham
status: active
source: wikipedia
---
```
Here `status` and `source` are custom fields stored in `entity_metadata`.
### Frontmatter Value Handling
YAML automatically converts some values to native types. Basic Memory normalizes them:
- Date strings (`2025-10-24`) → kept as ISO format strings
- Numbers (`1.0`) → converted to strings
- Booleans (`true`) → converted to strings (`"True"`)
- Lists and dicts → preserved, items normalized recursively
This prevents errors when downstream code expects string values.
## Observations
An observation is a categorized fact about the entity. Written as a Markdown list item.
**Syntax:**
```
- [category] content text #tag1 #tag2 (context)
```
| Part | Required | Description |
|------|----------|-------------|
| `[category]` | Yes | Classification in square brackets. Any text except `[]()` chars. |
| content | Yes | The fact or statement. |
| `#tags` | No | Inline tags. Space-separated, each starting with `#`. |
| `(context)` | No | Parenthesized text at end of line. Supporting details or source. |
### Examples
```markdown
- [tech] Uses SQLite for storage #database
- [design] Follows local-first architecture #architecture
- [decision] Selected bcrypt for passwords #security (based on OWASP audit)
- [name] Paul Graham
- [expertise] Startups
- [expertise] Lisp
- [expertise] Essay writing
```
Array-like fields use repeated categories — multiple `[expertise]` observations above.
### What Is Not an Observation
The parser excludes these list item patterns:
| Pattern | Example | Reason |
|---------|---------|--------|
| Checkboxes | `- [ ] Todo item`, `- [x] Done`, `- [-] Cancelled` | Task list syntax |
| Markdown links | `- [text](url)` | URL link syntax |
| Bare wiki links | `- [[Target]]` | Treated as a relation instead |
A list item with `#tags` but no `[category]` is still parsed — the tags are extracted and the category defaults to `Note`.
## Relations
Relations connect documents to form the knowledge graph. There are two kinds.
### Explicit Relations
Written as list items with a relation type and a `[[wiki link]]` target.
**Syntax:**
```
- relation_type [[Target Entity]] (context)
```
| Part | Required | Description |
|------|----------|-------------|
| `relation_type` | No | Text before `[[`. Defaults to `relates_to` if omitted. |
| `[[Target]]` | Yes | Wiki link to the target entity. Matched by title or permalink. |
| `(context)` | No | Parenthesized text after `]]`. Supporting details. |
### Examples
```markdown
- implements [[Search Design]]
- depends_on [[Database Schema]]
- works_at [[Y Combinator]] (co-founder)
- [[Some Entity]]
```
The last example — a bare `[[wiki link]]` in a list item — gets relation type `relates_to`.
Common relation types:
- `implements`, `depends_on`, `relates_to`, `inspired_by`
- `extends`, `part_of`, `contains`, `pairs_with`
- `works_at`, `authored`, `collaborated_with`
Any text works as a relation type. These are conventions, not a fixed set.
### Inline References
Wiki links appearing in regular prose (not as list items) create implicit `links_to` relations.
```markdown
This builds on [[Core Design]] and uses [[Utility Functions]].
```
This creates two relations: `links_to [[Core Design]]` and `links_to [[Utility Functions]]`.
### Forward References
Relations can link to entities that don't exist yet. Basic Memory resolves them when the target is created.
## Permalinks and memory:// URLs
Every document has a unique **permalink** — a stable identifier derived from its title. You can set one explicitly in frontmatter, or let the system generate it.
```yaml
permalink: auth-approaches-2024
```
Permalinks form the basis of `memory://` URLs:
```
memory://auth-approaches-2024 # By permalink
memory://Authentication Approaches # By title (auto-resolves)
memory://project/auth-approaches # By path
```
Pattern matching is supported:
```
memory://auth* # Starts with "auth"
memory://*/approaches # Ends with "approaches"
memory://project/*/requirements # Nested wildcard
```
## Schemas
Schemas declare the expected structure of a note — which observation categories and relation types a well-formed note should have. They use Picoschema, a compact notation from Google's Dotprompt that fits naturally in YAML frontmatter.
### Picoschema Syntax
```yaml
schema:
name: string, full name # required field with description
email?: string, contact email # ? = optional
role?: string, job title
works_at?: Organization, employer # capitalized type = entity reference
tags?(array): string, categories # array of type
status?(enum): [active, inactive] # enum with allowed values
metadata?(object): # nested object
updated_at?: string
source?: string
```
| Notation | Meaning | Example |
|----------|---------|---------|
| `field: type` | Required field | `name: string` |
| `field?: type` | Optional field | `role?: string` |
| `field(array): type` | Array of values | `expertise(array): string` |
| `field?(enum): [vals]` | Enum with allowed values | `status?(enum): [active, inactive]` |
| `field?(object):` | Nested object with sub-fields | `metadata?(object):` |
| `, description` | Description after comma | `name: string, full name` |
| `EntityName` | Capitalized type = entity reference | `works_at?: Organization` |
**Scalar types:** `string`, `integer`, `number`, `boolean`, `any`
Any type not in that set whose first letter is uppercase is treated as an entity reference (a relation target).
### Schema-to-Note Mapping
Schemas validate against existing observation/relation syntax. Note authors don't learn new syntax.
| Schema Declaration | Maps To | Example in Note |
|--------------------|---------|-----------------|
| `field: string` | Observation `[field] value` | `- [name] Paul Graham` |
| `field?(array): string` | Multiple `[field]` observations | `- [expertise] Lisp` (repeated) |
| `field?: EntityType` | Relation `field [[Target]]` | `- works_at [[Y Combinator]]` |
| `field?(array): EntityType` | Multiple `field` relations | `- authored [[Book]]` (repeated) |
| `tags` | Frontmatter `tags` array | `tags: [startups, essays]` |
| `field?(enum): [vals]` | Observation `[field] value` where value is in the set | `- [status] active` |
Observations and relations not covered by the schema are valid — schemas describe a subset, not a straitjacket.
### Schema Attachment
Three ways to attach a schema to a note, resolved in priority order:
**1. Inline schema**`schema` is a dict in frontmatter:
```yaml
---
title: Team Standup 2024-01-15
type: meeting
schema:
attendees(array): string, who was there
decisions(array): string, what was decided
action_items(array): string, follow-ups
blockers?(array): string, anything stuck
---
```
Good for one-off structured notes or prototyping a schema before extracting it.
**2. Explicit reference**`schema` is a string naming a schema note:
```yaml
---
title: Basic Memory
schema: SoftwareProject
---
```
or by permalink:
```yaml
---
title: LLM Memory Patterns
schema: schema/research-project
---
```
Use when the note's `type` differs from the schema it should validate against, or when multiple schema variants exist.
**3. Implicit by type** — no `schema` field, resolved by matching `type`:
```yaml
---
title: Paul Graham
type: Person
---
```
The system looks up a schema note where `entity: Person`. If found, it applies. If not, no validation occurs.
**4. No schema** — perfectly fine. Most notes don't need one.
### Schema Notes
A schema is itself a Basic Memory note with `type: schema`. It lives anywhere (though `schema/` is the conventional directory).
```yaml
# schema/Person.md
---
title: Person
type: schema
entity: Person
version: 1
schema:
name: string, full name
role?: string, job title or position
works_at?: Organization, employer
expertise?(array): string, areas of knowledge
email?: string, contact email
settings:
validation: warn
---
# Person
A human individual in the knowledge graph.
```
| Field | Required | Description |
|-------|----------|-------------|
| `type` | Yes | Must be `schema` |
| `entity` | Yes | The entity type this schema describes (e.g., `Person`) |
| `version` | No | Schema version number (default: `1`) |
| `schema` | Yes | Picoschema dict defining the fields |
| `settings.validation` | No | Validation mode (default: `warn`) |
Schema notes are regular notes — they show up in search, can have observations and relations, and participate in the knowledge graph.
### Validation Modes
| Mode | Behavior |
|------|----------|
| `warn` | Warnings in output, doesn't block (default) |
| `strict` | Errors that block sync, for CI/CD enforcement |
| `off` | No validation |
### Validation Output
```
$ bm schema validate people/ada-lovelace.md
⚠ Person schema validation:
- Missing required field: name (expected [name] observation)
- Missing optional field: role
- Missing optional field: works_at (no relation found)
Unmatched observations: [fact] ×2, [born] ×1
Unmatched relations: collaborated_with
```
"Unmatched" items are informational — observations and relations the schema doesn't cover.
### Schema Inference
Generate schemas from existing notes by analyzing observation and relation frequency:
```
$ bm schema infer Person
Analyzing 30 notes with type: Person...
Observations found:
[name] 30/30 100% → name: string
[role] 27/30 90% → role?: string
[expertise] 18/30 60% → expertise?(array): string
[email] 8/30 27% → email?: string
Relations found:
works_at 22/30 73% → works_at?: Organization
Suggested schema:
name: string, full name
role?: string, job title
expertise?(array): string, areas of knowledge
email?: string, contact email
works_at?: Organization, employer
Save to schema/Person.md? [y/n]
```
Frequency thresholds:
- **100% present** → required field
- **25%+ present** → optional field
- **Below 25%** → excluded from suggestion
### Schema Drift Detection
Track how usage patterns shift over time:
```
$ bm schema diff Person
Schema drift detected:
+ expertise: now in 81% of notes (was 12%)
- department: dropped to 3% of notes
~ works_at: cardinality changed (one → many)
Update schema? [y/n/review]
```
## Complete Examples
### Simple Note (No Schema)
```markdown
---
title: Project Ideas
type: note
tags: [ideas, brainstorm]
---
# Project Ideas
## Observations
- [idea] Build a CLI tool for markdown linting #tooling
- [idea] Create a recipe knowledge base #cooking
- [priority] Focus on developer tools first (Q1 goal)
## Relations
- inspired_by [[Developer Workflow Research]]
- part_of [[Q1 Planning]]
```
### Schema-Validated Note
Schema at `schema/Person.md`:
```yaml
---
title: Person
type: schema
entity: Person
version: 1
schema:
name: string, full name
role?: string, job title or position
works_at?: Organization, employer
expertise?(array): string, areas of knowledge
email?: string, contact email
settings:
validation: warn
---
# Person
A human individual in the knowledge graph.
```
Note at `people/paul-graham.md`:
```markdown
---
title: Paul Graham
type: Person
tags: [startups, essays, lisp]
---
# Paul Graham
## Observations
- [name] Paul Graham
- [role] Essayist and investor
- [expertise] Startups
- [expertise] Lisp
- [expertise] Essay writing
- [fact] Created Viaweb, the first web app
## Relations
- works_at [[Y Combinator]]
- authored [[Hackers and Painters]]
```
The `[fact]` observation and `authored` relation are not in the schema — they're valid, just unmatched. The schema only checks that `[name]` exists (required) and looks for optional fields like `[role]`, `[expertise]`, and `works_at`.
### Inline Schema Note
```markdown
---
title: Team Standup 2024-01-15
type: meeting
schema:
attendees(array): string, who was there
decisions(array): string, what was decided
action_items(array): string, follow-ups
blockers?(array): string, anything stuck
---
# Team Standup 2024-01-15
## Observations
- [attendees] Paul
- [attendees] Sarah
- [decisions] Ship v2 by Friday
- [action_items] Paul to review PR #42
- [blockers] Waiting on API credentials
```
+135
View File
@@ -0,0 +1,135 @@
# Simplified Local/Cloud Routing
## Context
Basic Memory now uses explicit, project-aware routing without a global cloud-mode toggle.
Routing is determined by command-level flags and project mode, not by a global `cloud_mode` state.
This document is the canonical contract for local/cloud routing behavior in CLI, MCP, and API-adjacent clients.
## Goals
1. Remove global `cloud_mode` from runtime/routing semantics.
2. Keep MCP stdio local-only and predictable.
3. Make CLI routing explicit and easy to reason about.
4. Support projects that exist in both local and cloud without ambiguity.
## Routing Contract
Routing is resolved in this order:
1. Injected client factory (for composition/integration contexts)
2. Explicit routing override (`--local` / `--cloud` or env vars below)
3. Project-scoped routing (`project.mode`) when a project is known
4. Default local routing
### Routing Environment Variables
- `BASIC_MEMORY_FORCE_LOCAL=true`: force local transport
- `BASIC_MEMORY_FORCE_CLOUD=true`: force cloud proxy transport
- `BASIC_MEMORY_EXPLICIT_ROUTING=true`: marks routing as explicitly chosen for this command
When explicit routing is active, project mode does not override the selected route.
## Config Semantics
- `project.mode` is the only config-based routing signal for project-scoped operations.
- Legacy `cloud_mode` values may be encountered during migration/loading but are not used for routing behavior.
- Normalization saves remove stale `cloud_mode` from `~/.basic-memory/config.json`.
### Example Config
```json
{
"projects": {
"main": {
"path": "/Users/me/basic-memory",
"mode": "local",
"cloud_sync_path": null,
"bisync_initialized": false,
"last_sync": null
},
"specs": {
"path": "specs",
"mode": "cloud",
"cloud_sync_path": "/Users/me/dev/specs",
"bisync_initialized": true,
"last_sync": "2026-02-06T17:36:38.544153"
}
},
"default_project": "main",
"cloud_api_key": "bmc_abc123...",
"cloud_host": "https://cloud.basicmemory.com"
}
```
## Cloud Commands Are Auth-Only
`bm cloud login`, `bm cloud logout`, and `bm cloud status` manage authentication state.
- `bm cloud login`
- performs OAuth device flow
- stores/refreshes token material
- may verify cloud health/subscription
- does not change routing defaults
- `bm cloud logout`
- removes stored OAuth session tokens
- does not change routing defaults
- `bm cloud status`
- reports auth state (API key, OAuth token validity)
- runs health checks only when credentials are available
## MCP Stdio Local Guarantee
`bm mcp --transport stdio` always routes locally.
The command sets explicit local routing (`BASIC_MEMORY_FORCE_LOCAL=true` and
`BASIC_MEMORY_EXPLICIT_ROUTING=true`) before starting the server. This prevents cloud routing for stdio MCP,
even if the selected project has `mode: cloud`.
## Project List UX for Dual Presence
Projects may exist in both local and cloud. `bm project list` should display that clearly in one row per logical
project identity, with explicit source/target signals.
Recommended display contract:
1. Keep one row per normalized project name/permalink.
2. Show both local and cloud presence as separate columns/indicators.
3. Show an explicit `MCP (stdio)` target column that always resolves to `local`.
4. Keep CLI route semantics explicit:
- no flags: default local for non-project commands
- `--cloud`: force cloud
- `--local`: force local
## Project LS Targeting
`bm project ls` should clearly identify which project instance is being listed.
Targeting rules:
1. No routing flags: list local project files.
2. `--cloud`: list cloud project files.
3. `--local`: list local project files (explicit override).
4. Output should label the active target (`LOCAL` or `CLOUD`) in heading or status line.
## Runtime Mode
Runtime mode is no longer a cloud/local routing switch for local app flows.
- `resolve_runtime_mode(is_test_env)` resolves to:
- `TEST` when running in test environment
- `LOCAL` otherwise
- `RuntimeMode.CLOUD` may remain for compatibility with existing tests/call sites but is not selected by normal local
runtime resolution.
## Verification Checklist
1. Loading config with legacy `cloud_mode` succeeds.
2. Saving config strips legacy `cloud_mode`.
3. `--local/--cloud` always override per-project mode for that command.
4. No-project + no-flags commands route local by default.
5. `bm cloud login/logout` do not toggle routing behavior.
6. `bm mcp` remains local-only in stdio mode.
7. `bm project list` communicates dual local/cloud presence without ambiguity.
8. `bm project ls` output identifies route target explicitly.
+49 -1
View File
@@ -1038,6 +1038,35 @@ recent_decisions = await search_notes(
)
```
**Structured frontmatter filters**:
```python
# Filter by tags and status
results = await search_notes(
query="authentication",
tags=["security"],
status="in-progress",
project="main"
)
# Complex metadata filters (supports $in, $gt, $gte, $lt, $lte, $between)
results = await search_notes(
query="api design",
metadata_filters={
"type": "spec",
"priority": {"$in": ["high", "critical"]},
"tags": ["architecture"]
},
project="main"
)
# Metadata-only search
results = await search_by_metadata(
filters={"type": "spec", "status": "in-progress"},
project="main"
)
```
### Search Types
**Text search (default)**:
@@ -2861,7 +2890,7 @@ contents = await list_directory(
### Search & Discovery
**search_notes(query, page, page_size, search_type, types, entity_types, after_date, project)**
**search_notes(query, page, page_size, search_type, types, entity_types, after_date, metadata_filters, tags, status, project)**
- Search across knowledge base
- Parameters:
- `query` (required): Search query
@@ -2871,6 +2900,9 @@ contents = await list_directory(
- `types` (optional): Entity type filter
- `entity_types` (optional): Observation category filter
- `after_date` (optional): Date filter (ISO format)
- `metadata_filters` (optional): Structured frontmatter filters (dict)
- `tags` (optional): Frontmatter tags filter (list)
- `status` (optional): Frontmatter status filter (string)
- `project` (required unless default_project_mode): Target project
- Returns: Matching entities with scores
- Example:
@@ -2883,6 +2915,22 @@ results = await search_notes(
)
```
**search_by_metadata(filters, limit, offset, project)**
- Metadata-only search using structured frontmatter
- Parameters:
- `filters` (required): Dict of field -> value (supports $in, $gt/$gte/$lt/$lte, $between)
- `limit` (optional): Max results (default: 20)
- `offset` (optional): Pagination offset (default: 0)
- `project` (required unless default_project_mode): Target project
- Returns: Matching entities
- Example:
```python
results = await search_by_metadata(
filters={"type": "spec", "status": "in-progress"},
project="main"
)
```
### Project Management
**list_memory_projects()**
+190 -61
View File
@@ -5,7 +5,7 @@ The Basic Memory Cloud CLI provides seamless integration between local and cloud
## Overview
The cloud CLI enables you to:
- **Toggle cloud mode** - All regular `bm` commands work with cloud when enabled
- **Authenticate cloud access** - OAuth/API key credentials are stored locally for cloud operations
- **Project-scoped sync** - Each project independently manages its sync configuration
- **Explicit operations** - Sync only what you want, when you want
- **Bidirectional sync** - Keep local and cloud in sync with rclone bisync
@@ -17,6 +17,8 @@ Before using Basic Memory Cloud, you need:
- **Active Subscription**: An active Basic Memory Cloud subscription is required to access cloud features
- **Subscribe**: Visit [https://basicmemory.com/subscribe](https://basicmemory.com/subscribe) to sign up
- **Optional**: Cloud is optional. Local-first open-source usage continues without cloud.
- **OSS Discount**: Use code `{{OSS_DISCOUNT_CODE}}` for 20% off for 3 months.
If you attempt to log in without an active subscription, you'll receive a "Subscription Required" error with a link to subscribe.
@@ -38,7 +40,7 @@ If you attempt to log in without an active subscription, you'll receive a "Subsc
1. **Cloud-only** - Project exists on cloud, no local copy
2. **Cloud + Local (synced)** - Project has a local working directory that syncs
3. **Local-only** - Project exists locally (when cloud mode is disabled)
3. **Local-only** - Project exists locally and is not routed to cloud
**Example:**
@@ -48,9 +50,9 @@ If you attempt to log in without an active subscription, you'll receive a "Subsc
# - work: wants local sync at ~/work-notes
# - temp: cloud-only, no local sync needed
bm project add research --local-path ~/Documents/research
bm project add work --local-path ~/work-notes
bm project add temp # No local sync
bm project add research --cloud --local-path ~/Documents/research
bm project add work --cloud --local-path ~/work-notes
bm project add temp --cloud # No local sync
# Now you can sync individually (after initial --resync):
bm project bisync --name research
@@ -66,9 +68,9 @@ bm project bisync --name work
## Quick Start
### 1. Enable Cloud Mode
### 1. Authenticate Cloud Access
Authenticate and enable cloud mode:
Authenticate with cloud:
```bash
bm cloud login
@@ -76,11 +78,12 @@ bm cloud login
**What this does:**
1. Opens browser to Basic Memory Cloud authentication page
2. Stores authentication token in `~/.basic-memory/auth/token`
3. **Enables cloud mode** - all CLI commands now work against cloud
4. Validates your subscription status
2. Stores authentication tokens in `~/.basic-memory/basic-memory-cloud.json`
3. Validates your subscription status
4. Leaves routing behavior unchanged (auth only)
**Result:** All `bm project`, `bm tools` commands now work with cloud.
**Result:** Cloud credentials are available for cloud-routed commands.
Apply OSS discount code `{{OSS_DISCOUNT_CODE}}` during checkout to receive 20% off for 3 months.
### 2. Set Up Sync
@@ -104,10 +107,10 @@ Create projects with optional local sync paths:
```bash
# Create cloud project without local sync
bm project add research
bm project add research --cloud
# Create cloud project WITH local sync
bm project add research --local-path ~/Documents/research
bm project add research --cloud --local-path ~/Documents/research
# Or configure sync for existing project
bm project sync-setup research ~/Documents/research
@@ -117,7 +120,7 @@ bm project sync-setup research ~/Documents/research
When you add a project with `--local-path`:
1. Project created on cloud at `/app/data/research`
2. Local path stored in config: `cloud_projects.research.local_path = "~/Documents/research"`
2. Local path stored in config for that project (`cloud_sync_path`)
3. Local directory created if it doesn't exist
4. Bisync state directory created at `~/.basic-memory/bisync-state/research/`
@@ -179,7 +182,8 @@ bm cloud status
```
You should see:
- `Mode: Cloud (enabled)`
- `OAuth: token valid` (or missing/expired)
- `API Key: configured` (or not set)
- `Cloud instance is healthy`
- Instructions for project sync commands
@@ -187,16 +191,16 @@ You should see:
### Understanding Project Commands
**Key concept:** When cloud mode is enabled, use regular `bm project` commands (not `bm cloud project`).
**Key concept:** Use regular `bm project` commands (not `bm cloud project`).
```bash
# In cloud mode:
bm project list # Lists cloud projects
bm project add research # Creates cloud project
# Local route
bm project list --local
bm project add research ~/Documents/research
# In local mode:
bm project list # Lists local projects
bm project add research ~/Documents/research # Creates local project
# Cloud route
bm project list --cloud
bm project add research --cloud
```
### Creating Projects
@@ -204,7 +208,7 @@ bm project add research ~/Documents/research # Creates local project
**Use case 1: Cloud-only project (no local sync)**
```bash
bm project add temp-notes
bm project add temp-notes --cloud
```
**What this does:**
@@ -217,7 +221,7 @@ bm project add temp-notes
**Use case 2: Cloud project with local sync**
```bash
bm project add research --local-path ~/Documents/research
bm project add research --cloud --local-path ~/Documents/research
```
**What this does:**
@@ -251,11 +255,35 @@ bm project list
```
**What you see:**
- All projects in cloud (when cloud mode enabled)
- Local projects always
- Cloud projects when credentials are available
- Default project marked
- Project paths shown
- Route-related metadata (for example, local/cloud presence and sync info)
**Future:** Will show sync status (synced/not synced, last sync time).
Example shape (single row for dual-presence projects):
```text
Name Path Local Path Cloud Path CLI Default MCP (stdio)
main /basic-memory ~/basic-memory /basic-memory local local
specs /specs ~/dev/specs /specs cloud local
```
### When a Project Exists in Both Local and Cloud
Use routing flags to disambiguate command targets:
```bash
# Force local target for this command
bm project info main --local
bm project ls --name main --local
# Force cloud target for this command
bm project info main --cloud
bm project ls --name main --cloud
```
Default behavior for no-project, no-flag commands is local.
For MCP stdio, routing is always local.
## File Synchronization
@@ -363,24 +391,28 @@ bm project bisync --name research --dry-run
**Result:** Safe preview of sync operations.
### Advanced: List Remote Files
### Advanced: List Project Files by Route
**Use case:** See what files exist on cloud without syncing.
**Use case:** Inspect local or cloud project files explicitly.
```bash
# List all files in project
# List local project files (default target when no route flag is given)
bm project ls --name research
bm project ls --name research --local
# List cloud project files
bm project ls --name research --cloud
# List files in subdirectory
bm project ls --name research --path subfolder
bm project ls --name research --cloud --path subfolder
```
**What happens:**
1. Connects to cloud via rclone
2. Lists files in remote project path
1. Resolves route from flags (or local default when no route is given)
2. Lists files for the chosen project instance
3. No files transferred
**Result:** See cloud file listing.
**Result:** See file listing for the target route.
## Multiple Projects
@@ -390,9 +422,9 @@ bm project ls --name research --path subfolder
```bash
# Setup multiple projects
bm project add research --local-path ~/Documents/research
bm project add work --local-path ~/work-notes
bm project add personal --local-path ~/personal
bm project add research --cloud --local-path ~/Documents/research
bm project add work --cloud --local-path ~/work-notes
bm project add personal --cloud --local-path ~/personal
# Establish baselines
bm project bisync --name research --resync
@@ -417,12 +449,12 @@ bm project bisync --all # Coming soon
```bash
# Projects with sync
bm project add research --local-path ~/Documents/research
bm project add work --local-path ~/work
bm project add research --cloud --local-path ~/Documents/research
bm project add work --cloud --local-path ~/work
# Cloud-only projects
bm project add archive
bm project add temp-notes
bm project add archive --cloud
bm project add temp-notes --cloud
# Sync only the configured ones
bm project bisync --name research
@@ -433,20 +465,101 @@ bm project bisync --name work
**Result:** Fine-grained control over what syncs.
## Disable Cloud Mode
## Per-Project Cloud Routing (API Key)
Return to local mode:
Route individual projects through cloud using an API key. This lets you keep some projects local while others route through cloud.
### Setting Up API Key Auth
**Option A: Create a key in the web app, then save it locally:**
```bash
bm cloud set-key bmc_abc123...
```
**Option B: Create a key via CLI (requires OAuth login first):**
```bash
bm cloud login # One-time OAuth login
bm cloud create-key "my-laptop" # Creates key and saves it locally
```
The API key is account-level — it grants access to all your cloud projects. It's stored in `~/.basic-memory/config.json` as `cloud_api_key`.
### Setting Project Modes
```bash
# Route a project through cloud
bm project set-cloud research
# Revert to local mode
bm project set-local research
# View project modes
bm project list
```
**What happens:**
- `set-cloud`: validates the API key exists, then sets the project mode to `cloud` in config
- `set-local`: reverts the project to local mode (removes the mode entry from config)
- MCP tools and CLI commands for that project will route to `cloud_host/proxy` with the API key as Bearer token
### How It Works
When an MCP tool or CLI command runs for a cloud-mode project:
1. `get_client(project_name="research")` checks the project's mode in config
2. If mode is `cloud`, creates an HTTP client pointed at `cloud_host/proxy` with `Authorization: Bearer bmc_...`
3. If mode is `local` (default), uses the in-process ASGI transport as usual
**Routing priority** (highest to lowest):
1. Factory injection (cloud app, tests)
2. Explicit route override (`--local` / `--cloud`)
3. Per-project cloud mode (API key)
4. Local ASGI transport (default)
Route override environment variables:
- `BASIC_MEMORY_FORCE_LOCAL=true`
- `BASIC_MEMORY_FORCE_CLOUD=true`
- `BASIC_MEMORY_EXPLICIT_ROUTING=true`
No-project, no-flag CLI commands default to local routing.
### Configuration Example
```json
{
"projects": {
"personal": "/Users/me/notes",
"research": "/Users/me/research"
},
"project_modes": {
"research": "cloud"
},
"cloud_api_key": "bmc_abc123...",
"cloud_host": "https://cloud.basicmemory.com",
"default_project": "personal"
}
```
In this example, `personal` stays local and `research` routes through cloud. Projects not listed in `project_modes` default to local.
### Sync Behavior
Cloud-mode projects are automatically skipped during local file sync (background sync and file watching). Their files live on the cloud instance, not locally.
## OAuth Logout
```bash
bm cloud logout
```
**What this does:**
1. Disables cloud mode in config
2. All commands now work locally
3. Auth token remains (can re-enable with login)
1. Removes stored OAuth token(s)
2. Does not change per-project route configuration
3. Does not change command routing defaults
**Result:** All `bm` commands work with local projects again.
**Result:** OAuth session is cleared. API-key-based routing still works if `cloud_api_key` is configured.
## Filter Configuration
@@ -653,12 +766,20 @@ If instance is down, wait a few minutes and retry.
## Command Reference
### Cloud Mode Management
### Cloud Authentication
```bash
bm cloud login # Authenticate and enable cloud mode
bm cloud logout # Disable cloud mode
bm cloud status # Check cloud mode and instance health
bm cloud login # Authenticate and store OAuth credentials
bm cloud logout # Remove stored OAuth credentials
bm cloud status # Check auth state and instance health
bm cloud promo --off # Disable CLI cloud promo notices
```
### API Key Management
```bash
bm cloud set-key <key> # Save a cloud API key (bmc_ prefixed)
bm cloud create-key <name> # Create API key via cloud API (requires OAuth login)
```
### Setup
@@ -669,16 +790,22 @@ bm cloud setup # Install rclone and configure credentials
### Project Management
When cloud mode is enabled:
```bash
bm project list # List cloud projects
bm project add <name> # Create cloud project (no sync)
bm project add <name> --local-path <path> # Create with local sync
bm project list --local # Local project list
bm project list --cloud # Cloud project list
bm project add <name> --cloud # Create cloud project (no sync)
bm project add <name> --cloud --local-path <path> # Create with local sync
bm project sync-setup <name> <path> # Add sync to existing project
bm project rm <name> # Delete project
```
### Per-Project Routing
```bash
bm project set-cloud <name> # Route project through cloud (requires API key)
bm project set-local <name> # Revert project to local mode
```
### File Synchronization
```bash
@@ -697,18 +824,20 @@ bm project bisync --name <project> --verbose
bm project check --name <project>
bm project check --name <project> --one-way
# List remote files
bm project ls --name <project>
bm project ls --name <project> --path <subpath>
# List project files by route
bm project ls --name <project> # Default target: local
bm project ls --name <project> --local
bm project ls --name <project> --cloud
bm project ls --name <project> --cloud --path <subpath>
```
## Summary
**Basic Memory Cloud uses project-scoped sync:**
1. **Enable cloud mode** - `bm cloud login`
1. **Authenticate cloud access** - `bm cloud login`
2. **Install rclone** - `bm cloud setup`
3. **Add projects with sync** - `bm project add research --local-path ~/Documents/research`
3. **Add projects with sync** - `bm project add research --cloud --local-path ~/Documents/research`
4. **Preview first sync** - `bm project bisync --name research --resync --dry-run`
5. **Establish baseline** - `bm project bisync --name research --resync`
6. **Daily workflow** - `bm project bisync --name research`
+91
View File
@@ -0,0 +1,91 @@
# Cloud Semantic Search Value (Customer-Facing Technical Story)
This document explains why teams should buy cloud semantic search even when local search exists.
## Core Promise
Markdown files remain the source of truth in both local and cloud modes.
- Files are portable.
- Search indexes are derived and rebuildable.
- You never get locked into proprietary document storage.
## The Customer Problem
Teams paying for cloud are usually not optimizing for "can this run locally." They are optimizing for:
- finding the right note the first time,
- keeping retrieval quality high as note volume grows,
- avoiding search slowdowns while content is actively changing,
- getting consistent results across users, agents, and sessions.
## Why Cloud Is the Aspirin
Cloud semantic search is the immediate pain reliever because it fixes the problems users feel right now.
### 1) Better hit rate on real queries
Cloud uses stronger managed embeddings than the default local model, which improves semantic recall for paraphrases and vague questions.
Customer outcome:
- fewer "I know this exists but search missed it" moments,
- less query rewording,
- faster time to answer.
### 2) Better behavior under active workloads
Cloud indexing runs out of band in workers, so indexing does not compete with interactive read/write traffic.
Customer outcome:
- stable search responsiveness during heavy updates,
- fresher semantic results shortly after edits,
- less user-visible performance variance.
### 3) Better consistency for shared knowledge
Cloud retrieval runs against a centralized tenant index, so teams and agents resolve against the same semantic state.
Customer outcome:
- fewer "works on my machine" search differences,
- more predictable agent behavior across environments,
- easier cross-user collaboration on large knowledge bases.
### 4) Better quality at higher scale
With Postgres + `pgvector` per tenant, cloud can sustain larger note collections and higher query volumes than typical local setups.
Customer outcome:
- confidence as repositories grow to tens of thousands of notes,
- less need for user-side tuning,
- fewer quality regressions as usage increases.
## Local Is the Vitamin
Local semantic search still matters and should stay strong.
- offline use,
- privacy-first operation,
- no cloud dependency,
- user-controlled runtime.
It compounds long-term ownership and resilience, but does not remove the immediate pain points cloud solves for teams at scale.
## Recommended Messaging
One-liner:
"Cloud semantic search is the aspirin: it fixes retrieval quality and performance pain now. Local semantic search is the vitamin: it builds long-term control and resilience."
Long form:
"Basic Memory keeps markdown as the source of truth everywhere. Local gives privacy and offline control. Cloud adds immediate, measurable improvements in search quality, consistency, and responsiveness for teams and agents running at scale."
## Packaging Guidance
- Base: local FTS plus optional local semantic search.
- Cloud value: higher semantic quality, stable performance under load, and consistent team-wide retrieval.
- Keep interfaces pluggable (`EmbeddingProvider`, vector backend protocol) so implementation can evolve without changing user workflows.
+138
View File
@@ -0,0 +1,138 @@
# MCP UI Bakeoff - Instructions & Test Plan
Last updated: 2026-02-02
## Scope
Compare three presentation paths for Basic Memory MCP tools:
1. **ToolUI (React)** via MCP App resources.
2. **MCPUI Python SDK** embedded UI resources (legacy host path).
3. **ASCII/ANSI** output for TUI clients.
This doc is the running instruction set and test plan. Update as implementation progresses.
---
## Prerequisites
- Repo: `basic-memory` (worktree: `basic-memory-mcp-ui-poc`)
- Node for toolui build (already used for POC)
- Python 3.12+ with `uv`
Optional (for MCPUI Python SDK path):
- Local repo: `/Users/phernandez/dev/mcp-ui`
- Install the server SDK into the Basic Memory venv:
- `uv pip install -e /Users/phernandez/dev/mcp-ui/sdks/python/server`
---
## Build / Refresh Steps
### ToolUI React bundle
```bash
cd ui/tool-ui-react
npm install
npm run build
```
This regenerates:
- `src/basic_memory/mcp/ui/html/search-results-tool-ui.html`
- `src/basic_memory/mcp/ui/html/note-preview-tool-ui.html`
---
## How to Run the MCP Server
```bash
basic-memory mcp --transport stdio
```
Optional to pick UI variant for MCP App resources:
```bash
export BASIC_MEMORY_MCP_UI_VARIANT=tool-ui # or vanilla | mcp-ui
```
---
## Test Cases
### 1) MCP App Resource UI (toolui / vanilla / mcpui)
Tools:
- `search_notes`
- `read_note`
Expect:
- Tool meta points to `ui://basic-memory/search-results` and `ui://basic-memory/note-preview`
- Resource content differs by `BASIC_MEMORY_MCP_UI_VARIANT`
- Variantspecific URIs also available:
- `ui://basic-memory/search-results/vanilla`
- `ui://basic-memory/search-results/tool-ui`
- `ui://basic-memory/search-results/mcp-ui`
- `ui://basic-memory/note-preview/vanilla`
- `ui://basic-memory/note-preview/tool-ui`
- `ui://basic-memory/note-preview/mcp-ui`
Manual check:
- Trigger tool in MCPAppcapable host and confirm UI renders.
---
### 2) Text / JSON Output Modes
Tools:
- `search_notes(output_format="text" | "json")`
- `read_note(output_format="text" | "json")`
- `write_note(output_format="text" | "json")`
- `edit_note(output_format="text" | "json")`
- `recent_activity(output_format="text" | "json")`
- `list_memory_projects(output_format="text" | "json")`
- `create_memory_project(output_format="text" | "json")`
- `delete_note(output_format="text" | "json")`
- `move_note(output_format="text" | "json")`
- `build_context(output_format="json" | "text")`
Expect:
- `text` mode preserves existing human-readable responses.
- `json` mode returns structured dict/list payloads for machine-readable clients.
Automated:
- `uv run pytest test-int/mcp/test_output_format_json_integration.py`
---
### 3) MCPUI Python SDK (embedded UI resource)
Tools (embedded resource responses):
- `search_notes_ui` (MCPUI SDK)
- `read_note_ui` (MCPUI SDK)
Expected output:
- Tool response content contains an EmbeddedResource (`type: "resource"`)
- `mimeType` is `text/html`
- `_meta` includes:
- `mcpui.dev/ui-preferred-frame-size`
- `mcpui.dev/ui-initial-render-data`
Manual check:
- Render tool responses using `UIResourceRenderer` (legacy host flow).
Automated (if SDK installed):
- `uv run pytest test-int/mcp/test_ui_sdk_integration.py`
---
## Bakeoff Notes Template
Fill in after running:
- ToolUI (React): __
- MCPUI SDK (embedded): __
- Text/JSON modes: __
Decision + rationale: __
+344
View File
@@ -0,0 +1,344 @@
# Post-v0.18.0 Test Plan and Acceptance Criteria
## Goal
Define a complete validation plan for all major features merged after `v0.18.0`, combining:
- Coverage-gap-driven automated tests
- Real MCP server integration tests (no mocks for target flows)
- Manual MCP verification via LLM-driven tool calls
This plan is based on commits in `v0.18.0..HEAD` and the latest `just check` coverage output.
## Scope Window
- Start tag: `v0.18.0` (2026-01-28)
- End: current `main`
- Change volume: 12 feature commits + 14 bug-fix commits (+ release chores/hotfixes)
## Execution Strategy
1. Stabilize all feature-level acceptance criteria in automated tests first.
2. Add black-box MCP integration tests for semantic search + schema (real server startup).
3. Run manual MCP tool-call verification to confirm real UX and routing behavior.
4. Re-run full gate: `just check` + targeted integration packs.
## Global Quality Gates
- Feature criteria below must all pass.
- No regressions in existing suites.
- Coverage improves in targeted low-coverage feature modules.
- SQLite and Postgres parity for search/semantic features.
## Priority Coverage Gaps (from latest run)
These are the most important post-`v0.18.0` feature modules currently under-covered:
- `src/basic_memory/mcp/tools/schema.py` (27%)
- `src/basic_memory/mcp/clients/schema.py` (36%)
- `src/basic_memory/mcp/tools/ui_sdk.py` (43%)
- `src/basic_memory/mcp/tools/search.py` (73%)
- `src/basic_memory/repository/postgres_search_repository.py` (63%)
- `src/basic_memory/mcp/async_client.py` (82%)
- `src/basic_memory/api/v2/routers/schema_router.py` (80%)
## Feature Acceptance Criteria and Test Plan
### 1) Schema System (`c97733d`) — DONE
### Acceptance criteria
- `schema_validate`, `schema_infer`, and `schema_diff` produce consistent outcomes across CLI/API/MCP for the same fixture set.
- Strict validation fails deterministically on required-field/type violations.
- Validation warnings are stable and machine-readable in non-strict mode.
- Inference output is deterministic for unchanged input corpus.
- Drift diff output is deterministic and identifies missing/extra/type-mismatch fields correctly.
### Existing coverage anchor points
- `tests/schema/*`
- `tests/api/v2/test_schema_router.py`
- `test-int/test_schema/*`
### Gaps to close — DONE
- ~~MCP schema tool branches (`src/basic_memory/mcp/tools/schema.py`)~~ — 18 tests in `tests/mcp/test_tool_schema.py`
- ~~MCP schema client behavior (`src/basic_memory/mcp/clients/schema.py`)~~ — `tests/mcp/test_client_schema.py`
- ~~Schema router error-path branches (`src/basic_memory/api/v2/routers/schema_router.py`)~~ — `tests/api/v2/test_schema_router.py`
### Planned additions — DONE
- ~~Add MCP tool tests for `schema_validate` strict + non-strict result shapes.~~ **DONE**
- ~~Add MCP tool tests for `schema_infer` with explicit `entity_type` and inferred type fallback.~~ **DONE**
- ~~Add MCP tool tests for `schema_diff` empty-diff and non-empty-diff paths.~~ **DONE**
- ~~Add API tests for schema router invalid payload/edge error handling.~~ **DONE**
- Add integration test that starts MCP server and calls schema tools end-to-end on fixture notes. — deferred to backlog item 4.
### 2) Semantic Search (`0777879`, `1428d18`, `344e651`) — DONE
### Acceptance criteria
- `search_type=text|vector|hybrid` returns expected ranked results on canonical semantic corpus.
- Missing semantic extras fail fast with actionable install guidance.
- Reindex and provider/model changes produce valid vectors without dimension mismatch.
- SQLite and Postgres produce equivalent behavior for semantic modes on the same dataset.
- Generated-column migration path is valid on SQLite environments in use.
### Existing coverage anchor points
- `tests/repository/test_sqlite_vector_search_repository.py`
- `tests/repository/test_postgres_search_repository.py`
- `tests/services/test_semantic_search.py`
- `tests/mcp/test_tool_search.py`
- `test-int/test_search_performance_benchmark.py`
### Gaps to close — DONE
- ~~Uncovered Postgres vector/hybrid branches~~ — 20 tests in `tests/repository/test_postgres_search_repository_unit.py` + 5 integration tests in `test-int/semantic/test_semantic_coverage.py`
- ~~MCP search semantic/output branches~~ — expanded `tests/mcp/test_tool_search.py`
### Planned additions — DONE
- ~~Expand Postgres repository tests for vector query composition edge cases.~~ **DONE**
- ~~Expand Postgres repository tests for hybrid fusion ranking and pagination branches.~~ **DONE**
- ~~Expand Postgres repository tests for embedding/provider error handling branches.~~ **DONE**
- ~~Expand MCP search tool tests for vector/hybrid output formatting branches.~~ **DONE**
- ~~Expand MCP search tool tests for semantic-disabled and missing-dependency failures.~~ **DONE**
- Add MCP integration tests that start server and execute semantic `search_notes` tool calls. — deferred to backlog item 4.
### Semantic search quality benchmarks (NEW)
Full benchmark suite in `test-int/semantic/` covering 5 backend×provider combinations:
- `sqlite-fts`, `sqlite-fastembed`, `postgres-fts`, `postgres-fastembed`, `postgres-openai`
- Quality metrics: hit@1, recall@5, MRR@10 with per-query timing
- Realistic corpus with cross-topic vocabulary overlap (240 notes, 4 topics)
- Rich CLI viewer: `just semantic-report`
- JSON artifact output: `just test-semantic-report`
Key finding: **FastEmbed (384-d local ONNX) matches or exceeds OpenAI (1536-d) quality at 30x lower latency.** Recommending FastEmbed as default for both local and cloud deployments.
### 3) Per-Project Local/Cloud Routing + API Key Auth (`d84708c`, `ed94877`, `312662f`) — DONE
### Acceptance criteria
- Project mode (`local`/`cloud`) persists and displays correctly.
- Routing selects ASGI for local projects and HTTP+Bearer for cloud projects.
- Cloud project without key fails with explicit remediation (`cloud set-key`/`cloud create-key`).
- Resolution precedence is correct (factory > force-local > per-project cloud > global fallback > local).
- Watch/sync only run for local projects.
### Existing coverage anchor points
- `tests/mcp/test_async_client_modes.py`
- `tests/cli/test_project_set_cloud_local.py`
- `tests/mcp/test_project_context.py`
- `tests/test_project_resolver.py`
- `tests/sync/test_watch_service_reload.py`
### Gaps to close — DONE
- ~~Cloud routing branch gaps in `src/basic_memory/mcp/async_client.py`~~ — expanded `tests/mcp/test_async_client_modes.py`
### Planned additions — DONE
- ~~Add branch-focused tests for all unresolved routing branches in `get_client()`.~~ **DONE**
- Add MCP integration scenario with mixed local/cloud project config — deferred to backlog item 4.
### 4) Project-Prefixed Permalinks + Memory URL Routing (`545804f`) — DONE
### Acceptance criteria
- Project-prefixed permalinks are generated consistently on create/update/import flows.
- Memory URLs resolve to the correct project/entity even with duplicate note titles.
- `read_note`, `search`, `build_context`, write/edit/move flows preserve project identity correctly.
- Link resolution remains correct for context-aware wikilinks.
### Existing coverage anchor points
- `tests/utils/test_permalink_formatting.py`
- `tests/mcp/test_tool_read_note.py`
- `tests/mcp/test_tool_search.py`
- `tests/services/test_context_service.py`
- `test-int/mcp/test_read_note_integration.py`
### Gaps to close
- No major coverage alarm in report, but keep as regression-critical due broad impact surface.
### Planned additions — DONE
- ~~Add one integration test with colliding titles across two projects and assert URL routing invariants.~~ **DONE**`test-int/mcp/test_permalink_collision_integration.py` (2 tests: collision across projects + memory:// URL routing with project prefix)
### 5) MCP UI Variants + TUI Output (`8bc03d1`) — DONE
### Acceptance criteria
- UI resource variant selection (`tool-ui`, `vanilla`, `mcp-ui`) follows env configuration.
- `search_notes` and `read_note` expose expected resource metadata for UI hosts.
- `ascii`/`ansi` outputs are deterministic and stable for terminal clients.
### Existing coverage anchor points
- `tests/mcp/test_tool_contracts.py`
- `test-int/mcp/test_output_format_json_integration.py`
- `test-int/mcp/test_ui_sdk_integration.py`
### Gaps to close — DONE
- ~~`src/basic_memory/mcp/tools/ui_sdk.py` branch coverage~~ — `tests/mcp/test_ui_sdk.py`
- ~~`src/basic_memory/mcp/ui/sdk.py` and `src/basic_memory/mcp/ui/templates.py` branch coverage~~ — `tests/mcp/test_ui_templates.py` + `tests/mcp/test_ui_resources.py`
### Planned additions — DONE
- ~~Add unit tests for UI SDK metadata generation and template selection branches.~~ **DONE** — 31 tests
- ~~Add integration assertion for variant-specific resource URIs and metadata payload shape.~~ **DONE**
### 6) Watch Command (`8df88e4`) — DONE
### Acceptance criteria
- `basic-memory watch` starts and processes create/update/delete events.
- Watch restart/reload path does not duplicate watchers.
- Cloud-mode projects are excluded from active watcher set.
### Existing coverage anchor points
- `tests/cli/test_watch.py`
- `tests/sync/test_coordinator.py`
- `tests/sync/test_watch_service_reload.py`
### Planned additions — DONE
- ~~Add one stress-style integration test for rapid file changes and watcher stability.~~ **DONE**`tests/sync/test_watch_service_stress.py` (3 tests: 50-file batch, mixed add/modify/delete batch, rapid modifications to same file)
### 7) CLI JSON Output (`a47c9c0`) — DONE
### Acceptance criteria
- `--format json` returns valid JSON with stable keys for success paths.
- Error paths also return JSON-shaped output with correct non-zero exits.
- Default human output remains unchanged.
### Existing coverage anchor points
- `tests/cli/test_cli_tool_json_output.py`
- `test-int/cli/test_cli_tool_json_integration.py`
### Planned additions — DONE
- ~~Add one failure-path integration test per high-use tool command.~~ **DONE**`test-int/cli/test_cli_tool_json_failure_integration.py` (4 tests: read-note not found, write-note missing content, write→read roundtrip, recent-activity empty project)
### 8) Search/Edit and Metadata Fixes (`530cbac`, `f1d50c2`, `8838571`, `009e849`) — DONE
### Acceptance criteria
- Metadata filters produce consistent results on SQLite and Postgres.
- `tag:` shorthand works alone and with mixed query terms.
- Fast write/edit paths preserve `external_id` and metadata integrity.
### Existing coverage anchor points
- `tests/repository/test_metadata_filters.py`
- `tests/repository/test_search_repository.py`
- `tests/services/test_search_service.py`
### Planned additions — DONE
- ~~Add Postgres-specific metadata filter edge-case tests to mirror SQLite assertions exactly.~~ **DONE**`tests/repository/test_metadata_filters_edge_cases.py` (6 tests: missing field, AND logic, contains single-element array, nested path missing intermediate, $gte/$lte boundaries, $between inclusive — all pass on both SQLite and Postgres)
### 9) Compatibility and Hotfix Regression Pack (`c46d7a6`, `a0e754b`, `343a6e1`, `24ca5f6`, `e3ced49`, `8489a3d`, `b609c4e`, `f6e0a5b`, `7624a20`)
### Acceptance criteria
- Legacy endpoints required by older CLI versions function without `405` (`GET /projects/projects`, `POST /projects/projects`, `POST /projects/config/sync`).
- Entity creation conflicts map to conflict status (not 500).
- `recent_activity` prompt defaults are correct.
- No spurious `metadata: {}` in serialized frontmatter.
- Tigris/rclone uses global consistency headers for all transaction types.
- `bm --version` fast path avoids heavy import path and remains responsive.
- Default SQLite DB path is isolated by config dir.
### Gaps to close
- ~~Commits with no direct tests added (`c46d7a6`, `344e651`, `f6e0a5b`) need explicit regression tests.~~ **DONE**
### Planned additions — DONE
- ~~Add API compat test covering all legacy endpoint methods and payloads.~~ **DONE**`test_legacy_v1_add_project_endpoint`, `test_legacy_v1_sync_config_endpoint`
- ~~Add CLI fast-path test for `--version` import behavior/performance guard.~~ **DONE**`test_bm_version_does_not_import_heavy_modules`
- ~~Add empty metadata serialization regression test.~~ **DONE**`test_schema_to_markdown_empty_metadata_no_metadata_key`
- Add migration safety test for SQLite generated columns (`VIRTUAL` expectation) — deferred, low risk.
## MCP Manual Verification Plan (LLM Tool Calls)
Run after automated tests pass.
### Setup
- Start MCP server: `basic-memory mcp --transport stdio`
- Use an MCP-capable client and issue tool calls directly.
### Manual scenarios
- Schema: call `schema_validate`, `schema_infer`, and `schema_diff` on known fixtures.
- Schema: verify error and success payloads match acceptance criteria.
- Semantic search: call `search_notes` with `search_type=text|vector|hybrid`.
- Semantic search: verify ranking relevance on semantic fixture queries.
- Routing: call tools with explicit project on mixed local/cloud setup.
- Routing: verify success/failure paths with and without API key.
- Permalink routing: read/write/search notes across projects with colliding titles.
- Permalink routing: verify memory URL routing correctness.
- UI/TUI: call `search_notes` and `read_note` with UI variants and `output_format=text|json`.
- UI/TUI: verify payload/resource format and metadata completeness.
## Implementation Backlog (Ordered)
1. ~~Fill schema MCP/client/router coverage gaps.~~ **DONE** — 18 tests in `test_tool_schema.py` + `test_client_schema.py`
2. ~~Fill semantic search MCP + Postgres repository gaps.~~ **DONE** — 20 tests in `test_postgres_search_repository_unit.py` + `test_tool_search.py`
3. ~~Add compatibility regression tests (legacy endpoints, migration, version fast path).~~ **DONE** — 5 tests across 3 files (see below)
4. ~~Add feature-level integration tests (permalinks, watch, CLI JSON, metadata filters).~~ **DONE** — 15 tests across 4 files (see items 4, 6, 7, 8 above)
5. ~~Expand UI SDK and template branch tests.~~ **DONE** — 31 tests in `test_ui_templates.py` + `test_ui_sdk.py` + `test_ui_resources.py`
6. ~~Run full gate and capture results in a short release readiness summary.~~ **DONE** — see results below
### Full Gate Results (`just check`)
| Phase | Result |
|-------|--------|
| lint | PASS |
| format | PASS |
| typecheck | PASS |
| Unit tests (SQLite) | 1788 passed, 15 skipped |
| Integration tests (SQLite) | 243 passed, 4 skipped, 10 deselected |
| Unit tests (Postgres) | 1760 passed, 28 skipped |
| Integration tests (Postgres) | 234 passed, 13 skipped, 10 deselected |
**0 failures. 10 deselected = semantic benchmark tests (run separately via `just test-semantic`).**
### Item 3 Details — Compatibility Regression Tests
| Test | File | What it covers |
|------|------|----------------|
| `test_legacy_v1_add_project_endpoint` | `tests/api/v2/test_project_router.py` | POST `/projects/projects` legacy route reachable (idempotent path) |
| `test_legacy_v1_sync_config_endpoint` | `tests/api/v2/test_project_router.py` | POST `/projects/config/sync` legacy route reachable |
| `test_bm_version_does_not_import_heavy_modules` | `tests/cli/test_cli_exit.py` | `bm --version` fast path does not load `basic_memory.mcp` |
| `test_schema_to_markdown_empty_metadata_no_metadata_key` | `tests/markdown/test_entity_parser_error_handling.py` | `schema_to_markdown()` with `entity_metadata={}` emits no `metadata:` key |
| `test_legacy_v1_list_projects_endpoint` | `tests/api/v2/test_project_router.py` | (pre-existing) GET `/projects/projects` legacy route |
**Suite totals after item 3: 1764 passed, 15 skipped, 0 failures.**
## Suggested Commands
- Full suite: `just check`
- Fast loop: `just fast-check`
- E2E consistency: `just doctor`
- SQLite focused: `just test-sqlite`
- Postgres focused: `just test-postgres`
- Schema integration: `pytest test-int/test_schema -q`
- Semantic + repo focus: `pytest tests/repository/test_postgres_search_repository.py tests/mcp/test_tool_search.py tests/services/test_semantic_search.py -q`
- MCP integration focus: `pytest test-int/mcp -q`
## Exit Criteria for This Plan
- All feature acceptance criteria above are validated.
- All identified high-priority coverage gaps are addressed or explicitly documented as intentional.
- Manual MCP verification scenarios complete with no P0/P1 findings.
+209
View File
@@ -0,0 +1,209 @@
# Semantic Search Manual Test Log
## Overview
Manual test session for semantic (vector) search on the main project.
- Date: 2026-02-15
- Database: ~/.basic-memory/memory.db (SQLite)
- Entities: 456 embedded, 2714 vector chunks
- Search index: 2390 FTS entries
- Embedding model: default (384-dim, sqlite-vec)
## Test Plan
1. **Search Type Routing** — verify vector/hybrid/text dispatch, invalid search_type handling
2. **Conceptual Queries** — natural language where vector should beat FTS
3. **Keyword Queries** — exact terms where FTS should be strong
4. **Hybrid Ranking** — queries where both FTS and vector contribute
5. **Result Types** — entities, observations, relations in vector results
6. **Filters + Vector** — combine vector with types/entity_types/after_date
7. **Edge Cases** — short queries, long queries, empty, special chars, no-match
8. **Pagination** — page > 1, page_size respected
---
## Test Results
### Test 1: Search Type Routing
#### 1a: search_type="semantic" (invalid value)
- **Input:** query="how does the knowledge graph work", search_type="semantic"
- **Expected:** error or explicit fallback
- **Actual:** Silently falls through to text search (else branch in search.py:430)
- **Verdict:** BUG — should either be a recognized alias for "vector" or return an error
#### 1b: search_type="vector"
- **Input:** query="keeping AI context between sessions", search_type="vector"
- **Actual:** 5 results, scores ~0.58-0.59, found "Maintaining context across conversation boundaries" observation
- **Verdict:** PASS
#### 1c: search_type="text" with conceptual query
- **Input:** query="keeping AI context between sessions", search_type="text"
- **Actual:** 0 results (no exact keyword match)
- **Verdict:** PASS (expected — FTS requires token overlap)
#### 1d: search_type="hybrid" with conceptual query
- **Input:** query="keeping AI context between sessions", search_type="hybrid"
- **Actual:** 5 results, same ranking as vector (FTS contributed nothing here)
- **Verdict:** PASS
#### 1e: search_type="text" with keyword query
- **Input:** query="OAuth authentication", search_type="text"
- **Actual:** 3 results — AUTH.md Supabase OAuth, OAuth Rip-and-Replace, OAuth Integration Analysis
- **Verdict:** PASS
#### 1f: search_type="vector" with keyword query
- **Input:** query="OAuth authentication", search_type="vector"
- **Actual:** Same top results as text (keyword-rich content also scores well in vector space)
- **Verdict:** PASS
---
### Test 2: Conceptual Queries (vector advantage)
#### 2a: Natural language question
- **Input:** query="why do AI assistants forget things", search_type="vector"
- **Actual:** 5 results — Manual Testing Session, "Balance security and usability" observation, "Tools should match thought patterns" observation. Scores ~0.56-0.57
- **Vector advantage:** Found conceptually related content despite no exact keyword overlap
- **Verdict:** PASS
#### 2b: Same query, text search
- **Input:** query="why do AI assistants forget things", search_type="text"
- **Actual:** 1 result — "What is Basic Memory?" (likely matched on "AI" token)
- **Verdict:** PASS (demonstrates vector advantage — text barely matched)
#### 2c: Domain concept with no jargon
- **Input:** query="pricing strategy for cloud product", search_type="vector"
- **Actual:** 3 results — SPEC-16 MCP Cloud Service Consolidation, knowledge architecture observation, Visual Knowledge Spaces relation. Scores ~0.56-0.57
- **Verdict:** PASS (found cloud-related content conceptually)
#### 2d: Technical concept, long query
- **Input:** query="SQLite performance optimization WAL mode concurrent writes", search_type="vector"
- **Actual:** 3 results — SPEC-11 API Performance Optimization, Real-Time Updates with WebSockets, marketing status update. Scores ~0.55-0.58
- **Verdict:** PASS (found performance-related content)
---
### Test 3: Keyword Queries (FTS strength)
#### 3a: Exact term match — "OAuth authentication"
- **Text:** 3 results with high relevance (exact matches in titles)
- **Vector:** Same top results (keyword overlap helps vector too)
- **Verdict:** PASS — FTS and vector converge on keyword-rich queries
#### 3b: "OAuth" single keyword, hybrid mode
- **Input:** query="OAuth", search_type="hybrid"
- **Actual:** 5 results — Basic Memory Coding Guide, AI Collaboration Examples, SPEC-18, daily note, Manual Testing Session. FTS + vector blended. Scores ~0.016-0.032
- **Note:** Top hybrid result is "Basic Memory Coding Guide" not an OAuth-specific doc — suggests hybrid scoring may dilute strong FTS matches
- **Verdict:** PASS but hybrid ranking questionable for single-keyword queries
---
### Test 4: Hybrid Ranking
#### 4a: Hybrid vs vector on "OAuth authentication"
- **Hybrid with entity_types=["entity"]:** 5 results — RLS Implementation Lessons, Cloud Readiness Assessment, AUTH.md OAuth, Core Service Implementation, OAuth Rip-and-Replace. Scores ~0.016-0.023
- **Vector with entity_types=["entity"]:** 5 results — Core Service Implementation, SPEC-13 CLI Auth, Coding Guide, Authentication Service, ADR Production Auth. Scores ~0.55-0.60
- **Observation:** Hybrid surfaces different top results than vector-only. Hybrid found RLS and Cloud Readiness docs that vector didn't prioritize. Different ranking is expected from RRF fusion.
- **Verdict:** PASS — hybrid produces meaningfully different ranking
---
### Test 5: Result Types
#### 5a: Vector returns all result types
- **Input:** query="keeping AI context between sessions", search_type="vector"
- **Entities:** SPEC-18 AI Memory Management Tool (type=entity)
- **Relations:** Prompt Builder integrates_with (type=relation)
- **Observations:** "Translation layer is key" (type=observation), "Maintaining context across conversation boundaries" (type=observation)
- **Verdict:** PASS — all three types appear in vector results
#### 5b: Observations carry metadata
- **Observation result:** category="challenge", content="Maintaining context across conversation boundaries", from_entity="research/ai-knowledge-management-research"
- **Verdict:** PASS — category, content, from_entity, tags all present
#### 5c: Relations carry link info
- **Relation result:** relation_type="integrates_with", from_entity="development/features/prompt-builder...", to_entity (present but truncated in some)
- **Verdict:** PASS — relation metadata present
---
### Test 6: Filters + Vector Search
#### 6a: entity_types=["entity"] with vector
- **Input:** query="OAuth authentication", search_type="vector", entity_types=["entity"]
- **Actual:** 5 results, all type="entity" (Core Service Implementation, SPEC-13, Coding Guide, Authentication Service, ADR Auth)
- **Verdict:** PASS — filter correctly restricts to entities only
#### 6b: types=["note"] with vector
- **Input:** query="OAuth authentication", search_type="vector", types=["note"]
- **Actual:** Same 5 results (all have entity_type="note" in metadata)
- **Verdict:** PASS — types filter works with vector search
#### 6c: after_date with vector
- **Input:** query="OAuth authentication", search_type="vector", after_date="2025-06-01"
- **Actual:** 3 results — Core Service Implementation, Cloud Web App analysis observation, SPEC-13. Filtered out older OAuth docs.
- **Verdict:** PASS — date filter applied correctly
#### 6d: entity_types=["entity"] with hybrid
- **Input:** query="OAuth authentication", search_type="hybrid", entity_types=["entity"]
- **Actual:** 5 results, all type="entity" — RLS lessons, Cloud Readiness, AUTH.md OAuth, Core Service, OAuth Rip-and-Replace
- **Verdict:** PASS — filter works with hybrid mode too
#### 6e: types=["entity"] with vector (WRONG filter name)
- **Input:** query="OAuth authentication", search_type="vector", types=["entity"]
- **Actual:** 0 results
- **Note:** `types` filters by entity_type metadata (e.g., "note", "person"), NOT by SearchItemType. Using types=["entity"] looks for entity_type="entity" which few/no notes have. This is a UX confusion point — the param names are ambiguous.
- **Verdict:** PASS (correct behavior) but USABILITY ISSUE — easy to confuse types vs entity_types
---
### Test 7: Edge Cases
#### 7a: Single character query
- **Input:** query="x", search_type="vector"
- **Actual:** 3 results — "Self-contained application bundle" observation, Non-Markdown File Support relation, quick-win-tools entity. Scores ~0.57-0.59
- **Note:** Single character still produces an embedding and returns results. Quality is low/random as expected.
- **Verdict:** PASS (no crash, returns results)
#### 7b: Whitespace-only query
- **Input:** query=" ", search_type="vector"
- **Actual:** 0 results
- **Verdict:** PASS (handled gracefully — _check_vector_eligible strips and rejects empty)
#### 7c: Query with no relevant content
- **Input:** query="quantum computing blockchain", search_type="vector"
- **Actual:** 3 results — Inter-Agent Communication relation, Self-contained bundle observation, JSON-LD interop observation. Scores ~0.54
- **Note:** Still returns results because vector search always finds nearest neighbors. Scores are lower (~0.54) than relevant queries (~0.58-0.60). No relevance threshold applied.
- **Verdict:** PASS (expected behavior) but NOTE — no relevance cutoff means irrelevant queries always return something
---
### Test 8: Pagination
#### 8a: Vector search page 2
- **Input:** query="keeping AI context between sessions", search_type="vector", page=2, page_size=3
- **Actual:** 3 results on page 2, current_page=2. Different results from page 1. Top: "Maintaining context across conversation boundaries" observation (score 0.587)
- **Note:** Interestingly, page 2 had a higher-scoring result than some page 1 results. This may indicate pagination doesn't sort globally — it might be paginating within a pre-scored set.
- **Verdict:** PASS (pagination works) but POSSIBLE ISSUE — result ordering across pages needs investigation
---
## Summary
### Passing Tests: 20/21
### Bugs Found
1. **search_type="semantic" silently falls through** (Test 1a) — Invalid search_type values fall to the `else` branch and default to text search without any warning. Should either alias "semantic" to "vector" or raise an error.
### Usability Issues
2. **types vs entity_types confusion** (Test 6e) — `types` filters by entity_type metadata (note, person, etc.) while `entity_types` filters by SearchItemType (entity, observation, relation). The naming is ambiguous and easy to mix up.
3. **No relevance threshold** (Test 7c) — Vector search always returns nearest neighbors even for completely irrelevant queries. Consider adding a minimum score threshold or at least documenting expected score ranges.
4. **Hybrid ranking for single keywords** (Test 3b) — Hybrid mode on simple keyword queries produced less intuitive rankings than pure FTS or pure vector. The RRF fusion may dilute strong FTS signals.
### Observations
- Vector search successfully finds conceptually related content that FTS misses entirely
- Score ranges: relevant queries ~0.56-0.60, irrelevant queries ~0.54 (narrow spread)
- All three result types (entity, observation, relation) appear correctly in vector results
- Filters (entity_types, types, after_date) all work correctly with vector and hybrid modes
- Pagination works but cross-page ordering may need investigation
+265
View File
@@ -0,0 +1,265 @@
# Semantic Search
This guide covers Basic Memory's optional semantic (vector) search feature, which adds meaning-based retrieval alongside the existing full-text search.
## Overview
Basic Memory's default search uses full-text search (FTS) — keyword matching with boolean operators. Semantic search adds vector embeddings that capture the *meaning* of your content, enabling:
- **Paraphrase matching**: Find "authentication flow" when searching for "login process"
- **Conceptual queries**: Search for "ways to improve performance" and find notes about caching, indexing, and optimization
- **Hybrid retrieval**: Combine the precision of keyword search with the recall of semantic similarity
Semantic search is **opt-in** — existing behavior is completely unchanged unless you enable it. It works on both SQLite (local) and Postgres (cloud) backends.
## Installation
Semantic search dependencies (fastembed, sqlite-vec, openai) are **optional extras** — they are not installed with the base `basic-memory` package. Install them with:
```bash
pip install 'basic-memory[semantic]'
```
This keeps the base install lightweight and avoids platform-specific issues with ONNX Runtime wheels.
### Platform Compatibility
| Platform | FastEmbed (local) | OpenAI (API) |
|---|---|---|
| macOS ARM64 (Apple Silicon) | Yes | Yes |
| macOS x86_64 (Intel Mac) | No — see workaround below | Yes |
| Linux x86_64 | Yes | Yes |
| Linux ARM64 | Yes | Yes |
| Windows x86_64 | Yes | Yes |
#### Intel Mac Workaround
The default FastEmbed provider uses ONNX Runtime, which dropped Intel Mac (x86_64) wheels starting in v1.24. Intel Mac users have two options:
**Option 1: Use OpenAI embeddings (recommended)**
Install only the OpenAI dependency manually — no ONNX Runtime or FastEmbed needed:
```bash
pip install openai sqlite-vec
export BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED=true
export BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER=openai
export OPENAI_API_KEY=sk-...
```
**Option 2: Pin an older ONNX Runtime**
FastEmbed's ONNX Runtime dependency is unpinned, so you can constrain it to an older version that still ships Intel Mac wheels by passing both requirements in the same install command:
```bash
pip install 'basic-memory[semantic]' 'onnxruntime<1.24'
```
## Quick Start
1. Install semantic extras:
```bash
pip install 'basic-memory[semantic]'
```
2. Enable semantic search:
```bash
export BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED=true
```
3. Build vector embeddings for your existing content:
```bash
bm reindex --embeddings
```
4. Search using semantic modes:
```python
# Pure vector similarity
search_notes("login process", search_type="vector")
# Hybrid: combines FTS precision with vector recall (recommended)
search_notes("login process", search_type="hybrid")
# Traditional full-text search (still the default)
search_notes("login process", search_type="text")
```
## Configuration Reference
All settings are fields on `BasicMemoryConfig` and can be set via environment variables (prefixed with `BASIC_MEMORY_`).
| Config Field | Env Var | Default | Description |
|---|---|---|---|
| `semantic_search_enabled` | `BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED` | `false` | Enable semantic search. Required before vector/hybrid modes work. |
| `semantic_embedding_provider` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER` | `"fastembed"` | Embedding provider: `"fastembed"` (local) or `"openai"` (API). |
| `semantic_embedding_model` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_MODEL` | `"bge-small-en-v1.5"` | Model identifier. Auto-adjusted per provider if left at default. |
| `semantic_embedding_dimensions` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_DIMENSIONS` | Auto-detected | Vector dimensions. 384 for FastEmbed, 1536 for OpenAI. Override only if using a non-default model. |
| `semantic_embedding_batch_size` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_BATCH_SIZE` | `64` | Number of texts to embed per batch. |
| `semantic_vector_k` | `BASIC_MEMORY_SEMANTIC_VECTOR_K` | `100` | Candidate count for vector nearest-neighbour retrieval. Higher values improve recall at the cost of latency. |
## Embedding Providers
### FastEmbed (default)
FastEmbed runs entirely locally using ONNX models — no API key, no network calls, no cost.
- **Model**: `BAAI/bge-small-en-v1.5`
- **Dimensions**: 384
- **Tradeoff**: Smaller model, fast inference, good quality for most use cases
```bash
# Install semantic extras and enable
pip install 'basic-memory[semantic]'
export BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED=true
```
### OpenAI
Uses OpenAI's embeddings API for higher-dimensional vectors. Requires an API key.
- **Model**: `text-embedding-3-small`
- **Dimensions**: 1536
- **Tradeoff**: Higher quality embeddings, requires API calls and an OpenAI key
```bash
export BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED=true
export BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER=openai
export OPENAI_API_KEY=sk-...
```
When switching from FastEmbed to OpenAI (or vice versa), you must rebuild embeddings since the vector dimensions differ:
```bash
bm reindex --embeddings
```
## Search Modes
### `text` (default)
Full-text keyword search using FTS5 (SQLite) or tsvector (Postgres). Supports boolean operators (`AND`, `OR`, `NOT`), phrase matching, and prefix wildcards.
```python
search_notes("project AND planning", search_type="text")
```
This is the existing default and does not require semantic search to be enabled.
### `vector`
Pure semantic similarity search. Embeds your query and finds the nearest content vectors. Good for conceptual or paraphrase queries where exact keywords may not appear in the content.
```python
search_notes("how to speed up the app", search_type="vector")
```
Returns results ranked by cosine similarity. Individual observations and relations surface as first-class results, not collapsed into parent entities.
### `hybrid`
Combines FTS and vector results using reciprocal rank fusion (RRF). This is generally the best mode when you want both keyword precision and semantic recall.
```python
search_notes("authentication security", search_type="hybrid")
```
RRF merges the two ranked lists so that items appearing in both get a score boost, while items found by only one method still appear.
### When to Use Which
| Mode | Best For |
|---|---|
| `text` | Exact keyword matching, boolean queries, tag/category searches |
| `vector` | Conceptual queries, paraphrase matching, exploratory searches |
| `hybrid` | General-purpose search combining precision and recall |
## The Reindex Command
The `bm reindex` command rebuilds search indexes without dropping the database.
```bash
# Rebuild everything (FTS + embeddings if semantic is enabled)
bm reindex
# Only rebuild vector embeddings
bm reindex --embeddings
# Only rebuild the full-text search index
bm reindex --search
# Target a specific project
bm reindex -p my-project
```
### When You Need to Reindex
- **First enable**: After turning on `semantic_search_enabled` for the first time
- **Provider change**: After switching between `fastembed` and `openai`
- **Model change**: After changing `semantic_embedding_model`
- **Dimension change**: After changing `semantic_embedding_dimensions`
The reindex command shows progress with embedded/skipped/error counts:
```
Project: main
Building vector embeddings...
✓ Embeddings complete: 142 entities embedded, 0 skipped, 0 errors
Reindex complete!
```
## How It Works
### Chunking
Each entity in the search index is split into semantic chunks before embedding:
- **Headers**: Markdown headers (`#`, `##`, etc.) start new chunks
- **Bullets**: Each bullet item (`-`, `*`) becomes its own chunk for granular fact retrieval
- **Prose sections**: Non-bullet text is merged up to ~900 characters per chunk
- **Long sections**: Oversized content is split with ~120 character overlap to preserve context at boundaries
Each search index item type (entity, observation, relation) is chunked independently, so observations and relations are embeddable as discrete facts.
### Deduplication
Each chunk has a `source_hash` (SHA-256 of the chunk text). On re-sync, unchanged chunks skip re-embedding entirely. This makes incremental updates fast — only modified content triggers API calls or model inference.
### Hybrid Fusion
Hybrid search uses reciprocal rank fusion (RRF) to merge FTS and vector results:
1. Run FTS search to get keyword-ranked results
2. Run vector search to get similarity-ranked results
3. For each result, compute: `score = 1/(k + fts_rank) + 1/(k + vector_rank)` where `k = 60`
4. Sort by fused score
Items found by both methods get a natural score boost. Items found by only one method still appear but rank lower.
### Observation-Level Results
Vector and hybrid modes return individual observations and relations as first-class search results, not just parent entities. This means a search for "water temperature for brewing" can surface the specific observation about 205°F without returning the entire "Coffee Brewing Methods" entity.
## Database Backends
### SQLite (local)
- **Vector storage**: [sqlite-vec](https://github.com/asg017/sqlite-vec) virtual table
- **Table creation**: At runtime when semantic search is first used — no migration needed
- **Embedding table**: `search_vector_embeddings` using `vec0(embedding float[N])` where N is the configured dimensions
- **Chunk metadata**: `search_vector_chunks` table stores chunk text, keys, and source hashes
The sqlite-vec extension is loaded per-connection. Vector tables are created lazily on first use.
### Postgres (cloud)
- **Vector storage**: [pgvector](https://github.com/pgvector/pgvector) with HNSW indexing
- **Chunk metadata table**: Created via Alembic migration (`search_vector_chunks` with `BIGSERIAL` primary key)
- **Embedding table**: `search_vector_embeddings` created at runtime (dimension-dependent, same pattern as SQLite)
- **Index**: HNSW index on the embedding column for fast approximate nearest-neighbour queries
The Alembic migration creates the dimension-independent chunks table. The embeddings table and HNSW index are deferred to runtime because they depend on the configured vector dimensions.
+225
View File
@@ -0,0 +1,225 @@
# SPEC-LOCAL-PLUS-PUBLISH: Local+ Published Notes and Privacy Tiers
**Status:** Draft
**Date:** 2026-02-14
**Owner:** Basic Memory
## Summary
Add a paid Local+ feature that lets users publish selected notes to shareable URLs while keeping the
main knowledge base local-first. Use this as a product wedge for users who do not want full cloud
hosting but do want collaboration and distribution features.
This spec also captures a practical position on "zero knowledge" for Local+.
## Context
Basic Memory already has strong local-first primitives and optional cloud routing/sync. A recurring
request is:
- keep knowledge local by default,
- pay for selective value-add,
- share specific outputs externally.
Published Notes fits this model: explicit per-note opt-in, reversible, and easy to understand.
## Goals
1. Provide an Obsidian Publish-style sharing experience for selected notes.
2. Keep local markdown files as source of truth.
3. Make sharing compatible with current cloud/auth/billing primitives.
4. Define clear Local+ packaging that does not degrade OSS local workflows.
5. Document zero-knowledge constraints so product decisions are explicit.
## Non-Goals
1. Full hosted editing for all notes (Cloud Full remains separate).
2. Public website builder/CMS features.
3. Strict cryptographic zero-knowledge server processing for MCP/search in v1.
## Local+ Feature Catalog (Sellable)
Core Local+ candidates:
1. Published Notes (share URL, revoke, expiry, password).
2. Snapshot Time Machine (point-in-time restore for local projects).
3. Recovery Drill Reports (automated restore verification).
4. Device/API Key Governance (per-device keys, revocation, audit trail).
5. BYO Storage Orchestration (managed setup for user-owned object storage).
6. Semantic Boost Add-on (higher quality retrieval options while files remain source-of-truth).
Team-oriented add-ons:
1. Team-owned shared links and domain branding.
2. Role-based publish permissions.
3. Shared workspace policies for what can be published.
## Proposed MVP: Published Notes
### User Experience
Per note actions:
1. Publish.
2. Unpublish.
3. Copy URL.
4. Regenerate URL.
5. Set visibility and controls.
Controls:
1. Visibility: `unlisted` (default) or `public`.
2. Optional password gate.
3. Optional expiration datetime.
4. Optional "disable indexing" flag for public mode.
Behavior:
1. Source note remains local markdown.
2. Publish is explicit opt-in per note.
3. Unpublish removes public access immediately.
4. Republish creates a new URL token unless user chooses to keep current URL.
### URL Model
1. Unlisted share URL: high-entropy token path.
2. Public URL: slug path (optional, later phase).
3. Team plans can support custom domain mapping in later phase.
### Content Model
v1 published page includes:
1. Rendered markdown body.
2. Optional metadata (title, updated_at).
v1 excludes:
1. Full graph traversal expansion.
2. Related note auto-discovery on public pages.
### Sync Model
1. Local file remains canonical.
2. Publish stores a rendered snapshot plus metadata in cloud.
3. Update path:
- manual "update published version", or
- optional auto-update on note change (plan-gated).
## Architecture (v1)
### High-Level Flow
1. Client selects a note to publish.
2. Client sends publish request with note identifier and policy.
3. Service resolves note content (local sync artifact or explicit upload payload).
4. Service stores published artifact and returns share URL.
### Data Model
`published_notes`
1. `id` (uuid)
2. `tenant_id` or `workspace_id`
3. `project_id`
4. `entity_permalink` (or stable external_id)
5. `share_token` (hashed in DB)
6. `visibility` (`unlisted`|`public`)
7. `password_hash` (nullable)
8. `expires_at` (nullable)
9. `is_active`
10. `published_content` (rendered snapshot or reference)
11. `published_at`
12. `updated_at`
### API Shape (Draft)
1. `POST /api/published-notes`
2. `GET /api/published-notes`
3. `GET /api/published-notes/{id}`
4. `PATCH /api/published-notes/{id}`
5. `DELETE /api/published-notes/{id}` (unpublish)
6. `POST /api/published-notes/{id}/regenerate-url`
7. `GET /p/{token}` (public resolver)
### CLI Shape (Draft)
1. `bm cloud publish <identifier>`
2. `bm cloud publish list`
3. `bm cloud publish update <id>`
4. `bm cloud publish unpublish <id>`
5. `bm cloud publish rotate-url <id>`
### Security
1. Default to unlisted URLs.
2. Store only hashed share tokens.
3. Passwords hashed server-side.
4. Enforce expiration at request time.
5. Log publish/unpublish/rotate events for auditability.
## Packaging and Pricing Direction
Suggested split:
1. OSS Local: no publish URLs.
2. Local+ Solo: publish URLs + snapshots + recovery.
3. Local+ Team: solo features + team governance and branding.
4. Cloud Full: hosted app + full cloud workflows.
Key message:
"Keep everything local. Publish only what you choose."
## Rollout Plan
1. Phase 1: Unlisted publish URLs + unpublish + regenerate URL.
2. Phase 2: Password/expiry controls.
3. Phase 3: Auto-update on note change and basic analytics.
4. Phase 4: Team branding/domains/policies.
## Zero-Knowledge Position
### Strict Zero-Knowledge Definition
Strict zero-knowledge means the server cannot decrypt note content at all.
### Why This Conflicts with MCP and Search
If server cannot decrypt:
1. MCP tool execution against cloud content cannot read/write semantic content.
2. Full-text search cannot index plaintext content.
3. Semantic/vector search cannot generate or query embeddings on plaintext.
4. Server-side relation resolution and context building become severely limited.
This matches earlier findings: strict zero-knowledge materially handicaps MCP-driven behavior and
search quality.
### Viable Alternatives (Not Strict Zero-Knowledge)
1. Encryption at rest/in transit with server-side decrypt in trusted runtime.
- Preserves MCP/search quality.
- Not zero-knowledge cryptographically.
2. Client-side retrieval mode.
- Keep MCP/search local; cloud is sync/share/backup relay.
- Best for privacy-first users.
- Requires local agent availability for advanced retrieval.
3. Limited encrypted indexing.
- Blind indexes for exact keywords only.
- No high-quality semantic search.
- Usually poor UX for natural-language memory recall.
### Recommendation
For Local+:
1. Do not promise strict zero-knowledge for cloud MCP/search paths.
2. Offer a privacy-first local mode where advanced retrieval stays local.
3. Clearly label tradeoffs:
- "Local private mode" (best privacy, best local retrieval).
- "Cloud-assisted mode" (best cross-device/MCP consistency, trusted-runtime decrypt).
This keeps messaging honest and avoids repeating the known incompatibility.
+365
View File
@@ -0,0 +1,365 @@
# SPEC-SCHEMA-IMPL: Schema System Implementation Plan
**Status:** Draft
**Created:** 2025-02-06
**Branch:** `feature/schema-system`
**Depends on:** [SPEC-SCHEMA](SPEC-SCHEMA.md)
## Overview
Implementation plan for the Basic Memory Schema System. The system is entirely programmatic —
no LLM agent runtime or API key required. The LLM already in the user's session (Claude Code,
Claude Desktop, etc.) provides the intelligence layer by reading schema notes via existing
MCP tools.
## Architecture
```
┌─────────────────────────────────────────────────┐
│ Entry Points │
│ CLI (bm schema ...) │ MCP (schema_validate) │
└──────────┬────────────┴──────────┬──────────────┘
│ │
▼ ▼
┌─────────────────────────────────────────────────┐
│ Schema Service Layer │
│ resolve_schema · validate · infer · diff │
└──────────┬────────────────────────┬──────────────┘
│ │
▼ ▼
┌──────────────────────┐ ┌────────────────────────┐
│ Picoschema Parser │ │ Note/Entity Access │
│ YAML → SchemaModel │ │ (existing repository) │
└──────────────────────┘ └────────────────────────┘
```
No new database tables. Schemas are notes with `type: schema` — they're already indexed.
Validation reads observations and relations from existing data.
## Components
### 1. Picoschema Parser
**Location:** `src/basic_memory/schema/parser.py`
Parses Picoschema YAML into an internal representation.
```python
@dataclass
class SchemaField:
name: str
type: str # string, integer, number, boolean, any, or EntityName
required: bool # True unless field name ends with ?
is_array: bool # True if (array) notation
is_enum: bool # True if (enum) notation
enum_values: list[str] # Populated for enums
description: str | None # Text after comma
is_entity_ref: bool # True if type is capitalized (entity reference)
children: list[SchemaField] # For (object) types
@dataclass
class SchemaDefinition:
entity: str # The entity type this schema describes
version: int # Schema version
fields: list[SchemaField] # Parsed fields
validation_mode: str # "warn" | "strict" | "off"
def parse_picoschema(yaml_dict: dict) -> list[SchemaField]:
"""Parse a Picoschema YAML dict into a list of SchemaField objects."""
def parse_schema_note(frontmatter: dict) -> SchemaDefinition:
"""Parse a full schema note's frontmatter into a SchemaDefinition."""
```
**Input/Output:**
```yaml
# Input (YAML dict from frontmatter)
schema:
name: string, full name
role?: string, job title
works_at?: Organization, employer
expertise?(array): string, areas of knowledge
```
```python
# Output
[
SchemaField(name="name", type="string", required=True, description="full name", ...),
SchemaField(name="role", type="string", required=False, description="job title", ...),
SchemaField(name="works_at", type="Organization", required=False, is_entity_ref=True, ...),
SchemaField(name="expertise", type="string", required=False, is_array=True, ...),
]
```
### 2. Schema Resolver
**Location:** `src/basic_memory/schema/resolver.py`
Finds the applicable schema for a note using the resolution order.
```python
async def resolve_schema(
note_frontmatter: dict,
search_fn: Callable, # injected search capability
) -> SchemaDefinition | None:
"""Resolve schema for a note.
Resolution order:
1. Inline schema (frontmatter['schema'] is a dict)
2. Explicit reference (frontmatter['schema'] is a string)
3. Implicit by type (frontmatter['type'] → schema note with matching entity)
4. No schema (returns None)
"""
```
### 3. Schema Validator
**Location:** `src/basic_memory/schema/validator.py`
Validates a note's observations and relations against a resolved schema.
```python
@dataclass
class FieldResult:
field: SchemaField
status: str # "present" | "missing" | "type_mismatch"
values: list[str] # Matched observation values or relation targets
message: str | None # Human-readable detail
@dataclass
class ValidationResult:
note_identifier: str
schema_entity: str
passed: bool # True if no errors (warnings are OK)
field_results: list[FieldResult]
unmatched_observations: dict[str, int] # category → count
unmatched_relations: list[str] # relation types not in schema
warnings: list[str]
errors: list[str]
async def validate_note(
note: Note,
schema: SchemaDefinition,
) -> ValidationResult:
"""Validate a note against a schema definition.
Mapping rules:
- field: string → observation [field] exists
- field?(array): type → multiple [field] observations
- field?: EntityType → relation 'field [[...]]' exists
- field?(enum): [v] → observation [field] value ∈ enum values
"""
```
### 4. Schema Inference Engine
**Location:** `src/basic_memory/schema/inference.py`
Analyzes notes of a given type and suggests a schema based on usage frequency.
```python
@dataclass
class FieldFrequency:
name: str
source: str # "observation" | "relation"
count: int # notes containing this field
total: int # total notes analyzed
percentage: float
sample_values: list[str] # representative values
is_array: bool # True if typically appears multiple times per note
target_type: str | None # For relations, the most common target entity type
@dataclass
class InferenceResult:
entity_type: str
notes_analyzed: int
field_frequencies: list[FieldFrequency]
suggested_schema: dict # Ready-to-use Picoschema YAML dict
suggested_required: list[str]
suggested_optional: list[str]
excluded: list[str] # Below threshold
async def infer_schema(
entity_type: str,
notes: list[Note],
required_threshold: float = 0.95, # 95%+ = required
optional_threshold: float = 0.25, # 25%+ = optional
) -> InferenceResult:
"""Analyze notes and suggest a Picoschema definition."""
```
### 5. Schema Diff
**Location:** `src/basic_memory/schema/diff.py`
Compares current note usage against an existing schema definition.
```python
@dataclass
class SchemaDrift:
new_fields: list[FieldFrequency] # Fields not in schema but common in notes
dropped_fields: list[FieldFrequency] # Fields in schema but rare in notes
cardinality_changes: list[str] # one → many or many → one
type_mismatches: list[str] # observation values don't match declared type
async def diff_schema(
schema: SchemaDefinition,
notes: list[Note],
) -> SchemaDrift:
"""Compare a schema against actual note usage to detect drift."""
```
## Entry Points
### CLI Commands
**Location:** `src/basic_memory/cli/schema.py`
```python
import typer
schema_app = typer.Typer(name="schema", help="Schema management commands")
@schema_app.command()
async def validate(
target: str = typer.Argument(None, help="Note path or entity type"),
strict: bool = typer.Option(False, help="Override to strict mode"),
):
"""Validate notes against their schemas."""
@schema_app.command()
async def infer(
entity_type: str = typer.Argument(..., help="Entity type to analyze"),
threshold: float = typer.Option(0.25, help="Minimum frequency for optional fields"),
save: bool = typer.Option(False, help="Save to schema/ directory"),
):
"""Infer schema from existing notes of a type."""
@schema_app.command()
async def diff(
entity_type: str = typer.Argument(..., help="Entity type to diff"),
):
"""Show drift between schema and actual usage."""
```
Registered as subcommand: `bm schema validate`, `bm schema infer`, `bm schema diff`.
### MCP Tools
**Location:** `src/basic_memory/mcp/tools/schema.py`
```python
@mcp_tool
async def schema_validate(
entity_type: str | None = None,
identifier: str | None = None,
project: str | None = None,
) -> str:
"""Validate notes against their resolved schema."""
@mcp_tool
async def schema_infer(
entity_type: str,
threshold: float = 0.25,
project: str | None = None,
) -> str:
"""Analyze existing notes and suggest a schema definition."""
```
### API Endpoints
**Location:** `src/basic_memory/api/schema_router.py`
```python
router = APIRouter(prefix="/schema", tags=["schema"])
@router.post("/validate")
async def validate_schema(...) -> ValidationReport: ...
@router.post("/infer")
async def infer_schema(...) -> InferenceResult: ...
@router.get("/diff/{entity_type}")
async def diff_schema(...) -> SchemaDrift: ...
```
MCP tools call these endpoints via the typed client pattern (consistent with existing
architecture).
## Implementation Phases
### Phase 1: Parser + Resolver
Build the foundation — can parse Picoschema and find schemas for notes.
**Deliverables:**
- `schema/parser.py` — Picoschema YAML → `SchemaDefinition`
- `schema/resolver.py` — Resolution order (inline → explicit ref → implicit by type → none)
- Unit tests for all Picoschema syntax variations
- Unit tests for resolution order
**No external dependencies.** Pure Python parsing of YAML dicts. Can develop and test
in isolation.
### Phase 2: Validator
Connect schemas to notes and produce validation results.
**Deliverables:**
- `schema/validator.py` — Validate note observations/relations against schema fields
- API endpoint: `POST /schema/validate`
- MCP tool: `schema_validate`
- CLI command: `bm schema validate`
- Integration tests with real notes and schemas
**Depends on:** Phase 1 (parser + resolver)
### Phase 3: Inference
Analyze existing notes to suggest schemas.
**Deliverables:**
- `schema/inference.py` — Frequency analysis across notes of a type
- API endpoint: `POST /schema/infer`
- MCP tool: `schema_infer`
- CLI command: `bm schema infer`
- Option to save inferred schema as a note via `write_note`
**Depends on:** Phase 1 (parser for output format)
### Phase 4: Diff
Compare schemas against current usage.
**Deliverables:**
- `schema/diff.py` — Drift detection between schema and actual notes
- API endpoint: `GET /schema/diff/{entity_type}`
- CLI command: `bm schema diff`
**Depends on:** Phase 1 (parser), Phase 3 (inference, for frequency analysis)
## Testing Strategy
- **Unit tests** (`tests/schema/`): Parser edge cases, resolution logic, validation mapping,
inference thresholds
- **Integration tests** (`test-int/schema/`): End-to-end with real markdown files, schema notes
on disk, CLI invocation
- Coverage target: 100% (consistent with project standard)
## What This Does NOT Include
- No new database tables or migrations
- No new markdown syntax (schemas validate existing observations/relations)
- No LLM agent runtime or API key management
- No hook integration (deferred)
- No schema composition/inheritance (deferred)
- No OWL/RDF export (deferred)
- No built-in templates (deferred)
+462
View File
@@ -0,0 +1,462 @@
# SPEC-SCHEMA: Basic Memory Schema System
**Status:** Draft
**Created:** 2025-02-06
**Branch:** `feature/schema-system`
## Summary
A schema system for Basic Memory that uses [Picoschema](https://genkit.dev/docs/dotprompt/)
syntax in YAML frontmatter. Schemas validate notes against their existing observation/relation
structure — no new data model, no migration, just a declarative lens over what's already there.
## Core Principles
1. **Schemas are just notes** — A schema is a note with `type: schema`, lives anywhere
2. **Use prior art** — Picoschema syntax in YAML frontmatter, no custom notation
3. **Validation maps to existing format** — Observations and relations, not a parallel data model
4. **Validation is soft** — Warnings by default, not blocking errors
5. **Inference over prescription** — Schemas describe reality, emerge from usage
6. **No built-in agent** — Programmatic core; the LLM already in the session provides intelligence
## Picoschema Syntax
Picoschema is a compact schema notation from Google's Dotprompt that fits naturally in YAML
frontmatter.
### Supported Types
| Type | Description |
|------|-------------|
| `string` | Text value |
| `integer` | Whole number |
| `number` | Decimal number |
| `boolean` | True/false |
| `any` | Any scalar type |
| `EntityName` | Reference to another entity (capitalized = entity reference) |
### Syntax Rules
```yaml
schema:
name: string, full name # required field with description
email?: string, contact email # ? = optional
role?: string, job title
works_at?: Organization, employer # capitalized type = entity reference
tags?(array): string, categories # array of type
status?(enum): [active, inactive] # enum with allowed values
metadata?(object): # nested object
updated_at?: string
source?: string
```
- `field: type` — required field
- `field?: type` — optional field
- `field(array): type` — array of values
- `field?(enum): [values]` — enumeration
- `field?(object):` — nested object with sub-fields
- `, description` — description after comma
- `EntityName` as type (capitalized) — reference to another entity
## Schema-to-Note Mapping
Schemas validate against the existing Basic Memory note format. No new syntax for note
authors to learn.
### Mapping Rules
| Schema Declaration | Grounded In | Example Match |
|--------------------|-------------|---------------|
| `field: string` | Observation `[field] value` | `- [name] Paul Graham` |
| `field?(array): string` | Multiple `[field]` observations | `- [expertise] Lisp` (×N) |
| `field?: EntityType` | Relation `field [[Target]]` | `- works_at [[Y Combinator]]` |
| `field?(array): EntityType` | Multiple `field` relations | `- authored [[Book]]` (×N) |
| `tags` | Frontmatter `tags` array | `tags: [startups, essays]` |
| `field?(enum): [values]` | Observation `[field] value` where value ∈ set | `- [status] active` |
### Key Insight
Schemas don't introduce a new way to store data. They describe the patterns already present
in observations and relations. A note doesn't have to change how it's written — the schema
just says "a good Person note has a `[name]` observation and a `works_at` relation."
## Schema Definition
### As a Dedicated Schema Note
```yaml
# schema/Person.md
---
title: Person
type: schema
entity: Person
version: 1
schema:
name: string, full name
email?: string, contact email
role?: string, job title
works_at?: Organization, employer
expertise?(array): string, areas of knowledge
settings:
validation: warn # warn | strict | off
---
# Person
A human individual in the knowledge graph.
Any documentation about this entity type goes here as prose.
```
Schema notes are regular Basic Memory notes. They show up in search, can have their own
observations and relations, and can be organized in any folder (though `schema/` is
the suggested convention).
### Inline Schema in a Note
Notes can carry their own schema directly:
```yaml
# meetings/2024-01-15-standup.md
---
title: Team Standup 2024-01-15
type: meeting
schema:
attendees(array): string, who was there
decisions(array): string, what was decided
action_items(array): string, follow-ups
blockers?(array): string, anything stuck
---
# Team Standup 2024-01-15
## Observations
- [attendees] Paul
- [attendees] Sarah
- [decisions] Ship v2 by Friday
- [action_items] Paul to review PR #42
- [blockers] Waiting on API credentials
```
Good for one-off structured notes or prototyping a schema before extracting it.
### Explicit Schema Reference
A note can reference a schema by entity name or permalink:
```yaml
# projects/basic-memory.md
---
title: Basic Memory
schema: SoftwareProject # by entity name
---
# research/llm-memory-patterns.md
---
title: LLM Memory Patterns
schema: schema/research-project # by permalink
---
```
Use cases:
- Note's `type` differs from the schema it should validate against
- Multiple schema variants exist for the same domain
- Applying structure to existing notes without changing their type
## Schema Resolution
When validating a note, schemas resolve in priority order:
```
1. Inline schema → schema: { ... } (dict in frontmatter)
2. Explicit ref → schema: Person (string in frontmatter)
3. Implicit by type → type: Person (lookup schema note with entity: Person)
4. No schema → no validation (perfectly fine)
```
```python
async def resolve_schema(note: Note) -> Schema | None:
schema_value = note.frontmatter.get('schema')
# 1. Inline schema (dict)
if isinstance(schema_value, dict):
return parse_picoschema(schema_value)
# 2. Explicit reference (string)
if isinstance(schema_value, str):
schema_note = await find_schema_note(schema_value)
if schema_note:
return parse_picoschema(schema_note.frontmatter['schema'])
# 3. Implicit by type
note_type = note.frontmatter.get('type')
if note_type:
results = await search_notes(f"type:schema entity:{note_type}")
if results:
return parse_picoschema(results[0].frontmatter['schema'])
# 4. No schema
return None
```
## Validation
### Modes
Configured in the schema's `settings.validation`:
| Mode | Behavior |
|------|----------|
| `off` | No validation |
| `warn` | Warnings in output, doesn't block (default) |
| `strict` | Errors that block sync, for CI/CD enforcement |
### Validation Output
For a note missing required fields:
```
$ bm schema validate people/ada-lovelace.md
⚠ Person schema validation:
- Missing required field: name (expected [name] observation)
- Missing optional field: role
- Missing optional field: works_at (no relation found)
Unmatched observations: [fact] ×2, [born] ×1
Unmatched relations: collaborated_with
```
"Unmatched" items are informational — observations and relations the schema doesn't cover.
They're valid. Schemas are a subset, not a straitjacket.
### Batch Validation
```
$ bm schema validate Person
Validating 30 notes against Person schema...
✓ people/paul-graham.md — all fields present
✓ people/rich-hickey.md — all fields present
⚠ people/ada-lovelace.md — missing: name
⚠ people/alan-kay.md — missing: name, role
✓ people/linus-torvalds.md — all fields present
...
Summary: 22/30 valid, 8 warnings, 0 errors
```
## Emerging Schemas
### The Problem with Traditional Schemas
Most schema systems require: define schema → create conforming content → fight the schema
when reality doesn't match. This is backwards. Knowledge grows organically.
### The Basic Memory Approach
```
Write notes freely → Patterns emerge → Crystallize into schema → Validate future notes
```
### Schema Inference
Generate schemas from existing notes by analyzing observation and relation frequency:
```
$ bm schema infer Person
Analyzing 30 notes with type: Person...
Observations found:
[name] 30/30 100% → name: string
[role] 27/30 90% → role?: string
[fact] 25/30 83% (generic — no single field)
[expertise] 18/30 60% → expertise?(array): string
[email] 8/30 27% → email?: string
[born] 6/30 20% (below threshold)
Relations found:
works_at 22/30 73% → works_at?: Organization
authored 11/30 37% → authored?(array): string
Suggested schema:
name: string, full name
role?: string, job title
expertise?(array): string, areas of knowledge
email?: string, contact email
works_at?: Organization, employer
Save to schema/Person.md? [y/n]
```
Frequency thresholds:
- 100% present → required field
- 25%+ present → optional field
- Below 25% → excluded from suggestion (but noted)
### Schema Drift Detection
Track how usage patterns shift over time:
```
$ bm schema diff Person
Schema drift detected:
+ expertise: now in 81% of notes (was 12%)
- department: dropped to 3% of notes
~ works_at: cardinality changed (one → many)
Update schema? [y/n/review]
```
## LLM Integration (AI Guidance)
No agent runtime or API key required. The LLM already in the session uses schemas as
context for note creation.
### Flow
1. User asks LLM to "write a note about Rich Hickey"
2. LLM determines `type: Person` is appropriate
3. LLM calls `search_notes("type:schema entity:Person")` → finds schema
4. LLM reads schema fields: required `name`, optional `role`, `works_at`, `expertise`
5. LLM calls `write_note` with observations and relations that satisfy the schema
The schema acts as a creation template. The LLM knows what a "complete" note looks like
without any custom agent infrastructure.
### MCP Tools
```python
@mcp_tool
async def schema_validate(
entity_type: str | None = None,
identifier: str | None = None,
project: str | None = None,
) -> ValidationReport:
"""Validate notes against their resolved schema.
Validates a specific note (by identifier) or all notes of a given type.
Returns warnings/errors based on the schema's validation mode.
"""
@mcp_tool
async def schema_infer(
entity_type: str,
threshold: float = 0.25,
project: str | None = None,
) -> SuggestedSchema:
"""Analyze existing notes and suggest a schema definition.
Examines observation categories and relation types across all notes
of the given type. Returns frequency analysis and suggested Picoschema.
"""
```
## CLI Commands
```bash
# Validate a specific note
bm schema validate people/ada-lovelace.md
# Validate all notes of a type
bm schema validate Person
# Validate everything with a schema
bm schema validate
# Infer schema from existing notes
bm schema infer Person
# Show schema drift from current definition
bm schema diff Person
# List all schema notes
bm search "type:schema"
```
## Examples
### Complete Person Workflow
**Schema:**
```yaml
# schema/Person.md
---
title: Person
type: schema
entity: Person
version: 1
schema:
name: string, full name
role?: string, job title or position
works_at?: Organization, employer
expertise?(array): string, areas of knowledge
email?: string, contact email
settings:
validation: warn
---
# Person
A human individual in the knowledge graph.
```
**Valid note:**
```yaml
# people/paul-graham.md
---
title: Paul Graham
type: Person
tags: [startups, essays, lisp]
---
# Paul Graham
## Observations
- [name] Paul Graham
- [role] Essayist and investor
- [expertise] Startups
- [expertise] Lisp
- [expertise] Essay writing
- [fact] Created Viaweb, the first web app
## Relations
- works_at [[Y Combinator]]
- authored [[Hackers and Painters]]
```
**Note with warnings:**
```yaml
# people/ada-lovelace.md
---
title: Ada Lovelace
type: Person
---
# Ada Lovelace
## Observations
- [fact] Wrote the first computer program
- [born] 1815
## Relations
- collaborated_with [[Charles Babbage]]
```
Validation: warns about missing required `[name]` observation. Everything else is optional
or unmatched (which is fine).
## Future Considerations (Deferred)
These are interesting but out of scope for the initial implementation:
- **Multiple schema inheritance** — `schema: [Person, Author]`
- **Hook integration** — Pre-write validation via the hooks system
- **OWL/RDF export** — `bm schema export --format owl`
- **SPARQL queries** — Schema-aware graph queries
- **Built-in templates** — `bm schema use gtd`, `bm schema use zettelkasten`
- **Schema versioning/migration** — Tracking breaking changes across versions
+64 -9
View File
@@ -2,8 +2,7 @@
# Install dependencies
install:
uv pip install -e ".[dev]"
uv sync
uv sync --extra semantic
@echo ""
@echo "💡 Remember to activate the virtual environment by running: source .venv/bin/activate"
@@ -43,9 +42,9 @@ test-unit-sqlite:
test-unit-postgres:
BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov tests
# Run integration tests against SQLite
# Run integration tests against SQLite (excludes semantic benchmarks — use just test-semantic)
test-int-sqlite:
uv run pytest -p pytest_mock -v --no-cov test-int
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m "not semantic" test-int
# Run integration tests against Postgres
# Note: Uses timeout due to FastMCP Client + asyncpg cleanup hang (tests pass, process hangs on exit)
@@ -56,12 +55,28 @@ test-int-postgres:
# Use gtimeout (macOS/Homebrew) or timeout (Linux)
TIMEOUT_CMD=$(command -v gtimeout || command -v timeout || echo "")
if [[ -n "$TIMEOUT_CMD" ]]; then
$TIMEOUT_CMD --signal=KILL 600 bash -c 'BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov test-int' || test $? -eq 137
$TIMEOUT_CMD --signal=KILL 600 bash -c 'BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov -m "not semantic" test-int' || test $? -eq 137
else
echo "⚠️ No timeout command found, running without timeout..."
BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov test-int
BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov -m "not semantic" test-int
fi
# Run tests impacted by recent changes (requires pytest-testmon)
testmon *args:
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov --testmon --testmon-forceselect {{args}}
# Run MCP smoke test (fast end-to-end loop)
test-smoke:
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m smoke test-int/mcp/test_smoke_integration.py
# Fast local loop: lint, format, typecheck, impacted tests
fast-check:
just fix
just format
just typecheck
just testmon
just test-smoke
# Reset Postgres test database (drops and recreates schema)
# Useful when Alembic migration state gets out of sync during development
# Uses credentials from docker-compose-postgres.yml
@@ -83,18 +98,43 @@ postgres-migrate:
# These tests verify Windows-specific database optimizations (locking mode, NullPool)
# Will be skipped automatically on non-Windows platforms
test-windows:
uv run pytest -p pytest_mock -v --no-cov -m windows tests test-int
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m windows tests test-int
# Run benchmark tests only (performance testing)
# These are slow tests that measure sync performance with various file counts
# Excluded from default test runs to keep CI fast
test-benchmark:
uv run pytest -p pytest_mock -v --no-cov -m benchmark tests test-int
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m benchmark tests test-int
# Run semantic search quality benchmarks (all combos)
test-semantic:
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m semantic test-int/semantic/
# Run semantic benchmarks with JSON artifact output, then show report
test-semantic-report:
BASIC_MEMORY_ENV=test BASIC_MEMORY_BENCHMARK_OUTPUT=.benchmarks/semantic-quality.jsonl uv run pytest -p pytest_mock -v -s --no-cov -m semantic test-int/semantic/
uv run python test-int/semantic/report.py .benchmarks/semantic-quality.jsonl
# Run semantic benchmarks (Postgres combos only)
test-semantic-postgres:
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m semantic -k postgres test-int/semantic/
# View semantic benchmark results (rich formatted table)
# Usage: just semantic-report [--filter-combo sqlite] [--filter-suite paraphrase] [--sort-by avg_latency_ms]
semantic-report *args:
uv run python test-int/semantic/report.py .benchmarks/semantic-quality.jsonl {{args}}
# Compare two search benchmark JSONL outputs
# Usage:
# just benchmark-compare .benchmarks/search-baseline.jsonl .benchmarks/search-candidate.jsonl
# just benchmark-compare .benchmarks/search-baseline.jsonl .benchmarks/search-candidate.jsonl --format markdown --show-missing
benchmark-compare baseline candidate *args:
uv run python test-int/compare_search_benchmarks.py "{{baseline}}" "{{candidate}}" --format table {{args}}
# Run all tests including Windows, Postgres, and Benchmarks (for CI/comprehensive testing)
# Use this before releasing to ensure everything works across all backends and platforms
test-all:
uv run pytest -p pytest_mock -v --no-cov tests test-int
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov tests test-int
# Generate HTML coverage report
coverage:
@@ -149,6 +189,18 @@ format:
run-inspector:
npx @modelcontextprotocol/inspector
# Run doctor checks in an isolated temp home/config
doctor:
#!/usr/bin/env bash
set -euo pipefail
TMP_HOME=$(mktemp -d)
TMP_CONFIG=$(mktemp -d)
HOME="$TMP_HOME" \
BASIC_MEMORY_ENV=test \
BASIC_MEMORY_HOME="$TMP_HOME/basic-memory" \
BASIC_MEMORY_CONFIG_DIR="$TMP_CONFIG" \
./.venv/bin/python -m basic_memory.cli.main doctor --local
# Update all dependencies to latest versions
update-deps:
@@ -157,6 +209,9 @@ update-deps:
# Run all code quality checks and tests
check: lint format typecheck test
# Run all code quality checks and all test suites, including semantic benchmarks
check-all: lint format typecheck test test-semantic
# Generate Alembic migration with descriptive message
migration message:
cd src/basic_memory/alembic && alembic revision --autogenerate -m "{{message}}"
+9
View File
@@ -46,6 +46,12 @@ dependencies = [
"httpx>=0.28.0",
]
[project.optional-dependencies]
semantic = [
"fastembed>=0.7.4",
"sqlite-vec>=0.1.6",
"openai>=1.100.2",
]
[project.urls]
Homepage = "https://github.com/basicmachines-co/basic-memory"
@@ -71,6 +77,8 @@ markers = [
"slow: Slow-running tests (deselect with '-m \"not slow\"')",
"postgres: Tests that run against Postgres backend (deselect with '-m \"not postgres\"')",
"windows: Windows-specific tests (deselect with '-m \"not windows\"')",
"smoke: Fast end-to-end smoke tests for MCP flows",
"semantic: Tests requiring [semantic] extras (fastembed, sqlite-vec, openai)",
]
[tool.ruff]
@@ -91,6 +99,7 @@ dev = [
"testcontainers[postgres]>=4.0.0",
"psycopg>=3.2.0",
"pyright>=1.1.408",
"pytest-testmon>=2.2.0",
]
[tool.hatch.version]
+6 -3
View File
@@ -6,14 +6,17 @@
"url": "https://github.com/basicmachines-co/basic-memory.git",
"source": "github"
},
"version": "0.17.7",
"version": "0.18.3",
"packages": [
{
"registryType": "pypi",
"identifier": "basic-memory",
"version": "0.17.7",
"version": "0.18.3",
"runtimeHint": "uvx",
"runtimeArguments": ["basic-memory", "mcp"],
"runtimeArguments": [
{"type": "positional", "value": "basic-memory"},
{"type": "positional", "value": "mcp"}
],
"transport": {
"type": "stdio"
}
+1 -1
View File
@@ -1,7 +1,7 @@
"""basic-memory - Local-first knowledge management combining Zettelkasten with knowledge graphs"""
# Package version - updated by release automation
__version__ = "0.17.7"
__version__ = "0.18.3"
# API version for FastAPI - independent of package version
__api_version__ = "v0"
@@ -0,0 +1,154 @@
"""Add structured metadata indexes for entity frontmatter
Revision ID: d7e8f9a0b1c2
Revises: g9a0b3c4d5e6
Create Date: 2026-01-31 12:00:00.000000
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy import text
def column_exists(connection, table: str, column: str) -> bool:
"""Check if a column exists in a table (idempotent migration support)."""
if connection.dialect.name == "postgresql":
result = connection.execute(
text(
"SELECT 1 FROM information_schema.columns "
"WHERE table_name = :table AND column_name = :column"
),
{"table": table, "column": column},
)
return result.fetchone() is not None
# SQLite
result = connection.execute(text(f"PRAGMA table_info({table})"))
columns = [row[1] for row in result]
return column in columns
def index_exists(connection, index_name: str) -> bool:
"""Check if an index exists (idempotent migration support)."""
if connection.dialect.name == "postgresql":
result = connection.execute(
text("SELECT 1 FROM pg_indexes WHERE indexname = :index_name"),
{"index_name": index_name},
)
return result.fetchone() is not None
# SQLite
result = connection.execute(
text("SELECT 1 FROM sqlite_master WHERE type='index' AND name = :index_name"),
{"index_name": index_name},
)
return result.fetchone() is not None
# revision identifiers, used by Alembic.
revision: str = "d7e8f9a0b1c2"
down_revision: Union[str, None] = "6830751f5fb6"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Add JSONB/GiN indexes for Postgres and generated columns for SQLite."""
connection = op.get_bind()
dialect = connection.dialect.name
if dialect == "postgresql":
# Ensure JSONB for efficient indexing
result = connection.execute(
text(
"SELECT data_type FROM information_schema.columns "
"WHERE table_name = 'entity' AND column_name = 'entity_metadata'"
)
).fetchone()
if result and result[0] != "jsonb":
op.execute(
"ALTER TABLE entity ALTER COLUMN entity_metadata "
"TYPE jsonb USING entity_metadata::jsonb"
)
# General JSONB GIN index
op.execute(
"CREATE INDEX IF NOT EXISTS idx_entity_metadata_gin "
"ON entity USING GIN (entity_metadata jsonb_path_ops)"
)
# Common field indexes
op.execute(
"CREATE INDEX IF NOT EXISTS idx_entity_tags_json "
"ON entity USING GIN ((entity_metadata -> 'tags'))"
)
op.execute(
"CREATE INDEX IF NOT EXISTS idx_entity_frontmatter_type "
"ON entity ((entity_metadata ->> 'type'))"
)
op.execute(
"CREATE INDEX IF NOT EXISTS idx_entity_frontmatter_status "
"ON entity ((entity_metadata ->> 'status'))"
)
return
# SQLite: add generated columns for common frontmatter fields
# Constraint: SQLite ALTER TABLE ADD COLUMN only supports VIRTUAL generated columns,
# not STORED. json_extract is deterministic so VIRTUAL columns can still be indexed.
if not column_exists(connection, "entity", "tags_json"):
op.add_column(
"entity",
sa.Column(
"tags_json",
sa.Text(),
sa.Computed("json_extract(entity_metadata, '$.tags')", persisted=False),
),
)
if not column_exists(connection, "entity", "frontmatter_status"):
op.add_column(
"entity",
sa.Column(
"frontmatter_status",
sa.Text(),
sa.Computed("json_extract(entity_metadata, '$.status')", persisted=False),
),
)
if not column_exists(connection, "entity", "frontmatter_type"):
op.add_column(
"entity",
sa.Column(
"frontmatter_type",
sa.Text(),
sa.Computed("json_extract(entity_metadata, '$.type')", persisted=False),
),
)
# Index generated columns
if not index_exists(connection, "idx_entity_tags_json"):
op.create_index("idx_entity_tags_json", "entity", ["tags_json"])
if not index_exists(connection, "idx_entity_frontmatter_status"):
op.create_index("idx_entity_frontmatter_status", "entity", ["frontmatter_status"])
if not index_exists(connection, "idx_entity_frontmatter_type"):
op.create_index("idx_entity_frontmatter_type", "entity", ["frontmatter_type"])
def downgrade() -> None:
"""Best-effort downgrade (drop indexes, revert JSONB on Postgres)."""
connection = op.get_bind()
dialect = connection.dialect.name
if dialect == "postgresql":
op.execute("DROP INDEX IF EXISTS idx_entity_frontmatter_status")
op.execute("DROP INDEX IF EXISTS idx_entity_frontmatter_type")
op.execute("DROP INDEX IF EXISTS idx_entity_tags_json")
op.execute("DROP INDEX IF EXISTS idx_entity_metadata_gin")
op.execute(
"ALTER TABLE entity ALTER COLUMN entity_metadata TYPE json USING entity_metadata::json"
)
return
# SQLite: drop indexes (dropping generated columns requires table rebuild)
op.execute("DROP INDEX IF EXISTS idx_entity_frontmatter_status")
op.execute("DROP INDEX IF EXISTS idx_entity_frontmatter_type")
op.execute("DROP INDEX IF EXISTS idx_entity_tags_json")
@@ -0,0 +1,68 @@
"""Add Postgres semantic vector search tables (pgvector-aware, optional)
Revision ID: h1b2c3d4e5f6
Revises: d7e8f9a0b1c2
Create Date: 2026-02-07 00:00:00.000000
"""
from typing import Sequence, Union
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "h1b2c3d4e5f6"
down_revision: Union[str, None] = "d7e8f9a0b1c2"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Create Postgres vector chunk metadata table.
Trigger: database backend is PostgreSQL.
Why: search_vector_chunks stores text metadata with no vector-dimension
dependency, so it's safe in a migration. search_vector_embeddings (which
requires pgvector and a provider-specific dimension) is created at runtime
by PostgresSearchRepository._ensure_vector_tables(), mirroring the SQLite
pattern where vector tables are created dynamically.
Outcome: creates the dimension-independent chunks table. The embeddings
table + HNSW index are deferred to runtime.
"""
connection = op.get_bind()
if connection.dialect.name != "postgresql":
return
op.execute(
"""
CREATE TABLE IF NOT EXISTS search_vector_chunks (
id BIGSERIAL PRIMARY KEY,
entity_id INTEGER NOT NULL,
project_id INTEGER NOT NULL,
chunk_key TEXT NOT NULL,
chunk_text TEXT NOT NULL,
source_hash TEXT NOT NULL,
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
UNIQUE (project_id, entity_id, chunk_key)
)
"""
)
op.execute(
"""
CREATE INDEX IF NOT EXISTS idx_search_vector_chunks_project_entity
ON search_vector_chunks (project_id, entity_id)
"""
)
def downgrade() -> None:
"""Remove Postgres vector chunk/embedding tables.
Does not drop pgvector extension because other schema objects may depend on it.
"""
connection = op.get_bind()
if connection.dialect.name != "postgresql":
return
op.execute("DROP TABLE IF EXISTS search_vector_embeddings")
op.execute("DROP TABLE IF EXISTS search_vector_chunks")
+44 -24
View File
@@ -2,23 +2,13 @@
from contextlib import asynccontextmanager
from fastapi import FastAPI, HTTPException
from fastapi import FastAPI, HTTPException, Request
from fastapi.exception_handlers import http_exception_handler
from fastapi.routing import APIRouter
from loguru import logger
from basic_memory import __version__ as version
from basic_memory.api.container import ApiContainer, set_container
from basic_memory.api.routers import (
directory_router,
importer_router,
knowledge,
management,
memory,
project,
resource,
search,
prompt_router,
)
from basic_memory.api.v2.routers import (
knowledge_router as v2_knowledge,
project_router as v2_project,
@@ -28,8 +18,15 @@ from basic_memory.api.v2.routers import (
directory_router as v2_directory,
prompt_router as v2_prompt,
importer_router as v2_importer,
schema_router as v2_schema,
)
from basic_memory.api.v2.routers.project_router import (
add_project,
list_projects,
synchronize_projects,
)
from basic_memory.config import init_api_logging
from basic_memory.services.exceptions import EntityAlreadyExistsError
from basic_memory.services.initialization import initialize_app
@@ -88,21 +85,44 @@ app.include_router(v2_resource, prefix="/v2/projects/{project_id}")
app.include_router(v2_directory, prefix="/v2/projects/{project_id}")
app.include_router(v2_prompt, prefix="/v2/projects/{project_id}")
app.include_router(v2_importer, prefix="/v2/projects/{project_id}")
app.include_router(v2_schema, prefix="/v2/projects/{project_id}")
app.include_router(v2_project, prefix="/v2")
# Include v1 routers (/{project} is a catch-all, must come after specific prefixes)
app.include_router(knowledge.router, prefix="/{project}")
app.include_router(memory.router, prefix="/{project}")
app.include_router(resource.router, prefix="/{project}")
app.include_router(search.router, prefix="/{project}")
app.include_router(project.project_router, prefix="/{project}")
app.include_router(directory_router.router, prefix="/{project}")
app.include_router(prompt_router.router, prefix="/{project}")
app.include_router(importer_router.router, prefix="/{project}")
# Legacy web app proxy paths (compat with /proxy/projects/projects)
app.include_router(v2_project, prefix="/proxy/projects")
# Project resource router works across projects
app.include_router(project.project_resource_router)
app.include_router(management.router)
# Legacy v1 compat: older CLI versions (v0.18.0 and earlier) call /projects/...
# Using router mount causes 307 redirect which proxy doesn't follow, so add explicit routes
legacy_router = APIRouter(tags=["legacy"])
legacy_router.add_api_route("/projects/projects", list_projects, methods=["GET"])
legacy_router.add_api_route("/projects/projects", add_project, methods=["POST"])
legacy_router.add_api_route("/projects/config/sync", synchronize_projects, methods=["POST"])
app.include_router(legacy_router)
# V2 routers are the only public API surface
@app.exception_handler(EntityAlreadyExistsError)
async def entity_already_exists_error_handler(request: Request, exc: EntityAlreadyExistsError):
"""Handle entity creation conflicts (e.g., file already exists).
This is expected behavior when users try to create notes that exist,
so log at INFO level instead of ERROR.
"""
logger.info(
"Entity already exists",
url=str(request.url),
method=request.method,
path=request.url.path,
error=str(exc),
)
return await http_exception_handler(
request,
HTTPException(
status_code=409,
detail="Note already exists. Use edit_note to modify it, or delete it first.",
),
)
@app.exception_handler(Exception)
-1
View File
@@ -47,7 +47,6 @@ class ApiContainer:
"""
config = ConfigManager().config
mode = resolve_runtime_mode(
cloud_mode_enabled=config.cloud_mode_enabled,
is_test_env=config.is_test_env,
)
return cls(config=config, mode=mode)
-11
View File
@@ -1,11 +0,0 @@
"""API routers."""
from . import knowledge_router as knowledge
from . import management_router as management
from . import memory_router as memory
from . import project_router as project
from . import resource_router as resource
from . import search_router as search
from . import prompt_router as prompt
__all__ = ["knowledge", "management", "memory", "project", "resource", "search", "prompt"]
@@ -1,84 +0,0 @@
"""Router for directory tree operations."""
from typing import List, Optional
from fastapi import APIRouter, Query
from basic_memory.deps import DirectoryServiceDep, ProjectIdDep
from basic_memory.schemas.directory import DirectoryNode
router = APIRouter(prefix="/directory", tags=["directory"])
@router.get("/tree", response_model=DirectoryNode, response_model_exclude_none=True)
async def get_directory_tree(
directory_service: DirectoryServiceDep,
project_id: ProjectIdDep,
):
"""Get hierarchical directory structure from the knowledge base.
Args:
directory_service: Service for directory operations
project_id: ID of the current project
Returns:
DirectoryNode representing the root of the hierarchical tree structure
"""
# Get a hierarchical directory tree for the specific project
tree = await directory_service.get_directory_tree()
# Return the hierarchical tree
return tree
@router.get("/structure", response_model=DirectoryNode, response_model_exclude_none=True)
async def get_directory_structure(
directory_service: DirectoryServiceDep,
project_id: ProjectIdDep,
):
"""Get folder structure for navigation (no files).
Optimized endpoint for folder tree navigation. Returns only directory nodes
without file metadata. For full tree with files, use /directory/tree.
Args:
directory_service: Service for directory operations
project_id: ID of the current project
Returns:
DirectoryNode tree containing only folders (type="directory")
"""
structure = await directory_service.get_directory_structure()
return structure
@router.get("/list", response_model=List[DirectoryNode], response_model_exclude_none=True)
async def list_directory(
directory_service: DirectoryServiceDep,
project_id: ProjectIdDep,
dir_name: str = Query("/", description="Directory path to list"),
depth: int = Query(1, ge=1, le=10, description="Recursion depth (1-10)"),
file_name_glob: Optional[str] = Query(
None, description="Glob pattern for filtering file names"
),
):
"""List directory contents with filtering and depth control.
Args:
directory_service: Service for directory operations
project_id: ID of the current project
dir_name: Directory path to list (default: root "/")
depth: Recursion depth (1-10, default: 1 for immediate children only)
file_name_glob: Optional glob pattern for filtering file names (e.g., "*.md", "*meeting*")
Returns:
List of DirectoryNode objects matching the criteria
"""
# Get directory listing with filtering
nodes = await directory_service.list_directory(
dir_name=dir_name,
depth=depth,
file_name_glob=file_name_glob,
)
return nodes
@@ -1,152 +0,0 @@
"""Import router for Basic Memory API."""
import json
import logging
from fastapi import APIRouter, Form, HTTPException, UploadFile, status
from basic_memory.deps import (
ChatGPTImporterDep,
ClaudeConversationsImporterDep,
ClaudeProjectsImporterDep,
MemoryJsonImporterDep,
)
from basic_memory.importers import Importer
from basic_memory.schemas.importer import (
ChatImportResult,
EntityImportResult,
ProjectImportResult,
)
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/import", tags=["import"])
@router.post("/chatgpt", response_model=ChatImportResult)
async def import_chatgpt(
importer: ChatGPTImporterDep,
file: UploadFile,
folder: str = Form("conversations"),
) -> ChatImportResult:
"""Import conversations from ChatGPT JSON export.
Args:
file: The ChatGPT conversations.json file.
folder: The folder to place the files in.
markdown_processor: MarkdownProcessor instance.
Returns:
ChatImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
return await import_file(importer, file, folder)
@router.post("/claude/conversations", response_model=ChatImportResult)
async def import_claude_conversations(
importer: ClaudeConversationsImporterDep,
file: UploadFile,
folder: str = Form("conversations"),
) -> ChatImportResult:
"""Import conversations from Claude conversations.json export.
Args:
file: The Claude conversations.json file.
folder: The folder to place the files in.
markdown_processor: MarkdownProcessor instance.
Returns:
ChatImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
return await import_file(importer, file, folder)
@router.post("/claude/projects", response_model=ProjectImportResult)
async def import_claude_projects(
importer: ClaudeProjectsImporterDep,
file: UploadFile,
folder: str = Form("projects"),
) -> ProjectImportResult:
"""Import projects from Claude projects.json export.
Args:
file: The Claude projects.json file.
base_folder: The base folder to place the files in.
markdown_processor: MarkdownProcessor instance.
Returns:
ProjectImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
return await import_file(importer, file, folder)
@router.post("/memory-json", response_model=EntityImportResult)
async def import_memory_json(
importer: MemoryJsonImporterDep,
file: UploadFile,
folder: str = Form("conversations"),
) -> EntityImportResult:
"""Import entities and relations from a memory.json file.
Args:
file: The memory.json file.
destination_folder: Optional destination folder within the project.
markdown_processor: MarkdownProcessor instance.
Returns:
EntityImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
try:
file_data = []
file_bytes = await file.read()
file_str = file_bytes.decode("utf-8")
for line in file_str.splitlines():
json_data = json.loads(line)
file_data.append(json_data)
result = await importer.import_data(file_data, folder)
if not result.success: # pragma: no cover
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=result.error_message or "Import failed",
)
except Exception as e:
logger.exception("Import failed")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Import failed: {str(e)}",
)
return result
async def import_file(importer: Importer, file: UploadFile, destination_folder: str):
try:
# Process file
json_data = json.load(file.file)
result = await importer.import_data(json_data, destination_folder)
if not result.success: # pragma: no cover
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=result.error_message or "Import failed",
)
return result
except Exception as e:
logger.exception("Import failed")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Import failed: {str(e)}",
)
@@ -1,319 +0,0 @@
"""Router for knowledge graph operations.
⚠️ DEPRECATED: This v1 API is deprecated and will be removed on June 30, 2026.
Please migrate to /v2/{project}/knowledge endpoints which use entity IDs instead
of path-based identifiers for improved performance and stability.
Migration guide: See docs/migration/v1-to-v2.md
"""
from typing import Annotated
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Query, Response
from loguru import logger
from basic_memory.deps import (
EntityServiceDep,
get_search_service,
SearchServiceDep,
LinkResolverDep,
ProjectPathDep,
FileServiceDep,
ProjectConfigDep,
AppConfigDep,
SyncServiceDep,
)
from basic_memory.schemas import (
EntityListResponse,
EntityResponse,
DeleteEntitiesResponse,
DeleteEntitiesRequest,
)
from basic_memory.schemas.request import EditEntityRequest, MoveEntityRequest
from basic_memory.schemas.base import Permalink, Entity
router = APIRouter(
prefix="/knowledge",
tags=["knowledge"],
deprecated=True, # Marks entire router as deprecated in OpenAPI docs
)
async def resolve_relations_background(sync_service, entity_id: int, entity_permalink: str) -> None:
"""Background task to resolve relations for a specific entity.
This runs asynchronously after the API response is sent, preventing
long delays when creating entities with many relations.
"""
try:
# Only resolve relations for the newly created entity
await sync_service.resolve_relations(entity_id=entity_id)
logger.debug(
f"Background: Resolved relations for entity {entity_permalink} (id={entity_id})"
)
except Exception as e: # pragma: no cover
# Log but don't fail - this is a background task.
# Avoid forcing synthetic failures just for coverage.
logger.warning( # pragma: no cover
f"Background: Failed to resolve relations for entity {entity_permalink}: {e}"
)
## Create endpoints
@router.post("/entities", response_model=EntityResponse)
async def create_entity(
data: Entity,
background_tasks: BackgroundTasks,
entity_service: EntityServiceDep,
search_service: SearchServiceDep,
) -> EntityResponse:
"""Create an entity."""
logger.info(
"API request", endpoint="create_entity", entity_type=data.entity_type, title=data.title
)
entity = await entity_service.create_entity(data)
# reindex
await search_service.index_entity(entity, background_tasks=background_tasks)
result = EntityResponse.model_validate(entity)
logger.info(
f"API response: endpoint='create_entity' title={result.title}, permalink={result.permalink}, status_code=201"
)
return result
@router.put("/entities/{permalink:path}", response_model=EntityResponse)
async def create_or_update_entity(
project: ProjectPathDep,
permalink: Permalink,
data: Entity,
response: Response,
background_tasks: BackgroundTasks,
entity_service: EntityServiceDep,
search_service: SearchServiceDep,
file_service: FileServiceDep,
sync_service: SyncServiceDep,
) -> EntityResponse:
"""Create or update an entity. If entity exists, it will be updated, otherwise created."""
logger.info(
f"API request: create_or_update_entity for {project=}, {permalink=}, {data.entity_type=}, {data.title=}"
)
# Validate permalink matches
if data.permalink != permalink:
logger.warning(
f"API validation error: creating/updating entity with permalink mismatch - url={permalink}, data={data.permalink}",
)
raise HTTPException(
status_code=400,
detail=f"Entity permalink {data.permalink} must match URL path: '{permalink}'",
)
# Try create_or_update operation
entity, created = await entity_service.create_or_update_entity(data)
response.status_code = 201 if created else 200
# reindex
await search_service.index_entity(entity, background_tasks=background_tasks)
# Schedule relation resolution as a background task for new entities
# This prevents blocking the API response while resolving potentially many relations
if created:
background_tasks.add_task(
resolve_relations_background, sync_service, entity.id, entity.permalink or ""
)
result = EntityResponse.model_validate(entity)
logger.info(
f"API response: {result.title=}, {result.permalink=}, {created=}, status_code={response.status_code}"
)
return result
@router.patch("/entities/{identifier:path}", response_model=EntityResponse)
async def edit_entity(
identifier: str,
data: EditEntityRequest,
background_tasks: BackgroundTasks,
entity_service: EntityServiceDep,
search_service: SearchServiceDep,
) -> EntityResponse:
"""Edit an existing entity using various operations like append, prepend, find_replace, or replace_section.
This endpoint allows for targeted edits without requiring the full entity content.
"""
logger.info(
f"API request: endpoint='edit_entity', identifier='{identifier}', operation='{data.operation}'"
)
try:
# Edit the entity using the service
entity = await entity_service.edit_entity(
identifier=identifier,
operation=data.operation,
content=data.content,
section=data.section,
find_text=data.find_text,
expected_replacements=data.expected_replacements,
)
# Reindex the updated entity
await search_service.index_entity(entity, background_tasks=background_tasks)
# Return the updated entity response
result = EntityResponse.model_validate(entity)
logger.info(
"API response",
endpoint="edit_entity",
identifier=identifier,
operation=data.operation,
permalink=result.permalink,
status_code=200,
)
return result
except Exception as e:
logger.error(f"Error editing entity: {e}")
raise HTTPException(status_code=400, detail=str(e))
@router.post("/move")
async def move_entity(
data: MoveEntityRequest,
background_tasks: BackgroundTasks,
entity_service: EntityServiceDep,
project_config: ProjectConfigDep,
app_config: AppConfigDep,
search_service: SearchServiceDep,
) -> EntityResponse:
"""Move an entity to a new file location with project consistency.
This endpoint moves a note to a different path while maintaining project
consistency and optionally updating permalinks based on configuration.
"""
logger.info(
f"API request: endpoint='move_entity', identifier='{data.identifier}', destination='{data.destination_path}'"
)
try:
# Move the entity using the service
moved_entity = await entity_service.move_entity(
identifier=data.identifier,
destination_path=data.destination_path,
project_config=project_config,
app_config=app_config,
)
# Get the moved entity to reindex it
entity = await entity_service.link_resolver.resolve_link(data.destination_path)
if entity:
await search_service.index_entity(entity, background_tasks=background_tasks)
logger.info(
"API response",
endpoint="move_entity",
identifier=data.identifier,
destination=data.destination_path,
status_code=200,
)
result = EntityResponse.model_validate(moved_entity)
return result
except Exception as e:
logger.error(f"Error moving entity: {e}")
raise HTTPException(status_code=400, detail=str(e))
## Read endpoints
@router.get("/entities/{identifier:path}", response_model=EntityResponse)
async def get_entity(
entity_service: EntityServiceDep,
link_resolver: LinkResolverDep,
identifier: str,
) -> EntityResponse:
"""Get a specific entity by file path or permalink..
Args:
identifier: Entity file path or permalink
:param entity_service: EntityService
:param link_resolver: LinkResolver
"""
logger.info(f"request: get_entity with identifier={identifier}")
entity = await link_resolver.resolve_link(identifier)
if not entity:
raise HTTPException(status_code=404, detail=f"Entity {identifier} not found")
result = EntityResponse.model_validate(entity)
return result
@router.get("/entities", response_model=EntityListResponse)
async def get_entities(
entity_service: EntityServiceDep,
permalink: Annotated[list[str] | None, Query()] = None,
) -> EntityListResponse:
"""Open specific entities"""
logger.info(f"request: get_entities with permalinks={permalink}")
entities = await entity_service.get_entities_by_permalinks(permalink) if permalink else []
result = EntityListResponse(
entities=[EntityResponse.model_validate(entity) for entity in entities]
)
return result
## Delete endpoints
@router.delete("/entities/{identifier:path}", response_model=DeleteEntitiesResponse)
async def delete_entity(
identifier: str,
background_tasks: BackgroundTasks,
entity_service: EntityServiceDep,
link_resolver: LinkResolverDep,
search_service=Depends(get_search_service),
) -> DeleteEntitiesResponse:
"""Delete a single entity and remove from search index."""
logger.info(f"request: delete_entity with identifier={identifier}")
entity = await link_resolver.resolve_link(identifier)
if entity is None:
return DeleteEntitiesResponse(deleted=False)
# Delete the entity
deleted = await entity_service.delete_entity(entity.permalink or entity.id)
# Remove from search index (entity, observations, and relations)
background_tasks.add_task(search_service.handle_delete, entity)
result = DeleteEntitiesResponse(deleted=deleted)
return result
@router.post("/entities/delete", response_model=DeleteEntitiesResponse)
async def delete_entities(
data: DeleteEntitiesRequest,
background_tasks: BackgroundTasks,
entity_service: EntityServiceDep,
search_service=Depends(get_search_service),
) -> DeleteEntitiesResponse:
"""Delete entities and remove from search index."""
logger.info(f"request: delete_entities with data={data}")
deleted = False
# Remove each deleted entity from search index
for permalink in data.permalinks:
deleted = await entity_service.delete_entity(permalink)
background_tasks.add_task(search_service.delete_by_permalink, permalink)
result = DeleteEntitiesResponse(deleted=deleted)
return result
@@ -1,80 +0,0 @@
"""Management router for basic-memory API."""
import asyncio
from fastapi import APIRouter, Request
from loguru import logger
from pydantic import BaseModel
from basic_memory.config import ConfigManager
from basic_memory.deps import SyncServiceDep, ProjectRepositoryDep
router = APIRouter(prefix="/management", tags=["management"])
class WatchStatusResponse(BaseModel):
"""Response model for watch status."""
running: bool
"""Whether the watch service is currently running."""
@router.get("/watch/status", response_model=WatchStatusResponse)
async def get_watch_status(request: Request) -> WatchStatusResponse:
"""Get the current status of the watch service."""
return WatchStatusResponse(
running=request.app.state.watch_task is not None and not request.app.state.watch_task.done()
)
@router.post("/watch/start", response_model=WatchStatusResponse)
async def start_watch_service(
request: Request, project_repository: ProjectRepositoryDep, sync_service: SyncServiceDep
) -> WatchStatusResponse:
"""Start the watch service if it's not already running."""
# needed because of circular imports from sync -> app
from basic_memory.sync import WatchService
from basic_memory.sync.background_sync import create_background_sync_task
if request.app.state.watch_task is not None and not request.app.state.watch_task.done():
# Watch service is already running
return WatchStatusResponse(running=True)
app_config = ConfigManager().config
# Create and start a new watch service
logger.info("Starting watch service via management API")
# Get services needed for the watch task
watch_service = WatchService(
app_config=app_config,
project_repository=project_repository,
)
# Create and store the task
watch_task = create_background_sync_task(sync_service, watch_service)
request.app.state.watch_task = watch_task
return WatchStatusResponse(running=True)
@router.post("/watch/stop", response_model=WatchStatusResponse)
async def stop_watch_service(request: Request) -> WatchStatusResponse: # pragma: no cover
"""Stop the watch service if it's running."""
if request.app.state.watch_task is None or request.app.state.watch_task.done():
# Watch service is not running
return WatchStatusResponse(running=False)
# Cancel the running task
logger.info("Stopping watch service via management API")
request.app.state.watch_task.cancel()
# Wait for it to be properly cancelled
try:
await request.app.state.watch_task
except asyncio.CancelledError:
pass
request.app.state.watch_task = None
return WatchStatusResponse(running=False)
@@ -1,90 +0,0 @@
"""Routes for memory:// URI operations."""
from typing import Annotated, Optional
from fastapi import APIRouter, Query
from loguru import logger
from basic_memory.deps import ContextServiceDep, EntityRepositoryDep
from basic_memory.schemas.base import TimeFrame, parse_timeframe
from basic_memory.schemas.memory import (
GraphContext,
normalize_memory_url,
)
from basic_memory.schemas.search import SearchItemType
from basic_memory.api.routers.utils import to_graph_context
router = APIRouter(prefix="/memory", tags=["memory"])
@router.get("/recent", response_model=GraphContext)
async def recent(
context_service: ContextServiceDep,
entity_repository: EntityRepositoryDep,
type: Annotated[list[SearchItemType] | None, Query()] = None,
depth: int = 1,
timeframe: TimeFrame = "7d",
page: int = 1,
page_size: int = 10,
max_related: int = 10,
) -> GraphContext:
# return all types by default
types = (
[SearchItemType.ENTITY, SearchItemType.RELATION, SearchItemType.OBSERVATION]
if not type
else type
)
logger.debug(
f"Getting recent context: `{types}` depth: `{depth}` timeframe: `{timeframe}` page: `{page}` page_size: `{page_size}` max_related: `{max_related}`"
)
# Parse timeframe
since = parse_timeframe(timeframe)
limit = page_size
offset = (page - 1) * page_size
# Build context
context = await context_service.build_context(
types=types, depth=depth, since=since, limit=limit, offset=offset, max_related=max_related
)
recent_context = await to_graph_context(
context, entity_repository=entity_repository, page=page, page_size=page_size
)
logger.debug(f"Recent context: {recent_context.model_dump_json()}")
return recent_context
# get_memory_context needs to be declared last so other paths can match
@router.get("/{uri:path}", response_model=GraphContext)
async def get_memory_context(
context_service: ContextServiceDep,
entity_repository: EntityRepositoryDep,
uri: str,
depth: int = 1,
timeframe: Optional[TimeFrame] = None,
page: int = 1,
page_size: int = 10,
max_related: int = 10,
) -> GraphContext:
"""Get rich context from memory:// URI."""
# add the project name from the config to the url as the "host
# Parse URI
logger.debug(
f"Getting context for URI: `{uri}` depth: `{depth}` timeframe: `{timeframe}` page: `{page}` page_size: `{page_size}` max_related: `{max_related}`"
)
memory_url = normalize_memory_url(uri)
# Parse timeframe
since = parse_timeframe(timeframe) if timeframe else None
limit = page_size
offset = (page - 1) * page_size
# Build context
context = await context_service.build_context(
memory_url, depth=depth, since=since, limit=limit, offset=offset, max_related=max_related
)
return await to_graph_context(
context, entity_repository=entity_repository, page=page, page_size=page_size
)
@@ -1,460 +0,0 @@
"""Router for project management."""
import os
from fastapi import APIRouter, HTTPException, Path, Body, BackgroundTasks, Response, Query
from typing import Optional
from loguru import logger
from basic_memory.deps import (
ProjectConfigDep,
ProjectServiceDep,
ProjectPathDep,
SyncServiceDep,
)
from basic_memory.schemas import ProjectInfoResponse, SyncReportResponse
from basic_memory.schemas.project_info import (
ProjectList,
ProjectItem,
ProjectInfoRequest,
ProjectStatusResponse,
)
from basic_memory.utils import normalize_project_path
# Router for resources in a specific project
# The ProjectPathDep is used in the path as a prefix, so the request path is like /{project}/project/info
project_router = APIRouter(prefix="/project", tags=["project"])
# Router for managing project resources
project_resource_router = APIRouter(prefix="/projects", tags=["project_management"])
@project_router.get("/info", response_model=ProjectInfoResponse)
async def get_project_info(
project_service: ProjectServiceDep,
project: ProjectPathDep,
) -> ProjectInfoResponse:
"""Get comprehensive information about the specified Basic Memory project."""
return await project_service.get_project_info(project)
@project_router.get("/item", response_model=ProjectItem)
async def get_project(
project_service: ProjectServiceDep,
project: ProjectPathDep,
) -> ProjectItem:
"""Get bassic info about the specified Basic Memory project."""
found_project = await project_service.get_project(project)
if not found_project:
raise HTTPException(
status_code=404, detail=f"Project: '{project}' does not exist"
) # pragma: no cover
return ProjectItem(
id=found_project.id,
external_id=found_project.external_id,
name=found_project.name,
path=normalize_project_path(found_project.path),
is_default=found_project.is_default or False,
)
# Update a project
@project_router.patch("/{name}", response_model=ProjectStatusResponse)
async def update_project(
project_service: ProjectServiceDep,
name: str = Path(..., description="Name of the project to update"),
path: Optional[str] = Body(None, description="New absolute path for the project"),
is_active: Optional[bool] = Body(None, description="Status of the project (active/inactive)"),
) -> ProjectStatusResponse:
"""Update a project's information in configuration and database.
Args:
name: The name of the project to update
path: Optional new absolute path for the project
is_active: Optional status update for the project
Returns:
Response confirming the project was updated
"""
try:
# Validate that path is absolute if provided
if path and not os.path.isabs(path):
raise HTTPException(status_code=400, detail="Path must be absolute")
# Get original project info for the response
old_project = await project_service.get_project(name)
if not old_project:
raise HTTPException(
status_code=400, detail=f"Project '{name}' not found in configuration"
)
old_project_info = ProjectItem(
id=old_project.id,
external_id=old_project.external_id,
name=old_project.name,
path=old_project.path,
is_default=old_project.is_default or False,
)
if path:
await project_service.move_project(name, path)
elif is_active is not None:
await project_service.update_project(name, is_active=is_active)
# Get updated project info
updated_project = await project_service.get_project(name)
if not updated_project:
raise HTTPException( # pragma: no cover
status_code=404, detail=f"Project '{name}' not found after update"
)
return ProjectStatusResponse(
message=f"Project '{name}' updated successfully",
status="success",
default=(name == project_service.default_project),
old_project=old_project_info,
new_project=ProjectItem(
id=updated_project.id,
external_id=updated_project.external_id,
name=updated_project.name,
path=updated_project.path,
is_default=updated_project.is_default or False,
),
)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e)) # pragma: no cover
# Sync project filesystem
@project_router.post("/sync")
async def sync_project(
background_tasks: BackgroundTasks,
sync_service: SyncServiceDep,
project_config: ProjectConfigDep,
force_full: bool = Query(
False, description="Force full scan, bypassing watermark optimization"
),
run_in_background: bool = Query(True, description="Run in background"),
):
"""Force project filesystem sync to database.
Scans the project directory and updates the database with any new or modified files.
Args:
background_tasks: FastAPI background tasks
sync_service: Sync service for this project
project_config: Project configuration
force_full: If True, force a full scan even if watermark exists
run_in_background: If True, run sync in background and return immediately
Returns:
Response confirming sync was initiated (background) or SyncReportResponse (foreground)
"""
if run_in_background:
background_tasks.add_task(
sync_service.sync, project_config.home, project_config.name, force_full=force_full
)
logger.info(
f"Filesystem sync initiated for project: {project_config.name} (force_full={force_full})"
)
return {
"status": "sync_started",
"message": f"Filesystem sync initiated for project '{project_config.name}'",
}
else:
report = await sync_service.sync(
project_config.home, project_config.name, force_full=force_full
)
logger.info(
f"Filesystem sync completed for project: {project_config.name} (force_full={force_full})"
)
return SyncReportResponse.from_sync_report(report)
@project_router.post("/status", response_model=SyncReportResponse)
async def project_sync_status(
sync_service: SyncServiceDep,
project_config: ProjectConfigDep,
) -> SyncReportResponse:
"""Scan directory for changes compared to database state.
Args:
sync_service: Sync service for this project
project_config: Project configuration
Returns:
Scan report with details on files that need syncing
"""
logger.info(f"Scanning filesystem for project: {project_config.name}") # pragma: no cover
sync_report = await sync_service.scan(project_config.home) # pragma: no cover
return SyncReportResponse.from_sync_report(sync_report) # pragma: no cover
# List all available projects
@project_resource_router.get("/projects", response_model=ProjectList)
async def list_projects(
project_service: ProjectServiceDep,
) -> ProjectList:
"""List all configured projects.
Returns:
A list of all projects with metadata
"""
projects = await project_service.list_projects()
default_project = project_service.default_project
project_items = [
ProjectItem(
id=project.id,
external_id=project.external_id,
name=project.name,
path=normalize_project_path(project.path),
is_default=project.is_default or False,
)
for project in projects
]
return ProjectList(
projects=project_items,
default_project=default_project,
)
# Add a new project
@project_resource_router.post("/projects", response_model=ProjectStatusResponse, status_code=201)
async def add_project(
response: Response,
project_data: ProjectInfoRequest,
project_service: ProjectServiceDep,
) -> ProjectStatusResponse:
"""Add a new project to configuration and database.
Args:
project_data: The project name and path, with option to set as default
Returns:
Response confirming the project was added
"""
# Check if project already exists before attempting to add
existing_project = await project_service.get_project(project_data.name)
if existing_project:
# Project exists - check if paths match for true idempotency
# Normalize paths for comparison (resolve symlinks, etc.)
from pathlib import Path
requested_path = Path(project_data.path).resolve()
existing_path = Path(existing_project.path).resolve()
if requested_path == existing_path:
# Same name, same path - return 200 OK (idempotent)
response.status_code = 200
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message=f"Project '{project_data.name}' already exists",
status="success",
default=existing_project.is_default or False,
new_project=ProjectItem(
id=existing_project.id,
external_id=existing_project.external_id,
name=existing_project.name,
path=existing_project.path,
is_default=existing_project.is_default or False,
),
)
else:
# Same name, different path - this is an error
raise HTTPException(
status_code=400,
detail=f"Project '{project_data.name}' already exists with different path. Existing: {existing_project.path}, Requested: {project_data.path}",
)
try: # pragma: no cover
# The service layer now handles cloud mode validation and path sanitization
await project_service.add_project(
project_data.name, project_data.path, set_default=project_data.set_default
)
# Fetch the newly created project to get its ID
new_project = await project_service.get_project(project_data.name)
if not new_project:
raise HTTPException(status_code=500, detail="Failed to retrieve newly created project")
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message=f"Project '{new_project.name}' added successfully",
status="success",
default=project_data.set_default,
new_project=ProjectItem(
id=new_project.id,
external_id=new_project.external_id,
name=new_project.name,
path=new_project.path,
is_default=new_project.is_default or False,
),
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e))
# Remove a project
@project_resource_router.delete("/{name}", response_model=ProjectStatusResponse)
async def remove_project(
project_service: ProjectServiceDep,
name: str = Path(..., description="Name of the project to remove"),
delete_notes: bool = Query(
False, description="If True, delete project directory from filesystem"
),
) -> ProjectStatusResponse:
"""Remove a project from configuration and database.
Args:
name: The name of the project to remove
delete_notes: If True, delete the project directory from the filesystem
Returns:
Response confirming the project was removed
"""
try:
old_project = await project_service.get_project(name)
if not old_project: # pragma: no cover
raise HTTPException(
status_code=404, detail=f"Project: '{name}' does not exist"
) # pragma: no cover
# Check if trying to delete the default project
if name == project_service.default_project:
available_projects = await project_service.list_projects()
other_projects = [p.name for p in available_projects if p.name != name]
detail = f"Cannot delete default project '{name}'. "
if other_projects:
detail += (
f"Set another project as default first. Available: {', '.join(other_projects)}"
)
else:
detail += "This is the only project in your configuration."
raise HTTPException(status_code=400, detail=detail)
await project_service.remove_project(name, delete_notes=delete_notes)
return ProjectStatusResponse(
message=f"Project '{old_project.name}' removed successfully",
status="success",
default=False,
old_project=ProjectItem(
id=old_project.id,
external_id=old_project.external_id,
name=old_project.name,
path=old_project.path,
is_default=old_project.is_default or False,
),
new_project=None,
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e))
# Set a project as default
@project_resource_router.put("/{name}/default", response_model=ProjectStatusResponse)
async def set_default_project(
project_service: ProjectServiceDep,
name: str = Path(..., description="Name of the project to set as default"),
) -> ProjectStatusResponse:
"""Set a project as the default project.
Args:
name: The name of the project to set as default
Returns:
Response confirming the project was set as default
"""
try:
# Get the old default project
default_name = project_service.default_project
default_project = await project_service.get_project(default_name)
if not default_project: # pragma: no cover
raise HTTPException( # pragma: no cover
status_code=404, detail=f"Default Project: '{default_name}' does not exist"
)
# get the new project
new_default_project = await project_service.get_project(name)
if not new_default_project: # pragma: no cover
raise HTTPException(
status_code=404, detail=f"Project: '{name}' does not exist"
) # pragma: no cover
await project_service.set_default_project(name)
return ProjectStatusResponse(
message=f"Project '{name}' set as default successfully",
status="success",
default=True,
old_project=ProjectItem(
id=default_project.id,
external_id=default_project.external_id,
name=default_name,
path=default_project.path,
is_default=False,
),
new_project=ProjectItem(
id=new_default_project.id,
external_id=new_default_project.external_id,
name=name,
path=new_default_project.path,
is_default=True,
),
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e))
# Get the default project
@project_resource_router.get("/default", response_model=ProjectItem)
async def get_default_project(
project_service: ProjectServiceDep,
) -> ProjectItem:
"""Get the default project.
Returns:
Response with project default information
"""
# Get the old default project
default_name = project_service.default_project
default_project = await project_service.get_project(default_name)
if not default_project: # pragma: no cover
raise HTTPException( # pragma: no cover
status_code=404, detail=f"Default Project: '{default_name}' does not exist"
)
return ProjectItem(
id=default_project.id,
external_id=default_project.external_id,
name=default_project.name,
path=default_project.path,
is_default=True,
)
# Synchronize projects between config and database
@project_resource_router.post("/config/sync", response_model=ProjectStatusResponse)
async def synchronize_projects(
project_service: ProjectServiceDep,
) -> ProjectStatusResponse:
"""Synchronize projects between configuration file and database.
Ensures that all projects in the configuration file exist in the database
and vice versa.
Returns:
Response confirming synchronization was completed
"""
try: # pragma: no cover
await project_service.synchronize_projects()
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message="Projects synchronized successfully between configuration and database",
status="success",
default=False,
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e))
@@ -1,260 +0,0 @@
"""Router for prompt-related operations.
This router is responsible for rendering various prompts using Handlebars templates.
It centralizes all prompt formatting logic that was previously in the MCP prompts.
"""
from datetime import datetime, timezone
from fastapi import APIRouter, HTTPException, status
from loguru import logger
from basic_memory.api.routers.utils import to_graph_context, to_search_results
from basic_memory.api.template_loader import template_loader
from basic_memory.schemas.base import parse_timeframe
from basic_memory.deps import (
ContextServiceDep,
EntityRepositoryDep,
SearchServiceDep,
EntityServiceDep,
)
from basic_memory.schemas.prompt import (
ContinueConversationRequest,
SearchPromptRequest,
PromptResponse,
PromptMetadata,
)
from basic_memory.schemas.search import SearchItemType, SearchQuery
router = APIRouter(prefix="/prompt", tags=["prompt"])
@router.post("/continue-conversation", response_model=PromptResponse)
async def continue_conversation(
search_service: SearchServiceDep,
entity_service: EntityServiceDep,
context_service: ContextServiceDep,
entity_repository: EntityRepositoryDep,
request: ContinueConversationRequest,
) -> PromptResponse:
"""Generate a prompt for continuing a conversation.
This endpoint takes a topic and/or timeframe and generates a prompt with
relevant context from the knowledge base.
Args:
request: The request parameters
Returns:
Formatted continuation prompt with context
"""
logger.info(
f"Generating continue conversation prompt, topic: {request.topic}, timeframe: {request.timeframe}"
)
since = parse_timeframe(request.timeframe) if request.timeframe else None
# Initialize search results
search_results = []
# Get data needed for template
if request.topic:
query = SearchQuery(text=request.topic, after_date=request.timeframe)
results = await search_service.search(query, limit=request.search_items_limit)
search_results = await to_search_results(entity_service, results)
# Build context from results
all_hierarchical_results = []
for result in search_results:
if hasattr(result, "permalink") and result.permalink:
# Get hierarchical context using the new dataclass-based approach
context_result = await context_service.build_context(
result.permalink,
depth=request.depth,
since=since,
max_related=request.related_items_limit,
include_observations=True, # Include observations for entities
)
# Process results into the schema format
graph_context = await to_graph_context(
context_result, entity_repository=entity_repository
)
# Add results to our collection (limit to top results for each permalink)
if graph_context.results:
all_hierarchical_results.extend(graph_context.results[:3])
# Limit to a reasonable number of total results
all_hierarchical_results = all_hierarchical_results[:10]
template_context = {
"topic": request.topic,
"timeframe": request.timeframe,
"hierarchical_results": all_hierarchical_results,
"has_results": len(all_hierarchical_results) > 0,
}
else:
# If no topic, get recent activity
context_result = await context_service.build_context(
types=[SearchItemType.ENTITY],
depth=request.depth,
since=since,
max_related=request.related_items_limit,
include_observations=True,
)
recent_context = await to_graph_context(context_result, entity_repository=entity_repository)
hierarchical_results = recent_context.results[:5] # Limit to top 5 recent items
template_context = {
"topic": f"Recent Activity from ({request.timeframe})",
"timeframe": request.timeframe,
"hierarchical_results": hierarchical_results,
"has_results": len(hierarchical_results) > 0,
}
try:
# Render template
rendered_prompt = await template_loader.render(
"prompts/continue_conversation.hbs", template_context
)
# Calculate metadata
# Count items of different types
observation_count = 0
relation_count = 0
entity_count = 0
# Get the hierarchical results from the template context
hierarchical_results_for_count = template_context.get("hierarchical_results", [])
# For topic-based search
if request.topic:
for item in hierarchical_results_for_count:
if hasattr(item, "observations"):
observation_count += len(item.observations) if item.observations else 0
if hasattr(item, "related_results"):
for related in item.related_results or []:
if hasattr(related, "type"):
if related.type == "relation":
relation_count += 1
elif related.type == "entity": # pragma: no cover
entity_count += 1 # pragma: no cover
# For recent activity
else:
for item in hierarchical_results_for_count:
if hasattr(item, "observations"):
observation_count += len(item.observations) if item.observations else 0
if hasattr(item, "related_results"):
for related in item.related_results or []:
if hasattr(related, "type"):
if related.type == "relation":
relation_count += 1
elif related.type == "entity": # pragma: no cover
entity_count += 1 # pragma: no cover
# Build metadata
metadata = {
"query": request.topic,
"timeframe": request.timeframe,
"search_count": len(search_results)
if request.topic
else 0, # Original search results count
"context_count": len(hierarchical_results_for_count),
"observation_count": observation_count,
"relation_count": relation_count,
"total_items": (
len(hierarchical_results_for_count)
+ observation_count
+ relation_count
+ entity_count
),
"search_limit": request.search_items_limit,
"context_depth": request.depth,
"related_limit": request.related_items_limit,
"generated_at": datetime.now(timezone.utc).isoformat(),
}
prompt_metadata = PromptMetadata(**metadata)
return PromptResponse(
prompt=rendered_prompt, context=template_context, metadata=prompt_metadata
)
except Exception as e:
logger.error(f"Error rendering continue conversation template: {e}")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Error rendering prompt template: {str(e)}",
)
@router.post("/search", response_model=PromptResponse)
async def search_prompt(
search_service: SearchServiceDep,
entity_service: EntityServiceDep,
request: SearchPromptRequest,
page: int = 1,
page_size: int = 10,
) -> PromptResponse:
"""Generate a prompt for search results.
This endpoint takes a search query and formats the results into a helpful
prompt with context and suggestions.
Args:
request: The search parameters
page: The page number for pagination
page_size: The number of results per page, defaults to 10
Returns:
Formatted search results prompt with context
"""
logger.info(f"Generating search prompt, query: {request.query}, timeframe: {request.timeframe}")
limit = page_size
offset = (page - 1) * page_size
query = SearchQuery(text=request.query, after_date=request.timeframe)
results = await search_service.search(query, limit=limit, offset=offset)
search_results = await to_search_results(entity_service, results)
template_context = {
"query": request.query,
"timeframe": request.timeframe,
"results": search_results,
"has_results": len(search_results) > 0,
"result_count": len(search_results),
}
try:
# Render template
rendered_prompt = await template_loader.render("prompts/search.hbs", template_context)
# Build metadata
metadata = {
"query": request.query,
"timeframe": request.timeframe,
"search_count": len(search_results),
"context_count": len(search_results),
"observation_count": 0, # Search results don't include observations
"relation_count": 0, # Search results don't include relations
"total_items": len(search_results),
"search_limit": limit,
"context_depth": 0, # No context depth for basic search
"related_limit": 0, # No related items for basic search
"generated_at": datetime.now(timezone.utc).isoformat(),
}
prompt_metadata = PromptMetadata(**metadata)
return PromptResponse(
prompt=rendered_prompt, context=template_context, metadata=prompt_metadata
)
except Exception as e:
logger.error(f"Error rendering search template: {e}")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Error rendering prompt template: {str(e)}",
)
@@ -1,252 +0,0 @@
"""Routes for getting entity content."""
import tempfile
import uuid
from pathlib import Path
from typing import Annotated, Union
from fastapi import APIRouter, HTTPException, BackgroundTasks, Body, Response
from fastapi.responses import FileResponse, JSONResponse
from loguru import logger
from basic_memory.deps import (
ProjectConfigDep,
LinkResolverDep,
SearchServiceDep,
EntityServiceDep,
FileServiceDep,
EntityRepositoryDep,
)
from basic_memory.repository.search_repository import SearchIndexRow
from basic_memory.schemas.memory import normalize_memory_url
from basic_memory.schemas.search import SearchQuery, SearchItemType
from basic_memory.models.knowledge import Entity as EntityModel
from datetime import datetime
router = APIRouter(prefix="/resource", tags=["resources"])
def _mtime_to_datetime(entity: EntityModel) -> datetime:
"""Convert entity mtime (file modification time) to datetime.
Returns the file's actual modification time, falling back to updated_at
if mtime is not available.
"""
if entity.mtime: # pragma: no cover
return datetime.fromtimestamp(entity.mtime).astimezone() # pragma: no cover
return entity.updated_at
def get_entity_ids(item: SearchIndexRow) -> set[int]:
match item.type:
case SearchItemType.ENTITY:
return {item.id}
case SearchItemType.OBSERVATION:
return {item.entity_id} # pyright: ignore [reportReturnType]
case SearchItemType.RELATION:
from_entity = item.from_id
to_entity = item.to_id # pyright: ignore [reportReturnType]
return {from_entity, to_entity} if to_entity else {from_entity} # pyright: ignore [reportReturnType]
case _: # pragma: no cover
raise ValueError(f"Unexpected type: {item.type}")
@router.get("/{identifier:path}", response_model=None)
async def get_resource_content(
config: ProjectConfigDep,
link_resolver: LinkResolverDep,
search_service: SearchServiceDep,
entity_service: EntityServiceDep,
file_service: FileServiceDep,
background_tasks: BackgroundTasks,
identifier: str,
page: int = 1,
page_size: int = 10,
) -> Union[Response, FileResponse]:
"""Get resource content by identifier: name or permalink."""
logger.debug(f"Getting content for: {identifier}")
# Find single entity by permalink
entity = await link_resolver.resolve_link(identifier)
results = [entity] if entity else []
# pagination for multiple results
limit = page_size
offset = (page - 1) * page_size
# search using the identifier as a permalink
if not results:
# if the identifier contains a wildcard, use GLOB search
query = (
SearchQuery(permalink_match=identifier)
if "*" in identifier
else SearchQuery(permalink=identifier)
)
search_results = await search_service.search(query, limit, offset)
if not search_results:
raise HTTPException(status_code=404, detail=f"Resource not found: {identifier}")
# get the deduplicated entities related to the search results
entity_ids = {id for result in search_results for id in get_entity_ids(result)}
results = await entity_service.get_entities_by_id(list(entity_ids))
# return single response
if len(results) == 1:
entity = results[0]
# Check file exists via file_service (for cloud compatibility)
if not await file_service.exists(entity.file_path):
raise HTTPException(
status_code=404,
detail=f"File not found: {entity.file_path}",
)
# Read content via file_service as bytes (works with both local and S3)
content = await file_service.read_file_bytes(entity.file_path)
content_type = file_service.content_type(entity.file_path)
return Response(content=content, media_type=content_type)
# for multiple files, initialize a temporary file for writing the results
with tempfile.NamedTemporaryFile(delete=False, mode="w", suffix=".md") as tmp_file:
temp_file_path = tmp_file.name
for result in results:
# Read content for each entity
content = await file_service.read_entity_content(result)
memory_url = normalize_memory_url(result.permalink)
modified_date = _mtime_to_datetime(result).isoformat()
checksum = result.checksum[:8] if result.checksum else ""
# Prepare the delimited content
response_content = f"--- {memory_url} {modified_date} {checksum}\n"
response_content += f"\n{content}\n"
response_content += "\n"
# Write content directly to the temporary file in append mode
tmp_file.write(response_content)
# Ensure all content is written to disk
tmp_file.flush()
# Schedule the temporary file to be deleted after the response
background_tasks.add_task(cleanup_temp_file, temp_file_path)
# Return the file response
return FileResponse(path=temp_file_path)
def cleanup_temp_file(file_path: str):
"""Delete the temporary file."""
try:
Path(file_path).unlink() # Deletes the file
logger.debug(f"Temporary file deleted: {file_path}")
except Exception as e: # pragma: no cover
logger.error(f"Error deleting temporary file {file_path}: {e}")
@router.put("/{file_path:path}")
async def write_resource(
config: ProjectConfigDep,
file_service: FileServiceDep,
entity_repository: EntityRepositoryDep,
search_service: SearchServiceDep,
file_path: str,
content: Annotated[str, Body()],
) -> JSONResponse:
"""Write content to a file in the project.
This endpoint allows writing content directly to a file in the project.
Also creates an entity record and indexes the file for search.
Args:
file_path: Path to write to, relative to project root
request: Contains the content to write
Returns:
JSON response with file information
"""
try:
# Get content from request body
# Defensive type checking: ensure content is a string
# FastAPI should validate this, but if a dict somehow gets through
# (e.g., via JSON body parsing), we need to catch it here
if isinstance(content, dict):
logger.error( # pragma: no cover
f"Error writing resource {file_path}: "
f"content is a dict, expected string. Keys: {list(content.keys())}"
)
raise HTTPException( # pragma: no cover
status_code=400,
detail="content must be a string, not a dict. "
"Ensure request body is sent as raw string content, not JSON object.",
)
# Ensure it's UTF-8 string content
if isinstance(content, bytes): # pragma: no cover
content_str = content.decode("utf-8")
else:
content_str = str(content)
# Cloud compatibility: do not assume a local filesystem path structure.
# Delegate directory creation + writes to the configured FileService (local or S3).
await file_service.ensure_directory(Path(file_path).parent)
checksum = await file_service.write_file(file_path, content_str)
# Get file info
file_metadata = await file_service.get_file_metadata(file_path)
# Determine file details
file_name = Path(file_path).name
content_type = file_service.content_type(file_path)
entity_type = "canvas" if file_path.endswith(".canvas") else "file"
# Check if entity already exists
existing_entity = await entity_repository.get_by_file_path(file_path)
if existing_entity:
# Update existing entity
entity = await entity_repository.update(
existing_entity.id,
{
"title": file_name,
"entity_type": entity_type,
"content_type": content_type,
"file_path": file_path,
"checksum": checksum,
"updated_at": file_metadata.modified_at,
},
)
status_code = 200
else:
# Create a new entity model
# Explicitly set external_id to ensure NOT NULL constraint is satisfied (fixes #512)
entity = EntityModel(
external_id=str(uuid.uuid4()),
title=file_name,
entity_type=entity_type,
content_type=content_type,
file_path=file_path,
checksum=checksum,
created_at=file_metadata.created_at,
updated_at=file_metadata.modified_at,
)
entity = await entity_repository.add(entity)
status_code = 201
# Index the file for search
await search_service.index_entity(entity) # pyright: ignore
# Return success response
return JSONResponse(
status_code=status_code,
content={
"file_path": file_path,
"checksum": checksum,
"size": file_metadata.size,
"created_at": file_metadata.created_at.timestamp(),
"modified_at": file_metadata.modified_at.timestamp(),
},
)
except Exception as e: # pragma: no cover
logger.error(f"Error writing resource {file_path}: {e}")
raise HTTPException(status_code=500, detail=f"Failed to write resource: {str(e)}")
@@ -1,36 +0,0 @@
"""Router for search operations."""
from fastapi import APIRouter, BackgroundTasks
from basic_memory.api.routers.utils import to_search_results
from basic_memory.schemas.search import SearchQuery, SearchResponse
from basic_memory.deps import SearchServiceDep, EntityServiceDep
router = APIRouter(prefix="/search", tags=["search"])
@router.post("/", response_model=SearchResponse)
async def search(
query: SearchQuery,
search_service: SearchServiceDep,
entity_service: EntityServiceDep,
page: int = 1,
page_size: int = 10,
):
"""Search across all knowledge and documents."""
limit = page_size
offset = (page - 1) * page_size
results = await search_service.search(query, limit=limit, offset=offset)
search_results = await to_search_results(entity_service, results)
return SearchResponse(
results=search_results,
current_page=page,
page_size=page_size,
)
@router.post("/reindex")
async def reindex(background_tasks: BackgroundTasks, search_service: SearchServiceDep):
"""Recreate and populate the search index."""
await search_service.reindex_all(background_tasks=background_tasks)
return {"status": "ok", "message": "Reindex initiated"}
@@ -8,6 +8,7 @@ from basic_memory.api.v2.routers.resource_router import router as resource_route
from basic_memory.api.v2.routers.directory_router import router as directory_router
from basic_memory.api.v2.routers.prompt_router import router as prompt_router
from basic_memory.api.v2.routers.importer_router import router as importer_router
from basic_memory.api.v2.routers.schema_router import router as schema_router
__all__ = [
"knowledge_router",
@@ -18,4 +19,5 @@ __all__ = [
"directory_router",
"prompt_router",
"importer_router",
"schema_router",
]
@@ -32,14 +32,14 @@ async def import_chatgpt(
importer: ChatGPTImporterV2ExternalDep,
file: UploadFile,
project_id: str = Path(..., description="Project external UUID"),
folder: str = Form("conversations"),
directory: str = Form("conversations"),
) -> ChatImportResult:
"""Import conversations from ChatGPT JSON export.
Args:
project_id: Project external UUID from URL path
file: The ChatGPT conversations.json file.
folder: The folder to place the files in.
directory: The directory to place the files in.
importer: ChatGPT importer instance.
Returns:
@@ -49,7 +49,7 @@ async def import_chatgpt(
HTTPException: If import fails.
"""
logger.info(f"V2 Importing ChatGPT conversations for project {project_id}")
return await import_file(importer, file, folder)
return await import_file(importer, file, directory)
@router.post("/claude/conversations", response_model=ChatImportResult)
@@ -57,14 +57,14 @@ async def import_claude_conversations(
importer: ClaudeConversationsImporterV2ExternalDep,
file: UploadFile,
project_id: str = Path(..., description="Project external UUID"),
folder: str = Form("conversations"),
directory: str = Form("conversations"),
) -> ChatImportResult:
"""Import conversations from Claude conversations.json export.
Args:
project_id: Project external UUID from URL path
file: The Claude conversations.json file.
folder: The folder to place the files in.
directory: The directory to place the files in.
importer: Claude conversations importer instance.
Returns:
@@ -74,7 +74,7 @@ async def import_claude_conversations(
HTTPException: If import fails.
"""
logger.info(f"V2 Importing Claude conversations for project {project_id}")
return await import_file(importer, file, folder)
return await import_file(importer, file, directory)
@router.post("/claude/projects", response_model=ProjectImportResult)
@@ -82,14 +82,14 @@ async def import_claude_projects(
importer: ClaudeProjectsImporterV2ExternalDep,
file: UploadFile,
project_id: str = Path(..., description="Project external UUID"),
folder: str = Form("projects"),
directory: str = Form("projects"),
) -> ProjectImportResult:
"""Import projects from Claude projects.json export.
Args:
project_id: Project external UUID from URL path
file: The Claude projects.json file.
folder: The base folder to place the files in.
directory: The base directory to place the files in.
importer: Claude projects importer instance.
Returns:
@@ -99,7 +99,7 @@ async def import_claude_projects(
HTTPException: If import fails.
"""
logger.info(f"V2 Importing Claude projects for project {project_id}")
return await import_file(importer, file, folder)
return await import_file(importer, file, directory)
@router.post("/memory-json", response_model=EntityImportResult)
@@ -107,14 +107,14 @@ async def import_memory_json(
importer: MemoryJsonImporterV2ExternalDep,
file: UploadFile,
project_id: str = Path(..., description="Project external UUID"),
folder: str = Form("conversations"),
directory: str = Form("conversations"),
) -> EntityImportResult:
"""Import entities and relations from a memory.json file.
Args:
project_id: Project external UUID from URL path
file: The memory.json file.
folder: Optional destination folder within the project.
directory: Optional destination directory within the project.
importer: Memory JSON importer instance.
Returns:
@@ -132,7 +132,7 @@ async def import_memory_json(
json_data = json.loads(line)
file_data.append(json_data)
result = await importer.import_data(file_data, folder)
result = await importer.import_data(file_data, directory)
if not result.success: # pragma: no cover
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
@@ -147,13 +147,13 @@ async def import_memory_json(
return result
async def import_file(importer: Importer, file: UploadFile, destination_folder: str):
async def import_file(importer: Importer, file: UploadFile, destination_directory: str):
"""Helper function to import a file using an importer instance.
Args:
importer: The importer instance to use
file: The file to import
destination_folder: Destination folder for imported content
destination_directory: Destination directory for imported content
Returns:
Import result from the importer
@@ -164,7 +164,7 @@ async def import_file(importer: Importer, file: UploadFile, destination_folder:
try:
# Process file
json_data = json.load(file.file)
result = await importer.import_data(json_data, destination_folder)
result = await importer.import_data(json_data, destination_directory)
if not result.success: # pragma: no cover
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
@@ -10,7 +10,7 @@ Key improvements:
- Simplified caching strategies
"""
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Response, Path
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Response, Path, Query
from loguru import logger
from basic_memory.deps import (
@@ -19,9 +19,10 @@ from basic_memory.deps import (
LinkResolverV2ExternalDep,
ProjectConfigV2ExternalDep,
AppConfigDep,
SyncServiceV2ExternalDep,
EntityRepositoryV2ExternalDep,
ProjectExternalIdPathDep,
TaskSchedulerDep,
FileServiceV2ExternalDep,
)
from basic_memory.schemas import DeleteEntitiesResponse
from basic_memory.schemas.base import Entity
@@ -31,27 +32,27 @@ from basic_memory.schemas.v2 import (
EntityResolveResponse,
EntityResponseV2,
MoveEntityRequestV2,
MoveDirectoryRequestV2,
DeleteDirectoryRequestV2,
)
from basic_memory.schemas.response import DirectoryMoveResult, DirectoryDeleteResult
router = APIRouter(prefix="/knowledge", tags=["knowledge-v2"])
async def resolve_relations_background(sync_service, entity_id: int, entity_permalink: str) -> None:
"""Background task to resolve relations for a specific entity.
This runs asynchronously after the API response is sent, preventing
long delays when creating entities with many relations.
"""
try: # pragma: no cover
# Only resolve relations for the newly created entity
await sync_service.resolve_relations(entity_id=entity_id) # pragma: no cover
logger.debug( # pragma: no cover
f"Background: Resolved relations for entity {entity_permalink} (id={entity_id})"
)
except Exception as e: # pragma: no cover
# Log but don't fail - this is a background task
logger.warning( # pragma: no cover
f"Background: Failed to resolve relations for entity {entity_permalink}: {e}"
def _schedule_vector_sync_if_enabled(
*,
task_scheduler,
app_config,
entity_id: int,
project_id: int,
) -> None:
"""Schedule out-of-band vector sync only when semantic search is enabled."""
if app_config.semantic_search_enabled:
task_scheduler.schedule(
"sync_entity_vectors",
entity_id=entity_id,
project_id=project_id,
)
@@ -100,8 +101,12 @@ async def resolve_identifier(
resolution_method = "external_id" if entity else "search"
# If not found by external_id, try other resolution methods
# Pass source_path for context-aware resolution (prefers notes closer to source)
# Pass strict to control fuzzy search fallback (default False allows fuzzy matching)
if not entity:
entity = await link_resolver.resolve_link(data.identifier)
entity = await link_resolver.resolve_link(
data.identifier, source_path=data.source_path, strict=data.strict
)
if entity:
# Determine resolution method
if entity.permalink == data.identifier:
@@ -179,24 +184,50 @@ async def create_entity(
background_tasks: BackgroundTasks,
entity_service: EntityServiceV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
task_scheduler: TaskSchedulerDep,
file_service: FileServiceV2ExternalDep,
app_config: AppConfigDep,
fast: bool = Query(
True, description="If true, write quickly and defer indexing to background tasks."
),
) -> EntityResponseV2:
"""Create a new entity.
Args:
data: Entity data to create
fast: If True, defer indexing to background tasks
Returns:
Created entity with generated external_id (UUID)
Created entity with generated external_id (UUID) and file content
"""
logger.info(
"API v2 request", endpoint="create_entity", entity_type=data.entity_type, title=data.title
)
entity = await entity_service.create_entity(data)
if fast:
entity = await entity_service.fast_write_entity(data)
task_scheduler.schedule(
"reindex_entity",
entity_id=entity.id,
project_id=project_id,
)
else:
entity = await entity_service.create_entity(data)
await search_service.index_entity(entity)
_schedule_vector_sync_if_enabled(
task_scheduler=task_scheduler,
app_config=app_config,
entity_id=entity.id,
project_id=project_id,
)
# reindex
await search_service.index_entity(entity, background_tasks=background_tasks)
result = EntityResponseV2.model_validate(entity)
if fast:
result = result.model_copy(update={"observations": [], "relations": []})
# Always read and return file content
content = await file_service.read_file_content(entity.file_path)
result = result.model_copy(update={"content": content})
logger.info(
f"API v2 response: endpoint='create_entity' external_id={entity.external_id}, title={result.title}, permalink={result.permalink}, status_code=201"
@@ -215,9 +246,14 @@ async def update_entity_by_id(
project_id: ProjectExternalIdPathDep,
entity_service: EntityServiceV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
sync_service: SyncServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
task_scheduler: TaskSchedulerDep,
file_service: FileServiceV2ExternalDep,
app_config: AppConfigDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
fast: bool = Query(
True, description="If true, write quickly and defer indexing to background tasks."
),
) -> EntityResponseV2:
"""Update an entity by external ID.
@@ -226,30 +262,61 @@ async def update_entity_by_id(
Args:
entity_id: External ID (UUID string)
data: Updated entity data
fast: If True, defer indexing to background tasks
Returns:
Updated entity
Updated entity with file content
"""
logger.info(f"API v2 request: update_entity_by_id entity_id={entity_id}")
# Check if entity exists
# Check if entity exists (external_id is the source of truth for v2)
existing = await entity_repository.get_by_external_id(entity_id)
created = existing is None
# Perform update or create
entity, _ = await entity_service.create_or_update_entity(data)
response.status_code = 201 if created else 200
if fast:
entity = await entity_service.fast_write_entity(data, external_id=entity_id)
response.status_code = 200 if existing else 201
task_scheduler.schedule(
"reindex_entity",
entity_id=entity.id,
project_id=project_id,
resolve_relations=created,
)
else:
if existing:
# Update the existing entity in-place to avoid path-based duplication
entity = await entity_service.update_entity(existing, data)
response.status_code = 200
else:
# Create new entity, then bind external_id to the requested UUID
entity = await entity_service.create_entity(data)
if entity.external_id != entity_id:
entity = await entity_repository.update(
entity.id,
{"external_id": entity_id},
)
if not entity:
raise HTTPException(
status_code=404,
detail=f"Entity with external_id '{entity_id}' not found",
)
response.status_code = 201
# reindex
await search_service.index_entity(entity, background_tasks=background_tasks)
# Schedule relation resolution for new entities
if created:
background_tasks.add_task( # pragma: no cover
resolve_relations_background, sync_service, entity.id, entity.permalink or ""
await search_service.index_entity(entity)
_schedule_vector_sync_if_enabled(
task_scheduler=task_scheduler,
app_config=app_config,
entity_id=entity.id,
project_id=project_id,
)
result = EntityResponseV2.model_validate(entity)
if fast:
result = result.model_copy(update={"observations": [], "relations": []})
# Always read and return file content
content = await file_service.read_file_content(entity.file_path)
result = result.model_copy(update={"content": content})
logger.info(
f"API v2 response: external_id={entity_id}, created={created}, status_code={response.status_code}"
@@ -265,16 +332,23 @@ async def edit_entity_by_id(
entity_service: EntityServiceV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
task_scheduler: TaskSchedulerDep,
file_service: FileServiceV2ExternalDep,
app_config: AppConfigDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
fast: bool = Query(
True, description="If true, write quickly and defer indexing to background tasks."
),
) -> EntityResponseV2:
"""Edit an existing entity by external ID using operations like append, prepend, etc.
Args:
entity_id: External ID (UUID string)
data: Edit operation details
fast: If True, defer indexing to background tasks
Returns:
Updated entity
Updated entity with file content
Raises:
HTTPException: 404 if entity not found, 400 if edit fails
@@ -291,21 +365,47 @@ async def edit_entity_by_id(
)
try:
# Edit using the entity's permalink or path
identifier = entity.permalink or entity.file_path
updated_entity = await entity_service.edit_entity(
identifier=identifier,
operation=data.operation,
content=data.content,
section=data.section,
find_text=data.find_text,
expected_replacements=data.expected_replacements,
)
if fast:
updated_entity = await entity_service.fast_edit_entity(
entity=entity,
operation=data.operation,
content=data.content,
section=data.section,
find_text=data.find_text,
expected_replacements=data.expected_replacements,
)
task_scheduler.schedule(
"reindex_entity",
entity_id=updated_entity.id,
project_id=project_id,
)
else:
# Edit using the entity's permalink or path
identifier = entity.permalink or entity.file_path
updated_entity = await entity_service.edit_entity(
identifier=identifier,
operation=data.operation,
content=data.content,
section=data.section,
find_text=data.find_text,
expected_replacements=data.expected_replacements,
)
# Reindex
await search_service.index_entity(updated_entity, background_tasks=background_tasks)
await search_service.index_entity(updated_entity)
_schedule_vector_sync_if_enabled(
task_scheduler=task_scheduler,
app_config=app_config,
entity_id=updated_entity.id,
project_id=project_id,
)
result = EntityResponseV2.model_validate(updated_entity)
if fast:
result = result.model_copy(update={"observations": [], "relations": []})
# Always read and return file content
content = await file_service.read_file_content(updated_entity.file_path)
result = result.model_copy(update={"content": content})
logger.info(
f"API v2 response: external_id={entity_id}, operation='{data.operation}', status_code=200"
@@ -372,6 +472,7 @@ async def move_entity(
project_config: ProjectConfigV2ExternalDep,
app_config: AppConfigDep,
search_service: SearchServiceV2ExternalDep,
task_scheduler: TaskSchedulerDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
) -> EntityResponseV2:
"""Move an entity to a new file location.
@@ -410,7 +511,13 @@ async def move_entity(
# Reindex at new location
reindexed_entity = await entity_service.link_resolver.resolve_link(data.destination_path)
if reindexed_entity:
await search_service.index_entity(reindexed_entity, background_tasks=background_tasks)
await search_service.index_entity(reindexed_entity)
_schedule_vector_sync_if_enabled(
task_scheduler=task_scheduler,
app_config=app_config,
entity_id=reindexed_entity.id,
project_id=project_id,
)
result = EntityResponseV2.model_validate(moved_entity)
@@ -423,3 +530,107 @@ async def move_entity(
except Exception as e:
logger.error(f"Error moving entity: {e}")
raise HTTPException(status_code=400, detail=str(e))
## Move directory endpoint
@router.post("/move-directory", response_model=DirectoryMoveResult)
async def move_directory(
data: MoveDirectoryRequestV2,
background_tasks: BackgroundTasks,
project_id: ProjectExternalIdPathDep,
entity_service: EntityServiceV2ExternalDep,
project_config: ProjectConfigV2ExternalDep,
app_config: AppConfigDep,
search_service: SearchServiceV2ExternalDep,
task_scheduler: TaskSchedulerDep,
) -> DirectoryMoveResult:
"""Move all entities in a directory to a new location.
V2 API uses project external_id in the URL path for stable references.
Moves all files within a source directory to a destination directory,
updating database records and optionally updating permalinks.
Args:
project_id: Project external ID from URL path
data: Move request with source and destination directories
Returns:
DirectoryMoveResult with counts and details of moved files
"""
logger.info(
f"API v2 request: move_directory source='{data.source_directory}', destination='{data.destination_directory}'"
)
try:
# Move the directory using the service
result = await entity_service.move_directory(
source_directory=data.source_directory,
destination_directory=data.destination_directory,
project_config=project_config,
app_config=app_config,
)
# Reindex moved entities
for file_path in result.moved_files:
entity = await entity_service.link_resolver.resolve_link(file_path)
if entity:
await search_service.index_entity(entity)
_schedule_vector_sync_if_enabled(
task_scheduler=task_scheduler,
app_config=app_config,
entity_id=entity.id,
project_id=project_id,
)
logger.info(
f"API v2 response: move_directory "
f"total={result.total_files}, success={result.successful_moves}, failed={result.failed_moves}"
)
return result
except Exception as e:
logger.error(f"Error moving directory: {e}")
raise HTTPException(status_code=400, detail=str(e))
## Delete directory endpoint
@router.post("/delete-directory", response_model=DirectoryDeleteResult)
async def delete_directory(
data: DeleteDirectoryRequestV2,
project_id: ProjectExternalIdPathDep,
entity_service: EntityServiceV2ExternalDep,
) -> DirectoryDeleteResult:
"""Delete all entities in a directory.
V2 API uses project external_id in the URL path for stable references.
Deletes all files within a directory, updating database records and
removing files from the filesystem.
Args:
project_id: Project external ID from URL path
data: Delete request with directory path
Returns:
DirectoryDeleteResult with counts and details of deleted files
"""
logger.info(f"API v2 request: delete_directory directory='{data.directory}'")
try:
# Delete the directory using the service
result = await entity_service.delete_directory(
directory=data.directory,
)
logger.info(
f"API v2 response: delete_directory "
f"total={result.total_files}, success={result.successful_deletes}, failed={result.failed_deletes}"
)
return result
except Exception as e:
logger.error(f"Error deleting directory: {e}")
raise HTTPException(status_code=400, detail=str(e))
@@ -16,7 +16,7 @@ from basic_memory.schemas.memory import (
normalize_memory_url,
)
from basic_memory.schemas.search import SearchItemType
from basic_memory.api.routers.utils import to_graph_context
from basic_memory.api.v2.utils import to_graph_context
# Note: No prefix here - it's added during registration as /v2/{project_id}/memory
router = APIRouter(tags=["memory"])
@@ -19,9 +19,17 @@ from loguru import logger
from basic_memory.deps import (
ProjectServiceDep,
ProjectRepositoryDep,
ProjectConfigV2ExternalDep,
SyncServiceV2ExternalDep,
TaskSchedulerDep,
ProjectExternalIdPathDep,
)
from basic_memory.schemas import SyncReportResponse
from basic_memory.schemas.project_info import (
ProjectItem,
ProjectList,
ProjectInfoRequest,
ProjectInfoResponse,
ProjectStatusResponse,
)
from basic_memory.schemas.v2 import ProjectResolveRequest, ProjectResolveResponse
@@ -30,6 +38,175 @@ from basic_memory.utils import normalize_project_path, generate_permalink
router = APIRouter(prefix="/projects", tags=["project_management-v2"])
@router.get("/", response_model=ProjectList)
async def list_projects(
project_service: ProjectServiceDep,
) -> ProjectList:
"""List all configured projects.
Returns:
A list of all projects with metadata
"""
projects = await project_service.list_projects()
default_project = project_service.default_project
project_items = [
ProjectItem(
id=project.id,
external_id=project.external_id,
name=project.name,
path=normalize_project_path(project.path),
is_default=project.is_default or False,
)
for project in projects
]
return ProjectList(
projects=project_items,
default_project=default_project,
)
@router.post("/", response_model=ProjectStatusResponse, status_code=201)
async def add_project(
project_data: ProjectInfoRequest,
project_service: ProjectServiceDep,
) -> ProjectStatusResponse:
"""Add a new project to configuration and database.
Args:
project_data: The project name and path, with option to set as default
Returns:
Response confirming the project was added
"""
# Check if project already exists before attempting to add
existing_project = await project_service.get_project(project_data.name)
if existing_project:
# Project exists - check if paths match for true idempotency
# Normalize paths for comparison (resolve symlinks, etc.)
requested_path = os.path.abspath(os.path.expanduser(project_data.path))
existing_path = os.path.abspath(os.path.expanduser(existing_project.path))
if requested_path == existing_path:
# Same name, same path - return 200 OK (idempotent)
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message=f"Project '{project_data.name}' already exists",
status="success",
default=existing_project.is_default or False,
new_project=ProjectItem(
id=existing_project.id,
external_id=existing_project.external_id,
name=existing_project.name,
path=existing_project.path,
is_default=existing_project.is_default or False,
),
)
else:
# Same name, different path - this is an error
raise HTTPException(
status_code=400,
detail=(
f"Project '{project_data.name}' already exists with different path. "
f"Existing: {existing_project.path}, Requested: {project_data.path}"
),
)
try: # pragma: no cover
# The service layer handles cloud mode validation and path sanitization
await project_service.add_project(
project_data.name, project_data.path, set_default=project_data.set_default
)
# Fetch the newly created project to get its ID
new_project = await project_service.get_project(project_data.name)
if not new_project:
raise HTTPException(status_code=500, detail="Failed to retrieve newly created project")
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message=f"Project '{new_project.name}' added successfully",
status="success",
default=project_data.set_default,
new_project=ProjectItem(
id=new_project.id,
external_id=new_project.external_id,
name=new_project.name,
path=new_project.path,
is_default=new_project.is_default or False,
),
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e))
@router.post("/config/sync", response_model=ProjectStatusResponse)
async def synchronize_projects(
project_service: ProjectServiceDep,
) -> ProjectStatusResponse:
"""Synchronize projects between configuration file and database."""
try: # pragma: no cover
await project_service.synchronize_projects()
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message="Projects synchronized successfully between configuration and database",
status="success",
default=False,
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e))
@router.post("/{project_id}/sync")
async def sync_project(
sync_service: SyncServiceV2ExternalDep,
project_config: ProjectConfigV2ExternalDep,
task_scheduler: TaskSchedulerDep,
project_internal_id: ProjectExternalIdPathDep,
force_full: bool = Query(
False, description="Force full scan, bypassing watermark optimization"
),
run_in_background: bool = Query(True, description="Run in background"),
):
"""Force project filesystem sync to database."""
if run_in_background:
task_scheduler.schedule(
"sync_project",
project_id=project_internal_id,
force_full=force_full,
)
logger.info(
f"Filesystem sync initiated for project: {project_config.name} (force_full={force_full})"
)
return {
"status": "sync_started",
"message": f"Filesystem sync initiated for project '{project_config.name}'",
}
report = await sync_service.sync(
project_config.home, project_config.name, force_full=force_full
)
logger.info(
f"Filesystem sync completed for project: {project_config.name} (force_full={force_full})"
)
return SyncReportResponse.from_sync_report(report)
@router.post("/{project_id}/status", response_model=SyncReportResponse)
async def get_project_status(
sync_service: SyncServiceV2ExternalDep,
project_config: ProjectConfigV2ExternalDep,
project_id: str = Path(..., description="Project external ID (UUID)"),
force_full: bool = Query(
False, description="Force full scan, bypassing watermark optimization"
),
) -> SyncReportResponse:
"""Get sync status of files vs database for a project."""
logger.info(f"API v2 request: get_project_status for project_id={project_id}")
report = await sync_service.scan(project_config.home, force_full=force_full)
return SyncReportResponse.from_sync_report(report)
@router.post("/resolve", response_model=ProjectResolveResponse)
async def resolve_project_identifier(
data: ProjectResolveRequest,
@@ -147,6 +324,22 @@ async def get_project_by_id(
)
@router.get("/{project_id}/info", response_model=ProjectInfoResponse)
async def get_project_info_by_id(
project_service: ProjectServiceDep,
project_repository: ProjectRepositoryDep,
project_id: str = Path(..., description="Project external ID (UUID)"),
) -> ProjectInfoResponse:
"""Get detailed project information by external ID."""
logger.info(f"API v2 request: get_project_info_by_id for project_id={project_id}")
project = await project_repository.get_by_external_id(project_id)
if not project:
raise HTTPException(
status_code=404, detail=f"Project with external_id '{project_id}' not found"
)
return await project_service.get_project_info(project.name)
@router.patch("/{project_id}", response_model=ProjectStatusResponse)
async def update_project_by_id(
project_service: ProjectServiceDep,
@@ -9,7 +9,7 @@ from datetime import datetime, timezone
from fastapi import APIRouter, HTTPException, status, Path
from loguru import logger
from basic_memory.api.routers.utils import to_graph_context, to_search_results
from basic_memory.api.v2.utils import to_graph_context, to_search_results
from basic_memory.api.template_loader import template_loader
from basic_memory.schemas.base import parse_timeframe
from basic_memory.deps import (
@@ -0,0 +1,351 @@
"""V2 router for schema operations.
Provides endpoints for schema validation, inference, and drift detection.
The schema system validates notes against Picoschema definitions without
introducing any new data model -- it works entirely with existing
observations and relations.
Flow: Entity loaded with eager observations/relations -> convert to tuples -> core functions.
"""
from pathlib import Path as FilePath
from fastapi import APIRouter, Path, Query
from basic_memory.deps import EntityRepositoryV2ExternalDep
from basic_memory.models.knowledge import Entity
from basic_memory.schemas.schema import (
ValidationReport,
InferenceReport,
DriftReport,
NoteValidationResponse,
FieldResultResponse,
FieldFrequencyResponse,
DriftFieldResponse,
)
from basic_memory.schema.resolver import resolve_schema
from basic_memory.schema.validator import validate_note
from basic_memory.schema.inference import infer_schema, NoteData, ObservationData, RelationData
from basic_memory.schema.diff import diff_schema
from basic_memory.utils import generate_permalink
# Note: No prefix here -- it's added during registration as /v2/{project_id}/schema
router = APIRouter(tags=["schema"])
# --- ORM to core data conversion ---
def _entity_observations(entity: Entity) -> list[ObservationData]:
"""Extract ObservationData from an entity's observations."""
return [ObservationData(obs.category, obs.content) for obs in entity.observations]
def _entity_relations(entity: Entity) -> list[RelationData]:
"""Extract RelationData from an entity's outgoing relations.
Carries the target entity's type on each relation so the inference engine
can suggest correct types (e.g. works_at -> Organization, not the source type).
"""
return [
RelationData(
relation_type=rel.relation_type,
target_name=rel.to_name,
target_entity_type=rel.to_entity.entity_type if rel.to_entity else None,
)
for rel in entity.outgoing_relations
]
def _entity_to_note_data(entity: Entity) -> NoteData:
"""Convert an ORM Entity to a NoteData for inference/diff analysis."""
return NoteData(
identifier=entity.permalink or entity.file_path,
observations=_entity_observations(entity),
relations=_entity_relations(entity),
)
def _entity_frontmatter(entity: Entity) -> dict:
"""Build a frontmatter dict from an entity for schema resolution."""
frontmatter = dict(entity.entity_metadata) if entity.entity_metadata else {}
if entity.entity_type:
frontmatter.setdefault("type", entity.entity_type)
return frontmatter
# --- Validation ---
@router.post("/schema/validate", response_model=ValidationReport)
async def validate_schema(
entity_repository: EntityRepositoryV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
entity_type: str | None = Query(None, description="Entity type to validate"),
identifier: str | None = Query(None, description="Specific note identifier"),
):
"""Validate notes against their resolved schemas.
Validates a specific note (by identifier) or all notes of a given type.
Returns warnings/errors based on the schema's validation mode.
"""
results: list[NoteValidationResponse] = []
# --- Single note validation ---
if identifier:
entity = await entity_repository.get_by_permalink(identifier)
if not entity:
return ValidationReport(entity_type=entity_type, total_notes=0, results=[])
frontmatter = _entity_frontmatter(entity)
schema_ref = frontmatter.get("schema")
async def search_fn(query: str) -> list[dict]:
entities = await _find_schema_entities(
entity_repository,
query,
allow_reference_match=isinstance(schema_ref, str) and query == schema_ref,
)
return [_entity_frontmatter(e) for e in entities]
schema_def = await resolve_schema(frontmatter, search_fn)
if schema_def:
result = validate_note(
entity.permalink or identifier,
schema_def,
_entity_observations(entity),
_entity_relations(entity),
)
results.append(_to_note_validation_response(result))
return ValidationReport(
entity_type=entity_type or entity.entity_type,
total_notes=1,
valid_count=1 if (results and results[0].passed) else 0,
warning_count=sum(len(r.warnings) for r in results),
error_count=sum(len(r.errors) for r in results),
results=results,
)
# --- Batch validation by entity type ---
entities = await _find_by_entity_type(entity_repository, entity_type) if entity_type else []
for entity in entities:
frontmatter = _entity_frontmatter(entity)
schema_ref = frontmatter.get("schema")
async def search_fn(query: str) -> list[dict]:
entities = await _find_schema_entities(
entity_repository,
query,
allow_reference_match=isinstance(schema_ref, str) and query == schema_ref,
)
return [_entity_frontmatter(e) for e in entities]
schema_def = await resolve_schema(frontmatter, search_fn)
if schema_def:
result = validate_note(
entity.permalink or entity.file_path,
schema_def,
_entity_observations(entity),
_entity_relations(entity),
)
results.append(_to_note_validation_response(result))
valid = sum(1 for r in results if r.passed)
return ValidationReport(
entity_type=entity_type,
total_notes=len(results),
total_entities=len(entities),
valid_count=valid,
warning_count=sum(len(r.warnings) for r in results),
error_count=sum(len(r.errors) for r in results),
results=results,
)
# --- Inference ---
@router.post("/schema/infer", response_model=InferenceReport)
async def infer_schema_endpoint(
entity_repository: EntityRepositoryV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
entity_type: str = Query(..., description="Entity type to analyze"),
threshold: float = Query(0.25, description="Minimum frequency for optional fields"),
):
"""Infer a schema from existing notes of a given type.
Examines observation categories and relation types across all notes
of the given type. Returns frequency analysis and suggested Picoschema.
"""
entities = await _find_by_entity_type(entity_repository, entity_type)
notes_data = [_entity_to_note_data(entity) for entity in entities]
result = infer_schema(entity_type, notes_data, optional_threshold=threshold)
return InferenceReport(
entity_type=result.entity_type,
notes_analyzed=result.notes_analyzed,
field_frequencies=[
FieldFrequencyResponse(
name=f.name,
source=f.source,
count=f.count,
total=f.total,
percentage=f.percentage,
sample_values=f.sample_values,
is_array=f.is_array,
target_type=f.target_type,
)
for f in result.field_frequencies
],
suggested_schema=result.suggested_schema,
suggested_required=result.suggested_required,
suggested_optional=result.suggested_optional,
excluded=result.excluded,
)
# --- Drift Detection ---
@router.get("/schema/diff/{entity_type}", response_model=DriftReport)
async def diff_schema_endpoint(
entity_repository: EntityRepositoryV2ExternalDep,
entity_type: str = Path(..., description="Entity type to check for drift"),
project_id: str = Path(..., description="Project external UUID"),
):
"""Show drift between a schema definition and actual note usage.
Compares the existing schema for an entity type against how notes
of that type are actually structured. Identifies new fields, dropped
fields, and cardinality changes.
"""
async def search_fn(query: str) -> list[dict]:
entities = await _find_schema_entities(entity_repository, query)
return [_entity_frontmatter(e) for e in entities]
# Resolve schema by entity type
schema_frontmatter = {"type": entity_type}
schema_def = await resolve_schema(schema_frontmatter, search_fn)
if not schema_def:
return DriftReport(entity_type=entity_type, schema_found=False)
# Collect all notes of this type
entities = await _find_by_entity_type(entity_repository, entity_type)
notes_data = [_entity_to_note_data(entity) for entity in entities]
result = diff_schema(schema_def, notes_data)
return DriftReport(
entity_type=entity_type,
new_fields=[
DriftFieldResponse(
name=f.name,
source=f.source,
count=f.count,
total=f.total,
percentage=f.percentage,
)
for f in result.new_fields
],
dropped_fields=[
DriftFieldResponse(
name=f.name,
source=f.source,
count=f.count,
total=f.total,
percentage=f.percentage,
)
for f in result.dropped_fields
],
cardinality_changes=result.cardinality_changes,
)
# --- Helpers ---
async def _find_by_entity_type(
entity_repository: EntityRepositoryV2ExternalDep,
entity_type: str,
) -> list[Entity]:
"""Find all entities of a given type using the repository's select pattern."""
query = entity_repository.select().where(Entity.entity_type == entity_type)
result = await entity_repository.execute_query(query)
return list(result.scalars().all())
async def _find_schema_entities(
entity_repository: EntityRepositoryV2ExternalDep,
target_entity_type: str,
*,
allow_reference_match: bool = False,
) -> list[Entity]:
"""Find schema entities for resolver lookups.
Resolution strategy:
1) Always try exact entity_metadata['entity'] match (for implicit type lookup
and explicit references that use entity names)
2) Only when allow_reference_match=True and no entity match was found, try
exact reference matching by title/permalink (explicit schema references)
"""
query = entity_repository.select().where(Entity.entity_type == "schema")
result = await entity_repository.execute_query(query)
entities = list(result.scalars().all())
normalized_target = generate_permalink(target_entity_type)
entity_matches = [
e
for e in entities
if e.entity_metadata
and isinstance(e.entity_metadata.get("entity"), str)
and generate_permalink(e.entity_metadata["entity"]) == normalized_target
]
if entity_matches:
return entity_matches
if not allow_reference_match:
return []
reference_matches: list[Entity] = []
for entity in entities:
candidate_refs: list[str] = []
if entity.title:
candidate_refs.append(entity.title)
if entity.permalink:
candidate_refs.append(entity.permalink)
candidate_refs.append(FilePath(entity.permalink).name)
if any(generate_permalink(ref) == normalized_target for ref in candidate_refs):
reference_matches.append(entity)
return reference_matches
def _to_note_validation_response(result) -> NoteValidationResponse:
"""Convert a core ValidationResult to a Pydantic response model."""
return NoteValidationResponse(
note_identifier=result.note_identifier,
schema_entity=result.schema_entity,
passed=result.passed,
field_results=[
FieldResultResponse(
field_name=fr.field.name,
field_type=fr.field.type,
required=fr.field.required,
status=fr.status,
values=fr.values,
message=fr.message,
)
for fr in result.field_results
],
unmatched_observations=result.unmatched_observations,
unmatched_relations=result.unmatched_relations,
warnings=result.warnings,
errors=result.errors,
)
@@ -4,11 +4,20 @@ This router uses external_id UUIDs for stable, API-friendly routing.
V1 uses string-based project names which are less efficient and less stable.
"""
from fastapi import APIRouter, BackgroundTasks, Path
from fastapi import APIRouter, HTTPException, Path
from basic_memory.api.routers.utils import to_search_results
from basic_memory.api.v2.utils import to_search_results
from basic_memory.repository.semantic_errors import (
SemanticDependenciesMissingError,
SemanticSearchDisabledError,
)
from basic_memory.schemas.search import SearchQuery, SearchResponse
from basic_memory.deps import SearchServiceV2ExternalDep, EntityServiceV2ExternalDep
from basic_memory.deps import (
SearchServiceV2ExternalDep,
EntityServiceV2ExternalDep,
TaskSchedulerDep,
ProjectExternalIdPathDep,
)
# Note: No prefix here - it's added during registration as /v2/{project_id}/search
router = APIRouter(tags=["search"])
@@ -40,7 +49,14 @@ async def search(
"""
limit = page_size
offset = (page - 1) * page_size
results = await search_service.search(query, limit=limit, offset=offset)
try:
results = await search_service.search(query, limit=limit, offset=offset)
except SemanticSearchDisabledError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
except SemanticDependenciesMissingError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
search_results = await to_search_results(entity_service, results)
return SearchResponse(
results=search_results,
@@ -51,9 +67,8 @@ async def search(
@router.post("/search/reindex")
async def reindex(
background_tasks: BackgroundTasks,
search_service: SearchServiceV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
task_scheduler: TaskSchedulerDep,
project_id: ProjectExternalIdPathDep,
):
"""Recreate and populate the search index for a project.
@@ -63,11 +78,10 @@ async def reindex(
Args:
project_id: Project external UUID from URL path
background_tasks: FastAPI background tasks handler
search_service: Search service scoped to project
task_scheduler: Task scheduler for background work
Returns:
Status message indicating reindex has been initiated
"""
await search_service.reindex_all(background_tasks=background_tasks)
task_scheduler.schedule("reindex_project", project_id=project_id)
return {"status": "ok", "message": "Reindex initiated"}
@@ -24,29 +24,42 @@ async def to_graph_context(
page: Optional[int] = None,
page_size: Optional[int] = None,
):
# First pass: collect all entity IDs needed for relations
# First pass: collect all entity IDs needed for external_id lookup
# This includes: entity primary results, observation parent entities, relation from/to entities
entity_ids_needed: set[int] = set()
for context_item in context_result.results:
for item in (
[context_item.primary_result] + context_item.observations + context_item.related_results
):
if item.type == SearchItemType.RELATION:
if item.type == SearchItemType.ENTITY:
# Entity's own ID for its external_id
entity_ids_needed.add(item.id)
elif item.type == SearchItemType.OBSERVATION:
# Parent entity ID for entity_external_id
if item.entity_id: # pyright: ignore
entity_ids_needed.add(item.entity_id) # pyright: ignore
elif item.type == SearchItemType.RELATION:
# Source and target entity IDs for external_ids
if item.from_id: # pyright: ignore
entity_ids_needed.add(item.from_id) # pyright: ignore
if item.to_id:
entity_ids_needed.add(item.to_id)
# Batch fetch all entities at once
entity_lookup: dict[int, str] = {}
# Batch fetch all entities at once - get both title and external_id
entity_title_lookup: dict[int, str] = {}
entity_external_id_lookup: dict[int, str] = {}
if entity_ids_needed:
entities = await entity_repository.find_by_ids(list(entity_ids_needed))
entity_lookup = {e.id: e.title for e in entities}
for e in entities:
entity_title_lookup[e.id] = e.title
entity_external_id_lookup[e.id] = e.external_id
# Helper function to convert items to summaries
def to_summary(item: SearchIndexRow | ContextResultRow):
match item.type:
case SearchItemType.ENTITY:
return EntitySummary(
external_id=entity_external_id_lookup.get(item.id, ""),
entity_id=item.id,
title=item.title, # pyright: ignore
permalink=item.permalink,
@@ -55,10 +68,14 @@ async def to_graph_context(
created_at=item.created_at,
)
case SearchItemType.OBSERVATION:
entity_ext_id = None
if item.entity_id: # pyright: ignore
entity_ext_id = entity_external_id_lookup.get(item.entity_id) # pyright: ignore
return ObservationSummary(
observation_id=item.id,
entity_id=item.entity_id, # pyright: ignore
title=item.title, # pyright: ignore
entity_external_id=entity_ext_id,
title=entity_title_lookup.get(item.entity_id), # pyright: ignore
file_path=item.file_path,
category=item.category, # pyright: ignore
content=item.content, # pyright: ignore
@@ -66,8 +83,10 @@ async def to_graph_context(
created_at=item.created_at,
)
case SearchItemType.RELATION:
from_title = entity_lookup.get(item.from_id) if item.from_id else None # pyright: ignore
to_title = entity_lookup.get(item.to_id) if item.to_id else None
from_title = entity_title_lookup.get(item.from_id) if item.from_id else None # pyright: ignore
to_title = entity_title_lookup.get(item.to_id) if item.to_id else None
from_ext_id = entity_external_id_lookup.get(item.from_id) if item.from_id else None # pyright: ignore
to_ext_id = entity_external_id_lookup.get(item.to_id) if item.to_id else None
return RelationSummary(
relation_id=item.id,
entity_id=item.entity_id, # pyright: ignore
@@ -77,8 +96,10 @@ async def to_graph_context(
relation_type=item.relation_type, # pyright: ignore
from_entity=from_title,
from_entity_id=item.from_id, # pyright: ignore
from_entity_external_id=from_ext_id,
to_entity=to_title,
to_entity_id=item.to_id,
to_entity_external_id=to_ext_id,
created_at=item.created_at,
)
case _: # pragma: no cover
+22 -1
View File
@@ -9,6 +9,7 @@ from typing import Optional # noqa: E402
import typer # noqa: E402
from basic_memory.cli.container import CliContainer, set_container # noqa: E402
from basic_memory.cli.promo import maybe_show_cloud_promo, maybe_show_init_line # noqa: E402
from basic_memory.config import init_cli_logging # noqa: E402
@@ -46,11 +47,31 @@ def app_callback(
container = CliContainer.create()
set_container(container)
# Trigger: first-run init confirmation before command output.
# Why: informational "initialized" message belongs above command results, not in the upsell panel.
# Outcome: one-time plain line printed before the subcommand runs.
maybe_show_init_line(ctx.invoked_subcommand)
# Trigger: register promo as a post-command callback.
# Why: promo output should appear after the command's own output, not before.
# Outcome: promo panel renders below the command results (status tree, table, etc.).
ctx.call_on_close(lambda: maybe_show_cloud_promo(ctx.invoked_subcommand))
# Run initialization for commands that don't use the API
# Skip for 'mcp' command - it has its own lifespan that handles initialization
# Skip for API-using commands (status, sync, etc.) - they handle initialization via deps.py
# Skip for 'reset' command - it manages its own database lifecycle
skip_init_commands = {"mcp", "status", "sync", "project", "tool", "reset"}
skip_init_commands = {
"doctor",
"mcp",
"status",
"sync",
"project",
"tool",
"reset",
"reindex",
"watch",
}
if (
not version
and ctx.invoked_subcommand is not None
+15 -2
View File
@@ -1,11 +1,21 @@
"""CLI commands for basic-memory."""
from . import status, db, import_memory_json, mcp, import_claude_conversations
from . import import_claude_projects, import_chatgpt, tool, project, format
from . import status, db, doctor, import_memory_json, mcp, import_claude_conversations
from . import (
import_claude_projects,
import_chatgpt,
tool,
project,
format,
schema,
watch,
workspace,
)
__all__ = [
"status",
"db",
"doctor",
"import_memory_json",
"mcp",
"import_claude_conversations",
@@ -14,4 +24,7 @@ __all__ = [
"tool",
"project",
"format",
"schema",
"watch",
"workspace",
]
@@ -1,6 +1,16 @@
"""Cloud commands package."""
from basic_memory.cli.app import cloud_app
# Import all commands to register them with typer
from basic_memory.cli.commands.cloud.core_commands import * # noqa: F401,F403
from basic_memory.cli.commands.cloud.api_client import get_authenticated_headers, get_cloud_config # noqa: F401
from basic_memory.cli.commands.cloud.upload_command import * # noqa: F401,F403
# Register snapshot sub-command group
from basic_memory.cli.commands.cloud.snapshot import snapshot_app
cloud_app.add_typer(snapshot_app, name="snapshot")
# Register restore command (directly on cloud_app via decorator)
from basic_memory.cli.commands.cloud.restore import restore # noqa: F401, E402
@@ -30,7 +30,7 @@ async def fetch_cloud_projects(
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
response = await api_request(method="GET", url=f"{host_url}/proxy/projects/projects")
response = await api_request(method="GET", url=f"{host_url}/proxy/v2/projects/")
return CloudProjectList.model_validate(response.json())
except Exception as e:
@@ -66,7 +66,7 @@ async def create_cloud_project(
response = await api_request(
method="POST",
url=f"{host_url}/proxy/projects/projects",
url=f"{host_url}/proxy/v2/projects/",
headers={"Content-Type": "application/json"},
json_data=project_data.model_dump(),
)
@@ -6,6 +6,7 @@ from rich.console import Console
from basic_memory.cli.app import cloud_app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.cli.auth import CLIAuth
from basic_memory.cli.promo import OSS_DISCOUNT_CODE
from basic_memory.config import ConfigManager
from basic_memory.cli.commands.cloud.api_client import (
CloudAPIError,
@@ -29,7 +30,7 @@ console = Console()
@cloud_app.command()
def login():
"""Authenticate with WorkOS using OAuth Device Authorization flow and enable cloud mode."""
"""Authenticate with WorkOS using OAuth Device Authorization flow."""
async def _login():
client_id, domain, host_url = get_cloud_config()
@@ -45,18 +46,15 @@ def login():
console.print("[dim]Verifying subscription access...[/dim]")
await make_api_request("GET", f"{host_url.rstrip('/')}/proxy/health")
# Enable cloud mode after successful login and subscription validation
config_manager = ConfigManager()
config = config_manager.load_config()
config.cloud_mode = True
config_manager.save_config(config)
console.print("[green]Cloud mode enabled[/green]")
console.print(f"[dim]All CLI commands now work against {host_url}[/dim]")
console.print("[green]Cloud authentication successful[/green]")
console.print(f"[dim]Cloud host ready: {host_url}[/dim]")
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(
f"OSS discount code: [bold]{OSS_DISCOUNT_CODE}[/bold] (20% off for 3 months)\n"
)
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
console.print(
"[dim]Once you have an active subscription, run [bold]bm cloud login[/bold] again.[/dim]"
@@ -68,50 +66,52 @@ def login():
@cloud_app.command()
def logout():
"""Disable cloud mode and return to local mode."""
# Disable cloud mode
config_manager = ConfigManager()
config = config_manager.load_config()
config.cloud_mode = False
config_manager.save_config(config)
console.print("[green]Cloud mode disabled[/green]")
console.print("[dim]All CLI commands now work locally[/dim]")
"""Remove stored OAuth tokens."""
config = ConfigManager().config
auth = CLIAuth(client_id=config.cloud_client_id, authkit_domain=config.cloud_domain)
auth.logout()
console.print("[dim]API key (if configured) remains available for cloud project routing.[/dim]")
@cloud_app.command("status")
def status() -> None:
"""Check cloud mode status and cloud instance health."""
# Check cloud mode
"""Check cloud authentication state and cloud instance health."""
config_manager = ConfigManager()
config = config_manager.load_config()
auth = CLIAuth(client_id=config.cloud_client_id, authkit_domain=config.cloud_domain)
tokens = auth.load_tokens()
console.print("[bold blue]Cloud Mode Status[/bold blue]")
if config.cloud_mode:
console.print(" Mode: [green]Cloud (enabled)[/green]")
console.print(f" Host: {config.cloud_host}")
console.print(" [dim]All CLI commands work against cloud[/dim]")
else:
console.print(" Mode: [yellow]Local (disabled)[/yellow]")
console.print(" [dim]All CLI commands work locally[/dim]")
console.print("\n[dim]To enable cloud mode, run: bm cloud login[/dim]")
return
console.print("[bold blue]Cloud Authentication Status[/bold blue]")
console.print(f" Host: {config.cloud_host}")
console.print(
f" API Key: {'[green]configured[/green]' if config.cloud_api_key else '[yellow]not set[/yellow]'}"
)
oauth_status = "[yellow]not logged in[/yellow]"
if tokens:
oauth_status = (
"[green]token valid[/green]"
if auth.is_token_valid(tokens)
else "[yellow]token expired[/yellow]"
)
console.print(f" OAuth: {oauth_status}")
# Get cloud configuration
_, _, host_url = get_cloud_config()
host_url = host_url.rstrip("/")
# Prepare headers
headers = {}
has_credentials = bool(config.cloud_api_key) or tokens is not None
if not has_credentials:
console.print(
"\n[dim]No cloud credentials found. Run: bm cloud login or bm cloud set-key <key>[/dim]"
)
return
try:
console.print("\n[blue]Checking cloud instance health...[/blue]")
# Make API request to check health
response = run_with_cleanup(
make_api_request(method="GET", url=f"{host_url}/proxy/health", headers=headers)
)
response = run_with_cleanup(make_api_request(method="GET", url=f"{host_url}/proxy/health"))
health_data = response.json()
@@ -128,11 +128,12 @@ def status() -> None:
console.print("\n[dim]To sync projects, use: bm project bisync --name <project>[/dim]")
except CloudAPIError as e:
console.print(f"[red]Error checking cloud health: {e}[/red]")
raise typer.Exit(1)
console.print(f"[yellow]Cloud health check failed: {e}[/yellow]")
console.print(
"[dim]Try re-authenticating with 'bm cloud login' or setting API key with 'bm cloud set-key'.[/dim]"
)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
console.print(f"[yellow]Unexpected health check error: {e}[/yellow]")
@cloud_app.command("setup")
@@ -191,3 +192,93 @@ def setup() -> None:
except Exception as e:
console.print(f"\n[red]Unexpected error during setup: {e}[/red]")
raise typer.Exit(1)
@cloud_app.command("promo")
def promo(enabled: bool = typer.Option(True, "--on/--off", help="Enable or disable CLI promos.")):
"""Enable or disable CLI cloud promo messages."""
config_manager = ConfigManager()
config = config_manager.load_config()
config.cloud_promo_opt_out = not enabled
config_manager.save_config(config)
if enabled:
console.print("[green]Cloud promo messages enabled[/green]")
else:
console.print("[yellow]Cloud promo messages disabled[/yellow]")
@cloud_app.command("set-key")
def set_key(
api_key: str = typer.Argument(..., help="API key (bmc_ prefixed) for cloud access"),
) -> None:
"""Save a cloud API key for per-project cloud routing.
The API key is account-level and used by projects set to cloud mode.
Create a key in the web app or use 'bm cloud create-key'.
Example:
bm cloud set-key bmc_abc123...
"""
if not api_key.startswith("bmc_"):
console.print("[red]Error: API key must start with 'bmc_'[/red]")
raise typer.Exit(1)
config_manager = ConfigManager()
config = config_manager.load_config()
config.cloud_api_key = api_key
config_manager.save_config(config)
console.print("[green]API key saved[/green]")
console.print("[dim]Projects set to cloud mode will use this key for authentication[/dim]")
console.print("[dim]Set a project to cloud mode: bm project set-cloud <name>[/dim]")
@cloud_app.command("create-key")
def create_key(
name: str = typer.Argument(..., help="Human-readable name for the API key"),
) -> None:
"""Create a new cloud API key and save it locally.
Requires active OAuth session (run 'bm cloud login' first).
The key is created via the cloud API and saved to local config.
Example:
bm cloud create-key "my-laptop"
"""
async def _create_key():
_, _, host_url = get_cloud_config()
host_url = host_url.rstrip("/")
console.print(f"[dim]Creating API key '{name}'...[/dim]")
response = await make_api_request(
method="POST",
url=f"{host_url}/api/keys",
json_data={"name": name},
)
key_data = response.json()
api_key = key_data.get("key")
if not api_key:
console.print("[red]Error: No key returned from API[/red]")
raise typer.Exit(1)
# Save to config
config_manager = ConfigManager()
config = config_manager.load_config()
config.cloud_api_key = api_key
config_manager.save_config(config)
console.print(f"[green]API key '{name}' created and saved[/green]")
console.print("[dim]Projects set to cloud mode will use this key for authentication[/dim]")
console.print("[dim]Set a project to cloud mode: bm project set-cloud <name>[/dim]")
try:
run_with_cleanup(_create_key())
except CloudAPIError as e:
console.print(f"[red]Error creating API key: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
@@ -27,6 +27,16 @@ console = Console()
# Minimum rclone version for --create-empty-src-dirs support
MIN_RCLONE_VERSION_EMPTY_DIRS = (1, 64, 0)
# Tigris edge caching returns stale data for users outside the origin region (iad).
# --header is rclone's global flag that applies to ALL HTTP transactions (list, download,
# upload). This is critical because bisync starts with S3 ListObjectsV2, which is neither
# a download nor upload — so --header-download/--header-upload would miss list requests.
# See: https://www.tigrisdata.com/docs/objects/consistency/
TIGRIS_CONSISTENCY_HEADERS = [
"--header",
"X-Tigris-Consistent: true",
]
class RunResult(Protocol):
returncode: int
@@ -210,6 +220,7 @@ def project_sync(
"sync",
str(local_path),
remote_path,
*TIGRIS_CONSISTENCY_HEADERS,
"--filter-from",
str(filter_path),
]
@@ -279,6 +290,7 @@ def project_bisync(
"bisync",
str(local_path),
remote_path,
*TIGRIS_CONSISTENCY_HEADERS,
"--resilient",
"--conflict-resolve=newer",
"--max-delete=25",
@@ -354,6 +366,7 @@ def project_check(
"check",
str(local_path),
remote_path,
*TIGRIS_CONSISTENCY_HEADERS,
"--filter-from",
str(filter_path),
]
@@ -393,6 +406,6 @@ def project_ls(
if path:
remote_path = f"{remote_path}/{path}"
cmd = ["rclone", "ls", remote_path]
cmd = ["rclone", "ls", *TIGRIS_CONSISTENCY_HEADERS, remote_path]
result = run(cmd, capture_output=True, text=True, check=True)
return result.stdout.splitlines()
@@ -0,0 +1,159 @@
"""Restore CLI commands for Basic Memory Cloud.
SPEC-29 Phase 3: CLI commands for restoring files from Tigris bucket snapshots.
"""
import asyncio
import typer
from rich.console import Console
from basic_memory.cli.app import cloud_app
from basic_memory.cli.commands.cloud.api_client import (
CloudAPIError,
SubscriptionRequiredError,
make_api_request,
)
from basic_memory.cli.commands.cloud.schemas import BucketSnapshotBrowseResponse
from basic_memory.config import ConfigManager
console = Console()
@cloud_app.command("restore")
def restore(
path: str = typer.Argument(
...,
help="Path to restore (file or folder, e.g., 'notes/project.md' or 'research/')",
),
snapshot_id: str = typer.Option(
...,
"--snapshot",
"-s",
help="ID of the snapshot to restore from",
),
force: bool = typer.Option(
False,
"--force",
"-f",
help="Skip confirmation prompt",
),
) -> None:
"""Restore a file or folder from a snapshot.
This command restores files from a previous snapshot to the current bucket.
The restored files will overwrite any existing files at the same path.
Examples:
bm cloud restore notes/project.md --snapshot abc123
bm cloud restore research/ --snapshot abc123
bm cloud restore notes/project.md --snapshot abc123 --force
"""
async def _restore():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
# Normalize path - remove leading slash if present
normalized_path = path.lstrip("/")
if not force:
# Show what will be restored
console.print(f"[blue]Preparing to restore from snapshot {snapshot_id}[/blue]")
console.print(f" Path: {normalized_path}")
# Try to browse the snapshot to show what files will be affected
try:
browse_url = f"{host_url}/api/bucket-snapshots/{snapshot_id}/browse"
if normalized_path:
browse_url += f"?prefix={normalized_path}"
response = await make_api_request(
method="GET",
url=browse_url,
)
browse_response = BucketSnapshotBrowseResponse.model_validate(response.json())
if browse_response.files:
if len(browse_response.files) <= 10:
console.print("\n Files to restore:")
for file_info in browse_response.files:
console.print(f" - {file_info.key}")
else:
console.print(
f"\n {len(browse_response.files)} files will be restored"
)
console.print(" First 5 files:")
for file_info in browse_response.files[:5]:
console.print(f" - {file_info.key}")
console.print(f" ... and {len(browse_response.files) - 5} more")
else:
console.print(
f"\n[yellow]No files found matching '{normalized_path}' "
f"in snapshot[/yellow]"
)
raise typer.Exit(0)
except CloudAPIError as browse_error:
if browse_error.status_code == 404:
console.print(f"[red]Snapshot not found: {snapshot_id}[/red]")
raise typer.Exit(1)
# If browse fails for other reasons, proceed with confirmation anyway
pass
console.print(
"\n[yellow]Warning: Restored files will overwrite existing files![/yellow]"
)
confirmed = typer.confirm("\nProceed with restore?")
if not confirmed:
console.print("[yellow]Restore cancelled[/yellow]")
raise typer.Exit(0)
console.print(f"[blue]Restoring from snapshot {snapshot_id}...[/blue]")
response = await make_api_request(
method="POST",
url=f"{host_url}/api/bucket-snapshots/{snapshot_id}/restore",
json_data={"path": normalized_path},
)
data = response.json()
restored_files = data.get("restored", [])
returned_snapshot_id = data.get("snapshot_id", snapshot_id)
if restored_files:
console.print(f"[green]Successfully restored {len(restored_files)} file(s)[/green]")
if len(restored_files) <= 10:
for file_path in restored_files:
console.print(f" - {file_path}")
else:
console.print(" First 5 restored files:")
for file_path in restored_files[:5]:
console.print(f" - {file_path}")
console.print(f" ... and {len(restored_files) - 5} more")
console.print(f"\n[dim]Snapshot ID: {returned_snapshot_id}[/dim]")
else:
console.print("[yellow]No files were restored[/yellow]")
console.print(f"[dim]No files matching '{normalized_path}' found in snapshot[/dim]")
except typer.Exit:
# Re-raise typer.Exit without modification - it's used for clean exits
raise
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
if e.status_code == 404:
console.print(f"[red]Snapshot not found: {snapshot_id}[/red]")
else:
console.print(f"[red]Failed to restore: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_restore())
@@ -0,0 +1,55 @@
"""Pydantic schemas for Basic Memory Cloud API responses.
These schemas mirror the API response models from basic-memory-cloud
for type-safe parsing of API responses in CLI commands.
"""
from datetime import datetime
from uuid import UUID
from pydantic import BaseModel
class BucketSnapshotFileInfo(BaseModel):
"""File info from snapshot browse response."""
key: str
size: int
last_modified: datetime
etag: str | None = None
class BucketSnapshotBrowseResponse(BaseModel):
"""Response from browsing snapshot contents."""
files: list[BucketSnapshotFileInfo]
prefix: str
snapshot_version: str
class BucketSnapshotResponse(BaseModel):
"""Response model for bucket snapshot data."""
id: UUID
bucket_name: str
snapshot_version: str
name: str
description: str | None
auto: bool
created_at: datetime
created_by: UUID | None = None
class BucketSnapshotListResponse(BaseModel):
"""Response from listing bucket snapshots."""
snapshots: list[BucketSnapshotResponse]
total: int
class BucketSnapshotRestoreResponse(BaseModel):
"""Response from restore operation."""
restored: list[str]
snapshot_version: str
snapshot_id: UUID
@@ -0,0 +1,370 @@
"""Snapshot CLI commands for Basic Memory Cloud.
SPEC-29 Phase 3: CLI commands for managing Tigris bucket snapshots.
"""
import asyncio
from datetime import datetime
from typing import Optional
import typer
from rich.console import Console
from rich.table import Table
from basic_memory.cli.commands.cloud.api_client import (
CloudAPIError,
SubscriptionRequiredError,
make_api_request,
)
from basic_memory.cli.commands.cloud.schemas import BucketSnapshotBrowseResponse
from basic_memory.config import ConfigManager
console = Console()
snapshot_app = typer.Typer(help="Manage bucket snapshots")
def _format_timestamp(iso_timestamp: str) -> str:
"""Format ISO timestamp to a human-readable format."""
try:
dt = datetime.fromisoformat(iso_timestamp.replace("Z", "+00:00"))
return dt.strftime("%Y-%m-%d %H:%M:%S")
except (ValueError, AttributeError):
return iso_timestamp
@snapshot_app.command("create")
def create(
description: str = typer.Argument(
...,
help="Description for the snapshot",
),
) -> None:
"""Create a new bucket snapshot.
Examples:
bm cloud snapshot create "before major refactor"
bm cloud snapshot create "daily backup"
"""
async def _create():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
console.print("[blue]Creating snapshot...[/blue]")
response = await make_api_request(
method="POST",
url=f"{host_url}/api/bucket-snapshots",
json_data={"description": description},
)
data = response.json()
snapshot_id = data.get("id", "unknown")
snapshot_version = data.get("snapshot_version", "unknown")
created_at = _format_timestamp(data.get("created_at", ""))
console.print("[green]Snapshot created successfully[/green]")
console.print(f" ID: {snapshot_id}")
console.print(f" Version: {snapshot_version}")
console.print(f" Created: {created_at}")
console.print(f" Description: {description}")
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
console.print(f"[red]Failed to create snapshot: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_create())
@snapshot_app.command("list")
def list_snapshots(
limit: int = typer.Option(
10,
"--limit",
"-l",
help="Maximum number of snapshots to display",
),
) -> None:
"""List all bucket snapshots.
Examples:
bm cloud snapshot list
bm cloud snapshot list --limit 20
"""
async def _list():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
console.print("[blue]Fetching snapshots...[/blue]")
response = await make_api_request(
method="GET",
url=f"{host_url}/api/bucket-snapshots",
)
data = response.json()
snapshots = data.get("snapshots", [])
total = data.get("total", len(snapshots))
if not snapshots:
console.print("[yellow]No snapshots found[/yellow]")
console.print(
'\n[dim]Create a snapshot with: bm cloud snapshot create "description"[/dim]'
)
return
# Create a table for displaying snapshots
table = Table(title=f"Bucket Snapshots ({total} total)")
table.add_column("ID", style="cyan", no_wrap=True)
table.add_column("Description", style="white")
table.add_column("Auto", style="dim")
table.add_column("Created", style="green")
for snapshot in snapshots[:limit]:
snapshot_id = snapshot.get("id", "unknown")
desc = snapshot.get("description") or snapshot.get("name", "-")
auto = "yes" if snapshot.get("auto", False) else "no"
created_at = _format_timestamp(snapshot.get("created_at", ""))
table.add_row(snapshot_id, desc, auto, created_at)
console.print(table)
if total > limit:
console.print(
f"\n[dim]Showing {limit} of {total} snapshots. Use --limit to see more.[/dim]"
)
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
console.print(f"[red]Failed to list snapshots: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_list())
@snapshot_app.command("delete")
def delete(
snapshot_id: str = typer.Argument(
...,
help="The ID of the snapshot to delete",
),
force: bool = typer.Option(
False,
"--force",
"-f",
help="Skip confirmation prompt",
),
) -> None:
"""Delete a bucket snapshot.
Examples:
bm cloud snapshot delete abc123
bm cloud snapshot delete abc123 --force
"""
async def _delete():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
if not force:
# Fetch snapshot details first to show what will be deleted
console.print("[blue]Fetching snapshot details...[/blue]")
try:
response = await make_api_request(
method="GET",
url=f"{host_url}/api/bucket-snapshots/{snapshot_id}",
)
data = response.json()
desc = data.get("description") or data.get("name", "unnamed")
created_at = _format_timestamp(data.get("created_at", ""))
console.print("\nSnapshot to delete:")
console.print(f" ID: {snapshot_id}")
console.print(f" Description: {desc}")
console.print(f" Created: {created_at}")
except CloudAPIError:
# If we can't fetch details, proceed with confirmation anyway
pass
confirmed = typer.confirm("\nAre you sure you want to delete this snapshot?")
if not confirmed:
console.print("[yellow]Deletion cancelled[/yellow]")
raise typer.Exit(0)
console.print("[blue]Deleting snapshot...[/blue]")
await make_api_request(
method="DELETE",
url=f"{host_url}/api/bucket-snapshots/{snapshot_id}",
)
console.print(f"[green]Snapshot {snapshot_id} deleted successfully[/green]")
except typer.Exit:
# Re-raise typer.Exit without modification - it's used for clean exits
raise
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
if e.status_code == 404:
console.print(f"[red]Snapshot not found: {snapshot_id}[/red]")
else:
console.print(f"[red]Failed to delete snapshot: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_delete())
@snapshot_app.command("show")
def show(
snapshot_id: str = typer.Argument(
...,
help="The ID of the snapshot to show",
),
) -> None:
"""Show details of a specific snapshot.
Examples:
bm cloud snapshot show abc123
"""
async def _show():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
response = await make_api_request(
method="GET",
url=f"{host_url}/api/bucket-snapshots/{snapshot_id}",
)
data = response.json()
console.print("[bold blue]Snapshot Details[/bold blue]")
console.print(f" ID: {data.get('id', 'unknown')}")
console.print(f" Bucket: {data.get('bucket_name', 'unknown')}")
console.print(f" Version: {data.get('snapshot_version', 'unknown')}")
console.print(f" Name: {data.get('name', '-')}")
console.print(f" Description: {data.get('description') or '-'}")
console.print(f" Auto: {'yes' if data.get('auto', False) else 'no'}")
console.print(f" Created: {_format_timestamp(data.get('created_at', ''))}")
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
if e.status_code == 404:
console.print(f"[red]Snapshot not found: {snapshot_id}[/red]")
else:
console.print(f"[red]Failed to get snapshot details: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_show())
@snapshot_app.command("browse")
def browse(
snapshot_id: str = typer.Argument(
...,
help="The ID of the snapshot to browse",
),
prefix: Optional[str] = typer.Option(
None,
"--prefix",
"-p",
help="Filter files by path prefix (e.g., 'notes/')",
),
) -> None:
"""Browse contents of a snapshot.
Examples:
bm cloud snapshot browse abc123
bm cloud snapshot browse abc123 --prefix notes/
"""
async def _browse():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
url = f"{host_url}/api/bucket-snapshots/{snapshot_id}/browse"
if prefix:
url += f"?prefix={prefix}"
response = await make_api_request(
method="GET",
url=url,
)
browse_response = BucketSnapshotBrowseResponse.model_validate(response.json())
if not browse_response.files:
if prefix:
console.print(f"[yellow]No files found with prefix '{prefix}'[/yellow]")
else:
console.print("[yellow]No files found in snapshot[/yellow]")
return
console.print(
f"[bold blue]Snapshot Contents ({len(browse_response.files)} files)[/bold blue]"
)
for file_info in browse_response.files:
size_kb = file_info.size // 1024
console.print(f" {file_info.key} ({size_kb} KB)")
console.print(
f"\n[dim]Use 'bm cloud restore <path> --snapshot {snapshot_id}' "
f"to restore files[/dim]"
)
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
if e.status_code == 404:
console.print(f"[red]Snapshot not found: {snapshot_id}[/red]")
else:
console.print(f"[red]Failed to browse snapshot: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_browse())
@@ -9,8 +9,8 @@ import typer
from rich.console import Console
from basic_memory import db
from basic_memory.config import ConfigManager
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.tools.utils import call_post, call_get
from basic_memory.mcp.project_context import get_active_project
from basic_memory.schemas import ProjectInfoResponse
@@ -55,10 +55,13 @@ async def run_sync(
run_in_background: If True, return immediately; if False, wait for completion
"""
# Resolve default project so get_client() can route per-project
project = project or ConfigManager().default_project
try:
async with get_client() as client:
async with get_client(project_name=project) as client:
project_item = await get_active_project(client, project, None)
url = f"{project_item.project_url}/project/sync"
url = f"/v2/projects/{project_item.external_id}/sync"
params = []
if force_full:
params.append("force_full=true")
@@ -88,11 +91,10 @@ async def run_sync(
async def get_project_info(project: str):
"""Get project information via API endpoint."""
try:
async with get_client() as client:
async with get_client(project_name=project) as client:
project_item = await get_active_project(client, project, None)
response = await call_get(client, f"{project_item.project_url}/project/info")
response = await call_get(client, f"/v2/projects/{project_item.external_id}/info")
return ProjectInfoResponse.model_validate(response.json())
except (ToolError, ValueError) as e:
console.print(f"[red]Sync failed: {e}[/red]")
+116
View File
@@ -5,6 +5,7 @@ from pathlib import Path
import typer
from loguru import logger
from rich.console import Console
from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn, TaskProgressColumn
from sqlalchemy.exc import OperationalError
from basic_memory import db
@@ -103,3 +104,118 @@ def reset(
# ensures db.shutdown_db() is called even if _reindex_projects changes
run_with_cleanup(_reindex_projects(app_config))
console.print("[green]Reindex complete[/green]")
@app.command()
def reindex(
embeddings: bool = typer.Option(
False, "--embeddings", "-e", help="Rebuild vector embeddings (requires semantic search)"
),
search: bool = typer.Option(False, "--search", "-s", help="Rebuild full-text search index"),
project: str = typer.Option(
None, "--project", "-p", help="Reindex a specific project (default: all)"
),
): # pragma: no cover
"""Rebuild search indexes and/or vector embeddings without dropping the database.
By default rebuilds everything (search + embeddings if semantic is enabled).
Use --search or --embeddings to rebuild only one.
Examples:
bm reindex # Rebuild everything
bm reindex --embeddings # Only rebuild vector embeddings
bm reindex --search # Only rebuild FTS index
bm reindex -p claw # Reindex only the 'claw' project
"""
# If neither flag is set, do both
if not embeddings and not search:
embeddings = True
search = True
config_manager = ConfigManager()
app_config = config_manager.config
if embeddings and not app_config.semantic_search_enabled:
console.print(
"[yellow]Semantic search is not enabled.[/yellow] "
"Set [cyan]semantic_search_enabled: true[/cyan] in config to use embeddings."
)
embeddings = False
if not search:
raise typer.Exit(0)
run_with_cleanup(_reindex(app_config, search=search, embeddings=embeddings, project=project))
async def _reindex(app_config, search: bool, embeddings: bool, project: str | None):
"""Run reindex operations."""
from basic_memory.repository import EntityRepository
from basic_memory.repository.search_repository import create_search_repository
from basic_memory.services.search_service import SearchService
from basic_memory.services.file_service import FileService
from basic_memory.markdown.markdown_processor import MarkdownProcessor
from basic_memory.markdown.entity_parser import EntityParser
try:
await reconcile_projects_with_config(app_config)
_, session_maker = await db.get_or_create_db(
db_path=app_config.database_path,
db_type=db.DatabaseType.FILESYSTEM,
)
project_repository = ProjectRepository(session_maker)
projects = await project_repository.get_active_projects()
if project:
projects = [p for p in projects if p.name == project]
if not projects:
console.print(f"[red]Project '{project}' not found.[/red]")
raise typer.Exit(1)
for proj in projects:
console.print(f"\n[bold]Project: [cyan]{proj.name}[/cyan][/bold]")
if search:
console.print(" Rebuilding full-text search index...")
sync_service = await get_sync_service(proj)
sync_dir = Path(proj.path)
await sync_service.sync(sync_dir, project_name=proj.name)
console.print(" [green]✓[/green] Full-text search index rebuilt")
if embeddings:
console.print(" Building vector embeddings...")
entity_repository = EntityRepository(session_maker, project_id=proj.id)
search_repository = create_search_repository(
session_maker, project_id=proj.id, app_config=app_config
)
project_path = Path(proj.path)
entity_parser = EntityParser(project_path)
markdown_processor = MarkdownProcessor(entity_parser, app_config=app_config)
file_service = FileService(project_path, markdown_processor, app_config=app_config)
search_service = SearchService(search_repository, entity_repository, file_service)
with Progress(
SpinnerColumn(),
TextColumn("[progress.description]{task.description}"),
BarColumn(),
TaskProgressColumn(),
console=console,
) as progress:
task = progress.add_task(" Embedding entities...", total=None)
def on_progress(entity_id, index, total):
progress.update(task, total=total, completed=index)
stats = await search_service.reindex_vectors(progress_callback=on_progress)
progress.update(task, completed=stats["total_entities"])
console.print(
f" [green]✓[/green] Embeddings complete: "
f"{stats['embedded']} entities embedded, "
f"{stats['skipped']} skipped, "
f"{stats['errors']} errors"
)
console.print("\n[green]Reindex complete![/green]")
finally:
await db.shutdown_db()
+153
View File
@@ -0,0 +1,153 @@
"""Doctor command for local consistency checks."""
from __future__ import annotations
import tempfile
import uuid
from pathlib import Path
from loguru import logger
from mcp.server.fastmcp.exceptions import ToolError
from rich.console import Console
import typer
from basic_memory.cli.app import app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.cli.commands.routing import force_routing, validate_routing_flags
from basic_memory.markdown.entity_parser import EntityParser
from basic_memory.markdown.markdown_processor import MarkdownProcessor
from basic_memory.markdown.schemas import EntityFrontmatter, EntityMarkdown
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.clients import KnowledgeClient, ProjectClient, SearchClient
from basic_memory.mcp.tools.utils import call_post
from basic_memory.schemas.base import Entity
from basic_memory.schemas.project_info import ProjectInfoRequest
from basic_memory.schemas.search import SearchQuery
from basic_memory.schemas import SyncReportResponse
console = Console()
async def run_doctor() -> None:
"""Run local consistency checks for file <-> database flows."""
console.print("[blue]Running Basic Memory doctor checks...[/blue]")
project_name = f"doctor-{uuid.uuid4().hex[:8]}"
api_note_title = "Doctor API Note"
manual_note_title = "Doctor Manual Note"
manual_permalink = "doctor/manual-note"
with tempfile.TemporaryDirectory() as temp_dir:
temp_path = Path(temp_dir)
async with get_client() as client:
project_client = ProjectClient(client)
project_request = ProjectInfoRequest(
name=project_name,
path=str(temp_path),
set_default=False,
)
project_id: str | None = None
try:
status = await project_client.create_project(project_request.model_dump())
if not status.new_project:
raise ValueError("Failed to create doctor project")
project_id = status.new_project.external_id
console.print(f"[green]OK[/green] Created doctor project: {project_name}")
# --- DB -> File: create an entity via API ---
knowledge_client = KnowledgeClient(client, project_id)
api_note = Entity(
title=api_note_title,
directory="doctor",
entity_type="note",
content_type="text/markdown",
content=f"# {api_note_title}\n\n- [note] API to file check",
entity_metadata={"tags": ["doctor"]},
)
api_result = await knowledge_client.create_entity(api_note.model_dump(), fast=False)
api_file = temp_path / api_result.file_path
if not api_file.exists():
raise ValueError(f"API note file missing: {api_result.file_path}")
api_text = api_file.read_text(encoding="utf-8")
if api_note_title not in api_text:
raise ValueError("API note content missing from file")
console.print("[green]OK[/green] API write created file")
# --- File -> DB: write markdown file directly, then sync ---
parser = EntityParser(temp_path)
processor = MarkdownProcessor(parser)
manual_markdown = EntityMarkdown(
frontmatter=EntityFrontmatter(
metadata={
"title": manual_note_title,
"type": "note",
"permalink": manual_permalink,
"tags": ["doctor"],
}
),
content=f"# {manual_note_title}\n\n- [note] File to DB check",
)
manual_path = temp_path / "doctor" / "manual-note.md"
await processor.write_file(manual_path, manual_markdown)
console.print("[green]OK[/green] Manual file written")
sync_response = await call_post(
client,
f"/v2/projects/{project_id}/sync?force_full=true&run_in_background=false",
)
sync_report = SyncReportResponse.model_validate(sync_response.json())
if sync_report.total == 0:
raise ValueError("Sync did not detect any changes")
console.print("[green]OK[/green] Sync indexed manual file")
search_client = SearchClient(client, project_id)
search_query = SearchQuery(title=manual_note_title)
search_results = await search_client.search(
search_query.model_dump(), page=1, page_size=5
)
if not any(result.title == manual_note_title for result in search_results.results):
raise ValueError("Manual note not found in search index")
console.print("[green]OK[/green] Search confirmed manual file")
status_response = await call_post(client, f"/v2/projects/{project_id}/status")
status_report = SyncReportResponse.model_validate(status_response.json())
if status_report.total != 0:
raise ValueError("Project status not clean after sync")
console.print("[green]OK[/green] Status clean after sync")
finally:
if project_id:
await project_client.delete_project(project_id)
console.print("[green]Doctor checks passed.[/green]")
@app.command()
def doctor(
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
) -> None:
"""Run local consistency checks to verify file/database sync."""
try:
validate_routing_flags(local, cloud)
with force_routing(local=local, cloud=cloud):
run_with_cleanup(run_doctor())
except (ToolError, ValueError) as e:
console.print(f"[red]Doctor failed: {e}[/red]")
raise typer.Exit(code=1)
except Exception as e:
logger.error(f"Doctor failed: {e}")
typer.echo(f"Doctor failed: {e}", err=True)
raise typer.Exit(code=1) # pragma: no cover
@@ -60,7 +60,9 @@ def import_chatgpt(
console.print(f"\nImporting chats from {conversations_json}...writing to {base_path}")
# Create importer and run import
importer = ChatGPTImporter(config.home, markdown_processor, file_service)
importer = ChatGPTImporter(
config.home, markdown_processor, file_service, project_name=config.name
)
with conversations_json.open("r", encoding="utf-8") as file:
json_data = json.load(file)
result = run_with_cleanup(importer.import_data(json_data, folder))
@@ -57,7 +57,9 @@ def import_claude(
markdown_processor, file_service = run_with_cleanup(get_importer_dependencies())
# Create the importer
importer = ClaudeConversationsImporter(config.home, markdown_processor, file_service)
importer = ClaudeConversationsImporter(
config.home, markdown_processor, file_service, project_name=config.name
)
# Process the file
base_path = config.home / folder
@@ -56,7 +56,9 @@ def import_projects(
markdown_processor, file_service = run_with_cleanup(get_importer_dependencies())
# Create the importer
importer = ClaudeProjectsImporter(config.home, markdown_processor, file_service)
importer = ClaudeProjectsImporter(
config.home, markdown_processor, file_service, project_name=config.name
)
# Process the file
base_path = config.home / base_folder if base_folder else config.home
@@ -55,7 +55,9 @@ def memory_json(
markdown_processor, file_service = run_with_cleanup(get_importer_dependencies())
# Create the importer
importer = MemoryJsonImporter(config.home, markdown_processor, file_service)
importer = MemoryJsonImporter(
config.home, markdown_processor, file_service, project_name=config.name
)
# Process the file
base_path = config.home if not destination_folder else config.home / destination_folder
+72 -56
View File
@@ -1,76 +1,92 @@
"""MCP server command with streamable HTTP transport."""
import os
from typing import Any, Optional
import typer
from typing import Optional
from loguru import logger
from basic_memory.cli.app import app
from basic_memory.config import ConfigManager, init_mcp_logging
# Import mcp instance (has lifespan that handles initialization and file sync)
from basic_memory.mcp.server import mcp as mcp_server # pragma: no cover
# Import mcp tools to register them
import basic_memory.mcp.tools # noqa: F401 # pragma: no cover
class _DeferredMcpServer:
def run(self, *args: Any, **kwargs: Any) -> None: # pragma: no cover
from basic_memory.mcp.server import mcp as live_mcp_server
# Import prompts to register them
import basic_memory.mcp.prompts # noqa: F401 # pragma: no cover
from loguru import logger
live_mcp_server.run(*args, **kwargs)
config = ConfigManager().config
if not config.cloud_mode_enabled:
# Keep module-level attribute for tests/monkeypatching while deferring heavy import.
mcp_server = _DeferredMcpServer()
@app.command()
def mcp(
transport: str = typer.Option(
"stdio", help="Transport type: stdio, streamable-http, or sse"
),
host: str = typer.Option(
"0.0.0.0", help="Host for HTTP transports (use 0.0.0.0 to allow external connections)"
),
port: int = typer.Option(8000, help="Port for HTTP transports"),
path: str = typer.Option("/mcp", help="Path prefix for streamable-http transport"),
project: Optional[str] = typer.Option(None, help="Restrict MCP server to single project"),
): # pragma: no cover
"""Run the MCP server with configurable transport options.
This command starts an MCP server using one of three transport options:
@app.command()
def mcp(
transport: str = typer.Option("stdio", help="Transport type: stdio, streamable-http, or sse"),
host: str = typer.Option(
"0.0.0.0", help="Host for HTTP transports (use 0.0.0.0 to allow external connections)"
),
port: int = typer.Option(8000, help="Port for HTTP transports"),
path: str = typer.Option("/mcp", help="Path prefix for streamable-http transport"),
project: Optional[str] = typer.Option(None, help="Restrict MCP server to single project"),
): # pragma: no cover
"""Run the MCP server with configurable transport options.
- stdio: Standard I/O (good for local usage)
- streamable-http: Recommended for web deployments (default)
- sse: Server-Sent Events (for compatibility with existing clients)
This command starts an MCP server using one of three transport options:
Initialization, file sync, and cleanup are handled by the MCP server's lifespan.
"""
# Initialize logging for MCP (file only, stdout breaks protocol)
init_mcp_logging()
- stdio: Standard I/O (good for local usage)
- streamable-http: Recommended for web deployments (default)
- sse: Server-Sent Events (for compatibility with existing clients)
# Validate and set project constraint if specified
if project:
config_manager = ConfigManager()
project_name, _ = config_manager.get_project(project)
if not project_name:
typer.echo(f"No project found named: {project}", err=True)
raise typer.Exit(1)
Initialization, file sync, and cleanup are handled by the MCP server's lifespan.
# Set env var with validated project name
os.environ["BASIC_MEMORY_MCP_PROJECT"] = project_name
logger.info(f"MCP server constrained to project: {project_name}")
Note: This command is available regardless of cloud mode setting.
Users who have cloud mode enabled can still use local MCP for Claude Code
and Claude Desktop while using cloud MCP for web and mobile access.
"""
# Force local routing for local MCP server.
# Trigger: MCP server command invocation (all transports).
# Why: local MCP must never route through cloud; stdio in particular must
# remain local-only to avoid cross-environment ambiguity.
# Outcome: explicit local override disables per-project cloud routing.
os.environ["BASIC_MEMORY_FORCE_LOCAL"] = "true"
os.environ.pop("BASIC_MEMORY_FORCE_CLOUD", None)
os.environ["BASIC_MEMORY_EXPLICIT_ROUTING"] = "true"
# Run the MCP server (blocks)
# Lifespan handles: initialization, migrations, file sync, cleanup
logger.info(f"Starting MCP server with {transport.upper()} transport")
# Import mcp tools/prompts to register them with the server
import basic_memory.mcp.tools # noqa: F401 # pragma: no cover
import basic_memory.mcp.prompts # noqa: F401 # pragma: no cover
import basic_memory.mcp.resources # noqa: F401 # pragma: no cover
if transport == "stdio":
mcp_server.run(
transport=transport,
)
elif transport == "streamable-http" or transport == "sse":
mcp_server.run(
transport=transport,
host=host,
port=port,
path=path,
log_level="INFO",
)
# Initialize logging for MCP (file only, stdout breaks protocol)
init_mcp_logging()
# Validate and set project constraint if specified
if project:
config_manager = ConfigManager()
project_name, _ = config_manager.get_project(project)
if not project_name:
typer.echo(f"No project found named: {project}", err=True)
raise typer.Exit(1)
# Set env var with validated project name
os.environ["BASIC_MEMORY_MCP_PROJECT"] = project_name
logger.info(f"MCP server constrained to project: {project_name}")
# Run the MCP server (blocks)
# Lifespan handles: initialization, migrations, file sync, cleanup
logger.info(f"Starting MCP server with {transport.upper()} transport")
if transport == "stdio":
mcp_server.run(
transport=transport,
)
elif transport == "streamable-http" or transport == "sse":
mcp_server.run(
transport=transport,
host=host,
port=port,
path=path,
log_level="INFO",
)
+433 -167
View File
@@ -1,32 +1,34 @@
"""Command module for basic-memory project management."""
import json
import os
from datetime import datetime
from pathlib import Path
import typer
from rich.console import Console
from rich.panel import Panel
from rich.table import Table
from basic_memory.cli.app import app
from basic_memory.cli.auth import CLIAuth
from basic_memory.cli.commands.command_utils import get_project_info, run_with_cleanup
from basic_memory.config import ConfigManager
import json
from datetime import datetime
from rich.panel import Panel
from basic_memory.cli.commands.routing import force_routing, validate_routing_flags
from basic_memory.config import ConfigManager, ProjectEntry, ProjectMode
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.tools.utils import call_get, call_post, call_delete, call_put, call_patch
from basic_memory.schemas.project_info import ProjectList, ProjectStatusResponse
from basic_memory.mcp.tools.utils import call_delete, call_get, call_patch, call_post, call_put
from basic_memory.schemas.project_info import ProjectItem, ProjectList, ProjectStatusResponse
from basic_memory.schemas.v2 import ProjectResolveResponse
from basic_memory.utils import generate_permalink, normalize_project_path
# Import rclone commands for project sync
from basic_memory.cli.commands.cloud.rclone_commands import (
SyncProject,
RcloneError,
project_sync,
SyncProject,
project_bisync,
project_check,
project_ls,
project_sync,
)
from basic_memory.cli.commands.cloud.bisync_commands import get_mount_info
@@ -45,54 +47,149 @@ def format_path(path: str) -> str:
return path
def _has_cloud_credentials(config) -> bool:
"""Return whether cloud credentials are available (API key or OAuth token)."""
if config.cloud_api_key:
return True
auth = CLIAuth(client_id=config.cloud_client_id, authkit_domain=config.cloud_domain)
return auth.load_tokens() is not None
def _require_cloud_credentials(config) -> None:
"""Exit with actionable guidance when cloud credentials are missing."""
if _has_cloud_credentials(config):
return
console.print("[red]Error: cloud credentials are required for this command[/red]")
console.print("[dim]Run 'bm cloud login' or 'bm cloud set-key <key>' first[/dim]")
raise typer.Exit(1)
@project_app.command("list")
def list_projects() -> None:
"""List all Basic Memory projects."""
def list_projects(
local: bool = typer.Option(False, "--local", help="Force local routing for this command"),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
) -> None:
"""List Basic Memory projects from local and (when available) cloud."""
try:
validate_routing_flags(local, cloud)
except ValueError as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
async def _list_projects():
async with get_client() as client:
response = await call_get(client, "/projects/projects")
response = await call_get(client, "/v2/projects/")
return ProjectList.model_validate(response.json())
try:
result = run_with_cleanup(_list_projects())
config = ConfigManager().config
local_result: ProjectList | None = None
cloud_result: ProjectList | None = None
cloud_error: Exception | None = None
if cloud:
with force_routing(cloud=True):
cloud_result = run_with_cleanup(_list_projects())
elif local:
with force_routing(local=True):
local_result = run_with_cleanup(_list_projects())
else:
# Default behavior: always show local projects first.
with force_routing(local=True):
local_result = run_with_cleanup(_list_projects())
if _has_cloud_credentials(config):
try:
with force_routing(cloud=True):
cloud_result = run_with_cleanup(_list_projects())
except Exception as exc: # pragma: no cover
cloud_error = exc
table = Table(title="Basic Memory Projects")
table.add_column("Name", style="cyan")
table.add_column("Path", style="green")
table.add_column("Local Path", style="yellow", no_wrap=True, overflow="fold")
table.add_column("Cloud Path", style="green")
table.add_column("CLI Route", style="blue")
table.add_column("MCP (stdio)", style="blue")
table.add_column("Sync", style="green")
table.add_column("Default", style="magenta")
# Add Local Path column if in cloud mode
if config.cloud_mode_enabled:
table.add_column("Local Path", style="yellow", no_wrap=True, overflow="fold")
project_names_by_permalink: dict[str, str] = {}
local_projects_by_permalink: dict[str, ProjectItem] = {}
cloud_projects_by_permalink: dict[str, ProjectItem] = {}
# Show Default column in local mode or if default_project_mode is enabled in cloud mode
show_default_column = not config.cloud_mode_enabled or config.default_project_mode
if show_default_column:
table.add_column("Default", style="magenta")
if local_result:
for project in local_result.projects:
permalink = generate_permalink(project.name)
project_names_by_permalink[permalink] = project.name
local_projects_by_permalink[permalink] = project
for project in result.projects:
is_default = "[X]" if project.is_default else ""
normalized_path = normalize_project_path(project.path)
if cloud_result:
for project in cloud_result.projects:
permalink = generate_permalink(project.name)
project_names_by_permalink[permalink] = project.name
cloud_projects_by_permalink[permalink] = project
# Build row based on mode
row = [project.name, format_path(normalized_path)]
for permalink in sorted(project_names_by_permalink):
project_name = project_names_by_permalink[permalink]
local_project = local_projects_by_permalink.get(permalink)
cloud_project = cloud_projects_by_permalink.get(permalink)
entry = config.projects.get(project_name)
# Add local path if in cloud mode
if config.cloud_mode_enabled:
local_path = ""
if project.name in config.cloud_projects:
local_path = config.cloud_projects[project.name].local_path or ""
local_path = format_path(local_path)
row.append(local_path)
local_path = ""
if local_project is not None:
local_path = format_path(normalize_project_path(local_project.path))
elif entry and entry.cloud_sync_path:
local_path = format_path(entry.cloud_sync_path)
elif entry and entry.mode == ProjectMode.LOCAL and entry.path:
local_path = format_path(normalize_project_path(entry.path))
# Add default indicator if showing default column
if show_default_column:
row.append(is_default)
cloud_path = ""
if cloud_project is not None:
cloud_path = normalize_project_path(cloud_project.path)
if local:
cli_route = "local (flag)"
elif cloud:
cli_route = "cloud (flag)"
elif entry:
cli_route = entry.mode.value
elif cloud_project is not None and local_project is None:
cli_route = ProjectMode.CLOUD.value
else:
cli_route = ProjectMode.LOCAL.value
is_default = ""
if config.default_project == project_name:
is_default = "[X]"
if local_project is not None and local_project.is_default:
is_default = "[X]"
if cloud_project is not None and cloud_project.is_default:
is_default = "[X]"
has_sync = "[X]" if entry and entry.cloud_sync_path else ""
mcp_stdio_target = "local" if local_project is not None else "n/a"
row = [
project_name,
local_path,
cloud_path,
cli_route,
mcp_stdio_target,
has_sync,
is_default,
]
table.add_row(*row)
console.print(table)
if cloud_error is not None:
console.print(
"[yellow]Cloud project discovery failed. "
"Showing local projects only. Run 'bm cloud login' or 'bm cloud set-key <key>'.[/yellow]"
)
except Exception as e:
console.print(f"[red]Error listing projects: {str(e)}[/red]")
raise typer.Exit(1)
@@ -108,9 +205,16 @@ def add_project(
None, "--local-path", help="Local sync path for cloud mode (optional)"
),
set_default: bool = typer.Option(False, "--default", help="Set as default project"),
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
) -> None:
"""Add a new project.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
Cloud mode examples:\n
bm project add research # No local sync\n
bm project add research --local-path ~/docs # With local sync\n
@@ -118,14 +222,24 @@ def add_project(
Local mode example:\n
bm project add research ~/Documents/research
"""
try:
validate_routing_flags(local, cloud)
except ValueError as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
config = ConfigManager().config
# Determine effective mode: default local, cloud only when explicitly requested.
effective_cloud_mode = cloud and not local
# Resolve local sync path early (needed for both cloud and local mode)
local_sync_path: str | None = None
if local_path:
local_sync_path = Path(os.path.abspath(os.path.expanduser(local_path))).as_posix()
if config.cloud_mode_enabled:
if effective_cloud_mode:
_require_cloud_credentials(config)
# Cloud mode: path auto-generated from name, local sync is optional
async def _add_project():
@@ -136,7 +250,7 @@ def add_project(
"local_sync_path": local_sync_path,
"set_default": set_default,
}
response = await call_post(client, "/projects/projects", json=data)
response = await call_post(client, "/v2/projects/", json=data)
return ProjectStatusResponse.model_validate(response.json())
else:
# Local mode: path is required
@@ -150,27 +264,30 @@ def add_project(
async def _add_project():
async with get_client() as client:
data = {"name": name, "path": resolved_path, "set_default": set_default}
response = await call_post(client, "/projects/projects", json=data)
response = await call_post(client, "/v2/projects/", json=data)
return ProjectStatusResponse.model_validate(response.json())
try:
result = run_with_cleanup(_add_project())
with force_routing(local=local, cloud=cloud):
result = run_with_cleanup(_add_project())
console.print(f"[green]{result.message}[/green]")
# Save local sync path to config if in cloud mode
if config.cloud_mode_enabled and local_sync_path:
from basic_memory.config import CloudProjectConfig
if effective_cloud_mode and local_sync_path:
# Create local directory if it doesn't exist
local_dir = Path(local_sync_path)
local_dir.mkdir(parents=True, exist_ok=True)
# Update config with sync path
config.cloud_projects[name] = CloudProjectConfig(
local_path=local_sync_path,
last_sync=None,
bisync_initialized=False,
)
# Update project entry with sync path
entry = config.projects.get(name)
if entry:
entry.cloud_sync_path = local_sync_path
else:
# Project may not be in local config yet (cloud-only add)
config.projects[name] = ProjectEntry(
path=local_sync_path,
cloud_sync_path=local_sync_path,
)
ConfigManager().save_config(config)
console.print(f"\n[green]Local sync path configured: {local_sync_path}[/green]")
@@ -194,15 +311,12 @@ def setup_project_sync(
"""
config_manager = ConfigManager()
config = config_manager.config
if not config.cloud_mode_enabled:
console.print("[red]Error: sync-setup only available in cloud mode[/red]")
raise typer.Exit(1)
_require_cloud_credentials(config)
async def _verify_project_exists():
"""Verify the project exists on cloud by listing all projects."""
async with get_client() as client:
response = await call_get(client, "/projects/projects")
response = await call_get(client, "/v2/projects/")
project_list = response.json()
project_names = [p["name"] for p in project_list["projects"]]
if name not in project_names:
@@ -211,20 +325,24 @@ def setup_project_sync(
try:
# Verify project exists on cloud
run_with_cleanup(_verify_project_exists())
with force_routing(cloud=True):
run_with_cleanup(_verify_project_exists())
# Resolve and create local path
resolved_path = Path(os.path.abspath(os.path.expanduser(local_path)))
resolved_path.mkdir(parents=True, exist_ok=True)
# Update local config with sync path
from basic_memory.config import CloudProjectConfig
config.cloud_projects[name] = CloudProjectConfig(
local_path=resolved_path.as_posix(),
last_sync=None,
bisync_initialized=False,
)
# Update project entry with sync path
entry = config.projects.get(name)
if entry:
entry.cloud_sync_path = resolved_path.as_posix()
entry.bisync_initialized = False
entry.last_sync = None
else:
config.projects[name] = ProjectEntry(
path=resolved_path.as_posix(),
cloud_sync_path=resolved_path.as_posix(),
)
config_manager.save_config(config)
console.print(f"[green]Sync configured for project '{name}'[/green]")
@@ -243,8 +361,21 @@ def remove_project(
delete_notes: bool = typer.Option(
False, "--delete-notes", help="Delete project files from disk"
),
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
) -> None:
"""Remove a project."""
"""Remove a project.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
try:
validate_routing_flags(local, cloud)
except ValueError as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
async def _remove_project():
async with get_client() as client:
@@ -265,11 +396,12 @@ def remove_project(
try:
# Get config to check for local sync path and bisync state
config = ConfigManager().config
local_path = None
local_path_config = None
has_bisync_state = False
if config.cloud_mode_enabled and name in config.cloud_projects:
local_path = config.cloud_projects[name].local_path
entry = config.projects.get(name)
if cloud and entry and entry.cloud_sync_path:
local_path_config = entry.cloud_sync_path
# Check for bisync state
from basic_memory.cli.commands.cloud.rclone_commands import get_project_bisync_state
@@ -278,17 +410,18 @@ def remove_project(
has_bisync_state = bisync_state_path.exists()
# Remove project from cloud/API
result = run_with_cleanup(_remove_project())
with force_routing(local=local, cloud=cloud):
result = run_with_cleanup(_remove_project())
console.print(f"[green]{result.message}[/green]")
# Clean up local sync directory if it exists and delete_notes is True
if delete_notes and local_path:
local_dir = Path(local_path)
if delete_notes and local_path_config:
local_dir = Path(local_path_config)
if local_dir.exists():
import shutil
shutil.rmtree(local_dir)
console.print(f"[green]Removed local sync directory: {local_path}[/green]")
console.print(f"[green]Removed local sync directory: {local_path_config}[/green]")
# Clean up bisync state if it exists
if has_bisync_state:
@@ -300,15 +433,17 @@ def remove_project(
shutil.rmtree(bisync_state_path)
console.print("[green]Removed bisync state[/green]")
# Clean up cloud_projects config entry
if config.cloud_mode_enabled and name in config.cloud_projects:
del config.cloud_projects[name]
# Clean up cloud sync fields on the project entry
if cloud and entry and entry.cloud_sync_path:
entry.cloud_sync_path = None
entry.bisync_initialized = False
entry.last_sync = None
ConfigManager().save_config(config)
# Show informative message if files were not deleted
if not delete_notes:
if local_path:
console.print(f"[yellow]Note: Local files remain at {local_path}[/yellow]")
if local_path_config:
console.print(f"[yellow]Note: Local files remain at {local_path_config}[/yellow]")
except Exception as e:
console.print(f"[red]Error removing project: {str(e)}[/red]")
@@ -318,16 +453,14 @@ def remove_project(
@project_app.command("default")
def set_default_project(
name: str = typer.Argument(..., help="Name of the project to set as CLI default"),
local: bool = typer.Option(
False, "--local", help="Force local API routing (required in cloud mode)"
),
) -> None:
"""Set the default project when 'config.default_project_mode' is set.
"""Set the default project used as fallback when no project is specified.
Note: This command is only available in local mode.
In cloud mode, use --local to modify the local configuration.
"""
config = ConfigManager().config
if config.cloud_mode_enabled:
console.print("[red]Error: 'default' command is not available in cloud mode[/red]")
raise typer.Exit(1)
async def _set_default():
async with get_client() as client:
@@ -346,7 +479,8 @@ def set_default_project(
return ProjectStatusResponse.model_validate(response.json())
try:
result = run_with_cleanup(_set_default())
with force_routing(local=local):
result = run_with_cleanup(_set_default())
console.print(f"[green]{result.message}[/green]")
except Exception as e:
console.print(f"[red]Error setting default project: {str(e)}[/red]")
@@ -354,24 +488,24 @@ def set_default_project(
@project_app.command("sync-config")
def synchronize_projects() -> None:
def synchronize_projects(
local: bool = typer.Option(
False, "--local", help="Force local API routing (required in cloud mode)"
),
) -> None:
"""Synchronize project config between configuration file and database.
Note: This command is only available in local mode.
In cloud mode, use --local to sync local configuration.
"""
config = ConfigManager().config
if config.cloud_mode_enabled:
console.print("[red]Error: 'sync-config' command is not available in cloud mode[/red]")
raise typer.Exit(1)
async def _sync_config():
async with get_client() as client:
response = await call_post(client, "/projects/config/sync")
response = await call_post(client, "/v2/projects/config/sync")
return ProjectStatusResponse.model_validate(response.json())
try:
result = run_with_cleanup(_sync_config())
with force_routing(local=local):
result = run_with_cleanup(_sync_config())
console.print(f"[green]{result.message}[/green]")
except Exception as e: # pragma: no cover
console.print(f"[red]Error synchronizing projects: {str(e)}[/red]")
@@ -382,31 +516,34 @@ def synchronize_projects() -> None:
def move_project(
name: str = typer.Argument(..., help="Name of the project to move"),
new_path: str = typer.Argument(..., help="New absolute path for the project"),
local: bool = typer.Option(
False, "--local", help="Force local API routing (required in cloud mode)"
),
) -> None:
"""Move a project to a new location.
Note: This command is only available in local mode.
In cloud mode, use --local to modify local project paths.
"""
config = ConfigManager().config
if config.cloud_mode_enabled:
console.print("[red]Error: 'move' command is not available in cloud mode[/red]")
raise typer.Exit(1)
# Resolve to absolute path
resolved_path = Path(os.path.abspath(os.path.expanduser(new_path))).as_posix()
async def _move_project():
async with get_client() as client:
data = {"path": resolved_path}
project_permalink = generate_permalink(name)
# TODO fix route to use ProjectPathDep
response = await call_patch(client, f"/{name}/project/{project_permalink}", json=data)
resolve_response = await call_post(
client,
"/v2/projects/resolve",
json={"identifier": name},
)
project_info = ProjectResolveResponse.model_validate(resolve_response.json())
response = await call_patch(
client, f"/v2/projects/{project_info.external_id}", json=data
)
return ProjectStatusResponse.model_validate(response.json())
try:
result = run_with_cleanup(_move_project())
with force_routing(local=local):
result = run_with_cleanup(_move_project())
console.print(f"[green]{result.message}[/green]")
# Show important file movement reminder
@@ -428,6 +565,74 @@ def move_project(
raise typer.Exit(1)
@project_app.command("set-cloud")
def set_cloud(
name: str = typer.Argument(..., help="Name of the project to route through cloud"),
) -> None:
"""Set a project to cloud mode (route through cloud API).
Requires either an API key or an active OAuth session.
Examples:
bm cloud set-key bmc_abc123... # save API key, then:
bm project set-cloud research # route "research" through cloud
bm cloud login # OAuth login, then:
bm project set-cloud research # route "research" through cloud
"""
from basic_memory.cli.auth import CLIAuth
config_manager = ConfigManager()
config = config_manager.config
# Validate project exists in config
if name not in config.projects:
console.print(f"[red]Error: Project '{name}' not found in config[/red]")
raise typer.Exit(1)
# Validate credentials: API key or OAuth session
has_api_key = bool(config.cloud_api_key)
has_oauth = False
if not has_api_key:
auth = CLIAuth(client_id=config.cloud_client_id, authkit_domain=config.cloud_domain)
has_oauth = auth.load_tokens() is not None
if not has_api_key and not has_oauth:
console.print("[red]Error: No cloud credentials found[/red]")
console.print("[dim]Run 'bm cloud set-key <key>' or 'bm cloud login' first[/dim]")
raise typer.Exit(1)
config.set_project_mode(name, ProjectMode.CLOUD)
config_manager.save_config(config)
console.print(f"[green]Project '{name}' set to cloud mode[/green]")
console.print("[dim]MCP tools and CLI commands for this project will route through cloud[/dim]")
@project_app.command("set-local")
def set_local(
name: str = typer.Argument(..., help="Name of the project to revert to local mode"),
) -> None:
"""Revert a project to local mode (use in-process ASGI transport).
Example:
bm project set-local research
"""
config_manager = ConfigManager()
config = config_manager.config
# Validate project exists in config
if name not in config.projects:
console.print(f"[red]Error: Project '{name}' not found in config[/red]")
raise typer.Exit(1)
config.set_project_mode(name, ProjectMode.LOCAL)
config_manager.save_config(config)
console.print(f"[green]Project '{name}' set to local mode[/green]")
console.print("[dim]MCP tools and CLI commands for this project will use local transport[/dim]")
@project_app.command("sync")
def sync_project_command(
name: str = typer.Option(..., "--name", help="Project name to sync"),
@@ -441,9 +646,7 @@ def sync_project_command(
bm project sync --name research --dry-run
"""
config = ConfigManager().config
if not config.cloud_mode_enabled:
console.print("[red]Error: sync only available in cloud mode[/red]")
raise typer.Exit(1)
_require_cloud_credentials(config)
try:
# Get tenant info for bucket name
@@ -453,22 +656,22 @@ def sync_project_command(
# Get project info
async def _get_project():
async with get_client() as client:
response = await call_get(client, "/projects/projects")
response = await call_get(client, "/v2/projects/")
projects_list = ProjectList.model_validate(response.json())
for proj in projects_list.projects:
if generate_permalink(proj.name) == generate_permalink(name):
return proj
return None
project_data = run_with_cleanup(_get_project())
with force_routing(cloud=True):
project_data = run_with_cleanup(_get_project())
if not project_data:
console.print(f"[red]Error: Project '{name}' not found[/red]")
raise typer.Exit(1)
# Get local_sync_path from cloud_projects config
local_sync_path = None
if name in config.cloud_projects:
local_sync_path = config.cloud_projects[name].local_path
# Get local_sync_path from project entry
sync_entry = config.projects.get(name)
local_sync_path = sync_entry.cloud_sync_path if sync_entry else None
if not local_sync_path:
console.print(f"[red]Error: Project '{name}' has no local_sync_path configured[/red]")
@@ -494,14 +697,16 @@ def sync_project_command(
async def _trigger_db_sync():
async with get_client() as client:
permalink = generate_permalink(name)
response = await call_post(
client, f"/{permalink}/project/sync?force_full=true", json={}
client,
f"/v2/projects/{project_data.external_id}/sync?force_full=true",
json={},
)
return response.json()
try:
result = run_with_cleanup(_trigger_db_sync())
with force_routing(cloud=True):
result = run_with_cleanup(_trigger_db_sync())
console.print(f"[dim]Database sync initiated: {result.get('message')}[/dim]")
except Exception as e:
console.print(f"[yellow]Warning: Could not trigger database sync: {e}[/yellow]")
@@ -532,9 +737,7 @@ def bisync_project_command(
bm project bisync --name research --dry-run # Preview changes
"""
config = ConfigManager().config
if not config.cloud_mode_enabled:
console.print("[red]Error: bisync only available in cloud mode[/red]")
raise typer.Exit(1)
_require_cloud_credentials(config)
try:
# Get tenant info for bucket name
@@ -544,22 +747,22 @@ def bisync_project_command(
# Get project info
async def _get_project():
async with get_client() as client:
response = await call_get(client, "/projects/projects")
response = await call_get(client, "/v2/projects/")
projects_list = ProjectList.model_validate(response.json())
for proj in projects_list.projects:
if generate_permalink(proj.name) == generate_permalink(name):
return proj
return None
project_data = run_with_cleanup(_get_project())
with force_routing(cloud=True):
project_data = run_with_cleanup(_get_project())
if not project_data:
console.print(f"[red]Error: Project '{name}' not found[/red]")
raise typer.Exit(1)
# Get local_sync_path from cloud_projects config
local_sync_path = None
if name in config.cloud_projects:
local_sync_path = config.cloud_projects[name].local_path
# Get local_sync_path from project entry
sync_entry = config.projects.get(name)
local_sync_path = sync_entry.cloud_sync_path if sync_entry else None
if not local_sync_path:
console.print(f"[red]Error: Project '{name}' has no local_sync_path configured[/red]")
@@ -582,9 +785,11 @@ def bisync_project_command(
if success:
console.print(f"[green]{name} bisync completed successfully[/green]")
# Update config
config.cloud_projects[name].last_sync = datetime.now()
config.cloud_projects[name].bisync_initialized = True
# Update config — sync_entry is guaranteed non-None because
# we checked local_sync_path above (which comes from sync_entry)
assert sync_entry is not None
sync_entry.last_sync = datetime.now()
sync_entry.bisync_initialized = True
ConfigManager().save_config(config)
# Trigger database sync if not a dry run
@@ -592,14 +797,16 @@ def bisync_project_command(
async def _trigger_db_sync():
async with get_client() as client:
permalink = generate_permalink(name)
response = await call_post(
client, f"/{permalink}/project/sync?force_full=true", json={}
client,
f"/v2/projects/{project_data.external_id}/sync?force_full=true",
json={},
)
return response.json()
try:
result = run_with_cleanup(_trigger_db_sync())
with force_routing(cloud=True):
result = run_with_cleanup(_trigger_db_sync())
console.print(f"[dim]Database sync initiated: {result.get('message')}[/dim]")
except Exception as e:
console.print(f"[yellow]Warning: Could not trigger database sync: {e}[/yellow]")
@@ -626,9 +833,7 @@ def check_project_command(
bm project check --name research
"""
config = ConfigManager().config
if not config.cloud_mode_enabled:
console.print("[red]Error: check only available in cloud mode[/red]")
raise typer.Exit(1)
_require_cloud_credentials(config)
try:
# Get tenant info for bucket name
@@ -638,22 +843,22 @@ def check_project_command(
# Get project info
async def _get_project():
async with get_client() as client:
response = await call_get(client, "/projects/projects")
response = await call_get(client, "/v2/projects/")
projects_list = ProjectList.model_validate(response.json())
for proj in projects_list.projects:
if generate_permalink(proj.name) == generate_permalink(name):
return proj
return None
project_data = run_with_cleanup(_get_project())
with force_routing(cloud=True):
project_data = run_with_cleanup(_get_project())
if not project_data:
console.print(f"[red]Error: Project '{name}' not found[/red]")
raise typer.Exit(1)
# Get local_sync_path from cloud_projects config
local_sync_path = None
if name in config.cloud_projects:
local_sync_path = config.cloud_projects[name].local_path
# Get local_sync_path from project entry
check_entry = config.projects.get(name)
local_sync_path = check_entry.cloud_sync_path if check_entry else None
if not local_sync_path:
console.print(f"[red]Error: Project '{name}' has no local_sync_path configured[/red]")
@@ -719,56 +924,102 @@ def bisync_reset(
def ls_project_command(
name: str = typer.Option(..., "--name", help="Project name to list files from"),
path: str = typer.Argument(None, help="Path within project (optional)"),
local: bool = typer.Option(False, "--local", help="List files from local project instance"),
cloud: bool = typer.Option(False, "--cloud", help="List files from cloud project instance"),
) -> None:
"""List files in remote project.
"""List files in a project.
Examples:
bm project ls --name research
bm project ls --name research --local
bm project ls --name research --cloud
bm project ls --name research subfolder
"""
config = ConfigManager().config
if not config.cloud_mode_enabled:
console.print("[red]Error: ls only available in cloud mode[/red]")
try:
validate_routing_flags(local, cloud)
except ValueError as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
try:
# Get tenant info for bucket name
tenant_info = run_with_cleanup(get_mount_info())
bucket_name = tenant_info.bucket_name
use_cloud_route = cloud and not local
def _list_local_files(project_path: str, subpath: str | None = None) -> list[str]:
project_root = Path(normalize_project_path(project_path)).expanduser().resolve()
target_dir = project_root
if subpath:
requested = Path(subpath)
if requested.is_absolute():
raise ValueError("Path must be relative to the project root")
target_dir = (project_root / requested).resolve()
if not target_dir.is_relative_to(project_root):
raise ValueError("Path must stay within the project root")
if not target_dir.exists():
raise ValueError(f"Path not found: {target_dir}")
if not target_dir.is_dir():
raise ValueError(f"Path is not a directory: {target_dir}")
files: list[str] = []
for file_path in sorted(target_dir.rglob("*")):
if file_path.is_file():
size = file_path.stat().st_size
relative = file_path.relative_to(project_root).as_posix()
files.append(f"{size:10d} {relative}")
return files
try:
# Get project info
async def _get_project():
async with get_client() as client:
response = await call_get(client, "/projects/projects")
response = await call_get(client, "/v2/projects/")
projects_list = ProjectList.model_validate(response.json())
for proj in projects_list.projects:
if generate_permalink(proj.name) == generate_permalink(name):
return proj
return None
project_data = run_with_cleanup(_get_project())
if not project_data:
console.print(f"[red]Error: Project '{name}' not found[/red]")
raise typer.Exit(1)
if use_cloud_route:
config = ConfigManager().config
_require_cloud_credentials(config)
# Create SyncProject (local_sync_path not needed for ls)
sync_project = SyncProject(
name=project_data.name,
path=normalize_project_path(project_data.path),
)
tenant_info = run_with_cleanup(get_mount_info())
bucket_name = tenant_info.bucket_name
# List files
files = project_ls(sync_project, bucket_name, path=path)
with force_routing(cloud=True):
project_data = run_with_cleanup(_get_project())
if not project_data:
console.print(f"[red]Error: Project '{name}' not found[/red]")
raise typer.Exit(1)
sync_project = SyncProject(
name=project_data.name,
path=normalize_project_path(project_data.path),
)
files = project_ls(sync_project, bucket_name, path=path)
target_label = "CLOUD"
else:
with force_routing(local=True):
project_data = run_with_cleanup(_get_project())
if not project_data:
console.print(f"[red]Error: Project '{name}' not found[/red]")
raise typer.Exit(1)
files = _list_local_files(project_data.path, path)
target_label = "LOCAL"
if files:
console.print(f"\n[bold]Files in {name}" + (f"/{path}" if path else "") + ":[/bold]")
heading = f"\n[bold]Files in {name} ({target_label})"
if path:
heading += f"/{path}"
heading += ":[/bold]"
console.print(heading)
for file in files:
console.print(f" {file}")
console.print(f"\n[dim]Total: {len(files)} files[/dim]")
else:
console.print(
f"[yellow]No files found in {name}" + (f"/{path}" if path else "") + "[/yellow]"
)
prefix = f"[yellow]No files found in {name} ({target_label})"
console.print(prefix + (f"/{path}" if path else "") + "[/yellow]")
except Exception as e:
console.print(f"[red]Error: {e}[/red]")
@@ -779,11 +1030,26 @@ def ls_project_command(
def display_project_info(
name: str = typer.Argument(..., help="Name of the project"),
json_output: bool = typer.Option(False, "--json", help="Output in JSON format"),
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Display detailed information and statistics about the current project."""
"""Display detailed information and statistics about the current project.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
try:
validate_routing_flags(local, cloud)
except ValueError as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
try:
# Get project info
info = run_with_cleanup(get_project_info(name))
with force_routing(local=local, cloud=cloud):
info = run_with_cleanup(get_project_info(name))
if json_output:
# Convert to JSON and print
+91
View File
@@ -0,0 +1,91 @@
"""CLI routing utilities for --local/--cloud flag handling.
This module provides utilities for CLI commands to override default routing.
This allows users to force local or cloud routing per-command.
The routing is controlled via environment variables:
- BASIC_MEMORY_FORCE_LOCAL: When "true", forces local ASGI transport
- BASIC_MEMORY_FORCE_CLOUD: When "true", forces cloud proxy transport
- BASIC_MEMORY_EXPLICIT_ROUTING: When "true", signals that --local/--cloud
was explicitly passed, overriding per-project routing in get_client()
- These are checked in basic_memory.mcp.async_client.get_client()
"""
import os
from contextlib import contextmanager
from typing import Generator
@contextmanager
def force_routing(local: bool = False, cloud: bool = False) -> Generator[None, None, None]:
"""Context manager to temporarily override routing mode.
Sets environment variables that are checked by get_client() to determine
whether to use local ASGI transport or cloud proxy transport.
When either flag is set, BASIC_MEMORY_EXPLICIT_ROUTING is also set so
that get_client() skips per-project routing and honors the flag directly.
This only affects CLI commands the MCP server sets FORCE_LOCAL directly
(without EXPLICIT_ROUTING), so per-project routing still works for MCP tools.
Args:
local: If True, force local ASGI transport
cloud: If True, force cloud proxy transport
Usage:
with force_routing(local=True):
# All API calls will use local ASGI transport
await some_api_call()
Raises:
ValueError: If both local and cloud are True
"""
if local and cloud:
raise ValueError("Cannot specify both --local and --cloud")
# Save original values
original_force_local = os.environ.get("BASIC_MEMORY_FORCE_LOCAL")
original_force_cloud = os.environ.get("BASIC_MEMORY_FORCE_CLOUD")
original_explicit = os.environ.get("BASIC_MEMORY_EXPLICIT_ROUTING")
try:
if local:
os.environ["BASIC_MEMORY_FORCE_LOCAL"] = "true"
os.environ.pop("BASIC_MEMORY_FORCE_CLOUD", None)
os.environ["BASIC_MEMORY_EXPLICIT_ROUTING"] = "true"
elif cloud:
os.environ.pop("BASIC_MEMORY_FORCE_LOCAL", None)
os.environ["BASIC_MEMORY_FORCE_CLOUD"] = "true"
os.environ["BASIC_MEMORY_EXPLICIT_ROUTING"] = "true"
# If neither is set, don't change anything (use default behavior)
yield
finally:
# Restore original values
if original_force_local is None:
os.environ.pop("BASIC_MEMORY_FORCE_LOCAL", None)
else:
os.environ["BASIC_MEMORY_FORCE_LOCAL"] = original_force_local
if original_force_cloud is None:
os.environ.pop("BASIC_MEMORY_FORCE_CLOUD", None)
else:
os.environ["BASIC_MEMORY_FORCE_CLOUD"] = original_force_cloud
if original_explicit is None:
os.environ.pop("BASIC_MEMORY_EXPLICIT_ROUTING", None)
else:
os.environ["BASIC_MEMORY_EXPLICIT_ROUTING"] = original_explicit
def validate_routing_flags(local: bool, cloud: bool) -> None:
"""Validate that --local and --cloud flags are not both specified.
Args:
local: Value of --local flag
cloud: Value of --cloud flag
Raises:
ValueError: If both flags are True
"""
if local and cloud:
raise ValueError("Cannot specify both --local and --cloud flags")
+362
View File
@@ -0,0 +1,362 @@
"""Schema management CLI commands for Basic Memory.
Provides CLI access to schema validation, inference, and drift detection.
Registered as a subcommand group: `bm schema validate`, `bm schema infer`, `bm schema diff`.
"""
import json
from typing import Annotated, Optional
import typer
from loguru import logger
from rich.console import Console
from rich.table import Table
from basic_memory.cli.app import app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.cli.commands.routing import force_routing, validate_routing_flags
from basic_memory.config import ConfigManager
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.project_context import get_active_project
console = Console()
schema_app = typer.Typer(help="Schema management commands")
app.add_typer(schema_app, name="schema")
def _resolve_project_name(project: Optional[str]) -> Optional[str]:
"""Resolve project name from CLI argument or config default."""
config_manager = ConfigManager()
if project is not None:
project_name, _ = config_manager.get_project(project)
if not project_name:
typer.echo(f"No project found named: {project}", err=True)
raise typer.Exit(1)
return project_name
return config_manager.default_project
# --- Validate ---
async def _run_validate(
target: Optional[str] = None,
project: Optional[str] = None,
strict: bool = False,
):
"""Run schema validation via the API."""
from basic_memory.mcp.clients.schema import SchemaClient
async with get_client(project_name=project) as client:
active_project = await get_active_project(client, project, None)
schema_client = SchemaClient(client, active_project.external_id)
# Determine if target is a note identifier or note type
# Heuristic: if target contains / or ., treat as identifier
entity_type = None
identifier = None
if target:
if "/" in target or "." in target:
identifier = target
else:
entity_type = target
report = await schema_client.validate(
entity_type=entity_type,
identifier=identifier,
)
# --- Display results ---
if report.total_notes == 0:
if report.total_entities == 0:
console.print(f"[yellow]No notes of type '{entity_type}' found.[/yellow]")
else:
console.print(
f"[yellow]Found {report.total_entities} notes but no schema "
f"defined for '{entity_type}'.[/yellow]"
)
return
table = Table(title=f"Schema Validation: {entity_type or identifier or 'all'}")
table.add_column("Note", style="cyan")
table.add_column("Status", justify="center")
table.add_column("Warnings", justify="right")
table.add_column("Errors", justify="right")
for result in report.results:
if result.passed and not result.warnings:
status = "[green]pass[/green]"
elif result.passed:
status = "[yellow]warn[/yellow]"
else:
status = "[red]fail[/red]"
table.add_row(
result.note_identifier,
status,
str(len(result.warnings)),
str(len(result.errors)),
)
console.print(table)
console.print(
f"\nSummary: {report.valid_count}/{report.total_notes} valid, "
f"{report.warning_count} warnings, {report.error_count} errors"
)
# Exit with error code in strict mode if there are failures
if strict and report.error_count > 0:
raise typer.Exit(1)
@schema_app.command()
def validate(
target: Annotated[
Optional[str],
typer.Argument(help="Note path or note type to validate"),
] = None,
project: Annotated[
Optional[str],
typer.Option(help="The project name."),
] = None,
strict: bool = typer.Option(False, "--strict", help="Exit with error on validation failures"),
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Validate notes against their schemas.
TARGET can be a note path (e.g., people/ada-lovelace.md) or a note type
(e.g., person). If omitted, validates all notes that have schemas.
Use --strict to exit with error code 1 if any validation errors are found.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
try:
validate_routing_flags(local, cloud)
project_name = _resolve_project_name(project)
with force_routing(local=local, cloud=cloud):
run_with_cleanup(_run_validate(target, project_name, strict))
except ValueError as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
if not isinstance(e, typer.Exit):
logger.error(f"Error during schema validate: {e}")
typer.echo(f"Error during schema validate: {e}", err=True)
raise typer.Exit(1)
raise
# --- Infer ---
async def _run_infer(
entity_type: str,
project: Optional[str] = None,
threshold: float = 0.25,
save: bool = False,
):
"""Run schema inference via the API."""
from basic_memory.mcp.clients.schema import SchemaClient
async with get_client(project_name=project) as client:
active_project = await get_active_project(client, project, None)
schema_client = SchemaClient(client, active_project.external_id)
report = await schema_client.infer(entity_type, threshold=threshold)
if report.notes_analyzed == 0:
console.print(f"[yellow]No notes found with type: {entity_type}[/yellow]")
return
# --- Empty schema guard ---
# Trigger: notes were analyzed but no fields met the threshold
# Why: dumping hundreds of excluded fields is not useful output
# Outcome: show count and suggest a more specific type
if not report.suggested_schema:
console.print(
f"\n[yellow]Analyzed {report.notes_analyzed} notes of type '{entity_type}', "
f"but no fields met the {threshold:.0%} threshold.[/yellow]\n"
)
console.print(
f"This usually means '{entity_type}' is too broad — "
f"the notes don't share a consistent structure.\n"
)
console.print("[bold]Suggestions:[/bold]")
console.print(" 1. Use a more specific type")
console.print(
f" 2. Lower the threshold: bm schema infer {entity_type} --threshold 0.1"
)
console.print(" 3. Create typed notes with write_note using a specific note_type")
return
# --- Display frequency analysis ---
console.print(
f"\n[bold]Analyzing {report.notes_analyzed} notes with type: {entity_type}...[/bold]\n"
)
table = Table(title="Field Frequencies")
table.add_column("Field", style="cyan")
table.add_column("Source")
table.add_column("Count", justify="right")
table.add_column("Percentage", justify="right")
table.add_column("Suggested")
for freq in report.field_frequencies:
pct = f"{freq.percentage:.0%}"
if freq.name in report.suggested_required:
suggested = "[green]required[/green]"
elif freq.name in report.suggested_optional:
suggested = "[yellow]optional[/yellow]"
else:
suggested = "[dim]excluded[/dim]"
table.add_row(
freq.name,
freq.source,
str(freq.count),
pct,
suggested,
)
console.print(table)
# --- Display suggested schema ---
console.print("\n[bold]Suggested schema:[/bold]")
console.print(json.dumps(report.suggested_schema, indent=2))
if save:
console.print(
f"\n[yellow]--save not yet implemented. "
f"Copy the schema above into schema/{entity_type}.md[/yellow]"
)
@schema_app.command()
def infer(
entity_type: Annotated[
str,
typer.Argument(help="Note type to analyze (e.g., person, meeting)"),
],
project: Annotated[
Optional[str],
typer.Option(help="The project name."),
] = None,
threshold: float = typer.Option(
0.25, "--threshold", help="Minimum frequency for optional fields (0-1)"
),
save: bool = typer.Option(False, "--save", help="Save inferred schema to schema/ directory"),
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Infer schema from existing notes of a type.
Analyzes all notes with the given type and suggests a Picoschema
definition based on observation and relation frequency.
Fields present in 95%+ of notes become required. Fields above the
threshold (default 25%) become optional. Fields below threshold are excluded.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
try:
validate_routing_flags(local, cloud)
project_name = _resolve_project_name(project)
with force_routing(local=local, cloud=cloud):
run_with_cleanup(_run_infer(entity_type, project_name, threshold, save))
except ValueError as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
if not isinstance(e, typer.Exit):
logger.error(f"Error during schema infer: {e}")
typer.echo(f"Error during schema infer: {e}", err=True)
raise typer.Exit(1)
raise
# --- Diff ---
async def _run_diff(
entity_type: str,
project: Optional[str] = None,
):
"""Run schema drift detection via the API."""
from basic_memory.mcp.clients.schema import SchemaClient
async with get_client(project_name=project) as client:
active_project = await get_active_project(client, project, None)
schema_client = SchemaClient(client, active_project.external_id)
report = await schema_client.diff(entity_type)
has_drift = report.new_fields or report.dropped_fields or report.cardinality_changes
if not has_drift:
console.print(f"[green]No drift detected for {entity_type} schema.[/green]")
return
console.print(f"\n[bold]Schema drift detected for {entity_type}:[/bold]\n")
if report.new_fields:
console.print("[green]+ New fields (common in notes, not in schema):[/green]")
for f in report.new_fields:
console.print(f" + {f.name}: {f.percentage:.0%} of notes ({f.source})")
if report.dropped_fields:
console.print("[red]- Dropped fields (in schema, rare in notes):[/red]")
for f in report.dropped_fields:
console.print(f" - {f.name}: {f.percentage:.0%} of notes ({f.source})")
if report.cardinality_changes:
console.print("[yellow]~ Cardinality changes:[/yellow]")
for change in report.cardinality_changes:
console.print(f" ~ {change}")
@schema_app.command()
def diff(
entity_type: Annotated[
str,
typer.Argument(help="Note type to check for drift"),
],
project: Annotated[
Optional[str],
typer.Option(help="The project name."),
] = None,
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Show drift between schema and actual usage.
Compares the existing schema definition against how notes of that type
are actually structured. Identifies new fields,
dropped fields, and cardinality changes.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
try:
validate_routing_flags(local, cloud)
project_name = _resolve_project_name(project)
with force_routing(local=local, cloud=cloud):
run_with_cleanup(_run_diff(entity_type, project_name))
except ValueError as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
if not isinstance(e, typer.Exit):
logger.error(f"Error during schema diff: {e}")
typer.echo(f"Error during schema diff: {e}", err=True)
raise typer.Exit(1)
raise
+21 -4
View File
@@ -11,6 +11,8 @@ from rich.panel import Panel
from rich.tree import Tree
from basic_memory.cli.app import app
from basic_memory.cli.commands.routing import force_routing, validate_routing_flags
from basic_memory.config import ConfigManager
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.tools.utils import call_post
from basic_memory.schemas import SyncReportResponse
@@ -141,11 +143,13 @@ def display_changes(
async def run_status(project: Optional[str] = None, verbose: bool = False): # pragma: no cover
"""Check sync status of files vs database."""
# Resolve default project so get_client() can route per-project
project = project or ConfigManager().default_project
try:
async with get_client() as client:
async with get_client(project_name=project) as client:
project_item = await get_active_project(client, project, None)
response = await call_post(client, f"{project_item.project_url}/project/status")
response = await call_post(client, f"/v2/projects/{project_item.external_id}/status")
sync_report = SyncReportResponse.model_validate(response.json())
display_changes(project_item.name, "Status", sync_report, verbose)
@@ -162,12 +166,25 @@ def status(
typer.Option(help="The project name."),
] = None,
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed file information"),
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Show sync status between files and database."""
"""Show sync status between files and database.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
from basic_memory.cli.commands.command_utils import run_with_cleanup
try:
run_with_cleanup(run_status(project, verbose)) # pragma: no cover
validate_routing_flags(local, cloud)
with force_routing(local=local, cloud=cloud):
run_with_cleanup(run_status(project, verbose)) # pragma: no cover
except ValueError as e:
console.print(f"[red]Error: {e}[/red]")
raise typer.Exit(code=1)
except Exception as e:
logger.error(f"Error checking status: {e}")
typer.echo(f"Error checking status: {e}", err=True)
+709 -120
View File
@@ -1,35 +1,285 @@
"""CLI tool commands for Basic Memory."""
import json
import sys
from typing import Annotated, List, Optional
from typing import Annotated, Any, List, Optional
import typer
import yaml
from loguru import logger
from rich import print as rprint
from basic_memory.cli.app import app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.cli.commands.routing import force_routing, validate_routing_flags
from basic_memory.config import ConfigManager
from basic_memory.mcp.clients import KnowledgeClient, ResourceClient
from basic_memory.mcp.project_context import get_project_client
from basic_memory.mcp.tools.utils import call_get
from basic_memory.schemas.base import Entity, TimeFrame
from basic_memory.schemas.memory import GraphContext, MemoryUrl, memory_url_path
from basic_memory.schemas.search import SearchItemType
# Import prompts
from basic_memory.mcp.prompts.continue_conversation import (
continue_conversation as mcp_continue_conversation,
)
from basic_memory.mcp.prompts.recent_activity import (
recent_activity_prompt as recent_activity_prompt,
)
from basic_memory.mcp.tools import build_context as mcp_build_context
from basic_memory.mcp.tools import edit_note as mcp_edit_note
from basic_memory.mcp.tools import read_note as mcp_read_note
from basic_memory.mcp.tools import recent_activity as mcp_recent_activity
from basic_memory.mcp.tools import search_notes as mcp_search
from basic_memory.mcp.tools import write_note as mcp_write_note
from basic_memory.schemas.base import TimeFrame
from basic_memory.schemas.memory import MemoryUrl
from basic_memory.schemas.search import SearchItemType
tool_app = typer.Typer()
app.add_typer(tool_app, name="tool", help="Access to MCP tools via CLI")
VALID_EDIT_OPERATIONS = ["append", "prepend", "find_replace", "replace_section"]
# --- Frontmatter helpers ---
def _parse_opening_frontmatter(content: str) -> tuple[str, dict[str, Any] | None]:
"""Parse and strip an opening YAML frontmatter block if valid.
Returns a tuple of (body_content_or_original, parsed_frontmatter_or_none).
Behavior:
- Only parses frontmatter if the first line is an opening '---' delimiter.
- Requires a closing '---' delimiter.
- Accepts mapping YAML only; malformed or non-mapping YAML is ignored.
- Supports UTF-8 BOM at document start.
"""
if not content:
return content, None
original_content = content
if content.startswith("\ufeff"):
content = content[1:]
lines = content.splitlines(keepends=True)
if not lines:
return original_content, None
if lines[0].rstrip("\r\n").strip() != "---":
return original_content, None
closing_index = None
for index in range(1, len(lines)):
if lines[index].rstrip("\r\n").strip() == "---":
closing_index = index
break
if closing_index is None:
return original_content, None
frontmatter_text = "".join(lines[1:closing_index])
try:
parsed = yaml.safe_load(frontmatter_text) if frontmatter_text else {}
except yaml.YAMLError:
return original_content, None
if parsed is None:
parsed = {}
if not isinstance(parsed, dict):
return original_content, None
body_content = "".join(lines[closing_index + 1 :])
return body_content, parsed
# --- JSON output helpers ---
# These async functions bypass the MCP tool (which returns formatted strings)
# and use API clients directly to return structured data for --format json.
async def _write_note_json(
title: str,
content: str,
folder: str,
project_name: Optional[str],
workspace: Optional[str],
tags: Optional[List[str]],
) -> dict:
"""Write a note and return structured JSON metadata."""
# Use the MCP tool to create/update the entity (handles create-or-update logic)
await mcp_write_note.fn(
title=title,
content=content,
directory=folder,
project=project_name,
workspace=workspace,
tags=tags,
)
# Resolve the entity to get metadata back
async with get_project_client(project_name, workspace) as (client, active_project):
knowledge_client = KnowledgeClient(client, active_project.external_id)
entity = Entity(title=title, directory=folder)
if not entity.permalink:
raise ValueError(f"Could not generate permalink for title={title}, folder={folder}")
entity_id = await knowledge_client.resolve_entity(entity.permalink)
entity = await knowledge_client.get_entity(entity_id)
return {
"title": entity.title,
"permalink": entity.permalink,
"content": content,
"file_path": entity.file_path,
}
async def _read_note_json(
identifier: str,
project_name: Optional[str],
workspace: Optional[str],
page: int,
page_size: int,
) -> dict:
"""Read a note and return structured JSON with content and metadata."""
async with get_project_client(project_name, workspace) as (client, active_project):
knowledge_client = KnowledgeClient(client, active_project.external_id)
resource_client = ResourceClient(client, active_project.external_id)
# Try direct resolution first (works for permalinks and memory URLs)
entity_path = memory_url_path(identifier)
entity_id = None
try:
entity_id = await knowledge_client.resolve_entity(entity_path)
except Exception:
logger.info(f"Direct lookup failed for '{entity_path}', trying title search")
# Fallback: title search (handles plain titles like "My Note")
if entity_id is None:
from basic_memory.mcp.tools.search import search_notes as mcp_search_tool
title_results = await mcp_search_tool.fn(
query=identifier,
search_type="title",
project=project_name,
workspace=workspace,
output_format="json",
)
results = title_results.get("results", []) if isinstance(title_results, dict) else []
if results:
result = results[0]
permalink = result.get("permalink")
if permalink:
entity_id = await knowledge_client.resolve_entity(permalink)
if entity_id is None:
raise ValueError(f"Could not find note matching: {identifier}")
entity = await knowledge_client.get_entity(entity_id)
response = await resource_client.read(entity_id, page=page, page_size=page_size)
return {
"title": entity.title,
"permalink": entity.permalink,
"content": response.text,
"file_path": entity.file_path,
}
async def _edit_note_json(
identifier: str,
operation: str,
content: str,
project_name: Optional[str],
workspace: Optional[str],
section: Optional[str],
find_text: Optional[str],
expected_replacements: int,
) -> dict:
"""Edit a note and return structured JSON metadata."""
async with get_project_client(project_name, workspace) as (client, active_project):
knowledge_client = KnowledgeClient(client, active_project.external_id)
entity_id = await knowledge_client.resolve_entity(identifier)
edit_data: dict[str, Any] = {
"operation": operation,
"content": content,
"expected_replacements": expected_replacements,
}
if section:
edit_data["section"] = section
if find_text:
edit_data["find_text"] = find_text
result = await knowledge_client.patch_entity(entity_id, edit_data, fast=False)
return {
"title": result.title,
"permalink": result.permalink,
"file_path": result.file_path,
"operation": operation,
"checksum": result.checksum,
}
def _validate_edit_note_args(
operation: str, find_text: Optional[str], section: Optional[str]
) -> None:
"""Validate operation-specific required arguments for edit-note."""
if operation not in VALID_EDIT_OPERATIONS:
raise ValueError(
f"Invalid operation '{operation}'. Must be one of: {', '.join(VALID_EDIT_OPERATIONS)}"
)
if operation == "find_replace" and not find_text:
raise ValueError("find_text parameter is required for find_replace operation")
if operation == "replace_section" and not section:
raise ValueError("section parameter is required for replace_section operation")
def _is_edit_note_failure_response(result: str) -> bool:
"""Check whether the MCP edit_note text response indicates a failed edit."""
return result.lstrip().startswith("# Edit Failed")
async def _recent_activity_json(
type: Optional[List[SearchItemType]],
depth: Optional[int],
timeframe: Optional[TimeFrame],
project_name: Optional[str] = None,
workspace: Optional[str] = None,
page: int = 1,
page_size: int = 50,
) -> list:
"""Get recent activity and return structured JSON list."""
async with get_project_client(project_name, workspace) as (client, active_project):
# Build query params matching the MCP tool's logic
params: dict = {"page": page, "page_size": page_size, "max_related": 10}
if depth:
params["depth"] = depth
if timeframe:
params["timeframe"] = timeframe
if type:
params["type"] = [t.value for t in type]
response = await call_get(
client,
f"/v2/projects/{active_project.external_id}/memory/recent",
params=params,
)
activity_data = GraphContext.model_validate(response.json())
# Extract entity results
results = []
for result in activity_data.results:
pr = result.primary_result
if pr.type == "entity":
results.append(
{
"title": pr.title,
"permalink": pr.permalink,
"file_path": pr.file_path,
"created_at": str(pr.created_at) if pr.created_at else None,
}
)
return results
@tool_app.command()
def write_note(
@@ -41,6 +291,10 @@ def write_note(
help="The project to write to. If not provided, the default project will be used."
),
] = None,
workspace: Annotated[
Optional[str],
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
] = None,
content: Annotated[
Optional[str],
typer.Option(
@@ -50,6 +304,11 @@ def write_note(
tags: Annotated[
Optional[List[str]], typer.Option(help="A list of tags to apply to the note")
] = None,
format: str = typer.Option("text", "--format", help="Output format: text or json"),
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Create or update a markdown note. Content can be provided as an argument or read from stdin.
@@ -57,6 +316,9 @@ def write_note(
1. Using the --content parameter
2. Piping content through stdin (if --content is not provided)
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
Examples:
# Using content parameter
@@ -77,8 +339,13 @@ def write_note(
# Reading from a file
cat document.md | basic-memory tools write-note --title "Document" --folder "docs"
# Force local routing in cloud mode
basic-memory tools write-note --title "My Note" --folder "notes" --content "..." --local
"""
try:
validate_routing_flags(local, cloud)
# If content is not provided, read from stdin
if content is None:
# Check if we're getting data from a pipe or redirect
@@ -109,8 +376,30 @@ def write_note(
# use the project name, or the default from the config
project_name = project_name or config_manager.default_project
note = run_with_cleanup(mcp_write_note.fn(title, content, folder, project_name, tags))
rprint(note)
# content is validated non-None above (stdin or --content)
assert content is not None
with force_routing(local=local, cloud=cloud):
if format == "json":
result = run_with_cleanup(
_write_note_json(title, content, folder, project_name, workspace, tags)
)
print(json.dumps(result, indent=2, ensure_ascii=True, default=str))
else:
note = run_with_cleanup(
mcp_write_note.fn(
title=title,
content=content,
directory=folder,
project=project_name,
workspace=workspace,
tags=tags,
)
)
rprint(note)
except ValueError as e:
typer.echo(f"Error: {e}", err=True)
raise typer.Exit(1)
except Exception as e: # pragma: no cover
if not isinstance(e, typer.Exit):
typer.echo(f"Error during write_note: {e}", err=True)
@@ -127,26 +416,73 @@ def read_note(
help="The project to use for the note. If not provided, the default project will be used."
),
] = None,
workspace: Annotated[
Optional[str],
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
] = None,
page: int = 1,
page_size: int = 10,
format: str = typer.Option("text", "--format", help="Output format: text or json"),
strip_frontmatter: bool = typer.Option(
False,
"--strip-frontmatter",
help=(
"Strip opening YAML frontmatter from content. "
"JSON output includes parsed frontmatter under 'frontmatter'."
),
),
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Read a markdown note from the knowledge base."""
# look for the project in the config
config_manager = ConfigManager()
project_name = None
if project is not None:
project_name, _ = config_manager.get_project(project)
if not project_name:
typer.echo(f"No project found named: {project}", err=True)
raise typer.Exit(1)
# use the project name, or the default from the config
project_name = project_name or config_manager.default_project
"""Read a markdown note from the knowledge base.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
Use --strip-frontmatter to return body-only markdown content.
"""
try:
note = run_with_cleanup(mcp_read_note.fn(identifier, project_name, page, page_size))
rprint(note)
validate_routing_flags(local, cloud)
# look for the project in the config
config_manager = ConfigManager()
project_name = None
if project is not None:
project_name, _ = config_manager.get_project(project)
if not project_name:
typer.echo(f"No project found named: {project}", err=True)
raise typer.Exit(1)
# use the project name, or the default from the config
project_name = project_name or config_manager.default_project
with force_routing(local=local, cloud=cloud):
if format == "json":
result = run_with_cleanup(
_read_note_json(identifier, project_name, workspace, page, page_size)
)
stripped_content, parsed_frontmatter = _parse_opening_frontmatter(result["content"])
result["frontmatter"] = parsed_frontmatter
if strip_frontmatter:
result["content"] = stripped_content
print(json.dumps(result, indent=2, ensure_ascii=True, default=str))
else:
note = run_with_cleanup(
mcp_read_note.fn(
identifier=identifier,
project=project_name,
workspace=workspace,
page=page,
page_size=page_size,
)
)
if strip_frontmatter:
note, _ = _parse_opening_frontmatter(note)
rprint(note)
except ValueError as e:
typer.echo(f"Error: {e}", err=True)
raise typer.Exit(1)
except Exception as e: # pragma: no cover
if not isinstance(e, typer.Exit):
typer.echo(f"Error during read_note: {e}", err=True)
@@ -154,6 +490,101 @@ def read_note(
raise
@tool_app.command()
def edit_note(
identifier: str,
operation: Annotated[str, typer.Option("--operation", help="Edit operation to apply")],
content: Annotated[str, typer.Option("--content", help="Content for the edit operation")],
project: Annotated[
Optional[str],
typer.Option(
help="The project to edit. If not provided, the default project will be used."
),
] = None,
workspace: Annotated[
Optional[str],
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
] = None,
find_text: Annotated[
Optional[str], typer.Option("--find-text", help="Text to find for find_replace operation")
] = None,
section: Annotated[
Optional[str],
typer.Option("--section", help="Section heading for replace_section operation"),
] = None,
expected_replacements: int = typer.Option(
1,
"--expected-replacements",
help="Expected replacement count for find_replace operation",
),
format: str = typer.Option("text", "--format", help="Output format: text or json"),
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Edit an existing markdown note using append/prepend/find_replace/replace_section.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
try:
validate_routing_flags(local, cloud)
_validate_edit_note_args(operation, find_text, section)
# look for the project in the config
config_manager = ConfigManager()
project_name = None
if project is not None:
project_name, _ = config_manager.get_project(project)
if not project_name:
typer.echo(f"No project found named: {project}", err=True)
raise typer.Exit(1)
# use the project name, or the default from the config
project_name = project_name or config_manager.default_project
with force_routing(local=local, cloud=cloud):
if format == "json":
result = run_with_cleanup(
_edit_note_json(
identifier=identifier,
operation=operation,
content=content,
project_name=project_name,
workspace=workspace,
section=section,
find_text=find_text,
expected_replacements=expected_replacements,
)
)
print(json.dumps(result, indent=2, ensure_ascii=True, default=str))
else:
result = run_with_cleanup(
mcp_edit_note.fn(
identifier=identifier,
operation=operation,
content=content,
project=project_name,
workspace=workspace,
section=section,
find_text=find_text,
expected_replacements=expected_replacements,
)
)
rprint(result)
if _is_edit_note_failure_response(result):
raise typer.Exit(1)
except ValueError as e:
typer.echo(f"Error: {e}", err=True)
raise typer.Exit(1)
except Exception as e: # pragma: no cover
if not isinstance(e, typer.Exit):
typer.echo(f"Error during edit_note: {e}", err=True)
raise typer.Exit(1)
raise
@tool_app.command()
def build_context(
url: MemoryUrl,
@@ -161,43 +592,62 @@ def build_context(
Optional[str],
typer.Option(help="The project to use. If not provided, the default project will be used."),
] = None,
workspace: Annotated[
Optional[str],
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
] = None,
depth: Optional[int] = 1,
timeframe: Optional[TimeFrame] = "7d",
page: int = 1,
page_size: int = 10,
max_related: int = 10,
format: str = typer.Option("json", "--format", help="Output format: text or json"),
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Get context needed to continue a discussion."""
# look for the project in the config
config_manager = ConfigManager()
project_name = None
if project is not None:
project_name, _ = config_manager.get_project(project)
if not project_name:
typer.echo(f"No project found named: {project}", err=True)
raise typer.Exit(1)
# use the project name, or the default from the config
project_name = project_name or config_manager.default_project
"""Get context needed to continue a discussion.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
try:
context = run_with_cleanup(
mcp_build_context.fn(
project=project_name,
url=url,
depth=depth,
timeframe=timeframe,
page=page,
page_size=page_size,
max_related=max_related,
)
)
# Use json module for more controlled serialization
import json
validate_routing_flags(local, cloud)
context_dict = context.model_dump(exclude_none=True)
print(json.dumps(context_dict, indent=2, ensure_ascii=True, default=str))
# look for the project in the config
config_manager = ConfigManager()
project_name = None
if project is not None:
project_name, _ = config_manager.get_project(project)
if not project_name:
typer.echo(f"No project found named: {project}", err=True)
raise typer.Exit(1)
# use the project name, or the default from the config
project_name = project_name or config_manager.default_project
with force_routing(local=local, cloud=cloud):
result = run_with_cleanup(
mcp_build_context.fn(
project=project_name,
workspace=workspace,
url=url,
depth=depth,
timeframe=timeframe,
page=page,
page_size=page_size,
max_related=max_related,
output_format="text" if format == "text" else "json",
)
)
if format == "json":
print(json.dumps(result, indent=2, ensure_ascii=True, default=str))
else:
print(result)
except ValueError as e:
typer.echo(f"Error: {e}", err=True)
raise typer.Exit(1)
except Exception as e: # pragma: no cover
if not isinstance(e, typer.Exit):
typer.echo(f"Error during build_context: {e}", err=True)
@@ -208,20 +658,73 @@ def build_context(
@tool_app.command()
def recent_activity(
type: Annotated[Optional[List[SearchItemType]], typer.Option()] = None,
project: Annotated[
Optional[str],
typer.Option(help="The project to use. If not provided, the default project will be used."),
] = None,
workspace: Annotated[
Optional[str],
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
] = None,
depth: Optional[int] = 1,
timeframe: Optional[TimeFrame] = "7d",
page: int = typer.Option(1, "--page", help="Page number for pagination (JSON format)"),
page_size: int = typer.Option(
50, "--page-size", help="Number of results per page (JSON format)"
),
format: str = typer.Option("text", "--format", help="Output format: text or json"),
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Get recent activity across the knowledge base."""
"""Get recent activity across the knowledge base.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
try:
result = run_with_cleanup(
mcp_recent_activity.fn(
type=type, # pyright: ignore [reportArgumentType]
depth=depth,
timeframe=timeframe,
)
)
# The tool now returns a formatted string directly
print(result)
validate_routing_flags(local, cloud)
# Resolve project from config for JSON mode
config_manager = ConfigManager()
project_name = None
if project is not None:
project_name, _ = config_manager.get_project(project)
if not project_name:
typer.echo(f"No project found named: {project}", err=True)
raise typer.Exit(1)
project_name = project_name or config_manager.default_project
with force_routing(local=local, cloud=cloud):
if format == "json":
result = run_with_cleanup(
_recent_activity_json(
type=type,
depth=depth,
timeframe=timeframe,
project_name=project_name,
workspace=workspace,
page=page,
page_size=page_size,
)
)
print(json.dumps(result, indent=2, ensure_ascii=True, default=str))
else:
result = run_with_cleanup(
mcp_recent_activity.fn(
type=type, # pyright: ignore [reportArgumentType]
depth=depth,
timeframe=timeframe,
project=project_name,
workspace=workspace,
)
)
# The tool returns a formatted string directly
print(result)
except ValueError as e:
typer.echo(f"Error: {e}", err=True)
raise typer.Exit(1)
except Exception as e: # pragma: no cover
if not isinstance(e, typer.Exit):
typer.echo(f"Error during recent_activity: {e}", err=True)
@@ -231,69 +734,158 @@ def recent_activity(
@tool_app.command("search-notes")
def search_notes(
query: str,
query: Annotated[
Optional[str],
typer.Argument(help="Search query string (optional when using metadata filters)"),
] = "",
permalink: Annotated[bool, typer.Option("--permalink", help="Search permalink values")] = False,
title: Annotated[bool, typer.Option("--title", help="Search title values")] = False,
vector: Annotated[bool, typer.Option("--vector", help="Use vector retrieval")] = False,
hybrid: Annotated[bool, typer.Option("--hybrid", help="Use hybrid retrieval")] = False,
project: Annotated[
Optional[str],
typer.Option(
help="The project to use for the note. If not provided, the default project will be used."
),
] = None,
workspace: Annotated[
Optional[str],
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
] = None,
after_date: Annotated[
Optional[str],
typer.Option("--after_date", help="Search results after date, eg. '2d', '1 week'"),
] = None,
tags: Annotated[
Optional[List[str]],
typer.Option("--tag", help="Filter by frontmatter tag (repeatable)"),
] = None,
status: Annotated[
Optional[str],
typer.Option("--status", help="Filter by frontmatter status"),
] = None,
note_types: Annotated[
Optional[List[str]],
typer.Option("--type", help="Filter by frontmatter type (repeatable)"),
] = None,
entity_types: Annotated[
Optional[List[str]],
typer.Option(
"--entity-type",
help="Filter by search item type: entity, observation, relation (repeatable)",
),
] = None,
meta: Annotated[
Optional[List[str]],
typer.Option("--meta", help="Filter by frontmatter key=value (repeatable)"),
] = None,
filter_json: Annotated[
Optional[str],
typer.Option("--filter", help="JSON metadata filter (advanced)"),
] = None,
page: int = 1,
page_size: int = 10,
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Search across all content in the knowledge base."""
# look for the project in the config
config_manager = ConfigManager()
project_name = None
if project is not None:
project_name, _ = config_manager.get_project(project)
if not project_name:
typer.echo(f"No project found named: {project}", err=True)
raise typer.Exit(1)
# use the project name, or the default from the config
project_name = project_name or config_manager.default_project
if permalink and title: # pragma: no cover
print("Cannot search both permalink and title")
raise typer.Abort()
"""Search across all content in the knowledge base.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
try:
if permalink and title: # pragma: no cover
validate_routing_flags(local, cloud)
# look for the project in the config
config_manager = ConfigManager()
project_name = None
if project is not None:
project_name, _ = config_manager.get_project(project)
if not project_name:
typer.echo(f"No project found named: {project}", err=True)
raise typer.Exit(1)
# use the project name, or the default from the config
project_name = project_name or config_manager.default_project
mode_flags = [permalink, title, vector, hybrid]
if sum(1 for enabled in mode_flags if enabled) > 1: # pragma: no cover
typer.echo(
"Use either --permalink or --title, not both. Exiting.",
"Use only one mode flag: --permalink, --title, --vector, or --hybrid. Exiting.",
err=True,
)
raise typer.Exit(1)
# set search type
search_type = ("permalink" if permalink else None,)
search_type = ("permalink_match" if permalink and "*" in query else None,)
search_type = ("title" if title else None,)
search_type = "text" if search_type is None else search_type
# Build metadata filters from --filter and --meta
metadata_filters = {}
if filter_json:
try:
metadata_filters = json.loads(filter_json)
if not isinstance(metadata_filters, dict):
raise ValueError("Metadata filter JSON must be an object")
except json.JSONDecodeError as e:
typer.echo(f"Invalid JSON for --filter: {e}", err=True)
raise typer.Exit(1)
results = run_with_cleanup(
mcp_search.fn(
query,
project_name,
search_type=search_type,
page=page,
after_date=after_date,
page_size=page_size,
if meta:
for item in meta:
if "=" not in item:
typer.echo(
f"Invalid --meta entry '{item}'. Use key=value format.",
err=True,
)
raise typer.Exit(1)
key, value = item.split("=", 1)
key = key.strip()
if not key:
typer.echo(f"Invalid --meta entry '{item}'.", err=True)
raise typer.Exit(1)
metadata_filters[key] = value
if not metadata_filters:
metadata_filters = None
# set search type
search_type = "text"
if permalink:
search_type = "permalink"
if query and "*" in query:
search_type = "permalink"
if title:
search_type = "title"
if vector:
search_type = "vector"
if hybrid:
search_type = "hybrid"
with force_routing(local=local, cloud=cloud):
results = run_with_cleanup(
mcp_search.fn(
query=query or "",
project=project_name,
workspace=workspace,
search_type=search_type,
page=page,
after_date=after_date,
page_size=page_size,
types=note_types,
entity_types=entity_types,
metadata_filters=metadata_filters,
tags=tags,
status=status,
)
)
)
# Use json module for more controlled serialization
import json
if isinstance(results, str):
print(results)
raise typer.Exit(1)
results_dict = results.model_dump(exclude_none=True)
print(json.dumps(results_dict, indent=2, ensure_ascii=True, default=str))
except ValueError as e:
typer.echo(f"Error: {e}", err=True)
raise typer.Exit(1)
except Exception as e: # pragma: no cover
if not isinstance(e, typer.Exit):
logger.exception("Error during search", e)
@@ -308,34 +900,31 @@ def continue_conversation(
timeframe: Annotated[
Optional[str], typer.Option(help="How far back to look for activity")
] = None,
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
):
"""Prompt to continue a previous conversation or work session."""
"""Prompt to continue a previous conversation or work session.
Use --local to force local routing when cloud mode is enabled.
Use --cloud to force cloud routing when cloud mode is disabled.
"""
try:
# Prompt functions return formatted strings directly
session = run_with_cleanup(mcp_continue_conversation.fn(topic=topic, timeframe=timeframe)) # type: ignore
validate_routing_flags(local, cloud)
with force_routing(local=local, cloud=cloud):
# Prompt functions return formatted strings directly
session = run_with_cleanup(
mcp_continue_conversation.fn(topic=topic, timeframe=timeframe) # type: ignore[arg-type]
)
rprint(session)
except ValueError as e:
typer.echo(f"Error: {e}", err=True)
raise typer.Exit(1)
except Exception as e: # pragma: no cover
if not isinstance(e, typer.Exit):
logger.exception("Error continuing conversation", e)
typer.echo(f"Error continuing conversation: {e}", err=True)
raise typer.Exit(1)
raise
# @tool_app.command(name="show-recent-activity")
# def show_recent_activity(
# timeframe: Annotated[
# str, typer.Option(help="How far back to look for activity")
# ] = "7d",
# ):
# """Prompt to show recent activity."""
# try:
# # Prompt functions return formatted strings directly
# session = asyncio.run(recent_activity_prompt(timeframe=timeframe))
# rprint(session)
# except Exception as e: # pragma: no cover
# if not isinstance(e, typer.Exit):
# logger.exception("Error continuing conversation", e)
# typer.echo(f"Error continuing conversation: {e}", err=True)
# raise typer.Exit(1)
# raise
+97
View File
@@ -0,0 +1,97 @@
"""Watch command - run file watcher as a standalone long-running process."""
import asyncio
import os
import signal
import sys
from typing import Optional
import typer
from loguru import logger
from basic_memory import db
from basic_memory.cli.app import app
from basic_memory.cli.container import get_container
from basic_memory.config import ConfigManager
from basic_memory.services.initialization import initialize_app
from basic_memory.sync.coordinator import SyncCoordinator
async def run_watch(project: Optional[str] = None) -> None:
"""Run the file watcher as a long-running process.
This is the async core of the watch command. It:
1. Initializes the app (DB migrations + project reconciliation)
2. Validates and sets project constraint if --project given
3. Creates a SyncCoordinator with quiet=False for Rich console output
4. Blocks until SIGINT/SIGTERM, then shuts down cleanly
"""
container = get_container()
config = container.config
# --- Initialization ---
# Wrapped in try/finally so DB resources are cleaned up on all exit paths,
# including early exits from invalid --project names.
await initialize_app(config)
sync_coordinator = None
try:
# --- Project constraint ---
if project:
config_manager = ConfigManager()
project_name, _ = config_manager.get_project(project)
if not project_name:
typer.echo(f"No project found named: {project}", err=True)
raise typer.Exit(1)
os.environ["BASIC_MEMORY_MCP_PROJECT"] = project_name
logger.info(f"Watch constrained to project: {project_name}")
# --- Sync coordinator ---
# quiet=False so file change events are printed to the terminal
sync_coordinator = SyncCoordinator(config=config, should_sync=True, quiet=False)
# --- Signal handling ---
shutdown_event = asyncio.Event()
def _signal_handler() -> None:
logger.info("Shutdown signal received")
shutdown_event.set()
loop = asyncio.get_running_loop()
# Windows ProactorEventLoop does not support add_signal_handler;
# fall back to the stdlib signal module which works cross-platform.
try:
for sig in (signal.SIGINT, signal.SIGTERM):
loop.add_signal_handler(sig, _signal_handler)
except NotImplementedError:
for sig in (signal.SIGINT, signal.SIGTERM):
signal.signal(sig, lambda _signum, _frame: _signal_handler())
# --- Run ---
await sync_coordinator.start()
logger.info("Watch service running, press Ctrl+C to stop")
await shutdown_event.wait()
finally:
if sync_coordinator is not None:
await sync_coordinator.stop()
await db.shutdown_db()
logger.info("Watch service stopped")
@app.command()
def watch(
project: Optional[str] = typer.Option(None, help="Restrict watcher to a single project"),
) -> None:
"""Run file watcher as a long-running process (no MCP server).
Watches for file changes in project directories and syncs them to the
database. Useful for running Basic Memory sync alongside external tools
that don't use the MCP server.
"""
# On Windows, use SelectorEventLoop to avoid ProactorEventLoop cleanup issues
if sys.platform == "win32": # pragma: no cover
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
asyncio.run(run_watch(project=project))
@@ -0,0 +1,57 @@
"""Workspace commands for Basic Memory cloud workspaces."""
import typer
from rich.console import Console
from rich.table import Table
from basic_memory.cli.app import app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.mcp.project_context import get_available_workspaces
console = Console()
workspace_app = typer.Typer(help="Manage cloud workspaces")
app.add_typer(workspace_app, name="workspace")
@workspace_app.command("list")
def list_workspaces() -> None:
"""List cloud workspaces available to the current OAuth session."""
async def _list():
return await get_available_workspaces()
try:
workspaces = run_with_cleanup(_list())
except RuntimeError as exc:
console.print(f"[red]Error: {exc}[/red]")
raise typer.Exit(1)
except Exception as exc: # pragma: no cover
console.print(f"[red]Error listing workspaces: {exc}[/red]")
raise typer.Exit(1)
if not workspaces:
console.print("[yellow]No accessible workspaces found.[/yellow]")
return
table = Table(title="Available Workspaces")
table.add_column("Name", style="cyan")
table.add_column("Type", style="blue")
table.add_column("Role", style="green")
table.add_column("Tenant ID", style="yellow")
for workspace in workspaces:
table.add_row(
workspace.name,
workspace.workspace_type,
workspace.role,
workspace.tenant_id,
)
console.print(table)
@app.command("workspaces")
def workspaces_alias() -> None:
"""Alias for `bm workspace list`."""
list_workspaces()
-1
View File
@@ -35,7 +35,6 @@ class CliContainer:
"""
config = ConfigManager().config
mode = resolve_runtime_mode(
cloud_mode_enabled=config.cloud_mode_enabled,
is_test_env=config.is_test_env,
)
return cls(config=config, mode=mode)
+27 -16
View File
@@ -1,24 +1,35 @@
"""Main CLI entry point for basic-memory.""" # pragma: no cover
import sys
import warnings
from basic_memory.cli.app import app # pragma: no cover
# Register commands
from basic_memory.cli.commands import ( # noqa: F401 # pragma: no cover
cloud,
db,
import_chatgpt,
import_claude_conversations,
import_claude_projects,
import_memory_json,
mcp,
project,
status,
tool,
)
# Re-apply warning filter AFTER all imports
# (authlib adds a DeprecationWarning filter that overrides ours)
import warnings # pragma: no cover
def _version_only_invocation(argv: list[str]) -> bool:
# Trigger: invocation is exactly `bm --version` or `bm -v`
# Why: avoid importing command modules on the hot version path
# Outcome: eager version callback exits quickly with minimal startup work
return len(argv) == 1 and argv[0] in {"--version", "-v"}
if not _version_only_invocation(sys.argv[1:]):
# Register commands only when not short-circuiting for --version
from basic_memory.cli.commands import ( # noqa: F401 # pragma: no cover
cloud,
db,
doctor,
import_chatgpt,
import_claude_conversations,
import_claude_projects,
import_memory_json,
mcp,
project,
schema,
status,
tool,
workspace,
)
warnings.filterwarnings("ignore") # pragma: no cover
+118
View File
@@ -0,0 +1,118 @@
"""Cloud promo messaging for CLI entrypoint."""
import os
import sys
from rich.console import Console
from rich.panel import Panel
import basic_memory
from basic_memory.config import ConfigManager
OSS_DISCOUNT_CODE = "BMFOSS"
CLOUD_LEARN_MORE_URL = "https://basicmemory.com"
def _promos_disabled_by_env() -> bool:
"""Check environment-level kill switch for promo output."""
value = os.getenv("BASIC_MEMORY_NO_PROMOS", "").strip().lower()
return value in {"1", "true", "yes"}
def _is_interactive_session() -> bool:
"""Return whether stdin/stdout are interactive terminals."""
return sys.stdin.isatty() and sys.stdout.isatty()
def _build_cloud_promo_message() -> str:
"""Build benefit-led cloud upsell copy with Rich markup."""
return (
"☁️ [bold]Your knowledge, everywhere.[/bold] ✨\n"
"Stop losing context when you switch machines.\n"
"Basic Memory Cloud syncs your memory across every device, including mobile and web.\n"
"Try it free for 7 days.\n"
f"Use [bold cyan]{OSS_DISCOUNT_CODE}[/bold cyan] for 20% off when you subscribe.\n"
"[bold green]→ bm cloud login[/bold green]"
)
def maybe_show_init_line(
invoked_subcommand: str | None,
*,
config_manager: ConfigManager | None = None,
is_interactive: bool | None = None,
console: Console | None = None,
) -> None:
"""Show a one-time init confirmation line before command output."""
manager = config_manager or ConfigManager()
config = manager.load_config()
interactive = _is_interactive_session() if is_interactive is None else is_interactive
# Same gates as the cloud promo — suppress in non-interactive, env kill-switch,
# mcp/root-help contexts, or when already shown.
if _promos_disabled_by_env() or not interactive:
return
if invoked_subcommand in {None, "mcp"}:
return
if config.cloud_promo_first_run_shown:
return
out = console or Console()
out.print("Basic Memory initialized ✓")
def maybe_show_cloud_promo(
invoked_subcommand: str | None,
*,
config_manager: ConfigManager | None = None,
is_interactive: bool | None = None,
console: Console | None = None,
) -> None:
"""Show cloud promo copy when discovery gates are satisfied."""
manager = config_manager or ConfigManager()
config = manager.load_config()
from basic_memory.cli.auth import CLIAuth
auth = CLIAuth(client_id=config.cloud_client_id, authkit_domain=config.cloud_domain)
has_cloud_access = bool(config.cloud_api_key) or auth.load_tokens() is not None
interactive = _is_interactive_session() if is_interactive is None else is_interactive
# Trigger: environment-level promo suppression or non-interactive execution.
# Why: avoid polluting scripts/CI output and support a hard opt-out.
# Outcome: skip all promo copy for this invocation.
if _promos_disabled_by_env() or not interactive:
return
# Trigger: command context where cloud promo is not actionable.
# Why: mcp/stdin protocol and root help flows should stay noise-free.
# Outcome: command continues without promo messaging.
if invoked_subcommand in {None, "mcp"}:
return
if has_cloud_access or config.cloud_promo_opt_out:
return
show_first_run = not config.cloud_promo_first_run_shown
show_version_notice = config.cloud_promo_last_version_shown != basic_memory.__version__
if not show_first_run and not show_version_notice:
return
out = console or Console()
out.print(
Panel(
_build_cloud_promo_message(),
title="Basic Memory Cloud",
border_style="cyan",
expand=False,
)
)
out.print(f"Learn more at [link={CLOUD_LEARN_MORE_URL}]{CLOUD_LEARN_MORE_URL}[/link]")
out.print("[dim]Disable with: bm cloud promo --off[/dim]")
config.cloud_promo_first_run_shown = True
config.cloud_promo_last_version_shown = basic_memory.__version__
manager.save_config(config)
+264 -48
View File
@@ -1,5 +1,6 @@
"""Configuration management for basic-memory."""
import importlib.util
import json
import os
from dataclasses import dataclass
@@ -24,6 +25,13 @@ WATCH_STATUS_JSON = "watch-status.json"
Environment = Literal["test", "dev", "user"]
class ProjectMode(str, Enum):
"""Per-project routing mode."""
LOCAL = "local"
CLOUD = "cloud"
class DatabaseBackend(str, Enum):
"""Supported database backends."""
@@ -31,12 +39,18 @@ class DatabaseBackend(str, Enum):
POSTGRES = "postgres"
def _default_semantic_search_enabled() -> bool:
"""Enable semantic search by default when semantic extras are installed."""
return importlib.util.find_spec("fastembed") is not None
@dataclass
class ProjectConfig:
"""Configuration for a specific basic-memory project."""
name: str
home: Path
mode: ProjectMode = ProjectMode.LOCAL
@property
def project(self):
@@ -52,6 +66,9 @@ class CloudProjectConfig(BaseModel):
This tracks the local working directory and sync state for a project
that is synced with Basic Memory Cloud.
DEPRECATED: Kept for backward-compatible migration only. New code should
use ProjectEntry fields (cloud_sync_path, bisync_initialized, last_sync).
"""
local_path: str = Field(description="Local working directory path for this cloud project")
@@ -63,26 +80,52 @@ class CloudProjectConfig(BaseModel):
)
class ProjectEntry(BaseModel):
"""Unified project configuration entry.
Replaces the old triple of projects (Dict[str, str]), project_modes
(Dict[str, ProjectMode]), and cloud_projects (Dict[str, CloudProjectConfig])
with a single structure per project.
"""
path: str = Field(description="Local filesystem path for the project")
mode: ProjectMode = Field(
default=ProjectMode.LOCAL,
description="Routing mode: local (in-process ASGI) or cloud (remote API)",
)
# Cloud sync state (replaces CloudProjectConfig)
cloud_sync_path: Optional[str] = Field(
default=None,
description="Local working directory for bisync (formerly CloudProjectConfig.local_path)",
)
bisync_initialized: bool = Field(
default=False,
description="Whether rclone bisync baseline has been established",
)
last_sync: Optional[datetime] = Field(
default=None,
description="Timestamp of last successful sync operation",
)
class BasicMemoryConfig(BaseSettings):
"""Pydantic model for Basic Memory global configuration."""
env: Environment = Field(default="dev", description="Environment name")
projects: Dict[str, str] = Field(
projects: Dict[str, ProjectEntry] = Field(
default_factory=lambda: {
"main": str(Path(os.getenv("BASIC_MEMORY_HOME", Path.home() / "basic-memory")))
"main": ProjectEntry(
path=str(Path(os.getenv("BASIC_MEMORY_HOME", Path.home() / "basic-memory")))
)
}
if os.getenv("BASIC_MEMORY_HOME")
else {},
description="Mapping of project names to their filesystem paths",
description="Mapping of project names to their ProjectEntry configuration",
)
default_project: str = Field(
default_project: Optional[str] = Field(
default="main",
description="Name of the default project to use",
)
default_project_mode: bool = Field(
default=False,
description="When True, MCP tools automatically use default_project when no project parameter is specified. Enables simplified UX for single-project workflows.",
description="Name of the default project to use. When set, acts as fallback when no project parameter is specified. Set to null to disable automatic project resolution.",
)
# overridden by ~/.basic-memory/config.json
@@ -99,6 +142,40 @@ class BasicMemoryConfig(BaseSettings):
description="Database connection URL. For Postgres, use postgresql+asyncpg://user:pass@host:port/db. If not set, SQLite will use default path.",
)
# Semantic search configuration
semantic_search_enabled: bool = Field(
default_factory=_default_semantic_search_enabled,
description="Enable semantic search (vector/hybrid retrieval). Works on both SQLite and Postgres backends. Requires semantic extras.",
)
semantic_embedding_provider: str = Field(
default="fastembed",
description="Embedding provider for local semantic indexing/search.",
)
semantic_embedding_model: str = Field(
default="bge-small-en-v1.5",
description="Embedding model identifier used by the local provider.",
)
semantic_embedding_dimensions: int | None = Field(
default=None,
description="Embedding vector dimensions. Auto-detected from provider if not set (384 for FastEmbed, 1536 for OpenAI).",
)
semantic_embedding_batch_size: int = Field(
default=64,
description="Batch size for embedding generation.",
gt=0,
)
semantic_vector_k: int = Field(
default=100,
description="Vector candidate count for vector and hybrid retrieval.",
gt=0,
)
semantic_min_similarity: float = Field(
default=0.55,
description="Minimum similarity score for vector search results. Results below this threshold are filtered out. 0.0 disables filtering.",
ge=0.0,
le=1.0,
)
# Database connection pool configuration (Postgres only)
db_pool_size: int = Field(
default=20,
@@ -160,6 +237,16 @@ class BasicMemoryConfig(BaseSettings):
description="Disable automatic permalink generation in frontmatter. When enabled, new notes won't have permalinks added and sync won't update permalinks. Existing permalinks will still work for reading.",
)
ensure_frontmatter_on_sync: bool = Field(
default=False,
description="Ensure markdown files have frontmatter during sync by adding derived title/type/permalink when missing. When combined with disable_permalinks=True, this setting takes precedence for missing-frontmatter files and still writes permalinks.",
)
permalinks_include_project: bool = Field(
default=True,
description="When True, generated permalinks are prefixed with the project slug (e.g., 'specs/search'). Existing permalinks remain unchanged unless explicitly updated.",
)
skip_initialization_sync: bool = Field(
default=False,
description="Skip expensive initialization synchronization. Useful for cloud/stateless deployments where project reconciliation is not needed.",
@@ -211,16 +298,99 @@ class BasicMemoryConfig(BaseSettings):
description="Basic Memory Cloud host URL",
)
cloud_mode: bool = Field(
cloud_promo_opt_out: bool = Field(
default=False,
description="Enable cloud mode - all requests go to cloud instead of local (config file value)",
description="Disable CLI cloud promo messages when true.",
)
cloud_projects: Dict[str, CloudProjectConfig] = Field(
default_factory=dict,
description="Cloud project sync configuration mapping project names to their local paths and sync state",
cloud_promo_first_run_shown: bool = Field(
default=False,
description="Tracks whether the first-run cloud promo message has been shown.",
)
cloud_promo_last_version_shown: Optional[str] = Field(
default=None,
description="Most recent cloud promo version shown in CLI.",
)
cloud_api_key: Optional[str] = Field(
default=None,
description="API key for cloud access (bmc_ prefixed). Account-level, not per-project.",
)
@model_validator(mode="before")
@classmethod
def migrate_legacy_projects(cls, data: Any) -> Any:
"""Migrate old-format config (Dict[str, str]) to new ProjectEntry format.
Old format stored projects as three separate dicts:
projects: {"name": "/path"}
project_modes: {"name": "cloud"}
cloud_projects: {"name": {"local_path": "...", ...}}
New format unifies them into:
projects: {"name": {"path": "/path", "mode": "cloud", ...}}
Also removes stale keys (default_project_mode, permalinks_include_project)
that are no longer part of the config model.
"""
if not isinstance(data, dict):
return data
# --- Remove stale keys from old config versions ---
data.pop("default_project_mode", None)
data.pop("cloud_mode", None)
projects = data.get("projects", {})
if not projects:
return data
# Check if already in new format — peek at first value
first_value = next(iter(projects.values()), None)
if isinstance(first_value, str):
# Old format: {"name": "/path"} → convert
project_modes = data.pop("project_modes", {})
cloud_projects = data.pop("cloud_projects", {})
new_projects: Dict[str, Any] = {}
for name, path in projects.items():
entry: Dict[str, Any] = {"path": path}
if name in project_modes:
entry["mode"] = project_modes[name]
if name in cloud_projects:
cp = cloud_projects[name]
if isinstance(cp, dict):
entry["cloud_sync_path"] = cp.get("local_path")
entry["bisync_initialized"] = cp.get("bisync_initialized", False)
entry["last_sync"] = cp.get("last_sync")
else:
# Already a CloudProjectConfig-like object
entry["cloud_sync_path"] = getattr(cp, "local_path", None)
entry["bisync_initialized"] = getattr(cp, "bisync_initialized", False)
entry["last_sync"] = getattr(cp, "last_sync", None)
new_projects[name] = entry
# Pick up cloud_projects entries not already in projects
# These are cloud-only projects — path is the cloud permalink,
# local_path goes into cloud_sync_path for bisync
for name, cp in cloud_projects.items():
if name not in new_projects:
if isinstance(cp, dict):
new_projects[name] = {
"path": generate_permalink(name),
"mode": project_modes.get(name, "cloud"),
"cloud_sync_path": cp.get("local_path"),
"bisync_initialized": cp.get("bisync_initialized", False),
"last_sync": cp.get("last_sync"),
}
data["projects"] = new_projects
else:
# New format or dict-based — just clean up stale keys
data.pop("project_modes", None)
data.pop("cloud_projects", None)
return data
@property
def is_test_env(self) -> bool:
"""Check if running in a test environment.
@@ -238,27 +408,31 @@ class BasicMemoryConfig(BaseSettings):
or os.getenv("PYTEST_CURRENT_TEST") is not None
)
@property
def cloud_mode_enabled(self) -> bool:
"""Check if cloud mode is enabled.
def get_project_mode(self, project_name: str) -> ProjectMode:
"""Get the routing mode for a project.
Priority:
1. BASIC_MEMORY_CLOUD_MODE environment variable
2. Config file value (cloud_mode)
Returns the per-project mode if set, otherwise LOCAL.
"""
env_value = os.environ.get("BASIC_MEMORY_CLOUD_MODE", "").lower()
if env_value in ("true", "1", "yes"):
return True
elif env_value in ("false", "0", "no"):
return False
# Fall back to config file value
return self.cloud_mode
entry = self.projects.get(project_name)
return entry.mode if entry else ProjectMode.LOCAL
def set_project_mode(self, project_name: str, mode: ProjectMode) -> None:
"""Set the routing mode for a project.
Creates a minimal ProjectEntry if the project doesn't already exist,
preserving backward compatibility with code that sets mode before
adding a full project entry.
"""
if project_name in self.projects:
self.projects[project_name].mode = mode
else:
self.projects[project_name] = ProjectEntry(path="", mode=mode)
@classmethod
def for_cloud_tenant(
cls,
database_url: str,
projects: Optional[Dict[str, str]] = None,
projects: Optional[Dict[str, "ProjectEntry"]] = None,
) -> "BasicMemoryConfig":
"""Create config for cloud tenant - no config.json, database is source of truth.
@@ -280,7 +454,6 @@ class BasicMemoryConfig(BaseSettings):
database_backend=DatabaseBackend.POSTGRES,
database_url=database_url,
projects=projects or {},
cloud_mode=True,
skip_initialization_sync=True,
)
@@ -296,7 +469,7 @@ class BasicMemoryConfig(BaseSettings):
if name not in self.projects:
raise ValueError(f"Project '{name}' not found in configuration")
return Path(self.projects[name])
return Path(self.projects[name].path)
def model_post_init(self, __context: Any) -> None:
"""Ensure configuration is valid after initialization."""
@@ -306,12 +479,15 @@ class BasicMemoryConfig(BaseSettings):
# Ensure at least one project exists; if none exist then create main
if not self.projects: # pragma: no cover
self.projects["main"] = str(
Path(os.getenv("BASIC_MEMORY_HOME", Path.home() / "basic-memory"))
self.projects["main"] = ProjectEntry(
path=str(Path(os.getenv("BASIC_MEMORY_HOME", Path.home() / "basic-memory")))
)
# Ensure default project is valid (i.e. points to an existing project)
if self.default_project not in self.projects: # pragma: no cover
# None means "no default" — intentionally left unset
if (
self.default_project is not None and self.default_project not in self.projects
): # pragma: no cover
# Set default to first available project
self.default_project = next(iter(self.projects.keys()))
@@ -321,8 +497,11 @@ class BasicMemoryConfig(BaseSettings):
This is the single database that will store all knowledge data
across all projects.
Uses BASIC_MEMORY_CONFIG_DIR when set so each process/worktree can
isolate both config and database state.
"""
database_path = Path.home() / DATA_DIR_NAME / APP_DATABASE_NAME
database_path = self.data_dir_path / APP_DATABASE_NAME
if not database_path.exists(): # pragma: no cover
database_path.parent.mkdir(parents=True, exist_ok=True)
database_path.touch()
@@ -344,7 +523,10 @@ class BasicMemoryConfig(BaseSettings):
@property
def project_list(self) -> List[ProjectConfig]: # pragma: no cover
"""Get all configured projects as ProjectConfig objects."""
return [ProjectConfig(name=name, home=Path(path)) for name, path in self.projects.items()]
return [
ProjectConfig(name=name, home=Path(entry.path), mode=entry.mode)
for name, entry in self.projects.items()
]
@model_validator(mode="after")
def ensure_project_paths_exists(self) -> "BasicMemoryConfig": # pragma: no cover
@@ -357,8 +539,8 @@ class BasicMemoryConfig(BaseSettings):
if self.database_backend == DatabaseBackend.POSTGRES:
return self
for name, path_value in self.projects.items():
path = Path(path_value)
for name, entry in self.projects.items():
path = Path(entry.path)
if not path.exists():
try:
path.mkdir(parents=True)
@@ -368,8 +550,13 @@ class BasicMemoryConfig(BaseSettings):
return self
@property
def data_dir_path(self):
return Path.home() / DATA_DIR_NAME
def data_dir_path(self) -> Path:
"""Get app state directory for config and default SQLite database."""
if config_dir := os.getenv("BASIC_MEMORY_CONFIG_DIR"):
return Path(config_dir)
home = os.getenv("HOME", Path.home())
return Path(home) / DATA_DIR_NAME
# Module-level cache for configuration
@@ -419,6 +606,22 @@ class ConfigManager:
try:
file_data = json.loads(self.config_file.read_text(encoding="utf-8"))
# Detect legacy format before model validators strip stale keys
_STALE_KEYS = {
"default_project_mode",
"project_modes",
"cloud_projects",
"cloud_mode",
}
needs_resave = bool(_STALE_KEYS & file_data.keys())
# Check if projects dict uses old string-value format
projects_raw = file_data.get("projects", {})
if projects_raw:
first_val = next(iter(projects_raw.values()), None)
if isinstance(first_val, str):
needs_resave = True
# First, create config from environment variables (Pydantic will read them)
# Then overlay with file data for fields that aren't set via env vars
# This ensures env vars take precedence
@@ -440,6 +643,12 @@ class ConfigManager:
merged_data[field_name] = env_dict[field_name]
_CONFIG_CACHE = BasicMemoryConfig(**merged_data)
# Re-save to normalize legacy config into current format
if needs_resave:
logger.info("Migrating config to current format")
save_basic_memory_config(self.config_file, _CONFIG_CACHE)
return _CONFIG_CACHE
except Exception as e: # pragma: no cover
logger.exception(f"Failed to load config: {e}")
@@ -458,11 +667,15 @@ class ConfigManager:
@property
def projects(self) -> Dict[str, str]:
"""Get all configured projects."""
return self.config.projects.copy()
"""Get all configured projects as name -> path mapping.
Returns the legacy Dict[str, str] format for backward compatibility
with code that expects project name -> filesystem path.
"""
return {name: entry.path for name, entry in self.config.projects.items()}
@property
def default_project(self) -> str:
def default_project(self) -> Optional[str]:
"""Get the default project name."""
return self.config.default_project
@@ -478,7 +691,7 @@ class ConfigManager:
# Load config, modify it, and save it
config = self.load_config()
config.projects[name] = str(project_path)
config.projects[name] = ProjectEntry(path=str(project_path))
self.save_config(config)
return ProjectConfig(name=name, home=project_path)
@@ -510,12 +723,15 @@ class ConfigManager:
self.save_config(config)
def get_project(self, name: str) -> Tuple[str, str] | Tuple[None, None]:
"""Look up a project from the configuration by name or permalink"""
"""Look up a project from the configuration by name or permalink.
Returns (project_name, path_string) for backward compatibility.
"""
project_permalink = generate_permalink(name)
app_config = self.config
for project_name, path in app_config.projects.items():
for project_name, entry in app_config.projects.items():
if project_permalink == generate_permalink(project_name):
return project_name, path
return project_name, entry.path
return None, None
@@ -550,9 +766,9 @@ def get_project_config(project_name: Optional[str] = None) -> ProjectConfig:
project_permalink = generate_permalink(actual_project_name)
for name, path in app_config.projects.items():
for name, entry in app_config.projects.items():
if project_permalink == generate_permalink(name):
return ProjectConfig(name=name, home=Path(path))
return ProjectConfig(name=name, home=Path(entry.path))
# otherwise raise error
raise ValueError(f"Project '{actual_project_name}' not found") # pragma: no cover
+16 -7
View File
@@ -344,24 +344,33 @@ async def engine_session_factory(
global _engine, _session_maker
# Use the same helper function as production code
_engine, _session_maker = _create_engine_and_session(db_path, db_type, config)
# Use the same helper function as production code.
#
# Keep local references so teardown can deterministically dispose the
# specific engine created by this context manager, even if other code calls
# shutdown_db() and mutates module-level globals mid-test.
created_engine, created_session_maker = _create_engine_and_session(db_path, db_type, config)
_engine, _session_maker = created_engine, created_session_maker
try:
# Verify that engine and session maker are initialized
if _engine is None: # pragma: no cover
if created_engine is None: # pragma: no cover
logger.error("Database engine is None in engine_session_factory")
raise RuntimeError("Database engine initialization failed")
if _session_maker is None: # pragma: no cover
if created_session_maker is None: # pragma: no cover
logger.error("Session maker is None in engine_session_factory")
raise RuntimeError("Session maker initialization failed")
yield _engine, _session_maker
yield created_engine, created_session_maker
finally:
if _engine:
await _engine.dispose()
await created_engine.dispose()
# Only clear module-level globals if they still point to this context's
# engine/session. This avoids clobbering newer globals from other callers.
if _engine is created_engine:
_engine = None
if _session_maker is created_session_maker:
_session_maker = None
+4
View File
@@ -91,6 +91,8 @@ from basic_memory.deps.services import (
FileServiceV2Dep,
get_file_service_v2_external,
FileServiceV2ExternalDep,
get_task_scheduler,
TaskSchedulerDep,
get_search_service,
SearchServiceDep,
get_search_service_v2,
@@ -227,6 +229,8 @@ __all__ = [
"FileServiceV2Dep",
"get_file_service_v2_external",
"FileServiceV2ExternalDep",
"get_task_scheduler",
"TaskSchedulerDep",
"get_search_service",
"SearchServiceDep",
"get_search_service_v2",
+72 -12
View File
@@ -41,7 +41,12 @@ async def get_chatgpt_importer(
file_service: FileServiceDep,
) -> ChatGPTImporter:
"""Create ChatGPTImporter with dependencies."""
return ChatGPTImporter(project_config.home, markdown_processor, file_service)
return ChatGPTImporter(
project_config.home,
markdown_processor,
file_service,
project_name=project_config.name,
)
ChatGPTImporterDep = Annotated[ChatGPTImporter, Depends(get_chatgpt_importer)]
@@ -53,7 +58,12 @@ async def get_chatgpt_importer_v2( # pragma: no cover
file_service: FileServiceV2Dep,
) -> ChatGPTImporter:
"""Create ChatGPTImporter with v2 dependencies."""
return ChatGPTImporter(project_config.home, markdown_processor, file_service)
return ChatGPTImporter(
project_config.home,
markdown_processor,
file_service,
project_name=project_config.name,
)
ChatGPTImporterV2Dep = Annotated[ChatGPTImporter, Depends(get_chatgpt_importer_v2)]
@@ -65,7 +75,12 @@ async def get_chatgpt_importer_v2_external(
file_service: FileServiceV2ExternalDep,
) -> ChatGPTImporter:
"""Create ChatGPTImporter with v2 external_id dependencies."""
return ChatGPTImporter(project_config.home, markdown_processor, file_service)
return ChatGPTImporter(
project_config.home,
markdown_processor,
file_service,
project_name=project_config.name,
)
ChatGPTImporterV2ExternalDep = Annotated[ChatGPTImporter, Depends(get_chatgpt_importer_v2_external)]
@@ -80,7 +95,12 @@ async def get_claude_conversations_importer(
file_service: FileServiceDep,
) -> ClaudeConversationsImporter:
"""Create ClaudeConversationsImporter with dependencies."""
return ClaudeConversationsImporter(project_config.home, markdown_processor, file_service)
return ClaudeConversationsImporter(
project_config.home,
markdown_processor,
file_service,
project_name=project_config.name,
)
ClaudeConversationsImporterDep = Annotated[
@@ -94,7 +114,12 @@ async def get_claude_conversations_importer_v2( # pragma: no cover
file_service: FileServiceV2Dep,
) -> ClaudeConversationsImporter:
"""Create ClaudeConversationsImporter with v2 dependencies."""
return ClaudeConversationsImporter(project_config.home, markdown_processor, file_service)
return ClaudeConversationsImporter(
project_config.home,
markdown_processor,
file_service,
project_name=project_config.name,
)
ClaudeConversationsImporterV2Dep = Annotated[
@@ -108,7 +133,12 @@ async def get_claude_conversations_importer_v2_external(
file_service: FileServiceV2ExternalDep,
) -> ClaudeConversationsImporter:
"""Create ClaudeConversationsImporter with v2 external_id dependencies."""
return ClaudeConversationsImporter(project_config.home, markdown_processor, file_service)
return ClaudeConversationsImporter(
project_config.home,
markdown_processor,
file_service,
project_name=project_config.name,
)
ClaudeConversationsImporterV2ExternalDep = Annotated[
@@ -125,7 +155,12 @@ async def get_claude_projects_importer(
file_service: FileServiceDep,
) -> ClaudeProjectsImporter:
"""Create ClaudeProjectsImporter with dependencies."""
return ClaudeProjectsImporter(project_config.home, markdown_processor, file_service)
return ClaudeProjectsImporter(
project_config.home,
markdown_processor,
file_service,
project_name=project_config.name,
)
ClaudeProjectsImporterDep = Annotated[ClaudeProjectsImporter, Depends(get_claude_projects_importer)]
@@ -137,7 +172,12 @@ async def get_claude_projects_importer_v2( # pragma: no cover
file_service: FileServiceV2Dep,
) -> ClaudeProjectsImporter:
"""Create ClaudeProjectsImporter with v2 dependencies."""
return ClaudeProjectsImporter(project_config.home, markdown_processor, file_service)
return ClaudeProjectsImporter(
project_config.home,
markdown_processor,
file_service,
project_name=project_config.name,
)
ClaudeProjectsImporterV2Dep = Annotated[
@@ -151,7 +191,12 @@ async def get_claude_projects_importer_v2_external(
file_service: FileServiceV2ExternalDep,
) -> ClaudeProjectsImporter:
"""Create ClaudeProjectsImporter with v2 external_id dependencies."""
return ClaudeProjectsImporter(project_config.home, markdown_processor, file_service)
return ClaudeProjectsImporter(
project_config.home,
markdown_processor,
file_service,
project_name=project_config.name,
)
ClaudeProjectsImporterV2ExternalDep = Annotated[
@@ -168,7 +213,12 @@ async def get_memory_json_importer(
file_service: FileServiceDep,
) -> MemoryJsonImporter:
"""Create MemoryJsonImporter with dependencies."""
return MemoryJsonImporter(project_config.home, markdown_processor, file_service)
return MemoryJsonImporter(
project_config.home,
markdown_processor,
file_service,
project_name=project_config.name,
)
MemoryJsonImporterDep = Annotated[MemoryJsonImporter, Depends(get_memory_json_importer)]
@@ -180,7 +230,12 @@ async def get_memory_json_importer_v2( # pragma: no cover
file_service: FileServiceV2Dep,
) -> MemoryJsonImporter:
"""Create MemoryJsonImporter with v2 dependencies."""
return MemoryJsonImporter(project_config.home, markdown_processor, file_service)
return MemoryJsonImporter(
project_config.home,
markdown_processor,
file_service,
project_name=project_config.name,
)
MemoryJsonImporterV2Dep = Annotated[MemoryJsonImporter, Depends(get_memory_json_importer_v2)]
@@ -192,7 +247,12 @@ async def get_memory_json_importer_v2_external(
file_service: FileServiceV2ExternalDep,
) -> MemoryJsonImporter:
"""Create MemoryJsonImporter with v2 external_id dependencies."""
return MemoryJsonImporter(project_config.home, markdown_processor, file_service)
return MemoryJsonImporter(
project_config.home,
markdown_processor,
file_service,
project_name=project_config.name,
)
MemoryJsonImporterV2ExternalDep = Annotated[
+7 -3
View File
@@ -13,6 +13,7 @@ from typing import Annotated
from fastapi import Depends
from basic_memory.deps.config import AppConfigDep
from basic_memory.deps.db import SessionMakerDep
from basic_memory.deps.projects import (
ProjectIdDep,
@@ -147,13 +148,14 @@ RelationRepositoryV2ExternalDep = Annotated[
async def get_search_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
app_config: AppConfigDep,
) -> SearchRepository:
"""Create a backend-specific SearchRepository instance for the current project.
Uses factory function to return SQLiteSearchRepository or PostgresSearchRepository
based on database backend configuration.
"""
return create_search_repository(session_maker, project_id=project_id)
return create_search_repository(session_maker, project_id=project_id, app_config=app_config)
SearchRepositoryDep = Annotated[SearchRepository, Depends(get_search_repository)]
@@ -162,9 +164,10 @@ SearchRepositoryDep = Annotated[SearchRepository, Depends(get_search_repository)
async def get_search_repository_v2( # pragma: no cover
session_maker: SessionMakerDep,
project_id: ProjectIdPathDep,
app_config: AppConfigDep,
) -> SearchRepository:
"""Create a SearchRepository instance for v2 API."""
return create_search_repository(session_maker, project_id=project_id)
return create_search_repository(session_maker, project_id=project_id, app_config=app_config)
SearchRepositoryV2Dep = Annotated[SearchRepository, Depends(get_search_repository_v2)]
@@ -173,9 +176,10 @@ SearchRepositoryV2Dep = Annotated[SearchRepository, Depends(get_search_repositor
async def get_search_repository_v2_external(
session_maker: SessionMakerDep,
project_id: ProjectExternalIdPathDep,
app_config: AppConfigDep,
) -> SearchRepository:
"""Create a SearchRepository instance for v2 API (uses external_id)."""
return create_search_repository(session_maker, project_id=project_id)
return create_search_repository(session_maker, project_id=project_id, app_config=app_config)
SearchRepositoryV2ExternalDep = Annotated[
+116 -2
View File
@@ -7,7 +7,9 @@ This module provides service-layer dependencies:
- SyncService, ProjectService, DirectoryService
"""
from typing import Annotated
import asyncio
import os
from typing import Annotated, Any, Callable, Coroutine, Mapping, Protocol
from fastapi import Depends
from loguru import logger
@@ -43,7 +45,6 @@ from basic_memory.services.link_resolver import LinkResolver
from basic_memory.services.search_service import SearchService
from basic_memory.sync import SyncService
# --- Entity Parser ---
@@ -430,6 +431,119 @@ async def get_sync_service_v2_external(
SyncServiceV2ExternalDep = Annotated[SyncService, Depends(get_sync_service_v2_external)]
# --- Background Task Scheduler ---
class TaskScheduler(Protocol):
def schedule(self, task_name: str, **payload: Any) -> None:
"""Schedule a background task by name."""
def _log_task_failure(completed: asyncio.Task) -> None:
try:
completed.result()
except Exception as exc: # pragma: no cover
logger.exception("Background task failed", error=str(exc))
class LocalTaskScheduler:
"""Default scheduler that runs tasks in-process via asyncio.create_task.
In test mode (BASIC_MEMORY_ENV=test), tasks run as no-ops to avoid
background asyncio tasks racing against test teardown and causing
SQLite 'cannot commit transaction' errors.
"""
def __init__(
self,
handlers: Mapping[str, Callable[..., Coroutine[Any, Any, None]]],
test_mode: bool | None = None,
) -> None:
self._handlers = handlers
self._test_mode = (
test_mode if test_mode is not None else os.environ.get("BASIC_MEMORY_ENV") == "test"
)
def schedule(self, task_name: str, **payload: Any) -> None:
handler = self._handlers.get(task_name)
# Trigger: task name is not registered
# Why: avoid silently dropping background work
# Outcome: fail fast to surface misconfiguration
if not handler:
raise ValueError(f"Unknown task name: {task_name}")
# Trigger: running inside pytest (BASIC_MEMORY_ENV=test)
# Why: background create_task() outlives test fixtures and races
# against engine disposal, causing flaky SQLite errors
# Outcome: skip background scheduling; tests exercise the sync
# codepaths directly when they need to
if self._test_mode:
return
task = asyncio.create_task(handler(**payload))
task.add_done_callback(_log_task_failure)
async def get_task_scheduler(
entity_service: EntityServiceV2ExternalDep,
sync_service: SyncServiceV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
project_config: ProjectConfigV2ExternalDep,
app_config: AppConfigDep,
) -> TaskScheduler:
"""Create a scheduler that maps task specs to coroutines."""
scheduler: LocalTaskScheduler | None = None
async def _reindex_entity(
entity_id: int,
resolve_relations: bool = False,
**_: Any,
) -> None:
await entity_service.reindex_entity(entity_id)
# Trigger: caller requests relation resolution
# Why: resolve forward references created before the entity existed
# Outcome: updates unresolved relations pointing to this entity
if resolve_relations:
await sync_service.resolve_relations(entity_id=entity_id)
# Trigger: semantic search enabled in local config.
# Why: vector chunks are derived and should refresh after canonical reindex completes.
# Outcome: schedules out-of-band vector sync without extending write latency.
if app_config.semantic_search_enabled and scheduler is not None:
scheduler.schedule("sync_entity_vectors", entity_id=entity_id)
async def _resolve_relations(entity_id: int, **_: Any) -> None:
await sync_service.resolve_relations(entity_id=entity_id)
async def _sync_entity_vectors(entity_id: int, **_: Any) -> None:
await search_service.sync_entity_vectors(entity_id)
async def _sync_project(force_full: bool = False, **_: Any) -> None:
await sync_service.sync(
project_config.home,
project_config.name,
force_full=force_full,
)
async def _reindex_project(**_: Any) -> None:
await search_service.reindex_all()
scheduler = LocalTaskScheduler(
{
"reindex_entity": _reindex_entity,
"resolve_relations": _resolve_relations,
"sync_entity_vectors": _sync_entity_vectors,
"sync_project": _sync_project,
"reindex_project": _reindex_project,
},
test_mode=app_config.is_test_env,
)
return scheduler
TaskSchedulerDep = Annotated[TaskScheduler, Depends(get_task_scheduler)]
# --- Project Service ---
+4 -4
View File
@@ -450,16 +450,16 @@ def sanitize_for_filename(text: str, replacement: str = "-") -> str:
return text.strip(replacement)
def sanitize_for_folder(folder: str) -> str:
def sanitize_for_directory(directory: str) -> str:
"""
Sanitize folder path to be safe for use in file system paths.
Sanitize directory path to be safe for use in file system paths.
Removes leading/trailing whitespace, compresses multiple slashes,
and removes special characters except for /, -, and _.
"""
if not folder:
if not directory:
return ""
sanitized = folder.strip()
sanitized = directory.strip()
if sanitized.startswith("./"):
sanitized = sanitized[2:]
+24
View File
@@ -8,6 +8,7 @@ from typing import TYPE_CHECKING, Any, Optional, TypeVar
from basic_memory.markdown.markdown_processor import MarkdownProcessor
from basic_memory.markdown.schemas import EntityMarkdown
from basic_memory.schemas.importer import ImportResult
from basic_memory.utils import build_canonical_permalink, generate_permalink
if TYPE_CHECKING: # pragma: no cover
from basic_memory.services.file_service import FileService
@@ -29,6 +30,7 @@ class Importer[T: ImportResult]:
base_path: Path,
markdown_processor: MarkdownProcessor,
file_service: "FileService",
project_name: Optional[str] = None,
):
"""Initialize the import service.
@@ -40,6 +42,8 @@ class Importer[T: ImportResult]:
self.base_path = base_path.resolve() # Get absolute path
self.markdown_processor = markdown_processor
self.file_service = file_service
self.project_name = project_name
self.project_permalink = generate_permalink(project_name) if project_name else None
@abstractmethod
async def import_data(self, source_data, destination_folder: str, **kwargs: Any) -> T:
@@ -73,6 +77,26 @@ class Importer[T: ImportResult]:
# FileService.write_file handles directory creation and returns checksum
return await self.file_service.write_file(file_path, content)
def canonical_permalink(self, path: str) -> str:
"""Build a canonical permalink for imported content."""
include_project = True
# Trigger: importer has app config with permalink prefixing flag
# Why: imported notes should align with canonical permalink format
# Outcome: include project prefix when enabled
if self.file_service.app_config is not None:
include_project = self.file_service.app_config.permalinks_include_project
return build_canonical_permalink(
self.project_permalink,
path,
include_project=include_project,
)
def build_import_paths(self, path: str) -> tuple[str, str]:
"""Return (permalink, file_path) for an imported entity."""
permalink = self.canonical_permalink(path)
return permalink, f"{path}.md"
async def ensure_folder_exists(self, folder: str) -> None:
"""Ensure folder exists using FileService.
+13 -8
View File
@@ -51,11 +51,20 @@ class ChatGPTImporter(Importer[ChatImportResult]):
chats_imported = 0
for chat in conversations:
created_at = chat["create_time"]
date_prefix = datetime.fromtimestamp(created_at).astimezone().strftime("%Y%m%d")
clean_title = clean_filename(chat["title"])
relative_path = (
f"{destination_folder}/{date_prefix}-{clean_title}"
if destination_folder
else f"{date_prefix}-{clean_title}"
)
permalink, file_path = self.build_import_paths(relative_path)
# Convert to entity
entity = self._format_chat_content(destination_folder, chat)
entity = self._format_chat_content(chat, permalink)
# Write file using relative path - FileService handles base_path
file_path = f"{entity.frontmatter.metadata['permalink']}.md"
await self.write_entity(entity, file_path)
# Count messages
@@ -83,7 +92,7 @@ class ChatGPTImporter(Importer[ChatImportResult]):
return self.handle_error("Failed to import ChatGPT conversations", e)
def _format_chat_content(
self, folder: str, conversation: Dict[str, Any]
self, conversation: Dict[str, Any], permalink: str
) -> EntityMarkdown: # pragma: no cover
"""Convert chat conversation to Basic Memory entity.
@@ -105,10 +114,6 @@ class ChatGPTImporter(Importer[ChatImportResult]):
root_id = node_id
break
# Generate permalink
date_prefix = datetime.fromtimestamp(created_at).astimezone().strftime("%Y%m%d")
clean_title = clean_filename(conversation["title"])
# Format content
content = self._format_chat_markdown(
title=conversation["title"],
@@ -126,7 +131,7 @@ class ChatGPTImporter(Importer[ChatImportResult]):
"title": conversation["title"],
"created": format_timestamp(created_at),
"modified": format_timestamp(modified_at),
"permalink": f"{folder}/{date_prefix}-{clean_title}",
"permalink": permalink,
}
),
content=content,
@@ -54,18 +54,27 @@ class ClaudeConversationsImporter(Importer[ChatImportResult]):
for chat in conversations:
# Get name, providing default for unnamed conversations
chat_name = chat.get("name") or f"Conversation {chat.get('uuid', 'untitled')}"
date_prefix = datetime.fromisoformat(
chat["created_at"].replace("Z", "+00:00")
).strftime("%Y%m%d")
clean_title = clean_filename(chat_name)
relative_path = (
f"{destination_folder}/{date_prefix}-{clean_title}"
if destination_folder
else f"{date_prefix}-{clean_title}"
)
permalink, file_path = self.build_import_paths(relative_path)
# Convert to entity
entity = self._format_chat_content(
folder=destination_folder,
name=chat_name,
messages=chat["chat_messages"],
created_at=chat["created_at"],
modified_at=chat["updated_at"],
permalink=permalink,
)
# Write file using relative path - FileService handles base_path
file_path = f"{entity.frontmatter.metadata['permalink']}.md"
await self.write_entity(entity, file_path)
chats_imported += 1
@@ -84,11 +93,11 @@ class ClaudeConversationsImporter(Importer[ChatImportResult]):
def _format_chat_content(
self,
folder: str,
name: str,
messages: List[Dict[str, Any]],
created_at: str,
modified_at: str,
permalink: str,
) -> EntityMarkdown:
"""Convert chat messages to Basic Memory entity format.
@@ -102,11 +111,6 @@ class ClaudeConversationsImporter(Importer[ChatImportResult]):
Returns:
EntityMarkdown instance representing the conversation.
"""
# Generate permalink using folder name (relative path)
date_prefix = datetime.fromisoformat(created_at.replace("Z", "+00:00")).strftime("%Y%m%d")
clean_title = clean_filename(name)
permalink = f"{folder}/{date_prefix}-{clean_title}"
# Format content
content = self._format_chat_markdown(
name=name,
@@ -63,17 +63,28 @@ class ClaudeProjectsImporter(Importer[ProjectImportResult]):
await self.file_service.ensure_directory(docs_dir)
# Import prompt template if it exists
if prompt_entity := self._format_prompt_markdown(project, destination_folder):
# Write file using relative path - FileService handles base_path
file_path = f"{prompt_entity.frontmatter.metadata['permalink']}.md"
await self.write_entity(prompt_entity, file_path)
if project.get("prompt_template"):
prompt_path = (
f"{destination_folder}/{project_dir}/prompt-template"
if destination_folder
else f"{project_dir}/prompt-template"
)
permalink, file_path = self.build_import_paths(prompt_path)
prompt_entity = self._format_prompt_markdown(project, permalink)
if prompt_entity:
await self.write_entity(prompt_entity, file_path)
prompts_imported += 1
# Import project documents
for doc in project.get("docs", []):
entity = self._format_project_markdown(project, doc, destination_folder)
# Write file using relative path - FileService handles base_path
file_path = f"{entity.frontmatter.metadata['permalink']}.md"
doc_file = clean_filename(doc["filename"])
doc_path = (
f"{destination_folder}/{project_dir}/docs/{doc_file}"
if destination_folder
else f"{project_dir}/docs/{doc_file}"
)
permalink, file_path = self.build_import_paths(doc_path)
entity = self._format_project_markdown(project, doc, permalink)
await self.write_entity(entity, file_path)
docs_imported += 1
@@ -89,7 +100,7 @@ class ClaudeProjectsImporter(Importer[ProjectImportResult]):
return self.handle_error("Failed to import Claude projects", e)
def _format_project_markdown(
self, project: Dict[str, Any], doc: Dict[str, Any], destination_folder: str = ""
self, project: Dict[str, Any], doc: Dict[str, Any], permalink: str
) -> EntityMarkdown:
"""Format a project document as a Basic Memory entity.
@@ -105,17 +116,6 @@ class ClaudeProjectsImporter(Importer[ProjectImportResult]):
created_at = doc.get("created_at") or project["created_at"]
modified_at = project["updated_at"]
# Generate clean names for organization
project_dir = clean_filename(project["name"])
doc_file = clean_filename(doc["filename"])
# Build permalink with optional destination folder prefix
permalink = (
f"{destination_folder}/{project_dir}/docs/{doc_file}"
if destination_folder
else f"{project_dir}/docs/{doc_file}"
)
# Create entity
entity = EntityMarkdown(
frontmatter=EntityFrontmatter(
@@ -136,7 +136,7 @@ class ClaudeProjectsImporter(Importer[ProjectImportResult]):
return entity
def _format_prompt_markdown(
self, project: Dict[str, Any], destination_folder: str = ""
self, project: Dict[str, Any], permalink: str
) -> Optional[EntityMarkdown]:
"""Format project prompt template as a Basic Memory entity.
@@ -155,16 +155,6 @@ class ClaudeProjectsImporter(Importer[ProjectImportResult]):
created_at = project["created_at"]
modified_at = project["updated_at"]
# Generate clean project directory name
project_dir = clean_filename(project["name"])
# Build permalink with optional destination folder prefix
permalink = (
f"{destination_folder}/{project_dir}/prompt-template"
if destination_folder
else f"{project_dir}/prompt-template"
)
# Create entity
entity = EntityMarkdown(
frontmatter=EntityFrontmatter(
@@ -80,11 +80,12 @@ class MemoryJsonImporter(Importer[EntityImportResult]):
entity_type = entity_data.get("entityType") or entity_data.get("type") or "entity"
# Build permalink with optional destination folder prefix
permalink = (
relative_path = (
f"{destination_folder}/{entity_type}/{name}"
if destination_folder
else f"{entity_type}/{name}"
)
permalink, file_path = self.build_import_paths(relative_path)
# Ensure entity type directory exists using FileService with relative path
entity_type_dir = (
@@ -109,7 +110,6 @@ class MemoryJsonImporter(Importer[EntityImportResult]):
)
# Write file using relative path - FileService handles base_path
file_path = f"{entity.frontmatter.metadata['permalink']}.md"
await self.write_entity(entity, file_path)
entities_created += 1
+3 -1
View File
@@ -241,7 +241,9 @@ class EntityParser:
f"Failed to parse YAML frontmatter in {file_path}: {e}. "
f"Treating file as plain markdown without frontmatter."
)
post = frontmatter.Post(content, metadata={})
# Use Post(content) not Post(content, metadata={})
# The latter creates {"metadata": {}} in the metadata dict (issue #528)
post = frontmatter.Post(content)
# Normalize frontmatter values
metadata = normalize_frontmatter_metadata(post.metadata)
+4 -2
View File
@@ -1,6 +1,8 @@
"""Markdown-it plugins for Basic Memory markdown parsing."""
from typing import List, Any, Dict
from basic_memory.utils import normalize_project_reference
from markdown_it import MarkdownIt
from markdown_it.token import Token
@@ -114,7 +116,7 @@ def parse_relation(token: Token) -> Dict[str, Any] | None:
rel_type = before
# Get target
target = content[start + 2 : end].strip()
target = normalize_project_reference(content[start + 2 : end].strip())
# Look for context after
after = content[end + 2 :].strip()
@@ -160,7 +162,7 @@ def parse_inline_relations(content: str) -> List[Dict[str, Any]]:
# No matching ]] found
break
target = content[start + 2 : end].strip()
target = normalize_project_reference(content[start + 2 : end].strip())
if target:
relations.append({"type": "links_to", "target": target, "context": None})
+4 -3
View File
@@ -9,6 +9,7 @@ from frontmatter import Post
from basic_memory.file_utils import has_frontmatter, remove_frontmatter, parse_frontmatter
from basic_memory.markdown import EntityMarkdown
from basic_memory.markdown.entity_parser import normalize_frontmatter_metadata
from basic_memory.models import Entity
from basic_memory.models import Observation as ObservationModel
@@ -58,9 +59,9 @@ def entity_model_from_markdown(
model.created_at = markdown.created
model.updated_at = markdown.modified
# Handle metadata - ensure all values are strings and filter None
metadata = markdown.frontmatter.metadata or {}
model.entity_metadata = {k: str(v) for k, v in metadata.items() if v is not None}
# Handle metadata - normalize values and filter None (preserve structured data)
metadata = normalize_frontmatter_metadata(markdown.frontmatter.metadata or {})
model.entity_metadata = {k: v for k, v in metadata.items() if v is not None}
# Get project_id from entity if not provided
obs_project_id = project_id or (model.project_id if hasattr(model, "project_id") else None)
+153 -102
View File
@@ -1,11 +1,99 @@
from contextlib import asynccontextmanager, AbstractAsyncContextManager
import os
from contextlib import AbstractAsyncContextManager, asynccontextmanager
from typing import AsyncIterator, Callable, Optional
from httpx import ASGITransport, AsyncClient, Timeout
from loguru import logger
from basic_memory.api.app import app as fastapi_app
from basic_memory.config import ConfigManager
from basic_memory.config import ConfigManager, ProjectMode
def _force_local_mode() -> bool:
"""Check if local mode is forced via environment variable."""
return os.environ.get("BASIC_MEMORY_FORCE_LOCAL", "").lower() in ("true", "1", "yes")
def _force_cloud_mode() -> bool:
"""Check if cloud mode is forced via environment variable."""
return os.environ.get("BASIC_MEMORY_FORCE_CLOUD", "").lower() in ("true", "1", "yes")
def _explicit_routing() -> bool:
"""Check if CLI --local/--cloud flag was explicitly passed."""
return os.environ.get("BASIC_MEMORY_EXPLICIT_ROUTING", "").lower() in ("true", "1", "yes")
def _build_timeout() -> Timeout:
"""Create a standard timeout config used across all clients."""
return Timeout(
connect=10.0,
read=30.0,
write=30.0,
pool=30.0,
)
def _asgi_client(timeout: Timeout) -> AsyncClient:
"""Create a local ASGI client."""
return AsyncClient(
transport=ASGITransport(app=fastapi_app), base_url="http://test", timeout=timeout
)
async def _resolve_cloud_token(config) -> str:
"""Resolve cloud token with API key preferred, OAuth fallback."""
token = config.cloud_api_key
if token:
return token
from basic_memory.cli.auth import CLIAuth
auth = CLIAuth(client_id=config.cloud_client_id, authkit_domain=config.cloud_domain)
token = await auth.get_valid_token()
if token:
return token
raise RuntimeError(
"Cloud routing requested but no credentials found. "
"Run 'bm cloud set-key <key>' or 'bm cloud login' first."
)
@asynccontextmanager
async def _cloud_client(
config,
timeout: Timeout,
workspace: Optional[str] = None,
) -> AsyncIterator[AsyncClient]:
"""Create a cloud proxy client with resolved credentials."""
token = await _resolve_cloud_token(config)
proxy_base_url = f"{config.cloud_host}/proxy"
headers = {"Authorization": f"Bearer {token}"}
if workspace:
headers["X-Workspace-ID"] = workspace
logger.info(f"Creating HTTP client for cloud proxy at: {proxy_base_url}")
async with AsyncClient(
base_url=proxy_base_url,
headers=headers,
timeout=timeout,
) as client:
yield client
@asynccontextmanager
async def get_cloud_control_plane_client() -> AsyncIterator[AsyncClient]:
"""Create a control-plane cloud client for endpoints outside /proxy."""
config = ConfigManager().config
timeout = _build_timeout()
token = await _resolve_cloud_token(config)
logger.info(f"Creating HTTP client for cloud control plane at: {config.cloud_host}")
async with AsyncClient(
base_url=config.cloud_host,
headers={"Authorization": f"Bearer {token}"},
timeout=timeout,
) as client:
yield client
# Optional factory override for dependency injection
@@ -13,127 +101,90 @@ _client_factory: Optional[Callable[[], AbstractAsyncContextManager[AsyncClient]]
def set_client_factory(factory: Callable[[], AbstractAsyncContextManager[AsyncClient]]) -> None:
"""Override the default client factory (for cloud app, testing, etc).
Args:
factory: An async context manager that yields an AsyncClient
Example:
@asynccontextmanager
async def custom_client_factory():
async with AsyncClient(...) as client:
yield client
set_client_factory(custom_client_factory)
"""
"""Override the default client factory (for cloud app, testing, etc)."""
global _client_factory
_client_factory = factory
@asynccontextmanager
async def get_client() -> AsyncIterator[AsyncClient]:
async def get_client(
project_name: Optional[str] = None,
workspace: Optional[str] = None,
) -> AsyncIterator[AsyncClient]:
"""Get an AsyncClient as a context manager.
This function provides proper resource management for HTTP clients,
ensuring connections are closed after use. It supports three modes:
1. **Factory injection** (cloud app, tests):
If a custom factory is set via set_client_factory(), use that.
2. **CLI cloud mode**:
When cloud_mode_enabled is True, create HTTP client with auth
token from CLIAuth for requests to cloud proxy endpoint.
3. **Local mode** (default):
Use ASGI transport for in-process requests to local FastAPI app.
Usage:
async with get_client() as client:
response = await client.get("/path")
Yields:
AsyncClient: Configured HTTP client for the current mode
Raises:
RuntimeError: If cloud mode is enabled but user is not authenticated
Routing priority:
1. Factory injection.
2. Explicit routing flags (--local/--cloud).
3. Per-project mode routing when project_name is provided.
4. Local ASGI transport by default.
"""
if _client_factory:
# Use injected factory (cloud app, tests)
async with _client_factory() as client:
yield client
else:
# Default: create based on config
config = ConfigManager().config
timeout = Timeout(
connect=10.0, # 10 seconds for connection
read=30.0, # 30 seconds for reading response
write=30.0, # 30 seconds for writing request
pool=30.0, # 30 seconds for connection pool
)
return
if config.cloud_mode_enabled:
# CLI cloud mode: inject auth when creating client
from basic_memory.cli.auth import CLIAuth
config = ConfigManager().config
timeout = _build_timeout()
auth = CLIAuth(client_id=config.cloud_client_id, authkit_domain=config.cloud_domain)
token = await auth.get_valid_token()
# --- Explicit routing override ---
# Trigger: user passed --local/--cloud.
# Why: command-level override should be deterministic and bypass project mode.
# Outcome: route strictly based on explicit flag.
if _explicit_routing():
if _force_local_mode():
logger.info("Explicit local routing enabled - using ASGI client")
async with _asgi_client(timeout) as client:
yield client
return
if not token:
if _force_cloud_mode():
logger.info("Explicit cloud routing enabled - using cloud proxy client")
async with _cloud_client(config, timeout, workspace=workspace) as client:
yield client
return
# --- Per-project routing ---
# Trigger: project_name provided without explicit routing override.
# Why: project mode is the source of truth for project-scoped commands.
# Outcome: route via project.mode (CLOUD/LOCAL).
if project_name is not None and not _explicit_routing():
project_mode = config.get_project_mode(project_name)
if project_mode == ProjectMode.CLOUD:
logger.info(f"Project '{project_name}' is cloud mode - using cloud proxy client")
try:
async with _cloud_client(config, timeout, workspace=workspace) as client:
yield client
except RuntimeError as exc:
raise RuntimeError(
"Cloud mode enabled but not authenticated. "
"Run 'basic-memory cloud login' first."
)
f"Project '{project_name}' is set to cloud mode but no credentials found. "
"Run 'bm cloud set-key <key>' or 'bm cloud login' first."
) from exc
return
# Auth header set ONCE at client creation
proxy_base_url = f"{config.cloud_host}/proxy"
logger.info(f"Creating HTTP client for cloud proxy at: {proxy_base_url}")
async with AsyncClient(
base_url=proxy_base_url,
headers={"Authorization": f"Bearer {token}"},
timeout=timeout,
) as client:
yield client
else:
# Local mode: ASGI transport for in-process calls
# Note: ASGI transport does NOT trigger FastAPI lifespan, so no special handling needed
logger.info("Creating ASGI client for local Basic Memory API")
async with AsyncClient(
transport=ASGITransport(app=fastapi_app), base_url="http://test", timeout=timeout
) as client:
yield client
logger.info(f"Project '{project_name}' is local mode - using ASGI client")
async with _asgi_client(timeout) as client:
yield client
return
# --- Default fallback ---
logger.info("Default routing - using ASGI client for local Basic Memory API")
async with _asgi_client(timeout) as client:
yield client
def create_client() -> AsyncClient:
"""Create an HTTP client based on configuration.
"""Create an HTTP client based on explicit routing flags.
DEPRECATED: Use get_client() context manager instead for proper resource management.
This function is kept for backward compatibility but will be removed in a future version.
The returned client should be closed manually by calling await client.aclose().
Returns:
AsyncClient configured for either local ASGI or remote proxy
"""
config_manager = ConfigManager()
config = config_manager.config
timeout = _build_timeout()
# Configure timeout for longer operations like write_note
# Default httpx timeout is 5 seconds which is too short for file operations
timeout = Timeout(
connect=10.0, # 10 seconds for connection
read=30.0, # 30 seconds for reading response
write=30.0, # 30 seconds for writing request
pool=30.0, # 30 seconds for connection pool
)
if config.cloud_mode_enabled:
# Use HTTP transport to proxy endpoint
proxy_base_url = f"{config.cloud_host}/proxy"
logger.info(f"Creating HTTP client for proxy at: {proxy_base_url}")
return AsyncClient(base_url=proxy_base_url, timeout=timeout)
else:
# Default: use ASGI transport for local API (development mode)
if _force_local_mode() or not _force_cloud_mode():
logger.info("Creating ASGI client for local Basic Memory API")
return AsyncClient(
transport=ASGITransport(app=fastapi_app), base_url="http://test", timeout=timeout
)
return _asgi_client(timeout)
logger.info("Creating HTTP client for cloud proxy (legacy create_client path)")
config = ConfigManager().config
proxy_base_url = f"{config.cloud_host}/proxy"
return AsyncClient(base_url=proxy_base_url, timeout=timeout)
+2
View File
@@ -17,6 +17,7 @@ from basic_memory.mcp.clients.memory import MemoryClient
from basic_memory.mcp.clients.directory import DirectoryClient
from basic_memory.mcp.clients.resource import ResourceClient
from basic_memory.mcp.clients.project import ProjectClient
from basic_memory.mcp.clients.schema import SchemaClient
__all__ = [
"KnowledgeClient",
@@ -25,4 +26,5 @@ __all__ = [
"DirectoryClient",
"ResourceClient",
"ProjectClient",
"SchemaClient",
]
+76 -6
View File
@@ -8,7 +8,12 @@ from typing import Any
from httpx import AsyncClient
from basic_memory.mcp.tools.utils import call_get, call_post, call_put, call_patch, call_delete
from basic_memory.schemas.response import EntityResponse, DeleteEntitiesResponse
from basic_memory.schemas.response import (
EntityResponse,
DeleteEntitiesResponse,
DirectoryMoveResult,
DirectoryDeleteResult,
)
class KnowledgeClient:
@@ -38,7 +43,9 @@ class KnowledgeClient:
# --- Entity CRUD Operations ---
async def create_entity(self, entity_data: dict[str, Any]) -> EntityResponse:
async def create_entity(
self, entity_data: dict[str, Any], *, fast: bool | None = None
) -> EntityResponse:
"""Create a new entity.
Args:
@@ -50,14 +57,22 @@ class KnowledgeClient:
Raises:
ToolError: If the request fails
"""
params = {"fast": fast} if fast is not None else None
response = await call_post(
self.http_client,
f"{self._base_path}/entities",
json=entity_data,
params=params,
)
return EntityResponse.model_validate(response.json())
async def update_entity(self, entity_id: str, entity_data: dict[str, Any]) -> EntityResponse:
async def update_entity(
self,
entity_id: str,
entity_data: dict[str, Any],
*,
fast: bool | None = None,
) -> EntityResponse:
"""Update an existing entity (full replacement).
Args:
@@ -70,10 +85,12 @@ class KnowledgeClient:
Raises:
ToolError: If the request fails
"""
params = {"fast": fast} if fast is not None else None
response = await call_put(
self.http_client,
f"{self._base_path}/entities/{entity_id}",
json=entity_data,
params=params,
)
return EntityResponse.model_validate(response.json())
@@ -95,7 +112,13 @@ class KnowledgeClient:
)
return EntityResponse.model_validate(response.json())
async def patch_entity(self, entity_id: str, patch_data: dict[str, Any]) -> EntityResponse:
async def patch_entity(
self,
entity_id: str,
patch_data: dict[str, Any],
*,
fast: bool | None = None,
) -> EntityResponse:
"""Partially update an entity.
Args:
@@ -108,10 +131,12 @@ class KnowledgeClient:
Raises:
ToolError: If the request fails
"""
params = {"fast": fast} if fast is not None else None
response = await call_patch(
self.http_client,
f"{self._base_path}/entities/{entity_id}",
json=patch_data,
params=params,
)
return EntityResponse.model_validate(response.json())
@@ -153,13 +178,58 @@ class KnowledgeClient:
)
return EntityResponse.model_validate(response.json())
async def move_directory(
self, source_directory: str, destination_directory: str
) -> DirectoryMoveResult:
"""Move all entities in a directory to a new location.
Args:
source_directory: Source directory path (relative to project root)
destination_directory: Destination directory path (relative to project root)
Returns:
DirectoryMoveResult with counts and details of moved files
Raises:
ToolError: If the request fails
"""
response = await call_post(
self.http_client,
f"{self._base_path}/move-directory",
json={
"source_directory": source_directory,
"destination_directory": destination_directory,
},
)
return DirectoryMoveResult.model_validate(response.json())
async def delete_directory(self, directory: str) -> DirectoryDeleteResult:
"""Delete all entities in a directory.
Args:
directory: Directory path to delete (relative to project root)
Returns:
DirectoryDeleteResult with counts and details of deleted files
Raises:
ToolError: If the request fails
"""
response = await call_post(
self.http_client,
f"{self._base_path}/delete-directory",
json={"directory": directory},
)
return DirectoryDeleteResult.model_validate(response.json())
# --- Resolution ---
async def resolve_entity(self, identifier: str) -> str:
async def resolve_entity(self, identifier: str, *, strict: bool = False) -> str:
"""Resolve a string identifier to an entity external_id.
Args:
identifier: The identifier to resolve (permalink, title, or path)
strict: If True, require exact matching (no fuzzy fallback)
Returns:
The resolved entity external_id (UUID)
@@ -170,7 +240,7 @@ class KnowledgeClient:
response = await call_post(
self.http_client,
f"{self._base_path}/resolve",
json={"identifier": identifier},
json={"identifier": identifier, "strict": strict},
)
data = response.json()
return data["external_id"]
+2 -2
View File
@@ -47,7 +47,7 @@ class ProjectClient:
"""
response = await call_get(
self.http_client,
"/projects/projects",
"/v2/projects/",
)
return ProjectList.model_validate(response.json())
@@ -65,7 +65,7 @@ class ProjectClient:
"""
response = await call_post(
self.http_client,
"/projects/projects",
"/v2/projects/",
json=project_data,
)
return ProjectStatusResponse.model_validate(response.json())
+113
View File
@@ -0,0 +1,113 @@
"""Typed client for schema API operations.
Encapsulates all /v2/projects/{project_id}/schema/* endpoints.
"""
from httpx import AsyncClient
from basic_memory.mcp.tools.utils import call_post, call_get
from basic_memory.schemas.schema import (
ValidationReport,
InferenceReport,
DriftReport,
)
class SchemaClient:
"""Typed client for schema operations.
Centralizes:
- API path construction for /v2/projects/{project_id}/schema/*
- Response validation via Pydantic models
- Consistent error handling through call_* utilities
Usage:
async with get_client() as http_client:
client = SchemaClient(http_client, project_id)
report = await client.validate(entity_type="Person")
"""
def __init__(self, http_client: AsyncClient, project_id: str):
"""Initialize the schema client.
Args:
http_client: HTTPX AsyncClient for making requests
project_id: Project external_id (UUID) for API calls
"""
self.http_client = http_client
self.project_id = project_id
self._base_path = f"/v2/projects/{project_id}/schema"
async def validate(
self,
*,
entity_type: str | None = None,
identifier: str | None = None,
) -> ValidationReport:
"""Validate notes against their resolved schemas.
Args:
entity_type: Optional entity type to batch-validate
identifier: Optional specific note to validate
Returns:
ValidationReport with per-note results
Raises:
ToolError: If the request fails
"""
params: dict[str, str] = {}
if entity_type:
params["entity_type"] = entity_type
if identifier:
params["identifier"] = identifier
response = await call_post(
self.http_client,
f"{self._base_path}/validate",
params=params,
)
return ValidationReport.model_validate(response.json())
async def infer(
self,
entity_type: str,
*,
threshold: float = 0.25,
) -> InferenceReport:
"""Infer a schema from existing notes of a given type.
Args:
entity_type: The entity type to analyze
threshold: Minimum frequency for optional fields (0-1)
Returns:
InferenceReport with frequency data and suggested schema
Raises:
ToolError: If the request fails
"""
response = await call_post(
self.http_client,
f"{self._base_path}/infer",
params={"entity_type": entity_type, "threshold": threshold},
)
return InferenceReport.model_validate(response.json())
async def diff(self, entity_type: str) -> DriftReport:
"""Show drift between schema definition and actual usage.
Args:
entity_type: The entity type to check for drift
Returns:
DriftReport with detected differences
Raises:
ToolError: If the request fails
"""
response = await call_get(
self.http_client,
f"{self._base_path}/diff/{entity_type}",
)
return DriftReport.model_validate(response.json())
-1
View File
@@ -39,7 +39,6 @@ class McpContainer:
"""
config = ConfigManager().config
mode = resolve_runtime_mode(
cloud_mode_enabled=config.cloud_mode_enabled,
is_test_env=config.is_test_env,
)
return cls(config=config, mode=mode)
+162
View File
@@ -0,0 +1,162 @@
"""Formatting helpers for MCP tool outputs."""
from __future__ import annotations
from typing import Sequence
from basic_memory.schemas.search import SearchResponse, SearchResult
ANSI_RESET = "\x1b[0m"
ANSI_BOLD = "\x1b[1m"
ANSI_DIM = "\x1b[2m"
ANSI_CYAN = "\x1b[36m"
def _apply_style(text: str, style: str, enabled: bool) -> str:
if not enabled:
return text
return f"{style}{text}{ANSI_RESET}"
def _strip_frontmatter(text: str) -> str:
lines = text.splitlines()
if not lines or lines[0].strip() != "---":
return text
for idx in range(1, len(lines)):
if lines[idx].strip() == "---":
return "\n".join(lines[idx + 1 :]).lstrip()
return text
def _parse_title(text: str) -> str | None:
for line in text.splitlines():
if line.startswith("# "):
return line[2:].strip()
return None
def _truncate(text: str, width: int) -> str:
if width <= 0:
return ""
if len(text) <= width:
return text
if width <= 3:
return text[:width]
return text[: width - 3] + "..."
def _make_separator(widths: Sequence[int]) -> str:
return "+" + "+".join("-" * (width + 2) for width in widths) + "+"
def _format_row(values: Sequence[str], widths: Sequence[int]) -> str:
cells = []
for value, width in zip(values, widths, strict=True):
cells.append(f" {_truncate(value, width).ljust(width)} ")
return "|" + "|".join(cells) + "|"
def _get_result_tags(result: SearchResult) -> str:
metadata = result.metadata or {}
if isinstance(metadata, dict):
tags = metadata.get("tags")
if isinstance(tags, list):
return ", ".join(str(tag) for tag in tags if tag)
return ""
def _get_result_path(result: SearchResult) -> str:
return result.permalink or result.file_path or ""
def format_search_results_ascii(
result: SearchResponse,
query: str | None = None,
color: bool = False,
) -> str:
"""Format search results as an ASCII table for TUI clients."""
results = result.results or []
header_line = _apply_style("Search results", f"{ANSI_BOLD}{ANSI_CYAN}", color)
lines = [header_line]
if query:
lines.append(f"Query: {query}")
summary = (
f"Results: {len(results)} | Page: {result.current_page} | Page size: {result.page_size}"
)
lines.append(_apply_style(summary, ANSI_DIM, color))
if not results:
lines.append("No results.")
return "\n".join(lines).strip()
headers = ["#", "Title", "Type", "Score", "Path", "Tags"]
rows = []
for idx, item in enumerate(results, start=1):
rows.append(
[
str(idx),
item.title or "Untitled",
item.type.value if hasattr(item.type, "value") else str(item.type),
f"{item.score:.2f}" if isinstance(item.score, (int, float)) else "",
_get_result_path(item),
_get_result_tags(item),
]
)
max_widths = [3, 32, 10, 7, 36, 24]
widths = []
for index, header in enumerate(headers):
column_values = [header] + [row[index] for row in rows]
max_len = max(len(value) for value in column_values)
widths.append(min(max_widths[index], max_len))
table = [_make_separator(widths)]
header_row = _format_row(headers, widths)
if color:
header_cells = []
for value, width in zip(headers, widths, strict=True):
padded = f" {_truncate(value, width).ljust(width)} "
header_cells.append(_apply_style(padded, f"{ANSI_BOLD}{ANSI_CYAN}", color))
header_row = "|" + "|".join(header_cells) + "|"
table.append(header_row)
table.append(_make_separator(widths))
for row in rows:
table.append(_format_row(row, widths))
table.append(_make_separator(widths))
lines.append("")
lines.extend(table)
return "\n".join(lines).rstrip()
def format_note_preview_ascii(
content: str,
identifier: str | None = None,
color: bool = False,
) -> str:
"""Format note content for ASCII/TUI display."""
identifier = identifier or ""
cleaned = _strip_frontmatter(content)
title = _parse_title(cleaned) or identifier or "Note Preview"
header = _apply_style("Note preview", f"{ANSI_BOLD}{ANSI_CYAN}", color)
lines = [header, f"Title: {title}"]
if identifier:
lines.append(f"Identifier: {identifier}")
lines.append(_apply_style("-" * 72, ANSI_DIM, color))
if content.strip():
lines.append(content.rstrip())
else:
lines.append("(empty note)")
return "\n".join(lines).rstrip()
+286 -31
View File
@@ -8,66 +8,59 @@ The resolve_project_parameter function is a thin wrapper for backwards
compatibility with existing MCP tools.
"""
from typing import Optional, List
from contextlib import asynccontextmanager
from typing import AsyncIterator, Optional, List, Tuple
from httpx import AsyncClient
from httpx._types import (
HeaderTypes,
)
from loguru import logger
from fastmcp import Context
from mcp.server.fastmcp.exceptions import ToolError
from basic_memory.config import ConfigManager
from basic_memory.config import ConfigManager, ProjectMode
from basic_memory.project_resolver import ProjectResolver
from basic_memory.schemas.cloud import WorkspaceInfo, WorkspaceListResponse
from basic_memory.schemas.project_info import ProjectItem, ProjectList
from basic_memory.utils import generate_permalink
from basic_memory.schemas.v2 import ProjectResolveResponse
from basic_memory.schemas.memory import memory_url_path
from basic_memory.utils import generate_permalink, normalize_project_reference
async def resolve_project_parameter(
project: Optional[str] = None,
allow_discovery: bool = False,
cloud_mode: Optional[bool] = None,
default_project_mode: Optional[bool] = None,
default_project: Optional[str] = None,
) -> Optional[str]:
"""Resolve project parameter using three-tier hierarchy.
"""Resolve project parameter using unified linear priority chain.
This is a thin wrapper around ProjectResolver for backwards compatibility.
New code should consider using ProjectResolver directly for more detailed
resolution information.
if cloud_mode:
project is required (unless allow_discovery=True for tools that support discovery mode)
else:
Resolution order:
1. Single Project Mode (--project cli arg, or BASIC_MEMORY_MCP_PROJECT env var) - highest priority
2. Explicit project parameter - medium priority
3. Default project if default_project_mode=true - lowest priority
Resolution order:
1. ENV_CONSTRAINT: BASIC_MEMORY_MCP_PROJECT env var (highest priority)
2. EXPLICIT: project parameter passed directly
3. DEFAULT: default_project from config (if set)
4. Fallback: discovery (if allowed) NONE
Args:
project: Optional explicit project parameter
allow_discovery: If True, allows returning None in cloud mode for discovery mode
allow_discovery: If True, allows returning None for discovery mode
(used by tools like recent_activity that can operate across all projects)
cloud_mode: Optional explicit cloud mode. If not provided, reads from ConfigManager.
default_project_mode: Optional explicit default project mode. If not provided, reads from ConfigManager.
default_project: Optional explicit default project. If not provided, reads from ConfigManager.
Returns:
Resolved project name or None if no resolution possible
"""
# Load config for any values not explicitly provided
if cloud_mode is None or default_project_mode is None or default_project is None:
if default_project is None:
config = ConfigManager().config
if cloud_mode is None:
cloud_mode = config.cloud_mode
if default_project_mode is None:
default_project_mode = config.default_project_mode
if default_project is None:
default_project = config.default_project
default_project = config.default_project
# Create resolver with configuration and resolve
resolver = ProjectResolver.from_env(
cloud_mode=cloud_mode,
default_project_mode=default_project_mode,
default_project=default_project,
)
result = resolver.resolve(project=project, allow_discovery=allow_discovery)
@@ -78,11 +71,105 @@ async def get_project_names(client: AsyncClient, headers: HeaderTypes | None = N
# Deferred import to avoid circular dependency with tools
from basic_memory.mcp.tools.utils import call_get
response = await call_get(client, "/projects/projects", headers=headers)
response = await call_get(client, "/v2/projects/", headers=headers)
project_list = ProjectList.model_validate(response.json())
return [project.name for project in project_list.projects]
def _workspace_matches_identifier(workspace: WorkspaceInfo, identifier: str) -> bool:
"""Return True when identifier matches workspace tenant_id or name."""
if workspace.tenant_id == identifier:
return True
return workspace.name.lower() == identifier.lower()
def _workspace_choices(workspaces: list[WorkspaceInfo]) -> str:
"""Format deterministic workspace choices for prompt-style errors."""
return "\n".join(
[
(
f"- {item.name} "
f"(type={item.workspace_type}, role={item.role}, tenant_id={item.tenant_id})"
)
for item in workspaces
]
)
async def get_available_workspaces(context: Optional[Context] = None) -> list[WorkspaceInfo]:
"""Load available cloud workspaces for the current authenticated user."""
if context:
cached_workspaces = context.get_state("available_workspaces")
if isinstance(cached_workspaces, list) and all(
isinstance(item, WorkspaceInfo) for item in cached_workspaces
):
return cached_workspaces
from basic_memory.mcp.async_client import get_cloud_control_plane_client
from basic_memory.mcp.tools.utils import call_get
async with get_cloud_control_plane_client() as client:
response = await call_get(client, "/workspaces/")
workspace_list = WorkspaceListResponse.model_validate(response.json())
if context:
context.set_state("available_workspaces", workspace_list.workspaces)
return workspace_list.workspaces
async def resolve_workspace_parameter(
workspace: Optional[str] = None,
context: Optional[Context] = None,
) -> WorkspaceInfo:
"""Resolve workspace using explicit input, session cache, and cloud discovery."""
if context:
cached_workspace = context.get_state("active_workspace")
if isinstance(cached_workspace, WorkspaceInfo) and (
workspace is None or _workspace_matches_identifier(cached_workspace, workspace)
):
logger.debug(f"Using cached workspace from context: {cached_workspace.tenant_id}")
return cached_workspace
workspaces = await get_available_workspaces(context=context)
if not workspaces:
raise ValueError(
"No accessible workspaces found for this account. "
"Ensure you have an active subscription and tenant access."
)
selected_workspace: WorkspaceInfo | None = None
if workspace:
matches = [item for item in workspaces if _workspace_matches_identifier(item, workspace)]
if not matches:
raise ValueError(
f"Workspace '{workspace}' was not found.\n"
f"Available workspaces:\n{_workspace_choices(workspaces)}"
)
if len(matches) > 1:
raise ValueError(
f"Workspace name '{workspace}' matches multiple workspaces. "
"Use tenant_id instead.\n"
f"Available workspaces:\n{_workspace_choices(workspaces)}"
)
selected_workspace = matches[0]
elif len(workspaces) == 1:
selected_workspace = workspaces[0]
else:
raise ValueError(
"Multiple workspaces are available. Ask the user which workspace to use, then retry "
"with the 'workspace' argument set to the tenant_id or unique name.\n"
f"Available workspaces:\n{_workspace_choices(workspaces)}"
)
if context:
context.set_state("active_workspace", selected_workspace)
logger.debug(f"Cached workspace in context: {selected_workspace.tenant_id}")
return selected_workspace
async def get_active_project(
client: AsyncClient,
project: Optional[str] = None,
@@ -104,14 +191,14 @@ async def get_active_project(
HTTPError: If project doesn't exist or is inaccessible
"""
# Deferred import to avoid circular dependency with tools
from basic_memory.mcp.tools.utils import call_get
from basic_memory.mcp.tools.utils import call_post
resolved_project = await resolve_project_parameter(project)
if not resolved_project:
project_names = await get_project_names(client, headers)
raise ValueError(
"No project specified. "
"Either set 'default_project_mode=true' in config, or use 'project' argument.\n"
"Either set 'default_project' in config, or use 'project' argument.\n"
f"Available projects: {project_names}"
)
@@ -126,9 +213,20 @@ async def get_active_project(
# Validate project exists by calling API
logger.debug(f"Validating project: {project}")
permalink = generate_permalink(project)
response = await call_get(client, f"/{permalink}/project/item", headers=headers)
active_project = ProjectItem.model_validate(response.json())
response = await call_post(
client,
"/v2/projects/resolve",
json={"identifier": project},
headers=headers,
)
resolved = ProjectResolveResponse.model_validate(response.json())
active_project = ProjectItem(
id=resolved.project_id,
external_id=resolved.external_id,
name=resolved.name,
path=resolved.path,
is_default=resolved.is_default,
)
# Cache in context if available
if context:
@@ -139,6 +237,96 @@ async def get_active_project(
return active_project
def _split_project_prefix(path: str) -> tuple[Optional[str], str]:
"""Split a possible project prefix from a memory URL path."""
if "/" not in path:
return None, path
project_prefix, remainder = path.split("/", 1)
if not project_prefix or not remainder:
return None, path
if "*" in project_prefix:
return None, path
return project_prefix, remainder
async def resolve_project_and_path(
client: AsyncClient,
identifier: str,
project: Optional[str] = None,
context: Optional[Context] = None,
headers: HeaderTypes | None = None,
) -> tuple[ProjectItem, str, bool]:
"""Resolve project and normalized path for memory:// identifiers.
Returns:
Tuple of (active_project, normalized_path, is_memory_url)
"""
is_memory_url = identifier.strip().startswith("memory://")
if not is_memory_url:
active_project = await get_active_project(client, project, context, headers)
return active_project, identifier, False
normalized_path = normalize_project_reference(memory_url_path(identifier))
project_prefix, remainder = _split_project_prefix(normalized_path)
include_project = ConfigManager().config.permalinks_include_project
# Trigger: memory URL begins with a potential project segment
# Why: allow project-scoped memory URLs without requiring a separate project parameter
# Outcome: attempt to resolve the prefix as a project and route to it
if project_prefix:
try:
from basic_memory.mcp.tools.utils import call_post
response = await call_post(
client,
"/v2/projects/resolve",
json={"identifier": project_prefix},
headers=headers,
)
resolved = ProjectResolveResponse.model_validate(response.json())
except ToolError as exc:
if "project not found" not in str(exc).lower():
raise
else:
resolved_project = await resolve_project_parameter(project_prefix)
if resolved_project and generate_permalink(resolved_project) != generate_permalink(
project_prefix
):
raise ValueError(
f"Project is constrained to '{resolved_project}', cannot use '{project_prefix}'."
)
active_project = ProjectItem(
id=resolved.project_id,
external_id=resolved.external_id,
name=resolved.name,
path=resolved.path,
is_default=resolved.is_default,
)
if context:
context.set_state("active_project", active_project)
resolved_path = f"{resolved.permalink}/{remainder}" if include_project else remainder
return active_project, resolved_path, True
# Trigger: no resolvable project prefix in the memory URL
# Why: preserve existing memory URL behavior within the active project
# Outcome: use the active project and normalize the path for lookup
active_project = await get_active_project(client, project, context, headers)
resolved_path = normalized_path
if include_project:
# Trigger: project-prefixed permalinks are enabled and the path lacks a prefix
# Why: ensure memory URL lookups align with canonical permalinks
# Outcome: prefix the path with the active project's permalink
project_prefix = active_project.permalink
if resolved_path != project_prefix and not resolved_path.startswith(f"{project_prefix}/"):
resolved_path = f"{project_prefix}/{resolved_path}"
return active_project, resolved_path, True
def add_project_metadata(result: str, project_name: str) -> str:
"""Add project context as metadata footer for assistant session tracking.
@@ -153,3 +341,70 @@ def add_project_metadata(result: str, project_name: str) -> str:
Result with project session tracking metadata
"""
return f"{result}\n\n[Session: Using project '{project_name}']"
@asynccontextmanager
async def get_project_client(
project: Optional[str] = None,
workspace: Optional[str] = None,
context: Optional[Context] = None,
) -> AsyncIterator[Tuple[AsyncClient, ProjectItem]]:
"""Resolve project, create correctly-routed client, and validate project.
Solves the bootstrap problem: we need to know the project name to choose
the right client (local vs cloud), but we need the client to validate
the project. This helper resolves the project from config first (no
network), creates the correctly-routed client, then validates via API.
Args:
project: Optional explicit project parameter
workspace: Optional cloud workspace selector (tenant_id or unique name)
context: Optional FastMCP context for caching
Yields:
Tuple of (client, active_project)
Raises:
ValueError: If no project can be resolved
RuntimeError: If cloud project but no API key configured
"""
# Deferred import to avoid circular dependency
from basic_memory.mcp.async_client import get_client
# Step 1: Resolve project name from config (no network call)
resolved_project = await resolve_project_parameter(project)
if not resolved_project:
# Fall back to local client to discover projects and raise helpful error
async with get_client() as client:
project_names = await get_project_names(client)
raise ValueError(
"No project specified. "
"Either set 'default_project' in config, or use 'project' argument.\n"
f"Available projects: {project_names}"
)
# Step 2: Resolve project mode and optional workspace selection
config = ConfigManager().config
project_mode = config.get_project_mode(resolved_project)
active_workspace: WorkspaceInfo | None = None
# Trigger: workspace provided for a local project
# Why: workspace selection is a cloud routing concern only
# Outcome: fail fast with a deterministic guidance message
if project_mode != ProjectMode.CLOUD and workspace is not None:
raise ValueError(
f"Workspace '{workspace}' cannot be used with local project '{resolved_project}'. "
"Workspace selection is only supported for cloud-mode projects."
)
if project_mode == ProjectMode.CLOUD:
active_workspace = await resolve_workspace_parameter(workspace=workspace, context=context)
# Step 2: Create client routed based on project's mode
async with get_client(
project_name=resolved_project,
workspace=active_workspace.tenant_id if active_workspace else None,
) as client:
# Step 3: Validate project exists via API
active_project = await get_active_project(client, resolved_project, context)
yield client, active_project
@@ -14,8 +14,8 @@ def ai_assistant_guide() -> str:
"""Return a concise guide on Basic Memory tools and how to use them.
Dynamically adapts instructions based on configuration:
- Default project mode: Simplified instructions with automatic project
- Regular mode: Project discovery and selection guidance
- Default project set: Simplified instructions with automatic project fallback
- No default project: Project discovery and selection guidance
- CLI constraint mode: Single project constraint information
Returns:
@@ -30,34 +30,32 @@ def ai_assistant_guide() -> str:
# Check configuration for mode-specific instructions
config = ConfigManager().config
# Add mode-specific header
mode_info = ""
if config.default_project_mode: # pragma: no cover
# Add mode-specific header based on whether a default project is configured
if config.default_project:
mode_info = f"""
# 🎯 Default Project Mode Active
# Default Project Active
**Current Configuration**: All operations automatically use project '{config.default_project}'
**Current Configuration**: Operations automatically fall back to project '{config.default_project}'
**Simplified Usage**: You don't need to specify the project parameter in tool calls.
- `write_note(title="Note", content="...", folder="docs")`
- Project parameter is optional and will default to '{config.default_project}'
- `write_note(title="Note", content="...", folder="docs")` - uses '{config.default_project}'
- To use a different project, explicitly specify: `project="other-project"`
---
"""
else: # pragma: no cover
mode_info = """
# 🔧 Multi-Project Mode Active
# Multi-Project Mode
**Current Configuration**: Project parameter required for all operations
**Current Configuration**: No default project set project parameter required for all operations
**Project Discovery Required**: Use these tools to select a project:
- `list_memory_projects()` - See all available projects
- `recent_activity()` - Get project activity and recommendations
- Remember the user's project choice throughout the conversation
---
"""
@@ -65,6 +63,7 @@ def ai_assistant_guide() -> str:
enhanced_content = mode_info + content
logger.info(
f"Loaded AI assistant guide ({len(enhanced_content)} chars) with mode: {'default_project' if config.default_project_mode else 'multi_project'}"
f"Loaded AI assistant guide ({len(enhanced_content)} chars) "
f"with default_project: {config.default_project or 'none'}"
)
return enhanced_content

Some files were not shown because too many files have changed in this diff Show More