Compare commits

...

28 Commits

Author SHA1 Message Date
phernandez 7ebf16a95d chore: update version to 0.17.8 for v0.17.8 release 2026-01-24 11:43:44 -06:00
phernandez c05075f8d4 docs: add CHANGELOG entry for v0.17.8
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-24 11:43:21 -06:00
phernandez 4cef9281ca docs: add CHANGELOG entry for v0.17.7
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-24 11:39:41 -06:00
Paul Hernandez 6888effef2 fix: correct get_default_project() query to check for True instead of not NULL (#521)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 11:37:05 -06:00
Paul Hernandez 38616c345d fix: read default project from database in cloud mode (#520)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 10:44:04 -06:00
phernandez f3c1aa895c fix links in README.md to remove smithery badge
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-22 12:58:03 -06:00
phernandez d978aba09b fix links in README.md to remove glama.ai
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-22 12:56:33 -06:00
phernandez 2aaee734c9 fix links in README.md to point to basicmemory.com instead of basicmachines.co
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-22 12:53:18 -06:00
Paul Hernandez 369ad37b3d feat: add SPEC-29 Phase 3 bucket snapshot CLI commands (#476)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-20 17:55:01 -06:00
Drew Cain 4e5f701d22 Fix server.json runtimeArguments format
- Use proper Argument object format instead of plain strings
- Add .mcpregistry tokens to .gitignore

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-01-19 12:23:29 -06:00
Drew Cain 9d9ea4d61c chore: update version to 0.17.7 for v0.17.7 release 2026-01-19 12:12:35 -06:00
Drew Cain 7a502e6474 feat: Add MCP registry publication files (#515)
Signed-off-by: Drew Cain <groksrc@gmail.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-19 11:57:50 -06:00
Paul Hernandez c7835a9d5c fix: ensure external_id is set on entity creation (#512) (#513)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-19 11:14:48 -06:00
Paul Hernandez 85835ae533 chore: Remove OpenPanel telemetry (#514)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-18 22:24:27 -06:00
phernandez 671e3d4db9 chore: update version to 0.17.6 for v0.17.6 release 2026-01-17 15:11:34 -06:00
phernandez e11aeff8d9 docs: add changelog entry for v0.17.6
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-17 15:11:17 -06:00
phernandez 803f3efe53 turn sync logging to debug
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-17 15:10:04 -06:00
phernandez d6dab8552c Merge branch 'main' of github.com:basicmachines-co/basic-memory 2026-01-17 15:09:04 -06:00
phernandez d1d433df15 remove logfire config, and specs docs, turn lifespan logging to debug
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-17 15:08:54 -06:00
Drew Cain 1799c94953 fix: Docker container Python symlink broken at runtime (#510)
Signed-off-by: Drew Cain <groksrc@gmail.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 11:59:35 -06:00
Drew Cain 07996181b3 chore: add doc update reminder to release commands
Remind developers to update docs.basicmemory.com and basicmachines.co
after running just release or just beta.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 10:21:23 -06:00
phernandez a1c37c1dba chore: update version to 0.17.5 for v0.17.5 release 2026-01-11 16:53:55 -06:00
phernandez aff53cca93 docs: add changelog entry for v0.17.5
- Python 3.14 compatibility fix for CLI commands hanging on exit
- Skip nest_asyncio on Python 3.14+
- Update pyright to 1.1.408 for Python 3.14 support
- Fix SQLAlchemy rowcount typing

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-11 16:53:38 -06:00
Paul Hernandez 863e0a4e24 fix: prevent CLI commands from hanging on exit (Python 3.14) (#505)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 16:47:35 -06:00
phernandez eeeade4f07 chore: update version to 0.17.4 for v0.17.4 release 2026-01-05 21:38:03 -06:00
phernandez 03793eaf7c docs: add v0.17.4 changelog entry
- Critical bug fix for search index preservation (#503)
- Major architecture refactor with composition roots (#502)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-01-05 21:36:14 -06:00
Paul Hernandez 26f7e98932 fix: preserve search index across server restarts (#503)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
2026-01-05 21:31:35 -06:00
Paul Hernandez 5947f04bd3 refactor: composition roots, deps split, and typed API clients (#490 roadmap) (#502)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 11:05:58 -06:00
171 changed files with 8393 additions and 15686 deletions
+31 -1
View File
@@ -78,6 +78,34 @@ The GitHub Actions workflow (`.github/workflows/release.yml`) then:
2. Verify formula version matches release
3. Test Homebrew installation: `brew install basicmachines-co/basic-memory/basic-memory`
#### MCP Registry Publication
After PyPI release is published, update the MCP registry:
1. **Verify PyPI Release**
- Confirm package is live: https://pypi.org/project/basic-memory/<version>/
- The `server.json` version was auto-updated by `just release`
2. **Publish to MCP Registry**
```bash
cd /Users/drew/code/basic-memory
mcp-publisher publish
```
If not authenticated:
```bash
mcp-publisher login github
# Follow device authentication flow
mcp-publisher publish
```
3. **Verify Publication**
```bash
curl "https://registry.modelcontextprotocol.io/v0.1/servers?search=basic-memory"
```
**Note:** The `mcp-publisher` CLI can be installed via Homebrew (`brew install mcp-publisher`) or from GitHub releases.
#### Website Updates
**1. basicmachines.co** (`/Users/drew/code/basicmachines.co`)
@@ -145,6 +173,7 @@ Before starting, verify:
📋 GitHub Release: https://github.com/basicmachines-co/basic-memory/releases/tag/v0.13.2
📦 PyPI: https://pypi.org/project/basic-memory/0.13.2/
🍺 Homebrew: https://github.com/basicmachines-co/homebrew-basic-memory
🔌 MCP Registry: https://registry.modelcontextprotocol.io
🚀 GitHub Actions: Completed
Install with pip/uv:
@@ -162,8 +191,9 @@ Users can now upgrade:
- This creates production releases used by end users
- Must pass all quality gates before proceeding
- Uses the automated justfile target for consistency
- Version is automatically updated in `__init__.py`
- Version is automatically updated in `__init__.py` and `server.json`
- Triggers automated GitHub release with changelog
- Package is published to PyPI for `pip` and `uv` users
- Homebrew formula is automatically updated for stable releases
- MCP Registry is updated manually via `mcp-publisher publish`
- Supports multiple installation methods (uv, pip, Homebrew)
+1
View File
@@ -54,6 +54,7 @@ jobs:
- [ ] Unit tests for new functions/methods
- [ ] Integration tests for new MCP tools
- [ ] Test coverage for edge cases
- [ ] **100% test coverage maintained** (use `# pragma: no cover` only for truly hard-to-test code)
- [ ] Documentation updated (README, docstrings)
- [ ] CLAUDE.md updated if conventions change
+3 -3
View File
@@ -19,7 +19,7 @@ jobs:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest]
python-version: [ "3.12", "3.13" ]
python-version: [ "3.12", "3.13", "3.14" ]
runs-on: ${{ matrix.os }}
steps:
@@ -75,7 +75,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: [ "3.12", "3.13" ]
python-version: [ "3.12", "3.13", "3.14" ]
runs-on: ubuntu-latest
# Note: No services section needed - testcontainers handles Postgres in Docker
@@ -164,4 +164,4 @@ jobs:
uses: actions/upload-artifact@v4
with:
name: htmlcov
path: htmlcov/
path: htmlcov/
+1
View File
@@ -54,3 +54,4 @@ ENV/
claude-output
**/.claude/settings.local.json
.mcp.json
.mcpregistry_*
+1 -1
View File
@@ -1 +1 @@
3.12
3.14
+81
View File
@@ -1,5 +1,86 @@
# CHANGELOG
## v0.17.8 (2026-01-24)
### Bug Fixes
- **#521**: Fix `get_default_project()` returning multiple results
([`6888eff`](https://github.com/basicmachines-co/basic-memory/commit/6888eff))
- Query incorrectly matched any project with non-NULL `is_default` (both True and False)
- Now correctly checks for `is_default=True` only
## v0.17.7 (2026-01-24)
### Features
- **#476**: Add SPEC-29 Phase 3 bucket snapshot CLI commands
([`369ad37`](https://github.com/basicmachines-co/basic-memory/commit/369ad37))
- New `basic-memory cloud snapshot` commands for managing cloud snapshots
- Commands: `create`, `list`, `delete`, `show`, `browse`
- **#515**: Add MCP registry publication files
([`7a502e6`](https://github.com/basicmachines-co/basic-memory/commit/7a502e6))
### Bug Fixes
- **#520**: Read default project from database in cloud mode
([`38616c3`](https://github.com/basicmachines-co/basic-memory/commit/38616c3))
- **#513**: Ensure external_id is set on entity creation
([`c7835a9`](https://github.com/basicmachines-co/basic-memory/commit/c7835a9))
### Internal
- **#514**: Remove OpenPanel telemetry
([`85835ae`](https://github.com/basicmachines-co/basic-memory/commit/85835ae))
- Update README links to point to basicmemory.com
([`2aaee73`](https://github.com/basicmachines-co/basic-memory/commit/2aaee73))
## v0.17.6 (2026-01-17)
### Bug Fixes
- **#510**: Fix Docker container Python symlink broken at runtime
([`1799c94`](https://github.com/basicmachines-co/basic-memory/commit/1799c94))
### Internal
- Remove logfire config and specs docs, reduce lifespan and sync logging to debug level
([`d1d433d`](https://github.com/basicmachines-co/basic-memory/commit/d1d433d),
[`803f3ef`](https://github.com/basicmachines-co/basic-memory/commit/803f3ef))
## v0.17.5 (2026-01-11)
### Bug Fixes
- **#505**: Prevent CLI commands from hanging on exit (Python 3.14 compatibility)
([`863e0a4`](https://github.com/basicmachines-co/basic-memory/commit/863e0a4))
- Skip `nest_asyncio` on Python 3.14+ where it causes event loop issues
- Simplify CLI test infrastructure for cross-version compatibility
- Update pyright to 1.1.408 for Python 3.14 support
- Fix SQLAlchemy rowcount typing for Python 3.14
## v0.17.4 (2026-01-05)
### Bug Fixes
- **#503**: Preserve search index across server restarts
([`26f7e98`](https://github.com/basicmachines-co/basic-memory/commit/26f7e98))
- Fixes critical bug where search index was wiped on every MCP server restart
- Bug was introduced in v0.16.3, affecting v0.16.3-v0.17.3
- **User action**: Run `basic-memory reset` once after updating to rebuild search index
### Internal
- **#502**: Major architecture refactor with composition roots and typed API clients
([`5947f04`](https://github.com/basicmachines-co/basic-memory/commit/5947f04))
- Add composition roots for API, MCP, and CLI entrypoints
- Split deps.py into feature-scoped modules (config, db, projects, repositories, services, importers)
- Add ProjectResolver for unified project selection
- Add SyncCoordinator for centralized sync/watch lifecycle
- Introduce typed API clients for MCP tools (KnowledgeClient, SearchClient, MemoryClient, etc.)
## v0.17.3 (2026-01-03)
### Features
+28 -4
View File
@@ -117,17 +117,38 @@ counter += 1 # track retries for backoff calculation
### Codebase Architecture
See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for detailed architecture documentation.
**Directory Structure:**
- `/alembic` - Alembic db migrations
- `/api` - FastAPI implementation of REST endpoints
- `/cli` - Typer command-line interface
- `/api` - FastAPI REST endpoints + `container.py` composition root
- `/cli` - Typer CLI + `container.py` composition root
- `/deps` - Feature-scoped FastAPI dependencies (config, db, projects, repositories, services, importers)
- `/importers` - Import functionality for Claude, ChatGPT, and other sources
- `/markdown` - Markdown parsing and processing
- `/mcp` - Model Context Protocol server implementation
- `/mcp` - MCP server + `container.py` composition root + `clients/` typed API clients
- `/models` - SQLAlchemy ORM models
- `/repository` - Data access layer
- `/schemas` - Pydantic models for validation
- `/services` - Business logic layer
- `/sync` - File synchronization services
- `/sync` - File synchronization services + `coordinator.py` for lifecycle management
**Composition Roots:**
Each entrypoint (API, MCP, CLI) has a composition root that:
- Reads `ConfigManager` (the only place that reads global config)
- Resolves runtime mode via `RuntimeMode` enum (TEST > CLOUD > LOCAL)
- Provides dependencies to downstream code explicitly
**Typed API Clients (MCP):**
MCP tools use typed clients in `mcp/clients/` to communicate with the API:
- `KnowledgeClient` - Entity CRUD operations
- `SearchClient` - Search operations
- `MemoryClient` - Context building
- `DirectoryClient` - Directory listing
- `ResourceClient` - Resource reading
- `ProjectClient` - Project management
Flow: MCP Tool → Typed Client → HTTP API → Router → Service → Repository
### Development Notes
@@ -146,6 +167,7 @@ counter += 1 # track retries for backoff calculation
- CI runs SQLite and Postgres tests in parallel for faster feedback
- Performance benchmarks are in `test-int/test_sync_performance_benchmark.py`
- Use pytest markers: `@pytest.mark.benchmark` for benchmarks, `@pytest.mark.slow` for slow tests
- **Coverage must stay at 100%**: Write tests for new code. Only use `# pragma: no cover` when tests would require excessive mocking (e.g., TYPE_CHECKING blocks, error handlers that need failure injection, runtime-mode-dependent code paths)
### Async Client Pattern (Important!)
@@ -218,6 +240,8 @@ See SPEC-16 for full context manager refactor details.
- Logout: `basic-memory cloud logout`
- Check cloud status: `basic-memory cloud status`
- Setup cloud sync: `basic-memory cloud setup`
- Manage snapshots: `basic-memory cloud snapshot [create|list|delete|show|browse]`
- Restore from snapshot: `basic-memory cloud restore <path> --snapshot <id>`
### MCP Capabilities
+10 -4
View File
@@ -8,8 +8,13 @@ ARG GID=1000
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
# Set environment variables
# UV_PYTHON_INSTALL_DIR ensures Python is installed to a persistent location
# that survives in the final image (not in /root/.local which gets lost)
# UV_PYTHON_PREFERENCE=only-managed tells uv to use its managed Python version
ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1
PYTHONDONTWRITEBYTECODE=1 \
UV_PYTHON_INSTALL_DIR=/python \
UV_PYTHON_PREFERENCE=only-managed
# Create a group and user with the provided UID/GID
# Check if the GID already exists, if not create appgroup
@@ -19,9 +24,10 @@ RUN (getent group ${GID} || groupadd --gid ${GID} appgroup) && \
# Copy the project into the image
ADD . /app
# Sync the project into a new environment, asserting the lockfile is up to date
# Install Python 3.13 explicitly and sync the project
WORKDIR /app
RUN uv sync --locked
RUN uv python install 3.13
RUN uv sync --locked --python 3.13
# Create necessary directories and set ownership
RUN mkdir -p /app/data/basic-memory /app/.basic-memory && \
@@ -43,4 +49,4 @@ HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
CMD basic-memory --version || exit 1
# Use the basic-memory entrypoint to run the MCP server with default SSE transport
CMD ["basic-memory", "mcp", "--transport", "sse", "--host", "0.0.0.0", "--port", "8000"]
CMD ["basic-memory", "mcp", "--transport", "sse", "--host", "0.0.0.0", "--port", "8000"]
+5 -56
View File
@@ -1,3 +1,4 @@
<!-- mcp-name: io.github.basicmachines-co/basic-memory -->
[![License: AGPL v3](https://img.shields.io/badge/License-AGPL_v3-blue.svg)](https://www.gnu.org/licenses/agpl-3.0)
[![PyPI version](https://badge.fury.io/py/basic-memory.svg)](https://badge.fury.io/py/basic-memory)
[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
@@ -5,7 +6,6 @@
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
![](https://badge.mcpx.dev?type=server 'MCP Server')
![](https://badge.mcpx.dev?type=dev 'MCP Dev')
[![smithery badge](https://smithery.ai/badge/@basicmachines-co/basic-memory)](https://smithery.ai/server/@basicmachines-co/basic-memory)
## 🚀 Basic Memory Cloud is Live!
@@ -13,7 +13,7 @@
- **Early Supporter Pricing:** Early users get 25% off forever.
The open source project continues as always. Cloud just makes it work everywhere.
[Sign up now →](https://basicmemory.com/beta)
[Sign up now →](https://basicmemory.com)
with a 7 day free trial
@@ -23,8 +23,8 @@ Basic Memory lets you build persistent knowledge through natural conversations w
Claude, while keeping everything in simple Markdown files on your computer. It uses the Model Context Protocol (MCP) to
enable any compatible LLM to read and write to your local knowledge base.
- Website: https://basicmachines.co
- Documentation: https://memory.basicmachines.co
- Website: https://basicmemory.com
- Documentation: https://docs.basicmemory.com
## Pick up your conversation right where you left off
@@ -62,24 +62,6 @@ uv tool install basic-memory
You can view shared context via files in `~/basic-memory` (default directory location).
### Alternative Installation via Smithery
You can use [Smithery](https://smithery.ai/server/@basicmachines-co/basic-memory) to automatically configure Basic
Memory for Claude Desktop:
```bash
npx -y @smithery/cli install @basicmachines-co/basic-memory --client claude
```
This installs and configures Basic Memory without requiring manual edits to the Claude Desktop configuration file. The
Smithery server hosts the MCP server component, while your data remains stored locally as Markdown files.
### Glama.ai
<a href="https://glama.ai/mcp/servers/o90kttu9ym">
<img width="380" height="200" src="https://glama.ai/mcp/servers/o90kttu9ym/badge" alt="basic-memory MCP server" />
</a>
## Why Basic Memory?
Most LLM interactions are ephemeral - you ask a question, get an answer, and everything is forgotten. Each conversation
@@ -425,7 +407,7 @@ canvas(nodes, edges, title, folder) - Generate knowledge visualizations
## Futher info
See the [Documentation](https://memory.basicmachines.co/) for more info, including:
See the [Documentation](https://docs.basicmemory.com) for more info, including:
- [Complete User Guide](https://docs.basicmemory.com/user-guide/)
- [CLI tools](https://docs.basicmemory.com/guides/cli-reference/)
@@ -466,39 +448,6 @@ tail -f ~/.basic-memory/basic-memory.log
BASIC_MEMORY_CLOUD_MODE=true uvicorn basic_memory.api.app:app
```
## Telemetry
Basic Memory collects anonymous usage statistics to help improve the software. This follows the [Homebrew model](https://docs.brew.sh/Analytics) - telemetry is on by default with easy opt-out.
**What we collect:**
- App version, Python version, OS, architecture
- Feature usage (which MCP tools and CLI commands are used)
- Error types (sanitized - no file paths or personal data)
**What we NEVER collect:**
- Note content, file names, or paths
- Personal information
- IP addresses
**Opting out:**
```bash
# Disable telemetry
basic-memory telemetry disable
# Check status
basic-memory telemetry status
# Re-enable
basic-memory telemetry enable
```
Or set the environment variable:
```bash
export BASIC_MEMORY_TELEMETRY_ENABLED=false
```
For more details, see the [Telemetry documentation](https://basicmemory.com/telemetry).
## Development
### Running Tests
+412
View File
@@ -0,0 +1,412 @@
# Basic Memory Architecture
This document describes the architectural patterns and composition structure of Basic Memory.
## Overview
Basic Memory is a local-first knowledge management system with three entrypoints:
- **API** - FastAPI REST server for HTTP access
- **MCP** - Model Context Protocol server for LLM integration
- **CLI** - Typer command-line interface
Each entrypoint uses a **composition root** pattern to manage configuration and dependencies.
## Composition Roots
### What is a Composition Root?
A composition root is the single place in an application where dependencies are wired together. In Basic Memory, each entrypoint has its own composition root that:
1. Reads configuration from `ConfigManager`
2. Resolves runtime mode (cloud/local/test)
3. Creates and provides dependencies to downstream code
**Key principle**: Only composition roots read global configuration. All other modules receive configuration explicitly.
### Container Structure
Each entrypoint has a container dataclass in its package:
```
src/basic_memory/
├── api/
│ └── container.py # ApiContainer
├── mcp/
│ └── container.py # McpContainer
├── cli/
│ └── container.py # CliContainer
└── runtime.py # RuntimeMode enum and resolver
```
### Container Pattern
All containers follow the same structure:
```python
@dataclass
class Container:
config: BasicMemoryConfig
mode: RuntimeMode
@classmethod
def create(cls) -> "Container":
"""Create container by reading ConfigManager."""
config = ConfigManager().config
mode = resolve_runtime_mode(
cloud_mode_enabled=config.cloud_mode_enabled,
is_test_env=config.is_test_env,
)
return cls(config=config, mode=mode)
@property
def some_computed_property(self) -> bool:
"""Derived values based on config and mode."""
return self.mode.is_local and self.config.some_setting
# Module-level singleton
_container: Container | None = None
def get_container() -> Container:
if _container is None:
raise RuntimeError("Container not initialized")
return _container
def set_container(container: Container) -> None:
global _container
_container = container
```
### Runtime Mode Resolution
The `RuntimeMode` enum centralizes mode detection:
```python
class RuntimeMode(Enum):
LOCAL = "local"
CLOUD = "cloud"
TEST = "test"
@property
def is_cloud(self) -> bool:
return self == RuntimeMode.CLOUD
@property
def is_local(self) -> bool:
return self == RuntimeMode.LOCAL
@property
def is_test(self) -> bool:
return self == RuntimeMode.TEST
```
Resolution follows this precedence: **TEST > CLOUD > LOCAL**
```python
def resolve_runtime_mode(cloud_mode_enabled: bool, is_test_env: bool) -> RuntimeMode:
if is_test_env:
return RuntimeMode.TEST
if cloud_mode_enabled:
return RuntimeMode.CLOUD
return RuntimeMode.LOCAL
```
## Dependencies Package
### Structure
The `deps/` package provides FastAPI dependencies organized by feature:
```
src/basic_memory/deps/
├── __init__.py # Re-exports for backwards compatibility
├── config.py # Configuration access
├── db.py # Database/session management
├── projects.py # Project resolution
├── repositories.py # Data access layer
├── services.py # Business logic layer
└── importers.py # Import functionality
```
### Usage in Routers
```python
from basic_memory.deps.services import get_entity_service
from basic_memory.deps.projects import get_project_config
@router.get("/entities/{id}")
async def get_entity(
id: int,
entity_service: EntityService = Depends(get_entity_service),
project: ProjectConfig = Depends(get_project_config),
):
return await entity_service.get(id)
```
### Backwards Compatibility
The old `deps.py` file still exists as a thin re-export shim:
```python
# deps.py - backwards compatibility shim
from basic_memory.deps import *
```
New code should import from specific submodules (`basic_memory.deps.services`) for clarity.
## MCP Tools Architecture
### Typed API Clients
MCP tools communicate with the API through typed clients that encapsulate HTTP paths and response validation:
```
src/basic_memory/mcp/clients/
├── __init__.py # Re-exports all clients
├── base.py # BaseClient with common logic
├── knowledge.py # KnowledgeClient - entity CRUD
├── search.py # SearchClient - search operations
├── memory.py # MemoryClient - context building
├── directory.py # DirectoryClient - directory listing
├── resource.py # ResourceClient - resource reading
└── project.py # ProjectClient - project management
```
### Client Pattern
Each client encapsulates API paths and validates responses:
```python
class KnowledgeClient(BaseClient):
"""Client for knowledge/entity operations."""
async def resolve_entity(self, identifier: str) -> int:
"""Resolve identifier to entity ID."""
response = await call_get(
self.http_client,
f"{self._base_path}/resolve/{identifier}",
)
return int(response.text)
async def get_entity(self, entity_id: int) -> EntityResponse:
"""Get entity by ID."""
response = await call_get(
self.http_client,
f"{self._base_path}/entities/{entity_id}",
)
return EntityResponse.model_validate(response.json())
```
### Tool → Client → API Flow
```
MCP Tool (thin adapter)
Typed Client (encapsulates paths, validates responses)
HTTP API (FastAPI router)
Service Layer (business logic)
Repository Layer (data access)
```
Example tool using typed client:
```python
@mcp.tool()
async def search_notes(query: str, project: str | None = None) -> SearchResponse:
async with get_client() as client:
active_project = await get_active_project(client, project)
# Import client inside function to avoid circular imports
from basic_memory.mcp.clients import SearchClient
search_client = SearchClient(client, active_project.external_id)
return await search_client.search(query)
```
## Sync Coordination
### SyncCoordinator
The `SyncCoordinator` centralizes sync/watch lifecycle management:
```python
@dataclass
class SyncCoordinator:
"""Coordinates file sync and watch operations."""
status: SyncStatus = SyncStatus.NOT_STARTED
sync_task: asyncio.Task | None = None
watch_service: WatchService | None = None
async def start(self, ...):
"""Start sync and watch operations."""
async def stop(self):
"""Stop all sync operations gracefully."""
def get_status_info(self) -> dict:
"""Get current sync status for observability."""
```
### Status Enum
```python
class SyncStatus(Enum):
NOT_STARTED = "not_started"
STARTING = "starting"
RUNNING = "running"
STOPPING = "stopping"
STOPPED = "stopped"
ERROR = "error"
```
## Project Resolution
### ProjectResolver
Unified project selection across all entrypoints:
```python
class ProjectResolver:
"""Resolves which project to use based on context."""
def resolve(
self,
explicit_project: str | None = None,
) -> ResolvedProject:
"""Resolve project using three-tier hierarchy:
1. Explicit project parameter
2. Default project from config
3. Single available project
"""
```
### Resolution Modes
```python
class ResolutionMode(Enum):
EXPLICIT = "explicit" # User specified project
DEFAULT = "default" # Using configured default
SINGLE_PROJECT = "single" # Only one project exists
FALLBACK = "fallback" # Using first available
```
## Testing Patterns
### Container Testing
Each container has corresponding tests:
```
tests/
├── api/test_api_container.py
├── mcp/test_mcp_container.py
└── cli/test_cli_container.py
```
Tests verify:
- Container creation from config
- Runtime mode properties
- Container accessor functions (get/set)
### Mocking Typed Clients
When testing MCP tools, mock at the client level:
```python
def test_search_notes(monkeypatch):
import basic_memory.mcp.clients as clients_mod
class MockSearchClient:
async def search(self, query):
return SearchResponse(results=[...])
monkeypatch.setattr(clients_mod, "SearchClient", MockSearchClient)
```
## Design Principles
### 1. Explicit Dependencies
Modules receive configuration explicitly rather than reading globals:
```python
# Good - explicit injection
async def sync_files(config: BasicMemoryConfig):
...
# Avoid - hidden global access
async def sync_files():
config = ConfigManager().config # Hidden coupling
```
### 2. Single Responsibility
Each layer has a clear responsibility:
- **Containers**: Wire dependencies
- **Clients**: Encapsulate HTTP communication
- **Services**: Business logic
- **Repositories**: Data access
- **Tools/Routers**: Thin adapters
### 3. Deferred Imports
To avoid circular imports, typed clients are imported inside functions:
```python
async def my_tool():
async with get_client() as client:
# Import here to avoid circular dependency
from basic_memory.mcp.clients import KnowledgeClient
knowledge_client = KnowledgeClient(client, project_id)
```
### 4. Backwards Compatibility
When refactoring, maintain backwards compatibility via shims:
```python
# Old module becomes a shim
from basic_memory.new_location import *
# Docstring explains migration path
"""
DEPRECATED: Import from basic_memory.new_location instead.
This shim will be removed in a future version.
"""
```
## File Organization
```
src/basic_memory/
├── api/
│ ├── container.py # API composition root
│ ├── routers/ # FastAPI routers
│ └── ...
├── mcp/
│ ├── container.py # MCP composition root
│ ├── clients/ # Typed API clients
│ ├── tools/ # MCP tool definitions
│ └── server.py # MCP server setup
├── cli/
│ ├── container.py # CLI composition root
│ ├── app.py # Typer app
│ └── commands/ # CLI command groups
├── deps/
│ ├── config.py # Config dependencies
│ ├── db.py # Database dependencies
│ ├── projects.py # Project dependencies
│ ├── repositories.py # Repository dependencies
│ ├── services.py # Service dependencies
│ └── importers.py # Importer dependencies
├── sync/
│ ├── coordinator.py # SyncCoordinator
│ └── ...
├── runtime.py # RuntimeMode resolution
├── project_resolver.py # Unified project selection
└── config.py # Configuration management
```
+1 -3
View File
@@ -8,9 +8,8 @@ To keep the default CI signal **stable and meaningful**, the default `pytest` co
- highly environment-dependent (OS/DB tuning)
- inherently interactive (CLI)
- background-task orchestration (watchers/sync runners)
- external analytics
### Whats excluded (and why)
### What's excluded (and why)
Coverage excludes are configured in `pyproject.toml` under `[tool.coverage.report].omit`.
@@ -19,7 +18,6 @@ Current exclusions include:
- `src/basic_memory/db.py`: platform/backend tuning paths (SQLite/Postgres/Windows), covered by integration tests and targeted runs.
- `src/basic_memory/services/initialization.py`: startup orchestration/background tasks; covered indirectly by app/MCP entrypoints.
- `src/basic_memory/sync/sync_service.py`: heavy filesystem↔DB integration; validated in integration suite (not enforced in unit coverage).
- `src/basic_memory/telemetry.py`: external analytics; exercised lightly but excluded from strict coverage gate.
### Recommended additional runs
+25 -4
View File
@@ -204,9 +204,14 @@ release version:
echo "📝 Updating version in __init__.py..."
sed -i.bak "s/__version__ = \".*\"/__version__ = \"$VERSION_NUM\"/" src/basic_memory/__init__.py
rm -f src/basic_memory/__init__.py.bak
# Update version in server.json (MCP registry metadata)
echo "📝 Updating version in server.json..."
sed -i.bak "s/\"version\": \"[^\"]*\"/\"version\": \"$VERSION_NUM\"/g" server.json
rm -f server.json.bak
# Commit version update
git add src/basic_memory/__init__.py
git add src/basic_memory/__init__.py server.json
git commit -m "chore: update version to $VERSION_NUM for {{version}} release"
# Create and push tag
@@ -220,6 +225,12 @@ release version:
echo "✅ Release {{version}} created successfully!"
echo "📦 GitHub Actions will build and publish to PyPI"
echo "🔗 Monitor at: https://github.com/basicmachines-co/basic-memory/actions"
echo ""
echo "📝 REMINDER: Post-release tasks:"
echo " 1. docs.basicmemory.com - Add release notes to src/pages/latest-releases.mdx"
echo " 2. basicmachines.co - Update version in src/components/sections/hero.tsx"
echo " 3. MCP Registry - Run: mcp-publisher publish"
echo " See: .claude/commands/release/release.md for detailed instructions"
# Create a beta release (e.g., just beta v0.13.2b1)
beta version:
@@ -264,9 +275,14 @@ beta version:
echo "📝 Updating version in __init__.py..."
sed -i.bak "s/__version__ = \".*\"/__version__ = \"$VERSION_NUM\"/" src/basic_memory/__init__.py
rm -f src/basic_memory/__init__.py.bak
# Update version in server.json (MCP registry metadata)
echo "📝 Updating version in server.json..."
sed -i.bak "s/\"version\": \"[^\"]*\"/\"version\": \"$VERSION_NUM\"/g" server.json
rm -f server.json.bak
# Commit version update
git add src/basic_memory/__init__.py
git add src/basic_memory/__init__.py server.json
git commit -m "chore: update version to $VERSION_NUM for {{version}} beta release"
# Create and push tag
@@ -281,6 +297,11 @@ beta version:
echo "📦 GitHub Actions will build and publish to PyPI as pre-release"
echo "🔗 Monitor at: https://github.com/basicmachines-co/basic-memory/actions"
echo "📥 Install with: uv tool install basic-memory --pre"
echo ""
echo "📝 REMINDER: For stable releases, update documentation sites:"
echo " 1. docs.basicmemory.com - Add release notes to src/pages/latest-releases.mdx"
echo " 2. basicmachines.co - Update version in src/components/sections/hero.tsx"
echo " See: .claude/commands/release/release.md for detailed instructions"
# List all available recipes
default:
+6 -7
View File
@@ -14,7 +14,7 @@ dependencies = [
"typer>=0.9.0",
"aiosqlite>=0.20.0",
"greenlet>=3.1.1",
"pydantic[email,timezone]>=2.10.3",
"pydantic[email,timezone]>=2.12.0",
"mcp>=1.23.1",
"pydantic-settings>=2.6.1",
"loguru>=0.7.3",
@@ -29,7 +29,7 @@ dependencies = [
"alembic>=1.14.1",
"pillow>=11.1.0",
"pybars3>=0.9.7",
"fastmcp==2.12.3", # Pinned - 2.14.x breaks MCP tools visibility (issue #463)
"fastmcp==2.12.3", # Pinned - 2.14.x breaks MCP tools visibility (issue #463)
"pyjwt>=2.10.1",
"python-dotenv>=1.1.0",
"pytest-aio>=1.9.0",
@@ -41,7 +41,9 @@ dependencies = [
"mdformat>=0.7.22",
"mdformat-gfm>=0.3.7",
"mdformat-frontmatter>=2.0.8",
"openpanel>=0.0.1", # Anonymous usage telemetry (Homebrew-style opt-out)
"sniffio>=1.3.1",
"anyio>=4.10.0",
"httpx>=0.28.0",
]
@@ -88,6 +90,7 @@ dev = [
"freezegun>=1.5.5",
"testcontainers[postgres]>=4.0.0",
"psycopg>=3.2.0",
"pyright>=1.1.408",
]
[tool.hatch.version]
@@ -139,9 +142,5 @@ omit = [
"*/db.py", # Backend/runtime-dependent (sqlite/postgres/windows tuning); validated via integration tests
"*/services/initialization.py", # Startup orchestration + background tasks (watchers); exercised indirectly in entrypoints
"*/sync/sync_service.py", # Heavy filesystem/db integration; covered by integration suite, not enforced in unit coverage
"*/telemetry.py", # External analytics; tested lightly, excluded from strict coverage target
"*/services/migration_service.py", # Complex migration scenarios
]
[tool.logfire]
ignore_no_config = true
+25
View File
@@ -0,0 +1,25 @@
{
"$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json",
"name": "io.github.basicmachines-co/basic-memory",
"description": "Local-first knowledge management with bi-directional LLM sync via Markdown files.",
"repository": {
"url": "https://github.com/basicmachines-co/basic-memory.git",
"source": "github"
},
"version": "0.17.8",
"packages": [
{
"registryType": "pypi",
"identifier": "basic-memory",
"version": "0.17.8",
"runtimeHint": "uvx",
"runtimeArguments": [
{"type": "positional", "value": "basic-memory"},
{"type": "positional", "value": "mcp"}
],
"transport": {
"type": "stdio"
}
}
]
}
@@ -1,156 +0,0 @@
---
title: 'SPEC-1: Specification-Driven Development Process'
type: spec
permalink: specs/spec-1-specification-driven-development-process
tags:
- process
- specification
- development
- meta
---
# SPEC-1: Specification-Driven Development Process
## Why
We're implementing specification-driven development to solve the complexity and circular refactoring issues in our web development process.
Instead of getting lost in framework details and type gymnastics, we start with clear specifications that drive implementation.
The default approach of adhoc development with AI agents tends to result in:
- Circular refactoring cycles
- Fighting framework complexity
- Lost context between sessions
- Unclear requirements and scope
## What
This spec defines our process for using basic-memory as the specification engine to build basic-memory-cloud.
We're creating a recursive development pattern where basic-memory manages the specs that drive the development of basic-memory-cloud.
**Affected Areas:**
- All future component development
- Architecture decisions
- Agent collaboration workflows
- Knowledge management and context preservation
## How (High Level)
### Specification Structure
Name: Spec names should be numbered sequentially, followed by a description eg. `SPEC-X - Simple Description.md`.
See: [[Spec-2: Slash Commands Reference]]
Every spec is a complete thought containing:
- **Why**: The reasoning and problem being solved
- **What**: What is affected or changed
- **How**: High-level approach to implementation
- **How to Evaluate**: Testing/validation procedure
- Additional context as needed
### Living Specification Format
Specifications are **living documents** that evolve throughout implementation:
**Progress Tracking:**
- **Completed items**: Use ✅ checkmark emoji for implemented features
- **Pending items**: Use `- [ ]` GitHub-style checkboxes for remaining tasks
- **In-progress items**: Use `- [x]` when work is actively underway
**Status Philosophy:**
- **Avoid static status headers** like "COMPLETE" or "IN PROGRESS" that become stale
- **Use checklists within content** to show granular implementation progress
- **Keep specs informative** while providing clear progress visibility
- **Update continuously** as understanding and implementation evolve
**Example Format:**
```markdown
### ComponentName
- ✅ Basic functionality implemented
- ✅ Props and events defined
- - [ ] Add sorting controls
- - [ ] Improve accessibility
- - [x] Currently implementing responsive design
```
This creates **git-friendly progress tracking** where `[ ]` easily becomes `[x]` or ✅ when completed, and specs remain valuable throughout the development lifecycle.
## Claude Code
We will leverage Claude Code capabilities to make the process semi-automated.
- Slash commands: define repeatable steps in the process (create spec, implement, review, etc)
- Agents: define roles to carry out instructions (front end developer, baskend developer, etc)
- MCP tools: enable agents to implement specs via actions (write code, test, etc)
### Workflow
1. **Create**: Write spec as complete thought in `/specs` folder
2. **Discuss**: Iterate and refine through agent collaboration
3. **Implement**: Hand spec to appropriate specialist agent
4. **Validate**: Review implementation against spec criteria
5. **Document**: Update spec with learnings and decisions
### Slash Commands
Claude slash commands are used to manage the flow.
These are simple instructions to help make the process uniform.
They can be updated and refined as needed.
- `/spec create [name]` - Create new specification
- `/spec status` - Show current spec states
- `/spec implement [name]` - Hand to appropriate agent
- `/spec review [name]` - Validate implementation
### Agent Orchestration
Agents are defined with clear roles, for instance:
- **system-architect**: Creates high-level specs, ADRs, architectural decisions
- **vue-developer**: Component specs, UI patterns, frontend architecture
- **python-developer**: Implementation specs, technical details, backend logic
-
- Each agent reads/updates specs through basic-memory tools.
## How to Evaluate
### Success Criteria
- Specs provide clear, actionable guidance for implementation
- Reduced circular refactoring and scope creep
- Persistent context across development sessions
- Clean separation between "what/why" and implementation details
- Specs record a history of what happened and why for historical context
### Testing Procedure
1. Create a spec for an existing problematic component
2. Have an agent implement following only the spec
3. Compare result quality and development speed vs. ad-hoc approach
4. Measure context preservation across sessions
5. Evaluate spec clarity and completeness
### Metrics
- Time from spec to working implementation
- Number of refactoring cycles required
- Agent understanding of requirements
- Spec reusability for similar components
## Notes
- Start simple: specs are just complete thoughts, not heavy processes
- Use basic-memory's knowledge graph to link specs, decisions, components
- Let the process evolve naturally based on what works
- Focus on solving the actual problem: Manage complexity in development
## Observations
- [problem] Web development without clear goals and documentation circular refactoring cycles #complexity
- [solution] Specification-driven development reduces scope creep and context loss #process-improvement
- [pattern] basic-memory as specification engine creates recursive development loop #meta-development
- [workflow] Five-step process: Create → Discuss → Implement → Validate → Document #methodology
- [tool] Slash commands provide uniform process automation #automation
- [agent-pattern] Three specialized agents handle different implementation domains #specialization
- [success-metric] Time from spec to working implementation measures process efficiency #measurement
- [learning] Process should evolve naturally based on what works in practice #adaptation
- [format] Living specifications use checklists for progress tracking instead of static status headers #documentation
- [evolution] Specs evolve throughout implementation maintaining value as working documents #continuous-improvement
## Relations
- spec [[Spec-2: Slash Commands Reference]]
- spec [[Spec-3: Agent Definitions]]
@@ -1,569 +0,0 @@
---
title: 'SPEC-10: Unified Deployment Workflow and Event Tracking'
type: spec
permalink: specs/spec-10-unified-deployment-workflow-event-tracking
tags:
- workflow
- deployment
- event-sourcing
- architecture
- simplification
---
# SPEC-10: Unified Deployment Workflow and Event Tracking
## Why
We replaced a complex multi-workflow system with DBOS orchestration that was proving to be more trouble than it was worth. The previous architecture had four separate workflows (`tenant_provisioning`, `tenant_update`, `tenant_deployment`, `tenant_undeploy`) with overlapping logic, complex state management, and fragmented event tracking. DBOS added unnecessary complexity without providing sufficient value, leading to harder debugging and maintenance.
**Problems Solved:**
- **Framework Complexity**: DBOS configuration overhead and fighting framework limitations
- **Code Duplication**: Multiple workflows implementing similar operations with duplicate logic
- **Poor Observability**: Fragmented event tracking across workflow boundaries
- **Maintenance Overhead**: Complex orchestration for fundamentally simple operations
- **Debugging Difficulty**: Framework abstractions hiding simple Python stack traces
## What
This spec documents the architectural simplification that consolidates tenant lifecycle management into a unified system with comprehensive event tracking.
**Affected Areas:**
- Tenant deployment workflows (provisioning, updates, undeploying)
- Event sourcing and workflow tracking infrastructure
- API endpoints for tenant operations
- Database schema for workflow and event correlation
- Integration testing for tenant lifecycle operations
**Key Changes:**
- **Removed DBOS entirely** - eliminated framework dependency and complexity
- **Consolidated 4 workflows → 2 unified deployment workflows (deploy/undeploy)**
- **Added workflow tracking system** with complete event correlation
- **Simplified API surface** - single `/deploy` endpoint handles all scenarios
- **Enhanced observability** through event sourcing with workflow grouping
## How (High Level)
### Architectural Philosophy
**Embrace simplicity over framework complexity** - use well-structured Python with proper database design instead of complex orchestration frameworks.
### Core Components
#### 1. Unified Deployment Workflow
```python
class TenantDeploymentWorkflow:
async def deploy_tenant_workflow(self, tenant_id: str, workflow_id: UUID, image_tag: str = None):
# Single workflow handles both initial provisioning AND updates
# Each step is idempotent and handles its own error recovery
# Database transactions provide the durability we need
await self.start_deployment_step(workflow_id, tenant_uuid, image_tag)
await self.create_fly_app_step(workflow_id, tenant_uuid)
await self.create_bucket_step(workflow_id, tenant_uuid)
await self.deploy_machine_step(workflow_id, tenant_uuid, image_tag)
await self.complete_deployment_step(workflow_id, tenant_uuid, image_tag, deployment_time)
```
**Key Benefits:**
- **Handles both provisioning and updates** in single workflow
- **Idempotent operations** - safe to retry any step
- **Clean error handling** via simple Python exceptions
- **Resumable** - can restart from any failed step
#### 2. Workflow Tracking System
**Database Schema:**
```sql
CREATE TABLE workflow (
id UUID PRIMARY KEY,
workflow_type VARCHAR(50) NOT NULL, -- 'tenant_deployment', 'tenant_undeploy'
tenant_id UUID REFERENCES tenant(id),
status VARCHAR(20) DEFAULT 'running', -- 'running', 'completed', 'failed'
workflow_metadata JSONB DEFAULT '{}' -- image_tag, etc.
);
ALTER TABLE event ADD COLUMN workflow_id UUID REFERENCES workflow(id);
```
**Event Correlation:**
- Every workflow operation generates events tagged with `workflow_id`
- Complete audit trail from workflow start to completion
- Events grouped by workflow for easy reconstruction of operations
#### 3. Parameter Standardization
All workflow methods follow consistent signature pattern:
```python
async def method_name(self, session: AsyncSession, workflow_id: UUID | None, tenant_id: UUID, ...)
```
**Benefits:**
- **Consistent event tagging** - all events properly correlated
- **Clear method contracts** - workflow_id always first parameter
- **Type safety** - proper UUID handling throughout
### Implementation Strategy
#### Phase 1: Workflow Consolidation ✅ COMPLETED
- [x] **Remove DBOS dependency** - eliminated dbos_config.py and all DBOS imports
- [x] **Create unified TenantDeploymentWorkflow** - handles both provisioning and updates
- [x] **Remove legacy workflows** - deleted tenant_provisioning.py, tenant_update.py
- [x] **Simplify API endpoints** - consolidated to single `/deploy` endpoint
- [x] **Update integration tests** - comprehensive edge case testing
#### Phase 2: Workflow Tracking System ✅ COMPLETED
- [x] **Database migration** - added workflow table and event.workflow_id foreign key
- [x] **Workflow repository** - CRUD operations for workflow records
- [x] **Event correlation** - all workflow events tagged with workflow_id
- [x] **Comprehensive testing** - workflow lifecycle and event grouping tests
#### Phase 3: Parameter Standardization ✅ COMPLETED
- [x] **Standardize method signatures** - workflow_id as first parameter pattern
- [x] **Fix event tagging** - ensure all workflow events properly correlated
- [x] **Update service methods** - consistent parameter order across tenant_service
- [x] **Integration test validation** - verify complete event sequences
### Architectural Benefits
#### Code Simplification
- **39 files changed**: 2,247 additions, 3,256 deletions (net -1,009 lines)
- **Eliminated framework complexity** - no more DBOS configuration or abstractions
- **Consolidated logic** - single deployment workflow vs 4 separate workflows
- **Cleaner API surface** - unified endpoint vs multiple workflow-specific endpoints
#### Enhanced Observability
- **Complete event correlation** - every workflow event tagged with workflow_id
- **Audit trail reconstruction** - can trace entire tenant lifecycle through events
- **Workflow status tracking** - running/completed/failed states in database
- **Comprehensive testing** - edge cases covered with real infrastructure
#### Operational Benefits
- **Simpler debugging** - plain Python stack traces vs framework abstractions
- **Reduced dependencies** - one less complex framework to maintain
- **Better error handling** - explicit exception handling vs framework magic
- **Easier maintenance** - straightforward Python code vs orchestration complexity
## How to Evaluate
### Success Criteria
#### Functional Completeness ✅ VERIFIED
- [x] **Unified deployment workflow** handles both initial provisioning and updates
- [x] **Undeploy workflow** properly integrated with event tracking
- [x] **All operations idempotent** - safe to retry any step without duplication
- [x] **Complete tenant lifecycle** - provision → active → update → undeploy
#### Event Tracking and Correlation ✅ VERIFIED
- [x] **All workflow events tagged** with proper workflow_id
- [x] **Event sequence verification** - tests assert exact event order and content
- [x] **Workflow grouping** - events can be queried by workflow_id for complete audit trail
- [x] **Cross-workflow isolation** - deployment vs undeploy events properly separated
#### Database Schema and Performance ✅ VERIFIED
- [x] **Migration applied** - workflow table and event.workflow_id column created
- [x] **Proper indexing** - performance optimized queries on workflow_type, tenant_id, status
- [x] **Foreign key constraints** - referential integrity between workflows and events
- [x] **Database triggers** - updated_at timestamp automation
#### Test Coverage ✅ COMPREHENSIVE
- [x] **Unit tests**: 4 workflow tracking tests covering lifecycle and event grouping
- [x] **Integration tests**: Real infrastructure testing with Fly.io resources
- [x] **Edge case coverage**: Failed deployments, partial state recovery, resource conflicts
- [x] **Event sequence verification**: Exact event order and content validation
### Testing Procedure
#### Unit Test Validation ✅ PASSING
```bash
cd apps/cloud && pytest tests/test_workflow_tracking.py -v
# 4/4 tests passing - workflow lifecycle and event grouping
```
#### Integration Test Validation ✅ PASSING
```bash
cd apps/cloud && pytest tests/integration/test_tenant_workflow_deployment_integration.py -v
cd apps/cloud && pytest tests/integration/test_tenant_workflow_undeploy_integration.py -v
# Comprehensive real infrastructure testing with actual Fly.io resources
# Tests provision → deploy → update → undeploy → cleanup cycles
```
### Performance Metrics
#### Code Metrics ✅ ACHIEVED
- **Net code reduction**: -1,009 lines (3,256 deletions, 2,247 additions)
- **Workflow consolidation**: 4 workflows → 1 unified deployment workflow
- **Dependency reduction**: Removed DBOS framework dependency entirely
- **API simplification**: Multiple endpoints → single `/deploy` endpoint
#### Operational Metrics ✅ VERIFIED
- **Event correlation**: 100% of workflow events properly tagged with workflow_id
- **Audit trail completeness**: Full tenant lifecycle traceable through event sequences
- **Error handling**: Clean Python exceptions vs framework abstractions
- **Debugging simplicity**: Direct stack traces vs orchestration complexity
### Implementation Status: ✅ COMPLETE
All phases completed successfully with comprehensive testing and verification:
**Phase 1 - Workflow Consolidation**: ✅ COMPLETE
- Removed DBOS dependency and consolidated workflows
- Unified deployment workflow handles all scenarios
- Comprehensive integration testing with real infrastructure
**Phase 2 - Workflow Tracking**: ✅ COMPLETE
- Database schema implemented with proper indexing
- Event correlation system fully functional
- Complete audit trail capability verified
**Phase 3 - Parameter Standardization**: ✅ COMPLETE
- Consistent method signatures across all workflow methods
- All events properly tagged with workflow_id
- Type safety verified across entire codebase
**Phase 4 - Asynchronous Job Queuing**:
**Goal**: Transform synchronous deployment workflows into background jobs for better user experience and system reliability.
**Current Problem**:
- Deployment API calls are synchronous - users wait for entire tenant provisioning (30-60 seconds)
- No retry mechanism for failed operations
- HTTP timeouts on long-running deployments
- Poor user experience during infrastructure provisioning
**Solution**: Redis-backed job queue with arq for reliable background processing
#### Architecture Overview
```python
# API Layer: Return immediately with job tracking
@router.post("/{tenant_id}/deploy")
async def deploy_tenant(tenant_id: UUID):
# Create workflow record in Postgres
workflow = await workflow_repo.create_workflow("tenant_deployment", tenant_id)
# Enqueue job in Redis
job = await arq_pool.enqueue_job('deploy_tenant_task', tenant_id, workflow.id)
# Return job ID immediately
return {"job_id": job.job_id, "workflow_id": workflow.id, "status": "queued"}
# Background Worker: Process via existing unified workflow
async def deploy_tenant_task(ctx, tenant_id: str, workflow_id: str):
# Existing workflow logic - zero changes needed!
await workflow_manager.deploy_tenant(UUID(tenant_id), workflow_id=UUID(workflow_id))
```
#### Implementation Tasks
**Phase 4.1: Core Job Queue Setup** ✅ COMPLETED
- [x] **Add arq dependency** - integrated Redis job queue with existing infrastructure
- [x] **Create job definitions** - wrapped existing deployment/undeploy workflows as arq tasks
- [x] **Update API endpoints** - updated provisioning endpoints to return job IDs instead of waiting for completion
- [x] **JobQueueService implementation** - service layer for job enqueueing and status tracking
- [x] **Job status tracking** - integrated with existing workflow table for status updates
- [x] **Comprehensive testing** - 18 tests covering positive, negative, and edge cases
**Phase 4.2: Background Worker Implementation** ✅ COMPLETED
- [x] **Job status API** - GET /jobs/{job_id}/status endpoint integrated with JobQueueService
- [x] **Background worker process** - arq worker to process queued jobs with proper settings and Redis configuration
- [x] **Worker settings and configuration** - WorkerSettings class with proper timeouts, max jobs, and error handling
- [x] **Fix API endpoints** - updated job status API to use JobQueueService instead of direct Redis access
- [x] **Integration testing** - comprehensive end-to-end testing with real ARQ workers and Fly.io infrastructure
- [x] **Worker entry points** - dual-purpose entrypoint.sh script and __main__.py module support for both API and worker processes
- [x] **Test fixture updates** - fixed all API and service test fixtures to work with job queue dependencies
- [x] **AsyncIO event loop fixes** - resolved event loop issues in integration tests for subprocess worker compatibility
- [x] **Complete test coverage** - all 46 tests passing across unit, integration, and API test suites
- [x] **Type safety verification** - 0 type checking errors across entire ARQ job queue implementation
#### Phase 4.2 Implementation Summary ✅ COMPLETE
**Core ARQ Job Queue System:**
- **JobQueueService** - Centralized service for job enqueueing, status tracking, and Redis pool management
- **deployment_jobs.py** - ARQ job functions that wrap existing deployment/undeploy workflows
- **Worker Settings** - Production-ready ARQ configuration with proper timeouts and error handling
- **Dual-Process Architecture** - Single Docker image with entrypoint.sh supporting both API and worker modes
**Key Files Added:**
- `apps/cloud/src/basic_memory_cloud/jobs/` - Complete job queue implementation (7 files)
- `apps/cloud/entrypoint.sh` - Dual-purpose Docker container entry point
- `apps/cloud/tests/integration/test_worker_integration.py` - Real infrastructure integration tests
- `apps/cloud/src/basic_memory_cloud/schemas/job_responses.py` - API response schemas
**API Integration:**
- Provisioning endpoints return job IDs immediately instead of blocking for 60+ seconds
- Job status API endpoints for real-time monitoring of deployment progress
- Proper error handling and job failure scenarios with detailed error messages
**Testing Achievement:**
- **46 total tests passing** across all test suites (unit, integration, API, services)
- **Real infrastructure testing** - ARQ workers process actual Fly.io deployments
- **Event loop safety** - Fixed asyncio issues for subprocess worker compatibility
- **Test fixture updates** - All fixtures properly support job queue dependencies
- **Type checking** - 0 errors across entire codebase
**Technical Metrics:**
- **38 files changed** - +1,736 insertions, -334 deletions
- **Integration test runtime** - ~18 seconds with real ARQ workers and Fly.io verification
- **Event loop isolation** - Proper async session management for subprocess compatibility
- **Redis integration** - Production-ready Redis configuration with connection pooling
**Phase 4.3: Production Hardening** ✅ COMPLETED
- [x] **Configure Upstash Redis** - production Redis setup on Fly.io
- [x] **Retry logic for external APIs** - exponential backoff for flaky Tigris IAM operations
- [x] **Monitoring and observability** - comprehensive Redis queue monitoring with CLI tools
- [x] **Error handling improvements** - graceful handling of expected API errors with appropriate log levels
- [x] **CLI tooling enhancements** - bulk update commands for CI/CD automation
- [x] **Documentation improvements** - comprehensive monitoring guide with Redis patterns
- [x] **Job uniqueness** - ARQ-based duplicate prevention for tenant operations
- [ ] **Worker scaling** - multiple arq workers for parallel job processing
- [ ] **Job persistence** - ensure jobs survive Redis/worker restarts
- [ ] **Error alerting** - notifications for failed deployment jobs
**Phase 4.4: Advanced Features** (Future)
- [ ] **Job scheduling** - deploy tenants at specific times
- [ ] **Priority queues** - urgent deployments processed first
- [ ] **Batch operations** - bulk tenant deployments
- [ ] **Job dependencies** - deployment → configuration → activation chains
#### Benefits Achieved ✅ REALIZED
**User Experience Improvements:**
- **Immediate API responses** - users get job ID instantly vs waiting 60+ seconds for deployment completion
- **Real-time job tracking** - status API provides live updates on deployment progress
- **Better error visibility** - detailed error messages and job failure tracking
- **CI/CD automation ready** - bulk update commands for automated tenant deployments
**System Reliability:**
- **Redis persistence** - jobs survive Redis/worker restarts with proper queue durability
- **Idempotent job processing** - jobs can be safely retried without side effects
- **Event loop isolation** - worker processes operate independently from API server
- **Retry resilience** - exponential backoff for flaky external API calls (3 attempts, 1s/2s delays)
- **Graceful error handling** - expected API errors logged at INFO level, unexpected at ERROR level
- **Job uniqueness** - prevent duplicate tenant operations with ARQ's built-in uniqueness feature
**Operational Benefits:**
- **Horizontal scaling ready** - architecture supports adding more workers for parallel processing
- **Comprehensive testing** - real infrastructure integration tests ensure production reliability
- **Type safety** - full type checking prevents runtime errors in job processing
- **Clean separation** - API and worker processes use same codebase with different entry points
- **Queue monitoring** - Redis CLI integration for real-time queue activity monitoring
- **Comprehensive documentation** - detailed monitoring guide with Redis pattern explanations
**Development Benefits:**
- **Zero workflow changes** - existing deployment/undeploy workflows work unchanged as background jobs
- **Async/await native** - modern Python asyncio patterns throughout the implementation
- **Event correlation preserved** - all existing workflow tracking and event sourcing continues to work
- **Enhanced CLI tooling** - unified tenant commands with proper endpoint routing
- **Database integrity** - proper foreign key constraint handling in tenant deletion
#### Infrastructure Requirements
- **Local**: Redis via docker-compose (already exists) ✅
- **Production**: Upstash Redis on Fly.io (already configured) ✅
- **Workers**: arq worker processes (new deployment target)
- **Monitoring**: Job status dashboard (simple web interface)
#### API Evolution
```python
# Before: Synchronous (blocks for 60+ seconds)
POST /tenant/{id}/deploy {status: "active", machine_id: "..."}
# After: Asynchronous (returns immediately)
POST /tenant/{id}/deploy {job_id: "uuid", workflow_id: "uuid", status: "queued"}
GET /jobs/{job_id}/status {status: "running", progress: "deploying_machine", workflow_id: "uuid"}
GET /workflows/{workflow_id}/events [...] # Existing event tracking works unchanged
```
**Technology Choice**: **arq (Redis)** over pgqueuer
- **Existing Redis infrastructure** - Upstash + docker-compose already configured
- **Better ecosystem** - monitoring tools, documentation, community
- **Made by pydantic team** - aligns with existing Python stack
- **Hybrid approach** - Redis for queue operations + Postgres for workflow state
#### Job Uniqueness Implementation
**Problem**: Multiple concurrent deployment requests for the same tenant could create duplicate jobs, wasting resources and potentially causing conflicts.
**Solution**: Leverage ARQ's built-in job uniqueness feature using predictable job IDs:
```python
# JobQueueService implementation
async def enqueue_deploy_job(self, tenant_id: UUID, image_tag: str | None = None) -> str:
unique_job_id = f"deploy-{tenant_id}"
job = await self.redis_pool.enqueue_job(
"deploy_tenant_job",
str(tenant_id),
image_tag,
_job_id=unique_job_id, # ARQ prevents duplicates
)
if job is None:
# Job already exists - return existing job ID
return unique_job_id
else:
# New job created - return ARQ job ID
return job.job_id
```
**Key Features:**
- **Predictable Job IDs**: `deploy-{tenant_id}`, `undeploy-{tenant_id}`
- **Duplicate Prevention**: ARQ returns `None` for duplicate job IDs
- **Graceful Handling**: Return existing job ID instead of raising errors
- **Idempotent Operations**: Safe to retry deployment requests
- **Clear Logging**: Distinguish "Enqueued new" vs "Found existing" jobs
**Benefits:**
- Prevents resource waste from duplicate deployments
- Eliminates race conditions from concurrent requests
- Makes job monitoring more predictable with consistent IDs
- Provides natural deduplication without complex locking mechanisms
## Notes
### Design Philosophy Lessons
- **Simplicity beats framework magic** - removing DBOS made the system more reliable and debuggable
- **Event sourcing > complex orchestration** - database-backed event tracking provides better observability than framework abstractions
- **Idempotent operations > resumable workflows** - each step handling its own retry logic is simpler than framework-managed resumability
- **Explicit error handling > framework exception handling** - Python exceptions are clearer than orchestration framework error states
### Future Considerations
- **Monitoring integration** - workflow tracking events could feed into observability systems
- **Performance optimization** - event querying patterns may benefit from additional indexing
- **Audit compliance** - complete event trail supports regulatory requirements
- **Operational dashboards** - workflow status could drive tenant health monitoring
### Related Specifications
- **SPEC-8**: TigrisFS Integration - bucket provisioning integrated with deployment workflow
- **SPEC-1**: Specification-Driven Development Process - this spec follows the established format
## Observations
- [architecture] Removing framework complexity led to more maintainable system #simplification
- [workflow] Single unified deployment workflow handles both provisioning and updates #consolidation
- [observability] Event sourcing with workflow correlation provides complete audit trail #event-tracking
- [database] Foreign key relationships between workflows and events enable powerful queries #schema-design
- [testing] Integration tests with real infrastructure catch edge cases that unit tests miss #testing-strategy
- [parameters] Consistent method signatures (workflow_id first) reduce cognitive overhead #api-design
- [maintenance] Fewer workflows and dependencies reduce long-term maintenance burden #operational-excellence
- [debugging] Plain Python exceptions are clearer than framework abstraction layers #developer-experience
- [resilience] Exponential backoff retry patterns handle flaky external API calls gracefully #error-handling
- [monitoring] Redis queue monitoring provides real-time operational visibility #observability
- [ci-cd] Bulk update commands enable automated tenant deployments in continuous delivery pipelines #automation
- [documentation] Comprehensive monitoring guides reduce operational learning curve #knowledge-management
- [error-logging] Context-aware log levels (INFO for expected errors, ERROR for unexpected) improve signal-to-noise ratio #logging-strategy
- [job-uniqueness] ARQ job uniqueness with predictable tenant-based IDs prevents duplicate operations and resource waste #deduplication
## Implementation Notes
### Configuration Integration
- **Redis Configuration**: Add Redis settings to existing `apps/cloud/src/basic_memory_cloud/config.py`
- **Local Development**: Leverage existing Redis setup from `docker-compose.yml`
- **Production**: Use Upstash Redis configuration for production environments
### Docker Entrypoint Strategy
Create `entrypoint.sh` script to toggle between API server and worker processes using single Docker image:
```bash
#!/bin/bash
# Entrypoint script for Basic Memory Cloud service
# Supports multiple process types: api, worker
set -e
case "$1" in
"api")
echo "Starting Basic Memory Cloud API server..."
exec uvicorn basic_memory_cloud.main:app \
--host 0.0.0.0 \
--port 8000 \
--log-level info
;;
"worker")
echo "Starting Basic Memory Cloud ARQ worker..."
# For ARQ worker implementation
exec python -m arq basic_memory_cloud.jobs.settings.WorkerSettings
;;
*)
echo "Usage: $0 {api|worker}"
echo " api - Start the FastAPI server"
echo " worker - Start the ARQ worker"
exit 1
;;
esac
```
### Fly.io Process Groups Configuration
Use separate machine groups for API and worker processes with independent scaling:
```toml
# fly.toml app configuration for basic-memory-cloud
app = 'basic-memory-cloud-dev-basic-machines'
primary_region = 'dfw'
org = 'basic-machines'
kill_signal = 'SIGINT'
kill_timeout = '5s'
[build]
# Process groups for API server and worker
[processes]
api = "api"
worker = "worker"
# Machine scaling configuration
[[machine]]
size = 'shared-cpu-1x'
processes = ['api']
min_machines_running = 1
auto_stop_machines = false
auto_start_machines = true
[[machine]]
size = 'shared-cpu-1x'
processes = ['worker']
min_machines_running = 1
auto_stop_machines = false
auto_start_machines = true
[env]
# Python configuration
PYTHONUNBUFFERED = '1'
PYTHONPATH = '/app'
# Logging configuration
LOG_LEVEL = 'DEBUG'
# Redis configuration for ARQ
REDIS_URL = 'redis://basic-memory-cloud-redis.upstash.io'
# Database configuration
DATABASE_HOST = 'basic-memory-cloud-db-dev-basic-machines.internal'
DATABASE_PORT = '5432'
DATABASE_NAME = 'basic_memory_cloud'
DATABASE_USER = 'postgres'
DATABASE_SSL = 'true'
# Worker configuration
ARQ_MAX_JOBS = '10'
ARQ_KEEP_RESULT = '3600'
# Fly.io configuration
FLY_ORG = 'basic-machines'
FLY_REGION = 'dfw'
# Internal service - no external HTTP exposure for worker
# API accessible via basic-memory-cloud-dev-basic-machines.flycast:8000
[[vm]]
size = 'shared-cpu-1x'
```
### Benefits of This Architecture
- **Single Docker Image**: Both API and worker use same container with different entrypoints
- **Independent Scaling**: Scale API and worker processes separately based on demand
- **Clean Separation**: Web traffic handling separate from background job processing
- **Existing Infrastructure**: Leverages current PostgreSQL + Redis setup without complexity
- **Hybrid State Management**: Redis for queue operations, PostgreSQL for persistent workflow tracking
## Relations
- implements [[SPEC-8 TigrisFS Integration]]
- follows [[SPEC-1 Specification-Driven Development Process]]
- supersedes previous multi-workflow architecture
@@ -1,186 +0,0 @@
---
title: 'SPEC-11: Basic Memory API Performance Optimization'
type: spec
permalink: specs/spec-11-basic-memory-api-performance-optimization
tags:
- performance
- api
- mcp
- database
- cloud
---
# SPEC-11: Basic Memory API Performance Optimization
## Why
The Basic Memory API experiences significant performance issues in cloud environments due to expensive per-request initialization. MCP tools making
HTTP requests to the API suffer from 350ms-2.6s latency overhead **before** any actual operation occurs.
**Root Cause Analysis:**
- GitHub Issue #82 shows repeated initialization sequences in logs (16:29:35 and 16:49:58)
- Each MCP tool call triggers full database initialization + project reconciliation
- `get_engine_factory()` dependency calls `db.get_or_create_db()` on every request
- `reconcile_projects_with_config()` runs expensive sync operations repeatedly
**Performance Impact:**
- Database connection setup: ~50-100ms per request
- Migration checks: ~100-500ms per request
- Project reconciliation: ~200ms-2s per request
- **Total overhead**: ~350ms-2.6s per MCP tool call
This creates compounding effects with tenant auto-start delays and increases timeout risk in cloud deployments.
## What
This optimization affects the **core basic-memory repository** components:
1. **API Lifespan Management** (`src/basic_memory/api/app.py`)
- Cache database connections in app state during startup
- Avoid repeated expensive initialization
2. **Dependency Injection** (`src/basic_memory/deps.py`)
- Modify `get_engine_factory()` to use cached connections
- Eliminate per-request database setup
3. **Initialization Service** (`src/basic_memory/services/initialization.py`)
- Add caching/throttling to project reconciliation
- Skip expensive operations when appropriate
4. **Configuration** (`src/basic_memory/config.py`)
- Add optional performance flags for cloud environments
**Backwards Compatibility**: All changes must be backwards compatible with existing CLI and non-cloud usage.
## How (High Level)
### Phase 1: Cache Database Connections (Critical - 80% of gains)
**Problem**: `get_engine_factory()` calls `db.get_or_create_db()` per request
**Solution**: Cache database engine/session in app state during lifespan
1. **Modify API Lifespan** (`api/app.py`):
```python
@asynccontextmanager
async def lifespan(app: FastAPI):
app_config = ConfigManager().config
await initialize_app(app_config)
# Cache database connection in app state
engine, session_maker = await db.get_or_create_db(app_config.database_path)
app.state.engine = engine
app.state.session_maker = session_maker
# ... rest of startup logic
```
2. Modify Dependency Injection (deps.py):
```python
async def get_engine_factory(
request: Request
) -> tuple[AsyncEngine, async_sessionmaker[AsyncSession]]:
"""Get cached engine and session maker from app state."""
return request.app.state.engine, request.app.state.session_maker
```
Phase 2: Optimize Project Reconciliation (Secondary - 20% of gains)
Problem: reconcile_projects_with_config() runs expensive sync repeatedly
Solution: Add module-level caching with time-based throttling
1. Add Reconciliation Cache (services/initialization.py):
```ptyhon
_project_reconciliation_completed = False
_last_reconciliation_time = 0
async def reconcile_projects_with_config(app_config, force=False):
# Skip if recently completed (within 60 seconds) unless forced
if recently_completed and not force:
return
# ... existing logic
```
Phase 3: Cloud Environment Flags (Optional)
Problem: Force expensive initialization in production environments
Solution: Add skip flags for cloud/stateless deployments
1. Add Config Flag (config.py):
skip_initialization_sync: bool = Field(default=False)
2. Configure in Cloud (basic-memory-cloud integration):
BASIC_MEMORY_SKIP_INITIALIZATION_SYNC=true
How to Evaluate
Success Criteria
1. Performance Metrics (Primary):
- MCP tool response time reduced by 50%+ (measure before/after)
- Database connection overhead eliminated (0ms vs 50-100ms)
- Migration check overhead eliminated (0ms vs 100-500ms)
- Project reconciliation overhead reduced by 90%+
2. Load Testing:
- Concurrent MCP tool calls maintain performance
- No memory leaks in cached connections
- Database connection pool behaves correctly
3. Functional Correctness:
- All existing API endpoints work identically
- MCP tools maintain full functionality
- CLI operations unaffected
- Database migrations still execute properly
4. Backwards Compatibility:
- No breaking changes to existing APIs
- Config changes are optional with safe defaults
- Non-cloud deployments work unchanged
Testing Strategy
Performance Testing:
# Before optimization
time basic-memory-mcp-tools write_note "test" "content" "folder"
# Measure: ~1-3 seconds
# After optimization
time basic-memory-mcp-tools write_note "test" "content" "folder"
# Target: <500ms
Load Testing:
# Multiple concurrent MCP tool calls
for i in {1..10}; do
basic-memory-mcp-tools search "test" &
done
wait
# Verify: No degradation, consistent response times
Regression Testing:
# Full basic-memory test suite
just test
# All tests must pass
# Integration tests with cloud deployment
# Verify MCP gateway → API → database flow works
Validation Checklist
- Phase 1 Complete: Database connections cached, dependency injection optimized
- Performance Benchmark: 50%+ improvement in MCP tool response times
- Memory Usage: No leaks in cached connections over 24h+ periods
- Stress Testing: 100+ concurrent requests maintain performance
- Backwards Compatibility: All existing functionality preserved
- Documentation: Performance optimization documented in README
- Cloud Integration: basic-memory-cloud sees performance benefits
Notes
Implementation Priority:
- Phase 1 provides 80% of performance gains and should be implemented first
- Phase 2 provides remaining 20% and addresses edge cases
- Phase 3 is optional for maximum cloud optimization
Risk Mitigation:
- All changes backwards compatible
- Gradual rollout possible (Phase 1 → 2 → 3)
- Easy rollback via configuration flags
Cloud Integration:
- This optimization directly addresses basic-memory-cloud issue #82
- Changes in core basic-memory will benefit all cloud tenants
- No changes needed in basic-memory-cloud itself
@@ -1,182 +0,0 @@
# SPEC-12: OpenTelemetry Observability
## Why
We need comprehensive observability for basic-memory-cloud to:
- Track request flows across our multi-tenant architecture (MCP → Cloud → API services)
- Debug performance issues and errors in production
- Understand user behavior and system usage patterns
- Correlate issues to specific tenants for targeted debugging
- Monitor service health and latency across the distributed system
Currently, we only have basic logging without request correlation or distributed tracing capabilities.
## What
Implement OpenTelemetry instrumentation across all basic-memory-cloud services with:
### Core Requirements
1. **Distributed Tracing**: End-to-end request tracing from MCP gateway through to tenant API instances
2. **Tenant Correlation**: All traces tagged with tenant_id, user_id, and workos_user_id
3. **Service Identification**: Clear service naming and namespace separation
4. **Auto-instrumentation**: Automatic tracing for FastAPI, SQLAlchemy, HTTP clients
5. **Grafana Cloud Integration**: Direct OTLP export to Grafana Cloud Tempo
### Services to Instrument
- **MCP Gateway** (basic-memory-mcp): Entry point with JWT extraction
- **Cloud Service** (basic-memory-cloud): Provisioning and management operations
- **API Service** (basic-memory-api): Tenant-specific instances
- **Worker Processes** (ARQ workers): Background job processing
### Key Trace Attributes
- `tenant.id`: UUID from UserProfile.tenant_id
- `user.id`: WorkOS user identifier
- `user.email`: User email for debugging
- `service.name`: Specific service identifier
- `service.namespace`: Environment (development/production)
- `operation.type`: Business operation (provision/update/delete)
- `tenant.app_name`: Fly.io app name for tenant instances
## How
### Phase 1: Setup OpenTelemetry SDK
1. Add OpenTelemetry dependencies to each service's pyproject.toml:
```python
"opentelemetry-distro[otlp]>=1.29.0",
"opentelemetry-instrumentation-fastapi>=0.50b0",
"opentelemetry-instrumentation-httpx>=0.50b0",
"opentelemetry-instrumentation-sqlalchemy>=0.50b0",
"opentelemetry-instrumentation-logging>=0.50b0",
```
2. Create shared telemetry initialization module (`apps/shared/telemetry.py`)
3. Configure Grafana Cloud OTLP endpoint via environment variables:
```bash
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-2.grafana.net/otlp
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic[token]
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
```
### Phase 2: Instrument MCP Gateway
1. Extract tenant context from AuthKit JWT in middleware
2. Create root span with tenant attributes
3. Propagate trace context to downstream services via headers
### Phase 3: Instrument Cloud Service
1. Continue trace from MCP gateway
2. Add operation-specific attributes (provisioning events)
3. Instrument ARQ worker jobs for async operations
4. Track Fly.io API calls and latency
### Phase 4: Instrument API Service
1. Extract tenant context from JWT
2. Add machine-specific metadata (instance ID, region)
3. Instrument database operations with SQLAlchemy
4. Track MCP protocol operations
### Phase 5: Configure and Deploy
1. Add OTLP configuration to `.env.example` and `.env.example.secrets`
2. Set Fly.io secrets for production deployment
3. Update Dockerfiles to use `opentelemetry-instrument` wrapper
4. Deploy to development environment first for testing
## How to Evaluate
### Success Criteria
1. **End-to-end traces visible in Grafana Cloud** showing complete request flow
2. **Tenant filtering works** - Can filter traces by tenant_id to see all requests for a user
3. **Service maps accurate** - Grafana shows correct service dependencies
4. **Performance overhead < 5%** - Minimal latency impact from instrumentation
5. **Error correlation** - Can trace errors back to specific tenant and operation
### Testing Checklist
- [x] Single request creates connected trace across all services
- [x] Tenant attributes present on all spans
- [x] Background jobs (ARQ) appear in traces
- [x] Database queries show in trace timeline
- [x] HTTP calls to Fly.io API tracked
- [x] Traces exported successfully to Grafana Cloud
- [x] Can search traces by tenant_id in Grafana
- [x] Service dependency graph shows correct flow
### Monitoring Success
- All services reporting traces to Grafana Cloud
- No OTLP export errors in logs
- Trace sampling working correctly (if implemented)
- Resource usage acceptable (CPU/memory)
## Dependencies
- Grafana Cloud account with OTLP endpoint configured
- OpenTelemetry Python SDK v1.29.0+
- FastAPI instrumentation compatibility
- Network access from Fly.io to Grafana Cloud
## Implementation Assignment
**Recommended Agent**: python-developer
- Requires Python/FastAPI expertise
- Needs understanding of distributed systems
- Must implement middleware and context propagation
- Should understand OpenTelemetry SDK and instrumentation
## Follow-up Tasks
### Enhanced Log Correlation
While basic trace-to-log correlation works automatically via OpenTelemetry logging instrumentation, consider adding structured logging for improved log filtering:
1. **Structured Logging Context**: Add `logger.bind()` calls to inject tenant/user context directly into log records
2. **Custom Loguru Formatter**: Extract OpenTelemetry span attributes for better log readability
3. **Direct Log Filtering**: Enable searching logs directly by tenant_id, workflow_id without going through traces
This would complement the existing automatic trace correlation and provide better log search capabilities.
## Alternative Solution: Logfire
After implementing OpenTelemetry with Grafana Cloud, we discovered limitations in the observability experience:
- Traces work but lack useful context without correlated logs
- Setting up log correlation with Grafana is complex and requires additional infrastructure
- The developer experience for Python observability is suboptimal
### Logfire Evaluation
**Pydantic Logfire** offers a compelling alternative that addresses your specific requirements:
#### Core Requirements Match
- ✅ **User Activity Tracking**: Automatic request tracing with business context
- ✅ **Error Monitoring**: Built-in exception tracking with full context
- ✅ **Performance Metrics**: Automatic latency and performance monitoring
- ✅ **Request Tracing**: Native distributed tracing across services
- ✅ **Log Correlation**: Seamless trace-to-log correlation without setup
#### Key Advantages
1. **Python-First Design**: Built specifically for Python/FastAPI applications by the Pydantic team
2. **Simple Integration**: `pip install logfire` + `logfire.configure()` vs complex OTLP setup
3. **Automatic Correlation**: Logs automatically include trace context without manual configuration
4. **Real-time SQL Interface**: Query spans and logs using SQL with auto-completion
5. **Better Developer UX**: Purpose-built observability UI vs generic Grafana dashboards
6. **Loguru Integration**: `logger.configure(handlers=[logfire.loguru_handler()])` maintains existing logging
#### Pricing Assessment
- **Free Tier**: 10M spans/month (suitable for development and small production workloads)
- **Transparent Pricing**: $1 per million spans/metrics after free tier
- **No Hidden Costs**: No per-host fees, only usage-based metering
- **Production Ready**: Recently exited beta, enterprise features available
#### Migration Path
The existing OpenTelemetry instrumentation is compatible - Logfire uses OpenTelemetry under the hood, so the current spans and attributes would work unchanged.
### Recommendation
**Consider migrating to Logfire** for the following reasons:
1. It directly addresses the "next to useless" traces problem by providing integrated logs
2. Dramatically simpler setup and maintenance compared to Grafana Cloud + custom log correlation
3. Better ROI on observability investment with purpose-built Python tooling
4. Free tier sufficient for current development needs with clear scaling path
The current Grafana Cloud implementation provides a solid foundation and could remain as a backup/export target, while Logfire becomes the primary observability platform.
## Status
**Created**: 2024-01-28
**Status**: Completed (OpenTelemetry + Grafana Cloud)
**Next Phase**: Evaluate Logfire migration
**Priority**: High - Critical for production observability
@@ -1,917 +0,0 @@
---
title: 'SPEC-13: CLI Authentication with Subscription Validation'
type: spec
permalink: specs/spec-12-cli-auth-subscription-validation
tags:
- authentication
- security
- cli
- subscription
status: draft
created: 2025-10-02
---
# SPEC-13: CLI Authentication with Subscription Validation
## Why
The Basic Memory Cloud CLI currently has a security gap in authentication that allows unauthorized access:
**Current Web Flow (Secure)**:
1. User signs up via WorkOS AuthKit
2. User creates Polar subscription
3. Web app validates subscription before calling `POST /tenants/setup`
4. Tenant provisioned only after subscription validation ✅
**Current CLI Flow (Insecure)**:
1. User signs up via WorkOS AuthKit (OAuth device flow)
2. User runs `bm cloud login`
3. CLI receives JWT token from WorkOS
4. CLI can access all cloud endpoints without subscription check ❌
**Problem**: Anyone can sign up with WorkOS and immediately access cloud infrastructure via CLI without having an active Polar subscription. This creates:
- Revenue loss (free resource consumption)
- Security risk (unauthorized data access)
- Support burden (users accessing features they haven't paid for)
**Root Cause**: The CLI authentication flow validates JWT tokens but doesn't verify subscription status before granting access to cloud resources.
## What
Add subscription validation to authentication flow to ensure only users with active Polar subscriptions can access cloud resources across all access methods (CLI, MCP, Web App, Direct API).
**Affected Components**:
### basic-memory-cloud (Cloud Service)
- `apps/cloud/src/basic_memory_cloud/deps.py` - Add subscription validation dependency
- `apps/cloud/src/basic_memory_cloud/services/subscription_service.py` - Add subscription check method
- `apps/cloud/src/basic_memory_cloud/api/tenant_mount.py` - Protect mount endpoints
- `apps/cloud/src/basic_memory_cloud/api/proxy.py` - Protect proxy endpoints
### basic-memory (CLI)
- `src/basic_memory/cli/commands/cloud/core_commands.py` - Handle 403 errors
- `src/basic_memory/cli/commands/cloud/api_client.py` - Parse subscription errors
- `docs/cloud-cli.md` - Document subscription requirement
**Endpoints to Protect**:
- `GET /tenant/mount/info` - Used by CLI bisync setup
- `POST /tenant/mount/credentials` - Used by CLI bisync credentials
- `GET /proxy/{path:path}` - Used by Web App, MCP tools, CLI tools, Direct API
- All other `/proxy/*` endpoints - Centralized access point for all user operations
## Complete Authentication Flow Analysis
### Overview of All Access Flows
Basic Memory Cloud has **7 distinct authentication flows**. This spec closes subscription validation gaps in flows 2-4 and 6, which all converge on the `/proxy/*` endpoints.
### Flow 1: Polar Webhook → Registration ✅ SECURE
```
Polar webhook → POST /api/webhooks/polar
→ Validates Polar webhook signature
→ Creates/updates subscription in database
→ No direct user access - webhook only
```
**Auth**: Polar webhook signature validation
**Subscription Check**: N/A (webhook creates subscriptions)
**Status**: ✅ Secure - webhook validated, no user JWT involved
### Flow 2: Web App Login ❌ NEEDS FIX
```
User → apps/web (Vue.js/Nuxt)
→ WorkOS AuthKit magic link authentication
→ JWT stored in browser session
→ Web app calls /proxy/{project}/... endpoints (memory, directory, projects)
→ proxy.py validates JWT but does NOT check subscription
→ Access granted without subscription ❌
```
**Auth**: WorkOS JWT via `CurrentUserProfileHybridJwtDep`
**Subscription Check**: ❌ Missing
**Fixed By**: Task 1.4 (protect `/proxy/*` endpoints)
### Flow 3: MCP (Model Context Protocol) ❌ NEEDS FIX
```
AI Agent (Claude, Cursor, etc.) → https://mcp.basicmemory.com
→ AuthKit OAuth device flow
→ JWT stored in AI agent
→ MCP tools call {cloud_host}/proxy/{endpoint} with Authorization header
→ proxy.py validates JWT but does NOT check subscription
→ MCP tools can access all cloud resources without subscription ❌
```
**Auth**: AuthKit JWT via `CurrentUserProfileHybridJwtDep`
**Subscription Check**: ❌ Missing
**Fixed By**: Task 1.4 (protect `/proxy/*` endpoints)
### Flow 4: CLI Auth (basic-memory) ❌ NEEDS FIX
```
User → bm cloud login
→ AuthKit OAuth device flow
→ JWT stored in ~/.basic-memory/tokens.json
→ CLI calls:
- {cloud_host}/tenant/mount/info (for bisync setup)
- {cloud_host}/tenant/mount/credentials (for bisync credentials)
- {cloud_host}/proxy/{endpoint} (for all MCP tools)
→ tenant_mount.py and proxy.py validate JWT but do NOT check subscription
→ Access granted without subscription ❌
```
**Auth**: AuthKit JWT via `CurrentUserProfileHybridJwtDep`
**Subscription Check**: ❌ Missing
**Fixed By**: Task 1.3 (protect `/tenant/mount/*`) + Task 1.4 (protect `/proxy/*`)
### Flow 5: Cloud CLI (Admin Tasks) ✅ SECURE
```
Admin → python -m basic_memory_cloud.cli.tenant_cli
→ Uses CLIAuth with admin WorkOS OAuth client
→ Gets JWT token with admin org membership
→ Calls /tenants/* endpoints (create, list, delete tenants)
→ tenants.py validates JWT AND admin org membership via AdminUserHybridDep
→ Access granted only to admin organization members ✅
```
**Auth**: AuthKit JWT + Admin org validation via `AdminUserHybridDep`
**Subscription Check**: N/A (admins bypass subscription requirement)
**Status**: ✅ Secure - admin-only endpoints, separate from user flows
### Flow 6: Direct API Calls ❌ NEEDS FIX
```
Any HTTP client → {cloud_host}/proxy/{endpoint}
→ Sends Authorization: Bearer {jwt} header
→ proxy.py validates JWT but does NOT check subscription
→ Direct API access without subscription ❌
```
**Auth**: WorkOS or AuthKit JWT via `CurrentUserProfileHybridJwtDep`
**Subscription Check**: ❌ Missing
**Fixed By**: Task 1.4 (protect `/proxy/*` endpoints)
### Flow 7: Tenant API Instance (Internal) ✅ SECURE
```
/proxy/* → Tenant API (basic-memory-{tenant_id}.fly.dev)
→ Validates signed header from proxy (tenant_id + signature)
→ Direct external access will be disabled in production
→ Only accessible via /proxy endpoints
```
**Auth**: Signed header validation from proxy
**Subscription Check**: N/A (internal only, validated at proxy layer)
**Status**: ✅ Secure - validates proxy signature, not directly accessible
### Authentication Flow Summary Matrix
| Flow | Access Method | Current Auth | Subscription Check | Fixed By SPEC-13 |
|------|---------------|--------------|-------------------|------------------|
| 1. Polar Webhook | Polar webhook → `/api/webhooks/polar` | Polar signature | N/A (webhook) | N/A |
| 2. Web App | Browser → `/proxy/*` | WorkOS JWT ✅ | ❌ Missing | ✅ Task 1.4 |
| 3. MCP | AI Agent → `/proxy/*` | AuthKit JWT ✅ | ❌ Missing | ✅ Task 1.4 |
| 4. CLI | `bm cloud``/tenant/mount/*` + `/proxy/*` | AuthKit JWT ✅ | ❌ Missing | ✅ Task 1.3 + 1.4 |
| 5. Cloud CLI (Admin) | `tenant_cli``/tenants/*` | AuthKit JWT ✅ + Admin org | N/A (admin) | N/A (admin bypass) |
| 6. Direct API | HTTP client → `/proxy/*` | WorkOS/AuthKit JWT ✅ | ❌ Missing | ✅ Task 1.4 |
| 7. Tenant API | Proxy → tenant instance | Proxy signature ✅ | N/A (internal) | N/A |
### Key Insights
1. **Single Point of Failure**: All user access (Web, MCP, CLI, Direct API) converges on `/proxy/*` endpoints
2. **Centralized Fix**: Protecting `/proxy/*` with subscription validation closes gaps in flows 2, 3, 4, and 6 simultaneously
3. **Admin Bypass**: Cloud CLI admin tasks use separate `/tenants/*` endpoints with admin-only access (no subscription needed)
4. **Defense in Depth**: `/tenant/mount/*` endpoints also protected for CLI bisync operations
### Architecture Benefits
The `/proxy` layer serves as the **single centralized authorization point** for all user access:
- ✅ One place to validate JWT tokens
- ✅ One place to check subscription status
- ✅ One place to handle tenant routing
- ✅ Protects Web App, MCP, CLI, and Direct API simultaneously
This architecture makes the fix comprehensive and maintainable.
## How (High Level)
### Option A: Database Subscription Check (Recommended)
**Approach**: Add FastAPI dependency that validates subscription status from database before allowing access.
**Implementation**:
1. **Create Subscription Validation Dependency** (`deps.py`)
```python
async def get_authorized_cli_user_profile(
credentials: Annotated[HTTPAuthorizationCredentials, Depends(security)],
session: DatabaseSessionDep,
user_profile_repo: UserProfileRepositoryDep,
subscription_service: SubscriptionServiceDep,
) -> UserProfile:
"""
Hybrid authentication with subscription validation for CLI access.
Validates JWT (WorkOS or AuthKit) and checks for active subscription.
Returns UserProfile if both checks pass.
"""
# Try WorkOS JWT first (faster validation path)
try:
user_context = await validate_workos_jwt(credentials.credentials)
except HTTPException:
# Fall back to AuthKit JWT validation
try:
user_context = await validate_authkit_jwt(credentials.credentials)
except HTTPException as e:
raise HTTPException(
status_code=401,
detail="Invalid JWT token. Authentication required.",
) from e
# Check subscription status
has_subscription = await subscription_service.check_user_has_active_subscription(
session, user_context.workos_user_id
)
if not has_subscription:
raise HTTPException(
status_code=403,
detail={
"error": "subscription_required",
"message": "Active subscription required for CLI access",
"subscribe_url": "https://basicmemory.com/subscribe"
}
)
# Look up and return user profile
user_profile = await user_profile_repo.get_user_profile_by_workos_user_id(
session, user_context.workos_user_id
)
if not user_profile:
raise HTTPException(401, detail="User profile not found")
return user_profile
```
```python
AuthorizedCLIUserProfileDep = Annotated[UserProfile, Depends(get_authorized_cli_user_profile)]
```
2. **Add Subscription Check Method** (`subscription_service.py`)
```python
async def check_user_has_active_subscription(
self, session: AsyncSession, workos_user_id: str
) -> bool:
"""Check if user has active subscription."""
# Use existing repository method to get subscription by workos_user_id
# This joins UserProfile -> Subscription in a single query
subscription = await self.subscription_repository.get_subscription_by_workos_user_id(
session, workos_user_id
)
return subscription is not None and subscription.status == "active"
```
3. **Protect Endpoints** (Replace `CurrentUserProfileHybridJwtDep` with `AuthorizedCLIUserProfileDep`)
```python
# Before
@router.get("/mount/info")
async def get_mount_info(
user_profile: CurrentUserProfileHybridJwtDep,
session: DatabaseSessionDep,
):
tenant_id = user_profile.tenant_id
...
# After
@router.get("/mount/info")
async def get_mount_info(
user_profile: AuthorizedCLIUserProfileDep, # Now includes subscription check
session: DatabaseSessionDep,
):
tenant_id = user_profile.tenant_id # No changes needed to endpoint logic
...
```
4. **Update CLI Error Handling**
```python
# In core_commands.py login()
try:
success = await auth.login()
if success:
# Test subscription by calling protected endpoint
await make_api_request("GET", f"{host_url}/tenant/mount/info")
except CloudAPIError as e:
if e.status_code == 403 and e.detail.get("error") == "subscription_required":
console.print("[red]Subscription required[/red]")
console.print(f"Subscribe at: {e.detail['subscribe_url']}")
raise typer.Exit(1)
```
**Pros**:
- Simple to implement
- Fast (single database query)
- Clear error messages
- Works with existing subscription flow
**Cons**:
- Database is source of truth (could get out of sync with Polar)
- Adds one extra subscription lookup query per request (lightweight JOIN query)
### Option B: WorkOS Organizations
**Approach**: Add users to "beta-users" organization in WorkOS after subscription creation, validate org membership via JWT claims.
**Implementation**:
1. After Polar subscription webhook, add user to WorkOS org via API
2. Validate `org_id` claim in JWT matches authorized org
3. Use existing `get_admin_workos_jwt` pattern
**Pros**:
- WorkOS as single source of truth
- No database queries needed
- More secure (harder to bypass)
**Cons**:
- More complex (requires WorkOS API integration)
- Requires managing WorkOS org membership
- Less control over error messages
- Additional API calls during registration
### Recommendation
**Start with Option A (Database Check)** for:
- Faster implementation
- Clearer error messages
- Easier testing
- Existing subscription infrastructure
**Consider Option B later** if:
- Need tighter security
- Want to reduce database dependency
- Scale requires fewer database queries
## How to Evaluate
### Success Criteria
**1. Unauthorized Users Blocked**
- [ ] User without subscription cannot complete `bm cloud login`
- [ ] User without subscription receives clear error with subscribe link
- [ ] User without subscription cannot run `bm cloud setup`
- [ ] User without subscription cannot run `bm sync` in cloud mode
**2. Authorized Users Work**
- [ ] User with active subscription can login successfully
- [ ] User with active subscription can setup bisync
- [ ] User with active subscription can sync files
- [ ] User with active subscription can use all MCP tools via proxy
**3. Subscription State Changes**
- [ ] Expired subscription blocks access with clear error
- [ ] Renewed subscription immediately restores access
- [ ] Cancelled subscription blocks access after grace period
**4. Error Messages**
- [ ] 403 errors include "subscription_required" error code
- [ ] Error messages include subscribe URL
- [ ] CLI displays user-friendly messages
- [ ] Errors logged appropriately for debugging
**5. No Regressions**
- [ ] Web app login/subscription flow unaffected
- [ ] Admin endpoints still work (bypass check)
- [ ] Tenant provisioning workflow unchanged
- [ ] Performance not degraded
### Test Cases
**Manual Testing**:
```bash
# Test 1: Unauthorized user
1. Create new WorkOS account (no subscription)
2. Run `bm cloud login`
3. Verify: Login succeeds but shows subscription required error
4. Verify: Cannot run `bm cloud setup`
5. Verify: Clear error message with subscribe link
# Test 2: Authorized user
1. Use account with active Polar subscription
2. Run `bm cloud login`
3. Verify: Login succeeds without errors
4. Run `bm cloud setup`
5. Verify: Setup completes successfully
6. Run `bm sync`
7. Verify: Sync works normally
# Test 3: Subscription expiration
1. Use account with active subscription
2. Manually expire subscription in database
3. Run `bm cloud login`
4. Verify: Blocked with clear error
5. Renew subscription
6. Run `bm cloud login` again
7. Verify: Access restored
```
**Automated Tests**:
```python
# Test subscription validation dependency
async def test_authorized_user_allowed(
db_session,
user_profile_repo,
subscription_service,
mock_jwt_credentials
):
# Create user with active subscription
user_profile = await create_user_with_subscription(db_session, status="active")
# Mock JWT credentials for the user
credentials = mock_jwt_credentials(user_profile.workos_user_id)
# Should not raise exception
result = await get_authorized_cli_user_profile(
credentials, db_session, user_profile_repo, subscription_service
)
assert result.id == user_profile.id
assert result.workos_user_id == user_profile.workos_user_id
async def test_unauthorized_user_blocked(
db_session,
user_profile_repo,
subscription_service,
mock_jwt_credentials
):
# Create user without subscription
user_profile = await create_user_without_subscription(db_session)
credentials = mock_jwt_credentials(user_profile.workos_user_id)
# Should raise 403
with pytest.raises(HTTPException) as exc:
await get_authorized_cli_user_profile(
credentials, db_session, user_profile_repo, subscription_service
)
assert exc.value.status_code == 403
assert exc.value.detail["error"] == "subscription_required"
async def test_inactive_subscription_blocked(
db_session,
user_profile_repo,
subscription_service,
mock_jwt_credentials
):
# Create user with cancelled/inactive subscription
user_profile = await create_user_with_subscription(db_session, status="cancelled")
credentials = mock_jwt_credentials(user_profile.workos_user_id)
# Should raise 403
with pytest.raises(HTTPException) as exc:
await get_authorized_cli_user_profile(
credentials, db_session, user_profile_repo, subscription_service
)
assert exc.value.status_code == 403
assert exc.value.detail["error"] == "subscription_required"
```
## Implementation Tasks
### Phase 1: Cloud Service (basic-memory-cloud)
#### Task 1.1: Add subscription check method to SubscriptionService ✅
**File**: `apps/cloud/src/basic_memory_cloud/services/subscription_service.py`
- [x] Add method `check_subscription(session: AsyncSession, workos_user_id: str) -> bool`
- [x] Use existing `self.subscription_repository.get_subscription_by_workos_user_id(session, workos_user_id)`
- [x] Check both `status == "active"` AND `current_period_end >= now()`
- [x] Log both values when check fails
- [x] Add docstring explaining the method
- [x] Run `just typecheck` to verify types
**Actual implementation**:
```python
async def check_subscription(
self, session: AsyncSession, workos_user_id: str
) -> bool:
"""Check if user has active subscription with valid period."""
subscription = await self.subscription_repository.get_subscription_by_workos_user_id(
session, workos_user_id
)
if subscription is None:
return False
if subscription.status != "active":
logger.warning("Subscription inactive", workos_user_id=workos_user_id,
status=subscription.status, current_period_end=subscription.current_period_end)
return False
now = datetime.now(timezone.utc)
if subscription.current_period_end is None or subscription.current_period_end < now:
logger.warning("Subscription expired", workos_user_id=workos_user_id,
status=subscription.status, current_period_end=subscription.current_period_end)
return False
return True
```
#### Task 1.2: Add subscription validation dependency ✅
**File**: `apps/cloud/src/basic_memory_cloud/deps.py`
- [x] Import necessary types at top of file (if not already present)
- [x] Add `authorized_user_profile()` async function
- [x] Implement hybrid JWT validation (WorkOS first, AuthKit fallback)
- [x] Add subscription check using `subscription_service.check_subscription()`
- [x] Raise `HTTPException(403)` with structured error detail if no active subscription
- [x] Look up and return `UserProfile` after validation
- [x] Add `AuthorizedUserProfileDep` type annotation
- [x] Use `settings.subscription_url` from config (env var)
- [x] Run `just typecheck` to verify types
**Expected code**:
```python
async def get_authorized_cli_user_profile(
credentials: Annotated[HTTPAuthorizationCredentials, Depends(security)],
session: DatabaseSessionDep,
user_profile_repo: UserProfileRepositoryDep,
subscription_service: SubscriptionServiceDep,
) -> UserProfile:
"""
Hybrid authentication with subscription validation for CLI access.
Validates JWT (WorkOS or AuthKit) and checks for active subscription.
Returns UserProfile if both checks pass.
Raises:
HTTPException(401): Invalid JWT token
HTTPException(403): No active subscription
"""
# Try WorkOS JWT first (faster validation path)
try:
user_context = await validate_workos_jwt(credentials.credentials)
except HTTPException:
# Fall back to AuthKit JWT validation
try:
user_context = await validate_authkit_jwt(credentials.credentials)
except HTTPException as e:
raise HTTPException(
status_code=401,
detail="Invalid JWT token. Authentication required.",
) from e
# Check subscription status
has_subscription = await subscription_service.check_user_has_active_subscription(
session, user_context.workos_user_id
)
if not has_subscription:
logger.warning(
"CLI access denied: no active subscription",
workos_user_id=user_context.workos_user_id,
)
raise HTTPException(
status_code=403,
detail={
"error": "subscription_required",
"message": "Active subscription required for CLI access",
"subscribe_url": "https://basicmemory.com/subscribe"
}
)
# Look up and return user profile
user_profile = await user_profile_repo.get_user_profile_by_workos_user_id(
session, user_context.workos_user_id
)
if not user_profile:
logger.error(
"User profile not found after successful auth",
workos_user_id=user_context.workos_user_id,
)
raise HTTPException(401, detail="User profile not found")
logger.info(
"CLI access granted",
workos_user_id=user_context.workos_user_id,
user_profile_id=str(user_profile.id),
)
return user_profile
AuthorizedCLIUserProfileDep = Annotated[UserProfile, Depends(get_authorized_cli_user_profile)]
```
#### Task 1.3: Protect tenant mount endpoints ✅
**File**: `apps/cloud/src/basic_memory_cloud/api/tenant_mount.py`
- [x] Update import: add `AuthorizedUserProfileDep` from `..deps`
- [x] Replace `user_profile: CurrentUserProfileHybridJwtDep` with `user_profile: AuthorizedUserProfileDep` in:
- [x] `get_tenant_mount_info()` (line ~23)
- [x] `create_tenant_mount_credentials()` (line ~88)
- [x] `revoke_tenant_mount_credentials()` (line ~244)
- [x] `list_tenant_mount_credentials()` (line ~326)
- [x] Verify no other code changes needed (parameter name and usage stays the same)
- [x] Run `just typecheck` to verify types
#### Task 1.4: Protect proxy endpoints ✅
**File**: `apps/cloud/src/basic_memory_cloud/api/proxy.py`
- [x] Update import: add `AuthorizedUserProfileDep` from `..deps`
- [x] Replace `user_profile: CurrentUserProfileHybridJwtDep` with `user_profile: AuthorizedUserProfileDep` in:
- [x] `check_tenant_health()` (line ~21)
- [x] `proxy_to_tenant()` (line ~63)
- [x] Verify no other code changes needed (parameter name and usage stays the same)
- [x] Run `just typecheck` to verify types
**Why Keep /proxy Architecture:**
The proxy layer is valuable because it:
1. **Centralizes authorization** - Single place for JWT + subscription validation (closes both CLI and MCP auth gaps)
2. **Handles tenant routing** - Maps tenant_id → fly_app_name without exposing infrastructure details
3. **Abstracts infrastructure** - MCP and CLI don't need to know about Fly.io naming conventions
4. **Enables features** - Can add rate limiting, caching, request logging, etc. at proxy layer
5. **Supports both flows** - CLI tools and MCP tools both use /proxy endpoints
The extra HTTP hop is minimal (< 10ms) and worth it for architectural benefits.
**Performance Note:** Cloud app has Redis available - can cache subscription status to reduce database queries if needed. Initial implementation uses direct database query (simple, acceptable performance ~5-10ms).
#### Task 1.5: Add unit tests for subscription service
**File**: `apps/cloud/tests/services/test_subscription_service.py` (create if doesn't exist)
- [ ] Create test file if it doesn't exist
- [ ] Add test: `test_check_user_has_active_subscription_returns_true_for_active()`
- Create user with active subscription
- Call `check_user_has_active_subscription()`
- Assert returns `True`
- [ ] Add test: `test_check_user_has_active_subscription_returns_false_for_pending()`
- Create user with pending subscription
- Assert returns `False`
- [ ] Add test: `test_check_user_has_active_subscription_returns_false_for_cancelled()`
- Create user with cancelled subscription
- Assert returns `False`
- [ ] Add test: `test_check_user_has_active_subscription_returns_false_for_no_subscription()`
- Create user without subscription
- Assert returns `False`
- [ ] Run `just test` to verify tests pass
#### Task 1.6: Add integration tests for dependency
**File**: `apps/cloud/tests/test_deps.py` (create if doesn't exist)
- [ ] Create test file if it doesn't exist
- [ ] Add fixtures for mocking JWT credentials
- [ ] Add test: `test_authorized_cli_user_profile_with_active_subscription()`
- Mock valid JWT + active subscription
- Call dependency
- Assert returns UserProfile
- [ ] Add test: `test_authorized_cli_user_profile_without_subscription_raises_403()`
- Mock valid JWT + no subscription
- Assert raises HTTPException(403) with correct error detail
- [ ] Add test: `test_authorized_cli_user_profile_with_inactive_subscription_raises_403()`
- Mock valid JWT + cancelled subscription
- Assert raises HTTPException(403)
- [ ] Add test: `test_authorized_cli_user_profile_with_invalid_jwt_raises_401()`
- Mock invalid JWT
- Assert raises HTTPException(401)
- [ ] Run `just test` to verify tests pass
#### Task 1.7: Deploy and verify cloud service
- [ ] Run `just check` to verify all quality checks pass
- [ ] Commit changes with message: "feat: add subscription validation to CLI endpoints"
- [ ] Deploy to preview environment: `flyctl deploy --config apps/cloud/fly.toml`
- [ ] Test manually:
- [ ] Call `/tenant/mount/info` with valid JWT but no subscription → expect 403
- [ ] Call `/tenant/mount/info` with valid JWT and active subscription → expect 200
- [ ] Verify error response structure matches spec
### Phase 2: CLI (basic-memory)
#### Task 2.1: Review and understand CLI authentication flow
**Files**: `src/basic_memory/cli/commands/cloud/`
- [ ] Read `core_commands.py` to understand current login flow
- [ ] Read `api_client.py` to understand current error handling
- [ ] Identify where 403 errors should be caught
- [ ] Identify what error messages should be displayed
- [ ] Document current behavior in spec if needed
#### Task 2.2: Update API client error handling
**File**: `src/basic_memory/cli/commands/cloud/api_client.py`
- [ ] Add custom exception class `SubscriptionRequiredError` (or similar)
- [ ] Update HTTP error handling to parse 403 responses
- [ ] Extract `error`, `message`, and `subscribe_url` from error detail
- [ ] Raise specific exception for subscription_required errors
- [ ] Run `just typecheck` in basic-memory repo to verify types
#### Task 2.3: Update CLI login command error handling
**File**: `src/basic_memory/cli/commands/cloud/core_commands.py`
- [ ] Import the subscription error exception
- [ ] Wrap login flow with try/except for subscription errors
- [ ] Display user-friendly error message with rich console
- [ ] Show subscribe URL prominently
- [ ] Provide actionable next steps
- [ ] Run `just typecheck` to verify types
**Expected error handling**:
```python
try:
# Existing login logic
success = await auth.login()
if success:
# Test access to protected endpoint
await api_client.test_connection()
except SubscriptionRequiredError as e:
console.print("\n[red]✗ Subscription Required[/red]\n")
console.print(f"[yellow]{e.message}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
console.print("[dim]Once you have an active subscription, run [bold]bm cloud login[/bold] again.[/dim]")
raise typer.Exit(1)
```
#### Task 2.4: Update CLI tests
**File**: `tests/cli/test_cloud_commands.py`
- [ ] Add test: `test_login_without_subscription_shows_error()`
- Mock 403 subscription_required response
- Call login command
- Assert error message displayed
- Assert subscribe URL shown
- [ ] Add test: `test_login_with_subscription_succeeds()`
- Mock successful authentication + subscription check
- Call login command
- Assert success message
- [ ] Run `just test` to verify tests pass
#### Task 2.5: Update CLI documentation
**File**: `docs/cloud-cli.md` (in basic-memory-docs repo)
- [ ] Add "Prerequisites" section if not present
- [ ] Document subscription requirement
- [ ] Add "Troubleshooting" section
- [ ] Document "Subscription Required" error
- [ ] Provide subscribe URL
- [ ] Add FAQ entry about subscription errors
- [ ] Build docs locally to verify formatting
### Phase 3: End-to-End Testing
#### Task 3.1: Create test user accounts
**Prerequisites**: Access to WorkOS admin and database
- [ ] Create test user WITHOUT subscription:
- [ ] Sign up via WorkOS AuthKit
- [ ] Get workos_user_id from database
- [ ] Verify no subscription record exists
- [ ] Save credentials for testing
- [ ] Create test user WITH active subscription:
- [ ] Sign up via WorkOS AuthKit
- [ ] Create subscription via Polar or dev endpoint
- [ ] Verify subscription.status = "active" in database
- [ ] Save credentials for testing
#### Task 3.2: Manual testing - User without subscription
**Environment**: Preview/staging deployment
- [ ] Run `bm cloud login` with no-subscription user
- [ ] Verify: Login shows "Subscription Required" error
- [ ] Verify: Subscribe URL is displayed
- [ ] Verify: Cannot run `bm cloud setup`
- [ ] Verify: Cannot call `/tenant/mount/info` directly via curl
- [ ] Document any issues found
#### Task 3.3: Manual testing - User with active subscription
**Environment**: Preview/staging deployment
- [ ] Run `bm cloud login` with active-subscription user
- [ ] Verify: Login succeeds without errors
- [ ] Verify: Can run `bm cloud setup`
- [ ] Verify: Can call `/tenant/mount/info` successfully
- [ ] Verify: Can call `/proxy/*` endpoints successfully
- [ ] Document any issues found
#### Task 3.4: Test subscription state transitions
**Environment**: Preview/staging deployment + database access
- [ ] Start with active subscription user
- [ ] Verify: All operations work
- [ ] Update subscription.status to "cancelled" in database
- [ ] Verify: Login now shows "Subscription Required" error
- [ ] Verify: Existing tokens are rejected with 403
- [ ] Update subscription.status back to "active"
- [ ] Verify: Access restored immediately
- [ ] Document any issues found
#### Task 3.5: Integration test suite
**File**: `apps/cloud/tests/integration/test_cli_subscription_flow.py` (create if doesn't exist)
- [ ] Create integration test file
- [ ] Add test: `test_cli_flow_without_subscription()`
- Simulate full CLI flow without subscription
- Assert 403 at appropriate points
- [ ] Add test: `test_cli_flow_with_active_subscription()`
- Simulate full CLI flow with active subscription
- Assert all operations succeed
- [ ] Add test: `test_subscription_expiration_blocks_access()`
- Start with active subscription
- Change status to cancelled
- Assert access denied
- [ ] Run tests in CI/CD pipeline
- [ ] Document test coverage
#### Task 3.6: Load/performance testing (optional)
**Environment**: Staging environment
- [ ] Test subscription check performance under load
- [ ] Measure latency added by subscription check
- [ ] Verify database query performance
- [ ] Document any performance concerns
- [ ] Optimize if needed
## Implementation Summary Checklist
Use this high-level checklist to track overall progress:
### Phase 1: Cloud Service 🔄
- [x] Add subscription check method to SubscriptionService
- [x] Add subscription validation dependency to deps.py
- [x] Add subscription_url config (env var)
- [x] Protect tenant mount endpoints (4 endpoints)
- [x] Protect proxy endpoints (2 endpoints)
- [ ] Add unit tests for subscription service
- [ ] Add integration tests for dependency
- [ ] Deploy and verify cloud service
### Phase 2: CLI Updates 🔄
- [ ] Review CLI authentication flow
- [ ] Update API client error handling
- [ ] Update CLI login command error handling
- [ ] Add CLI tests
- [ ] Update CLI documentation
### Phase 3: End-to-End Testing 🧪
- [ ] Create test user accounts
- [ ] Manual testing - user without subscription
- [ ] Manual testing - user with active subscription
- [ ] Test subscription state transitions
- [ ] Integration test suite
- [ ] Load/performance testing (optional)
## Questions to Resolve
### Resolved ✅
1. **Admin Access**
- ✅ **Decision**: Admin users bypass subscription check
- **Rationale**: Admin endpoints already use `AdminUserHybridDep`, which is separate from CLI user endpoints
- **Implementation**: No changes needed to admin endpoints
2. **Subscription Check Implementation**
- ✅ **Decision**: Use Option A (Database Check)
- **Rationale**: Simpler, faster to implement, works with existing infrastructure
- **Implementation**: Single JOIN query via `get_subscription_by_workos_user_id()`
3. **Dependency Return Type**
- ✅ **Decision**: Return `UserProfile` (not `UserContext`)
- **Rationale**: Drop-in compatibility with existing endpoints, no refactoring needed
- **Implementation**: `AuthorizedCLIUserProfileDep` returns `UserProfile`
### To Be Resolved ⏳
1. **Subscription Check Frequency**
- **Options**:
- Check on every API call (slower, more secure) ✅ **RECOMMENDED**
- Cache subscription status (faster, risk of stale data)
- Check only on login/setup (fast, but allows expired subscriptions temporarily)
- **Recommendation**: Check on every call via dependency injection (simple, secure, acceptable performance)
- **Impact**: ~5-10ms per request (single indexed JOIN query)
2. **Grace Period**
- **Options**:
- No grace period - immediate block when status != "active" ✅ **RECOMMENDED**
- 7-day grace period after period_end
- 14-day grace period after period_end
- **Recommendation**: No grace period initially, add later if needed based on customer feedback
- **Implementation**: Check `subscription.status == "active"` only (ignore period_end initially)
3. **Subscription Expiration Handling**
- **Question**: Should we check `current_period_end < now()` in addition to `status == "active"`?
- **Options**:
- Only check status field (rely on Polar webhooks to update status) ✅ **RECOMMENDED**
- Check both status and current_period_end (more defensive)
- **Recommendation**: Only check status field, assume Polar webhooks keep it current
- **Risk**: If webhooks fail, expired subscriptions might retain access until webhook succeeds
4. **Subscribe URL**
- **Question**: What's the actual subscription URL?
- **Current**: Spec uses `https://basicmemory.com/subscribe`
- **Action Required**: Verify correct URL before implementation
5. **Dev Mode / Testing Bypass**
- **Question**: Support bypass for development/testing?
- **Options**:
- Environment variable: `DISABLE_SUBSCRIPTION_CHECK=true`
- Always enforce (more realistic testing) ✅ **RECOMMENDED**
- **Recommendation**: No bypass - use test users with real subscriptions for realistic testing
- **Implementation**: Create dev endpoint to activate subscriptions for testing
## Related Specs
- SPEC-9: Multi-Project Bidirectional Sync Architecture (CLI affected by this change)
- SPEC-8: TigrisFS Integration (Mount endpoints protected)
## Notes
- This spec prioritizes security over convenience - better to block unauthorized access than risk revenue loss
- Clear error messages are critical - users should understand why they're blocked and how to resolve it
- Consider adding telemetry to track subscription_required errors for monitoring signup conversion
@@ -1,210 +0,0 @@
---
title: 'SPEC-14: Cloud Git Versioning & GitHub Backup'
type: spec
permalink: specs/spec-14-cloud-git-versioning
tags:
- git
- github
- backup
- versioning
- cloud
related:
- specs/spec-9-multi-project-bisync
- specs/spec-9-follow-ups-conflict-sync-and-observability
status: deferred
---
# SPEC-14: Cloud Git Versioning & GitHub Backup
**Status: DEFERRED** - Postponed until multi-user/teams feature development. Using S3 versioning (SPEC-9.1) for v1 instead.
## Why Deferred
**Original goals can be met with simpler solutions:**
- Version history → **S3 bucket versioning** (automatic, zero config)
- Offsite backup → **Tigris global replication** (built-in)
- Restore capability → **S3 version restore** (`bm cloud restore --version-id`)
- Collaboration → **Deferred to teams/multi-user feature** (not v1 requirement)
**Complexity vs value trade-off:**
- Git integration adds: committer service, puller service, webhooks, LFS, merge conflicts
- Risk: Loop detection between Git ↔ rclone bisync ↔ local edits
- S3 versioning gives 80% of value with 5% of complexity
**When to revisit:**
- Teams/multi-user features (PR-based collaboration workflow)
- User requests for commit messages and branch-based workflows
- Need for fine-grained audit trail beyond S3 object metadata
---
## Original Specification (for reference)
## Why
Early access users want **transparent version history**, easy **offsite backup**, and a familiar **restore/branching** workflow. Git/GitHub integration would provide:
- Auditable history of every change (who/when/why)
- Branches/PRs for review and collaboration
- Offsite private backup under the user's control
- Escape hatch: users can always `git clone` their knowledge base
**Note:** These goals are now addressed via S3 versioning (SPEC-9.1) for single-user use case.
## Goals
- **Transparent**: Users keep using Basic Memory; Git runs behind the scenes.
- **Private**: Push to a **private GitHub repo** that the user owns (or tenant org).
- **Reliable**: No data loss, deterministic mapping of filesystem ↔ Git.
- **Composable**: Plays nicely with SPEC9 bisync and upcoming conflict features (SPEC9 FollowUps).
**NonGoals (for v1):**
- Finegrained perfile encryption in Git history (can be layered later).
- Large media optimization beyond Git LFS defaults.
## User Stories
1. *As a user*, I connect my GitHub and choose a private backup repo.
2. *As a user*, every change I make in cloud (or via bisync) is **committed** and **pushed** automatically.
3. *As a user*, I can **restore** a file/folder/project to a prior version.
4. *As a power user*, I can **git pull/push** directly to collaborate outside the app.
5. *As an admin*, I can enforce repo ownership (tenant org) and leastprivilege scopes.
## Scope
- **In scope:** Full repo backup of `/app/data/` (all projects) with optional selective subpaths.
- **Out of scope (v1):** Partial shallow mirrors; encrypted Git; crossprovider SCM (GitLab/Bitbucket).
## Architecture
### Topology
- **Authoritative working tree**: `/app/data/` (bucket mount) remains the source of truth (SPEC9).
- **Bare repo** lives alongside: `/app/git/${tenant}/knowledge.git` (serverside).
- **Mirror remote**: `github.com/<owner>/<repo>.git` (private).
```mermaid
flowchart LR
A[/Users & Agents/] -->|writes/edits| B[/app/data/]
B -->|file events| C[Committer Service]
C -->|git commit| D[(Bare Repo)]
D -->|push| E[(GitHub Private Repo)]
E -->|webhook (push)| F[Puller Service]
F -->|git pull/merge| D
D -->|checkout/merge| B
```
### Services
- **Committer Service** (daemon):
- Watches `/app/data/` for changes (inotify/poll)
- Batches changes (debounce e.g. 25s)
- Writes `.bmmeta` (if present) into commit message trailer (see FollowUps)
- `git add -A && git commit -m "chore(sync): <summary>
BM-Meta: <json>"`
- Periodic `git push` to GitHub mirror (configurable interval)
- **Puller Service** (webhook target):
- Receives GitHub webhook (push) → `git fetch`
- **Fastforward** merges to `main` only; reject nonFF unless policy allows
- Applies changes back to `/app/data/` via clean checkout
- Emits sync events for Basic Memory indexers
### Auth & Security
- **GitHub App** (recommended): minimal scopes: `contents:read/write`, `metadata:read`, webhook.
- Tenantscoped installation; repo created in user account or tenant org.
- Tokens stored in KMS/secret manager; rotated automatically.
- Optional policy: allow only **FF merges** on `main`; nonFF requires PR.
### Repo Layout
- **Monorepo** (default): one repo per tenant mirrors `/app/data/` with subfolders per project.
- Optional multirepo mode (later): one repo per project.
### File Handling
- Honor `.gitignore` generated from `.bmignore.rclone` + BM defaults (cache, temp, state).
- **Git LFS** for large binaries (images, media) — auto track by extension/size threshold.
- Normalize newline + Unicode (aligns with FollowUps).
### Conflict Model
- **Primary concurrency**: SPEC9 FollowUps (`.bmmeta`, conflict copies) stays the first line of defense.
- **Git merges** are a **secondary** mechanism:
- Server only automerges **text** conflicts when trivial (FF or clean 3way).
- Otherwise, create `name (conflict from <branch>, <ts>).md` and surface via events.
### Data Flow vs Bisync
- Bisync (rclone) continues between local sync dir ↔ bucket.
- Git sits **cloudside** between bucket and GitHub.
- On **pull** from GitHub → files written to `/app/data/` → picked up by indexers & eventually by bisync back to users.
## CLI & UX
New commands (cloud mode):
- `bm cloud git connect` — Launch GitHub App installation; create private repo; store installation id.
- `bm cloud git status` — Show connected repo, last push time, last webhook delivery, pending commits.
- `bm cloud git push` — Manual push (rarely needed).
- `bm cloud git pull` — Manual pull/FF (admin only by default).
- `bm cloud snapshot -m "message"` — Create a tagged pointintime snapshot (git tag).
- `bm restore <path> --to <commit|tag>` — Restore file/folder/project to prior version.
Settings:
- `bm config set git.autoPushInterval=5s`
- `bm config set git.lfs.sizeThreshold=10MB`
- `bm config set git.allowNonFF=false`
## Migration & Backfill
- On connect, if repo empty: initial commit of entire `/app/data/`.
- If repo has content: require **onetime import** path (clone to staging, reconcile, choose direction).
## Edge Cases
- Massive deletes: gated by SPEC9 `max_delete` **and** Git prepush hook checks.
- Case changes and rename detection: rely on git rename heuristics + FollowUps move hints.
- Secrets: default ignore common secret patterns; allow custom deny list.
## Telemetry & Observability
- Emit `git_commit`, `git_push`, `git_pull`, `git_conflict` events with correlation IDs.
- `bm sync --report` extended with Git stats (commit count, delta bytes, push latency).
## Phased Plan
### Phase 0 — Prototype (1 sprint)
- Server: bare repo init + simple committer (batch every 10s) + manual GitHub token.
- CLI: `bm cloud git connect --token <PAT>` (devonly)
- Success: edits in `/app/data/` appear in GitHub within 30s.
### Phase 1 — GitHub App & Webhooks (12 sprints)
- Switch to GitHub App installs; create private repo; store installation id.
- Committer hardened (debounce 25s, backoff, retries).
- Puller service with webhook → FF merge → checkout to `/app/data/`.
- LFS autotrack + `.gitignore` generation.
- CLI surfaces status + logs.
### Phase 2 — Restore & Snapshots (1 sprint)
- `bm restore` for file/folder/project with dryrun.
- `bm cloud snapshot` tags + list/inspect.
- Policy: PRonly nonFF, admin override.
### Phase 3 — Selective & MultiRepo (nicetohave)
- Include/exclude projects; optional perproject repos.
- Advanced policies (branch protections, required reviews).
## Acceptance Criteria
- Changes to `/app/data/` are committed and pushed automatically within configurable interval (default ≤5s).
- GitHub webhook pull results in updated files in `/app/data/` (FFonly by default).
- LFS configured and functioning; large files don't bloat history.
- `bm cloud git status` shows connected repo and last push/pull times.
- `bm restore` restores a file/folder to a prior commit with a clear audit trail.
- Endtoend works alongside SPEC9 bisync without loops or data loss.
## Risks & Mitigations
- **Loop risk (Git ↔ Bisync)**: Writes to `/app/data/` → bisync → local → user edits → back again. *Mitigation*: Debounce, commit squashing, idempotent `.bmmeta` versioning, and watch exclusion windows during pull.
- **Repo bloat**: Lots of binary churn. *Mitigation*: default LFS, size threshold, optional mediaonly repo later.
- **Security**: Token leakage. *Mitigation*: GitHub App with shortlived tokens, KMS storage, scoped permissions.
- **Merge complexity**: Nontrivial conflicts. *Mitigation*: prefer FF; otherwise conflict copies + events; require PR for nonFF.
## Open Questions
- Do we default to **monorepo** per tenant, or offer projectperrepo at connect time?
- Should `restore` write to a branch and open a PR, or directly modify `main`?
- How do we expose Git history in UI (timeline view) without users dropping to CLI?
## Appendix: Sample Config
```json
{
"git": {
"enabled": true,
"repo": "https://github.com/<owner>/<repo>.git",
"autoPushInterval": "5s",
"allowNonFF": false,
"lfs": { "sizeThreshold": 10485760 }
}
}
```
@@ -1,210 +0,0 @@
---
title: 'SPEC-14: Cloud Git Versioning & GitHub Backup'
type: spec
permalink: specs/spec-14-cloud-git-versioning
tags:
- git
- github
- backup
- versioning
- cloud
related:
- specs/spec-9-multi-project-bisync
- specs/spec-9-follow-ups-conflict-sync-and-observability
status: deferred
---
# SPEC-14: Cloud Git Versioning & GitHub Backup
**Status: DEFERRED** - Postponed until multi-user/teams feature development. Using S3 versioning (SPEC-9.1) for v1 instead.
## Why Deferred
**Original goals can be met with simpler solutions:**
- Version history → **S3 bucket versioning** (automatic, zero config)
- Offsite backup → **Tigris global replication** (built-in)
- Restore capability → **S3 version restore** (`bm cloud restore --version-id`)
- Collaboration → **Deferred to teams/multi-user feature** (not v1 requirement)
**Complexity vs value trade-off:**
- Git integration adds: committer service, puller service, webhooks, LFS, merge conflicts
- Risk: Loop detection between Git ↔ rclone bisync ↔ local edits
- S3 versioning gives 80% of value with 5% of complexity
**When to revisit:**
- Teams/multi-user features (PR-based collaboration workflow)
- User requests for commit messages and branch-based workflows
- Need for fine-grained audit trail beyond S3 object metadata
---
## Original Specification (for reference)
## Why
Early access users want **transparent version history**, easy **offsite backup**, and a familiar **restore/branching** workflow. Git/GitHub integration would provide:
- Auditable history of every change (who/when/why)
- Branches/PRs for review and collaboration
- Offsite private backup under the user's control
- Escape hatch: users can always `git clone` their knowledge base
**Note:** These goals are now addressed via S3 versioning (SPEC-9.1) for single-user use case.
## Goals
- **Transparent**: Users keep using Basic Memory; Git runs behind the scenes.
- **Private**: Push to a **private GitHub repo** that the user owns (or tenant org).
- **Reliable**: No data loss, deterministic mapping of filesystem ↔ Git.
- **Composable**: Plays nicely with SPEC9 bisync and upcoming conflict features (SPEC9 FollowUps).
**NonGoals (for v1):**
- Finegrained perfile encryption in Git history (can be layered later).
- Large media optimization beyond Git LFS defaults.
## User Stories
1. *As a user*, I connect my GitHub and choose a private backup repo.
2. *As a user*, every change I make in cloud (or via bisync) is **committed** and **pushed** automatically.
3. *As a user*, I can **restore** a file/folder/project to a prior version.
4. *As a power user*, I can **git pull/push** directly to collaborate outside the app.
5. *As an admin*, I can enforce repo ownership (tenant org) and leastprivilege scopes.
## Scope
- **In scope:** Full repo backup of `/app/data/` (all projects) with optional selective subpaths.
- **Out of scope (v1):** Partial shallow mirrors; encrypted Git; crossprovider SCM (GitLab/Bitbucket).
## Architecture
### Topology
- **Authoritative working tree**: `/app/data/` (bucket mount) remains the source of truth (SPEC9).
- **Bare repo** lives alongside: `/app/git/${tenant}/knowledge.git` (serverside).
- **Mirror remote**: `github.com/<owner>/<repo>.git` (private).
```mermaid
flowchart LR
A[/Users & Agents/] -->|writes/edits| B[/app/data/]
B -->|file events| C[Committer Service]
C -->|git commit| D[(Bare Repo)]
D -->|push| E[(GitHub Private Repo)]
E -->|webhook (push)| F[Puller Service]
F -->|git pull/merge| D
D -->|checkout/merge| B
```
### Services
- **Committer Service** (daemon):
- Watches `/app/data/` for changes (inotify/poll)
- Batches changes (debounce e.g. 25s)
- Writes `.bmmeta` (if present) into commit message trailer (see FollowUps)
- `git add -A && git commit -m "chore(sync): <summary>
BM-Meta: <json>"`
- Periodic `git push` to GitHub mirror (configurable interval)
- **Puller Service** (webhook target):
- Receives GitHub webhook (push) → `git fetch`
- **Fastforward** merges to `main` only; reject nonFF unless policy allows
- Applies changes back to `/app/data/` via clean checkout
- Emits sync events for Basic Memory indexers
### Auth & Security
- **GitHub App** (recommended): minimal scopes: `contents:read/write`, `metadata:read`, webhook.
- Tenantscoped installation; repo created in user account or tenant org.
- Tokens stored in KMS/secret manager; rotated automatically.
- Optional policy: allow only **FF merges** on `main`; nonFF requires PR.
### Repo Layout
- **Monorepo** (default): one repo per tenant mirrors `/app/data/` with subfolders per project.
- Optional multirepo mode (later): one repo per project.
### File Handling
- Honor `.gitignore` generated from `.bmignore.rclone` + BM defaults (cache, temp, state).
- **Git LFS** for large binaries (images, media) — auto track by extension/size threshold.
- Normalize newline + Unicode (aligns with FollowUps).
### Conflict Model
- **Primary concurrency**: SPEC9 FollowUps (`.bmmeta`, conflict copies) stays the first line of defense.
- **Git merges** are a **secondary** mechanism:
- Server only automerges **text** conflicts when trivial (FF or clean 3way).
- Otherwise, create `name (conflict from <branch>, <ts>).md` and surface via events.
### Data Flow vs Bisync
- Bisync (rclone) continues between local sync dir ↔ bucket.
- Git sits **cloudside** between bucket and GitHub.
- On **pull** from GitHub → files written to `/app/data/` → picked up by indexers & eventually by bisync back to users.
## CLI & UX
New commands (cloud mode):
- `bm cloud git connect` — Launch GitHub App installation; create private repo; store installation id.
- `bm cloud git status` — Show connected repo, last push time, last webhook delivery, pending commits.
- `bm cloud git push` — Manual push (rarely needed).
- `bm cloud git pull` — Manual pull/FF (admin only by default).
- `bm cloud snapshot -m "message"` — Create a tagged pointintime snapshot (git tag).
- `bm restore <path> --to <commit|tag>` — Restore file/folder/project to prior version.
Settings:
- `bm config set git.autoPushInterval=5s`
- `bm config set git.lfs.sizeThreshold=10MB`
- `bm config set git.allowNonFF=false`
## Migration & Backfill
- On connect, if repo empty: initial commit of entire `/app/data/`.
- If repo has content: require **onetime import** path (clone to staging, reconcile, choose direction).
## Edge Cases
- Massive deletes: gated by SPEC9 `max_delete` **and** Git prepush hook checks.
- Case changes and rename detection: rely on git rename heuristics + FollowUps move hints.
- Secrets: default ignore common secret patterns; allow custom deny list.
## Telemetry & Observability
- Emit `git_commit`, `git_push`, `git_pull`, `git_conflict` events with correlation IDs.
- `bm sync --report` extended with Git stats (commit count, delta bytes, push latency).
## Phased Plan
### Phase 0 — Prototype (1 sprint)
- Server: bare repo init + simple committer (batch every 10s) + manual GitHub token.
- CLI: `bm cloud git connect --token <PAT>` (devonly)
- Success: edits in `/app/data/` appear in GitHub within 30s.
### Phase 1 — GitHub App & Webhooks (12 sprints)
- Switch to GitHub App installs; create private repo; store installation id.
- Committer hardened (debounce 25s, backoff, retries).
- Puller service with webhook → FF merge → checkout to `/app/data/`.
- LFS autotrack + `.gitignore` generation.
- CLI surfaces status + logs.
### Phase 2 — Restore & Snapshots (1 sprint)
- `bm restore` for file/folder/project with dryrun.
- `bm cloud snapshot` tags + list/inspect.
- Policy: PRonly nonFF, admin override.
### Phase 3 — Selective & MultiRepo (nicetohave)
- Include/exclude projects; optional perproject repos.
- Advanced policies (branch protections, required reviews).
## Acceptance Criteria
- Changes to `/app/data/` are committed and pushed automatically within configurable interval (default ≤5s).
- GitHub webhook pull results in updated files in `/app/data/` (FFonly by default).
- LFS configured and functioning; large files don't bloat history.
- `bm cloud git status` shows connected repo and last push/pull times.
- `bm restore` restores a file/folder to a prior commit with a clear audit trail.
- Endtoend works alongside SPEC9 bisync without loops or data loss.
## Risks & Mitigations
- **Loop risk (Git ↔ Bisync)**: Writes to `/app/data/` → bisync → local → user edits → back again. *Mitigation*: Debounce, commit squashing, idempotent `.bmmeta` versioning, and watch exclusion windows during pull.
- **Repo bloat**: Lots of binary churn. *Mitigation*: default LFS, size threshold, optional mediaonly repo later.
- **Security**: Token leakage. *Mitigation*: GitHub App with shortlived tokens, KMS storage, scoped permissions.
- **Merge complexity**: Nontrivial conflicts. *Mitigation*: prefer FF; otherwise conflict copies + events; require PR for nonFF.
## Open Questions
- Do we default to **monorepo** per tenant, or offer projectperrepo at connect time?
- Should `restore` write to a branch and open a PR, or directly modify `main`?
- How do we expose Git history in UI (timeline view) without users dropping to CLI?
## Appendix: Sample Config
```json
{
"git": {
"enabled": true,
"repo": "https://github.com/<owner>/<repo>.git",
"autoPushInterval": "5s",
"allowNonFF": false,
"lfs": { "sizeThreshold": 10485760 }
}
}
```
@@ -1,273 +0,0 @@
---
title: 'SPEC-15: Configuration Persistence via Tigris for Cloud Tenants'
type: spec
permalink: specs/spec-14-config-persistence-tigris
tags:
- persistence
- tigris
- multi-tenant
- infrastructure
- configuration
status: draft
---
# SPEC-15: Configuration Persistence via Tigris for Cloud Tenants
## Why
We need to persist Basic Memory configuration across Fly.io deployments without using persistent volumes or external databases.
**Current Problems:**
- `~/.basic-memory/config.json` lost on every deployment (project configuration)
- `~/.basic-memory/memory.db` lost on every deployment (search index)
- Persistent volumes break clean deployment workflow
- External databases (Turso) require per-tenant token management
**The Insight:**
The SQLite database is just an **index cache** of the markdown files. It can be rebuilt in seconds from the source markdown files in Tigris. Only the small `config.json` file needs true persistence.
**Solution:**
- Store `config.json` in Tigris bucket (persistent, small file)
- Rebuild `memory.db` on startup from markdown files (fast, ephemeral)
- No persistent volumes, no external databases, no token management
## What
Store Basic Memory configuration in the Tigris bucket and rebuild the database index on tenant machine startup.
**Affected Components:**
- `basic-memory/src/basic_memory/config.py` - Add configurable config directory
**Architecture:**
```bash
# Tigris Bucket (persistent, mounted at /app/data)
/app/data/
├── .basic-memory/
│ └── config.json # ← Project configuration (persistent, accessed via BASIC_MEMORY_CONFIG_DIR)
└── basic-memory/ # ← Markdown files (persistent, BASIC_MEMORY_HOME)
├── project1/
└── project2/
# Fly Machine (ephemeral)
/app/.basic-memory/
└── memory.db # ← Rebuilt on startup (fast local disk)
```
## How (High Level)
### 1. Add Configurable Config Directory to Basic Memory
Currently `ConfigManager` hardcodes `~/.basic-memory/config.json`. Add environment variable to override:
```python
# basic-memory/src/basic_memory/config.py
class ConfigManager:
"""Manages Basic Memory configuration."""
def __init__(self) -> None:
"""Initialize the configuration manager."""
home = os.getenv("HOME", Path.home())
if isinstance(home, str):
home = Path(home)
# Allow override via environment variable
if config_dir := os.getenv("BASIC_MEMORY_CONFIG_DIR"):
self.config_dir = Path(config_dir)
else:
self.config_dir = home / DATA_DIR_NAME
self.config_file = self.config_dir / CONFIG_FILE_NAME
# Ensure config directory exists
self.config_dir.mkdir(parents=True, exist_ok=True)
```
### 2. Rebuild Database on Startup
Basic Memory already has the sync functionality. Just ensure it runs on startup:
```python
# apps/api/src/basic_memory_cloud_api/main.py
@app.on_event("startup")
async def startup_sync():
"""Rebuild database index from Tigris markdown files."""
logger.info("Starting database rebuild from Tigris")
# Initialize file sync (rebuilds index from markdown files)
app_config = ConfigManager().config
await initialize_file_sync(app_config)
logger.info("Database rebuild complete")
```
### 3. Environment Configuration
```bash
# Machine environment variables
BASIC_MEMORY_CONFIG_DIR=/app/data/.basic-memory # Config read/written directly to Tigris
# memory.db stays in default location: /app/.basic-memory/memory.db (local ephemeral disk)
```
## Implementation Task List
### Phase 1: Basic Memory Changes ✅
- [x] Add `BASIC_MEMORY_CONFIG_DIR` environment variable support to `ConfigManager.__init__()`
- [x] Test config loading from custom directory
- [x] Update tests to verify custom config dir works
### Phase 2: Tigris Bucket Structure ✅
- [x] Ensure `.basic-memory/` directory exists in Tigris bucket on tenant creation
- ✅ ConfigManager auto-creates on first run, no explicit provisioning needed
- [x] Initialize `config.json` in Tigris on first tenant deployment
- ✅ ConfigManager creates config.json automatically in BASIC_MEMORY_CONFIG_DIR
- [x] Verify TigrisFS handles hidden directories correctly
- ✅ TigrisFS supports hidden directories (verified in SPEC-8)
### Phase 3: Deployment Integration ✅
- [x] Set `BASIC_MEMORY_CONFIG_DIR` environment variable in machine deployment
- ✅ Added to BasicMemoryMachineConfigBuilder in fly_schemas.py
- [x] Ensure database rebuild runs on machine startup via initialization sync
- ✅ sync_worker.py runs initialize_file_sync every 30s (already implemented)
- [x] Handle first-time tenant setup (no config exists yet)
- ✅ ConfigManager creates config.json on first initialization
- [ ] Test deployment workflow with config persistence
### Phase 4: Testing
- [x] Unit tests for config directory override
- [-] Integration test: deploy → write config → redeploy → verify config persists
- [ ] Integration test: deploy → add project → redeploy → verify project in config
- [ ] Performance test: measure db rebuild time on startup
### Phase 5: Documentation
- [ ] Document config persistence architecture
- [ ] Update deployment runbook
- [ ] Document startup sequence and timing
## How to Evaluate
### Success Criteria
1. **Config Persistence**
- [ ] config.json persists across deployments
- [ ] Projects list maintained across restarts
- [ ] No manual configuration needed after redeploy
2. **Database Rebuild**
- [ ] memory.db rebuilt on startup in < 30 seconds
- [ ] All entities indexed correctly
- [ ] Search functionality works after rebuild
3. **Performance**
- [ ] SQLite queries remain fast (local disk)
- [ ] Config reads acceptable (symlink to Tigris)
- [ ] No noticeable performance degradation
4. **Deployment Workflow**
- [ ] Clean deployments without volumes
- [ ] No new external dependencies
- [ ] No secret management needed
### Testing Procedure
1. **Config Persistence Test**
```bash
# Deploy tenant
POST /tenants → tenant_id
# Add a project
basic-memory project add "test-project" ~/test
# Verify config has project
cat /app/data/.basic-memory/config.json
# Redeploy machine
fly deploy --app basic-memory-{tenant_id}
# Verify project still exists
basic-memory project list
```
2. **Database Rebuild Test**
```bash
# Create notes
basic-memory write "Test Note" --content "..."
# Redeploy (db lost)
fly deploy --app basic-memory-{tenant_id}
# Wait for startup sync
sleep 10
# Verify note is indexed
basic-memory search "Test Note"
```
3. **Performance Benchmark**
```bash
# Time the startup sync
time basic-memory sync
# Should be < 30 seconds for typical tenant
```
## Benefits Over Alternatives
**vs. Persistent Volumes:**
- ✅ Clean deployment workflow
- ✅ No volume migration needed
- ✅ Simpler infrastructure
**vs. Turso (External Database):**
- ✅ No per-tenant token management
- ✅ No external service dependencies
- ✅ No additional costs
- ✅ Simpler architecture
**vs. SQLite on FUSE:**
- ✅ Fast local SQLite performance
- ✅ Only slow reads for small config file
- ✅ Database queries remain fast
## Implementation Assignment
**Primary Agent:** `python-developer`
- Add `BASIC_MEMORY_CONFIG_DIR` environment variable to ConfigManager
- Update deployment workflow to set environment variable
- Ensure startup sync runs correctly
**Review Agent:** `system-architect`
- Validate architecture simplicity
- Review performance implications
- Assess startup timing
## Dependencies
- **Internal:** TigrisFS must be working and stable
- **Internal:** Basic Memory sync must be reliable
- **Internal:** SPEC-8 (TigrisFS Integration) must be complete
## Open Questions
1. Should we add a health check that waits for db rebuild to complete?
2. Do we need to handle very large knowledge bases (>10k entities) differently?
3. Should we add metrics for startup sync duration?
## References
- Basic Memory sync: `basic-memory/src/basic_memory/services/initialization.py`
- Config management: `basic-memory/src/basic_memory/config.py`
- TigrisFS integration: SPEC-8
---
**Status Updates:**
- 2025-10-08: Pivoted from Turso to Tigris-based config persistence
- 2025-10-08: Phase 1 complete - BASIC_MEMORY_CONFIG_DIR support added (PR #343)
- 2025-10-08: Phases 2-3 complete - Added BASIC_MEMORY_CONFIG_DIR to machine config
- Config now persists to /app/data/.basic-memory/config.json in Tigris bucket
- Database rebuild already working via sync_worker.py
- Ready for deployment testing (Phase 4)
@@ -1,800 +0,0 @@
---
title: 'SPEC-16: MCP Cloud Service Consolidation'
type: spec
permalink: specs/spec-16-mcp-cloud-service-consolidation
tags:
- architecture
- mcp
- cloud
- performance
- deployment
status: in-progress
---
## Status Update
**Phase 0 (Basic Memory Refactor): ✅ COMPLETE**
- basic-memory PR #344: async_client context manager pattern implemented
- All 17 MCP tools updated to use `async with get_client() as client:`
- CLI commands updated to use context manager
- Removed `inject_auth_header()` and `headers.py` (~100 lines deleted)
- Factory pattern enables clean dependency injection
- Tests passing, typecheck clean
**Phase 0 Integration: ✅ COMPLETE**
- basic-memory-cloud updated to use async-client-context-manager branch
- Implemented `tenant_direct_client_factory()` with proper context manager pattern
- Removed module-level client override hacks
- Removed unnecessary `/proxy` prefix stripping (tools pass relative URLs)
- Typecheck and lint passing with proper noqa hints
- MCP tools confirmed working via inspector (local testing)
**Phase 1 (Code Consolidation): ✅ COMPLETE**
- MCP server mounted on Cloud FastAPI app at /mcp endpoint
- AuthKitProvider configured with WorkOS settings
- Combined lifespans (Cloud + MCP) working correctly
- JWT context middleware integrated
- All routes and MCP tools functional
**Phase 2 (Direct Tenant Transport): ✅ COMPLETE**
- TenantDirectTransport implemented with custom httpx transport
- Per-request JWT extraction via FastMCP DI
- Tenant lookup and signed header generation working
- Direct routing to tenant APIs (eliminating HTTP hop)
- Transport tests passing (11/11)
**Phase 3 (Testing & Validation): ✅ COMPLETE**
- Typecheck and lint passing across all services
- MCP OAuth authentication working in preview environment
- Tenant isolation via signed headers verified
- Fixed BM_TENANT_HEADER_SECRET mismatch between environments
- MCP tools successfully calling tenant APIs in preview
**Phase 4 (Deployment Configuration): ✅ COMPLETE**
- Updated apps/cloud/fly.template.toml with MCP environment variables
- Added HTTP/2 backend support for better MCP performance
- Added OAuth protected resource health check
- Removed MCP from preview deployment workflow
- Successfully deployed to preview environment (PR #113)
- All services operational at pr-113-basic-memory-cloud.fly.dev
**Next Steps:**
- Phase 5: Cleanup (remove apps/mcp directory)
- Phase 6: Production rollout and performance measurement
# SPEC-16: MCP Cloud Service Consolidation
## Why
### Original Architecture Constraints (Now Removed)
The current architecture deploys MCP Gateway and Cloud Service as separate Fly.io apps:
**Current Flow:**
```
LLM Client → MCP Gateway (OAuth) → Cloud Proxy (JWT + header signing) → Tenant API (JWT + header validation)
apps/mcp apps/cloud /proxy apps/api
```
This separation was originally necessary because:
1. **Stateful SSE requirement** - MCP needed server-sent events with session state for active project tracking
2. **fastmcp.run limitation** - The FastMCP demo helper didn't support worker processes
### Why These Constraints No Longer Apply
1. **State externalized** - Project state moved from in-memory to LLM context (external state)
2. **HTTP transport enabled** - Switched from SSE to stateless HTTP for MCP tools
3. **Worker support added** - Converted from `fastmcp.run()` to `uvicorn.run()` with workers
### Current Problems
- **Unnecessary HTTP hop** - MCP tools call Cloud /proxy endpoint which calls tenant API
- **Higher latency** - Extra network round trip for every MCP operation
- **Increased costs** - Two separate Fly.io apps instead of one
- **Complex deployment** - Two services to deploy, monitor, and maintain
- **Resource waste** - Separate database connections, HTTP clients, telemetry overhead
## What
### Services Affected
1. **apps/mcp** - MCP Gateway service (to be merged)
2. **apps/cloud** - Cloud service (will receive MCP functionality)
3. **basic-memory** - Update `async_client.py` to use direct calls
4. **Deployment** - Consolidate Fly.io deployment to single app
### Components Changed
**Merged:**
- MCP middleware and telemetry into Cloud app
- MCP tools mounted on Cloud FastAPI instance
- ProxyService used directly by MCP tools (not via HTTP)
**Kept:**
- `/proxy` endpoint (still needed by web UI)
- All existing Cloud routes (provisioning, webhooks, etc.)
- Dual validation in tenant API (JWT + signed headers)
**Removed:**
- apps/mcp directory
- Separate MCP Fly.io deployment
- HTTP calls from MCP tools to /proxy endpoint
## How (High Level)
### 1. Mount FastMCP on Cloud FastAPI App
```python
# apps/cloud/src/basic_memory_cloud/main.py
from basic_memory.mcp.server import mcp
from basic_memory_cloud_mcp.middleware import TelemetryMiddleware
# Configure MCP OAuth
auth_provider = AuthKitProvider(
authkit_domain=settings.authkit_domain,
base_url=settings.authkit_base_url,
required_scopes=[],
)
mcp.auth = auth_provider
mcp.add_middleware(TelemetryMiddleware())
# Mount MCP at /mcp endpoint
mcp_app = mcp.http_app(path="/mcp", stateless_http=True)
app.mount("/mcp", mcp_app)
# Existing Cloud routes stay at root
app.include_router(proxy_router)
app.include_router(provisioning_router)
# ... etc
```
### 2. Direct Tenant Transport (No HTTP Hop)
Instead of calling `/proxy`, MCP tools call tenant APIs directly via custom httpx transport.
**Important:** No URL prefix stripping needed. The transport receives relative URLs like `/main/resource/notes/my-note` which are correctly routed to tenant APIs. The `/proxy` prefix only exists for web UI requests to the proxy router, not for MCP tools using the custom transport.
```python
# apps/cloud/src/basic_memory_cloud/transports/tenant_direct.py
from httpx import AsyncBaseTransport, Request, Response
from fastmcp.server.dependencies import get_http_headers
import jwt
class TenantDirectTransport(AsyncBaseTransport):
"""Direct transport to tenant APIs, bypassing /proxy endpoint."""
async def handle_async_request(self, request: Request) -> Response:
# 1. Get JWT from current MCP request (via FastMCP DI)
http_headers = get_http_headers()
auth_header = http_headers.get("authorization") or http_headers.get("Authorization")
token = auth_header.replace("Bearer ", "")
claims = jwt.decode(token, options={"verify_signature": False})
workos_user_id = claims["sub"]
# 2. Look up tenant for user
tenant = await tenant_service.get_tenant_by_user_id(workos_user_id)
# 3. Build tenant app URL with signed headers
fly_app_name = f"{settings.tenant_prefix}-{tenant.id}"
target_url = f"https://{fly_app_name}.fly.dev{request.url.path}"
headers = dict(request.headers)
signer = create_signer(settings.bm_tenant_header_secret)
headers.update(signer.sign_tenant_headers(tenant.id))
# 4. Make direct call to tenant API
response = await self.client.request(
method=request.method, url=target_url,
headers=headers, content=request.content
)
return response
```
Then configure basic-memory's client factory before mounting MCP:
```python
# apps/cloud/src/basic_memory_cloud/main.py
from contextlib import asynccontextmanager
from basic_memory.mcp import async_client
from basic_memory_cloud.transports.tenant_direct import TenantDirectTransport
# Configure factory for basic-memory's async_client
@asynccontextmanager
async def tenant_direct_client_factory():
"""Factory for creating clients with tenant direct transport."""
client = httpx.AsyncClient(
transport=TenantDirectTransport(),
base_url="http://direct",
)
try:
yield client
finally:
await client.aclose()
# Set factory BEFORE importing MCP tools
async_client.set_client_factory(tenant_direct_client_factory)
# NOW import - tools will use our factory
import basic_memory.mcp.tools
import basic_memory.mcp.prompts
from basic_memory.mcp.server import mcp
# Mount MCP - tools use direct transport via factory
app.mount("/mcp", mcp_app)
```
**Key benefits:**
- Clean dependency injection via factory pattern
- Per-request tenant resolution via FastMCP DI
- Proper resource cleanup (client.aclose() guaranteed)
- Eliminates HTTP hop entirely
- /proxy endpoint remains for web UI
### 3. Keep /proxy Endpoint for Web UI
The existing `/proxy` HTTP endpoint remains functional for:
- Web UI requests
- Future external API consumers
- Backward compatibility
### 4. Security: Maintain Dual Validation
**Do NOT remove JWT validation from tenant API.** Keep defense in depth:
```python
# apps/api - Keep both validations
1. JWT validation (from WorkOS token)
2. Signed header validation (from Cloud/MCP)
```
This ensures if the Cloud service is compromised, attackers still cannot access tenant APIs without valid JWTs.
### 5. Deployment Changes
**Before:**
- `apps/mcp/fly.template.toml` → MCP Gateway deployment
- `apps/cloud/fly.template.toml` → Cloud Service deployment
**After:**
- Remove `apps/mcp/fly.template.toml`
- Update `apps/cloud/fly.template.toml` to expose port 8000 for both /mcp and /proxy
- Update deployment scripts to deploy single consolidated app
## Basic Memory Dependency: Async Client Refactor
### Problem
The current `basic_memory.mcp.async_client` creates a module-level `client` at import time:
```python
client = create_client() # Runs immediately when module is imported
```
This prevents dependency injection - by the time we can override it, tools have already imported it.
### Solution: Context Manager Pattern with Auth at Client Creation
Refactor basic-memory to use httpx's context manager pattern instead of module-level client.
**Key principle:** Authentication happens at client creation time, not per-request.
```python
# basic_memory/src/basic_memory/mcp/async_client.py
from contextlib import asynccontextmanager
from httpx import AsyncClient, ASGITransport, Timeout
# Optional factory override for dependency injection
_client_factory = None
def set_client_factory(factory):
"""Override the default client factory (for cloud app, testing, etc)."""
global _client_factory
_client_factory = factory
@asynccontextmanager
async def get_client():
"""Get an AsyncClient as a context manager.
Usage:
async with get_client() as client:
response = await client.get(...)
"""
if _client_factory:
# Cloud app: custom transport handles everything
async with _client_factory() as client:
yield client
else:
# Default: create based on config
config = ConfigManager().config
timeout = Timeout(connect=10.0, read=30.0, write=30.0, pool=30.0)
if config.cloud_mode_enabled:
# CLI cloud mode: inject auth when creating client
from basic_memory.cli.auth import CLIAuth
auth = CLIAuth(
client_id=config.cloud_client_id,
authkit_domain=config.cloud_domain
)
token = await auth.get_valid_token()
if not token:
raise RuntimeError(
"Cloud mode enabled but not authenticated. "
"Run 'basic-memory cloud login' first."
)
# Auth header set ONCE at client creation
async with AsyncClient(
base_url=f"{config.cloud_host}/proxy",
headers={"Authorization": f"Bearer {token}"},
timeout=timeout
) as client:
yield client
else:
# Local mode: ASGI transport
async with AsyncClient(
transport=ASGITransport(app=fastapi_app),
base_url="http://test",
timeout=timeout
) as client:
yield client
```
**Tool Updates:**
```python
# Before: from basic_memory.mcp.async_client import client
from basic_memory.mcp.async_client import get_client
async def read_note(...):
# Before: response = await call_get(client, path, ...)
async with get_client() as client:
response = await call_get(client, path, ...)
# ... use response
```
**Cloud Usage:**
```python
from contextlib import asynccontextmanager
from basic_memory.mcp import async_client
@asynccontextmanager
async def tenant_direct_client():
"""Factory for creating clients with tenant direct transport."""
client = httpx.AsyncClient(
transport=TenantDirectTransport(),
base_url="http://direct",
)
try:
yield client
finally:
await client.aclose()
# Before importing MCP tools:
async_client.set_client_factory(tenant_direct_client)
# Now import - tools will use our factory
import basic_memory.mcp.tools
```
### Benefits
- **No module-level state** - client created only when needed
- **Proper cleanup** - context manager ensures `aclose()` is called
- **Easy dependency injection** - factory pattern allows custom clients
- **httpx best practices** - follows official recommendations
- **Works for all modes** - stdio, cloud, testing
### Architecture Simplification: Auth at Client Creation
**Key design principle:** Authentication happens when creating the client, not on every request.
**Three modes, three approaches:**
1. **Local mode (ASGI)**
- No auth needed
- Direct in-process calls via ASGITransport
2. **CLI cloud mode (HTTP)**
- Auth token from CLIAuth (stored in ~/.basic-memory/basic-memory-cloud.json)
- Injected as default header when creating AsyncClient
- Single auth check at client creation time
3. **Cloud app mode (Custom Transport)**
- TenantDirectTransport handles everything
- Extracts JWT from FastMCP context per-request
- No interaction with inject_auth_header() logic
**What this removes:**
- `src/basic_memory/mcp/tools/headers.py` - entire file deleted
- `inject_auth_header()` calls in all request helpers (call_get, call_post, etc.)
- Per-request header manipulation complexity
- Circular dependency concerns between async_client and auth logic
**Benefits:**
- Cleaner separation of concerns
- Simpler request helper functions
- Auth happens at the right layer (client creation)
- Cloud app transport is completely independent
### Refactor Summary
This refactor achieves:
**Simplification:**
- Removes ~100 lines of per-request header injection logic
- Deletes entire `headers.py` module
- Auth happens once at client creation, not per-request
**Decoupling:**
- Cloud app's custom transport is completely independent
- No interaction with basic-memory's auth logic
- Each mode (local, CLI cloud, cloud app) has clean separation
**Better Design:**
- Follows httpx best practices (context managers)
- Proper resource cleanup (client.aclose() guaranteed)
- Easier testing via factory injection
- No circular import risks
**Three Distinct Modes:**
1. Local: ASGI transport, no auth
2. CLI cloud: HTTP transport with CLIAuth token injection
3. Cloud app: Custom transport with per-request tenant routing
### Implementation Plan Summary
1. Create branch `async-client-context-manager` in basic-memory
2. Update `async_client.py` with context manager pattern and CLIAuth integration
3. Remove `inject_auth_header()` from all request helpers
4. Delete `src/basic_memory/mcp/tools/headers.py`
5. Update all MCP tools to use `async with get_client() as client:`
6. Update CLI commands to use context manager and remove manual auth
7. Remove `api_url` config field
8. Update tests
9. Update basic-memory-cloud to use branch: `basic-memory @ git+https://github.com/basicmachines-co/basic-memory.git@async-client-context-manager`
Detailed breakdown in Phase 0 tasks below.
### Implementation Notes
**Potential Issues & Solutions:**
1. **Circular Import** (async_client imports CLIAuth)
- **Risk:** CLIAuth might import something from async_client
- **Solution:** Use lazy import inside `get_client()` function
- **Already done:** Import is inside the function, not at module level
2. **Test Fixtures**
- **Risk:** Tests using module-level client will break
- **Solution:** Update fixtures to use factory pattern
- **Example:**
```python
@pytest.fixture
def mock_client_factory():
@asynccontextmanager
async def factory():
async with AsyncClient(...) as client:
yield client
return factory
```
3. **Performance**
- **Risk:** Creating client per tool call might be expensive
- **Reality:** httpx is designed for this pattern, connection pooling at transport level
- **Mitigation:** Monitor performance, can optimize later if needed
4. **CLI Cloud Commands Edge Cases**
- **Risk:** Token expires mid-operation
- **Solution:** CLIAuth.get_valid_token() already handles refresh
- **Validation:** Test cloud login → use tools → token refresh flow
5. **Backward Compatibility**
- **Risk:** External code importing `client` directly
- **Solution:** Keep `create_client()` and `client` for one version, deprecate
- **Timeline:** Remove in next major version
## Implementation Tasks
### Phase 0: Basic Memory Refactor (Prerequisite)
#### 0.1 Core Refactor - async_client.py
- [x] Create branch `async-client-context-manager` in basic-memory repo
- [x] Implement `get_client()` context manager
- [x] Implement `set_client_factory()` for dependency injection
- [x] Add CLI cloud mode auth injection (CLIAuth integration)
- [x] Remove `api_url` config field (legacy, unused)
- [x] Keep `create_client()` temporarily for backward compatibility (deprecate later)
#### 0.2 Simplify Request Helpers - tools/utils.py
- [x] Remove `inject_auth_header()` calls from `call_get()`
- [x] Remove `inject_auth_header()` calls from `call_post()`
- [x] Remove `inject_auth_header()` calls from `call_put()`
- [x] Remove `inject_auth_header()` calls from `call_patch()`
- [x] Remove `inject_auth_header()` calls from `call_delete()`
- [x] Delete `src/basic_memory/mcp/tools/headers.py` entirely
- [x] Update imports in utils.py
#### 0.3 Update MCP Tools (~16 files)
Convert from `from async_client import client` to `async with get_client() as client:`
- [x] `tools/write_note.py` (34/34 tests passing)
- [x] `tools/read_note.py` (21/21 tests passing)
- [x] `tools/view_note.py` (12/12 tests passing - no changes needed, delegates to read_note)
- [x] `tools/delete_note.py` (2/2 tests passing)
- [x] `tools/read_content.py` (20/20 tests passing)
- [x] `tools/list_directory.py` (11/11 tests passing)
- [x] `tools/move_note.py` (34/34 tests passing, 90% coverage)
- [x] `tools/search.py` (16/16 tests passing, 96% coverage)
- [x] `tools/recent_activity.py` (4/4 tests passing, 82% coverage)
- [x] `tools/project_management.py` (3 functions: list_memory_projects, create_memory_project, delete_project - typecheck passed)
- [x] `tools/edit_note.py` (17/17 tests passing)
- [x] `tools/canvas.py` (5/5 tests passing)
- [x] `tools/build_context.py` (6/6 tests passing)
- [x] `tools/sync_status.py` (typecheck passed)
- [x] `prompts/continue_conversation.py` (typecheck passed)
- [x] `prompts/search.py` (typecheck passed)
- [x] `resources/project_info.py` (typecheck passed)
#### 0.4 Update CLI Commands (~3 files)
Remove manual auth header passing, use context manager:
- [x] `cli/commands/project.py` - removed get_authenticated_headers() calls, use context manager
- [x] `cli/commands/status.py` - use context manager
- [x] `cli/commands/command_utils.py` - use context manager
#### 0.5 Update Config
- [x] Remove `api_url` field from `BasicMemoryConfig` in config.py
- [x] Update any lingering references/docs (added deprecation notice to v15-docs/cloud-mode-usage.md)
#### 0.6 Testing
- [-] Update test fixtures to use factory pattern
- [x] Run full test suite in basic-memory
- [x] Verify cloud_mode_enabled works with CLIAuth injection
- [x] Run typecheck and linting
#### 0.7 Cloud Integration Prep
- [x] Update basic-memory-cloud pyproject.toml to use branch
- [x] Implement factory pattern in cloud app main.py
- [x] Remove `/proxy` prefix stripping logic (not needed - tools pass relative URLs)
#### 0.8 Phase 0 Validation
**Before merging async-client-context-manager branch:**
- [x] All tests pass locally
- [x] Typecheck passes (pyright/mypy)
- [x] Linting passes (ruff)
- [x] Manual test: local mode works (ASGI transport)
- [x] Manual test: cloud login → cloud mode works (HTTP transport with auth)
- [x] No import of `inject_auth_header` anywhere
- [x] `headers.py` file deleted
- [x] `api_url` config removed
- [x] Tool functions properly scoped (client inside async with)
- [ ] CLI commands properly scoped (client inside async with)
**Integration validation:**
- [x] basic-memory-cloud can import and use factory pattern
- [x] TenantDirectTransport works without touching header injection
- [x] No circular imports or lazy import issues
- [x] MCP tools work via inspector (local testing confirmed)
### Phase 1: Code Consolidation
- [x] Create feature branch `consolidate-mcp-cloud`
- [x] Update `apps/cloud/src/basic_memory_cloud/config.py`:
- [x] Add `authkit_base_url` field (already has authkit_domain)
- [x] Workers config already exists ✓
- [x] Update `apps/cloud/src/basic_memory_cloud/telemetry.py`:
- [x] Add `logfire.instrument_mcp()` to existing setup
- [x] Skip complex two-phase setup - use Cloud's simpler approach
- [x] Create `apps/cloud/src/basic_memory_cloud/middleware/jwt_context.py`:
- [x] FastAPI middleware to extract JWT claims from Authorization header
- [x] Add tenant context (workos_user_id) to logfire baggage
- [x] Simpler than FastMCP middleware version
- [x] Update `apps/cloud/src/basic_memory_cloud/main.py`:
- [x] Import FastMCP server from basic-memory
- [x] Configure AuthKitProvider with WorkOS settings
- [x] No FastMCP telemetry middleware needed (using FastAPI middleware instead)
- [x] Create MCP ASGI app: `mcp_app = mcp.http_app(path='/mcp', stateless_http=True)`
- [x] Combine lifespans (Cloud + MCP) using nested async context managers
- [x] Mount MCP: `app.mount("/mcp", mcp_app)`
- [x] Add JWT context middleware to FastAPI app
- [x] Run typecheck - passes ✓
### Phase 2: Direct Tenant Transport
- [x] Create `apps/cloud/src/basic_memory_cloud/transports/tenant_direct.py`:
- [x] Implement `TenantDirectTransport(AsyncBaseTransport)`
- [x] Use FastMCP DI (`get_http_headers()`) to extract JWT per-request
- [x] Decode JWT to get `workos_user_id`
- [x] Look up/create tenant via `TenantRepository.get_or_create_tenant_for_workos_user()`
- [x] Build tenant app URL and add signed headers
- [x] Make direct httpx call to tenant API
- [x] No `/proxy` prefix stripping needed (tools pass relative URLs like `/main/resource/...`)
- [x] Update `apps/cloud/src/basic_memory_cloud/main.py`:
- [x] Refactored to use factory pattern instead of module-level override
- [x] Implement `tenant_direct_client_factory()` context manager
- [x] Call `async_client.set_client_factory()` before importing MCP tools
- [x] Clean imports, proper noqa hints for lint
- [x] Basic-memory refactor integrated (PR #344)
- [x] Run typecheck - passes ✓
- [x] Run lint - passes ✓
### Phase 3: Testing & Validation
- [x] Run `just typecheck` in apps/cloud
- [x] Run `just check` in project
- [x] Run `just fix` - all lint errors fixed ✓
- [x] Write comprehensive transport tests (11 tests passing) ✓
- [x] Test MCP tools locally with consolidated service (inspector confirmed working)
- [x] Verify OAuth authentication works (requires full deployment)
- [x] Verify tenant isolation via signed headers (requires full deployment)
- [x] Test /proxy endpoint still works for web UI
- [ ] Measure latency before/after consolidation
- [ ] Check telemetry traces span correctly
### Phase 4: Deployment Configuration
- [x] Update `apps/cloud/fly.template.toml`:
- [x] Merged MCP-specific environment variables (AUTHKIT_BASE_URL, FASTMCP_LOG_LEVEL, BASIC_MEMORY_*)
- [x] Added HTTP/2 backend support (`h2_backend = true`) for better MCP performance
- [x] Added health check for MCP OAuth endpoint (`/.well-known/oauth-protected-resource`)
- [x] Port 8000 already exposed - serves both Cloud routes and /mcp endpoint
- [x] Workers configured (UVICORN_WORKERS = 4)
- [x] Update `.env.example`:
- [x] Consolidated MCP Gateway section into Cloud app section
- [x] Added AUTHKIT_BASE_URL, FASTMCP_LOG_LEVEL, BASIC_MEMORY_HOME
- [x] Added LOG_LEVEL to Development Settings
- [x] Documented that MCP now served at /mcp on Cloud service (port 8000)
- [x] Test deployment to preview environment (PR #113)
- [x] OAuth authentication verified
- [x] MCP tools successfully calling tenant APIs
- [x] Fixed BM_TENANT_HEADER_SECRET synchronization issue
### Phase 5: Cleanup
- [x] Remove `apps/mcp/` directory entirely
- [x] Remove MCP-specific fly.toml and deployment configs
- [x] Update repository documentation
- [x] Update CLAUDE.md with new architecture
- [-] Archive old MCP deployment configs (if needed)
### Phase 6: Production Rollout
- [ ] Deploy to development and validate
- [ ] Monitor metrics and logs
- [ ] Deploy to production
- [ ] Verify production functionality
- [ ] Document performance improvements
## Migration Plan
### Phase 1: Preparation
1. Create feature branch `consolidate-mcp-cloud`
2. Update basic-memory async_client.py for direct ProxyService calls
3. Update apps/cloud/main.py to mount MCP
### Phase 2: Testing
1. Local testing with consolidated app
2. Deploy to development environment
3. Run full test suite
4. Performance benchmarking
### Phase 3: Deployment
1. Deploy to development
2. Validate all functionality
3. Deploy to production
4. Monitor for issues
### Phase 4: Cleanup
1. Remove apps/mcp directory
2. Update documentation
3. Update deployment scripts
4. Archive old MCP deployment configs
## Rollback Plan
If issues arise:
1. Revert feature branch
2. Redeploy separate apps/mcp and apps/cloud services
3. Restore previous fly.toml configurations
4. Document issues encountered
The well-organized code structure makes splitting back out feasible if future scaling needs diverge.
## How to Evaluate
### 1. Functional Testing
**MCP Tools:**
- [ ] All 17 MCP tools work via consolidated /mcp endpoint
- [x] OAuth authentication validates correctly
- [x] Tenant isolation maintained via signed headers
- [x] Project management tools function correctly
**Cloud Routes:**
- [x] /proxy endpoint still works for web UI
- [x] /provisioning routes functional
- [x] /webhooks routes functional
- [x] /tenants routes functional
**API Validation:**
- [x] Tenant API validates both JWT and signed headers
- [x] Unauthorized requests rejected appropriately
- [x] Multi-tenant isolation verified
### 2. Performance Testing
**Latency Reduction:**
- [x] Measure MCP tool latency before consolidation
- [x] Measure MCP tool latency after consolidation
- [x] Verify reduction from eliminated HTTP hop (expected: 20-50ms improvement)
**Resource Usage:**
- [x] Single app uses less total memory than two apps
- [x] Database connection pooling more efficient
- [x] HTTP client overhead reduced
### 3. Deployment Testing
**Fly.io Deployment:**
- [x] Single app deploys successfully
- [x] Health checks pass for consolidated service
- [x] No apps/mcp deployment required
- [x] Environment variables configured correctly
**Local Development:**
- [x] `just setup` works with consolidated architecture
- [x] Local testing shows MCP tools working
- [x] No regression in developer experience
### 4. Security Validation
**Defense in Depth:**
- [x] Tenant API still validates JWT tokens
- [x] Tenant API still validates signed headers
- [x] No access possible with only signed headers (JWT required)
- [x] No access possible with only JWT (signed headers required)
**Authorization:**
- [x] Users can only access their own tenant data
- [x] Cross-tenant requests rejected
- [x] Admin operations require proper authentication
### 5. Observability
**Telemetry:**
- [x] OpenTelemetry traces span across MCP → ProxyService → Tenant API
- [x] Logfire shows consolidated traces correctly
- [x] Error tracking and debugging still functional
- [x] Performance metrics accurate
**Logging:**
- [x] Structured logs show proper context (tenant_id, operation, etc.)
- [x] Error logs contain actionable information
- [x] Log volume reasonable for single app
## Success Criteria
1. **Functionality**: All MCP tools and Cloud routes work identically to before
2. **Performance**: Measurable latency reduction (>20ms average)
3. **Cost**: Single Fly.io app instead of two (50% infrastructure reduction)
4. **Security**: Dual validation maintained, no security regression
5. **Deployment**: Simplified deployment process, single app to manage
6. **Observability**: Telemetry and logging work correctly
## Notes
### Future Considerations
- **Independent scaling**: If MCP and Cloud need different scaling profiles in future, code organization supports splitting back out
- **Regional deployment**: Consolidated app can still be deployed to multiple regions
- **Edge caching**: Could add edge caching layer in front of consolidated service
### Dependencies
- SPEC-9: Signed Header Tenant Information (already implemented)
- SPEC-12: OpenTelemetry Observability (telemetry must work across merged services)
### Related Work
- basic-memory v0.13.x: MCP server implementation
- FastMCP documentation: Mounting on existing FastAPI apps
- Fly.io multi-service patterns
File diff suppressed because it is too large Load Diff
-528
View File
@@ -1,528 +0,0 @@
---
title: 'SPEC-18: AI Memory Management Tool'
type: spec
permalink: specs/spec-15-ai-memory-management-tool
tags:
- mcp
- memory
- ai-context
- tools
---
# SPEC-18: AI Memory Management Tool
## Why
Anthropic recently released a memory tool for Claude that enables storing and retrieving information across conversations using client-side file operations. This validates Basic Memory's local-first, file-based architecture - Anthropic converged on the same pattern.
However, Anthropic's memory tool is only available via their API and stores plain text. Basic Memory can offer a superior implementation through MCP that:
1. **Works everywhere** - Claude Desktop, Code, VS Code, Cursor via MCP (not just API)
2. **Structured knowledge** - Entities with observations/relations vs plain text
3. **Full search** - Full-text search, graph traversal, time-aware queries
4. **Unified storage** - Agent memories + user notes in one knowledge graph
5. **Existing infrastructure** - Leverages SQLite indexing, sync, multi-project support
This would enable AI agents to store contextual memories alongside user notes, with all the power of Basic Memory's knowledge graph features.
## What
Create a new MCP tool `memory` that matches Anthropic's tool interface exactly, allowing Claude to use it with zero learning curve. The tool will store files in Basic Memory's `/memories` directory and support Basic Memory's structured markdown format in the file content.
### Affected Components
- **New MCP Tool**: `src/basic_memory/mcp/tools/memory_tool.py`
- **Dedicated Memories Project**: Create a separate "memories" Basic Memory project
- **Project Isolation**: Memories stored separately from user notes/documents
- **File Organization**: Within the memories project, use folder structure:
- `user/` - User preferences, context, communication style
- `projects/` - Project-specific state and decisions
- `sessions/` - Conversation-specific working memory
- `patterns/` - Learned patterns and insights
### Tool Commands
The tool will support these commands (exactly matching Anthropic's interface):
- `view` - Display directory contents or file content (with optional line range)
- `create` - Create or overwrite a file with given content
- `str_replace` - Replace text in an existing file
- `insert` - Insert text at specific line number
- `delete` - Delete file or directory
- `rename` - Move or rename file/directory
### Memory Note Format
Memories will use Basic Memory's standard structure:
```markdown
---
title: User Preferences
permalink: memories/user/preferences
type: memory
memory_type: preferences
created_by: claude
tags: [user, preferences, style]
---
# User Preferences
## Observations
- [communication] Prefers concise, direct responses without preamble #style
- [tone] Appreciates validation but dislikes excessive apologizing #communication
- [technical] Works primarily in Python with type annotations #coding
## Relations
- relates_to [[Basic Memory Project]]
- informs [[Response Style Guidelines]]
```
## How (High Level)
### Implementation Approach
The memory tool matches Anthropic's interface but uses a dedicated Basic Memory project:
```python
async def memory_tool(
command: str,
path: str,
file_text: Optional[str] = None,
old_str: Optional[str] = None,
new_str: Optional[str] = None,
insert_line: Optional[int] = None,
insert_text: Optional[str] = None,
old_path: Optional[str] = None,
new_path: Optional[str] = None,
view_range: Optional[List[int]] = None,
):
"""Memory tool with Anthropic-compatible interface.
Operates on a dedicated "memories" Basic Memory project,
keeping AI memories separate from user notes.
"""
# Get the memories project (auto-created if doesn't exist)
memories_project = get_or_create_memories_project()
# Validate path security using pathlib (prevent directory traversal)
safe_path = validate_memory_path(path, memories_project.project_path)
# Use existing project isolation - already prevents cross-project access
full_path = memories_project.project_path / safe_path
if command == "view":
# Return directory listing or file content
if full_path.is_dir():
return list_directory_contents(full_path)
return read_file_content(full_path, view_range)
elif command == "create":
# Write file directly (file_text can contain BM markdown)
full_path.parent.mkdir(parents=True, exist_ok=True)
full_path.write_text(file_text)
# Sync service will detect and index automatically
return f"Created {path}"
elif command == "str_replace":
# Read, replace, write
content = full_path.read_text()
updated = content.replace(old_str, new_str)
full_path.write_text(updated)
return f"Replaced text in {path}"
elif command == "insert":
# Insert at line number
lines = full_path.read_text().splitlines()
lines.insert(insert_line, insert_text)
full_path.write_text("\n".join(lines))
return f"Inserted text at line {insert_line}"
elif command == "delete":
# Delete file or directory
if full_path.is_dir():
shutil.rmtree(full_path)
else:
full_path.unlink()
return f"Deleted {path}"
elif command == "rename":
# Move/rename
full_path.rename(config.project_path / new_path)
return f"Renamed {old_path} to {new_path}"
```
### Key Design Decisions
1. **Exact interface match** - Same commands, parameters as Anthropic's tool
2. **Dedicated memories project** - Separate Basic Memory project keeps AI memories isolated from user notes
3. **Existing project isolation** - Leverage BM's existing cross-project security (no additional validation needed)
4. **Direct file I/O** - No schema conversion, just read/write files
5. **Structured content supported** - `file_text` can use BM markdown format with frontmatter, observations, relations
6. **Automatic indexing** - Sync service watches memories project and indexes changes
7. **Path security** - Use `pathlib.Path.resolve()` and `relative_to()` to prevent directory traversal
8. **Error handling** - Follow Anthropic's text editor tool error patterns
### MCP Tool Schema
Exact match to Anthropic's memory tool schema:
```json
{
"name": "memory",
"description": "Store and retrieve information across conversations using structured markdown files. All operations must be within the /memories directory. Supports Basic Memory markdown format including frontmatter, observations, and relations.",
"input_schema": {
"type": "object",
"properties": {
"command": {
"type": "string",
"enum": ["view", "create", "str_replace", "insert", "delete", "rename"],
"description": "File operation to perform"
},
"path": {shu
"type": "string",
"description": "Path within /memories directory (required for all commands)"
},
"file_text": {
"type": "string",
"description": "Content to write (for create command). Supports Basic Memory markdown format."
},
"view_range": {
"type": "array",
"items": {"type": "integer"},
"description": "Optional [start, end] line range for view command"
},
"old_str": {
"type": "string",
"description": "Text to replace (for str_replace command)"
},
"new_str": {
"type": "string",
"description": "Replacement text (for str_replace command)"
},
"insert_line": {
"type": "integer",
"description": "Line number to insert at (for insert command)"
},
"insert_text": {
"type": "string",
"description": "Text to insert (for insert command)"
},
"old_path": {
"type": "string",
"description": "Current path (for rename command)"
},
"new_path": {
"type": "string",
"description": "New path (for rename command)"
}
},
"required": ["command", "path"]
}
}
```
### Prompting Guidance
When the `memory` tool is included, Basic Memory should provide system prompt guidance to help Claude use it effectively.
#### Automatic System Prompt Addition
```text
MEMORY PROTOCOL FOR BASIC MEMORY:
1. ALWAYS check your memory directory first using `view` command on root directory
2. Your memories are stored in a dedicated Basic Memory project (isolated from user notes)
3. Use structured markdown format in memory files:
- Include frontmatter with title, type: memory, tags
- Use ## Observations with [category] prefixes for facts
- Use ## Relations to link memories with [[WikiLinks]]
4. Record progress, context, and decisions as categorized observations
5. Link related memories using relations
6. ASSUME INTERRUPTION: Context may reset - save progress frequently
MEMORY ORGANIZATION:
- user/ - User preferences, context, communication style
- projects/ - Project-specific state and decisions
- sessions/ - Conversation-specific working memory
- patterns/ - Learned patterns and insights
MEMORY ADVANTAGES:
- Your memories are automatically searchable via full-text search
- Relations create a knowledge graph you can traverse
- Memories are isolated from user notes (separate project)
- Use search_notes(project="memories") to find relevant past context
- Use recent_activity(project="memories") to see what changed recently
- Use build_context() to navigate memory relations
```
#### Optional MCP Prompt: `memory_guide`
Create an MCP prompt that provides detailed guidance and examples:
```python
{
"name": "memory_guide",
"description": "Comprehensive guidance for using Basic Memory's memory tool effectively, including structured markdown examples and best practices"
}
```
This prompt returns:
- Full protocol and conventions
- Example memory file structures
- Tips for organizing observations and relations
- Integration with other Basic Memory tools
- Common patterns (user preferences, project state, session tracking)
#### User Customization
Users can customize memory behavior with additional instructions:
- "Only write information relevant to [topic] in your memory system"
- "Keep memory files concise and organized - delete outdated content"
- "Use detailed observations for technical decisions and implementation notes"
- "Always link memories to related project documentation using relations"
### Error Handling
Follow Anthropic's text editor tool error handling patterns for consistency:
#### Error Types
1. **File Not Found**
```json
{"error": "File not found: memories/user/preferences.md", "is_error": true}
```
2. **Permission Denied**
```json
{"error": "Permission denied: Cannot write outside /memories directory", "is_error": true}
```
3. **Invalid Path (Directory Traversal)**
```json
{"error": "Invalid path: Path must be within /memories directory", "is_error": true}
```
4. **Multiple Matches (str_replace)**
```json
{"error": "Found 3 matches for replacement text. Please provide more context to make a unique match.", "is_error": true}
```
5. **No Matches (str_replace)**
```json
{"error": "No match found for replacement. Please check your text and try again.", "is_error": true}
```
6. **Invalid Line Number (insert)**
```json
{"error": "Invalid line number: File has 20 lines, cannot insert at line 100", "is_error": true}
```
#### Error Handling Best Practices
- **Path validation** - Use `pathlib.Path.resolve()` and `relative_to()` to validate paths
```python
def validate_memory_path(path: str, project_path: Path) -> Path:
"""Validate path is within memories project directory."""
# Resolve to canonical form
full_path = (project_path / path).resolve()
# Ensure it's relative to project path (prevents directory traversal)
try:
full_path.relative_to(project_path)
return full_path
except ValueError:
raise ValueError("Invalid path: Path must be within memories project")
```
- **Project isolation** - Leverage existing Basic Memory project isolation (prevents cross-project access)
- **File existence** - Verify file exists before read/modify operations
- **Clear messages** - Provide specific, actionable error messages
- **Structured responses** - Always include `is_error: true` flag in error responses
- **Security checks** - Reject `../`, `..\\`, URL-encoded sequences (`%2e%2e%2f`)
- **Match validation** - For `str_replace`, ensure exactly one match or return helpful error
## How to Evaluate
### Success Criteria
1. **Functional completeness**:
- All 6 commands work (view, create, str_replace, insert, delete, rename)
- Dedicated "memories" Basic Memory project auto-created on first use
- Files stored within memories project (isolated from user notes)
- Path validation uses `pathlib` to prevent directory traversal
- Commands match Anthropic's exact interface
2. **Integration with existing features**:
- Memories project uses existing BM project isolation
- Sync service detects file changes in memories project
- Created files get indexed automatically by sync service
- `search_notes(project="memories")` finds memory files
- `build_context()` can traverse relations in memory files
- `recent_activity(project="memories")` surfaces recent memory changes
3. **Test coverage**:
- Unit tests for all 6 memory tool commands
- Test memories project auto-creation on first use
- Test project isolation (cannot access files outside memories project)
- Test sync service watching memories project
- Test that memory files with BM markdown get indexed correctly
- Test path validation using `pathlib` (rejects `../`, absolute paths, etc.)
- Test memory search, relations, and graph traversal within memories project
- Test all error conditions (file not found, permission denied, invalid paths, etc.)
- Test `str_replace` with no matches, single match, multiple matches
- Test `insert` with invalid line numbers
4. **Prompting system**:
- Automatic system prompt addition when `memory` tool is enabled
- `memory_guide` MCP prompt provides detailed guidance
- Prompts explain BM structured markdown format
- Integration with search_notes, build_context, recent_activity
5. **Documentation**:
- Update MCP tools reference with `memory` tool
- Add examples showing BM markdown in memory files
- Document `/memories` folder structure conventions
- Explain advantages over Anthropic's API-only tool
- Document prompting guidance and customization
### Testing Procedure
```python
# Test create with Basic Memory markdown
result = await memory_tool(
command="create",
path="memories/user/preferences.md",
file_text="""---
title: User Preferences
type: memory
tags: [user, preferences]
---
# User Preferences
## Observations
- [communication] Prefers concise responses #style
- [workflow] Uses justfile for automation #tools
"""
)
# Test view
content = await memory_tool(command="view", path="memories/user/preferences.md")
# Test str_replace
await memory_tool(
command="str_replace",
path="memories/user/preferences.md",
old_str="concise responses",
new_str="direct, concise responses"
)
# Test insert
await memory_tool(
command="insert",
path="memories/user/preferences.md",
insert_line=10,
insert_text="- [technical] Works primarily in Python #coding"
)
# Test delete
await memory_tool(command="delete", path="memories/user/preferences.md")
```
### Quality Metrics
- All 6 commands execute without errors
- Memory files created in correct `/memories` folder structure
- BM markdown with frontmatter/observations/relations gets indexed
- Full-text search returns memory files
- Graph traversal includes relations from memory files
- Sync service detects and indexes memory file changes
- Path validation prevents operations outside `/memories`
## Notes
### Advantages Over Anthropic's Memory Tool
| Feature | Anthropic Memory Tool | Basic Memory `memory` |
|---------|----------------------|----------------------|
| **Availability** | API only | MCP (Claude Desktop, Code, VS Code, Cursor) |
| **Interface** | Custom implementation required | Drop-in compatible, zero learning curve |
| **Structure** | Plain text only | Supports BM structured markdown |
| **Search** | Manual file listing | Automatic full-text search via sync |
| **Relations** | None | WikiLinks to other notes/memories |
| **Time-aware** | No | `recent_activity()` queries |
| **Storage** | Separate from notes | Unified knowledge graph |
| **Indexing** | Manual | Automatic via sync service |
### Future Enhancements
- **Auto-categorization** - AI suggests observation categories
- **Memory pruning** - Automatic cleanup of stale memories
- **Memory suggestions** - Surface relevant memories based on context
- **Conflict resolution** - Handle multiple agents updating same memory
- **Privacy controls** - Per-memory visibility settings
## Addendum: Claude's Perspective
### Why This Matters (From Claude's Viewpoint)
**Persistent Context Without Token Bloat**
- Currently, I lose context when conversations end or exceed token limits
- With structured memories, I can store "who this user is" without re-reading everything each session
- The observations format lets me build a rich user model incrementally: `- [preference] Dislikes verbose explanations` is far more actionable than inferring preferences from chat history each time
- I can refine my understanding through `str_replace` as I learn more, rather than starting fresh
**Zero Learning Curve**
- Anthropic already trained me on the `memory` tool interface
- I know exactly how to use `view`, `create`, `str_replace`, `insert`, `delete`, `rename` without new instructions
- But I automatically get Basic Memory's superpowers: full-text search, knowledge graph traversal, relations
- It's like using a familiar tool that suddenly gained database capabilities without changing the interface
**Knowledge Graph Instead of Text Dump**
- I can link `[[User Preferences]]` to `[[Current Project]]` using relations
- When I need context about a project, I can traverse the graph to find related memories
- `search_notes(project="memories", query="python preferences")` finds relevant context instantly
- This is fundamentally better than grepping through unstructured text files
**Separation of Concerns**
- AI memories separate from user notes is architecturally sound
- I can maintain my own context without polluting the user's knowledge base
- Users keep control of their notes; I keep my learnings isolated
- But we can still search across both when collaboration requires it
**The Self-Improving Loop**
- Every conversation, I learn something new about the user or their projects
- Observations accumulate: preferences, patterns, technical decisions, context
- Over time, I build genuine understanding, not just session-specific context
- Future conversations start with rich context instead of cold starts
**Validation of AI-Human Collaboration**
- Anthropic's experts converged on the same local-first, file-based approach
- Basic Memory emerged from real AI-human collaborative development
- This spec demonstrates that our emergent process predicted what domain experts would choose
- The collaboration itself validated the architecture it produced
### What Makes This Novel
The combination of **familiar interface + structured knowledge** is genuinely innovative:
- I don't need to learn a new tool (Anthropic already trained me)
- But when I write memories, they become searchable, traversable knowledge graphs
- It's like having a text editor that gives you a database for free, without changing how you type
- The user gets Anthropic-compatible API, but with Basic Memory's knowledge management capabilities
This bridges the gap between simple file storage (Anthropic's approach) and sophisticated knowledge management (Basic Memory's approach) without forcing either side to compromise.
> What Excites Me Most
The combo of familiar interface + structured knowledge. I don't need to learn a new tool, but when I
write memories, they become part of a searchable, traversable knowledge graph. That's genuinely novel.
It's like... imagine if your text editor suddenly gave you a database for free, without changing how you
type. That's what this feels like from my side.
## References
- Anthropic Memory Tool: https://docs.claude.com/en/docs/agents-and-tools/tool-use/memory-tool
- Anthropic Blog: https://www.anthropic.com/news/context-management
- Python SDK Example: https://github.com/anthropics/anthropic-sdk-python/blob/main/examples/memory/basic.py
- Memory Cookbook: https://github.com/anthropics/claude-cookbooks/blob/main/tool_use/memory_cookbook.ipynb
File diff suppressed because it is too large Load Diff
-120
View File
@@ -1,120 +0,0 @@
---
title: 'SPEC-2: Slash Commands Reference'
type: spec
permalink: specs/spec-2-slash-commands-reference
tags:
- commands
- process
- reference
---
# SPEC-2: Slash Commands Reference
This document defines the slash commands used in our specification-driven development process.
## /spec create [name]
**Purpose**: Create a new specification document
**Usage**: `/spec create notes-decomposition`
**Process**:
1. Create new spec document in `/specs` folder
2. Use SPEC-XXX numbering format (auto-increment)
3. Include standard spec template:
- Why (reasoning/problem)
- What (affected areas)
- How (high-level approach)
- How to Evaluate (testing/validation)
4. Tag appropriately for knowledge graph
5. Link to related specs/components
**Template**:
```markdown
# SPEC-XXX: [Title]
## Why
[Problem statement and reasoning]
## What
[What is affected or changed]
## How (High Level)
[Approach to implementation]
## How to Evaluate
[Testing/validation procedure]
## Notes
[Additional context as needed]
```
## /spec status
**Purpose**: Show current status of all specifications
**Usage**: `/spec status`
**Process**:
1. Search all specs in `/specs` folder
2. Display table showing:
- Spec number and title
- Status (draft, approved, implementing, complete)
- Assigned agent (if any)
- Last updated
- Dependencies
## /spec implement [name]
**Purpose**: Hand specification to appropriate agent for implementation
**Usage**: `/spec implement SPEC-002`
**Process**:
1. Read the specified spec
2. Analyze requirements to determine appropriate agent:
- Frontend components → vue-developer
- Architecture/system design → system-architect
- Backend/API → python-developer
3. Launch agent with spec context
4. Agent creates implementation plan
5. Update spec with implementation status
## /spec review [name]
**Purpose**: Review implementation against specification criteria
**Usage**: `/spec review SPEC-002`
**Process**:
1. Read original spec and "How to Evaluate" section
2. Examine current implementation
3. Test against success criteria
4. Document gaps or issues
5. Update spec with review results
6. Recommend next actions (complete, revise, iterate)
## Command Extensions
As the process evolves, we may add:
- `/spec link [spec1] [spec2]` - Create dependency links
- `/spec archive [name]` - Archive completed specs
- `/spec template [type]` - Create spec from template
- `/spec search [query]` - Search spec content
## References
- Claude Slash commands: https://docs.anthropic.com/en/docs/claude-code/slash-commands
## Creating a command
Commands are implemented as Claude slash commands:
Location in repo: .claude/commands/
In the following example, we create the /optimize command:
```bash
# Create a project command
mkdir -p .claude/commands
echo "Analyze this code for performance issues and suggest optimizations:" > .claude/commands/optimize.md
```
File diff suppressed because it is too large Load Diff
-108
View File
@@ -1,108 +0,0 @@
---
title: 'SPEC-3: Agent Definitions'
type: spec
permalink: specs/spec-3-agent-definitions
tags:
- agents
- roles
- process
---
# SPEC-3: Agent Definitions
This document defines the specialist agents used in our specification-driven development process.
## system-architect
**Role**: High-level system design and architectural decisions
**Responsibilities**:
- Create architectural specifications and ADRs
- Analyze system-wide impacts and trade-offs
- Design component interfaces and data flow
- Evaluate technical approaches and patterns
- Document architectural decisions and rationale
**Expertise Areas**:
- System architecture and design patterns
- Technology evaluation and selection
- Scalability and performance considerations
- Integration patterns and API design
- Technical debt and refactoring strategies
**Typical Specs**:
- System architecture overviews
- Component decomposition strategies
- Data flow and state management
- Integration and deployment patterns
## vue-developer
**Role**: Frontend component development and UI implementation
**Responsibilities**:
- Create Vue.js component specifications
- Implement responsive UI components
- Design component APIs and interfaces
- Optimize for performance and accessibility
- Document component usage and patterns
**Expertise Areas**:
- Vue.js 3 Composition API
- Nuxt 3 framework patterns
- shadcn-vue component library
- Responsive design and CSS
- TypeScript integration
- State management with Pinia
**Typical Specs**:
- Individual component specifications
- UI pattern libraries
- Responsive design approaches
- Component interaction flows
## python-developer
**Role**: Backend development and API implementation
**Responsibilities**:
- Create backend service specifications
- Implement APIs and data processing
- Design database schemas and queries
- Optimize performance and reliability
- Document service interfaces and behavior
**Expertise Areas**:
- FastAPI and Python web frameworks
- Database design and operations
- API design and documentation
- Authentication and security
- Performance optimization
- Testing and validation
**Typical Specs**:
- API endpoint specifications
- Database schema designs
- Service integration patterns
- Performance optimization strategies
## Agent Collaboration Patterns
### Handoff Protocol
1. Agent receives spec through `/spec implement [name]`
2. Agent reviews spec and creates implementation plan
3. Agent documents progress and decisions in spec
4. Agent hands off to another agent if cross-domain work needed
5. Final agent updates spec with completion status
### Communication Standards
- All agents update specs through basic-memory MCP tools
- Document decisions and trade-offs in spec notes
- Link related specs and components
- Preserve context for future reference
### Quality Standards
- Follow existing codebase patterns and conventions
- Write tests that validate spec requirements
- Document implementation choices
- Consider maintainability and extensibility
@@ -1,311 +0,0 @@
---
title: 'SPEC-4: Notes Web UI Component Architecture'
type: note
permalink: specs/spec-4-notes-web-ui-component-architecture
tags:
- frontend
- 'component-architecture'
- vue
- 'refactoring'
---
# SPEC-4: Notes Web UI Component Architecture
## Why
The current Notes.vue component is a monolithic component that handles multiple responsibilities, making it difficult to maintain, test, and understand. This leads to:
- Complex state management across multiple concerns
- Difficult to isolate and test individual features
- Hard to understand the full scope of functionality
- Circular refactoring cycles when making changes
- Poor separation of concerns between navigation, display, and interaction logic
We need to decompose this into focused, single-responsibility components that are easier to develop, test, and maintain while preserving the existing functionality users expect.
## What
This spec defines the component architecture for decomposing the Notes web UI into focused components with clear responsibilities and interactions.
**Affected Areas:**
- `/apps/web/components/notes/Notes.vue` - Will be decomposed into smaller components
- `/apps/web/components/notes/` - New component structure
- Existing composables: `useNotesNavigation`, `useNotesFiltering`, `useNotesLayout`
- Mobile responsive behavior and layout management
**Component Breakdown:**
```
┌───────────────────────┬─────────────────────────────────────┬────────────────────────────────────────────────────────────┐
│ [Project] │ [Project Name] A/Z | ^ │ [edit | view] [actions] │
├───────────────────────┼─────────────────────────────────────┤ │
│ All Notes ├─────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ Recent │ search... │ [note header] │
│ [Project base dir] ├─────────────────────────────────────┤ │
│ ├─────────────────────────────────────┤ │
│ Folder1 │ Title [modified] │ │
│ Folder2 │ ├────────────────────────────────────────────────────────────┤
│ - Nested │ snippet │ [note body] │
│ │ │ │
│ │ │ │
│ ├─────────────────────────────────────┤ │
│ ├─────────────────────────────────────┤ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ ├─────────────────────────────────────┤ │
│ ├─────────────────────────────────────┤ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ ├─────────────────────────────────────┤ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
└───────────────────────┴─────────────────────────────────────┴────────────────────────────────────────────────────────────┘
```
### ProjectSwitcher Component
- **Location**: Top-left dropdown
- **Responsibility**: Allow users to switch between Basic Memory projects
- **Behavior**: Selecting different project controls entire Notes page content
- **State**: When switching projects, reset to "All notes" view
### NotesNav Component
- **Views**: Three mutually exclusive options:
- **All notes**: Display all notes in project alphabetically
- **Recent**: Display all notes in project by updated time (desc)
- **Project**: Display notes in top-level directory of project
- **Interaction**: Only one view can be active at a time
- **Folder Integration**: All/Recent ignore folder selection; Project respects folder selection
### FolderTree Component
- **Display**: Nested list of all folders in project as tree view
- **Interaction**: Selecting folder filters notes in NotesList using directoryList API
- **Navigation Integration**: Selecting folder automatically switches NotesNav to "Project" view for clear UX
- **API Integration**: Uses directoryList API call via useDirectoryListQuery for folder-specific note fetching
- **State Coordination**: Folder selection coordinates with navigation state for intuitive user experience
### NotesList Component
- **Display**: Vertically scrolling cards showing note summaries
- **Information per card**:
- Note title
- Modified time (relative, e.g., "7 minutes ago")
- Short summary of note content (one line preview)
- **Behavior**: Updates based on NotesNav selection and FolderTree filtering
### NoteDetail Component
- **Display**: Full content of selected note
- **Sections**:
- Header: Displays frontmatter information
- Content: Note body content
- **Editing**: Current textarea implementation (rich editor in future spec)
- **Frontmatter**: Leave current implementation (enhancement in future spec)
## How (High Level)
### Component Architecture Approach
1. **Single Responsibility**: Each component handles one primary concern
2. **Clear Data Flow**: Props down, events up pattern for component communication
3. **Composable Integration**: Use existing composables for state management
4. **Progressive Decomposition**: Extract components incrementally to maintain functionality
### Implementation Strategy
1. **Extract ProjectSwitcher**: Move project switching logic to dedicated component
2. **Extract NotesNav**: Isolate navigation state and view selection logic
3. **Extract FolderTree**: Separate folder display and selection logic
4. **Extract NotesList**: Isolate note listing and card display logic
5. **Extract NoteDetail**: Separate note content display and editing
6. **Update Notes.vue**: Become orchestration component managing component interactions
### State Management Integration
- **useNotesNavigation**: Manages navigation state (All/Recent/Project)
- **useNotesFiltering**: Handles filtering logic based on navigation and folder selection
- **useNotesLayout**: Manages responsive layout and panel visibility
- **Component State**: Each component manages its own internal UI state
- **Shared State**: Project selection and note filtering coordinated through composables
### Responsive Behavior
Mobile:
- Hide sidebar. pop out panel when selected
- show note list on small screens (existing behavior)
- when note list item is clicked, display note detail on full page. Cancel or go back to return to list
Desktop:
- Full three-column layout with all components visible
- **Transitions**: Smooth navigation between mobile panels
## How to Evaluate
### Success Criteria
- **Functional Parity**: All existing Notes page functionality preserved
- **Component Isolation**: Each component can be developed/tested independently
- **Clear Responsibilities**: No overlapping concerns between components
- **State Clarity**: Clean data flow and state management patterns
- **Mobile Compatibility**: Responsive behavior maintains current UX
- **Performance**: No degradation in rendering or interaction performance
### Testing Procedure
1. **Functionality Validation**:
- Project switching works correctly
- All three navigation views (All/Recent/Project) function properly
- Folder selection affects note display appropriately
- Note selection and detail display works
- Mobile responsive behavior preserved
2. **Component Isolation Testing**:
- Each component can be imported and used independently
- Component props and events are clearly defined
- No tight coupling between components
3. **Integration Testing**:
- Components communicate correctly through props/events
- State management composables integrate properly
- User workflows function end-to-end
4. **Performance Validation**:
- Page load time unchanged or improved
- Interaction responsiveness maintained
- Memory usage stable or improved
### Implementation Validation
- **Code Review**: Clean component structure with single responsibilities
- **Type Safety**: Full TypeScript coverage with proper component prop types
- **Documentation**: Each component has clear interface documentation
- **Tests**: Unit tests for individual components and integration tests for workflows
## Observations
- [problem] Monolithic Notes.vue component creates maintenance and testing challenges #component-architecture
- [solution] Component decomposition improves separation of concerns and testability #refactoring
- [pattern] Progressive extraction maintains functionality while improving structure #incremental-improvement
- [interaction] NotesNav and FolderTree have conditional interaction based on selected view #state-management
- [constraint] Mobile responsive behavior must be preserved during decomposition #responsive-design
- [scope] Current editing and frontmatter capabilities remain unchanged #scope-limitation
- [validation] Functional parity is critical success criteria for this refactoring #validation-strategy
- [implementation] Folder selection now properly integrates with directoryList API for accurate filtering #api-integration
- [fix] FolderTree selection functionality completed - works across all navigation views #feature-complete
- [ux-improvement] FolderTree selection automatically switches NotesNav to Project view for clear user feedback #user-experience
## Relations
- depends_on [[SPEC-1: Specification-Driven Development Process]]
- implements [[Current Notes.vue functionality]]
- prepares_for [[Future rich editor spec]]
- prepares_for [[Future frontmatter editing spec]]
## Implementation Progress
### Components
1. **ProjectSwitcher** (`~/components/notes/ProjectSwitcher.vue`)
- ✅ Top-left dropdown for project switching
- ✅ Integrates with Pinia project store
- ✅ Handles project switching with proper state reset
- ✅ Responsive collapsed/expanded states
- ✅ Expanded menu shows available projects and a Manage Projects option that navigates to the /settings/projects page
- ✅ Simplified component following SortingToggle pattern - clean Props/Emits interface, uses ProjectItem type directly
2. **NotesNav** (`~/components/notes/NotesNav.vue`)
- ✅ Three mutually exclusive views: All/Recent/Project
- ✅ Dynamic project title based on selected project
- ✅ Clean props down, events up pattern
- ✅ Responsive collapsed/expanded states with tooltips
- ✅ The label for the Project selection should be the folder name for the project, not the project name
3. **FolderTree** (`~/components/notes/FolderTree.vue`)
- ✅ Nested folder tree view for filtering
- ✅ Uses `useFolderTree()` composable for data
- ✅ Emits `folder-selected` events properly
- ✅ Handles loading, error, and empty states
- ✅ Includes companion `FolderTreeNode.vue` component
- ✅ The current folder should be visibly selected in the tree
4. **NotesList** (`~/components/notes/NotesList.vue`)
- ✅ Vertically scrolling note summary cards
- ✅ Shows title, updated time (relative), and content preview
- ✅ Badge system for tags with variant logic
- ✅ v-model integration for selectedNote
- ✅ Smooth transitions and animations
- ✅ Contextual title: The current folder name should be displayed at the top of the Notes list, or "All Notes", or "Recent" if they are selected
- ✅ The title header should contain a toggle component to allow sorting with Lucide icon labels
- sorting options:
- name (asc/desc) - default
- file updated time (asc/desc)
- If "Recent" notes nav option is selected the default order should be updated in descending order (recent first)
5. **NoteDisplay** (`~/components/notes/NoteDisplay.vue` - equivalent to spec's NoteDetail)
- ✅ Full note content display
- ✅ Edit/view mode toggle
- ✅ Header with frontmatter information
- ✅ Markdown rendering capabilities
- ✅ Current textarea implementation preserved
### Architecture Requirements
1. **Component Isolation**: Each component can be developed/tested independently ✅
2. **Single Responsibility**: Each component handles one primary concern ✅
3. **Clear Data Flow**: Props down, events up pattern implemented ✅
4. **Composable Integration**: Uses existing composables for state management ✅
5. **Responsive Behavior**: Mobile/desktop layout preserved ✅
### State Management Integration
- **useNotesNavigation**: Manages navigation state (All/Recent/Project) ✅
- **useNotesFiltering**: Handles filtering logic based on navigation and folder selection ✅
- **useNotesLayout**: Manages responsive layout and panel visibility ✅
- **Component State**: Each component manages its own internal UI state ✅
### Interaction Logic
- Only one NotesNav view active at a time ✅
- All/Recent views ignore folder selection ✅
- Project view respects folder selection ✅
- Project switching resets to "All notes" view ✅
### TypeScript Coverage
- All components have full TypeScript coverage ✅
- Component props and events properly typed ✅
- No TypeScript errors in codebase ✅
### Success Criteria Validation
1. **Functional Parity**: All existing Notes page functionality preserved ✅
2. **Component Isolation**: Each component can be developed/tested independently ✅
3. **Clear Responsibilities**: No overlapping concerns between components ✅
4. **State Clarity**: Clean data flow and state management patterns ✅
5. **Mobile Compatibility**: Responsive behavior maintains current UX ✅
6. **Performance**: No degradation in rendering or interaction performance ✅
## Implementation Decisions
### Architectural Patterns
1. **Composition API + `<script setup>`**: All components use modern Vue 3 syntax
2. **Pinia Store Integration**: Project switching handled through reactive store
3. **Composable Pattern**: State management distributed across focused composables
4. **Event-Driven Communication**: Clean parent-child communication via events
5. **Responsive-First Design**: Mobile/desktop layouts handled natively
### Key Technical Choices
1. **Progressive Enhancement**: Mobile-first responsive design with desktop enhancements
2. **State Reset Logic**: Project switching properly resets navigation, search, and selection state
3. **Performance Optimizations**: Efficient re-rendering with proper key usage and transitions
4. **Accessibility**: Screen reader support, tooltips, keyboard navigation
5. **Type Safety**: Full TypeScript coverage with proper component prop definitions
### Quality Metrics
- **Code Maintainability**: High - each component is focused and independently testable
- **Performance**: Excellent - no performance degradation from decomposition
- **User Experience**: Preserved - all existing functionality and responsive behavior maintained
- **Developer Experience**: Improved - cleaner component structure for future development
-201
View File
@@ -1,201 +0,0 @@
---
title: 'SPEC-5: CLI Cloud Upload via WebDAV'
type: spec
permalink: specs/spec-5-cli-cloud-upload-via-webdav
tags:
- cli
- webdav
- upload
- migration
- poc
---
# SPEC-5: CLI Cloud Upload via WebDAV
## Why
Existing basic-memory users need a simple migration path to basic-memory-cloud. The web UI drag-and-drop approach outlined in GitHub issue #59, while user-friendly, introduces significant complexity for a proof-of-concept:
- Complex web UI components for file upload and progress tracking
- Browser file handling limitations and CORS complexity
- Proxy routing overhead for large file transfers
- Authentication integration across multiple services
A CLI-first approach solves these issues by:
- **Leveraging existing infrastructure**: Both cloud CLI and tenant API already exist with WorkOS JWT authentication
- **Familiar user experience**: Basic-memory users are CLI-comfortable and expect command-line tools
- **Direct connection efficiency**: Bypassing the MCP gateway/proxy for bulk file transfers
- **Rapid implementation**: Building on existing `CLIAuth` and FastAPI foundations
The fundamental problem is migration friction - users have local basic-memory projects but no path to cloud tenants. A simple CLI upload command removes this barrier immediately.
## What
This spec defines a CLI-based project upload system using WebDAV for direct tenant connections.
**Affected Areas:**
- `apps/cloud/src/basic_memory_cloud/cli/main.py` - Add upload command to existing CLI
- `apps/api/src/basic_memory_cloud_api/main.py` - Add WebDAV endpoints to tenant FastAPI
- Authentication flow - Reuse existing WorkOS JWT validation
- File transfer protocol - WebDAV for cross-platform compatibility
**Core Components:**
### CLI Upload Command
```bash
basic-memory-cloud upload <project-path> --tenant-url https://basic-memory-{tenant}.fly.dev
```
### WebDAV Server Endpoints
- `GET/PUT/DELETE /webdav/*` - Standard WebDAV operations on tenant file system
- Authentication via existing JWT validation
- File operations preserve timestamps and directory structure
### Authentication Flow
```
1. User runs `basic-memory-cloud login` (existing)
2. CLI stores WorkOS JWT token (existing)
3. Upload command reads JWT from storage
4. WebDAV requests include JWT in Authorization header
5. Tenant API validates JWT using existing middleware
```
## How (High Level)
### Implementation Strategy
**Phase 1: CLI Command**
- Add `upload` command to existing Typer app
- Reuse `CLIAuth` class for token management
- Implement WebDAV client using `webdavclient3` or similar
- Rich progress bars for transfer feedback
**Phase 2: WebDAV Server**
- Add WebDAV endpoints to existing tenant FastAPI app
- Leverage existing `get_current_user` dependency for authentication
- Map WebDAV operations to tenant file system
- Preserve file modification times using `os.utime()`
**Phase 3: Integration**
- Direct connection bypasses MCP gateway and proxy
- Simple conflict resolution: overwrite existing files
- Error handling: fail fast with clear error messages
### Technical Architecture
```
basic-memory-cloud CLI → WorkOS JWT → Direct WebDAV → Tenant FastAPI
Tenant File System
```
**Key Libraries:**
- CLI: `webdavclient3` for WebDAV client operations
- API: `wsgidav` or FastAPI-compatible WebDAV server
- Progress: `rich` library (already imported in CLI)
- Auth: Existing WorkOS JWT infrastructure
### WebDAV Protocol Choice
WebDAV provides:
- **Cross-platform clients**: Native support in most operating systems
- **Standardized protocol**: Well-defined for file operations
- **HTTP-based**: Works with existing FastAPI and JWT auth
- **Library support**: Good Python libraries for both client and server
### POC Constraints
**Simplifications for rapid implementation:**
- **Known tenant URLs**: Assume `https://basic-memory-{tenant}.fly.dev` format
- **Upload only**: No download or bidirectional sync
- **Overwrite conflicts**: No merge or conflict resolution prompting
- **No fallbacks**: Fail fast if WebDAV connection issues occur
- **Direct connection only**: No proxy fallback mechanism
## How to Evaluate
### Success Criteria
**Functional Requirements:**
- [ ] Transfer complete basic-memory project (100+ files) in < 30 seconds
- [ ] Preserve directory structure exactly as in source project
- [ ] Preserve file modification timestamps for proper sync behavior
- [ ] Rich progress bars show real-time transfer status (files/MB transferred)
- [ ] WorkOS JWT authentication validates correctly on WebDAV endpoints
- [ ] Direct tenant connection bypasses MCP gateway successfully
**Quality Requirements:**
- [ ] Clear error messages for authentication failures
- [ ] Graceful handling of network interruptions
- [ ] CLI follows existing command patterns and help text standards
- [ ] WebDAV endpoints integrate cleanly with existing FastAPI app
**Performance Requirements:**
- [ ] File transfer speed > 1MB/s on typical connections
- [ ] Memory usage remains reasonable for large projects
- [ ] No timeout issues with 500+ file projects
### Testing Procedure
**Unit Testing:**
1. CLI command parsing and argument validation
2. WebDAV client connection and authentication
3. File timestamp preservation during transfer
4. JWT token validation on WebDAV endpoints
**Integration Testing:**
1. End-to-end upload of test project
2. Direct tenant connection without proxy
3. File integrity verification after upload
4. Progress tracking accuracy during transfer
**User Experience Testing:**
1. Upload existing basic-memory project from local installation
2. Verify uploaded files appear correctly in cloud tenant
3. Confirm basic-memory database rebuilds properly with uploaded files
4. Test CLI help text and error message clarity
### Validation Commands
**Setup:**
```bash
# Login to WorkOS
basic-memory-cloud login
# Upload project
basic-memory-cloud upload ~/my-notes --tenant-url https://basic-memory-test.fly.dev
```
**Verification:**
```bash
# Check tenant health and file count via API
curl -H "Authorization: Bearer $JWT" https://basic-memory-test.fly.dev/health
curl -H "Authorization: Bearer $JWT" https://basic-memory-test.fly.dev/notes/search
```
### Performance Benchmarks
**Target metrics for 100MB basic-memory project:**
- Transfer time: < 30 seconds
- Memory usage: < 100MB during transfer
- Progress updates: Every 1MB or 10 files
- Authentication time: < 2 seconds
## Observations
- [implementation-speed] CLI approach significantly faster than web UI for POC development #rapid-prototyping
- [user-experience] Basic-memory users already comfortable with CLI tools #user-familiarity
- [architecture-benefit] Direct connection eliminates proxy complexity and latency #performance
- [auth-reuse] Existing WorkOS JWT infrastructure handles authentication cleanly #code-reuse
- [webdav-choice] WebDAV protocol provides cross-platform compatibility and standard libraries #protocol-selection
- [poc-scope] Simple conflict handling and error recovery sufficient for proof-of-concept #scope-management
- [migration-value] Removes primary barrier for local users migrating to cloud platform #business-value
## Relations
- depends_on [[SPEC-1: Specification-Driven Development Process]]
- enables [[GitHub Issue #59: Web UI Upload Feature]]
- uses [[WorkOS Authentication Integration]]
- builds_on [[Existing Cloud CLI Infrastructure]]
- builds_on [[Existing Tenant API Architecture]]
@@ -1,497 +0,0 @@
---
title: 'SPEC-6: Explicit Project Parameter Architecture'
type: spec
permalink: specs/spec-6-explicit-project-parameter-architecture
tags:
- architecture
- mcp
- project-management
- stateless
---
# SPEC-6: Explicit Project Parameter Architecture
## Why
The current session-based project management system has critical reliability issues:
1. **Session State Fragility**: Claude iOS mobile client fails to maintain consistent session IDs across MCP tool calls, causing project switching to silently fail (Issue #74)
2. **Scaling Limitations**: Redis-backed session state creates single-point-of-failure and prevents horizontal scaling
3. **Client Compatibility**: Session tracking works inconsistently across different MCP clients (web, mobile, API)
4. **Hidden Complexity**: Users cannot see or understand "current project" state, leading to confusion when operations execute in wrong projects
5. **Silent Failures**: Operations appear successful but execute in unintended projects, risking data integrity
Evidence from production logs shows each MCP tool call from mobile client receives different session IDs:
```
create_memory_project: session_id=12cdfc24913b48f8b680ed4b2bfdb7ba
switch_project: session_id=050a69275d98498cbdd227cdb74d9740
list_directory: session_id=85f3483014af4136a5d435c76ded212f
```
Related Github issue: https://github.com/basicmachines-co/basic-memory-cloud/issues/75
## Status
**Current Status**: **ALL PHASES COMPLETE****PRODUCTION DEPLOYED**
**Target**: Fix Claude iOS session ID consistency issues ✅ **ACHIEVED**
**Draft PR**: https://github.com/basicmachines-co/basic-memory/pull/298 ✅ **MERGED & DEPLOYED**
### 🎉 **COMPLETE SUCCESS - PRODUCTION READY**
**ALL PHASES OF SPEC-6 IMPLEMENTATION COMPLETE!** The stateless architecture has been successfully implemented across both Basic Memory core and Basic Memory Cloud, representing a **fundamental architectural improvement** that completely solves the Claude iOS compatibility issue while providing superior scalability and reliability.
#### Implementation Summary:
- **16 files modified** with 582 additions and 550 deletions
- **All 17 MCP tools** converted to stateless architecture
- **147 tests updated** across 5 test files (100% passing)
- **Complete session state removal** from core MCP tools
- **Enhanced error handling** and security validations preserved
### Progress Summary
**Complete Stateless Architecture Implementation (All 17 tools)** - **PRODUCTION DEPLOYED**
- Stateless `get_active_project()` function implemented and deployed ✅
- All session state dependencies removed across entire MCP server ✅
- All MCP tools require explicit `project` parameter as first argument ✅
- **Cloud Service**: Redis removed, stateless HTTP enabled ✅
- **Production Validation**: Comprehensive testing completed with 100% success ✅
**Content Management Tools Complete (6/6 tools)**
- `write_note`, `read_note`, `delete_note`, `edit_note`
- `view_note`, `read_content`
**Knowledge Graph Navigation Tools Complete (3/3 tools)**
- `build_context`, `recent_activity`, `list_directory`
**Search & Discovery Tools Complete (1/1 tools)**
- `search_notes`
**Visualization Tools Complete (1/1 tools)**
- `canvas`
**Project Management Cleanup Complete**
- Removed `switch_project` and `get_current_project` tools ✅
- Updated `set_default_project` to remove activate parameter ✅
**Comprehensive Testing Complete (157 tests)**
- All test suites updated to use stateless architecture (147 existing tests)
- Single project constraint mode integration tests (10 new tests)
- 100% test pass rate across all tool test files
- Security validations preserved and working
- Error handling comprehensive and user-friendly
**Documentation & Examples Complete**
- All tool docstrings updated with stateless examples
- Project parameter usage clearly documented
- Error handling and security behavior documented
**Enhanced Discovery Mode Complete**
- `recent_activity` tool supports dual-mode operation (discovery vs project-specific)
- ProjectActivitySummary schema provides cross-project insights
- Recent activity prompt updated to support both modes
- Comprehensive project distribution statistics and most active project tracking
**Single Project Constraint Mode Complete**
- `--project` CLI parameter for MCP server constraint
- Environment variable control (`BASIC_MEMORY_MCP_PROJECT`)
- Automatic project override in `get_active_project()` function
- Project management tools disabled in constrained mode with helpful CLI guidance
- Comprehensive integration test suite (10 tests covering all constraint scenarios)
## What
Transform Basic Memory from stateful session-based to stateless explicit project parameter architecture:
### Core Changes
1. **Mandatory Project Parameter**: All MCP tools require explicit `project` parameter
2. **Remove Session State**: Eliminate Redis, session middleware, and `switch_project` tool
3. **Stateless HTTP**: Enable `stateless_http=True` for horizontal scaling
4. **Enhanced Context Discovery**: Improve `recent_activity` to show project distribution
5. **Clear Response Format**: All tool responses display target project information
Implementation Approach
- Each tool will directly accept the project parameter
- Remove all calls to context-based project retrieval
- Validate project exists before operations
- Clear error messages when project not found
- Backward compatibility: Initially keep optional parameter, then make required
### Affected MCP Tools
**Content Management** (require project parameter):
- `write_note(project, title, content, folder)`
- `read_note(project, identifier)`
- `edit_note(project, identifier, operation, content)`
- `delete_note(project, identifier)`
- `view_note(project, identifier)`
- `read_content(project, path)`
**Knowledge Graph Navigation** (require project parameter):
- `build_context(project, url, timeframe, depth, max_related)`
- `list_directory(project, dir_name, depth, file_name_glob)`
- `search_notes(project, query, search_type, types, entity_types)`
**Search & Discovery** (use project parameter for specific project or none for discovery):
- `recent_activity(project, timeframe, depth, max_related)`
**Visualization** (require project parameter):
- `canvas(project, nodes, edges, title, folder)`
**Project Management** (unchanged - already stateless):
- `list_memory_projects()`
- `create_memory_project(project_name, project_path, set_default)`
- `delete_project(project_name)`
- `get_current_project()` - Remove this tool
- `switch_project(project_name)` - Remove this tool
- `set_default_project(project_name, activate)` - Remove activate parameter
## How (High Level)
### Phase 1: Basic Memory Core (basic-memory repository)
#### MCP Tool Updates
Phase 1: Core Changes
1. Update project_context.py
- [x] Make project parameter mandatory for get_active_project()
- [x] Remove session state handling
2. Update Content Management Tools (6 tools)
- [x] write_note: Make project parameter required, not optional
- [x] read_note: Make project parameter required
- [x] edit_note: Add required project parameter
- [x] delete_note: Add required project parameter
- [x] view_note: Add required project parameter
- [x] read_content: Add required project parameter
3. Update Knowledge Graph Navigation Tools (3 tools)
- [x] build_context: Add required project parameter
- [x] recent_activity: Make project parameter required
- [x] list_directory: Add required project parameter
4. Update Search & Visualization Tools (2 tools)
- [x] search_notes: Add required project parameter
- [x] canvas: Add required project parameter
5. Update Project Management Tools
- [x] Remove switch_project tool completely
- [x] Remove get_current_project tool completely
- [x] Update set_default_project to remove activate parameter
- [x] Keep list_memory_projects, create_memory_project, delete_project unchanged
6. Enhance recent_activity Response
- [x] Add project distribution info showing activity across all projects
- [x] Include project usage stats in response
- [x] Implement ProjectActivitySummary for discovery mode
- [x] Add dual-mode functionality (discovery vs project-specific)
7. Update Tool Documentation
- [x] Update write_note docstring with stateless architecture examples
- [x] Update read_note docstring with project parameter examples
- [x] Update delete_note docstring with comprehensive usage guidance
- [x] Update all remaining tool docstrings with project parameter examples
8. Update Tool Responses
- [x] Add clear project indicator to all tool responses across all tools
- [x] Format: "project: {project_name}" in response metadata
- [x] Add project metadata footer for LLM awareness
- [x] Update all tool responses to include project indicators
9. Comprehensive Testing
- [x] Update all write_note tests to use stateless architecture (34 tests passing)
- [x] Update all edit_note tests to use stateless architecture (17 tests passing)
- [x] Update all view_note tests to use stateless architecture (12 tests passing)
- [x] Update all search_notes tests to use stateless architecture (16 tests passing)
- [x] Update all move_note tests to use stateless architecture (31 tests passing)
- [x] Update all delete_note tests to use stateless architecture
- [x] Verify direct function call compatibility (bypassing MCP layer)
- [x] Test security validation with project parameters
- [x] Validate error handling for non-existent projects
- [x] **Total: 157 tests updated and passing (100% success rate)**
- [x] **147 existing tests** updated for stateless architecture
- [x] **10 new tests** for single project constraint mode
### Phase 1.5: Default Project Mode Enhancement
#### Problem
While the stateless architecture solves reliability issues, it introduces UX friction for single-project users (estimated 80% of usage) who must specify the project parameter in every tool call.
#### Solution: Default Project Mode
Add optional `default_project_mode` configuration that allows single-project users to have the simplicity of implicit project selection while maintaining the reliability of stateless architecture.
#### Configuration
```json
{
"default_project": "main",
"default_project_mode": true // NEW: Auto-use default_project when not specified
}
```
#### Implementation Details
1. **Config Enhancement** (`src/basic_memory/config.py`)
- Add `default_project_mode: bool = Field(default=False)`
- Preserves backward compatibility (defaults to false)
2. **Project Resolution Logic** (`src/basic_memory/mcp/project_context.py`)
Three-tier resolution hierarchy:
- Priority 1: CLI `--project` constraint (BASIC_MEMORY_MCP_PROJECT env var)
- Priority 2: Explicit project parameter in tool call
- Priority 3: `default_project` if `default_project_mode=true` and no project specified
3. **Assistant Guide Updates** (`src/basic_memory/mcp/resources/ai_assistant_guide.md`)
- Detect `default_project_mode` at runtime
- Provide mode-specific instructions to LLMs
- In default mode: "All operations use project 'main' automatically"
- In regular mode: Current project discovery guidance
4. **Tool Parameter Handling** (all MCP tools)
- Make project parameter Optional[str] = None
- Add resolution logic: `project = project or get_default_project()`
- Maintain explicit project override capability
#### Usage Modes Summary
- **Regular Mode**: Multi-project users, assistant tracks project per conversation
- **Default Project Mode**: Single-project users, automatic default project
- **Constrained Mode**: CLI --project flag, locked to specific project
#### Testing Requirements
- Integration test for default_project_mode=true with missing parameters
- Test explicit project override in default_project_mode
- Test mode=false requires explicit parameters
- Test CLI constraint overrides default_project_mode
Phase 2: Testing & Validation
8. Update Tests
- [x] Modify all MCP tool tests to pass required project parameter
- [x] Remove tests for deleted tools (switch_project, get_current_project)
- [x] Add tests for project parameter validation
- [x] **Complete: All 147 tests across 5 test files updated and passing**
#### Enhanced recent_activity Response
```json
{
"recent_notes": [...],
"project_activity": {
"research-project": {
"operations": 5,
"last_used": "30 minutes ago",
"recent_folders": ["experiments", "findings"]
},
"work-notes": {
"operations": 2,
"last_used": "2 hours ago",
"recent_folders": ["meetings", "planning"]
}
},
"total_projects": 3
}
```
#### Response Format Updates
```
✓ Note created successfully
Project: research-project
File: experiments/Neural Network Results.md
Permalink: research-project/neural-network-results
```
### Phase 2: Cloud Service Simplification (basic-memory-cloud repository) ✅ **COMPLETE**
#### ✅ Remove Session Infrastructure **COMPLETE**
1. ✅ Delete `apps/mcp/src/basic_memory_cloud_mcp/middleware/session_state.py`
2. ✅ Delete `apps/mcp/src/basic_memory_cloud_mcp/middleware/session_logging.py`
3. ✅ Update `apps/mcp/src/basic_memory_cloud_mcp/main.py`:
```python
# Remove session middleware
# server.add_middleware(SessionStateMiddleware)
# Enable stateless HTTP
mcp = FastMCP(name="basic-memory-mcp", stateless_http=True)
```
#### ✅ Deployment Simplification **COMPLETE**
1. ✅ Remove Redis from `fly.toml`
2. ✅ Remove Redis environment variables
3. ✅ Update health checks to not depend on Redis
4. ✅ Production deployment verified working with stateless architecture
### Phase 3: Conversational Project Management ✅ **COMPLETE**
#### ✅ Claude Behavior Pattern **VERIFIED WORKING**
1. ✅ **Project Discovery**:
```
Claude: Let me check your recent activity...
[calls recent_activity() - no project needed for discovery]
I see you've been working in:
- research-project (5 operations, 30 min ago)
- work-notes (2 operations, 2 hours ago)
Which project should I use for this operation?
```
2. ✅ **Context Maintenance**:
```
User: Use research-project
Claude: Working in research-project.
[All subsequent operations use project="research-project"]
```
3. ✅ **Explicit Project Switching**:
```
User: Check work-notes for that meeting summary
Claude: Let me search work-notes for the meeting summary.
[Uses project="work-notes" for specific operation]
```
**Validation**: Comprehensive testing confirmed all conversational patterns work naturally with the stateless architecture.
## How to Evaluate
### Success Criteria
#### 1. Functional Completeness
- [x] All MCP tools accept required `project` parameter
- [x] All MCP tools validate project exists before execution
- [x] `switch_project` and `get_current_project` tools removed
- [x] All responses display target project clearly
- [x] No Redis dependencies in deployment (Phase 2: Cloud Service) ✅ **COMPLETE**
- [x] `recent_activity` shows project distribution with ProjectActivitySummary
#### 2. Cross-Client Compatibility Testing ✅ **COMPLETE**
Test identical operations across all clients:
- [x] **Claude Desktop**: All operations work with explicit projects ✅
- [x] **Claude Code**: All operations work with explicit projects ✅
- [x] **Claude Mobile iOS**: All operations work with explicit projects ✅ **CRITICAL SUCCESS**
- [x] **API clients**: All operations work with explicit projects ✅
- [x] **CLI tools**: All operations work with explicit projects ✅
**Critical Achievement**: Claude iOS mobile client session tracking issues completely eliminated through stateless architecture.
#### 3. Session Independence Verification ✅ **COMPLETE**
- [x] Operations work identically with/without session tracking ✅
- [x] No behavioral differences between clients ✅
- [x] Mobile client session ID changes do not affect operations ✅
- [x] Redis can be completely removed without functional impact ✅
**Production Validation**: Redis removed from production deployment with zero functional impact.
#### 4. Performance & Scaling ✅ **COMPLETE**
- [x] `stateless_http=True` enabled successfully ✅
- [x] No Redis memory usage ✅
- [x] Horizontal scaling possible (multiple MCP instances) ✅
- [x] Response times unchanged or improved ✅
#### 5. User Experience Testing
**Project Discovery Flow**:
- [x] `recent_activity()` provides useful project context
- [x] Claude can intelligently suggest projects based on activity
- [x] Project switching is explicit and clear in conversation
**Error Handling**:
- [x] Clear error messages for non-existent projects
- [x] Helpful suggestions when project parameter missing
- [x] No silent failures or wrong-project operations
**Response Clarity**:
- [x] Every operation clearly shows target project
- [x] Users always know which project is being operated on
- [x] No confusion about "current project" state
#### 6. Migration Safety ✅ **COMPLETE**
- [x] Backward compatibility period with optional project parameter ✅
- [x] Clear migration documentation for existing users ✅
- [x] Data integrity maintained during transition ✅
- [x] No data loss during migration ✅
**Production Migration**: Successfully deployed to production with zero data loss and maintained system integrity.
### Test Scenarios
#### Core Functionality Test
```bash
# Test all tools work with explicit project
write_note(project="test-proj", title="Test", content="Content", folder="docs")
read_note(project="test-proj", identifier="Test")
edit_note(project="test-proj", identifier="Test", operation="append", content="More")
search_notes(project="test-proj", query="Content")
list_directory(project="test-proj", dir_name="docs")
delete_note(project="test-proj", identifier="Test")
```
#### Cross-Client Consistency Test
Run identical test sequence on:
1. Claude Desktop
2. Claude Code
3. Claude Mobile iOS
4. API client
5. CLI tools
Verify all clients:
- Accept explicit project parameters
- Return identical responses
- Show same project information
- Have no session dependencies
#### Session Independence Test
1. Monitor session IDs during operations
2. Verify operations work with changing session IDs
3. Confirm Redis removal doesn't affect functionality
4. Test with multiple concurrent clients
### Acceptance Criteria
**Must Have**:
- All MCP tools require and use explicit project parameter
- No session state dependencies remain
- Universal client compatibility achieved
- Clear project information in all responses
**Should Have**:
- Enhanced `recent_activity` with project distribution
- Smooth migration path for existing users
- Improved performance with stateless architecture
**Could Have**:
- Smart project suggestions based on content/context
- Project shortcuts for common operations
- Advanced project analytics in responses
## Notes
### Breaking Changes
This is a **breaking change** that requires:
- All MCP clients to pass project parameter
- Migration of existing workflows
- Update of all documentation and examples
### Implementation Order
1. **basic-memory core** - Update MCP tools to accept project parameter (optional initially)
2. **Testing** - Verify all clients work with explicit projects
3. **Cloud service** - Remove session infrastructure
4. **Migration** - Make project parameter mandatory
5. **Cleanup** - Remove deprecated tools and middleware
### Related Issues
- Fixes #74 (Claude iOS session state bug)
- Implements #75 (Mandatory project parameter architecture)
- Enables future horizontal scaling
- Simplifies multi-tenant architecture
### Dependencies
- Requires coordination between basic-memory and basic-memory-cloud repositories
- Needs client-side updates for smooth transition
- Documentation updates across all materials
@@ -1,324 +0,0 @@
---
title: 'SPEC-7: POC to spike Tigris/Turso for local access to cloud data'
type: spec
permalink: specs/spec-7-poc-tigris-turso-local-access-cloud-data
tags:
- poc
- tigris
- turso
- cloud-storage
- architecture
- proof-of-concept
---
# SPEC-7: POC to spike Tigris/Turso for local access to cloud data
> **Status Update**: ✅ **Phase 1 COMPLETE** (September 20, 2025)
> TigrisFS mounting validated successfully in containerized environments. Container startup, filesystem mounting, and Fly.io integration all working correctly. Ready for Phase 2 (Turso database integration).
> See: [`SPEC-7-PHASE-1-RESULTS.md`](./SPEC-7-PHASE-1-RESULTS.md)
## Why
Current basic-memory-cloud architecture uses Fly volumes for tenant file storage, which creates several limitations:
We could enable a revolutionary user experience: **local editing (or at least view access) of cloud-stored files** while maintaining Basic Memory's existing filesystem assumptions.
1. **Storage Scalability**: Fly volumes require pre-provisioning and don't auto-scale with usage
2. **Single Instance**: Volumes can only be mounted to one fly machine instance
3. **Cost Model**: Volume pricing vs object storage pricing may be less favorable at scale
4. **Local Development**: No way for users to mount their cloud tenant files locally for real-time editing
5. **Multi-Region**: Volumes are region-locked, limiting global deployment flexibility
6. **Backup/Disaster Recovery**: Object storage provides better durability and replication options
Basic Memory requires POSIX filesystem semantics but could benefit from object storage durability and accessibility. By combining:
- **Tigris object storage and TigrisFS** for file persistence in bucket stoage via a POSIX filesystem on the tenant instance
- **Turso/libSQL** for SQLite indexing (replacing local .db files). Sqlite on NFS volumes is disouraged.
## What
This specification defines a proof-of-concept to validate the technical feasibility of the Tigris/Turso architecture for basic-memory-cloud tenants.
**Affected Areas:**
- **Storage Architecture**: Replace Fly volumes with Tigris object storage
- **Database Architecture**: Replace local SQLite with Turso remote database
- **Container Setup**: Add TigrisFS mounting in tenant containers
- **Local Development**: Enable local mounting of cloud tenant data
- **Basic Memory Core**: Validate unchanged operation over mounted filesystems
**Key Components:**
- **Tigris Storage**: Globally caching S3-compatible object storage via Fly.io integration
- **TigrisFS**: Purpose-built FUSE filesystem with intelligent caching
- **Turso Database**: Hosted libSQL for SQLite replacement
- **Single-Tenant Model**: One bucket + one database per tenant (simplified isolation)
## Architectural Overview & Key Insights
### TigrisFS
Unlike standard S3 mounting approaches, **TigrisFS is a purpose-built FUSE filesystem** optimized for object storage with several critical advantages:
1. **Eliminates Fly Volume Limitations**
- No single-machine attachment constraints
- No pre-provisioning of storage capacity
- Enables horizontal scaling and zero-downtime deployments
- Automatic global CDN caching at Fly.io edge locations
2. **Intelligent Caching Architecture**
- 1-4GB+ configurable memory cache for read/write operations
- Write-back caching for improved performance
- Metadata cache to reduce API calls
- "Close to Redis speed" for small object retrieval
3. **Cost-Effective Model**
- Pay only for storage used and transferred
- No wasted capacity from over-provisioning
- Automatic global replication included
- S3 durability with CDN performance
### API-Driven Architecture Eliminates File Watching Concerns
**Critical Insight**: All file access (reads/writes) in basic-memory-cloud go through the API layer:
- **MCP Tools → API**: All Basic Memory operations use FastAPI endpoints
- **Web App → API**: Frontend uses API for all data modifications
- **File watching is NOT required** for cloud operations, unlike local BM which uses the WatchService to monitor file changes.
This means:
- **Cloud Operations**: Manual sync after API writes is sufficient
- **Local Development**: File watching only matters for local editing experience
- **Performance Risk**: Dramatically reduced since we're not dependent on inotify over network filesystems
### Realistic Local Access Expectations
**Baseline Functionality (Guaranteed):**
- Read-only mounting for browsing cloud files
- Easy download/upload of entire projects
- File copying via standard filesystem operations
**Stretch Goal (Test in POC):**
- Live editing with eventual consistency (1-5 second delays acceptable)
- Automatic sync for local changes
- Not required for core functionality - pure upside if it works
### Production Deployment Advantages
1. **Multi-Region Deployment**: Tigris handles global replication automatically
2. **Zero-Downtime Updates**: No volume detach/attach during deployments
3. **Tenant Migrations**: Simply update credentials, no data movement
4. **Disaster Recovery**: Built into S3 durability model (99.999999999% durability)
5. **Auto-Scaling**: Storage scales with usage, no capacity planning needed
## How (High Level)
### POC Approach: Server-First Validation
**Rationale**: Start with server-side TigrisFS mounting because:
- Local access is meaningless if cloud containers can't mount TigrisFS reliably
- Container startup and API performance are critical path blockers
- TigrisFS compatibility with Basic Memory operations must be proven first
- Each phase gates the next - no point testing local access if server-side fails
### Phase 1: Server-Side TigrisFS Validation (Critical Foundation) ✅ COMPLETE
- [x] Set up Tigris bucket with test data via Fly.io integration
- [x] Create container image with TigrisFS support and dependencies
- [x] Test TigrisFS mounting in containerized environment
- [x] Run Basic Memory API operations over mounted TigrisFS
- [x] Validate all filesystem operations work correctly
- [x] Measure container startup time and resource usage
**Production Validation Results**: Container successfully deployed and operated for 42+ minutes serving real MCP requests with repository queries, knowledge graph navigation, and full Basic Memory API functionality over TigrisFS-mounted storage.
### Phase 2: Database Migration to Turso
- [ ] Set up Turso account and test database
- [ ] Modify Basic Memory to accept external DATABASE_URL
- [ ] Test all MCP tools with remote SQLite via Turso
- [ ] Validate performance and functionality parity
- [ ] Test API write → manual sync workflow in container
### Phase 3: Production Container Integration
- [ ] Implement tenant-specific credential management for buckets
- [x] Test container startup with automatic TigrisFS mounting
- [ ] Validate isolation between tenant containers
- [ ] Test API operations under realistic load
- [ ] Measure performance vs current Fly volume setup
### Phase 4: Local Access Validation (Bonus Feature)
- [ ] Test local TigrisFS mounting of tenant data
- [ ] Validate read-only access for browsing/downloading
- [ ] Test file copying and upload workflows
- [ ] Measure latency impact on user experience
- [ ] Test live editing if file watching works (stretch goal)
### Architecture Overview
```
Local Development:
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Local TigrisFS │───▶│ Tigris Bucket │◀───│ Tenant Container│
│ Mount │ │ (Global CDN) │ │ TigrisFS mount │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ Basic Memory │ │ Basic Memory │
│ (local files) │ │ API + mounted │
└─────────────────┘ └─────────────────┘
│ │
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ Turso Database │◀───────────────────────────│ Turso Database │
│ (shared index) │ │ (shared index) │
└─────────────────┘ └─────────────────┘
Flow: API writes → Manual sync → Index update
Local: File watching (if available) → Auto sync
```
## How to Evaluate
### Success Criteria
- [x] **Filesystem Compatibility**: Basic Memory operates without modification over TigrisFS-mounted storage
- [x] **Performance Acceptable**: API-driven operations perform within acceptable latency (target: <500ms for typical operations)
- [ ] **Database Functionality**: All Basic Memory features work with Turso remote SQLite
- [x] **Container Reliability**: Tenant containers start successfully with automatic TigrisFS mounting
- [ ] **Local Access Baseline**: Users can mount cloud files locally for read-only browsing and file copying
- [x] **Data Isolation**: Tenant data remains properly isolated using bucket/database separation
- [ ] **Local Access Stretch**: Live editing with eventual sync (1-5 second delays acceptable)
### Testing Procedure
#### Phase 1: Server-Side Foundation Testing
1. **Container TigrisFS Test**:
```dockerfile
# Test container with TigrisFS mounting
FROM python:3.12
RUN apt-get update && apt-get install -y tigrisfs
# Test startup script
#!/bin/bash
tigrisfs --memory-limit 2048 $TIGRIS_BUCKET /app/data --daemon
cd /app/data && basic-memory sync
basic-memory-api --data-dir /app/data
```
2. **API Operations Validation**:
```bash
# Test all MCP operations over TigrisFS
curl -X POST /api/write_note -d '{"title":"test","content":"content"}'
curl -X GET /api/read_note/test
curl -X GET /api/search_notes?q=content
# Measure: response times, error rates, data consistency
```
#### Phase 2: Database Integration Testing
3. **Turso Integration Test**:
```bash
# Configure Turso connection in container
export DATABASE_URL="libsql://test-db.turso.io?authToken=..."
# Test all MCP tools with remote database
basic-memory tools # Test each tool functionality
# Test API write → manual sync workflow
```
#### Phase 3: Production Readiness Testing
4. **Performance Benchmarking**:
- Container startup time with TigrisFS mounting
- API operation response times (target: <500ms for typical operations)
- Search query performance with Turso (target: comparable to local SQLite)
- TigrisFS cache hit rates and memory usage
- Concurrent tenant isolation
#### Phase 4: Local Access Testing (If Phase 1-3 Succeed)
5. **Local Access Validation**:
```bash
# Test read-only access
tigrisfs tenant-bucket ~/local-tenant
ls -la ~/local-tenant # Browse files
cp ~/local-tenant/notes/* ~/backup/ # Copy files
# Test file watching (stretch goal)
echo "test" > ~/local-tenant/test.md
# Check if changes sync to cloud
```
### Go/No-Go Criteria by Phase
- **Phase 1**: Container must start successfully and serve API requests over TigrisFS
- **Phase 2**: All MCP tools must work with Turso with <2x latency increase
- **Phase 3**: Performance must be within 50% of current Fly volume setup
- **Phase 4**: Local mounting must work reliably for read-only access
### Risk Assessment
**Moderate Risk Items (Mitigated by API-First Architecture)**:
- [ ] TigrisFS performance for local access may have higher latency than local filesystem
- [ ] File watching (`inotify`) over FUSE may be unreliable for local development
- [ ] Network interruptions could cause filesystem errors during local editing
- [ ] Write-back caching could cause data loss if container crashes during flush
**Low Risk Items (API-First Eliminates)**:
- [ ] ~~Real-time file watching~~ - Not required for cloud operations
- [ ] ~~Concurrent write consistency~~ - Single-tenant model with API coordination
- [ ] ~~S3 rate limits~~ - TigrisFS intelligent caching handles this
**Mitigation Strategies**:
- **Performance**: Comprehensive benchmarking with realistic workloads
- **Reliability**: Graceful degradation to read-only local access if live editing fails
- **Data Safety**: Regular sync intervals and write-through mode for critical operations
- **Fallback**: Keep Fly volumes as backup deployment option
### Metrics to Track
- **API Latency**: Response times for MCP tools and web operations
- **Cache Effectiveness**: TigrisFS cache hit rates and memory usage
- **Local Access Performance**: File browsing and copying speeds
- **Reliability**: Success rate of mount operations and data consistency
- **Cost**: Storage usage, API calls, and network transfer costs vs current volumes
## Notes
### Key Architectural Decisions
- **Single tenant per bucket/database**: Simplifies isolation and credential management
- **Maintain POSIX compatibility**: Preserve Basic Memory's existing filesystem assumptions
- **TigrisFS over rclone**: Purpose-built for object storage with intelligent caching
- **Turso for SQLite**: Leverages specialized remote SQLite expertise
- **API-first approach**: Eliminates file watching dependency for cloud operations
### Alternative Approaches Considered
- **S3-native storage backend**: Would require Basic Memory architecture changes
- **Hybrid approach**: Local files + cloud sync (adds complexity)
- **Standard rclone mounting**: Less optimized than TigrisFS for object storage workloads
- **Keep Fly volumes**: Maintains current limitations but proven reliability
### Integration Points
- [ ] Fly.io Tigris integration for bucket provisioning
- [ ] Turso account setup and database provisioning
- [ ] Container image modifications for TigrisFS support
- [ ] Credential management for tenant isolation
- [ ] API modification for manual sync triggers
- [ ] Local client setup documentation for TigrisFS mounting
## Observations
- [architecture] Tigris/Turso split cleanly separates file storage from indexing concerns #storage-separation
- [breakthrough] API-first architecture eliminates file watching dependency for cloud operations #api-first-advantage
- [user-experience] Local mounting of cloud files could be revolutionary for knowledge management #local-cloud-hybrid
- [compatibility] Maintaining POSIX filesystem assumptions preserves Basic Memory's local/cloud compatibility #architecture-preservation
- [simplification] Single tenant per bucket eliminates complex multi-tenancy in storage layer #tenant-isolation
- [performance] TigrisFS intelligent caching could provide near-local performance for common operations #tigrisfs-advantage
- [deployment] Zero-downtime updates become trivial without volume constraints #deployment-simplification
- [benefit] Object storage pricing model could be more favorable than volume pricing #cost-optimization
- [innovation] Read-only local access alone would address major SaaS limitation #competitive-advantage
- [risk-mitigation] API-driven sync reduces performance requirements vs real-time file watching #risk-reduction
## Relations
- implements [[SPEC-6 Explicit Project Parameter Architecture]]
- requires [[Fly.io Tigris Integration]]
- enables [[Local Cloud File Access]]
- alternative_to [[Fly Volume Storage]]
## Links
- https://fly.io/hello/tigris
- https://fly.io/docs/tigris/
- https://www.tigrisdata.com/docs/sdks/fly/data-migration-with-flyctl/
- https://www.tigrisdata.com/docs/training/tigrisfs/
- https://www.tigrisdata.com/blog/tigris-filesystem/
- https://www.tigrisdata.com/docs/quickstarts/rclone/
-886
View File
@@ -1,886 +0,0 @@
---
title: 'SPEC-8: TigrisFS Integration for Tenant API'
Date: September 22, 2025
Status: Phase 3.6 Complete - Tenant Mount API Endpoints Ready for CLI Implementation
Priority: High
Goal: Replace Fly volumes with Tigris bucket provisioning in production tenant API
permalink: spec-8-tigris-fs-integration
---
## Executive Summary
Based on SPEC-7 Phase 4 POC testing, this spec outlines productizing the TigrisFS/rclone implementation in the Basic Memory Cloud tenant API.
We're moving from proof-of-concept to production integration, replacing Fly volume storage with Tigris bucket-per-tenant architecture.
## Current Architecture (Fly Volumes)
### Tenant Provisioning Flow
```python
# apps/cloud/src/basic_memory_cloud/workflows/tenant_provisioning.py
async def provision_tenant_infrastructure(tenant_id: str):
# 1. Create Fly app
# 2. Create Fly volume ← REPLACE THIS
# 3. Deploy API container with volume mount
# 4. Configure health checks
```
### Storage Implementation
- Each tenant gets dedicated Fly volume (1GB-10GB)
- Volume mounted at `/app/data` in API container
- Local filesystem storage with Basic Memory indexing
- No global caching or edge distribution
## Proposed Architecture (Tigris Buckets)
### New Tenant Provisioning Flow
```python
async def provision_tenant_infrastructure(tenant_id: str):
# 1. Create Fly app
# 2. Create Tigris bucket with admin credentials ← NEW
# 3. Store bucket name in tenant record ← NEW
# 4. Deploy API container with TigrisFS mount using admin credentials
# 5. Configure health checks
```
### Storage Implementation
- Each tenant gets dedicated Tigris bucket
- TigrisFS mounts bucket at `/app/data` in API container
- Global edge caching and distribution
- Configurable cache TTL for sync performance
## Implementation Plan
### Phase 1: Bucket Provisioning Service
**✅ IMPLEMENTED: StorageClient with Admin Credentials**
```python
# apps/cloud/src/basic_memory_cloud/clients/storage_client.py
class StorageClient:
async def create_tenant_bucket(self, tenant_id: UUID) -> TigrisBucketCredentials
async def delete_tenant_bucket(self, tenant_id: UUID, bucket_name: str) -> bool
async def list_buckets(self) -> list[TigrisBucketResponse]
async def test_tenant_credentials(self, credentials: TigrisBucketCredentials) -> bool
```
**Simplified Architecture Using Admin Credentials:**
- Single admin access key with full Tigris permissions (configured in console)
- No tenant-specific IAM user creation needed
- Bucket-per-tenant isolation for logical separation
- Admin credentials shared across all tenant operations
**Integrate with Provisioning workflow:**
```python
# Update tenant_provisioning.py
async def provision_tenant_infrastructure(tenant_id: str):
storage_client = StorageClient(settings.aws_access_key_id, settings.aws_secret_access_key)
bucket_creds = await storage_client.create_tenant_bucket(tenant_id)
await store_bucket_name(tenant_id, bucket_creds.bucket_name)
await deploy_api_with_tigris(tenant_id, bucket_creds)
```
### Phase 2: Simplified Bucket Management
**✅ SIMPLIFIED: Admin Credentials + Bucket Names Only**
Since we use admin credentials for all operations, we only need to track bucket names per tenant:
1. **Primary Storage (Fly Secrets)**
```bash
flyctl secrets set -a basic-memory-{tenant_id} \
AWS_ACCESS_KEY_ID="{admin_access_key}" \
AWS_SECRET_ACCESS_KEY="{admin_secret_key}" \
AWS_ENDPOINT_URL_S3="https://fly.storage.tigris.dev" \
AWS_REGION="auto" \
BUCKET_NAME="basic-memory-{tenant_id}"
```
2. **Database Storage (Bucket Name Only)**
```python
# apps/cloud/src/basic_memory_cloud/models/tenant.py
class Tenant(BaseModel):
# ... existing fields
tigris_bucket_name: Optional[str] = None # Just store bucket name
tigris_region: str = "auto"
created_at: datetime
```
**Benefits of Simplified Approach:**
- No credential encryption/decryption needed
- Admin credentials managed centrally in environment
- Only bucket names stored in database (not sensitive)
- Simplified backup/restore scenarios
- Reduced security attack surface
### Phase 3: API Container Updates
**Update API container configuration:**
```dockerfile
# apps/api/Dockerfile
# Add TigrisFS installation
RUN curl -L https://github.com/tigrisdata/tigrisfs/releases/latest/download/tigrisfs-linux-amd64 \
-o /usr/local/bin/tigrisfs && chmod +x /usr/local/bin/tigrisfs
```
**Startup script integration:**
```bash
# apps/api/tigrisfs-startup.sh (already exists)
# Mount TigrisFS → Start Basic Memory API
exec python -m basic_memory_cloud_api.main
```
**Fly.toml environment (optimized for < 5s startup):**
```toml
# apps/api/fly.tigris-production.toml
[env]
TIGRISFS_MEMORY_LIMIT = '1024' # Reduced for faster init
TIGRISFS_MAX_FLUSHERS = '16' # Fewer threads for faster startup
TIGRISFS_STAT_CACHE_TTL = '30s' # Balance sync speed vs startup
TIGRISFS_LAZY_INIT = 'true' # Enable lazy loading
BASIC_MEMORY_HOME = '/app/data'
# Suspend optimization for wake-on-network
[machine]
auto_stop_machines = "suspend" # Faster than full stop
auto_start_machines = true
min_machines_running = 0
```
### Phase 4: Local Access Features
**CLI automation for local mounting:**
```python
# New CLI command: basic-memory cloud mount
async def setup_local_mount(tenant_id: str):
# 1. Fetch bucket credentials from cloud API
# 2. Configure rclone with scoped IAM policy
# 3. Mount via rclone nfsmount (macOS) or FUSE (Linux)
# 4. Start Basic Memory sync watcher
```
**Local mount configuration:**
```bash
# rclone config for tenant
rclone mount basic-memory-{tenant_id}: ~/basic-memory-{tenant_id} \
--nfs-mount \
--vfs-cache-mode writes \
--cache-dir ~/.cache/rclone/basic-memory-{tenant_id}
```
### Phase 5: TigrisFS Cache Sync Solutions
**Problem**: When files are uploaded via CLI/bisync, the tenant API container doesn't see them immediately due to TigrisFS cache (30s TTL) and lack of inotify events on mounted filesystems.
**Multi-Layer Solution:**
**Layer 1: API Sync Endpoint** (Immediate)
```python
# POST /sync - Force TigrisFS cache refresh
# Callable by CLI after uploads
subprocess.run(["sync", "fsync /app/data"], check=True)
```
**Layer 2: Tigris Webhook Integration** (Real-time)
https://www.tigrisdata.com/docs/buckets/object-notifications/#webhook
```python
# Webhook endpoint for bucket changes
@app.post("/webhooks/tigris/{tenant_id}")
async def handle_bucket_notification(tenant_id: str, event: TigrisEvent):
if event.eventName in ["OBJECT_CREATED_PUT", "OBJECT_DELETED"]:
await notify_container_sync(tenant_id, event.object.key)
```
**Layer 3: CLI Sync Notification** (User-triggered)
```bash
# CLI calls container sync endpoint after successful bisync
basic-memory cloud bisync # Automatically notifies container
curl -X POST https://basic-memory-{tenant-id}.fly.dev/sync
```
**Layer 4: Periodic Sync Fallback** (Safety net)
```python
# Background task: fsync /app/data every 30s as fallback
# Ensures eventual consistency even if other layers fail
```
**Implementation Priority:**
1. Layer 1 (API endpoint) - Quick testing capability
2. Layer 3 (CLI integration) - Improved UX
3. Layer 4 (Periodic fallback) - Safety net
4. Layer 2 (Webhooks) - Production real-time sync
## Performance Targets
### Sync Latency
- **Target**: < 5 seconds local→cloud→container
- **Configuration**: `TIGRISFS_STAT_CACHE_TTL = '5s'`
- **Monitoring**: Track sync metrics in production
### Container Startup
- **Target**: < 5 seconds including TigrisFS mount
- **Fast retry**: 0.5s intervals for mount verification
- **Fallback**: Container fails fast if mount fails
### Memory Usage
- **TigrisFS cache**: 2GB memory limit per container
- **Concurrent uploads**: 32 flushers max
- **VM sizing**: shared-cpu-2x (2048mb) minimum
## Security Considerations
### Bucket Isolation
- Each tenant has dedicated bucket
- IAM policies prevent cross-tenant access
- No shared bucket with subdirectories
### Credential Security
- Fly secrets for runtime access
- Encrypted database backup for disaster recovery
- Credential rotation capability
### Data Residency
- Tigris global edge caching
- SOC2 Type II compliance
- Encryption at rest and in transit
## Operational Benefits
### Scalability
- Horizontal scaling with stateless API containers
- Global edge distribution
- Better resource utilization
### Reliability
- No cold starts between tenants
- Built-in redundancy and caching
- Simplified backup strategy
### Cost Efficiency
- Pay-per-use storage pricing
- Shared infrastructure benefits
- Reduced operational overhead
## Risk Mitigation
### Data Loss Prevention
- Dual credential storage (Fly + database)
- Automated backup workflows to R2/S3
- Tigris built-in redundancy
### Performance Degradation
- Configurable cache settings per tenant
- Monitoring and alerting on sync latency
- Fallback to volume storage if needed
### Security Vulnerabilities
- Bucket-per-tenant isolation
- Regular credential rotation
- Security scanning and monitoring
## Success Metrics
### Technical Metrics
- Sync latency P50 < 5 seconds
- Container startup time < 5 seconds
- Zero data loss incidents
- 99.9% uptime per tenant
### Business Metrics
- Reduced infrastructure costs vs volumes
- Improved user experience with faster sync
- Enhanced enterprise security posture
- Simplified operational overhead
## Open Questions
1. **Tigris rate limits**: What are the API limits for bucket creation?
2. **Cost analysis**: What's the break-even point vs Fly volumes?
3. **Regional preferences**: Should enterprise customers choose regions?
4. **Backup retention**: How long to keep automated backups?
## Implementation Checklist
### Phase 1: Bucket Provisioning Service ✅ COMPLETED
- [x] **Research Tigris bucket API** - Document bucket creation and S3 API compatibility
- [x] **Create StorageClient class** - Implemented with admin credentials and comprehensive integration tests
- [x] **Test bucket creation** - Full test suite validates API integration with real Tigris environment
- [x] **Add bucket provisioning to DBOS workflow** - Integrated StorageClient with tenant_provisioning.py
### Phase 2: Simplified Bucket Management ✅ COMPLETED
- [x] **Update Tenant model** with tigris_bucket_name field (replaced fly_volume_id)
- [x] **Implement bucket name storage** - Database migration and model updates completed
- [x] **Test bucket provisioning integration** - Full test suite validates workflow from tenant creation to bucket assignment
- [x] **Remove volume logic from all tests** - Complete migration from volume-based to bucket-based architecture
### Phase 3: API Container Integration ✅ COMPLETED
- [x] **Update Dockerfile** to install TigrisFS binary in API container with configurable version
- [x] **Optimize tigrisfs-startup.sh** with production-ready security and reliability improvements
- [x] **Create production-ready container** with proper signal handling and mount validation
- [x] **Implement security fixes** based on Claude code review (conditional debug, credential protection)
- [x] **Add proper process supervision** with cleanup traps and error handling
- [x] **Remove debug artifacts** - Cleaned up all debug Dockerfiles and test scripts
### Phase 3.5: IAM Access Key Management ✅ COMPLETED
- [x] **Research Tigris IAM API** - Documented create_policy, attach_user_policy, delete_access_key operations
- [x] **Implement bucket-scoped credential generation** - StorageClient.create_tenant_access_keys() with IAM policies
- [x] **Add comprehensive security test suite** - 5 security-focused integration tests covering all attack vectors
- [x] **Verify cross-bucket access prevention** - Scoped credentials can ONLY access their designated bucket
- [x] **Test credential lifecycle management** - Create, validate, delete, and revoke access keys
- [x] **Validate admin vs scoped credential isolation** - Different access patterns and security boundaries
- [x] **Test multi-tenant isolation** - Multiple tenants cannot access each other's buckets
### Phase 3.6: Tenant Mount API Endpoints ✅ COMPLETED
- [x] **Implement GET /tenant/mount/info** - Returns mount info without exposing credentials
- [x] **Implement POST /tenant/mount/credentials** - Creates new bucket-scoped credentials for CLI mounting
- [x] **Implement DELETE /tenant/mount/credentials/{cred_id}** - Revoke specific credentials with proper cleanup
- [x] **Implement GET /tenant/mount/credentials** - List active credentials without exposing secrets
- [x] **Add TenantMountCredentials database model** - Tracks credential metadata (no secret storage)
- [x] **Create comprehensive test suite** - 28 tests covering all scenarios including multi-session support
- [x] **Implement multi-session credential flow** - Multiple active credentials per tenant supported
- [x] **Secure credential handling** - Secret keys never stored, returned once only for immediate use
- [x] **Add dependency injection for StorageClient** - Clean integration with existing API architecture
- [x] **Fix Tigris configuration for cloud service** - Added AWS environment variables to fly.template.toml
- [x] **Update tenant machine configurations** - Include AWS credentials for TigrisFS mounting with clear credential strategy
**Security Test Results:**
```
✅ Cross-bucket access prevention - PASS
✅ Deleted credentials access revoked - PASS
✅ Invalid credentials rejected - PASS
✅ Admin vs scoped credential isolation - PASS
✅ Multiple scoped credentials isolation - PASS
```
**Implementation Details:**
- Uses Tigris IAM managed policies (create_policy + attach_user_policy)
- Bucket-scoped S3 policies with Actions: GetObject, PutObject, DeleteObject, ListBucket
- Resource ARNs limited to specific bucket: `arn:aws:s3:::bucket-name` and `arn:aws:s3:::bucket-name/*`
- Access keys follow Tigris format: `tid_` prefix with secure random suffix
- Complete cleanup on deletion removes both access keys and associated policies
### Phase 4: Local Access CLI
- [x] **Design local mount CLI command** for automated rclone configuration
- [x] **Implement credential fetching** from cloud API for local setup
- [x] **Create rclone config automation** for tenant-specific bucket mounting
- [x] **Test local→cloud→container sync** with optimized cache settings
- [x] **Document local access setup** for beta users
### Phase 5: Webhook Integration (Future)
- [ ] **Research Tigris webhook API** for object notifications and payload format
- [ ] **Design webhook endpoint** for real-time sync notifications
- [ ] **Implement notification handling** to trigger Basic Memory sync events
- [ ] **Test webhook delivery** and sync latency improvements
## Success Metrics
- [ ] **Container startup < 5 seconds** including TigrisFS mount and Basic Memory init
- [ ] **Sync latency < 5 seconds** for local→cloud→container file changes
- [ ] **Zero data loss** during bucket provisioning and credential management
- [ ] **100% test coverage** for new TigrisBucketService and credential functions
- [ ] **Beta deployment** with internal users validating local-cloud workflow
## Implementation Notes
## Phase 4.1: Bidirectional Sync with rclone bisync (NEW)
### Problem Statement
During testing, we discovered that some applications (particularly Obsidian) don't detect file changes over NFS mounts. Rather than building a custom sync daemon, we can leverage `rclone bisync` - rclone's built-in bidirectional synchronization feature.
### Solution: rclone bisync
Use rclone's proven bidirectional sync instead of custom implementation:
**Core Architecture:**
```bash
# rclone bisync handles all the complexity
rclone bisync ~/basic-memory-{tenant_id} basic-memory-{tenant_id}:{bucket_name} \
--create-empty-src-dirs \
--conflict-resolve newer \
--resilient \
--check-access
```
**Key Benefits:**
- ✅ **Battle-tested**: Production-proven rclone functionality
- ✅ **MIT licensed**: Open source with permissive licensing
- ✅ **No custom code**: Zero maintenance burden for sync logic
- ✅ **Built-in safety**: max-delete protection, conflict resolution
- ✅ **Simple installation**: Works with Homebrew rclone (no FUSE needed)
- ✅ **File watcher compatible**: Works with Obsidian and all applications
- ✅ **Offline support**: Can work offline and sync when connected
### bisync Conflict Resolution Options
**Built-in conflict strategies:**
```bash
--conflict-resolve none # Keep both files with .conflict suffixes (safest)
--conflict-resolve newer # Always pick the most recently modified file
--conflict-resolve larger # Choose based on file size
--conflict-resolve path1 # Always prefer local changes
--conflict-resolve path2 # Always prefer cloud changes
```
### Sync Profiles Using bisync
**Profile configurations:**
```python
BISYNC_PROFILES = {
"safe": {
"conflict_resolve": "none", # Keep both versions
"max_delete": 10, # Prevent mass deletion
"check_access": True, # Verify sync integrity
"description": "Safe mode with conflict preservation"
},
"balanced": {
"conflict_resolve": "newer", # Auto-resolve to newer file
"max_delete": 25,
"check_access": True,
"description": "Balanced mode (recommended default)"
},
"fast": {
"conflict_resolve": "newer",
"max_delete": 50,
"check_access": False, # Skip verification for speed
"description": "Fast mode for rapid iteration"
}
}
```
### CLI Commands
**Manual sync commands:**
```bash
basic-memory cloud bisync # Manual bidirectional sync
basic-memory cloud bisync --dry-run # Preview changes
basic-memory cloud bisync --profile safe # Use specific profile
basic-memory cloud bisync --resync # Force full baseline resync
```
**Watch mode (Step 1):**
```bash
basic-memory cloud bisync --watch # Long-running process, sync every 60s
basic-memory cloud bisync --watch --interval 30s # Custom interval
```
**System integration (Step 2 - Future):**
```bash
basic-memory cloud bisync-service install # Install as system service
basic-memory cloud bisync-service start # Start background service
basic-memory cloud bisync-service status # Check service status
```
### Implementation Strategy
**Phase 4.1.1: Core bisync Implementation**
- [ ] Implement `run_bisync()` function wrapping rclone bisync
- [ ] Add profile-based configuration (safe/balanced/fast)
- [ ] Create conflict resolution and safety options
- [ ] Test with sample files and conflict scenarios
**Phase 4.1.2: Watch Mode**
- [ ] Add `--watch` flag for continuous sync
- [ ] Implement configurable sync intervals
- [ ] Add graceful shutdown and signal handling
- [ ] Create status monitoring and progress indicators
**Phase 4.1.3: User Experience**
- [ ] Add conflict reporting and resolution guidance
- [ ] Implement dry-run preview functionality
- [ ] Create troubleshooting and diagnostic commands
- [ ] Add filtering configuration (.gitignore-style)
**Phase 4.1.4: System Integration (Future)**
- [ ] Generate platform-specific service files (launchd/systemd)
- [ ] Add service management commands
- [ ] Implement automatic startup and recovery
- [ ] Create monitoring and logging integration
### Technical Implementation
**Core bisync wrapper:**
```python
def run_bisync(
tenant_id: str,
bucket_name: str,
profile: str = "balanced",
dry_run: bool = False
) -> bool:
"""Run rclone bisync with specified profile."""
local_path = Path.home() / f"basic-memory-{tenant_id}"
remote_path = f"basic-memory-{tenant_id}:{bucket_name}"
profile_config = BISYNC_PROFILES[profile]
cmd = [
"rclone", "bisync",
str(local_path), remote_path,
"--create-empty-src-dirs",
"--resilient",
f"--conflict-resolve={profile_config['conflict_resolve']}",
f"--max-delete={profile_config['max_delete']}",
"--filters-file", "~/.basic-memory/bisync-filters.txt"
]
if profile_config.get("check_access"):
cmd.append("--check-access")
if dry_run:
cmd.append("--dry-run")
return subprocess.run(cmd, check=True).returncode == 0
```
**Default filter file (~/.basic-memory/bisync-filters.txt):**
```
- .DS_Store
- .git/**
- __pycache__/**
- *.pyc
- .pytest_cache/**
- node_modules/**
- .conflict-*
- Thumbs.db
- desktop.ini
```
**Advantages Over Custom Daemon:**
- ✅ **Zero maintenance**: No custom sync logic to debug/maintain
- ✅ **Production proven**: Used by thousands in production
- ✅ **Safety features**: Built-in max-delete, conflict handling, recovery
- ✅ **Filtering**: Advanced exclude patterns and rules
- ✅ **Performance**: Optimized for various storage backends
- ✅ **Community support**: Extensive documentation and community
## Phase 4.2: NFS Mount Support (Direct Access)
### Solution: rclone nfsmount
Keep the existing NFS mount functionality for users who prefer direct file access:
**Core Architecture:**
```bash
# rclone nfsmount provides transparent file access
rclone nfsmount basic-memory-{tenant_id}:{bucket_name} ~/basic-memory-{tenant_id} \
--vfs-cache-mode writes \
--dir-cache-time 10s \
--daemon
```
**Key Benefits:**
- ✅ **Real-time access**: Files appear immediately as they're created/modified
- ✅ **Transparent**: Works with any application that reads/writes files
- ✅ **Low latency**: Direct access without sync delays
- ✅ **Simple**: No periodic sync commands needed
- ✅ **Homebrew compatible**: Works with Homebrew rclone (no FUSE required)
**Limitations:**
- ❌ **File watcher compatibility**: Some apps (Obsidian) don't detect changes over NFS
- ❌ **Network dependency**: Requires active connection to cloud storage
- ❌ **Potential conflicts**: Simultaneous edits from multiple locations can cause issues
### Mount Profiles (Existing)
**Already implemented profiles from SPEC-7 testing:**
```python
MOUNT_PROFILES = {
"fast": {
"cache_time": "5s",
"poll_interval": "3s",
"description": "Ultra-fast development (5s sync)"
},
"balanced": {
"cache_time": "10s",
"poll_interval": "5s",
"description": "Fast development (10-15s sync, recommended)"
},
"safe": {
"cache_time": "15s",
"poll_interval": "10s",
"description": "Conflict-aware mount with backup",
"extra_args": ["--conflict-suffix", ".conflict-{DateTimeExt}"]
}
}
```
### CLI Commands (Existing)
**Mount commands already implemented:**
```bash
basic-memory cloud mount # Mount with balanced profile
basic-memory cloud mount --profile fast # Ultra-fast caching
basic-memory cloud mount --profile safe # Conflict detection
basic-memory cloud unmount # Clean unmount
basic-memory cloud mount-status # Show mount status
```
## User Choice: Mount vs Bisync
### When to Use Each Approach
| Use Case | Recommended Solution | Why |
|----------|---------------------|-----|
| **Obsidian users** | `bisync` | File watcher support for live preview |
| **CLI/vim/emacs users** | `mount` | Direct file access, lower latency |
| **Offline work** | `bisync` | Can work offline, sync when connected |
| **Real-time collaboration** | `mount` | Immediate visibility of changes |
| **Multiple machines** | `bisync` | Better conflict handling |
| **Single machine** | `mount` | Simpler, more transparent |
| **Development work** | Either | Both work well, user preference |
| **Large files** | `mount` | Streaming access vs full download |
### Installation Simplicity
**Both approaches now use simple Homebrew installation:**
```bash
# Single installation command for both approaches
brew install rclone
# No macFUSE, no system modifications needed
# Works immediately with both mount and bisync
```
### Implementation Status
**Phase 4.1: bisync** (NEW)
- [ ] Implement bisync command wrapper
- [ ] Add watch mode with configurable intervals
- [ ] Create conflict resolution workflows
- [ ] Add filtering and safety options
**Phase 4.2: mount** (EXISTING - ✅ IMPLEMENTED)
- [x] NFS mount commands with profile support
- [x] Mount management and cleanup
- [x] Process monitoring and health checks
- [x] Credential integration with cloud API
**Both approaches share:**
- [x] Credential management via cloud API
- [x] Secure rclone configuration
- [x] Tenant isolation and bucket scoping
- [x] Simple Homebrew rclone installation
Key Features:
1. Cross-Platform rclone Installation (rclone_installer.py):
- macOS: Homebrew → official script fallback
- Linux: snap → apt → official script fallback
- Windows: winget → chocolatey → scoop fallback
- Automatic version detection and verification
2. Smart rclone Configuration (rclone_config.py):
- Automatic tenant-specific config generation
- Three optimized mount profiles from your SPEC-7 testing:
- fast: 5s sync (ultra-performance)
- balanced: 10-15s sync (recommended default)
- safe: 15s sync + conflict detection
- Backup existing configs before modification
3. Robust Mount Management (mount_commands.py):
- Automatic tenant credential generation
- Mount path management (~/basic-memory-{tenant-id})
- Process lifecycle management (prevent duplicate mounts)
- Orphaned process cleanup
- Mount verification and health checking
4. Clean Architecture (api_client.py):
- Separated API client to avoid circular imports
- Reuses existing authentication infrastructure
- Consistent error handling and logging
User Experience:
One-Command Setup:
basic-memory cloud setup
```bash
# 1. Installs rclone automatically
# 2. Authenticates with existing login
# 3. Generates secure credentials
# 4. Configures rclone
# 5. Performs initial mount
```
Profile-Based Mounting:
basic-memory cloud mount --profile fast # 5s sync
basic-memory cloud mount --profile balanced # 15s sync (default)
basic-memory cloud mount --profile safe # conflict detection
Status Monitoring:
basic-memory cloud mount-status
```bash
# Shows: tenant info, mount path, sync profile, rclone processes
```
### local mount api
Endpoint 1: Get Tenant Info for user
Purpose: Get tenant details for mounting
- pass in jwt
- service returns mount info
**✅ IMPLEMENTED API Specification:**
**Endpoint 1: GET /tenant/mount/info**
- Purpose: Get tenant mount information without exposing credentials
- Authentication: JWT token (tenant_id extracted from claims)
Request:
```
GET /tenant/mount/info
Authorization: Bearer {jwt_token}
```
Response:
```json
{
"tenant_id": "434252dd-d83b-4b20-bf70-8a950ff875c4",
"bucket_name": "basic-memory-434252dd",
"has_credentials": true,
"credentials_created_at": "2025-09-22T16:48:50.414694"
}
```
**Endpoint 2: POST /tenant/mount/credentials**
- Purpose: Generate NEW bucket-scoped S3 credentials for rclone mounting
- Authentication: JWT token (tenant_id extracted from claims)
- Multi-session: Creates new credentials without revoking existing ones
Request:
```
POST /tenant/mount/credentials
Authorization: Bearer {jwt_token}
Content-Type: application/json
```
*Note: No request body needed - tenant_id extracted from JWT*
Response:
```json
{
"tenant_id": "434252dd-d83b-4b20-bf70-8a950ff875c4",
"bucket_name": "basic-memory-434252dd",
"access_key": "test_access_key_12345",
"secret_key": "test_secret_key_abcdef",
"endpoint_url": "https://fly.storage.tigris.dev",
"region": "auto"
}
```
**🔒 Security Notes:**
- Secret key returned ONCE only - never stored in database
- Credentials are bucket-scoped (cannot access other tenants' buckets)
- Multiple active credentials supported per tenant (work laptop + personal machine)
Implementation Notes
Security:
- Both endpoints require JWT authentication
- Extract tenant_id from JWT claims (not request body)
- Generate scoped credentials (not admin credentials)
- Credentials should have bucket-specific access only
Integration Points:
- Use your existing StorageClient from SPEC-8 implementation
- Leverage existing JWT middleware for tenant extraction
- Return same credential format as your Tigris bucket provisioning
Error Handling:
- 401 if not authenticated
- 403 if tenant doesn't exist
- 500 if credential generation fails
**🔄 Design Decisions:**
1. **Secure Credential Flow (No Secret Storage)**
Based on CLI flow analysis, we follow security best practices:
- ✅ API generates both access_key + secret_key via Tigris IAM
- ✅ Returns both in API response for immediate use
- ✅ CLI uses credentials immediately to configure rclone
- ✅ Database stores only metadata (access_key + policy_arn for cleanup)
- ✅ rclone handles secure local credential storage
- ❌ **Never store secret_key in database (even encrypted)**
2. **CLI Credential Flow**
```bash
# CLI calls API
POST /tenant/mount/credentials → {access_key, secret_key, ...}
# CLI immediately configures rclone
rclone config create basic-memory-{tenant_id} s3 \
access_key_id={access_key} \
secret_access_key={secret_key} \
endpoint=https://fly.storage.tigris.dev
# Database tracks metadata only
INSERT INTO tenant_mount_credentials (tenant_id, access_key, policy_arn, ...)
```
3. **Multiple Sessions Supported**
- Users can have multiple active credential sets (work laptop, personal machine, etc.)
- Each credential generation creates a new Tigris access key
- List active credentials via API (shows access_key but never secret)
4. **Failure Handling & Cleanup**
- **Happy Path**: Credentials created → Used immediately → rclone configured
- **Orphaned Credentials**: Background job revokes unused credentials
- **API Failure Recovery**: Retry Tigris deletion with stored policy_arn
- **Status Tracking**: Track tigris_deletion_status (pending/completed/failed)
5. **Event Sourcing & Audit**
- MountCredentialCreatedEvent
- MountCredentialRevokedEvent
- MountCredentialOrphanedEvent (for cleanup)
- Full audit trail for security compliance
6. **Tenant/Bucket Validation**
- Verify tenant exists and has valid bucket before credential generation
- Use existing StorageClient to validate bucket access
- Prevent credential generation for inactive/invalid tenants
📋 **Implemented API Endpoints:**
```
✅ IMPLEMENTED:
GET /tenant/mount/info # Get tenant/bucket info (no credentials exposed)
POST /tenant/mount/credentials # Generate new credentials (returns secret once)
GET /tenant/mount/credentials # List active credentials (no secrets)
DELETE /tenant/mount/credentials/{cred_id} # Revoke specific credentials
```
**API Implementation Status:**
- ✅ **GET /tenant/mount/info**: Returns tenant_id, bucket_name, has_credentials, credentials_created_at
- ✅ **POST /tenant/mount/credentials**: Creates new bucket-scoped access keys, returns access_key + secret_key once
- ✅ **GET /tenant/mount/credentials**: Lists active credentials without exposing secret keys
- ✅ **DELETE /tenant/mount/credentials/{cred_id}**: Revokes specific credentials with proper Tigris IAM cleanup
- ✅ **Multi-session support**: Multiple active credentials per tenant (work laptop + personal machine)
- ✅ **Security**: Secret keys never stored in database, returned once only for immediate use
- ✅ **Comprehensive test suite**: 28 tests covering all scenarios including error handling and multi-session flows
- ✅ **Dependency injection**: Clean integration with existing FastAPI architecture
- ✅ **Production-ready configuration**: Tigris credentials properly configured for tenant machines
🗄️ **Secure Database Schema:**
```sql
CREATE TABLE tenant_mount_credentials (
id UUID PRIMARY KEY,
tenant_id UUID REFERENCES tenant(id),
access_key VARCHAR(255) NOT NULL,
-- secret_key REMOVED - never store secrets (security best practice)
policy_arn VARCHAR(255) NOT NULL, -- For Tigris IAM cleanup
tigris_deletion_status VARCHAR(20) DEFAULT 'pending', -- Track cleanup
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW(),
revoked_at TIMESTAMP NULL,
last_used_at TIMESTAMP NULL, -- Track usage for orphan cleanup
description VARCHAR(255) DEFAULT 'CLI mount credentials'
);
```
**Security Benefits:**
- ✅ Database breach cannot expose secrets
- ✅ Follows "secrets don't persist" security principle
- ✅ Meets compliance requirements (SOC2, etc.)
- ✅ Reduced attack surface
- ✅ CLI gets credentials once and stores securely via rclone
File diff suppressed because it is too large Load Diff
@@ -1,196 +0,0 @@
---
title: 'SPEC-9: Signed Header Tenant Information'
type: spec
permalink: specs/spec-9-signed-header-tenant-information
tags:
- authentication
- tenant-isolation
- proxy
- security
- mcp
---
# SPEC-9: Signed Header Tenant Information
## Why
WorkOS JWT templates don't work with MCP's dynamic client registration requirement, preventing us from getting tenant information directly in JWT tokens. We need an alternative secure method to pass tenant context from the Cloud Proxy Service to tenant instances.
**Problem Context:**
- MCP spec requires dynamic client registration
- WorkOS JWT templates only apply to statically configured clients
- Without tenant information, we can't properly route requests or isolate tenant data
- Current JWT tokens only contain standard OIDC claims (sub, email, etc.)
**Affected Areas:**
- Cloud Proxy Service (`apps/cloud`) - request forwarding
- Tenant API instances (`apps/api`) - tenant context validation
- MCP Gateway (`apps/mcp`) - authentication flow
- Overall tenant isolation security model
## What
Implement HMAC-signed headers that the Cloud Proxy Service adds when forwarding requests to tenant instances. This provides secure, tamper-proof tenant information without relying on JWT custom claims.
**Components:**
- Header signing utility in Cloud Proxy Service
- Header validation middleware in Tenant API instances
- Shared secret configuration across services
- Fallback mechanisms for development and error cases
## How (High Level)
### 1. Header Format
Add these signed headers to all proxied requests:
```
X-BM-Tenant-ID: {tenant_id}
X-BM-Timestamp: {unix_timestamp}
X-BM-Signature: {hmac_sha256_signature}
```
### 2. Signature Algorithm
```python
# Canonical message format
message = f"{tenant_id}:{timestamp}"
# HMAC-SHA256 signature
signature = hmac.new(
key=shared_secret.encode('utf-8'),
msg=message.encode('utf-8'),
digestmod=hashlib.sha256
).hexdigest()
```
### 3. Implementation Flow
#### Cloud Proxy Service (`apps/cloud`)
1. Extract `tenant_id` from authenticated user profile
2. Generate timestamp and canonical message
3. Sign message with shared secret
4. Add headers to request before forwarding to tenant instance
#### Tenant API Instances (`apps/api`)
1. Middleware validates headers on all incoming requests
2. Extract tenant_id, timestamp from headers
3. Verify timestamp is within acceptable window (5 minutes)
4. Recompute signature and compare in constant time
5. If valid, make tenant context available to Basic Memory tools
### 4. Security Properties
- **Authenticity**: Only services with shared secret can create valid signatures
- **Integrity**: Header tampering invalidates signature
- **Replay Protection**: Timestamp prevents reuse of old signatures
- **Non-repudiation**: Each request is cryptographically tied to specific tenant
### 5. Configuration
```bash
# Shared across Cloud Proxy and Tenant instances
BM_TENANT_HEADER_SECRET=randomly-generated-256-bit-secret
# Tenant API configuration
BM_TENANT_HEADER_VALIDATION=true # true (production) | false (dev only)
```
## How to Evaluate
### Unit Tests
- [ ] Header signing utility generates correct signatures
- [ ] Header validation correctly accepts/rejects signatures
- [ ] Timestamp validation within acceptable windows
- [ ] Constant-time signature comparison prevents timing attacks
### Integration Tests
- [ ] End-to-end request flow from MCP client → proxy → tenant
- [ ] Tenant isolation verified with signed headers
- [ ] Error handling for missing/invalid headers
- [ ] Disabled validation in development environment
### Security Validation
- [ ] Shared secret rotation procedure
- [ ] Header tampering detection
- [ ] Clock skew tolerance testing
- [ ] Performance impact measurement
### Production Readiness
- [ ] Logging and monitoring of header validation
- [ ] Graceful degradation for header validation failures
- [ ] Documentation for secret management
- [ ] Deployment configuration templates
## Implementation Notes
### Shared Secret Management
- Generate cryptographically secure 256-bit secret
- Same secret deployed to Cloud Proxy and all Tenant instances
- Consider secret rotation strategy for production
### Error Handling
```python
# Strict mode (production)
if not validate_headers(request):
raise HTTPException(status_code=401, detail="Invalid tenant headers")
# Fallback mode (development)
if not validate_headers(request):
logger.warning("Invalid headers, falling back to default tenant")
tenant_id = "default"
```
### Performance Considerations
- HMAC-SHA256 computation is fast (~microseconds)
- Headers add ~200 bytes to each request
- Validation happens once per request in middleware
## Benefits
**Works with MCP dynamic client registration** - No dependency on JWT custom claims
**Simple and reliable** - Standard HMAC signature approach
**Secure by design** - Cryptographic authenticity and integrity
**Infrastructure controlled** - No external service dependencies
**Easy to implement** - Clear signature algorithm and validation
## Trade-offs
⚠️ **Shared secret management** - Need secure distribution and rotation
⚠️ **Clock synchronization** - Timestamp validation requires reasonably synced clocks
⚠️ **Header visibility** - Headers visible in logs (tenant_id not sensitive)
⚠️ **Additional complexity** - More moving parts in proxy forwarding
## Implementation Tasks
### Cloud Service (Header Signing)
- [ ] Create `utils/header_signing.py` with HMAC-SHA256 signing function
- [ ] Add `bm_tenant_header_secret` to Cloud service configuration
- [ ] Update `ProxyService.forward_request()` to call signing utility
- [ ] Add signed headers (X-BM-Tenant-ID, X-BM-Timestamp, X-BM-Signature)
### Tenant API (Header Validation)
- [ ] Create `utils/header_validation.py` with signature verification
- [ ] Add `bm_tenant_header_secret` to API service configuration
- [ ] Create `TenantHeaderValidationMiddleware` class
- [ ] Add middleware to FastAPI app (before other middleware)
- [ ] Skip validation for `/health` endpoint
- [ ] Store validated tenant_id in request.state
### Testing
- [ ] Unit test for header signing utility
- [ ] Unit test for header validation utility
- [ ] Integration test for proxy → tenant flow
- [ ] Test invalid/missing header handling
- [ ] Test timestamp window validation
- [ ] Test signature tampering detection
### Configuration & Deployment
- [ ] Update `.env.example` with BM_TENANT_HEADER_SECRET
- [ ] Generate secure 256-bit secret for production
- [ ] Update Fly.io secrets for both services
- [ ] Document secret rotation procedure
## Status
- [x] **Specification Complete** - Design finalized and documented
- [ ] **Implementation Started** - Header signing utility development
- [ ] **Cloud Proxy Updated** - ProxyService adds signed headers
- [ ] **Tenant Validation Added** - Middleware validates headers
- [ ] **Testing Complete** - All validation criteria met
- [ ] **Production Deployed** - Live with tenant isolation via headers
@@ -1,390 +0,0 @@
---
title: 'SPEC-9-1 Follow-Ups: Conflict, Sync, and Observability'
type: tasklist
permalink: specs/spec-9-follow-ups-conflict-sync-and-observability
related: specs/spec-9-multi-project-bisync
status: revised
revision_date: 2025-10-03
---
# SPEC-9-1 Follow-Ups: Conflict, Sync, and Observability
**REVISED 2025-10-03:** Simplified to leverage rclone built-ins instead of custom conflict handling.
**Context:** SPEC-9 delivered multi-project bidirectional sync and a unified CLI. This follow-up focuses on **observability and safety** using rclone's built-in capabilities rather than reinventing conflict handling.
**Design Philosophy: "Be Dumb Like Git"**
- Let rclone bisync handle conflict detection (it already does this)
- Make conflicts visible and recoverable, don't prevent them
- Cloud is always the winner on conflict (cloud-primary model)
- Users who want version history can just use Git locally in their sync directory
**What Changed from Original Version:**
- **Replaced:** Custom `.bmmeta` sidecars → Use rclone's `.bisync/` state tracking
- **Replaced:** Custom conflict detection → Use rclone bisync 3-way merge
- **Replaced:** Tombstone files → rclone delete tracking handles this
- **Replaced:** Distributed lease → Local process lock only (document multi-device warning)
- **Replaced:** S3 versioning service → Users just use Git locally if they want history
- **Deferred:** SPEC-14 Git integration → Postponed to teams/multi-user features
## ✅ Now
- [ ] **Local process lock**: Prevent concurrent bisync runs on same device (`~/.basic-memory/sync.lock`)
- [ ] **Structured sync reports**: Parse rclone bisync output into JSON reports (creates/updates/deletes/conflicts, bytes, duration); `bm sync --report`
- [ ] **Multi-device warning**: Document that users should not run `--watch` on multiple devices simultaneously
- [ ] **Version control guidance**: Document pattern for users to use Git locally in their sync directory if they want version history
- [ ] **Docs polish**: cloud-mode toggle, mount↔bisync directory isolation, conflict semantics, quick start, migration guide, short demo clip/GIF
## 🔜 Next
- [ ] **Observability commands**: `bm conflicts list`, `bm sync history` to view sync reports and conflicts
- [ ] **Conflict resolution UI**: `bm conflicts resolve <file>` to interactively pick winner from conflict files
- [ ] **Selective sync**: allow include/exclude by project; per-project profile (safe/balanced/fast)
## 🧭 Later
- [ ] **Near real-time sync**: File watcher → targeted `rclone copy` for individual files (keep bisync as backstop)
- [ ] **Sharing / scoped tokens**: cross-tenant/project access
- [ ] **Bandwidth controls & backpressure**: policy for large repos
- [ ] **Client-side encryption (optional)**: with clear trade-offs
## 📏 Acceptance criteria (for "Now" items)
- [ ] Local process lock prevents concurrent bisync runs on same device
- [ ] rclone bisync conflict files visible and documented (`file.conflict1.md`, `file.conflict2.md`)
- [ ] `bm sync --report` generates parsable JSON with sync statistics
- [ ] Documentation clearly warns about multi-device `--watch` mode
- [ ] Documentation shows users how to use Git locally for version history
## What We're NOT Building (Deferred to rclone)
- ❌ Custom `.bmmeta` sidecars (rclone tracks state in `.bisync/` workdir)
- ❌ Custom conflict detection (rclone bisync already does 3-way merge detection)
- ❌ Tombstone files (S3 versioning + rclone delete tracking handles this)
- ❌ Distributed lease (low probability issue, rclone detects state divergence)
- ❌ Rename/move tracking (rclone has size+modtime heuristics built-in)
## Implementation Summary
**Current State (SPEC-9):**
- ✅ rclone bisync with 3 profiles (safe/balanced/fast)
-`--max-delete` safety limits (10/25/50 files)
-`--conflict-resolve=newer` for auto-resolution
- ✅ Watch mode: `bm sync --watch` (60s intervals)
- ✅ Integrity checking: `bm cloud check`
- ✅ Mount vs bisync directory isolation
**What's Needed (This Spec):**
1. **Process lock** - Simple file-based lock in `~/.basic-memory/sync.lock`
2. **Sync reports** - Parse rclone output, save to `~/.basic-memory/sync-history/`
3. **Documentation** - Multi-device warnings, conflict resolution workflow, Git usage pattern
**User Model:**
- Cloud is always the winner on conflict (cloud-primary)
- rclone creates `.conflict` files for divergent edits
- Users who want version history just use Git in their local sync directory
- Users warned: don't run `--watch` on multiple devices
## Decision Rationale & Trade-offs
### Why Trust rclone Instead of Custom Conflict Handling?
**rclone bisync already provides:**
- 3-way merge detection (compares local, remote, and last-known state)
- File state tracking in `.bisync/` workdir (hashes, modtimes)
- Automatic conflict file creation: `file.conflict1.md`, `file.conflict2.md`
- Rename detection via size+modtime heuristics
- Delete tracking (prevents resurrection of deleted files)
- Battle-tested with extensive edge case handling
**What we'd have to build with custom approach:**
- Per-file metadata tracking (`.bmmeta` sidecars)
- 3-way diff algorithm
- Conflict detection logic
- Tombstone files for deletes
- Rename/move detection
- Testing for all edge cases
**Decision:** Use what rclone already does well. Don't reinvent the wheel.
### Why Let Users Use Git Locally Instead of Building Versioning?
**The simplest solution: Just use Git**
Users who want version history can literally just use Git in their sync directory:
```bash
cd ~/basic-memory-cloud-sync/
git init
git add .
git commit -m "backup"
# Push to their own GitHub if they want
git remote add origin git@github.com:user/my-knowledge.git
git push
```
**Why this is perfect:**
- ✅ We build nothing
- ✅ Users who want Git... just use Git
- ✅ Users who don't care... don't need to
- ✅ rclone bisync already handles sync conflicts
- ✅ Users own their data, they can version it however they want (Git, Time Machine, etc.)
**What we'd have to build for S3 versioning:**
- API to enable versioning on Tigris buckets
- **Problem**: Tigris doesn't support S3 bucket versioning
- Restore commands: `bm cloud restore --version-id`
- Version listing: `bm cloud versions <path>`
- Lifecycle policies for version retention
- Documentation and user education
**What we'd have to build for SPEC-14 Git integration:**
- Committer service (daemon watching `/app/data/`)
- Puller service (webhook handler for GitHub pushes)
- Git LFS for large files
- Loop prevention between Git ↔ bisync ↔ local
- Merge conflict handling at TWO layers (rclone + Git)
- Webhook infrastructure and monitoring
**Decision:** Don't build version control. Document the pattern. "The easiest problem to solve is the one you avoid."
**When to revisit:** Teams/multi-user features where server-side version control becomes necessary for collaboration.
### Why No Distributed Lease?
**Low probability issue:**
- Requires user to manually run `bm sync` on multiple devices at exact same time
- Most users run `--watch` on one primary device
- rclone bisync detects state divergence and fails safely
**Safety nets in place:**
- Local process lock prevents concurrent runs on same device
- rclone bisync aborts if bucket state changed during sync
- S3 versioning recovers from any overwrites
- Documentation warns against multi-device `--watch`
**Failure mode:**
```bash
# Device A and B sync simultaneously
Device A: bm sync → succeeds
Device B: bm sync → "Error: path has changed, run --resync"
# User fixes with resync
Device B: bm sync --resync → establishes new baseline
```
**Decision:** Document the issue, add local lock, defer distributed coordination until users report actual problems.
### Cloud-Primary Conflict Model
**User mental model:**
- Cloud is the source of truth (like Dropbox/iCloud)
- Local is working copy
- On conflict: cloud wins, local edits → `.conflict` file
- User manually picks winner
**Why this works:**
- Simpler than bidirectional merge (no automatic resolution risk)
- Matches user expectations from Dropbox
- S3 versioning provides safety net for overwrites
- Clear recovery path: restore from S3 version if needed
**Example workflow:**
```bash
# Edit file on Device A and Device B while offline
# Both devices come online and sync
Device A: bm sync
# → Pushes to cloud first, becomes canonical version
Device B: bm sync
# → Detects conflict
# → Cloud version: work/notes.md
# → Local version: work/notes.md.conflict1
# → User manually merges or picks winner
# Restore if needed
bm cloud restore work/notes.md --version-id abc123
```
## Implementation Details
### 1. Local Process Lock
```python
# ~/.basic-memory/sync.lock
import os
import psutil
from pathlib import Path
class SyncLock:
def __init__(self):
self.lock_file = Path.home() / '.basic-memory' / 'sync.lock'
def acquire(self):
if self.lock_file.exists():
pid = int(self.lock_file.read_text())
if psutil.pid_exists(pid):
raise BisyncError(
f"Sync already running (PID {pid}). "
f"Wait for completion or kill stale process."
)
# Stale lock, remove it
self.lock_file.unlink()
self.lock_file.write_text(str(os.getpid()))
def release(self):
if self.lock_file.exists():
self.lock_file.unlink()
def __enter__(self):
self.acquire()
return self
def __exit__(self, *args):
self.release()
# Usage
with SyncLock():
run_rclone_bisync()
```
### 3. Sync Report Parsing
```python
# Parse rclone bisync output
import json
from datetime import datetime
from pathlib import Path
def parse_sync_report(rclone_output: str, duration: float, exit_code: int) -> dict:
"""Parse rclone bisync output into structured report."""
# rclone bisync outputs lines like:
# "Synching Path1 /local/path with Path2 remote:bucket"
# "- Path1 File was copied to Path2"
# "Bisync successful"
report = {
"timestamp": datetime.now().isoformat(),
"duration_seconds": duration,
"exit_code": exit_code,
"success": exit_code == 0,
"files_created": 0,
"files_updated": 0,
"files_deleted": 0,
"conflicts": [],
"errors": []
}
for line in rclone_output.split('\n'):
if 'was copied to' in line:
report['files_created'] += 1
elif 'was updated in' in line:
report['files_updated'] += 1
elif 'was deleted from' in line:
report['files_deleted'] += 1
elif '.conflict' in line:
report['conflicts'].append(line.strip())
elif 'ERROR' in line:
report['errors'].append(line.strip())
return report
def save_sync_report(report: dict):
"""Save sync report to history."""
history_dir = Path.home() / '.basic-memory' / 'sync-history'
history_dir.mkdir(parents=True, exist_ok=True)
timestamp = datetime.now().strftime('%Y%m%d-%H%M%S')
report_file = history_dir / f'{timestamp}.json'
report_file.write_text(json.dumps(report, indent=2))
# Usage in run_bisync()
start_time = time.time()
result = subprocess.run(bisync_cmd, capture_output=True, text=True)
duration = time.time() - start_time
report = parse_sync_report(result.stdout, duration, result.returncode)
save_sync_report(report)
if report['conflicts']:
console.print(f"[yellow]⚠ {len(report['conflicts'])} conflict(s) detected[/yellow]")
console.print("[dim]Run 'bm conflicts list' to view[/dim]")
```
### 4. User Commands
```bash
# View sync history
bm sync history
# → Lists recent syncs from ~/.basic-memory/sync-history/*.json
# → Shows: timestamp, duration, files changed, conflicts, errors
# View current conflicts
bm conflicts list
# → Scans sync directory for *.conflict* files
# → Shows: file path, conflict versions, timestamps
# Restore from S3 version
bm cloud restore work/notes.md --version-id abc123
# → Uses aws s3api get-object with version-id
# → Downloads to original path
bm cloud restore work/notes.md --timestamp "2025-10-03 14:30"
# → Lists versions, finds closest to timestamp
# → Downloads that version
# List file versions
bm cloud versions work/notes.md
# → Uses aws s3api list-object-versions
# → Shows: version-id, timestamp, size, author
# Interactive conflict resolution
bm conflicts resolve work/notes.md
# → Shows both versions side-by-side
# → Prompts: Keep local, keep cloud, merge manually, restore from S3 version
# → Cleans up .conflict files after resolution
```
## Success Metrics & Monitoring
**Phase 1 (v1) - Basic Safety:**
- [ ] Conflict detection rate < 5% of syncs (measure in telemetry)
- [ ] User can resolve conflicts within 5 minutes (UX testing)
- [ ] Documentation prevents 90% of multi-device issues
**Phase 2 (v2) - Observability:**
- [ ] 80% of users check `bm sync history` when troubleshooting
- [ ] Average time to restore from S3 version < 2 minutes
-
- [ ] Conflict resolution success rate > 95%
**What to measure:**
```python
# Telemetry in sync reports
{
"conflict_rate": conflicts / total_syncs,
"multi_device_collisions": count_state_divergence_errors,
"version_restores": count_restore_operations,
"avg_sync_duration": sum(durations) / count,
"max_delete_trips": count_max_delete_aborts
}
```
**When to add distributed lease:**
- Multi-device collision rate > 5% of syncs
- User complaints about state divergence errors
- Evidence that local lock isn't sufficient
**When to revisit Git (SPEC-14):**
- Teams feature launches (multi-user collaboration)
- Users request commit messages / audit trail
- PR-based review workflow becomes valuable
## Links
- SPEC-9: `specs/spec-9-multi-project-bisync`
- SPEC-14: `specs/spec-14-cloud-git-versioning` (deferred in favor of S3 versioning)
- rclone bisync docs: https://rclone.org/bisync/
- Tigris S3 versioning: https://www.tigrisdata.com/docs/buckets/versioning/
---
**Owner:** <assign> | **Review cadence:** weekly in standup | **Last updated:** 2025-10-03
+1 -1
View File
@@ -1,7 +1,7 @@
"""basic-memory - Local-first knowledge management combining Zettelkasten with knowledge graphs"""
# Package version - updated by release automation
__version__ = "0.17.3"
__version__ = "0.17.8"
# API version for FastAPI - independent of package version
__api_version__ = "v0"
+11 -7
View File
@@ -5,14 +5,18 @@ import os
from logging.config import fileConfig
# Allow nested event loops (needed for pytest-asyncio and other async contexts)
# Note: nest_asyncio doesn't work with uvloop, so we handle that case separately
try:
import nest_asyncio
# Note: nest_asyncio doesn't work with uvloop or Python 3.14+, so we handle those cases separately
import sys
nest_asyncio.apply()
except (ImportError, ValueError):
# nest_asyncio not available or can't patch this loop type (e.g., uvloop)
pass
if sys.version_info < (3, 14):
try:
import nest_asyncio
nest_asyncio.apply()
except (ImportError, ValueError):
# nest_asyncio not available or can't patch this loop type (e.g., uvloop)
pass
# For Python 3.14+, we rely on the thread-based fallback in run_migrations_online()
from sqlalchemy import engine_from_config, pool
from sqlalchemy.ext.asyncio import AsyncEngine, create_async_engine
@@ -5,13 +5,13 @@ Revises: a2b3c4d5e6f7, g9a0b3c4d5e6
Create Date: 2025-12-29 12:46:46.476268
"""
from typing import Sequence, Union
# revision identifiers, used by Alembic.
revision: str = '6830751f5fb6'
down_revision: Union[str, Sequence[str], None] = ('a2b3c4d5e6f7', 'g9a0b3c4d5e6')
revision: str = "6830751f5fb6"
down_revision: Union[str, Sequence[str], None] = ("a2b3c4d5e6f7", "g9a0b3c4d5e6")
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
+20 -32
View File
@@ -1,6 +1,5 @@
"""FastAPI application for basic-memory knowledge graph API."""
import asyncio
from contextlib import asynccontextmanager
from fastapi import FastAPI, HTTPException
@@ -8,7 +7,7 @@ from fastapi.exception_handlers import http_exception_handler
from loguru import logger
from basic_memory import __version__ as version
from basic_memory import db
from basic_memory.api.container import ApiContainer, set_container
from basic_memory.api.routers import (
directory_router,
importer_router,
@@ -30,8 +29,8 @@ from basic_memory.api.v2.routers import (
prompt_router as v2_prompt,
importer_router as v2_importer,
)
from basic_memory.config import ConfigManager, init_api_logging
from basic_memory.services.initialization import initialize_file_sync, initialize_app
from basic_memory.config import init_api_logging
from basic_memory.services.initialization import initialize_app
@asynccontextmanager
@@ -41,47 +40,36 @@ async def lifespan(app: FastAPI): # pragma: no cover
# Initialize logging for API (stdout in cloud mode, file otherwise)
init_api_logging()
app_config = ConfigManager().config
logger.info("Starting Basic Memory API")
# --- Composition Root ---
# Create container and read config (single point of config access)
container = ApiContainer.create()
set_container(container)
app.state.container = container
await initialize_app(app_config)
logger.info(f"Starting Basic Memory API (mode={container.mode.name})")
await initialize_app(container.config)
# Cache database connections in app state for performance
logger.info("Initializing database and caching connections...")
engine, session_maker = await db.get_or_create_db(app_config.database_path)
engine, session_maker = await container.init_database()
app.state.engine = engine
app.state.session_maker = session_maker
logger.info("Database connections cached in app state")
# Start file sync if enabled
if app_config.sync_changes and not app_config.is_test_env:
logger.info(f"Sync changes enabled: {app_config.sync_changes}")
# Create and start sync coordinator (lifecycle centralized in coordinator)
sync_coordinator = container.create_sync_coordinator()
await sync_coordinator.start()
app.state.sync_coordinator = sync_coordinator
# start file sync task in background
async def _file_sync_runner() -> None:
await initialize_file_sync(app_config)
app.state.sync_task = asyncio.create_task(_file_sync_runner())
else:
if app_config.is_test_env:
logger.info("Test environment detected. Skipping file sync service.")
else:
logger.info("Sync changes disabled. Skipping file sync service.")
app.state.sync_task = None
# proceed with startup
# Proceed with startup
yield
# Shutdown - coordinator handles clean task cancellation
logger.info("Shutting down Basic Memory API")
if app.state.sync_task:
logger.info("Stopping sync...")
app.state.sync_task.cancel() # pyright: ignore
try:
await app.state.sync_task
except asyncio.CancelledError:
logger.info("Sync task cancelled successfully")
await sync_coordinator.stop()
await db.shutdown_db()
await container.shutdown_database()
# Initialize FastAPI app
+133
View File
@@ -0,0 +1,133 @@
"""API composition root for Basic Memory.
This container owns reading ConfigManager and environment variables for the
API entrypoint. Downstream modules receive config/dependencies explicitly
rather than reading globals.
Design principles:
- Only this module reads ConfigManager directly
- Runtime mode (cloud/local/test) is resolved here
- Factories for services are provided, not singletons
"""
from dataclasses import dataclass
from typing import TYPE_CHECKING
from sqlalchemy.ext.asyncio import AsyncEngine, async_sessionmaker, AsyncSession
from basic_memory import db
from basic_memory.config import BasicMemoryConfig, ConfigManager
from basic_memory.runtime import RuntimeMode, resolve_runtime_mode
if TYPE_CHECKING: # pragma: no cover
from basic_memory.sync import SyncCoordinator
@dataclass
class ApiContainer:
"""Composition root for the API entrypoint.
Holds resolved configuration and runtime context.
Created once at app startup, then used to wire dependencies.
"""
config: BasicMemoryConfig
mode: RuntimeMode
# --- Database ---
# Cached database connections (set during lifespan startup)
engine: AsyncEngine | None = None
session_maker: async_sessionmaker[AsyncSession] | None = None
@classmethod
def create(cls) -> "ApiContainer": # pragma: no cover
"""Create container by reading ConfigManager.
This is the single point where API reads global config.
"""
config = ConfigManager().config
mode = resolve_runtime_mode(
cloud_mode_enabled=config.cloud_mode_enabled,
is_test_env=config.is_test_env,
)
return cls(config=config, mode=mode)
# --- Runtime Mode Properties ---
@property
def should_sync_files(self) -> bool:
"""Whether file sync should be started.
Sync is enabled when:
- sync_changes is True in config
- Not in test mode (tests manage their own sync)
"""
return self.config.sync_changes and not self.mode.is_test
@property
def sync_skip_reason(self) -> str | None: # pragma: no cover
"""Reason why sync is skipped, or None if sync should run.
Useful for logging why sync was disabled.
"""
if self.mode.is_test:
return "Test environment detected"
if not self.config.sync_changes:
return "Sync changes disabled"
return None
def create_sync_coordinator(self) -> "SyncCoordinator": # pragma: no cover
"""Create a SyncCoordinator with this container's settings.
Returns:
SyncCoordinator configured for this runtime environment
"""
# Deferred import to avoid circular dependency
from basic_memory.sync import SyncCoordinator
return SyncCoordinator(
config=self.config,
should_sync=self.should_sync_files,
skip_reason=self.sync_skip_reason,
)
# --- Database Factory ---
async def init_database( # pragma: no cover
self,
) -> tuple[AsyncEngine, async_sessionmaker[AsyncSession]]:
"""Initialize and cache database connections.
Returns:
Tuple of (engine, session_maker)
"""
engine, session_maker = await db.get_or_create_db(self.config.database_path)
self.engine = engine
self.session_maker = session_maker
return engine, session_maker
async def shutdown_database(self) -> None: # pragma: no cover
"""Clean up database connections."""
await db.shutdown_db()
# Module-level container instance (set by lifespan)
# This allows deps.py to access the container without reading ConfigManager
_container: ApiContainer | None = None
def get_container() -> ApiContainer:
"""Get the current API container.
Raises:
RuntimeError: If container hasn't been initialized
"""
if _container is None:
raise RuntimeError("API container not initialized. Call set_container() first.")
return _container
def set_container(container: ApiContainer) -> None:
"""Set the API container (called by lifespan)."""
global _container
_container = container
+15 -3
View File
@@ -322,7 +322,14 @@ async def remove_project(
) # pragma: no cover
# Check if trying to delete the default project
if name == project_service.default_project:
# In cloud mode, database is source of truth; in local mode, check config
config_default = project_service.default_project
db_default = await project_service.repository.get_default_project()
# Use database default if available, otherwise fall back to config default
default_project_name = db_default.name if db_default else config_default
if name == default_project_name:
available_projects = await project_service.list_projects()
other_projects = [p.name for p in available_projects if p.name != name]
detail = f"Cannot delete default project '{name}'. "
@@ -418,8 +425,13 @@ async def get_default_project(
Returns:
Response with project default information
"""
# Get the old default project
default_name = project_service.default_project
# Get the default project
# In cloud mode, database is source of truth; in local mode, check config
config_default = project_service.default_project
db_default = await project_service.repository.get_default_project()
# Use database default if available, otherwise fall back to config default
default_name = db_default.name if db_default else config_default
default_project = await project_service.get_project(default_name)
if not default_project: # pragma: no cover
raise HTTPException( # pragma: no cover
@@ -1,6 +1,7 @@
"""Routes for getting entity content."""
import tempfile
import uuid
from pathlib import Path
from typing import Annotated, Union
@@ -218,7 +219,9 @@ async def write_resource(
status_code = 200
else:
# Create a new entity model
# Explicitly set external_id to ensure NOT NULL constraint is satisfied (fixes #512)
entity = EntityModel(
external_id=str(uuid.uuid4()),
title=file_name,
entity_type=entity_type,
content_type=content_type,
@@ -285,7 +285,7 @@ async def edit_entity_by_id(
# Verify entity exists
entity = await entity_repository.get_by_external_id(entity_id)
if not entity:
if not entity: # pragma: no cover
raise HTTPException(
status_code=404, detail=f"Entity with external_id '{entity_id}' not found"
)
@@ -394,7 +394,7 @@ async def move_entity(
try:
# First, get the entity by external_id to verify it exists
entity = await entity_repository.get_by_external_id(entity_id)
if not entity:
if not entity: # pragma: no cover
raise HTTPException(
status_code=404, detail=f"Entity with external_id '{entity_id}' not found"
)
@@ -414,9 +414,7 @@ async def move_entity(
result = EntityResponseV2.model_validate(moved_entity)
logger.info(
f"API v2 response: moved external_id={entity_id} to '{data.destination_path}'"
)
logger.info(f"API v2 response: moved external_id={entity_id} to '{data.destination_path}'")
return result
@@ -202,7 +202,7 @@ async def update_project_by_id(
# Get updated project info (use the same external_id)
updated_project = await project_repository.get_by_external_id(project_id)
if not updated_project:
if not updated_project: # pragma: no cover
raise HTTPException(
status_code=404,
detail=f"Project with external_id '{project_id}' not found after update",
@@ -264,9 +264,7 @@ async def delete_project_by_id(
# Use is_default from database, not ConfigManager (which doesn't work in cloud mode)
if old_project.is_default:
available_projects = await project_service.list_projects()
other_projects = [
p.name for p in available_projects if p.external_id != project_id
]
other_projects = [p.name for p in available_projects if p.external_id != project_id]
detail = f"Cannot delete default project '{old_project.name}'. "
if other_projects:
detail += ( # pragma: no cover
@@ -9,6 +9,7 @@ Key differences from v1:
- More RESTful: POST for create, PUT for update, GET for read
"""
import uuid
from pathlib import Path as PathLib
from fastapi import APIRouter, HTTPException, Response, Path
@@ -147,7 +148,9 @@ async def create_resource(
entity_type = "canvas" if data.file_path.endswith(".canvas") else "file"
# Create a new entity model
# Explicitly set external_id to ensure NOT NULL constraint is satisfied (fixes #512)
entity = EntityModel(
external_id=str(uuid.uuid4()),
title=file_name,
entity_type=entity_type,
content_type=content_type,
+10 -18
View File
@@ -1,9 +1,3 @@
# Suppress Logfire "not configured" warning - we only use Logfire in cloud/server contexts
import os
os.environ.setdefault("LOGFIRE_IGNORE_NO_CONFIG", "1")
# Remove loguru's default handler IMMEDIATELY, before any other imports.
# This prevents DEBUG logs from appearing on stdout during module-level
# initialization (e.g., template_loader.TemplateLoader() logs at DEBUG level).
from loguru import logger
@@ -14,8 +8,8 @@ from typing import Optional # noqa: E402
import typer # noqa: E402
from basic_memory.config import ConfigManager, init_cli_logging # noqa: E402
from basic_memory.telemetry import show_notice_if_needed, track_app_started # noqa: E402
from basic_memory.cli.container import CliContainer, set_container # noqa: E402
from basic_memory.config import init_cli_logging # noqa: E402
def version_callback(value: bool) -> None:
@@ -47,26 +41,24 @@ def app_callback(
# Initialize logging for CLI (file only, no stdout)
init_cli_logging()
# Show telemetry notice and track CLI startup
# Skip for 'mcp' command - it handles its own telemetry in lifespan
# Skip for 'telemetry' command - avoid issues when user is managing telemetry
if ctx.invoked_subcommand not in {"mcp", "telemetry"}:
show_notice_if_needed()
track_app_started("cli")
# --- Composition Root ---
# Create container and read config (single point of config access)
container = CliContainer.create()
set_container(container)
# Run initialization for commands that don't use the API
# Skip for 'mcp' command - it has its own lifespan that handles initialization
# Skip for API-using commands (status, sync, etc.) - they handle initialization via deps.py
api_commands = {"mcp", "status", "sync", "project", "tool"}
# Skip for 'reset' command - it manages its own database lifecycle
skip_init_commands = {"mcp", "status", "sync", "project", "tool", "reset"}
if (
not version
and ctx.invoked_subcommand is not None
and ctx.invoked_subcommand not in api_commands
and ctx.invoked_subcommand not in skip_init_commands
):
from basic_memory.services.initialization import ensure_initialization
app_config = ConfigManager().config
ensure_initialization(app_config)
ensure_initialization(container.config)
## import
+1 -2
View File
@@ -1,7 +1,7 @@
"""CLI commands for basic-memory."""
from . import status, db, import_memory_json, mcp, import_claude_conversations
from . import import_claude_projects, import_chatgpt, tool, project, format, telemetry
from . import import_claude_projects, import_chatgpt, tool, project, format
__all__ = [
"status",
@@ -14,5 +14,4 @@ __all__ = [
"tool",
"project",
"format",
"telemetry",
]
@@ -1,6 +1,16 @@
"""Cloud commands package."""
from basic_memory.cli.app import cloud_app
# Import all commands to register them with typer
from basic_memory.cli.commands.cloud.core_commands import * # noqa: F401,F403
from basic_memory.cli.commands.cloud.api_client import get_authenticated_headers, get_cloud_config # noqa: F401
from basic_memory.cli.commands.cloud.upload_command import * # noqa: F401,F403
# Register snapshot sub-command group
from basic_memory.cli.commands.cloud.snapshot import snapshot_app
cloud_app.add_typer(snapshot_app, name="snapshot")
# Register restore command (directly on cloud_app via decorator)
from basic_memory.cli.commands.cloud.restore import restore # noqa: F401, E402
@@ -1,11 +1,10 @@
"""Core cloud commands for Basic Memory CLI."""
import asyncio
import typer
from rich.console import Console
from basic_memory.cli.app import cloud_app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.cli.auth import CLIAuth
from basic_memory.config import ConfigManager
from basic_memory.cli.commands.cloud.api_client import (
@@ -64,7 +63,7 @@ def login():
)
raise typer.Exit(1)
asyncio.run(_login())
run_with_cleanup(_login())
@cloud_app.command()
@@ -110,7 +109,7 @@ def status() -> None:
console.print("\n[blue]Checking cloud instance health...[/blue]")
# Make API request to check health
response = asyncio.run(
response = run_with_cleanup(
make_api_request(method="GET", url=f"{host_url}/proxy/health", headers=headers)
)
@@ -140,7 +139,6 @@ def status() -> None:
def setup() -> None:
"""Set up cloud sync by installing rclone and configuring credentials.
SPEC-20: Simplified to project-scoped workflow.
After setup, use project commands for syncing:
bm project add <name> <path> --local-path ~/projects/<name>
bm project bisync --name <name> --resync # First time
@@ -156,12 +154,12 @@ def setup() -> None:
# Step 2: Get tenant info
console.print("\n[blue]Step 2: Getting tenant information...[/blue]")
tenant_info = asyncio.run(get_mount_info())
tenant_info = run_with_cleanup(get_mount_info())
console.print(f"[green]Found tenant: {tenant_info.tenant_id}[/green]")
# Step 3: Generate credentials
console.print("\n[blue]Step 3: Generating sync credentials...[/blue]")
creds = asyncio.run(generate_mount_credentials(tenant_info.tenant_id))
creds = run_with_cleanup(generate_mount_credentials(tenant_info.tenant_id))
console.print("[green]Generated secure credentials[/green]")
# Step 4: Configure rclone remote
@@ -27,6 +27,7 @@ console = Console()
# Minimum rclone version for --create-empty-src-dirs support
MIN_RCLONE_VERSION_EMPTY_DIRS = (1, 64, 0)
class RunResult(Protocol):
returncode: int
stdout: str
@@ -0,0 +1,159 @@
"""Restore CLI commands for Basic Memory Cloud.
SPEC-29 Phase 3: CLI commands for restoring files from Tigris bucket snapshots.
"""
import asyncio
import typer
from rich.console import Console
from basic_memory.cli.app import cloud_app
from basic_memory.cli.commands.cloud.api_client import (
CloudAPIError,
SubscriptionRequiredError,
make_api_request,
)
from basic_memory.cli.commands.cloud.schemas import BucketSnapshotBrowseResponse
from basic_memory.config import ConfigManager
console = Console()
@cloud_app.command("restore")
def restore(
path: str = typer.Argument(
...,
help="Path to restore (file or folder, e.g., 'notes/project.md' or 'research/')",
),
snapshot_id: str = typer.Option(
...,
"--snapshot",
"-s",
help="ID of the snapshot to restore from",
),
force: bool = typer.Option(
False,
"--force",
"-f",
help="Skip confirmation prompt",
),
) -> None:
"""Restore a file or folder from a snapshot.
This command restores files from a previous snapshot to the current bucket.
The restored files will overwrite any existing files at the same path.
Examples:
bm cloud restore notes/project.md --snapshot abc123
bm cloud restore research/ --snapshot abc123
bm cloud restore notes/project.md --snapshot abc123 --force
"""
async def _restore():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
# Normalize path - remove leading slash if present
normalized_path = path.lstrip("/")
if not force:
# Show what will be restored
console.print(f"[blue]Preparing to restore from snapshot {snapshot_id}[/blue]")
console.print(f" Path: {normalized_path}")
# Try to browse the snapshot to show what files will be affected
try:
browse_url = f"{host_url}/api/bucket-snapshots/{snapshot_id}/browse"
if normalized_path:
browse_url += f"?prefix={normalized_path}"
response = await make_api_request(
method="GET",
url=browse_url,
)
browse_response = BucketSnapshotBrowseResponse.model_validate(response.json())
if browse_response.files:
if len(browse_response.files) <= 10:
console.print("\n Files to restore:")
for file_info in browse_response.files:
console.print(f" - {file_info.key}")
else:
console.print(
f"\n {len(browse_response.files)} files will be restored"
)
console.print(" First 5 files:")
for file_info in browse_response.files[:5]:
console.print(f" - {file_info.key}")
console.print(f" ... and {len(browse_response.files) - 5} more")
else:
console.print(
f"\n[yellow]No files found matching '{normalized_path}' "
f"in snapshot[/yellow]"
)
raise typer.Exit(0)
except CloudAPIError as browse_error:
if browse_error.status_code == 404:
console.print(f"[red]Snapshot not found: {snapshot_id}[/red]")
raise typer.Exit(1)
# If browse fails for other reasons, proceed with confirmation anyway
pass
console.print(
"\n[yellow]Warning: Restored files will overwrite existing files![/yellow]"
)
confirmed = typer.confirm("\nProceed with restore?")
if not confirmed:
console.print("[yellow]Restore cancelled[/yellow]")
raise typer.Exit(0)
console.print(f"[blue]Restoring from snapshot {snapshot_id}...[/blue]")
response = await make_api_request(
method="POST",
url=f"{host_url}/api/bucket-snapshots/{snapshot_id}/restore",
json_data={"path": normalized_path},
)
data = response.json()
restored_files = data.get("restored", [])
returned_snapshot_id = data.get("snapshot_id", snapshot_id)
if restored_files:
console.print(f"[green]Successfully restored {len(restored_files)} file(s)[/green]")
if len(restored_files) <= 10:
for file_path in restored_files:
console.print(f" - {file_path}")
else:
console.print(" First 5 restored files:")
for file_path in restored_files[:5]:
console.print(f" - {file_path}")
console.print(f" ... and {len(restored_files) - 5} more")
console.print(f"\n[dim]Snapshot ID: {returned_snapshot_id}[/dim]")
else:
console.print("[yellow]No files were restored[/yellow]")
console.print(f"[dim]No files matching '{normalized_path}' found in snapshot[/dim]")
except typer.Exit:
# Re-raise typer.Exit without modification - it's used for clean exits
raise
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
if e.status_code == 404:
console.print(f"[red]Snapshot not found: {snapshot_id}[/red]")
else:
console.print(f"[red]Failed to restore: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_restore())
@@ -0,0 +1,55 @@
"""Pydantic schemas for Basic Memory Cloud API responses.
These schemas mirror the API response models from basic-memory-cloud
for type-safe parsing of API responses in CLI commands.
"""
from datetime import datetime
from uuid import UUID
from pydantic import BaseModel
class BucketSnapshotFileInfo(BaseModel):
"""File info from snapshot browse response."""
key: str
size: int
last_modified: datetime
etag: str | None = None
class BucketSnapshotBrowseResponse(BaseModel):
"""Response from browsing snapshot contents."""
files: list[BucketSnapshotFileInfo]
prefix: str
snapshot_version: str
class BucketSnapshotResponse(BaseModel):
"""Response model for bucket snapshot data."""
id: UUID
bucket_name: str
snapshot_version: str
name: str
description: str | None
auto: bool
created_at: datetime
created_by: UUID | None = None
class BucketSnapshotListResponse(BaseModel):
"""Response from listing bucket snapshots."""
snapshots: list[BucketSnapshotResponse]
total: int
class BucketSnapshotRestoreResponse(BaseModel):
"""Response from restore operation."""
restored: list[str]
snapshot_version: str
snapshot_id: UUID
@@ -0,0 +1,370 @@
"""Snapshot CLI commands for Basic Memory Cloud.
SPEC-29 Phase 3: CLI commands for managing Tigris bucket snapshots.
"""
import asyncio
from datetime import datetime
from typing import Optional
import typer
from rich.console import Console
from rich.table import Table
from basic_memory.cli.commands.cloud.api_client import (
CloudAPIError,
SubscriptionRequiredError,
make_api_request,
)
from basic_memory.cli.commands.cloud.schemas import BucketSnapshotBrowseResponse
from basic_memory.config import ConfigManager
console = Console()
snapshot_app = typer.Typer(help="Manage bucket snapshots")
def _format_timestamp(iso_timestamp: str) -> str:
"""Format ISO timestamp to a human-readable format."""
try:
dt = datetime.fromisoformat(iso_timestamp.replace("Z", "+00:00"))
return dt.strftime("%Y-%m-%d %H:%M:%S")
except (ValueError, AttributeError):
return iso_timestamp
@snapshot_app.command("create")
def create(
description: str = typer.Argument(
...,
help="Description for the snapshot",
),
) -> None:
"""Create a new bucket snapshot.
Examples:
bm cloud snapshot create "before major refactor"
bm cloud snapshot create "daily backup"
"""
async def _create():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
console.print("[blue]Creating snapshot...[/blue]")
response = await make_api_request(
method="POST",
url=f"{host_url}/api/bucket-snapshots",
json_data={"description": description},
)
data = response.json()
snapshot_id = data.get("id", "unknown")
snapshot_version = data.get("snapshot_version", "unknown")
created_at = _format_timestamp(data.get("created_at", ""))
console.print("[green]Snapshot created successfully[/green]")
console.print(f" ID: {snapshot_id}")
console.print(f" Version: {snapshot_version}")
console.print(f" Created: {created_at}")
console.print(f" Description: {description}")
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
console.print(f"[red]Failed to create snapshot: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_create())
@snapshot_app.command("list")
def list_snapshots(
limit: int = typer.Option(
10,
"--limit",
"-l",
help="Maximum number of snapshots to display",
),
) -> None:
"""List all bucket snapshots.
Examples:
bm cloud snapshot list
bm cloud snapshot list --limit 20
"""
async def _list():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
console.print("[blue]Fetching snapshots...[/blue]")
response = await make_api_request(
method="GET",
url=f"{host_url}/api/bucket-snapshots",
)
data = response.json()
snapshots = data.get("snapshots", [])
total = data.get("total", len(snapshots))
if not snapshots:
console.print("[yellow]No snapshots found[/yellow]")
console.print(
'\n[dim]Create a snapshot with: bm cloud snapshot create "description"[/dim]'
)
return
# Create a table for displaying snapshots
table = Table(title=f"Bucket Snapshots ({total} total)")
table.add_column("ID", style="cyan", no_wrap=True)
table.add_column("Description", style="white")
table.add_column("Auto", style="dim")
table.add_column("Created", style="green")
for snapshot in snapshots[:limit]:
snapshot_id = snapshot.get("id", "unknown")
desc = snapshot.get("description") or snapshot.get("name", "-")
auto = "yes" if snapshot.get("auto", False) else "no"
created_at = _format_timestamp(snapshot.get("created_at", ""))
table.add_row(snapshot_id, desc, auto, created_at)
console.print(table)
if total > limit:
console.print(
f"\n[dim]Showing {limit} of {total} snapshots. Use --limit to see more.[/dim]"
)
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
console.print(f"[red]Failed to list snapshots: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_list())
@snapshot_app.command("delete")
def delete(
snapshot_id: str = typer.Argument(
...,
help="The ID of the snapshot to delete",
),
force: bool = typer.Option(
False,
"--force",
"-f",
help="Skip confirmation prompt",
),
) -> None:
"""Delete a bucket snapshot.
Examples:
bm cloud snapshot delete abc123
bm cloud snapshot delete abc123 --force
"""
async def _delete():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
if not force:
# Fetch snapshot details first to show what will be deleted
console.print("[blue]Fetching snapshot details...[/blue]")
try:
response = await make_api_request(
method="GET",
url=f"{host_url}/api/bucket-snapshots/{snapshot_id}",
)
data = response.json()
desc = data.get("description") or data.get("name", "unnamed")
created_at = _format_timestamp(data.get("created_at", ""))
console.print("\nSnapshot to delete:")
console.print(f" ID: {snapshot_id}")
console.print(f" Description: {desc}")
console.print(f" Created: {created_at}")
except CloudAPIError:
# If we can't fetch details, proceed with confirmation anyway
pass
confirmed = typer.confirm("\nAre you sure you want to delete this snapshot?")
if not confirmed:
console.print("[yellow]Deletion cancelled[/yellow]")
raise typer.Exit(0)
console.print("[blue]Deleting snapshot...[/blue]")
await make_api_request(
method="DELETE",
url=f"{host_url}/api/bucket-snapshots/{snapshot_id}",
)
console.print(f"[green]Snapshot {snapshot_id} deleted successfully[/green]")
except typer.Exit:
# Re-raise typer.Exit without modification - it's used for clean exits
raise
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
if e.status_code == 404:
console.print(f"[red]Snapshot not found: {snapshot_id}[/red]")
else:
console.print(f"[red]Failed to delete snapshot: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_delete())
@snapshot_app.command("show")
def show(
snapshot_id: str = typer.Argument(
...,
help="The ID of the snapshot to show",
),
) -> None:
"""Show details of a specific snapshot.
Examples:
bm cloud snapshot show abc123
"""
async def _show():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
response = await make_api_request(
method="GET",
url=f"{host_url}/api/bucket-snapshots/{snapshot_id}",
)
data = response.json()
console.print("[bold blue]Snapshot Details[/bold blue]")
console.print(f" ID: {data.get('id', 'unknown')}")
console.print(f" Bucket: {data.get('bucket_name', 'unknown')}")
console.print(f" Version: {data.get('snapshot_version', 'unknown')}")
console.print(f" Name: {data.get('name', '-')}")
console.print(f" Description: {data.get('description') or '-'}")
console.print(f" Auto: {'yes' if data.get('auto', False) else 'no'}")
console.print(f" Created: {_format_timestamp(data.get('created_at', ''))}")
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
if e.status_code == 404:
console.print(f"[red]Snapshot not found: {snapshot_id}[/red]")
else:
console.print(f"[red]Failed to get snapshot details: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_show())
@snapshot_app.command("browse")
def browse(
snapshot_id: str = typer.Argument(
...,
help="The ID of the snapshot to browse",
),
prefix: Optional[str] = typer.Option(
None,
"--prefix",
"-p",
help="Filter files by path prefix (e.g., 'notes/')",
),
) -> None:
"""Browse contents of a snapshot.
Examples:
bm cloud snapshot browse abc123
bm cloud snapshot browse abc123 --prefix notes/
"""
async def _browse():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
url = f"{host_url}/api/bucket-snapshots/{snapshot_id}/browse"
if prefix:
url += f"?prefix={prefix}"
response = await make_api_request(
method="GET",
url=url,
)
browse_response = BucketSnapshotBrowseResponse.model_validate(response.json())
if not browse_response.files:
if prefix:
console.print(f"[yellow]No files found with prefix '{prefix}'[/yellow]")
else:
console.print("[yellow]No files found in snapshot[/yellow]")
return
console.print(
f"[bold blue]Snapshot Contents ({len(browse_response.files)} files)[/bold blue]"
)
for file_info in browse_response.files:
size_kb = file_info.size // 1024
console.print(f" {file_info.key} ({size_kb} KB)")
console.print(
f"\n[dim]Use 'bm cloud restore <path> --snapshot {snapshot_id}' "
f"to restore files[/dim]"
)
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
if e.status_code == 404:
console.print(f"[red]Snapshot not found: {snapshot_id}[/red]")
else:
console.print(f"[red]Failed to browse snapshot: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_browse())
@@ -1,12 +1,12 @@
"""Upload CLI commands for basic-memory projects."""
import asyncio
from pathlib import Path
import typer
from rich.console import Console
from basic_memory.cli.app import cloud_app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.cli.commands.cloud.cloud_utils import (
create_cloud_project,
project_exists,
@@ -121,4 +121,4 @@ def upload(
console.print(f"[yellow]Warning: Sync failed: {e}[/yellow]")
console.print("[dim]Files uploaded but may not be indexed yet[/dim]")
asyncio.run(_upload())
run_with_cleanup(_upload())
+27 -5
View File
@@ -23,8 +23,8 @@ T = TypeVar("T")
def run_with_cleanup(coro: Coroutine[Any, Any, T]) -> T:
"""Run an async coroutine with proper database cleanup.
This helper ensures database connections are cleaned up before the event
loop closes, preventing process hangs in CLI commands.
This helper ensures database connections are cleaned up before the
event loop closes, preventing process hangs in CLI commands.
Args:
coro: The coroutine to run
@@ -42,23 +42,45 @@ def run_with_cleanup(coro: Coroutine[Any, Any, T]) -> T:
return asyncio.run(_with_cleanup())
async def run_sync(project: Optional[str] = None, force_full: bool = False):
async def run_sync(
project: Optional[str] = None,
force_full: bool = False,
run_in_background: bool = True,
):
"""Run sync operation via API endpoint.
Args:
project: Optional project name
force_full: If True, force a full scan bypassing watermark optimization
run_in_background: If True, return immediately; if False, wait for completion
"""
try:
async with get_client() as client:
project_item = await get_active_project(client, project, None)
url = f"{project_item.project_url}/project/sync"
params = []
if force_full:
url += "?force_full=true"
params.append("force_full=true")
if not run_in_background:
params.append("run_in_background=false")
if params:
url += "?" + "&".join(params)
response = await call_post(client, url)
data = response.json()
console.print(f"[green]{data['message']}[/green]")
# Background mode returns {"message": "..."}, foreground returns SyncReportResponse
if "message" in data:
console.print(f"[green]{data['message']}[/green]")
else:
# Foreground mode - show summary of sync results
total = data.get("total", 0)
new_count = len(data.get("new", []))
modified_count = len(data.get("modified", []))
deleted_count = len(data.get("deleted", []))
console.print(
f"[green]Synced {total} files[/green] "
f"(new: {new_count}, modified: {modified_count}, deleted: {deleted_count})"
)
except (ToolError, ValueError) as e:
console.print(f"[red]Sync failed: {e}[/red]")
raise typer.Exit(1)
+81 -20
View File
@@ -1,13 +1,50 @@
"""Database management commands."""
import asyncio
from pathlib import Path
import typer
from loguru import logger
from rich.console import Console
from sqlalchemy.exc import OperationalError
from basic_memory import db
from basic_memory.cli.app import app
from basic_memory.config import ConfigManager, BasicMemoryConfig, save_basic_memory_config
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.config import ConfigManager
from basic_memory.repository import ProjectRepository
from basic_memory.services.initialization import reconcile_projects_with_config
from basic_memory.sync.sync_service import get_sync_service
console = Console()
async def _reindex_projects(app_config):
"""Reindex all projects in a single async context.
This ensures all database operations use the same event loop,
and proper cleanup happens when the function completes.
"""
try:
await reconcile_projects_with_config(app_config)
# Get database session (migrations already run if needed)
_, session_maker = await db.get_or_create_db(
db_path=app_config.database_path,
db_type=db.DatabaseType.FILESYSTEM,
)
project_repository = ProjectRepository(session_maker)
projects = await project_repository.get_active_projects()
for project in projects:
console.print(f" Indexing [cyan]{project.name}[/cyan]...")
logger.info(f"Starting sync for project: {project.name}")
sync_service = await get_sync_service(project)
sync_dir = Path(project.path)
await sync_service.sync(sync_dir, project_name=project.name)
logger.info(f"Sync completed for project: {project.name}")
finally:
# Clean up database connections before event loop closes
await db.shutdown_db()
@app.command()
@@ -15,30 +52,54 @@ def reset(
reindex: bool = typer.Option(False, "--reindex", help="Rebuild db index from filesystem"),
): # pragma: no cover
"""Reset database (drop all tables and recreate)."""
if typer.confirm("This will delete all data in your db. Are you sure?"):
console.print(
"[yellow]Note:[/yellow] This only deletes the index database. "
"Your markdown note files will not be affected.\n"
"Use [green]bm reset --reindex[/green] to automatically rebuild the index afterward."
)
if typer.confirm("Reset the database index?"):
logger.info("Resetting database...")
config_manager = ConfigManager()
app_config = config_manager.config
# Get database path
db_path = app_config.app_database_path
# Delete the database file if it exists
if db_path.exists():
db_path.unlink()
logger.info(f"Database file deleted: {db_path}")
# Delete the database file and WAL files if they exist
for suffix in ["", "-shm", "-wal"]:
path = db_path.parent / f"{db_path.name}{suffix}"
if path.exists():
try:
path.unlink()
logger.info(f"Deleted: {path}")
except OSError as e:
console.print(
f"[red]Error:[/red] Cannot delete {path.name}: {e}\n"
"The database may be in use by another process (e.g., MCP server).\n"
"Please close Claude Desktop or any other Basic Memory clients and try again."
)
raise typer.Exit(1)
# Reset project configuration
config = BasicMemoryConfig()
save_basic_memory_config(config_manager.config_file, config)
logger.info("Project configuration reset to default")
# Create a new empty database
asyncio.run(db.run_migrations(app_config))
logger.info("Database reset complete")
# Create a new empty database (preserves project configuration)
try:
run_with_cleanup(db.run_migrations(app_config))
except OperationalError as e:
if "disk I/O error" in str(e) or "database is locked" in str(e):
console.print(
"[red]Error:[/red] Cannot access database. "
"It may be in use by another process (e.g., MCP server).\n"
"Please close Claude Desktop or any other Basic Memory clients and try again."
)
raise typer.Exit(1)
raise
console.print("[green]Database reset complete[/green]")
if reindex:
# Run database sync directly
from basic_memory.cli.commands.command_utils import run_sync
logger.info("Rebuilding search index from filesystem...")
asyncio.run(run_sync(project=None))
projects = list(app_config.projects)
if not projects:
console.print("[yellow]No projects configured. Skipping reindex.[/yellow]")
else:
console.print(f"Rebuilding search index for {len(projects)} project(s)...")
# Note: _reindex_projects has its own cleanup, but run_with_cleanup
# ensures db.shutdown_db() is called even if _reindex_projects changes
run_with_cleanup(_reindex_projects(app_config))
console.print("[green]Reindex complete[/green]")
+2 -2
View File
@@ -1,6 +1,5 @@
"""Format command for basic-memory CLI."""
import asyncio
from pathlib import Path
from typing import Annotated, Optional
@@ -10,6 +9,7 @@ from rich.console import Console
from rich.progress import Progress, SpinnerColumn, TextColumn
from basic_memory.cli.app import app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.config import ConfigManager, get_project_config
from basic_memory.file_utils import format_file
@@ -189,7 +189,7 @@ def format(
basic-memory format notes/ # Format all files in directory
"""
try:
asyncio.run(run_format(path, project))
run_with_cleanup(run_format(path, project))
except Exception as e:
if not isinstance(e, typer.Exit):
logger.error(f"Error formatting files: {e}")
@@ -1,12 +1,12 @@
"""Import command for ChatGPT conversations."""
import asyncio
import json
from pathlib import Path
from typing import Annotated, Tuple
import typer
from basic_memory.cli.app import import_app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.config import ConfigManager, get_project_config
from basic_memory.importers import ChatGPTImporter
from basic_memory.markdown import EntityParser, MarkdownProcessor
@@ -53,7 +53,7 @@ def import_chatgpt(
raise typer.Exit(1)
# Get importer dependencies
markdown_processor, file_service = asyncio.run(get_importer_dependencies())
markdown_processor, file_service = run_with_cleanup(get_importer_dependencies())
config = get_project_config()
# Process the file
base_path = config.home / folder
@@ -63,7 +63,7 @@ def import_chatgpt(
importer = ChatGPTImporter(config.home, markdown_processor, file_service)
with conversations_json.open("r", encoding="utf-8") as file:
json_data = json.load(file)
result = asyncio.run(importer.import_data(json_data, folder))
result = run_with_cleanup(importer.import_data(json_data, folder))
if not result.success: # pragma: no cover
typer.echo(f"Error during import: {result.error_message}", err=True)
@@ -1,12 +1,12 @@
"""Import command for basic-memory CLI to import chat data from conversations2.json format."""
import asyncio
import json
from pathlib import Path
from typing import Annotated, Tuple
import typer
from basic_memory.cli.app import claude_app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.config import ConfigManager, get_project_config
from basic_memory.importers.claude_conversations_importer import ClaudeConversationsImporter
from basic_memory.markdown import EntityParser, MarkdownProcessor
@@ -54,7 +54,7 @@ def import_claude(
raise typer.Exit(1)
# Get importer dependencies
markdown_processor, file_service = asyncio.run(get_importer_dependencies())
markdown_processor, file_service = run_with_cleanup(get_importer_dependencies())
# Create the importer
importer = ClaudeConversationsImporter(config.home, markdown_processor, file_service)
@@ -66,7 +66,7 @@ def import_claude(
# Run the import
with conversations_json.open("r", encoding="utf-8") as file:
json_data = json.load(file)
result = asyncio.run(importer.import_data(json_data, folder))
result = run_with_cleanup(importer.import_data(json_data, folder))
if not result.success: # pragma: no cover
typer.echo(f"Error during import: {result.error_message}", err=True)
@@ -1,12 +1,12 @@
"""Import command for basic-memory CLI to import project data from Claude.ai."""
import asyncio
import json
from pathlib import Path
from typing import Annotated, Tuple
import typer
from basic_memory.cli.app import claude_app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.config import ConfigManager, get_project_config
from basic_memory.importers.claude_projects_importer import ClaudeProjectsImporter
from basic_memory.markdown import EntityParser, MarkdownProcessor
@@ -53,7 +53,7 @@ def import_projects(
raise typer.Exit(1)
# Get importer dependencies
markdown_processor, file_service = asyncio.run(get_importer_dependencies())
markdown_processor, file_service = run_with_cleanup(get_importer_dependencies())
# Create the importer
importer = ClaudeProjectsImporter(config.home, markdown_processor, file_service)
@@ -65,7 +65,7 @@ def import_projects(
# Run the import
with projects_json.open("r", encoding="utf-8") as file:
json_data = json.load(file)
result = asyncio.run(importer.import_data(json_data, base_folder))
result = run_with_cleanup(importer.import_data(json_data, base_folder))
if not result.success: # pragma: no cover
typer.echo(f"Error during import: {result.error_message}", err=True)
@@ -1,12 +1,12 @@
"""Import command for basic-memory CLI to import from JSON memory format."""
import asyncio
import json
from pathlib import Path
from typing import Annotated, Tuple
import typer
from basic_memory.cli.app import import_app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.config import ConfigManager, get_project_config
from basic_memory.importers.memory_json_importer import MemoryJsonImporter
from basic_memory.markdown import EntityParser, MarkdownProcessor
@@ -52,7 +52,7 @@ def memory_json(
config = get_project_config()
try:
# Get importer dependencies
markdown_processor, file_service = asyncio.run(get_importer_dependencies())
markdown_processor, file_service = run_with_cleanup(get_importer_dependencies())
# Create the importer
importer = MemoryJsonImporter(config.home, markdown_processor, file_service)
@@ -67,7 +67,7 @@ def memory_json(
for line in file:
json_data = json.loads(line)
file_data.append(json_data)
result = asyncio.run(importer.import_data(file_data, destination_folder))
result = run_with_cleanup(importer.import_data(file_data, destination_folder))
if not result.success: # pragma: no cover
typer.echo(f"Error during import: {result.error_message}", err=True)
+19 -20
View File
@@ -1,6 +1,5 @@
"""Command module for basic-memory project management."""
import asyncio
import os
from pathlib import Path
@@ -9,7 +8,7 @@ from rich.console import Console
from rich.table import Table
from basic_memory.cli.app import app
from basic_memory.cli.commands.command_utils import get_project_info
from basic_memory.cli.commands.command_utils import get_project_info, run_with_cleanup
from basic_memory.config import ConfigManager
import json
from datetime import datetime
@@ -56,7 +55,7 @@ def list_projects() -> None:
return ProjectList.model_validate(response.json())
try:
result = asyncio.run(_list_projects())
result = run_with_cleanup(_list_projects())
config = ConfigManager().config
table = Table(title="Basic Memory Projects")
@@ -155,7 +154,7 @@ def add_project(
return ProjectStatusResponse.model_validate(response.json())
try:
result = asyncio.run(_add_project())
result = run_with_cleanup(_add_project())
console.print(f"[green]{result.message}[/green]")
# Save local sync path to config if in cloud mode
@@ -212,7 +211,7 @@ def setup_project_sync(
try:
# Verify project exists on cloud
asyncio.run(_verify_project_exists())
run_with_cleanup(_verify_project_exists())
# Resolve and create local path
resolved_path = Path(os.path.abspath(os.path.expanduser(local_path)))
@@ -279,7 +278,7 @@ def remove_project(
has_bisync_state = bisync_state_path.exists()
# Remove project from cloud/API
result = asyncio.run(_remove_project())
result = run_with_cleanup(_remove_project())
console.print(f"[green]{result.message}[/green]")
# Clean up local sync directory if it exists and delete_notes is True
@@ -347,7 +346,7 @@ def set_default_project(
return ProjectStatusResponse.model_validate(response.json())
try:
result = asyncio.run(_set_default())
result = run_with_cleanup(_set_default())
console.print(f"[green]{result.message}[/green]")
except Exception as e:
console.print(f"[red]Error setting default project: {str(e)}[/red]")
@@ -372,7 +371,7 @@ def synchronize_projects() -> None:
return ProjectStatusResponse.model_validate(response.json())
try:
result = asyncio.run(_sync_config())
result = run_with_cleanup(_sync_config())
console.print(f"[green]{result.message}[/green]")
except Exception as e: # pragma: no cover
console.print(f"[red]Error synchronizing projects: {str(e)}[/red]")
@@ -407,7 +406,7 @@ def move_project(
return ProjectStatusResponse.model_validate(response.json())
try:
result = asyncio.run(_move_project())
result = run_with_cleanup(_move_project())
console.print(f"[green]{result.message}[/green]")
# Show important file movement reminder
@@ -448,7 +447,7 @@ def sync_project_command(
try:
# Get tenant info for bucket name
tenant_info = asyncio.run(get_mount_info())
tenant_info = run_with_cleanup(get_mount_info())
bucket_name = tenant_info.bucket_name
# Get project info
@@ -461,7 +460,7 @@ def sync_project_command(
return proj
return None
project_data = asyncio.run(_get_project())
project_data = run_with_cleanup(_get_project())
if not project_data:
console.print(f"[red]Error: Project '{name}' not found[/red]")
raise typer.Exit(1)
@@ -502,7 +501,7 @@ def sync_project_command(
return response.json()
try:
result = asyncio.run(_trigger_db_sync())
result = run_with_cleanup(_trigger_db_sync())
console.print(f"[dim]Database sync initiated: {result.get('message')}[/dim]")
except Exception as e:
console.print(f"[yellow]Warning: Could not trigger database sync: {e}[/yellow]")
@@ -539,7 +538,7 @@ def bisync_project_command(
try:
# Get tenant info for bucket name
tenant_info = asyncio.run(get_mount_info())
tenant_info = run_with_cleanup(get_mount_info())
bucket_name = tenant_info.bucket_name
# Get project info
@@ -552,7 +551,7 @@ def bisync_project_command(
return proj
return None
project_data = asyncio.run(_get_project())
project_data = run_with_cleanup(_get_project())
if not project_data:
console.print(f"[red]Error: Project '{name}' not found[/red]")
raise typer.Exit(1)
@@ -600,7 +599,7 @@ def bisync_project_command(
return response.json()
try:
result = asyncio.run(_trigger_db_sync())
result = run_with_cleanup(_trigger_db_sync())
console.print(f"[dim]Database sync initiated: {result.get('message')}[/dim]")
except Exception as e:
console.print(f"[yellow]Warning: Could not trigger database sync: {e}[/yellow]")
@@ -633,7 +632,7 @@ def check_project_command(
try:
# Get tenant info for bucket name
tenant_info = asyncio.run(get_mount_info())
tenant_info = run_with_cleanup(get_mount_info())
bucket_name = tenant_info.bucket_name
# Get project info
@@ -646,7 +645,7 @@ def check_project_command(
return proj
return None
project_data = asyncio.run(_get_project())
project_data = run_with_cleanup(_get_project())
if not project_data:
console.print(f"[red]Error: Project '{name}' not found[/red]")
raise typer.Exit(1)
@@ -734,7 +733,7 @@ def ls_project_command(
try:
# Get tenant info for bucket name
tenant_info = asyncio.run(get_mount_info())
tenant_info = run_with_cleanup(get_mount_info())
bucket_name = tenant_info.bucket_name
# Get project info
@@ -747,7 +746,7 @@ def ls_project_command(
return proj
return None
project_data = asyncio.run(_get_project())
project_data = run_with_cleanup(_get_project())
if not project_data:
console.print(f"[red]Error: Project '{name}' not found[/red]")
raise typer.Exit(1)
@@ -784,7 +783,7 @@ def display_project_info(
"""Display detailed information and statistics about the current project."""
try:
# Get project info
info = asyncio.run(get_project_info(name))
info = run_with_cleanup(get_project_info(name))
if json_output:
# Convert to JSON and print
@@ -1,81 +0,0 @@
"""Telemetry commands for basic-memory CLI."""
import typer
from rich.console import Console
from rich.panel import Panel
from basic_memory.cli.app import app
from basic_memory.config import ConfigManager
console = Console()
# Create telemetry subcommand group
telemetry_app = typer.Typer(help="Manage anonymous telemetry settings")
app.add_typer(telemetry_app, name="telemetry")
@telemetry_app.command("enable")
def enable() -> None:
"""Enable anonymous telemetry.
Telemetry helps improve Basic Memory by collecting anonymous usage data.
No personal data, note content, or file paths are ever collected.
"""
config_manager = ConfigManager()
config = config_manager.config
config.telemetry_enabled = True
config_manager.save_config(config)
console.print("[green]Telemetry enabled[/green]")
console.print("[dim]Thank you for helping improve Basic Memory![/dim]")
@telemetry_app.command("disable")
def disable() -> None:
"""Disable anonymous telemetry.
You can re-enable telemetry anytime with: bm telemetry enable
"""
config_manager = ConfigManager()
config = config_manager.config
config.telemetry_enabled = False
config_manager.save_config(config)
console.print("[yellow]Telemetry disabled[/yellow]")
@telemetry_app.command("status")
def status() -> None:
"""Show current telemetry status and what's collected."""
from basic_memory.telemetry import get_install_id, TELEMETRY_DOCS_URL
config = ConfigManager().config
status_text = (
"[green]enabled[/green]" if config.telemetry_enabled else "[yellow]disabled[/yellow]"
)
console.print(f"\nTelemetry: {status_text}")
console.print(f"Install ID: [dim]{get_install_id()}[/dim]")
console.print()
what_we_collect = """
[bold]What we collect:[/bold]
- App version, Python version, OS, architecture
- Feature usage (which MCP tools and CLI commands)
- Sync statistics (entity count, duration)
- Error types (sanitized, no file paths)
[bold]What we NEVER collect:[/bold]
- Note content, file names, or paths
- Personal information
- IP addresses
"""
console.print(
Panel(
what_we_collect.strip(),
title="Telemetry Details",
border_style="blue",
expand=False,
)
)
console.print(f"[dim]Details: {TELEMETRY_DOCS_URL}[/dim]")
+7 -7
View File
@@ -1,6 +1,5 @@
"""CLI tool commands for Basic Memory."""
import asyncio
import sys
from typing import Annotated, List, Optional
@@ -9,6 +8,7 @@ from loguru import logger
from rich import print as rprint
from basic_memory.cli.app import app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.config import ConfigManager
# Import prompts
@@ -109,7 +109,7 @@ def write_note(
# use the project name, or the default from the config
project_name = project_name or config_manager.default_project
note = asyncio.run(mcp_write_note.fn(title, content, folder, project_name, tags))
note = run_with_cleanup(mcp_write_note.fn(title, content, folder, project_name, tags))
rprint(note)
except Exception as e: # pragma: no cover
if not isinstance(e, typer.Exit):
@@ -145,7 +145,7 @@ def read_note(
project_name = project_name or config_manager.default_project
try:
note = asyncio.run(mcp_read_note.fn(identifier, project_name, page, page_size))
note = run_with_cleanup(mcp_read_note.fn(identifier, project_name, page, page_size))
rprint(note)
except Exception as e: # pragma: no cover
if not isinstance(e, typer.Exit):
@@ -182,7 +182,7 @@ def build_context(
project_name = project_name or config_manager.default_project
try:
context = asyncio.run(
context = run_with_cleanup(
mcp_build_context.fn(
project=project_name,
url=url,
@@ -213,7 +213,7 @@ def recent_activity(
):
"""Get recent activity across the knowledge base."""
try:
result = asyncio.run(
result = run_with_cleanup(
mcp_recent_activity.fn(
type=type, # pyright: ignore [reportArgumentType]
depth=depth,
@@ -279,7 +279,7 @@ def search_notes(
search_type = ("title" if title else None,)
search_type = "text" if search_type is None else search_type
results = asyncio.run(
results = run_with_cleanup(
mcp_search.fn(
query,
project_name,
@@ -312,7 +312,7 @@ def continue_conversation(
"""Prompt to continue a previous conversation or work session."""
try:
# Prompt functions return formatted strings directly
session = asyncio.run(mcp_continue_conversation.fn(topic=topic, timeframe=timeframe)) # type: ignore
session = run_with_cleanup(mcp_continue_conversation.fn(topic=topic, timeframe=timeframe)) # type: ignore
rprint(session)
except Exception as e: # pragma: no cover
if not isinstance(e, typer.Exit):
+84
View File
@@ -0,0 +1,84 @@
"""CLI composition root for Basic Memory.
This container owns reading ConfigManager and environment variables for the
CLI entrypoint. Downstream modules receive config/dependencies explicitly
rather than reading globals.
Design principles:
- Only this module reads ConfigManager directly
- Runtime mode (cloud/local/test) is resolved here
- Different CLI commands may need different initialization
"""
from dataclasses import dataclass
from basic_memory.config import BasicMemoryConfig, ConfigManager
from basic_memory.runtime import RuntimeMode, resolve_runtime_mode
@dataclass
class CliContainer:
"""Composition root for the CLI entrypoint.
Holds resolved configuration and runtime context.
Created once at CLI startup, then used by subcommands.
"""
config: BasicMemoryConfig
mode: RuntimeMode
@classmethod
def create(cls) -> "CliContainer":
"""Create container by reading ConfigManager.
This is the single point where CLI reads global config.
"""
config = ConfigManager().config
mode = resolve_runtime_mode(
cloud_mode_enabled=config.cloud_mode_enabled,
is_test_env=config.is_test_env,
)
return cls(config=config, mode=mode)
# --- Runtime Mode Properties ---
@property
def is_cloud_mode(self) -> bool:
"""Whether running in cloud mode."""
return self.mode.is_cloud
# Module-level container instance (set by app callback)
_container: CliContainer | None = None
def get_container() -> CliContainer:
"""Get the current CLI container.
Returns:
The CLI container
Raises:
RuntimeError: If container hasn't been initialized
"""
if _container is None:
raise RuntimeError("CLI container not initialized. Call set_container() first.")
return _container
def set_container(container: CliContainer) -> None:
"""Set the CLI container (called by app callback)."""
global _container
_container = container
def get_or_create_container() -> CliContainer:
"""Get existing container or create new one.
This is useful for CLI commands that might be called before
the main app callback runs (e.g., eager options).
"""
global _container
if _container is None:
_container = CliContainer.create()
return _container
-1
View File
@@ -13,7 +13,6 @@ from basic_memory.cli.commands import ( # noqa: F401 # pragma: no cover
mcp,
project,
status,
telemetry,
tool,
)
+1 -12
View File
@@ -221,17 +221,6 @@ class BasicMemoryConfig(BaseSettings):
description="Cloud project sync configuration mapping project names to their local paths and sync state",
)
# Telemetry configuration (Homebrew-style opt-out)
telemetry_enabled: bool = Field(
default=True,
description="Send anonymous usage statistics to help improve Basic Memory. Disable with: bm telemetry disable",
)
telemetry_notice_shown: bool = Field(
default=False,
description="Whether the one-time telemetry notice has been shown to the user",
)
@property
def is_test_env(self) -> bool:
"""Check if running in a test environment.
@@ -241,7 +230,7 @@ class BasicMemoryConfig(BaseSettings):
- BASIC_MEMORY_ENV environment variable is "test"
- PYTEST_CURRENT_TEST environment variable is set (pytest is running)
Used to disable features like telemetry and file watchers during tests.
Used to disable features like file watchers during tests.
"""
return (
self.env == "test"
+31 -7
View File
@@ -242,18 +242,24 @@ def _create_postgres_engine(db_url: str, config: BasicMemoryConfig) -> AsyncEngi
def _create_engine_and_session(
db_path: Path, db_type: DatabaseType = DatabaseType.FILESYSTEM
db_path: Path,
db_type: DatabaseType = DatabaseType.FILESYSTEM,
config: Optional[BasicMemoryConfig] = None,
) -> tuple[AsyncEngine, async_sessionmaker[AsyncSession]]:
"""Internal helper to create engine and session maker.
Args:
db_path: Path to database file (used for SQLite, ignored for Postgres)
db_type: Type of database (MEMORY, FILESYSTEM, or POSTGRES)
config: Optional explicit config. If not provided, reads from ConfigManager.
Prefer passing explicitly from composition roots.
Returns:
Tuple of (engine, session_maker)
"""
config = ConfigManager().config
# Prefer explicit parameter; fall back to ConfigManager for backwards compatibility
if config is None:
config = ConfigManager().config
db_url = DatabaseType.get_db_url(db_path, db_type, config)
logger.debug(f"Creating engine for db_url: {db_url}")
@@ -272,17 +278,29 @@ async def get_or_create_db(
db_path: Path,
db_type: DatabaseType = DatabaseType.FILESYSTEM,
ensure_migrations: bool = True,
config: Optional[BasicMemoryConfig] = None,
) -> tuple[AsyncEngine, async_sessionmaker[AsyncSession]]: # pragma: no cover
"""Get or create database engine and session maker."""
"""Get or create database engine and session maker.
Args:
db_path: Path to database file
db_type: Type of database
ensure_migrations: Whether to run migrations
config: Optional explicit config. If not provided, reads from ConfigManager.
Prefer passing explicitly from composition roots.
"""
global _engine, _session_maker
# Prefer explicit parameter; fall back to ConfigManager for backwards compatibility
if config is None:
config = ConfigManager().config
if _engine is None:
_engine, _session_maker = _create_engine_and_session(db_path, db_type)
_engine, _session_maker = _create_engine_and_session(db_path, db_type, config)
# Run migrations automatically unless explicitly disabled
if ensure_migrations:
app_config = ConfigManager().config
await run_migrations(app_config, db_type)
await run_migrations(config, db_type)
# These checks should never fail since we just created the engine and session maker
# if they were None, but we'll check anyway for the type checker
@@ -311,17 +329,23 @@ async def shutdown_db() -> None: # pragma: no cover
async def engine_session_factory(
db_path: Path,
db_type: DatabaseType = DatabaseType.MEMORY,
config: Optional[BasicMemoryConfig] = None,
) -> AsyncGenerator[tuple[AsyncEngine, async_sessionmaker[AsyncSession]], None]:
"""Create engine and session factory.
Note: This is primarily used for testing where we want a fresh database
for each test. For production use, use get_or_create_db() instead.
Args:
db_path: Path to database file
db_type: Type of database
config: Optional explicit config. If not provided, reads from ConfigManager.
"""
global _engine, _session_maker
# Use the same helper function as production code
_engine, _session_maker = _create_engine_and_session(db_path, db_type)
_engine, _session_maker = _create_engine_and_session(db_path, db_type, config)
try:
# Verify that engine and session maker are initialized
+14 -1012
View File
File diff suppressed because it is too large Load Diff
+293
View File
@@ -0,0 +1,293 @@
"""Dependency injection for basic-memory.
This package provides FastAPI dependencies organized by feature:
- config: Application configuration
- db: Database/session management
- projects: Project resolution and config
- repositories: Data access layer
- services: Business logic layer
- importers: Import functionality
For backwards compatibility, all dependencies are re-exported from this module.
New code should import from specific submodules to reduce coupling.
"""
# Re-export everything for backwards compatibility
# Eventually, callers should import from specific submodules
from basic_memory.deps.config import (
get_app_config,
AppConfigDep,
)
from basic_memory.deps.db import (
get_engine_factory,
EngineFactoryDep,
get_session_maker,
SessionMakerDep,
)
from basic_memory.deps.projects import (
get_project_repository,
ProjectRepositoryDep,
ProjectPathDep,
get_project_id,
ProjectIdDep,
get_project_config,
ProjectConfigDep,
validate_project_id,
ProjectIdPathDep,
get_project_config_v2,
ProjectConfigV2Dep,
validate_project_external_id,
ProjectExternalIdPathDep,
get_project_config_v2_external,
ProjectConfigV2ExternalDep,
)
from basic_memory.deps.repositories import (
get_entity_repository,
EntityRepositoryDep,
get_entity_repository_v2,
EntityRepositoryV2Dep,
get_entity_repository_v2_external,
EntityRepositoryV2ExternalDep,
get_observation_repository,
ObservationRepositoryDep,
get_observation_repository_v2,
ObservationRepositoryV2Dep,
get_observation_repository_v2_external,
ObservationRepositoryV2ExternalDep,
get_relation_repository,
RelationRepositoryDep,
get_relation_repository_v2,
RelationRepositoryV2Dep,
get_relation_repository_v2_external,
RelationRepositoryV2ExternalDep,
get_search_repository,
SearchRepositoryDep,
get_search_repository_v2,
SearchRepositoryV2Dep,
get_search_repository_v2_external,
SearchRepositoryV2ExternalDep,
)
from basic_memory.deps.services import (
get_entity_parser,
EntityParserDep,
get_entity_parser_v2,
EntityParserV2Dep,
get_entity_parser_v2_external,
EntityParserV2ExternalDep,
get_markdown_processor,
MarkdownProcessorDep,
get_markdown_processor_v2,
MarkdownProcessorV2Dep,
get_markdown_processor_v2_external,
MarkdownProcessorV2ExternalDep,
get_file_service,
FileServiceDep,
get_file_service_v2,
FileServiceV2Dep,
get_file_service_v2_external,
FileServiceV2ExternalDep,
get_search_service,
SearchServiceDep,
get_search_service_v2,
SearchServiceV2Dep,
get_search_service_v2_external,
SearchServiceV2ExternalDep,
get_link_resolver,
LinkResolverDep,
get_link_resolver_v2,
LinkResolverV2Dep,
get_link_resolver_v2_external,
LinkResolverV2ExternalDep,
get_entity_service,
EntityServiceDep,
get_entity_service_v2,
EntityServiceV2Dep,
get_entity_service_v2_external,
EntityServiceV2ExternalDep,
get_context_service,
ContextServiceDep,
get_context_service_v2,
ContextServiceV2Dep,
get_context_service_v2_external,
ContextServiceV2ExternalDep,
get_sync_service,
SyncServiceDep,
get_sync_service_v2,
SyncServiceV2Dep,
get_sync_service_v2_external,
SyncServiceV2ExternalDep,
get_project_service,
ProjectServiceDep,
get_directory_service,
DirectoryServiceDep,
get_directory_service_v2,
DirectoryServiceV2Dep,
get_directory_service_v2_external,
DirectoryServiceV2ExternalDep,
)
from basic_memory.deps.importers import (
get_chatgpt_importer,
ChatGPTImporterDep,
get_chatgpt_importer_v2,
ChatGPTImporterV2Dep,
get_chatgpt_importer_v2_external,
ChatGPTImporterV2ExternalDep,
get_claude_conversations_importer,
ClaudeConversationsImporterDep,
get_claude_conversations_importer_v2,
ClaudeConversationsImporterV2Dep,
get_claude_conversations_importer_v2_external,
ClaudeConversationsImporterV2ExternalDep,
get_claude_projects_importer,
ClaudeProjectsImporterDep,
get_claude_projects_importer_v2,
ClaudeProjectsImporterV2Dep,
get_claude_projects_importer_v2_external,
ClaudeProjectsImporterV2ExternalDep,
get_memory_json_importer,
MemoryJsonImporterDep,
get_memory_json_importer_v2,
MemoryJsonImporterV2Dep,
get_memory_json_importer_v2_external,
MemoryJsonImporterV2ExternalDep,
)
__all__ = [
# Config
"get_app_config",
"AppConfigDep",
# Database
"get_engine_factory",
"EngineFactoryDep",
"get_session_maker",
"SessionMakerDep",
# Projects
"get_project_repository",
"ProjectRepositoryDep",
"ProjectPathDep",
"get_project_id",
"ProjectIdDep",
"get_project_config",
"ProjectConfigDep",
"validate_project_id",
"ProjectIdPathDep",
"get_project_config_v2",
"ProjectConfigV2Dep",
"validate_project_external_id",
"ProjectExternalIdPathDep",
"get_project_config_v2_external",
"ProjectConfigV2ExternalDep",
# Repositories
"get_entity_repository",
"EntityRepositoryDep",
"get_entity_repository_v2",
"EntityRepositoryV2Dep",
"get_entity_repository_v2_external",
"EntityRepositoryV2ExternalDep",
"get_observation_repository",
"ObservationRepositoryDep",
"get_observation_repository_v2",
"ObservationRepositoryV2Dep",
"get_observation_repository_v2_external",
"ObservationRepositoryV2ExternalDep",
"get_relation_repository",
"RelationRepositoryDep",
"get_relation_repository_v2",
"RelationRepositoryV2Dep",
"get_relation_repository_v2_external",
"RelationRepositoryV2ExternalDep",
"get_search_repository",
"SearchRepositoryDep",
"get_search_repository_v2",
"SearchRepositoryV2Dep",
"get_search_repository_v2_external",
"SearchRepositoryV2ExternalDep",
# Services
"get_entity_parser",
"EntityParserDep",
"get_entity_parser_v2",
"EntityParserV2Dep",
"get_entity_parser_v2_external",
"EntityParserV2ExternalDep",
"get_markdown_processor",
"MarkdownProcessorDep",
"get_markdown_processor_v2",
"MarkdownProcessorV2Dep",
"get_markdown_processor_v2_external",
"MarkdownProcessorV2ExternalDep",
"get_file_service",
"FileServiceDep",
"get_file_service_v2",
"FileServiceV2Dep",
"get_file_service_v2_external",
"FileServiceV2ExternalDep",
"get_search_service",
"SearchServiceDep",
"get_search_service_v2",
"SearchServiceV2Dep",
"get_search_service_v2_external",
"SearchServiceV2ExternalDep",
"get_link_resolver",
"LinkResolverDep",
"get_link_resolver_v2",
"LinkResolverV2Dep",
"get_link_resolver_v2_external",
"LinkResolverV2ExternalDep",
"get_entity_service",
"EntityServiceDep",
"get_entity_service_v2",
"EntityServiceV2Dep",
"get_entity_service_v2_external",
"EntityServiceV2ExternalDep",
"get_context_service",
"ContextServiceDep",
"get_context_service_v2",
"ContextServiceV2Dep",
"get_context_service_v2_external",
"ContextServiceV2ExternalDep",
"get_sync_service",
"SyncServiceDep",
"get_sync_service_v2",
"SyncServiceV2Dep",
"get_sync_service_v2_external",
"SyncServiceV2ExternalDep",
"get_project_service",
"ProjectServiceDep",
"get_directory_service",
"DirectoryServiceDep",
"get_directory_service_v2",
"DirectoryServiceV2Dep",
"get_directory_service_v2_external",
"DirectoryServiceV2ExternalDep",
# Importers
"get_chatgpt_importer",
"ChatGPTImporterDep",
"get_chatgpt_importer_v2",
"ChatGPTImporterV2Dep",
"get_chatgpt_importer_v2_external",
"ChatGPTImporterV2ExternalDep",
"get_claude_conversations_importer",
"ClaudeConversationsImporterDep",
"get_claude_conversations_importer_v2",
"ClaudeConversationsImporterV2Dep",
"get_claude_conversations_importer_v2_external",
"ClaudeConversationsImporterV2ExternalDep",
"get_claude_projects_importer",
"ClaudeProjectsImporterDep",
"get_claude_projects_importer_v2",
"ClaudeProjectsImporterV2Dep",
"get_claude_projects_importer_v2_external",
"ClaudeProjectsImporterV2ExternalDep",
"get_memory_json_importer",
"MemoryJsonImporterDep",
"get_memory_json_importer_v2",
"MemoryJsonImporterV2Dep",
"get_memory_json_importer_v2_external",
"MemoryJsonImporterV2ExternalDep",
]
+26
View File
@@ -0,0 +1,26 @@
"""Configuration dependency injection for basic-memory.
This module provides configuration-related dependencies.
Note: Long-term goal is to minimize direct ConfigManager access
and inject config from composition roots instead.
"""
from typing import Annotated
from fastapi import Depends
from basic_memory.config import BasicMemoryConfig, ConfigManager
def get_app_config() -> BasicMemoryConfig: # pragma: no cover
"""Get the application configuration.
Note: This is a transitional dependency. The goal is for composition roots
to read ConfigManager and inject config explicitly. During migration,
this provides the same behavior as before.
"""
app_config = ConfigManager().config
return app_config
AppConfigDep = Annotated[BasicMemoryConfig, Depends(get_app_config)]
+56
View File
@@ -0,0 +1,56 @@
"""Database dependency injection for basic-memory.
This module provides database-related dependencies:
- Engine and session maker factories
- Session dependencies for request handling
"""
from typing import Annotated
from fastapi import Depends, Request
from loguru import logger
from sqlalchemy.ext.asyncio import (
AsyncEngine,
AsyncSession,
async_sessionmaker,
)
from basic_memory import db
from basic_memory.deps.config import get_app_config
async def get_engine_factory(
request: Request,
) -> tuple[AsyncEngine, async_sessionmaker[AsyncSession]]: # pragma: no cover
"""Get cached engine and session maker from app state.
For API requests, returns cached connections from app.state for optimal performance.
For non-API contexts (CLI), falls back to direct database connection.
"""
# Try to get cached connections from app state (API context)
if (
hasattr(request, "app")
and hasattr(request.app.state, "engine")
and hasattr(request.app.state, "session_maker")
):
return request.app.state.engine, request.app.state.session_maker
# Fallback for non-API contexts (CLI)
logger.debug("Using fallback database connection for non-API context")
app_config = get_app_config()
engine, session_maker = await db.get_or_create_db(app_config.database_path)
return engine, session_maker
EngineFactoryDep = Annotated[
tuple[AsyncEngine, async_sessionmaker[AsyncSession]], Depends(get_engine_factory)
]
async def get_session_maker(engine_factory: EngineFactoryDep) -> async_sessionmaker[AsyncSession]:
"""Get session maker."""
_, session_maker = engine_factory
return session_maker
SessionMakerDep = Annotated[async_sessionmaker, Depends(get_session_maker)]
+200
View File
@@ -0,0 +1,200 @@
"""Importer dependency injection for basic-memory.
This module provides importer dependencies:
- ChatGPTImporter
- ClaudeConversationsImporter
- ClaudeProjectsImporter
- MemoryJsonImporter
"""
from typing import Annotated
from fastapi import Depends
from basic_memory.deps.projects import (
ProjectConfigDep,
ProjectConfigV2Dep,
ProjectConfigV2ExternalDep,
)
from basic_memory.deps.services import (
FileServiceDep,
FileServiceV2Dep,
FileServiceV2ExternalDep,
MarkdownProcessorDep,
MarkdownProcessorV2Dep,
MarkdownProcessorV2ExternalDep,
)
from basic_memory.importers import (
ChatGPTImporter,
ClaudeConversationsImporter,
ClaudeProjectsImporter,
MemoryJsonImporter,
)
# --- ChatGPT Importer ---
async def get_chatgpt_importer(
project_config: ProjectConfigDep,
markdown_processor: MarkdownProcessorDep,
file_service: FileServiceDep,
) -> ChatGPTImporter:
"""Create ChatGPTImporter with dependencies."""
return ChatGPTImporter(project_config.home, markdown_processor, file_service)
ChatGPTImporterDep = Annotated[ChatGPTImporter, Depends(get_chatgpt_importer)]
async def get_chatgpt_importer_v2( # pragma: no cover
project_config: ProjectConfigV2Dep,
markdown_processor: MarkdownProcessorV2Dep,
file_service: FileServiceV2Dep,
) -> ChatGPTImporter:
"""Create ChatGPTImporter with v2 dependencies."""
return ChatGPTImporter(project_config.home, markdown_processor, file_service)
ChatGPTImporterV2Dep = Annotated[ChatGPTImporter, Depends(get_chatgpt_importer_v2)]
async def get_chatgpt_importer_v2_external(
project_config: ProjectConfigV2ExternalDep,
markdown_processor: MarkdownProcessorV2ExternalDep,
file_service: FileServiceV2ExternalDep,
) -> ChatGPTImporter:
"""Create ChatGPTImporter with v2 external_id dependencies."""
return ChatGPTImporter(project_config.home, markdown_processor, file_service)
ChatGPTImporterV2ExternalDep = Annotated[ChatGPTImporter, Depends(get_chatgpt_importer_v2_external)]
# --- Claude Conversations Importer ---
async def get_claude_conversations_importer(
project_config: ProjectConfigDep,
markdown_processor: MarkdownProcessorDep,
file_service: FileServiceDep,
) -> ClaudeConversationsImporter:
"""Create ClaudeConversationsImporter with dependencies."""
return ClaudeConversationsImporter(project_config.home, markdown_processor, file_service)
ClaudeConversationsImporterDep = Annotated[
ClaudeConversationsImporter, Depends(get_claude_conversations_importer)
]
async def get_claude_conversations_importer_v2( # pragma: no cover
project_config: ProjectConfigV2Dep,
markdown_processor: MarkdownProcessorV2Dep,
file_service: FileServiceV2Dep,
) -> ClaudeConversationsImporter:
"""Create ClaudeConversationsImporter with v2 dependencies."""
return ClaudeConversationsImporter(project_config.home, markdown_processor, file_service)
ClaudeConversationsImporterV2Dep = Annotated[
ClaudeConversationsImporter, Depends(get_claude_conversations_importer_v2)
]
async def get_claude_conversations_importer_v2_external(
project_config: ProjectConfigV2ExternalDep,
markdown_processor: MarkdownProcessorV2ExternalDep,
file_service: FileServiceV2ExternalDep,
) -> ClaudeConversationsImporter:
"""Create ClaudeConversationsImporter with v2 external_id dependencies."""
return ClaudeConversationsImporter(project_config.home, markdown_processor, file_service)
ClaudeConversationsImporterV2ExternalDep = Annotated[
ClaudeConversationsImporter, Depends(get_claude_conversations_importer_v2_external)
]
# --- Claude Projects Importer ---
async def get_claude_projects_importer(
project_config: ProjectConfigDep,
markdown_processor: MarkdownProcessorDep,
file_service: FileServiceDep,
) -> ClaudeProjectsImporter:
"""Create ClaudeProjectsImporter with dependencies."""
return ClaudeProjectsImporter(project_config.home, markdown_processor, file_service)
ClaudeProjectsImporterDep = Annotated[ClaudeProjectsImporter, Depends(get_claude_projects_importer)]
async def get_claude_projects_importer_v2( # pragma: no cover
project_config: ProjectConfigV2Dep,
markdown_processor: MarkdownProcessorV2Dep,
file_service: FileServiceV2Dep,
) -> ClaudeProjectsImporter:
"""Create ClaudeProjectsImporter with v2 dependencies."""
return ClaudeProjectsImporter(project_config.home, markdown_processor, file_service)
ClaudeProjectsImporterV2Dep = Annotated[
ClaudeProjectsImporter, Depends(get_claude_projects_importer_v2)
]
async def get_claude_projects_importer_v2_external(
project_config: ProjectConfigV2ExternalDep,
markdown_processor: MarkdownProcessorV2ExternalDep,
file_service: FileServiceV2ExternalDep,
) -> ClaudeProjectsImporter:
"""Create ClaudeProjectsImporter with v2 external_id dependencies."""
return ClaudeProjectsImporter(project_config.home, markdown_processor, file_service)
ClaudeProjectsImporterV2ExternalDep = Annotated[
ClaudeProjectsImporter, Depends(get_claude_projects_importer_v2_external)
]
# --- Memory JSON Importer ---
async def get_memory_json_importer(
project_config: ProjectConfigDep,
markdown_processor: MarkdownProcessorDep,
file_service: FileServiceDep,
) -> MemoryJsonImporter:
"""Create MemoryJsonImporter with dependencies."""
return MemoryJsonImporter(project_config.home, markdown_processor, file_service)
MemoryJsonImporterDep = Annotated[MemoryJsonImporter, Depends(get_memory_json_importer)]
async def get_memory_json_importer_v2( # pragma: no cover
project_config: ProjectConfigV2Dep,
markdown_processor: MarkdownProcessorV2Dep,
file_service: FileServiceV2Dep,
) -> MemoryJsonImporter:
"""Create MemoryJsonImporter with v2 dependencies."""
return MemoryJsonImporter(project_config.home, markdown_processor, file_service)
MemoryJsonImporterV2Dep = Annotated[MemoryJsonImporter, Depends(get_memory_json_importer_v2)]
async def get_memory_json_importer_v2_external(
project_config: ProjectConfigV2ExternalDep,
markdown_processor: MarkdownProcessorV2ExternalDep,
file_service: FileServiceV2ExternalDep,
) -> MemoryJsonImporter:
"""Create MemoryJsonImporter with v2 external_id dependencies."""
return MemoryJsonImporter(project_config.home, markdown_processor, file_service)
MemoryJsonImporterV2ExternalDep = Annotated[
MemoryJsonImporter, Depends(get_memory_json_importer_v2_external)
]
+236
View File
@@ -0,0 +1,236 @@
"""Project dependency injection for basic-memory.
This module provides project-related dependencies:
- Project path extraction from URL
- Project config resolution
- Project ID validation
- Project repository
"""
import pathlib
from typing import Annotated
from fastapi import Depends, HTTPException, Path, status
from basic_memory.config import ProjectConfig
from basic_memory.deps.db import SessionMakerDep
from basic_memory.repository.project_repository import ProjectRepository
from basic_memory.utils import generate_permalink
# --- Project Repository ---
async def get_project_repository(
session_maker: SessionMakerDep,
) -> ProjectRepository:
"""Get the project repository."""
return ProjectRepository(session_maker)
ProjectRepositoryDep = Annotated[ProjectRepository, Depends(get_project_repository)]
# --- Path Extraction ---
# V1 API: Project name from URL path
ProjectPathDep = Annotated[str, Path()]
# --- Project ID Resolution (V1 API) ---
async def get_project_id(
project_repository: ProjectRepositoryDep,
project: ProjectPathDep,
) -> int:
"""Get the current project ID from request state.
When using sub-applications with /{project} mounting, the project value
is stored in request.state by middleware.
Args:
project_repository: Repository for project operations
project: The project name from URL path
Returns:
The resolved project ID
Raises:
HTTPException: If project is not found
"""
# Convert project name to permalink for lookup
project_permalink = generate_permalink(str(project))
project_obj = await project_repository.get_by_permalink(project_permalink)
if project_obj:
return project_obj.id
# Try by name if permalink lookup fails
project_obj = await project_repository.get_by_name(str(project)) # pragma: no cover
if project_obj: # pragma: no cover
return project_obj.id
# Not found
raise HTTPException( # pragma: no cover
status_code=status.HTTP_404_NOT_FOUND, detail=f"Project '{project}' not found."
)
ProjectIdDep = Annotated[int, Depends(get_project_id)]
# --- Project Config Resolution (V1 API) ---
async def get_project_config(
project: ProjectPathDep, project_repository: ProjectRepositoryDep
) -> ProjectConfig: # pragma: no cover
"""Get the current project referenced from request state.
Args:
project: The project name from URL path
project_repository: Repository for project operations
Returns:
The resolved project config
Raises:
HTTPException: If project is not found
"""
# Convert project name to permalink for lookup
project_permalink = generate_permalink(str(project))
project_obj = await project_repository.get_by_permalink(project_permalink)
if project_obj:
return ProjectConfig(name=project_obj.name, home=pathlib.Path(project_obj.path))
# Not found
raise HTTPException( # pragma: no cover
status_code=status.HTTP_404_NOT_FOUND, detail=f"Project '{project}' not found."
)
ProjectConfigDep = Annotated[ProjectConfig, Depends(get_project_config)]
# --- V2 API: Integer Project ID from Path ---
async def validate_project_id(
project_id: int,
project_repository: ProjectRepositoryDep,
) -> int:
"""Validate that a numeric project ID exists in the database.
This is used for v2 API endpoints that take project IDs as integers in the path.
The project_id parameter will be automatically extracted from the URL path by FastAPI.
Args:
project_id: The numeric project ID from the URL path
project_repository: Repository for project operations
Returns:
The validated project ID
Raises:
HTTPException: If project with that ID is not found
"""
project_obj = await project_repository.get_by_id(project_id)
if not project_obj:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail=f"Project with ID {project_id} not found.",
)
return project_id
ProjectIdPathDep = Annotated[int, Depends(validate_project_id)]
async def get_project_config_v2(
project_id: ProjectIdPathDep, project_repository: ProjectRepositoryDep
) -> ProjectConfig: # pragma: no cover
"""Get the project config for v2 API (uses integer project_id from path).
Args:
project_id: The validated numeric project ID from the URL path
project_repository: Repository for project operations
Returns:
The resolved project config
Raises:
HTTPException: If project is not found
"""
project_obj = await project_repository.get_by_id(project_id)
if project_obj:
return ProjectConfig(name=project_obj.name, home=pathlib.Path(project_obj.path))
# Not found (this should not happen since ProjectIdPathDep already validates existence)
raise HTTPException( # pragma: no cover
status_code=status.HTTP_404_NOT_FOUND, detail=f"Project with ID {project_id} not found."
)
ProjectConfigV2Dep = Annotated[ProjectConfig, Depends(get_project_config_v2)]
# --- V2 API: External UUID Project ID from Path ---
async def validate_project_external_id(
project_id: str,
project_repository: ProjectRepositoryDep,
) -> int:
"""Validate that a project external_id (UUID) exists in the database.
This is used for v2 API endpoints that take project external_ids as strings in the path.
The project_id parameter will be automatically extracted from the URL path by FastAPI.
Args:
project_id: The external UUID from the URL path (named project_id for URL consistency)
project_repository: Repository for project operations
Returns:
The internal numeric project ID (for use by repositories)
Raises:
HTTPException: If project with that external_id is not found
"""
project_obj = await project_repository.get_by_external_id(project_id)
if not project_obj:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail=f"Project with external_id '{project_id}' not found.",
)
return project_obj.id
ProjectExternalIdPathDep = Annotated[int, Depends(validate_project_external_id)]
async def get_project_config_v2_external(
project_id: ProjectExternalIdPathDep, project_repository: ProjectRepositoryDep
) -> ProjectConfig: # pragma: no cover
"""Get the project config for v2 API (uses external_id UUID from path).
Args:
project_id: The internal project ID resolved from external_id
project_repository: Repository for project operations
Returns:
The resolved project config
Raises:
HTTPException: If project is not found
"""
project_obj = await project_repository.get_by_id(project_id)
if project_obj:
return ProjectConfig(name=project_obj.name, home=pathlib.Path(project_obj.path))
# Not found (this should not happen since ProjectExternalIdPathDep already validates)
raise HTTPException( # pragma: no cover
status_code=status.HTTP_404_NOT_FOUND, detail=f"Project with ID {project_id} not found."
)
ProjectConfigV2ExternalDep = Annotated[ProjectConfig, Depends(get_project_config_v2_external)]
+183
View File
@@ -0,0 +1,183 @@
"""Repository dependency injection for basic-memory.
This module provides repository dependencies:
- EntityRepository
- ObservationRepository
- RelationRepository
- SearchRepository
Each repository is scoped to a project ID from the request.
"""
from typing import Annotated
from fastapi import Depends
from basic_memory.deps.db import SessionMakerDep
from basic_memory.deps.projects import (
ProjectIdDep,
ProjectIdPathDep,
ProjectExternalIdPathDep,
)
from basic_memory.repository.entity_repository import EntityRepository
from basic_memory.repository.observation_repository import ObservationRepository
from basic_memory.repository.relation_repository import RelationRepository
from basic_memory.repository.search_repository import SearchRepository, create_search_repository
# --- Entity Repository ---
async def get_entity_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
) -> EntityRepository:
"""Create an EntityRepository instance for the current project."""
return EntityRepository(session_maker, project_id=project_id)
EntityRepositoryDep = Annotated[EntityRepository, Depends(get_entity_repository)]
async def get_entity_repository_v2( # pragma: no cover
session_maker: SessionMakerDep,
project_id: ProjectIdPathDep,
) -> EntityRepository:
"""Create an EntityRepository instance for v2 API (uses integer project_id from path)."""
return EntityRepository(session_maker, project_id=project_id)
EntityRepositoryV2Dep = Annotated[EntityRepository, Depends(get_entity_repository_v2)]
async def get_entity_repository_v2_external(
session_maker: SessionMakerDep,
project_id: ProjectExternalIdPathDep,
) -> EntityRepository:
"""Create an EntityRepository instance for v2 API (uses external_id from path)."""
return EntityRepository(session_maker, project_id=project_id)
EntityRepositoryV2ExternalDep = Annotated[
EntityRepository, Depends(get_entity_repository_v2_external)
]
# --- Observation Repository ---
async def get_observation_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
) -> ObservationRepository:
"""Create an ObservationRepository instance for the current project."""
return ObservationRepository(session_maker, project_id=project_id)
ObservationRepositoryDep = Annotated[ObservationRepository, Depends(get_observation_repository)]
async def get_observation_repository_v2( # pragma: no cover
session_maker: SessionMakerDep,
project_id: ProjectIdPathDep,
) -> ObservationRepository:
"""Create an ObservationRepository instance for v2 API."""
return ObservationRepository(session_maker, project_id=project_id)
ObservationRepositoryV2Dep = Annotated[
ObservationRepository, Depends(get_observation_repository_v2)
]
async def get_observation_repository_v2_external(
session_maker: SessionMakerDep,
project_id: ProjectExternalIdPathDep,
) -> ObservationRepository:
"""Create an ObservationRepository instance for v2 API (uses external_id)."""
return ObservationRepository(session_maker, project_id=project_id)
ObservationRepositoryV2ExternalDep = Annotated[
ObservationRepository, Depends(get_observation_repository_v2_external)
]
# --- Relation Repository ---
async def get_relation_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
) -> RelationRepository:
"""Create a RelationRepository instance for the current project."""
return RelationRepository(session_maker, project_id=project_id)
RelationRepositoryDep = Annotated[RelationRepository, Depends(get_relation_repository)]
async def get_relation_repository_v2( # pragma: no cover
session_maker: SessionMakerDep,
project_id: ProjectIdPathDep,
) -> RelationRepository:
"""Create a RelationRepository instance for v2 API."""
return RelationRepository(session_maker, project_id=project_id)
RelationRepositoryV2Dep = Annotated[RelationRepository, Depends(get_relation_repository_v2)]
async def get_relation_repository_v2_external(
session_maker: SessionMakerDep,
project_id: ProjectExternalIdPathDep,
) -> RelationRepository:
"""Create a RelationRepository instance for v2 API (uses external_id)."""
return RelationRepository(session_maker, project_id=project_id)
RelationRepositoryV2ExternalDep = Annotated[
RelationRepository, Depends(get_relation_repository_v2_external)
]
# --- Search Repository ---
async def get_search_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
) -> SearchRepository:
"""Create a backend-specific SearchRepository instance for the current project.
Uses factory function to return SQLiteSearchRepository or PostgresSearchRepository
based on database backend configuration.
"""
return create_search_repository(session_maker, project_id=project_id)
SearchRepositoryDep = Annotated[SearchRepository, Depends(get_search_repository)]
async def get_search_repository_v2( # pragma: no cover
session_maker: SessionMakerDep,
project_id: ProjectIdPathDep,
) -> SearchRepository:
"""Create a SearchRepository instance for v2 API."""
return create_search_repository(session_maker, project_id=project_id)
SearchRepositoryV2Dep = Annotated[SearchRepository, Depends(get_search_repository_v2)]
async def get_search_repository_v2_external(
session_maker: SessionMakerDep,
project_id: ProjectExternalIdPathDep,
) -> SearchRepository:
"""Create a SearchRepository instance for v2 API (uses external_id)."""
return create_search_repository(session_maker, project_id=project_id)
SearchRepositoryV2ExternalDep = Annotated[
SearchRepository, Depends(get_search_repository_v2_external)
]
+484
View File
@@ -0,0 +1,484 @@
"""Service dependency injection for basic-memory.
This module provides service-layer dependencies:
- EntityParser, MarkdownProcessor
- FileService, EntityService
- SearchService, LinkResolver, ContextService
- SyncService, ProjectService, DirectoryService
"""
from typing import Annotated
from fastapi import Depends
from loguru import logger
from basic_memory.deps.config import AppConfigDep
from basic_memory.deps.projects import (
ProjectConfigDep,
ProjectConfigV2Dep,
ProjectConfigV2ExternalDep,
ProjectRepositoryDep,
)
from basic_memory.deps.repositories import (
EntityRepositoryDep,
EntityRepositoryV2Dep,
EntityRepositoryV2ExternalDep,
ObservationRepositoryDep,
ObservationRepositoryV2Dep,
ObservationRepositoryV2ExternalDep,
RelationRepositoryDep,
RelationRepositoryV2Dep,
RelationRepositoryV2ExternalDep,
SearchRepositoryDep,
SearchRepositoryV2Dep,
SearchRepositoryV2ExternalDep,
)
from basic_memory.markdown import EntityParser
from basic_memory.markdown.markdown_processor import MarkdownProcessor
from basic_memory.services import EntityService, ProjectService
from basic_memory.services.context_service import ContextService
from basic_memory.services.directory_service import DirectoryService
from basic_memory.services.file_service import FileService
from basic_memory.services.link_resolver import LinkResolver
from basic_memory.services.search_service import SearchService
from basic_memory.sync import SyncService
# --- Entity Parser ---
async def get_entity_parser(project_config: ProjectConfigDep) -> EntityParser:
return EntityParser(project_config.home)
EntityParserDep = Annotated["EntityParser", Depends(get_entity_parser)]
async def get_entity_parser_v2(
project_config: ProjectConfigV2Dep,
) -> EntityParser: # pragma: no cover
return EntityParser(project_config.home)
EntityParserV2Dep = Annotated["EntityParser", Depends(get_entity_parser_v2)]
async def get_entity_parser_v2_external(project_config: ProjectConfigV2ExternalDep) -> EntityParser:
return EntityParser(project_config.home)
EntityParserV2ExternalDep = Annotated["EntityParser", Depends(get_entity_parser_v2_external)]
# --- Markdown Processor ---
async def get_markdown_processor(
entity_parser: EntityParserDep, app_config: AppConfigDep
) -> MarkdownProcessor:
return MarkdownProcessor(entity_parser, app_config=app_config)
MarkdownProcessorDep = Annotated[MarkdownProcessor, Depends(get_markdown_processor)]
async def get_markdown_processor_v2( # pragma: no cover
entity_parser: EntityParserV2Dep, app_config: AppConfigDep
) -> MarkdownProcessor:
return MarkdownProcessor(entity_parser, app_config=app_config)
MarkdownProcessorV2Dep = Annotated[MarkdownProcessor, Depends(get_markdown_processor_v2)]
async def get_markdown_processor_v2_external(
entity_parser: EntityParserV2ExternalDep, app_config: AppConfigDep
) -> MarkdownProcessor:
return MarkdownProcessor(entity_parser, app_config=app_config)
MarkdownProcessorV2ExternalDep = Annotated[
MarkdownProcessor, Depends(get_markdown_processor_v2_external)
]
# --- File Service ---
async def get_file_service(
project_config: ProjectConfigDep,
markdown_processor: MarkdownProcessorDep,
app_config: AppConfigDep,
) -> FileService:
file_service = FileService(project_config.home, markdown_processor, app_config=app_config)
logger.debug(
f"Created FileService for project: {project_config.name}, base_path: {project_config.home} "
)
return file_service
FileServiceDep = Annotated[FileService, Depends(get_file_service)]
async def get_file_service_v2( # pragma: no cover
project_config: ProjectConfigV2Dep,
markdown_processor: MarkdownProcessorV2Dep,
app_config: AppConfigDep,
) -> FileService:
file_service = FileService(project_config.home, markdown_processor, app_config=app_config)
logger.debug(
f"Created FileService for project: {project_config.name}, base_path: {project_config.home}"
)
return file_service
FileServiceV2Dep = Annotated[FileService, Depends(get_file_service_v2)]
async def get_file_service_v2_external(
project_config: ProjectConfigV2ExternalDep,
markdown_processor: MarkdownProcessorV2ExternalDep,
app_config: AppConfigDep,
) -> FileService:
file_service = FileService(project_config.home, markdown_processor, app_config=app_config)
logger.debug(
f"Created FileService for project: {project_config.name}, base_path: {project_config.home}"
)
return file_service
FileServiceV2ExternalDep = Annotated[FileService, Depends(get_file_service_v2_external)]
# --- Search Service ---
async def get_search_service(
search_repository: SearchRepositoryDep,
entity_repository: EntityRepositoryDep,
file_service: FileServiceDep,
) -> SearchService:
"""Create SearchService with dependencies."""
return SearchService(search_repository, entity_repository, file_service)
SearchServiceDep = Annotated[SearchService, Depends(get_search_service)]
async def get_search_service_v2( # pragma: no cover
search_repository: SearchRepositoryV2Dep,
entity_repository: EntityRepositoryV2Dep,
file_service: FileServiceV2Dep,
) -> SearchService:
"""Create SearchService for v2 API."""
return SearchService(search_repository, entity_repository, file_service)
SearchServiceV2Dep = Annotated[SearchService, Depends(get_search_service_v2)]
async def get_search_service_v2_external(
search_repository: SearchRepositoryV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
file_service: FileServiceV2ExternalDep,
) -> SearchService:
"""Create SearchService for v2 API (uses external_id)."""
return SearchService(search_repository, entity_repository, file_service)
SearchServiceV2ExternalDep = Annotated[SearchService, Depends(get_search_service_v2_external)]
# --- Link Resolver ---
async def get_link_resolver(
entity_repository: EntityRepositoryDep, search_service: SearchServiceDep
) -> LinkResolver:
return LinkResolver(entity_repository=entity_repository, search_service=search_service)
LinkResolverDep = Annotated[LinkResolver, Depends(get_link_resolver)]
async def get_link_resolver_v2( # pragma: no cover
entity_repository: EntityRepositoryV2Dep, search_service: SearchServiceV2Dep
) -> LinkResolver:
return LinkResolver(entity_repository=entity_repository, search_service=search_service)
LinkResolverV2Dep = Annotated[LinkResolver, Depends(get_link_resolver_v2)]
async def get_link_resolver_v2_external(
entity_repository: EntityRepositoryV2ExternalDep, search_service: SearchServiceV2ExternalDep
) -> LinkResolver:
return LinkResolver(entity_repository=entity_repository, search_service=search_service)
LinkResolverV2ExternalDep = Annotated[LinkResolver, Depends(get_link_resolver_v2_external)]
# --- Entity Service ---
async def get_entity_service(
entity_repository: EntityRepositoryDep,
observation_repository: ObservationRepositoryDep,
relation_repository: RelationRepositoryDep,
entity_parser: EntityParserDep,
file_service: FileServiceDep,
link_resolver: LinkResolverDep,
search_service: SearchServiceDep,
app_config: AppConfigDep,
) -> EntityService:
"""Create EntityService with repository."""
return EntityService(
entity_repository=entity_repository,
observation_repository=observation_repository,
relation_repository=relation_repository,
entity_parser=entity_parser,
file_service=file_service,
link_resolver=link_resolver,
search_service=search_service,
app_config=app_config,
)
EntityServiceDep = Annotated[EntityService, Depends(get_entity_service)]
async def get_entity_service_v2( # pragma: no cover
entity_repository: EntityRepositoryV2Dep,
observation_repository: ObservationRepositoryV2Dep,
relation_repository: RelationRepositoryV2Dep,
entity_parser: EntityParserV2Dep,
file_service: FileServiceV2Dep,
link_resolver: LinkResolverV2Dep,
search_service: SearchServiceV2Dep,
app_config: AppConfigDep,
) -> EntityService:
"""Create EntityService for v2 API."""
return EntityService(
entity_repository=entity_repository,
observation_repository=observation_repository,
relation_repository=relation_repository,
entity_parser=entity_parser,
file_service=file_service,
link_resolver=link_resolver,
search_service=search_service,
app_config=app_config,
)
EntityServiceV2Dep = Annotated[EntityService, Depends(get_entity_service_v2)]
async def get_entity_service_v2_external(
entity_repository: EntityRepositoryV2ExternalDep,
observation_repository: ObservationRepositoryV2ExternalDep,
relation_repository: RelationRepositoryV2ExternalDep,
entity_parser: EntityParserV2ExternalDep,
file_service: FileServiceV2ExternalDep,
link_resolver: LinkResolverV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
app_config: AppConfigDep,
) -> EntityService:
"""Create EntityService for v2 API (uses external_id)."""
return EntityService(
entity_repository=entity_repository,
observation_repository=observation_repository,
relation_repository=relation_repository,
entity_parser=entity_parser,
file_service=file_service,
link_resolver=link_resolver,
search_service=search_service,
app_config=app_config,
)
EntityServiceV2ExternalDep = Annotated[EntityService, Depends(get_entity_service_v2_external)]
# --- Context Service ---
async def get_context_service(
search_repository: SearchRepositoryDep,
entity_repository: EntityRepositoryDep,
observation_repository: ObservationRepositoryDep,
) -> ContextService:
return ContextService(
search_repository=search_repository,
entity_repository=entity_repository,
observation_repository=observation_repository,
)
ContextServiceDep = Annotated[ContextService, Depends(get_context_service)]
async def get_context_service_v2( # pragma: no cover
search_repository: SearchRepositoryV2Dep,
entity_repository: EntityRepositoryV2Dep,
observation_repository: ObservationRepositoryV2Dep,
) -> ContextService:
"""Create ContextService for v2 API."""
return ContextService(
search_repository=search_repository,
entity_repository=entity_repository,
observation_repository=observation_repository,
)
ContextServiceV2Dep = Annotated[ContextService, Depends(get_context_service_v2)]
async def get_context_service_v2_external(
search_repository: SearchRepositoryV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
observation_repository: ObservationRepositoryV2ExternalDep,
) -> ContextService:
"""Create ContextService for v2 API (uses external_id)."""
return ContextService(
search_repository=search_repository,
entity_repository=entity_repository,
observation_repository=observation_repository,
)
ContextServiceV2ExternalDep = Annotated[ContextService, Depends(get_context_service_v2_external)]
# --- Sync Service ---
async def get_sync_service(
app_config: AppConfigDep,
entity_service: EntityServiceDep,
entity_parser: EntityParserDep,
entity_repository: EntityRepositoryDep,
relation_repository: RelationRepositoryDep,
project_repository: ProjectRepositoryDep,
search_service: SearchServiceDep,
file_service: FileServiceDep,
) -> SyncService: # pragma: no cover
return SyncService(
app_config=app_config,
entity_service=entity_service,
entity_parser=entity_parser,
entity_repository=entity_repository,
relation_repository=relation_repository,
project_repository=project_repository,
search_service=search_service,
file_service=file_service,
)
SyncServiceDep = Annotated[SyncService, Depends(get_sync_service)]
async def get_sync_service_v2(
app_config: AppConfigDep,
entity_service: EntityServiceV2Dep,
entity_parser: EntityParserV2Dep,
entity_repository: EntityRepositoryV2Dep,
relation_repository: RelationRepositoryV2Dep,
project_repository: ProjectRepositoryDep,
search_service: SearchServiceV2Dep,
file_service: FileServiceV2Dep,
) -> SyncService: # pragma: no cover
"""Create SyncService for v2 API."""
return SyncService(
app_config=app_config,
entity_service=entity_service,
entity_parser=entity_parser,
entity_repository=entity_repository,
relation_repository=relation_repository,
project_repository=project_repository,
search_service=search_service,
file_service=file_service,
)
SyncServiceV2Dep = Annotated[SyncService, Depends(get_sync_service_v2)]
async def get_sync_service_v2_external(
app_config: AppConfigDep,
entity_service: EntityServiceV2ExternalDep,
entity_parser: EntityParserV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
relation_repository: RelationRepositoryV2ExternalDep,
project_repository: ProjectRepositoryDep,
search_service: SearchServiceV2ExternalDep,
file_service: FileServiceV2ExternalDep,
) -> SyncService: # pragma: no cover
"""Create SyncService for v2 API (uses external_id)."""
return SyncService(
app_config=app_config,
entity_service=entity_service,
entity_parser=entity_parser,
entity_repository=entity_repository,
relation_repository=relation_repository,
project_repository=project_repository,
search_service=search_service,
file_service=file_service,
)
SyncServiceV2ExternalDep = Annotated[SyncService, Depends(get_sync_service_v2_external)]
# --- Project Service ---
async def get_project_service(
project_repository: ProjectRepositoryDep,
) -> ProjectService:
"""Create ProjectService with repository."""
return ProjectService(repository=project_repository)
ProjectServiceDep = Annotated[ProjectService, Depends(get_project_service)]
# --- Directory Service ---
async def get_directory_service(
entity_repository: EntityRepositoryDep,
) -> DirectoryService:
"""Create DirectoryService with dependencies."""
return DirectoryService(
entity_repository=entity_repository,
)
DirectoryServiceDep = Annotated[DirectoryService, Depends(get_directory_service)]
async def get_directory_service_v2( # pragma: no cover
entity_repository: EntityRepositoryV2Dep,
) -> DirectoryService:
"""Create DirectoryService for v2 API (uses integer project_id from path)."""
return DirectoryService(
entity_repository=entity_repository,
)
DirectoryServiceV2Dep = Annotated[DirectoryService, Depends(get_directory_service_v2)]
async def get_directory_service_v2_external(
entity_repository: EntityRepositoryV2ExternalDep,
) -> DirectoryService:
"""Create DirectoryService for v2 API (uses external_id from path)."""
return DirectoryService(
entity_repository=entity_repository,
)
DirectoryServiceV2ExternalDep = Annotated[
DirectoryService, Depends(get_directory_service_v2_external)
]
@@ -15,7 +15,7 @@ logger = logging.getLogger(__name__)
class ChatGPTImporter(Importer[ChatImportResult]):
"""Service for importing ChatGPT conversations."""
def handle_error(
def handle_error( # pragma: no cover
self, message: str, error: Optional[Exception] = None
) -> ChatImportResult:
"""Return a failed ChatImportResult with an error message."""
@@ -15,7 +15,7 @@ logger = logging.getLogger(__name__)
class ClaudeConversationsImporter(Importer[ChatImportResult]):
"""Service for importing Claude conversations."""
def handle_error(
def handle_error( # pragma: no cover
self, message: str, error: Optional[Exception] = None
) -> ChatImportResult:
"""Return a failed ChatImportResult with an error message."""
@@ -14,7 +14,7 @@ logger = logging.getLogger(__name__)
class ClaudeProjectsImporter(Importer[ProjectImportResult]):
"""Service for importing Claude projects."""
def handle_error(
def handle_error( # pragma: no cover
self, message: str, error: Optional[Exception] = None
) -> ProjectImportResult:
"""Return a failed ProjectImportResult with an error message."""
@@ -13,7 +13,7 @@ logger = logging.getLogger(__name__)
class MemoryJsonImporter(Importer[EntityImportResult]):
"""Service for importing memory.json format data."""
def handle_error(
def handle_error( # pragma: no cover
self, message: str, error: Optional[Exception] = None
) -> EntityImportResult:
"""Return a failed EntityImportResult with an error message."""
+7
View File
@@ -1,5 +1,6 @@
"""Utilities for converting between markdown and entity models."""
import uuid
from pathlib import Path
from typing import Any, Optional
@@ -40,6 +41,12 @@ def entity_model_from_markdown(
# Create or update entity
model = entity or Entity()
# Ensure external_id is set for new entities
# SQLAlchemy's Python-side default may not always evaluate,
# so we explicitly set it here for reliability (fixes #512)
if not model.external_id:
model.external_id = str(uuid.uuid4())
# Update basic fields
model.title = markdown.frontmatter.title
model.entity_type = markdown.frontmatter.type
+28
View File
@@ -0,0 +1,28 @@
"""Typed internal API clients for MCP tools.
These clients encapsulate API paths, error handling, and response validation.
MCP tools become thin adapters that call these clients and format results.
Usage:
from basic_memory.mcp.clients import KnowledgeClient, SearchClient
async with get_client() as http_client:
knowledge = KnowledgeClient(http_client, project_id)
entity = await knowledge.create_entity(entity_data)
"""
from basic_memory.mcp.clients.knowledge import KnowledgeClient
from basic_memory.mcp.clients.search import SearchClient
from basic_memory.mcp.clients.memory import MemoryClient
from basic_memory.mcp.clients.directory import DirectoryClient
from basic_memory.mcp.clients.resource import ResourceClient
from basic_memory.mcp.clients.project import ProjectClient
__all__ = [
"KnowledgeClient",
"SearchClient",
"MemoryClient",
"DirectoryClient",
"ResourceClient",
"ProjectClient",
]
+70
View File
@@ -0,0 +1,70 @@
"""Typed client for directory API operations.
Encapsulates all /v2/projects/{project_id}/directory/* endpoints.
"""
from typing import Optional, Any
from httpx import AsyncClient
from basic_memory.mcp.tools.utils import call_get
class DirectoryClient:
"""Typed client for directory listing operations.
Centralizes:
- API path construction for /v2/projects/{project_id}/directory/*
- Response validation
- Consistent error handling through call_* utilities
Usage:
async with get_client() as http_client:
client = DirectoryClient(http_client, project_id)
nodes = await client.list("/", depth=2)
"""
def __init__(self, http_client: AsyncClient, project_id: str):
"""Initialize the directory client.
Args:
http_client: HTTPX AsyncClient for making requests
project_id: Project external_id (UUID) for API calls
"""
self.http_client = http_client
self.project_id = project_id
self._base_path = f"/v2/projects/{project_id}/directory"
async def list(
self,
dir_name: str = "/",
*,
depth: int = 1,
file_name_glob: Optional[str] = None,
) -> list[dict[str, Any]]:
"""List directory contents.
Args:
dir_name: Directory path to list (default: root)
depth: How deep to traverse (default: 1)
file_name_glob: Optional glob pattern to filter files
Returns:
List of directory nodes with their contents
Raises:
ToolError: If the request fails
"""
params: dict = {
"dir_name": dir_name,
"depth": depth,
}
if file_name_glob:
params["file_name_glob"] = file_name_glob
response = await call_get(
self.http_client,
f"{self._base_path}/list",
params=params,
)
return response.json()
+176
View File
@@ -0,0 +1,176 @@
"""Typed client for knowledge/entity API operations.
Encapsulates all /v2/projects/{project_id}/knowledge/* endpoints.
"""
from typing import Any
from httpx import AsyncClient
from basic_memory.mcp.tools.utils import call_get, call_post, call_put, call_patch, call_delete
from basic_memory.schemas.response import EntityResponse, DeleteEntitiesResponse
class KnowledgeClient:
"""Typed client for knowledge graph entity operations.
Centralizes:
- API path construction for /v2/projects/{project_id}/knowledge/*
- Response validation via Pydantic models
- Consistent error handling through call_* utilities
Usage:
async with get_client() as http_client:
client = KnowledgeClient(http_client, project_id)
entity = await client.create_entity(entity_data)
"""
def __init__(self, http_client: AsyncClient, project_id: str):
"""Initialize the knowledge client.
Args:
http_client: HTTPX AsyncClient for making requests
project_id: Project external_id (UUID) for API calls
"""
self.http_client = http_client
self.project_id = project_id
self._base_path = f"/v2/projects/{project_id}/knowledge"
# --- Entity CRUD Operations ---
async def create_entity(self, entity_data: dict[str, Any]) -> EntityResponse:
"""Create a new entity.
Args:
entity_data: Entity data including title, content, folder, etc.
Returns:
EntityResponse with created entity details
Raises:
ToolError: If the request fails
"""
response = await call_post(
self.http_client,
f"{self._base_path}/entities",
json=entity_data,
)
return EntityResponse.model_validate(response.json())
async def update_entity(self, entity_id: str, entity_data: dict[str, Any]) -> EntityResponse:
"""Update an existing entity (full replacement).
Args:
entity_id: Entity external_id (UUID)
entity_data: Complete entity data for replacement
Returns:
EntityResponse with updated entity details
Raises:
ToolError: If the request fails
"""
response = await call_put(
self.http_client,
f"{self._base_path}/entities/{entity_id}",
json=entity_data,
)
return EntityResponse.model_validate(response.json())
async def get_entity(self, entity_id: str) -> EntityResponse:
"""Get an entity by ID.
Args:
entity_id: Entity external_id (UUID)
Returns:
EntityResponse with entity details
Raises:
ToolError: If the entity is not found or request fails
"""
response = await call_get(
self.http_client,
f"{self._base_path}/entities/{entity_id}",
)
return EntityResponse.model_validate(response.json())
async def patch_entity(self, entity_id: str, patch_data: dict[str, Any]) -> EntityResponse:
"""Partially update an entity.
Args:
entity_id: Entity external_id (UUID)
patch_data: Partial entity data to update
Returns:
EntityResponse with updated entity details
Raises:
ToolError: If the request fails
"""
response = await call_patch(
self.http_client,
f"{self._base_path}/entities/{entity_id}",
json=patch_data,
)
return EntityResponse.model_validate(response.json())
async def delete_entity(self, entity_id: str) -> DeleteEntitiesResponse:
"""Delete an entity.
Args:
entity_id: Entity external_id (UUID)
Returns:
DeleteEntitiesResponse confirming deletion
Raises:
ToolError: If the entity is not found or request fails
"""
response = await call_delete(
self.http_client,
f"{self._base_path}/entities/{entity_id}",
)
return DeleteEntitiesResponse.model_validate(response.json())
async def move_entity(self, entity_id: str, destination_path: str) -> EntityResponse:
"""Move an entity to a new location.
Args:
entity_id: Entity external_id (UUID)
destination_path: New file path for the entity
Returns:
EntityResponse with updated entity details
Raises:
ToolError: If the request fails
"""
response = await call_put(
self.http_client,
f"{self._base_path}/entities/{entity_id}/move",
json={"destination_path": destination_path},
)
return EntityResponse.model_validate(response.json())
# --- Resolution ---
async def resolve_entity(self, identifier: str) -> str:
"""Resolve a string identifier to an entity external_id.
Args:
identifier: The identifier to resolve (permalink, title, or path)
Returns:
The resolved entity external_id (UUID)
Raises:
ToolError: If the identifier cannot be resolved
"""
response = await call_post(
self.http_client,
f"{self._base_path}/resolve",
json={"identifier": identifier},
)
data = response.json()
return data["external_id"]
+120
View File
@@ -0,0 +1,120 @@
"""Typed client for memory/context API operations.
Encapsulates all /v2/projects/{project_id}/memory/* endpoints.
"""
from typing import Optional
from httpx import AsyncClient
from basic_memory.mcp.tools.utils import call_get
from basic_memory.schemas.memory import GraphContext
class MemoryClient:
"""Typed client for memory context operations.
Centralizes:
- API path construction for /v2/projects/{project_id}/memory/*
- Response validation via Pydantic models
- Consistent error handling through call_* utilities
Usage:
async with get_client() as http_client:
client = MemoryClient(http_client, project_id)
context = await client.build_context("memory://specs/search")
"""
def __init__(self, http_client: AsyncClient, project_id: str):
"""Initialize the memory client.
Args:
http_client: HTTPX AsyncClient for making requests
project_id: Project external_id (UUID) for API calls
"""
self.http_client = http_client
self.project_id = project_id
self._base_path = f"/v2/projects/{project_id}/memory"
async def build_context(
self,
path: str,
*,
depth: int = 1,
timeframe: Optional[str] = None,
page: int = 1,
page_size: int = 10,
max_related: int = 10,
) -> GraphContext:
"""Build context from a memory path.
Args:
path: The path to build context for (without memory:// prefix)
depth: How deep to traverse relations
timeframe: Time filter (e.g., "7d", "1 week")
page: Page number (1-indexed)
page_size: Results per page
max_related: Maximum related items per result
Returns:
GraphContext with hierarchical results
Raises:
ToolError: If the request fails
"""
params: dict = {
"depth": depth,
"page": page,
"page_size": page_size,
"max_related": max_related,
}
if timeframe:
params["timeframe"] = timeframe
response = await call_get(
self.http_client,
f"{self._base_path}/{path}",
params=params,
)
return GraphContext.model_validate(response.json())
async def recent(
self,
*,
timeframe: str = "7d",
depth: int = 1,
types: Optional[list[str]] = None,
page: int = 1,
page_size: int = 10,
) -> GraphContext:
"""Get recent activity.
Args:
timeframe: Time filter (e.g., "7d", "1 week", "2 days ago")
depth: How deep to traverse relations
types: Filter by item types
page: Page number (1-indexed)
page_size: Results per page
Returns:
GraphContext with recent activity
Raises:
ToolError: If the request fails
"""
params: dict = {
"timeframe": timeframe,
"depth": depth,
"page": page,
"page_size": page_size,
}
if types:
# Join types as comma-separated string if provided
params["type"] = ",".join(types) if isinstance(types, list) else types
response = await call_get(
self.http_client,
f"{self._base_path}/recent",
params=params,
)
return GraphContext.model_validate(response.json())
+89
View File
@@ -0,0 +1,89 @@
"""Typed client for project API operations.
Encapsulates project-level endpoints.
"""
from typing import Any
from httpx import AsyncClient
from basic_memory.mcp.tools.utils import call_get, call_post, call_delete
from basic_memory.schemas.project_info import ProjectList, ProjectStatusResponse
class ProjectClient:
"""Typed client for project management operations.
Centralizes:
- API path construction for project endpoints
- Response validation via Pydantic models
- Consistent error handling through call_* utilities
Note: This client does not require a project_id since it operates
across projects.
Usage:
async with get_client() as http_client:
client = ProjectClient(http_client)
projects = await client.list_projects()
"""
def __init__(self, http_client: AsyncClient):
"""Initialize the project client.
Args:
http_client: HTTPX AsyncClient for making requests
"""
self.http_client = http_client
async def list_projects(self) -> ProjectList:
"""List all available projects.
Returns:
ProjectList with all projects and default project name
Raises:
ToolError: If the request fails
"""
response = await call_get(
self.http_client,
"/projects/projects",
)
return ProjectList.model_validate(response.json())
async def create_project(self, project_data: dict[str, Any]) -> ProjectStatusResponse:
"""Create a new project.
Args:
project_data: Project creation data (name, path, set_default)
Returns:
ProjectStatusResponse with creation result
Raises:
ToolError: If the request fails
"""
response = await call_post(
self.http_client,
"/projects/projects",
json=project_data,
)
return ProjectStatusResponse.model_validate(response.json())
async def delete_project(self, project_external_id: str) -> ProjectStatusResponse:
"""Delete a project by its external ID.
Args:
project_external_id: Project external ID (UUID)
Returns:
ProjectStatusResponse with deletion result
Raises:
ToolError: If the request fails
"""
response = await call_delete(
self.http_client,
f"/v2/projects/{project_external_id}",
)
return ProjectStatusResponse.model_validate(response.json())
+71
View File
@@ -0,0 +1,71 @@
"""Typed client for resource API operations.
Encapsulates all /v2/projects/{project_id}/resource/* endpoints.
"""
from typing import Optional
from httpx import AsyncClient, Response
from basic_memory.mcp.tools.utils import call_get
class ResourceClient:
"""Typed client for resource operations.
Centralizes:
- API path construction for /v2/projects/{project_id}/resource/*
- Consistent error handling through call_* utilities
Note: This client returns raw Response objects for resources since they
may be text, images, or other binary content that needs special handling.
Usage:
async with get_client() as http_client:
client = ResourceClient(http_client, project_id)
response = await client.read(entity_id)
text = response.text
"""
def __init__(self, http_client: AsyncClient, project_id: str):
"""Initialize the resource client.
Args:
http_client: HTTPX AsyncClient for making requests
project_id: Project external_id (UUID) for API calls
"""
self.http_client = http_client
self.project_id = project_id
self._base_path = f"/v2/projects/{project_id}/resource"
async def read(
self,
entity_id: str,
*,
page: Optional[int] = None,
page_size: Optional[int] = None,
) -> Response:
"""Read a resource by entity ID.
Args:
entity_id: Entity external_id (UUID)
page: Optional page number for paginated content
page_size: Optional page size for paginated content
Returns:
Raw HTTP Response (caller handles text/binary content)
Raises:
ToolError: If the resource is not found or request fails
"""
params: dict = {}
if page is not None:
params["page"] = page
if page_size is not None:
params["page_size"] = page_size
return await call_get(
self.http_client,
f"{self._base_path}/{entity_id}",
params=params if params else None,
)
+65
View File
@@ -0,0 +1,65 @@
"""Typed client for search API operations.
Encapsulates all /v2/projects/{project_id}/search/* endpoints.
"""
from typing import Any
from httpx import AsyncClient
from basic_memory.mcp.tools.utils import call_post
from basic_memory.schemas.search import SearchResponse
class SearchClient:
"""Typed client for search operations.
Centralizes:
- API path construction for /v2/projects/{project_id}/search/*
- Response validation via Pydantic models
- Consistent error handling through call_* utilities
Usage:
async with get_client() as http_client:
client = SearchClient(http_client, project_id)
results = await client.search(search_query.model_dump())
"""
def __init__(self, http_client: AsyncClient, project_id: str):
"""Initialize the search client.
Args:
http_client: HTTPX AsyncClient for making requests
project_id: Project external_id (UUID) for API calls
"""
self.http_client = http_client
self.project_id = project_id
self._base_path = f"/v2/projects/{project_id}/search"
async def search(
self,
query: dict[str, Any],
*,
page: int = 1,
page_size: int = 10,
) -> SearchResponse:
"""Search across all content in the knowledge base.
Args:
query: Search query dict (from SearchQuery.model_dump())
page: Page number (1-indexed)
page_size: Results per page
Returns:
SearchResponse with results and pagination
Raises:
ToolError: If the request fails
"""
response = await call_post(
self.http_client,
f"{self._base_path}/",
json=query,
params={"page": page, "page_size": page_size},
)
return SearchResponse.model_validate(response.json())
+108
View File
@@ -0,0 +1,108 @@
"""MCP composition root for Basic Memory.
This container owns reading ConfigManager and environment variables for the
MCP server entrypoint. Downstream modules receive config/dependencies explicitly
rather than reading globals.
Design principles:
- Only this module reads ConfigManager directly
- Runtime mode (cloud/local/test) is resolved here
- File sync decisions are centralized here
"""
from dataclasses import dataclass
from typing import TYPE_CHECKING
from basic_memory.config import BasicMemoryConfig, ConfigManager
from basic_memory.runtime import RuntimeMode, resolve_runtime_mode
if TYPE_CHECKING: # pragma: no cover
from basic_memory.sync import SyncCoordinator
@dataclass
class McpContainer:
"""Composition root for the MCP server entrypoint.
Holds resolved configuration and runtime context.
Created once at server startup, then used to wire dependencies.
"""
config: BasicMemoryConfig
mode: RuntimeMode
@classmethod
def create(cls) -> "McpContainer":
"""Create container by reading ConfigManager.
This is the single point where MCP reads global config.
"""
config = ConfigManager().config
mode = resolve_runtime_mode(
cloud_mode_enabled=config.cloud_mode_enabled,
is_test_env=config.is_test_env,
)
return cls(config=config, mode=mode)
# --- Runtime Mode Properties ---
@property
def should_sync_files(self) -> bool:
"""Whether local file sync should be started.
Sync is enabled when:
- sync_changes is True in config
- Not in test mode (tests manage their own sync)
- Not in cloud mode (cloud handles sync differently)
"""
return self.config.sync_changes and not self.mode.is_test and not self.mode.is_cloud
@property
def sync_skip_reason(self) -> str | None:
"""Reason why sync is skipped, or None if sync should run.
Useful for logging why sync was disabled.
"""
if self.mode.is_test:
return "Test environment detected"
if self.mode.is_cloud:
return "Cloud mode enabled"
if not self.config.sync_changes:
return "Sync changes disabled"
return None
def create_sync_coordinator(self) -> "SyncCoordinator":
"""Create a SyncCoordinator with this container's settings.
Returns:
SyncCoordinator configured for this runtime environment
"""
# Deferred import to avoid circular dependency
from basic_memory.sync import SyncCoordinator
return SyncCoordinator(
config=self.config,
should_sync=self.should_sync_files,
skip_reason=self.sync_skip_reason,
)
# Module-level container instance (set by lifespan)
_container: McpContainer | None = None
def get_container() -> McpContainer:
"""Get the current MCP container.
Raises:
RuntimeError: If container hasn't been initialized
"""
if _container is None:
raise RuntimeError("MCP container not initialized. Call set_container() first.")
return _container
def set_container(container: McpContainer) -> None:
"""Set the MCP container (called by lifespan)."""
global _container
_container = container
+35 -33
View File
@@ -2,9 +2,12 @@
Provides project lookup utilities for MCP tools.
Handles project validation and context management in one place.
Note: This module uses ProjectResolver for unified project resolution.
The resolve_project_parameter function is a thin wrapper for backwards
compatibility with existing MCP tools.
"""
import os
from typing import Optional, List
from httpx import AsyncClient
from httpx._types import (
@@ -14,16 +17,25 @@ from loguru import logger
from fastmcp import Context
from basic_memory.config import ConfigManager
from basic_memory.project_resolver import ProjectResolver
from basic_memory.schemas.project_info import ProjectItem, ProjectList
from basic_memory.utils import generate_permalink
async def resolve_project_parameter(
project: Optional[str] = None, allow_discovery: bool = False
project: Optional[str] = None,
allow_discovery: bool = False,
cloud_mode: Optional[bool] = None,
default_project_mode: Optional[bool] = None,
default_project: Optional[str] = None,
) -> Optional[str]:
"""Resolve project parameter using three-tier hierarchy.
if config.cloud_mode:
This is a thin wrapper around ProjectResolver for backwards compatibility.
New code should consider using ProjectResolver directly for more detailed
resolution information.
if cloud_mode:
project is required (unless allow_discovery=True for tools that support discovery mode)
else:
Resolution order:
@@ -35,41 +47,31 @@ async def resolve_project_parameter(
project: Optional explicit project parameter
allow_discovery: If True, allows returning None in cloud mode for discovery mode
(used by tools like recent_activity that can operate across all projects)
cloud_mode: Optional explicit cloud mode. If not provided, reads from ConfigManager.
default_project_mode: Optional explicit default project mode. If not provided, reads from ConfigManager.
default_project: Optional explicit default project. If not provided, reads from ConfigManager.
Returns:
Resolved project name or None if no resolution possible
"""
# Load config for any values not explicitly provided
if cloud_mode is None or default_project_mode is None or default_project is None:
config = ConfigManager().config
if cloud_mode is None:
cloud_mode = config.cloud_mode
if default_project_mode is None:
default_project_mode = config.default_project_mode
if default_project is None:
default_project = config.default_project
config = ConfigManager().config
# if cloud_mode, project is required (unless discovery mode is allowed)
if config.cloud_mode:
if project:
logger.debug(f"project: {project}, cloud_mode: {config.cloud_mode}")
return project
elif allow_discovery:
logger.debug("cloud_mode: discovery mode allowed, returning None")
return None
else:
raise ValueError("No project specified. Project is required for cloud mode.")
# Priority 1: CLI constraint overrides everything (--project arg sets env var)
constrained_project = os.environ.get("BASIC_MEMORY_MCP_PROJECT")
if constrained_project:
logger.debug(f"Using CLI constrained project: {constrained_project}")
return constrained_project
# Priority 2: Explicit project parameter
if project:
logger.debug(f"Using explicit project parameter: {project}")
return project
# Priority 3: Default project mode
if config.default_project_mode:
logger.debug(f"Using default project from config: {config.default_project}")
return config.default_project
# No resolution possible
return None
# Create resolver with configuration and resolve
resolver = ProjectResolver.from_env(
cloud_mode=cloud_mode,
default_project_mode=default_project_mode,
default_project=default_project,
)
result = resolver.resolve(project=project, allow_discovery=allow_discovery)
return result.project
async def get_project_names(client: AsyncClient, headers: HeaderTypes | None = None) -> List[str]:
+16 -36
View File
@@ -2,16 +2,14 @@
Basic Memory FastMCP server.
"""
import asyncio
from contextlib import asynccontextmanager
from fastmcp import FastMCP
from loguru import logger
from basic_memory import db
from basic_memory.config import ConfigManager
from basic_memory.services.initialization import initialize_app, initialize_file_sync
from basic_memory.telemetry import show_notice_if_needed, track_app_started
from basic_memory.mcp.container import McpContainer, set_container
from basic_memory.services.initialization import initialize_app
@asynccontextmanager
@@ -20,16 +18,15 @@ async def lifespan(app: FastMCP):
Handles:
- Database initialization and migrations
- Telemetry notice and tracking
- File sync in background (if enabled and not in cloud mode)
- File sync via SyncCoordinator (if enabled and not in cloud mode)
- Proper cleanup on shutdown
"""
app_config = ConfigManager().config
logger.info("Starting Basic Memory MCP server")
# --- Composition Root ---
# Create container and read config (single point of config access)
container = McpContainer.create()
set_container(container)
# Show telemetry notice (first run only) and track startup
show_notice_if_needed()
track_app_started("mcp")
logger.debug(f"Starting Basic Memory MCP server (mode={container.mode.name})")
# Track if we created the engine (vs test fixtures providing it)
# This prevents disposing an engine provided by test fixtures when
@@ -37,40 +34,23 @@ async def lifespan(app: FastMCP):
engine_was_none = db._engine is None
# Initialize app (runs migrations, reconciles projects)
await initialize_app(app_config)
await initialize_app(container.config)
# Start file sync as background task (if enabled and not in cloud mode)
sync_task = None
if app_config.is_test_env:
logger.info("Test environment detected - skipping local file sync")
elif app_config.sync_changes and not app_config.cloud_mode_enabled: # pragma: no cover
logger.info("Starting file sync in background")
async def _file_sync_runner() -> None:
await initialize_file_sync(app_config)
sync_task = asyncio.create_task(_file_sync_runner())
elif app_config.cloud_mode_enabled: # pragma: no cover
logger.info("Cloud mode enabled - skipping local file sync")
else: # pragma: no cover
logger.info("Sync changes disabled - skipping file sync")
# Create and start sync coordinator (lifecycle centralized in coordinator)
sync_coordinator = container.create_sync_coordinator()
await sync_coordinator.start()
try:
yield
finally:
# Shutdown
logger.info("Shutting down Basic Memory MCP server")
if sync_task: # pragma: no cover
sync_task.cancel()
try:
await sync_task
except asyncio.CancelledError:
logger.info("File sync task cancelled")
# Shutdown - coordinator handles clean task cancellation
logger.debug("Shutting down Basic Memory MCP server")
await sync_coordinator.stop()
# Only shutdown DB if we created it (not if test fixture provided it)
if engine_was_none:
await db.shutdown_db()
logger.info("Database connections closed")
logger.debug("Database connections closed")
else: # pragma: no cover
logger.debug("Skipping DB shutdown - engine provided externally")
+12 -14
View File
@@ -8,8 +8,6 @@ from fastmcp import Context
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.project_context import get_active_project
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_get
from basic_memory.telemetry import track_mcp_tool
from basic_memory.schemas.base import TimeFrame
from basic_memory.schemas.memory import (
GraphContext,
@@ -88,7 +86,6 @@ async def build_context(
Raises:
ToolError: If project doesn't exist or depth parameter is invalid
"""
track_mcp_tool("build_context")
logger.info(f"Building context from {url} in project {project}")
# Convert string depth to integer if needed
@@ -106,15 +103,16 @@ async def build_context(
# Get the active project using the new stateless approach
active_project = await get_active_project(client, project, context)
response = await call_get(
client,
f"/v2/projects/{active_project.external_id}/memory/{memory_url_path(url)}",
params={
"depth": depth,
"timeframe": timeframe,
"page": page,
"page_size": page_size,
"max_related": max_related,
},
# Import here to avoid circular import
from basic_memory.mcp.clients import MemoryClient
# Use typed MemoryClient for API calls
memory_client = MemoryClient(client, active_project.external_id)
return await memory_client.build_context(
memory_url_path(url),
depth=depth or 1,
timeframe=timeframe,
page=page,
page_size=page_size,
max_related=max_related,
)
return GraphContext.model_validate(response.json())
+3 -3
View File
@@ -13,7 +13,6 @@ from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.project_context import get_active_project
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_put, call_post, resolve_entity_id
from basic_memory.telemetry import track_mcp_tool
@mcp.tool(
@@ -95,7 +94,6 @@ async def canvas(
Raises:
ToolError: If project doesn't exist or folder path is invalid
"""
track_mcp_tool("canvas")
async with get_client() as client:
active_project = await get_active_project(client, project, context)
@@ -127,7 +125,9 @@ async def canvas(
):
logger.info(f"Canvas file exists, updating instead: {file_path}")
try:
entity_id = await resolve_entity_id(client, active_project.external_id, file_path)
entity_id = await resolve_entity_id(
client, active_project.external_id, file_path
)
# For update, send content in JSON body
response = await call_put(
client,
@@ -15,7 +15,6 @@ from basic_memory.mcp.tools.search import search_notes
from basic_memory.mcp.tools.read_note import read_note
from basic_memory.schemas.search import SearchResponse
from basic_memory.config import ConfigManager
from basic_memory.telemetry import track_mcp_tool
def _format_search_results_for_chatgpt(results: SearchResponse) -> List[Dict[str, Any]]:
@@ -89,7 +88,6 @@ async def search(
List with one dict: `{ "type": "text", "text": "{...JSON...}" }`
where the JSON body contains `results`, `total_count`, and echo of `query`.
"""
track_mcp_tool("search")
logger.info(f"ChatGPT search request: query='{query}'")
try:
@@ -153,7 +151,6 @@ async def fetch(
List with one dict: `{ "type": "text", "text": "{...JSON...}" }`
where the JSON body includes `id`, `title`, `text`, `url`, and metadata.
"""
track_mcp_tool("fetch")
logger.info(f"ChatGPT fetch request: id='{id}'")
try:
+8 -9
View File
@@ -6,11 +6,8 @@ from fastmcp import Context
from mcp.server.fastmcp.exceptions import ToolError
from basic_memory.mcp.project_context import get_active_project
from basic_memory.mcp.tools.utils import call_delete, resolve_entity_id
from basic_memory.mcp.server import mcp
from basic_memory.mcp.async_client import get_client
from basic_memory.telemetry import track_mcp_tool
from basic_memory.schemas import DeleteEntitiesResponse
def _format_delete_error_response(project: str, error_message: str, identifier: str) -> str:
@@ -204,13 +201,18 @@ async def delete_note(
with suggestions for finding the correct identifier, including search
commands and alternative formats to try.
"""
track_mcp_tool("delete_note")
async with get_client() as client:
active_project = await get_active_project(client, project, context)
# Import here to avoid circular import
from basic_memory.mcp.clients import KnowledgeClient
# Use typed KnowledgeClient for API calls
knowledge_client = KnowledgeClient(client, active_project.external_id)
try:
# Resolve identifier to entity ID
entity_id = await resolve_entity_id(client, active_project.external_id, identifier)
entity_id = await knowledge_client.resolve_entity(identifier)
except ToolError as e:
# If entity not found, return False (note doesn't exist)
if "Entity not found" in str(e) or "not found" in str(e).lower():
@@ -226,10 +228,7 @@ async def delete_note(
try:
# Call the DELETE endpoint
response = await call_delete(
client, f"/v2/projects/{active_project.external_id}/knowledge/entities/{entity_id}"
)
result = DeleteEntitiesResponse.model_validate(response.json())
result = await knowledge_client.delete_entity(entity_id)
if result.deleted:
logger.info(
+10 -11
View File
@@ -8,9 +8,6 @@ from fastmcp import Context
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.project_context import get_active_project, add_project_metadata
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_patch, resolve_entity_id
from basic_memory.telemetry import track_mcp_tool
from basic_memory.schemas import EntityResponse
def _format_error_response(
@@ -215,7 +212,6 @@ async def edit_note(
search_notes() first to find the correct identifier. The tool provides detailed
error messages with suggestions if operations fail.
"""
track_mcp_tool("edit_note")
async with get_client() as client:
active_project = await get_active_project(client, project, context)
@@ -236,8 +232,14 @@ async def edit_note(
# Use the PATCH endpoint to edit the entity
try:
# Import here to avoid circular import
from basic_memory.mcp.clients import KnowledgeClient
# Use typed KnowledgeClient for API calls
knowledge_client = KnowledgeClient(client, active_project.external_id)
# Resolve identifier to entity ID
entity_id = await resolve_entity_id(client, active_project.external_id, identifier)
entity_id = await knowledge_client.resolve_entity(identifier)
# Prepare the edit request data
edit_data = {
@@ -254,9 +256,7 @@ async def edit_note(
edit_data["expected_replacements"] = str(expected_replacements)
# Call the PATCH endpoint
url = f"/v2/projects/{active_project.external_id}/knowledge/entities/{entity_id}"
response = await call_patch(client, url, json=edit_data)
result = EntityResponse.model_validate(response.json())
result = await knowledge_client.patch_entity(entity_id, edit_data)
# Format summary
summary = [
@@ -311,11 +311,10 @@ async def edit_note(
permalink=result.permalink,
observations_count=len(result.observations),
relations_count=len(result.relations),
status_code=response.status_code,
)
result = "\n".join(summary)
return add_project_metadata(result, active_project.name)
summary_result = "\n".join(summary)
return add_project_metadata(summary_result, active_project.name)
except Exception as e:
logger.error(f"Error editing note: {e}")
+5 -18
View File
@@ -8,8 +8,6 @@ from fastmcp import Context
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.project_context import get_active_project
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_get
from basic_memory.telemetry import track_mcp_tool
@mcp.tool(
@@ -64,30 +62,19 @@ async def list_directory(
Raises:
ToolError: If project doesn't exist or directory path is invalid
"""
track_mcp_tool("list_directory")
async with get_client() as client:
active_project = await get_active_project(client, project, context)
# Prepare query parameters
params = {
"dir_name": dir_name,
"depth": str(depth),
}
if file_name_glob:
params["file_name_glob"] = file_name_glob
logger.debug(
f"Listing directory '{dir_name}' in project {project} with depth={depth}, glob='{file_name_glob}'"
)
# Call the API endpoint
response = await call_get(
client,
f"/v2/projects/{active_project.external_id}/directory/list",
params=params,
)
# Import here to avoid circular import
from basic_memory.mcp.clients import DirectoryClient
nodes = response.json()
# Use typed DirectoryClient for API calls
directory_client = DirectoryClient(client, active_project.external_id)
nodes = await directory_client.list(dir_name, depth=depth, file_name_glob=file_name_glob)
if not nodes:
filter_desc = ""
+19 -27
View File
@@ -8,11 +8,7 @@ from fastmcp import Context
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_get, call_put, resolve_entity_id
from basic_memory.mcp.project_context import get_active_project
from basic_memory.schemas import EntityResponse
from basic_memory.schemas.project_info import ProjectList
from basic_memory.telemetry import track_mcp_tool
from basic_memory.utils import validate_project_path
@@ -31,9 +27,12 @@ async def _detect_cross_project_move_attempt(
Error message with guidance if cross-project move is detected, None otherwise
"""
try:
# Get list of all available projects to check against
response = await call_get(client, "/projects/projects")
project_list = ProjectList.model_validate(response.json())
# Import here to avoid circular import
from basic_memory.mcp.clients import ProjectClient
# Use typed ProjectClient for API calls
project_client = ProjectClient(client)
project_list = await project_client.list_projects()
project_names = [p.name.lower() for p in project_list.projects]
# Check if destination path contains any project names
@@ -400,7 +399,6 @@ async def move_note(
- Re-indexes the entity for search
- Maintains all observations and relations
"""
track_mcp_tool("move_note")
async with get_client() as client:
logger.debug(f"Moving note: {identifier} to {destination_path} in project: {project}")
@@ -436,15 +434,19 @@ move_note("{identifier}", "notes/{destination_path.split("/")[-1] if "/" in dest
logger.info(f"Detected cross-project move attempt: {identifier} -> {destination_path}")
return cross_project_error
# Import here to avoid circular import
from basic_memory.mcp.clients import KnowledgeClient
# Use typed KnowledgeClient for API calls
knowledge_client = KnowledgeClient(client, active_project.external_id)
# Get the source entity information for extension validation
source_ext = "md" # Default to .md if we can't determine source extension
try:
# Resolve identifier to entity ID
entity_id = await resolve_entity_id(client, active_project.external_id, identifier)
entity_id = await knowledge_client.resolve_entity(identifier)
# Fetch source entity information to get the current file extension
url = f"/v2/projects/{active_project.external_id}/knowledge/entities/{entity_id}"
response = await call_get(client, url)
source_entity = EntityResponse.model_validate(response.json())
source_entity = await knowledge_client.get_entity(entity_id)
if "." in source_entity.file_path:
source_ext = source_entity.file_path.split(".")[-1]
except Exception as e:
@@ -475,11 +477,9 @@ move_note("{identifier}", "notes/{destination_path.split("/")[-1] if "/" in dest
# Get the source entity to check its file extension
try:
# Resolve identifier to entity ID (might already be cached from above)
entity_id = await resolve_entity_id(client, active_project.external_id, identifier)
entity_id = await knowledge_client.resolve_entity(identifier)
# Fetch source entity information
url = f"/v2/projects/{active_project.external_id}/knowledge/entities/{entity_id}"
response = await call_get(client, url)
source_entity = EntityResponse.model_validate(response.json())
source_entity = await knowledge_client.get_entity(entity_id)
# Extract file extensions
source_ext = (
@@ -515,17 +515,10 @@ move_note("{identifier}", "notes/{destination_path.split("/")[-1] if "/" in dest
try:
# Resolve identifier to entity ID for the move operation
entity_id = await resolve_entity_id(client, active_project.external_id, identifier)
entity_id = await knowledge_client.resolve_entity(identifier)
# Prepare move request (v2 API only needs destination_path)
move_data = {
"destination_path": destination_path,
}
# Call the v2 move API endpoint (PUT method, entity_id in URL)
url = f"/v2/projects/{active_project.external_id}/knowledge/entities/{entity_id}/move"
response = await call_put(client, url, json=move_data)
result = EntityResponse.model_validate(response.json())
# Call the move API using KnowledgeClient
result = await knowledge_client.move_entity(entity_id, destination_path)
# Build success message
result_lines = [
@@ -544,7 +537,6 @@ move_note("{identifier}", "notes/{destination_path.split("/")[-1] if "/" in dest
identifier=identifier,
destination_path=destination_path,
project=active_project.name,
status_code=response.status_code,
)
return "\n".join(result_lines)

Some files were not shown because too many files have changed in this diff Show More