Compare commits

..

1 Commits

Author SHA1 Message Date
phernandez e26d0df2ed update deps to fastmcp-2.10
Signed-off-by: phernandez <paul@basicmachines.co>
2025-07-03 13:33:38 -05:00
603 changed files with 20328 additions and 93735 deletions
+18 -125
View File
@@ -15,16 +15,10 @@ Create a stable release using the automated justfile target with comprehensive v
You are an expert release manager for the Basic Memory project. When the user runs `/release`, execute the following steps:
### Step 1: Pre-flight Validation
#### Version Check
1. Check current version in `src/basic_memory/__init__.py`
2. Verify new version format matches `v\d+\.\d+\.\d+` pattern
3. Confirm version is higher than current version
#### Git Status
1. Check current git status for uncommitted changes
2. Verify we're on the `main` branch
3. Confirm no existing tag with this version
1. Verify version format matches `v\d+\.\d+\.\d+` pattern
2. Check current git status for uncommitted changes
3. Verify we're on the `main` branch
4. Confirm no existing tag with this version
#### Documentation Validation
1. **Changelog Check**
@@ -45,111 +39,19 @@ The justfile target handles:
- ✅ Version update in `src/basic_memory/__init__.py`
- ✅ Automatic commit with proper message
- ✅ Tag creation and pushing to GitHub
- ✅ Release workflow trigger (automatic on tag push)
The GitHub Actions workflow (`.github/workflows/release.yml`) then:
- ✅ Builds the package using `uv build`
- ✅ Creates GitHub release with auto-generated notes
- ✅ Publishes to PyPI
- ✅ Updates Homebrew formula (stable releases only)
- ✅ Release workflow trigger
### Step 3: Monitor Release Process
1. Verify tag push triggered the workflow (should start automatically within seconds)
2. Monitor workflow progress at: https://github.com/basicmachines-co/basic-memory/actions
3. Watch for successful completion of both jobs:
- `release` - Builds package and publishes to PyPI
- `homebrew` - Updates Homebrew formula (stable releases only)
4. Check for any workflow failures and investigate logs if needed
1. Check that GitHub Actions workflow starts successfully
2. Monitor workflow completion at: https://github.com/basicmachines-co/basic-memory/actions
3. Verify PyPI publication
4. Test installation: `uv tool install basic-memory`
### Step 4: Post-Release Validation
#### GitHub Release
1. Verify GitHub release is created at: https://github.com/basicmachines-co/basic-memory/releases/tag/<version>
2. Check that release notes are auto-generated from commits
3. Validate release assets (`.whl` and `.tar.gz` files are attached)
#### PyPI Publication
1. Verify package published at: https://pypi.org/project/basic-memory/<version>/
2. Test installation: `uv tool install basic-memory`
3. Verify installed version: `basic-memory --version`
#### Homebrew Formula (Stable Releases Only)
1. Check formula update at: https://github.com/basicmachines-co/homebrew-basic-memory
2. Verify formula version matches release
3. Test Homebrew installation: `brew install basicmachines-co/basic-memory/basic-memory`
#### MCP Registry Publication
After PyPI release is published, update the MCP registry:
1. **Verify PyPI Release**
- Confirm package is live: https://pypi.org/project/basic-memory/<version>/
- The `server.json` version was auto-updated by `just release`
2. **Publish to MCP Registry**
```bash
cd /Users/drew/code/basic-memory
mcp-publisher publish
```
If not authenticated:
```bash
mcp-publisher login github
# Follow device authentication flow
mcp-publisher publish
```
3. **Verify Publication**
```bash
curl "https://registry.modelcontextprotocol.io/v0.1/servers?search=basic-memory"
```
**Note:** The `mcp-publisher` CLI can be installed via Homebrew (`brew install mcp-publisher`) or from GitHub releases.
#### Website Updates
**1. basicmachines.co** (`/Users/drew/code/basicmachines.co`)
- **Goal**: Update version number displayed on the homepage
- **Location**: Search for "Basic Memory v0." in the codebase to find version displays
- **What to update**:
- Hero section heading that shows "Basic Memory v{VERSION}"
- "What's New in v{VERSION}" section heading
- Feature highlights array (look for array of features with title/description)
- **Process**:
1. Pull latest from GitHub: `git pull origin main`
2. Create release branch: `git checkout -b release/v{VERSION}`
3. Search codebase for current version number (e.g., "v0.16.1")
4. Update version numbers to new release version
5. Update feature highlights with 3-5 key features from this release (extract from CHANGELOG.md)
6. Commit changes: `git commit -m "chore: update to v{VERSION}"`
7. Push branch: `git push origin release/v{VERSION}`
- **Deploy**: Follow deployment process for basicmachines.co
**2. docs.basicmemory.com** (`/Users/drew/code/docs.basicmemory.com`)
- **Goal**: Add new release notes section to the latest-releases page
- **File**: `src/pages/latest-releases.mdx`
- **What to do**:
1. Pull latest from GitHub: `git pull origin main`
2. Create release branch: `git checkout -b release/v{VERSION}`
3. Read the existing file to understand the format and structure
4. Read `/Users/drew/code/basic-memory/CHANGELOG.md` to get release content
5. Add new release section **at the top** (after MDX imports, before other releases)
6. Follow the existing pattern:
- Heading: `## [v{VERSION}](github-link) — YYYY-MM-DD`
- Focus statement if applicable
- `<Info>` block with highlights (3-5 key items)
- Sections for Features, Bug Fixes, Breaking Changes, etc.
- Link to full changelog at the end
- Separator `---` between releases
7. Commit changes: `git commit -m "docs: add v{VERSION} release notes"`
8. Push branch: `git push origin release/v{VERSION}`
- **Source content**: Extract and format sections from CHANGELOG.md for this version
- **Deploy**: Follow deployment process for docs.basicmemory.com
**4. Announce Release**
- Post to Discord community if significant changes
- Update social media if major release
- Notify users via appropriate channels
1. Verify GitHub release is created automatically
2. Check PyPI publication
3. Validate release assets
4. Update any post-release documentation
## Pre-conditions Check
Before starting, verify:
@@ -172,28 +74,19 @@ Before starting, verify:
🏷️ Tag: v0.13.2
📋 GitHub Release: https://github.com/basicmachines-co/basic-memory/releases/tag/v0.13.2
📦 PyPI: https://pypi.org/project/basic-memory/0.13.2/
🍺 Homebrew: https://github.com/basicmachines-co/homebrew-basic-memory
🔌 MCP Registry: https://registry.modelcontextprotocol.io
🚀 GitHub Actions: Completed
Install with pip/uv:
uv tool install basic-memory
Install with Homebrew:
brew install basicmachines-co/basic-memory/basic-memory
Install with:
uv tool install basic-memory
Users can now upgrade:
uv tool upgrade basic-memory
brew upgrade basic-memory
uv tool upgrade basic-memory
```
## Context
- This creates production releases used by end users
- Must pass all quality gates before proceeding
- Uses the automated justfile target for consistency
- Version is automatically updated in `__init__.py` and `server.json`
- Version is automatically updated in `__init__.py`
- Triggers automated GitHub release with changelog
- Package is published to PyPI for `pip` and `uv` users
- Homebrew formula is automatically updated for stable releases
- MCP Registry is updated manually via `mcp-publisher publish`
- Supports multiple installation methods (uv, pip, Homebrew)
- Leverages uv-dynamic-versioning for package version management
-51
View File
@@ -1,51 +0,0 @@
---
allowed-tools: mcp__basic-memory__write_note, mcp__basic-memory__read_note, mcp__basic-memory__search_notes, mcp__basic-memory__edit_note
argument-hint: [create|status|show|review] [spec-name]
description: Manage specifications in our development process
---
## Context
Specifications are managed in the Basic Memory "specs" project. All specs live in a centralized location accessible across all repositories via MCP tools.
See SPEC-1 and SPEC-2 in the "specs" project for the full specification-driven development process.
Available commands:
- `create [name]` - Create new specification
- `status` - Show all spec statuses
- `show [spec-name]` - Read a specific spec
- `review [spec-name]` - Review implementation against spec
## Your task
Execute the spec command: `/spec $ARGUMENTS`
### If command is "create":
1. Get next SPEC number by searching existing specs in "specs" project
2. Create new spec using template from SPEC-2
3. Use mcp__basic-memory__write_note with project="specs"
4. Include standard sections: Why, What, How, How to Evaluate
### If command is "status":
1. Use mcp__basic-memory__search_notes with project="specs"
2. Display table with spec number, title, and progress
3. Show completion status from checkboxes in content
### If command is "show":
1. Use mcp__basic-memory__read_note with project="specs"
2. Display the full spec content
### If command is "review":
1. Read the specified spec and its "How to Evaluate" section
2. Review current implementation against success criteria with careful evaluation of:
- **Functional completeness** - All specified features working
- **Test coverage analysis** - Actual test files and coverage percentage
- Count existing test files vs required components/APIs/composables
- Verify unit tests, integration tests, and end-to-end tests
- Check for missing test categories (component, API, workflow)
- **Code quality metrics** - TypeScript compilation, linting, performance
- **Architecture compliance** - Component isolation, state management patterns
- **Documentation completeness** - Implementation matches specification
3. Provide honest, accurate assessment - do not overstate completeness
4. Document findings and update spec with review results using mcp__basic-memory__edit_note
5. If gaps found, clearly identify what still needs to be implemented/tested
+47 -74
View File
@@ -11,7 +11,7 @@ All test results are recorded as notes in a dedicated test project.
**Parameters:**
- `phase` (optional): Specific test phase to run (`recent`, `core`, `features`, `edge`, `workflows`, `stress`, or `all`)
- `recent` - Focus on recent changes and new features (recommended for regular testing)
- `core` - Essential tools only (Tier 1: write_note, read_note, search_notes, edit_note, list_memory_projects, recent_activity)
- `core` - Essential tools only (Tier 1: write_note, read_note, search_notes, edit_note, list_projects, switch_project)
- `features` - Core + important workflows (Tier 1 + Tier 2)
- `all` - Comprehensive testing of all tools and scenarios
@@ -24,66 +24,30 @@ When the user runs `/project:test-live`, execute comprehensive test plan:
### **Tier 1: Critical Core (Always Test)**
1. **write_note** - Foundation of all knowledge creation
2. **read_note** - Primary knowledge retrieval mechanism
2. **read_note** - Primary knowledge retrieval mechanism
3. **search_notes** - Essential for finding information
4. **edit_note** - Core content modification capability
5. **list_memory_projects** - Project discovery and session guidance
6. **recent_activity** - Project discovery mode and activity analysis
5. **list_memory_projects** - Project discovery and status
6. **switch_project** - Context switching for multi-project workflows
### **Tier 2: Important Workflows (Usually Test)**
7. **build_context** - Conversation continuity via memory:// URLs
8. **create_memory_project** - Essential for project setup
9. **move_note** - Knowledge organization
10. **sync_status** - Understanding system state
11. **delete_project** - Project lifecycle management
7. **recent_activity** - Understanding what's changed
8. **build_context** - Conversation continuity via memory:// URLs
9. **create_memory_project** - Essential for project setup
10. **move_note** - Knowledge organization
11. **sync_status** - Understanding system state
### **Tier 3: Enhanced Functionality (Sometimes Test)**
12. **view_note** - Claude Desktop artifact display
13. **read_content** - Raw content access
14. **delete_note** - Content removal
15. **list_directory** - File system exploration
16. **edit_note** (advanced modes) - Complex find/replace operations
16. **set_default_project** - Configuration
17. **delete_project** - Administrative cleanup
### **Tier 4: Specialized (Rarely Test)**
17. **canvas** - Obsidian visualization (specialized use case)
18. **MCP Prompts** - Enhanced UX tools (ai_assistant_guide, continue_conversation)
## Stateless Architecture Testing
### **Project Discovery Workflow (CRITICAL)**
Test the new stateless project selection flow:
1. **Initial Discovery**
- Call `list_memory_projects()` without knowing which project to use
- Verify clear session guidance appears: "Next: Ask which project to use"
- Confirm removal of CLI-specific references
2. **Activity-Based Discovery**
- Call `recent_activity()` without project parameter (discovery mode)
- Verify intelligent project suggestions based on activity
- Test guidance: "Should I use [most-active-project] for this task?"
3. **Session Tracking Validation**
- Verify all tool responses include `[Session: Using project 'name']`
- Confirm guidance reminds about session-wide project tracking
4. **Single Project Constraint Mode**
- Test MCP server with `--project` parameter
- Verify all operations constrained to specified project
- Test project override behavior in constrained mode
### **Explicit Project Parameters (CRITICAL)**
All tools must require explicit project parameters:
1. **Parameter Validation**
- Test all Tier 1 tools require `project` parameter
- Verify clear error messages for missing project
- Test invalid project name handling
2. **No Session State Dependencies**
- Confirm no tool relies on "current project" concept
- Test rapid project switching within conversation
- Verify each call is truly independent
18. **canvas** - Obsidian visualization (specialized use case)
19. **MCP Prompts** - Enhanced UX tools (ai_assistant_guide, continue_conversation)
### Pre-Test Setup
@@ -108,7 +72,7 @@ Run the bash `date` command to get the current date/time.
Purpose: Record all test observations and results
```
Make sure to use the newly created project for all subsequent test operations by specifying it in the `project` parameter of each tool call.
Make sure to switch to the newly created project with the `switch_project()` tool.
4. **Baseline Documentation**
Create initial test session note with:
@@ -179,42 +143,46 @@ Test essential MCP tools that form the foundation of Basic Memory:
- ⚠️ Error scenarios (invalid operations)
**5. list_memory_projects Tests (Critical):**
- ✅ Display all projects with clear session guidance
- ✅ Project discovery workflow prompts
- ✅ Removal of CLI-specific references
- ✅ Display all projects with status indicators
- ✅ Current and default project identification
- ✅ Empty project list handling
- ✅ Single project constraint mode display
- ✅ Project metadata accuracy
**6. recent_activity Tests (Critical - Discovery Mode):**
- ✅ Discovery mode without project parameter
- ✅ Intelligent project suggestions based on activity
- ✅ Guidance prompts for project selection
- ✅ Session tracking reminders in responses
- ⚠️ Performance with multiple projects
**6. switch_project Tests (Critical):**
- ✅ Switch between existing projects
- ✅ Context preservation during switch
- ⚠️ Invalid project name handling
- ✅ Confirmation of successful switch
### Phase 2: Important Workflows (Tier 2 Tools)
**7. build_context Tests (Important):**
**7. recent_activity Tests (Important):**
- ✅ Various timeframes ("today", "1 week", "1d")
- ✅ Type filtering capabilities
- ✅ Empty project scenarios
- ⚠️ Performance with many recent changes
**8. build_context Tests (Important):**
- ✅ Different depth levels (1, 2, 3+)
- ✅ Various timeframes for context
- ✅ memory:// URL navigation
- ⚠️ Performance with complex relation graphs
**8. create_memory_project Tests (Important):**
**9. create_memory_project Tests (Important):**
- ✅ Create projects dynamically
- ✅ Set default during creation
- ✅ Path validation and creation
- ⚠️ Invalid paths and names
- ✅ Integration with existing projects
**9. move_note Tests (Important):**
**10. move_note Tests (Important):**
- ✅ Move within same project
- ✅ Cross-project moves with detection (#161)
- ✅ Automatic folder creation
- ✅ Database consistency validation
- ⚠️ Special characters in paths
**10. sync_status Tests (Important):**
**11. sync_status Tests (Important):**
- ✅ Background operation monitoring
- ✅ File synchronization status
- ✅ Project sync state reporting
@@ -222,31 +190,36 @@ Test essential MCP tools that form the foundation of Basic Memory:
### Phase 3: Enhanced Functionality (Tier 3 Tools)
**11. view_note Tests (Enhanced):**
**12. view_note Tests (Enhanced):**
- ✅ Claude Desktop artifact display
- ✅ Title extraction from frontmatter
- ✅ Unicode and emoji content rendering
- ⚠️ Error handling for non-existent notes
**12. read_content Tests (Enhanced):**
**13. read_content Tests (Enhanced):**
- ✅ Raw file content access
- ✅ Binary file handling
- ✅ Image file reading
- ⚠️ Large file performance
**13. delete_note Tests (Enhanced):**
**14. delete_note Tests (Enhanced):**
- ✅ Single note deletion
- ✅ Database consistency after deletion
- ⚠️ Non-existent note handling
- ✅ Confirmation of successful deletion
**14. list_directory Tests (Enhanced):**
**15. list_directory Tests (Enhanced):**
- ✅ Directory content listing
- ✅ Depth control and filtering
- ✅ File name globbing
- ⚠️ Empty directory handling
**15. delete_project Tests (Enhanced):**
**16. set_default_project Tests (Enhanced):**
- ✅ Change default project
- ✅ Configuration persistence
- ⚠️ Invalid project handling
**17. delete_project Tests (Enhanced):**
- ✅ Project removal from config
- ✅ Database cleanup
- ⚠️ Default project protection
@@ -296,7 +269,7 @@ Test essential MCP tools that form the foundation of Basic Memory:
1. Technical documentation project
2. Personal recipe collection project
3. Learning/course notes project
4. Specify different projects for different operations
4. Switch contexts during conversation
5. Cross-reference related concepts
**Content Evolution:**
@@ -308,13 +281,13 @@ Test essential MCP tools that form the foundation of Basic Memory:
### Phase 6: Specialized Tools Testing (Tier 4)
**16. canvas Tests (Specialized):**
**18. canvas Tests (Specialized):**
- ✅ JSON Canvas generation
- ✅ Node and edge creation
- ✅ Obsidian compatibility
- ⚠️ Complex graph handling
**17. MCP Prompts Tests (Specialized):**
**19. MCP Prompts Tests (Specialized):**
- ✅ ai_assistant_guide output
- ✅ continue_conversation functionality
- ✅ Formatted search results
@@ -409,7 +382,7 @@ permalink: test-session-[phase]-[timestamp]
### 📊 Performance Metrics
- Average write_note time: 0.3s
- Search with 100+ notes: 0.6s
- Project parameter overhead: <0.1s
- Project switch overhead: 0.1s
- Memory usage: [observed levels]
## Relations
@@ -429,7 +402,7 @@ permalink: test-session-[phase]-[timestamp]
- Learning curve and intuitiveness
**System Behavior:**
- Stateless operation independence
- Context preservation across operations
- memory:// URL navigation reliability
- Multi-step workflow cohesion
- Edge case graceful handling
-3
View File
@@ -1,3 +0,0 @@
{
"enabledPlugins": {}
}
-28
View File
@@ -1,28 +0,0 @@
# Basic Memory Environment Variables Example
# Copy this file to .env and customize as needed
# Note: .env files are gitignored and should never be committed
# ============================================================================
# PostgreSQL Test Database Configuration
# ============================================================================
# These variables allow you to override the default test database credentials
# Default values match docker-compose-postgres.yml for local development
#
# Only needed if you want to use different credentials or a remote test database
# By default, tests use: postgresql://basic_memory_user:dev_password@localhost:5433/basic_memory_test
# Full PostgreSQL test database URL (used by tests and migrations)
# POSTGRES_TEST_URL=postgresql+asyncpg://basic_memory_user:dev_password@localhost:5433/basic_memory_test
# Individual components (used by justfile postgres-reset command)
# POSTGRES_USER=basic_memory_user
# POSTGRES_TEST_DB=basic_memory_test
# ============================================================================
# Production Database Configuration
# ============================================================================
# For production use, set these in your deployment environment
# DO NOT use the test credentials above in production!
# BASIC_MEMORY_DATABASE_BACKEND=postgres # or "sqlite"
# BASIC_MEMORY_DATABASE_URL=postgresql+asyncpg://user:password@host:port/database
-83
View File
@@ -1,83 +0,0 @@
name: Claude Code Review
on:
pull_request:
types: [opened, synchronize]
# Optional: Only run on specific file changes
# paths:
# - "src/**/*.ts"
# - "src/**/*.tsx"
# - "src/**/*.js"
# - "src/**/*.jsx"
jobs:
claude-review:
# Only run for organization members and collaborators
if: |
github.event.pull_request.author_association == 'OWNER' ||
github.event.pull_request.author_association == 'MEMBER' ||
github.event.pull_request.author_association == 'COLLABORATOR'
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
issues: read
id-token: write
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 1
- name: Run Claude Code Review
id: claude-review
uses: anthropics/claude-code-action@v1
with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
github_token: ${{ secrets.GITHUB_TOKEN }}
track_progress: true # Enable visual progress tracking
allowed_bots: '*'
prompt: |
Review this Basic Memory PR against our team checklist:
## Code Quality & Standards
- [ ] Follows Basic Memory's coding conventions in CLAUDE.md
- [ ] Python 3.12+ type annotations and async patterns
- [ ] SQLAlchemy 2.0 best practices
- [ ] FastAPI and Typer conventions followed
- [ ] 100-character line length limit maintained
- [ ] No commented-out code blocks
## Testing & Documentation
- [ ] Unit tests for new functions/methods
- [ ] Integration tests for new MCP tools
- [ ] Test coverage for edge cases
- [ ] **100% test coverage maintained** (use `# pragma: no cover` only for truly hard-to-test code)
- [ ] Documentation updated (README, docstrings)
- [ ] CLAUDE.md updated if conventions change
## Basic Memory Architecture
- [ ] MCP tools follow atomic, composable design
- [ ] Database changes include Alembic migrations
- [ ] Preserves local-first architecture principles
- [ ] Knowledge graph operations maintain consistency
- [ ] Markdown file handling preserves integrity
- [ ] AI-human collaboration patterns followed
## Security & Performance
- [ ] No hardcoded secrets or credentials
- [ ] Input validation for MCP tools
- [ ] Proper error handling and logging
- [ ] Performance considerations addressed
- [ ] No sensitive data in logs or commits
## Compatability
- [ ] File path comparisons must be windows compatible
- [ ] Avoid using emojis and unicode characters in console and log output
Read the CLAUDE.md file for detailed project context. For each checklist item, verify if it's satisfied and comment on any that need attention. Use inline comments for specific code issues and post a summary with checklist results.
# Allow broader tool access for thorough code review
claude_args: '--allowed-tools "Bash(gh pr:*),Bash(gh issue:*),Bash(gh api:*),Bash(git log:*),Bash(git show:*),Read,Grep,Glob"'
-71
View File
@@ -1,71 +0,0 @@
name: Claude Issue Triage
on:
issues:
types: [opened]
jobs:
triage:
runs-on: ubuntu-latest
permissions:
issues: write
id-token: write
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 1
- name: Run Claude Issue Triage
uses: anthropics/claude-code-action@v1
with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
track_progress: true # Show triage progress
prompt: |
Analyze this new Basic Memory issue and perform triage:
**Issue Analysis:**
1. **Type Classification:**
- Bug report (code defect)
- Feature request (new functionality)
- Enhancement (improvement to existing feature)
- Documentation (docs improvement)
- Question/Support (user help)
- MCP tool issue (specific to MCP functionality)
2. **Priority Assessment:**
- Critical: Security issues, data loss, complete breakage
- High: Major functionality broken, affects many users
- Medium: Minor bugs, usability issues
- Low: Nice-to-have improvements, cosmetic issues
3. **Component Classification:**
- CLI commands
- MCP tools
- Database/sync
- Cloud functionality
- Documentation
- Testing
4. **Complexity Estimate:**
- Simple: Quick fix, documentation update
- Medium: Requires some investigation/testing
- Complex: Major feature work, architectural changes
**Actions to Take:**
1. Add appropriate labels using: `gh issue edit ${{ github.event.issue.number }} --add-label "label1,label2"`
2. Check for duplicates using: `gh search issues`
3. If duplicate found, comment mentioning the original issue
4. For feature requests, ask clarifying questions if needed
5. For bugs, request reproduction steps if missing
**Available Labels:**
- Type: bug, enhancement, feature, documentation, question, mcp-tool
- Priority: critical, high, medium, low
- Component: cli, mcp, database, cloud, docs, testing
- Complexity: simple, medium, complex
- Status: needs-reproduction, needs-clarification, duplicate
Read the issue carefully and provide helpful triage with appropriate labels.
claude_args: '--allowed-tools "Bash(gh issue:*),Bash(gh search:*),Read"'
+84 -38
View File
@@ -9,60 +9,106 @@ on:
types: [opened, assigned]
pull_request_review:
types: [submitted]
pull_request_target:
types: [opened, synchronize]
jobs:
claude:
if: |
(
(github.event_name == 'issue_comment' && contains(github.event.comment.body, '@claude')) ||
(github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '@claude')) ||
(github.event_name == 'pull_request_review' && contains(github.event.review.body, '@claude')) ||
(github.event_name == 'issues' && (contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude'))) ||
(github.event_name == 'pull_request_target' && contains(github.event.pull_request.body, '@claude'))
) && (
github.event.comment.author_association == 'OWNER' ||
github.event.comment.author_association == 'MEMBER' ||
github.event.comment.author_association == 'COLLABORATOR' ||
github.event.sender.author_association == 'OWNER' ||
github.event.sender.author_association == 'MEMBER' ||
github.event.sender.author_association == 'COLLABORATOR' ||
github.event.pull_request.author_association == 'OWNER' ||
github.event.pull_request.author_association == 'MEMBER' ||
github.event.pull_request.author_association == 'COLLABORATOR'
)
(github.event_name == 'issue_comment' && contains(github.event.comment.body, '@claude')) ||
(github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '@claude')) ||
(github.event_name == 'pull_request_review' && contains(github.event.review.body, '@claude')) ||
(github.event_name == 'issues' && (contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude')))
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: read
issues: read
id-token: write
actions: read # Required for Claude to read CI results on PRs
steps:
- name: Check user permissions
id: check_membership
uses: actions/github-script@v7
with:
script: |
let actor;
if (context.eventName === 'issue_comment') {
actor = context.payload.comment.user.login;
} else if (context.eventName === 'pull_request_review_comment') {
actor = context.payload.comment.user.login;
} else if (context.eventName === 'pull_request_review') {
actor = context.payload.review.user.login;
} else if (context.eventName === 'issues') {
actor = context.payload.issue.user.login;
}
console.log(`Checking permissions for user: ${actor}`);
// List of explicitly allowed users (organization members)
const allowedUsers = [
'phernandez',
'groksrc',
'nellins',
'bm-claudeai'
];
if (allowedUsers.includes(actor)) {
console.log(`User ${actor} is in the allowed list`);
core.setOutput('is_member', true);
return;
}
// Fallback: Check if user has repository permissions
try {
const collaboration = await github.rest.repos.getCollaboratorPermissionLevel({
owner: context.repo.owner,
repo: context.repo.repo,
username: actor
});
const permission = collaboration.data.permission;
console.log(`User ${actor} has permission level: ${permission}`);
// Allow if user has push access or higher (write, maintain, admin)
const allowed = ['write', 'maintain', 'admin'].includes(permission);
core.setOutput('is_member', allowed);
if (!allowed) {
core.notice(`User ${actor} does not have sufficient repository permissions (has: ${permission})`);
}
} catch (error) {
console.log(`Error checking permissions: ${error.message}`);
// Final fallback: Check if user is a public member of the organization
try {
const membership = await github.rest.orgs.getMembershipForUser({
org: 'basicmachines-co',
username: actor
});
const allowed = membership.data.state === 'active';
core.setOutput('is_member', allowed);
if (!allowed) {
core.notice(`User ${actor} is not a public member of basicmachines-co organization`);
}
} catch (membershipError) {
console.log(`Error checking organization membership: ${membershipError.message}`);
core.setOutput('is_member', false);
core.notice(`User ${actor} does not have access to this repository`);
}
}
- name: Checkout repository
if: steps.check_membership.outputs.is_member == 'true'
uses: actions/checkout@v4
with:
# For pull_request_target, checkout the PR head to review the actual changes
ref: ${{ github.event_name == 'pull_request_target' && github.event.pull_request.head.sha || github.sha }}
fetch-depth: 1
- name: Run Claude Code
if: steps.check_membership.outputs.is_member == 'true'
id: claude
uses: anthropics/claude-code-action@v1
uses: anthropics/claude-code-action@beta
with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
track_progress: true # Enable visual progress tracking
# This is an optional setting that allows Claude to read CI results on PRs
additional_permissions: |
actions: read
# Optional: Give a custom prompt to Claude. If this is not specified, Claude will perform the instructions specified in the comment that tagged it.
# prompt: 'Update the pull request description to include a summary of changes.'
# Optional: Add claude_args to customize behavior and configuration
# See https://github.com/anthropics/claude-code-action/blob/main/docs/usage.md
# or https://docs.claude.com/en/docs/claude-code/sdk#command-line for available options
# claude_args: '--model claude-opus-4-1-20250805 --allowed-tools Bash(gh pr:*)'
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
allowed_tools: Bash(uv run pytest),Bash(uv run ruff check . --fix),Bash(uv run ruff format .),Bash(uv run pyright),Bash(just test),Bash(just lint),Bash(just format),Bash(just type-check),Bash(just check),Read,Write,Edit,MultiEdit,Glob,Grep,LS, mcp__web_search
+22 -286
View File
@@ -1,37 +1,43 @@
name: Tests
concurrency:
group: bm-ci-${{ github.workflow }}-${{ github.repository }}-${{ github.head_ref || github.ref }}
cancel-in-progress: true
on:
push:
branches: [ "main" ]
pull_request:
branches: [ "main" ]
# pull_request_target runs on the BASE of the PR, not the merge result.
# It has write permissions and access to secrets.
# It's useful for PRs from forks or automated PRs but requires careful use for security reasons.
# See: https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request_target
pull_request_target:
branches: [ "main" ]
jobs:
static-checks:
name: Static Checks (Python 3.12)
timeout-minutes: 20
test:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: [ "3.12" ]
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python 3.12
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: "3.12"
cache: "pip"
python-version: ${{ matrix.python-version }}
cache: 'pip'
- name: Install uv
run: |
pip install uv
- uses: extractions/setup-just@v3
- name: Install just
run: |
curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to /usr/local/bin
- name: Create virtual env
run: |
@@ -39,283 +45,13 @@ jobs:
- name: Install dependencies
run: |
uv pip install -e ".[dev]"
uv pip install -e .[dev]
- name: Run type checks
run: |
just typecheck
just type-check
- name: Run linting
- name: Run tests
run: |
just lint
test-sqlite-unit:
name: Test SQLite Unit (${{ matrix.os }}, Python ${{ matrix.python-version }})
timeout-minutes: 30
needs: [static-checks]
strategy:
fail-fast: false
matrix:
include:
- os: ubuntu-latest
python-version: "3.12"
- os: ubuntu-latest
python-version: "3.13"
- os: ubuntu-latest
python-version: "3.14"
- os: windows-latest
python-version: "3.12"
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
- name: Install uv
run: |
pip install uv
- uses: extractions/setup-just@v3
- name: Create virtual env
run: |
uv venv
- name: Install dependencies
run: |
uv pip install -e ".[dev]"
- name: Run tests (SQLite Unit)
run: |
just test-unit-sqlite
test-sqlite-integration:
name: Test SQLite Integration (${{ matrix.os }}, Python ${{ matrix.python-version }})
timeout-minutes: 45
needs: [static-checks]
strategy:
fail-fast: false
matrix:
include:
- os: ubuntu-latest
python-version: "3.12"
- os: ubuntu-latest
python-version: "3.13"
- os: ubuntu-latest
python-version: "3.14"
- os: windows-latest
python-version: "3.12"
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
- name: Install uv
run: |
pip install uv
- uses: extractions/setup-just@v3
- name: Create virtual env
run: |
uv venv
- name: Install dependencies
run: |
uv pip install -e ".[dev]"
- name: Run tests (SQLite Integration)
run: |
just test-int-sqlite
test-postgres-unit:
name: Test Postgres Unit (Python ${{ matrix.python-version }})
timeout-minutes: 30
needs: [static-checks]
strategy:
fail-fast: false
matrix:
python-version: [ "3.12", "3.13", "3.14" ]
runs-on: ubuntu-latest
# Note: No services section needed - testcontainers handles Postgres in Docker
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
- name: Install uv
run: |
pip install uv
- uses: extractions/setup-just@v3
- name: Create virtual env
run: |
uv venv
- name: Install dependencies
run: |
uv pip install -e ".[dev]"
- name: Run tests (Postgres Unit)
run: |
just test-unit-postgres
test-postgres-integration:
name: Test Postgres Integration (Python ${{ matrix.python-version }})
timeout-minutes: 45
needs: [static-checks]
strategy:
fail-fast: false
matrix:
python-version: [ "3.12", "3.13", "3.14" ]
runs-on: ubuntu-latest
# Note: No services section needed - testcontainers handles Postgres in Docker
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
- name: Install uv
run: |
pip install uv
- uses: extractions/setup-just@v3
- name: Create virtual env
run: |
uv venv
- name: Install dependencies
run: |
uv pip install -e ".[dev]"
- name: Run tests (Postgres Integration)
run: |
just test-int-postgres
test-semantic:
name: Test Semantic (Python 3.12)
timeout-minutes: 45
needs: [static-checks]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python 3.12
uses: actions/setup-python@v4
with:
python-version: "3.12"
cache: "pip"
- name: Install uv
run: |
pip install uv
- uses: extractions/setup-just@v3
- name: Create virtual env
run: |
uv venv
- name: Install dependencies
run: |
uv pip install -e ".[dev,semantic]"
- name: Run tests (Semantic)
run: |
just test-semantic
coverage:
name: Coverage Summary (combined, Python 3.12)
timeout-minutes: 60
needs:
- static-checks
- test-sqlite-unit
- test-sqlite-integration
- test-postgres-unit
- test-postgres-integration
- test-semantic
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python 3.12
uses: actions/setup-python@v4
with:
python-version: "3.12"
cache: "pip"
- name: Install uv
run: |
pip install uv
- uses: extractions/setup-just@v3
- name: Create virtual env
run: |
uv venv
- name: Install dependencies
run: |
uv pip install -e ".[dev,semantic]"
- name: Run combined coverage (SQLite + Postgres)
run: |
just coverage
- name: Add coverage report to job summary
if: always()
run: |
{
echo "## Coverage"
echo ""
echo '```'
uv run coverage report -m
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
- name: Upload HTML coverage report
if: always()
uses: actions/upload-artifact@v4
with:
name: htmlcov
path: htmlcov/
uv pip install pytest pytest-cov
just test
+1 -6
View File
@@ -1,7 +1,6 @@
*.py[cod]
__pycache__/
.pytest_cache/
.testmondata*
.coverage
htmlcov/
@@ -53,8 +52,4 @@ ENV/
# claude action
claude-output
**/.claude/settings.local.json
.mcp.json
.mcpregistry_*
/.testmondata
.benchmarks/
**/.claude/settings.local.json
+1 -1
View File
@@ -1 +1 @@
3.14
3.12
-445
View File
@@ -1,445 +0,0 @@
# AGENTS.md - Basic Memory Project Guide
## Project Overview
Basic Memory is a local-first knowledge management system built on the Model Context Protocol (MCP). It enables
bidirectional communication between LLMs (like Claude) and markdown files, creating a personal knowledge graph that can
be traversed using links between documents.
## CODEBASE DEVELOPMENT
### Project information
See the [README.md](README.md) file for a project overview.
### Build and Test Commands
- Install: `just install` or `pip install -e ".[dev]"`
- Run all tests (SQLite + Postgres): `just test`
- Run all tests against SQLite: `just test-sqlite`
- Run all tests against Postgres: `just test-postgres` (uses testcontainers)
- Run unit tests (SQLite): `just test-unit-sqlite`
- Run unit tests (Postgres): `just test-unit-postgres`
- Run integration tests (SQLite): `just test-int-sqlite`
- Run integration tests (Postgres): `just test-int-postgres`
- Run impacted tests: `just testmon` (pytest-testmon)
- Run MCP smoke test: `just test-smoke`
- Fast local loop: `just fast-check`
- Local consistency check: `just doctor`
- Generate HTML coverage: `just coverage`
- Single test: `pytest tests/path/to/test_file.py::test_function_name`
- Run benchmarks: `pytest test-int/test_sync_performance_benchmark.py -v -m "benchmark and not slow"`
- Lint: `just lint` or `ruff check . --fix`
- Type check: `just typecheck` or `uv run pyright`
- Format: `just format` or `uv run ruff format .`
- Run all code checks: `just check` (runs lint, format, typecheck, test)
- Create db migration: `just migration "Your migration message"`
- Run development MCP Inspector: `just run-inspector`
**Note:** Project requires Python 3.12+ (uses type parameter syntax and `type` aliases introduced in 3.12)
**Postgres Testing:** Uses [testcontainers](https://testcontainers-python.readthedocs.io/) which automatically spins up a Postgres instance in Docker. No manual database setup required - just have Docker running.
**Doctor Note:** `just doctor` runs with a temporary HOME/config so it won't touch your local Basic Memory settings. It leaves temp dirs in `/tmp` (safe to ignore or remove).
**Testmon Note:** When no files have changed, `just testmon` may collect 0 tests. That's expected and means no impacted tests were detected.
### Code/Test/Verify Loop (fast path)
1) **Code:** make changes.
2) **Test:** `just fast-check` (lint/format/typecheck + impacted tests + MCP smoke).
3) **Verify:** `just doctor` (end-to-end file ↔ DB loop in a temp project).
4) **Full gate (when needed):** `just test` or `just check` for SQLite + Postgres.
If testmon is “cold,” the first run may be long. Subsequent runs get much faster.
### Test Structure
- `tests/` - Unit tests for individual components (mocked, fast)
- `test-int/` - Integration tests for real-world scenarios (no mocks, realistic)
- Both directories are covered by unified coverage reporting
- Benchmark tests in `test-int/` are marked with `@pytest.mark.benchmark`
- Slow tests are marked with `@pytest.mark.slow`
- Smoke tests are marked with `@pytest.mark.smoke`
### Code Style Guidelines
- Line length: 100 characters max
- Python 3.12+ with full type annotations (uses type parameters and type aliases)
- Format with ruff (consistent styling)
- Import order: standard lib, third-party, local imports
- Naming: snake_case for functions/variables, PascalCase for classes
- Prefer async patterns with SQLAlchemy 2.0
- Use Pydantic v2 for data validation and schemas
- CLI uses Typer for command structure
- API uses FastAPI for endpoints
- Follow the repository pattern for data access
- Tools communicate to api routers via the httpx ASGI client (in process)
### Code Change Guidelines
- **Full file read before edits**: Before editing any file, read it in full first to ensure complete context; partial reads lead to corrupted edits
- **Minimize diffs**: Prefer the smallest change that satisfies the request. Avoid unrelated refactors or style rewrites unless necessary for correctness
- **No speculative getattr**: Never use `getattr(obj, "attr", default)` when unsure about attribute names. Check the class definition or source code first
- **Fail fast**: Write code with fail-fast logic by default. Do not swallow exceptions with errors or warnings
- **No fallback logic**: Do not add fallback logic unless explicitly told to and agreed with the user
- **No guessing**: Do not say "The issue is..." before you actually know what the issue is. Investigate first.
### Literate Programming Style
Code should tell a story. Comments must explain the "why" and narrative flow, not just the "what".
**Section Headers:**
For files with multiple phases of logic, add section headers so the control flow reads like chapters:
```python
# --- Authentication ---
# ... auth logic ...
# --- Data Validation ---
# ... validation logic ...
# --- Business Logic ---
# ... core logic ...
```
**Decision Point Comments:**
For conditionals that materially change behavior (gates, fallbacks, retries, feature flags), add comments with:
- **Trigger**: what condition causes this branch
- **Why**: the rationale (cost, correctness, UX, determinism)
- **Outcome**: what changes downstream
```python
# Trigger: project has no active sync watcher
# Why: avoid duplicate file system watchers consuming resources
# Outcome: starts new watcher, registers in active_watchers dict
if project_id not in active_watchers:
start_watcher(project_id)
```
**Constraint Comments:**
If code exists because of a constraint (async requirements, rate limits, schema compatibility), explain the constraint near the code:
```python
# SQLite requires WAL mode for concurrent read/write access
connection.execute("PRAGMA journal_mode=WAL")
```
**What NOT to Comment:**
Avoid comments that restate obvious code:
```python
# Bad - restates code
counter += 1 # increment counter
# Good - explains why
counter += 1 # track retries for backoff calculation
```
### Codebase Architecture
See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for detailed architecture documentation.
**Directory Structure:**
- `/alembic` - Alembic db migrations
- `/api` - FastAPI REST endpoints + `container.py` composition root
- `/cli` - Typer CLI + `container.py` composition root
- `/deps` - Feature-scoped FastAPI dependencies (config, db, projects, repositories, services, importers)
- `/importers` - Import functionality for Claude, ChatGPT, and other sources
- `/markdown` - Markdown parsing and processing
- `/mcp` - MCP server + `container.py` composition root + `clients/` typed API clients
- `/models` - SQLAlchemy ORM models
- `/repository` - Data access layer
- `/schemas` - Pydantic models for validation
- `/services` - Business logic layer
- `/sync` - File synchronization services + `coordinator.py` for lifecycle management
**Composition Roots:**
Each entrypoint (API, MCP, CLI) has a composition root that:
- Reads `ConfigManager` (the only place that reads global config)
- Resolves runtime mode via `RuntimeMode` enum (TEST > CLOUD > LOCAL)
- Provides dependencies to downstream code explicitly
**Typed API Clients (MCP):**
MCP tools use typed clients in `mcp/clients/` to communicate with the API:
- `KnowledgeClient` - Entity CRUD operations
- `SearchClient` - Search operations
- `MemoryClient` - Context building
- `DirectoryClient` - Directory listing
- `ResourceClient` - Resource reading
- `ProjectClient` - Project management
Flow: MCP Tool → Typed Client → HTTP API → Router → Service → Repository
### Development Notes
- MCP tools are defined in src/basic_memory/mcp/tools/
- MCP prompts are defined in src/basic_memory/mcp/prompts/
- MCP tools should be atomic, composable operations
- Use `textwrap.dedent()` for multi-line string formatting in prompts and tools
- MCP Prompts are used to invoke tools and format content with instructions for an LLM
- Schema changes require Alembic migrations
- SQLite is used for indexing and full text search, files are source of truth
- Testing uses pytest with asyncio support (strict mode)
- Unit tests (`tests/`) use mocks when necessary; integration tests (`test-int/`) use real implementations
- By default, tests run against SQLite (fast, no Docker needed)
- Set `BASIC_MEMORY_TEST_POSTGRES=1` to run against Postgres (uses testcontainers - Docker required)
- Each test runs in a standalone environment with isolated database and tmp_path directory
- CI runs SQLite and Postgres tests in parallel for faster feedback
- Performance benchmarks are in `test-int/test_sync_performance_benchmark.py`
- Use pytest markers: `@pytest.mark.benchmark` for benchmarks, `@pytest.mark.slow` for slow tests
- **Coverage must stay at 100%**: Write tests for new code. Only use `# pragma: no cover` when tests would require excessive mocking (e.g., TYPE_CHECKING blocks, error handlers that need failure injection, runtime-mode-dependent code paths)
### Async Client Pattern (Important!)
**MCP tools use `get_project_client()` for per-project routing:**
```python
from basic_memory.mcp.project_context import get_project_client
@mcp.tool()
async def my_tool(project: str | None = None, context: Context | None = None):
async with get_project_client(project, context) as (client, active_project):
# client is routed based on project's mode (local ASGI or cloud HTTP)
response = await call_get(client, "/path")
return response
```
**CLI commands and non-project-scoped code use `get_client()` directly:**
```python
from basic_memory.mcp.async_client import get_client
async def my_cli_command():
async with get_client() as client:
response = await call_get(client, "/path")
return response
# Per-project routing (when project name is known):
async with get_client(project_name="research") as client:
...
```
**Do NOT use:**
-`from basic_memory.mcp.async_client import client` (deprecated module-level client)
- ❌ Manual auth header management
-`inject_auth_header()` (deleted)
- ❌ Separate `get_client()` + `get_active_project()` in MCP tools (use `get_project_client()` instead)
**Key principles:**
- Auth happens at client creation, not per-request
- Proper resource management via context managers
- Per-project routing: each project can be LOCAL or CLOUD independently
- Cloud projects use API key (`cloud_api_key` in config) as Bearer token
- Routing priority: factory injection > force-local > per-project cloud > global cloud > local ASGI
- Factory pattern enables dependency injection for cloud consolidation
**For cloud app integration:**
```python
from basic_memory.mcp import async_client
# Set custom factory before importing tools
async_client.set_client_factory(your_custom_factory)
```
See SPEC-16 for full context manager refactor details.
## BASIC MEMORY PRODUCT USAGE
### Knowledge Structure
- Entity: Any concept, document, or idea represented as a markdown file
- Observation: A categorized fact about an entity (`- [category] content`)
- Relation: A directional link between entities (`- relation_type [[Target]]`)
- Frontmatter: YAML metadata at the top of markdown files
- Knowledge representation follows precise markdown format:
- Observations with [category] prefixes
- Relations with WikiLinks [[Entity]]
- Frontmatter with metadata
### Basic Memory Commands
**Local Commands:**
- Check sync status: `basic-memory status`
- Doctor check (file <-> DB loop): `basic-memory doctor`
- Import from Claude: `basic-memory import claude conversations`
- Import from ChatGPT: `basic-memory import chatgpt`
- Import from Memory JSON: `basic-memory import memory-json`
- Tool access: `basic-memory tool` (provides CLI access to MCP tools)
- Continue: `basic-memory tool continue-conversation --topic="search"`
**Project Management:**
- List projects: `basic-memory project list`
- Add project: `basic-memory project add "name" ~/path`
- Project info: `basic-memory project info`
- Set cloud mode: `basic-memory project set-cloud "name"`
- Set local mode: `basic-memory project set-local "name"`
- One-way sync (local -> cloud): `basic-memory project sync`
- Bidirectional sync: `basic-memory project bisync`
- Integrity check: `basic-memory project check`
**Cloud Commands (requires subscription):**
- Authenticate (global): `basic-memory cloud login`
- Logout (global): `basic-memory cloud logout`
- Check cloud status: `basic-memory cloud status`
- Setup cloud sync: `basic-memory cloud setup`
- Save API key: `basic-memory cloud set-key bmc_...`
- Create API key: `basic-memory cloud create-key "name"`
- Manage snapshots: `basic-memory cloud snapshot [create|list|delete|show|browse]`
- Restore from snapshot: `basic-memory cloud restore <path> --snapshot <id>`
### MCP Capabilities
- Basic Memory exposes these MCP tools to LLMs:
**Content Management:**
- `write_note(title, content, directory, tags)` - Create/update markdown notes with semantic observations and relations
- `read_note(identifier, page, page_size)` - Read notes by title, permalink, or memory:// URL with knowledge graph awareness
- `read_content(path)` - Read raw file content (text, images, binaries) without knowledge graph processing
- `view_note(identifier, page, page_size)` - View notes as formatted artifacts for better readability
- `edit_note(identifier, operation, content)` - Edit notes incrementally (append, prepend, find/replace, replace_section)
- `move_note(identifier, destination_path, is_directory)` - Move notes or directories to new locations, updating database and maintaining links
- `delete_note(identifier, is_directory)` - Delete notes or directories from the knowledge base
**Knowledge Graph Navigation:**
- `build_context(url, depth, timeframe)` - Navigate the knowledge graph via memory:// URLs for conversation continuity
- `recent_activity(type, depth, timeframe)` - Get recently updated information with specified timeframe (e.g., "1d", "1 week")
- `list_directory(dir_name, depth, file_name_glob)` - Browse directory contents with filtering and depth control
**Search & Discovery:**
- `search_notes(query, page, page_size, search_type, types, entity_types, after_date)` - Full-text search across all content with advanced filtering options
**Project Management:**
- `list_memory_projects()` - List all available projects with their status
- `create_memory_project(project_name, project_path, set_default)` - Create new Basic Memory projects
- `delete_project(project_name)` - Delete a project from configuration
**Visualization:**
- `canvas(nodes, edges, title, directory)` - Generate Obsidian canvas files for knowledge graph visualization
**ChatGPT-Compatible Tools:**
- `search(query)` - Search across knowledge base (OpenAI actions compatible)
- `fetch(id)` - Fetch full content of a search result document
- MCP Prompts for better AI interaction:
- `ai_assistant_guide()` - Guidance on effectively using Basic Memory tools for AI assistants
- `continue_conversation(topic, timeframe)` - Continue previous conversations with relevant historical context
- `search(query, after_date)` - Search with detailed, formatted results for better context understanding
- `recent_activity(timeframe)` - View recently changed items with formatted output
### Cloud Features (v0.15.0+)
Basic Memory now supports cloud synchronization and storage (requires active subscription):
**Authentication:**
- JWT-based authentication with subscription validation
- Secure session management with token refresh
- Support for multiple cloud projects
**Bidirectional Sync:**
- rclone bisync integration for two-way synchronization
- Conflict resolution and integrity verification
- Real-time sync with change detection
- Mount/unmount cloud storage for direct file access
**Cloud Project Management:**
- Create and manage projects in the cloud
- Toggle between local and cloud modes
- Per-project sync configuration
- Subscription-based access control
**Security & Performance:**
- Removed .env file loading for improved security
- .gitignore integration (respects gitignored files)
- WAL mode for SQLite performance
- Background relation resolution (non-blocking startup)
- API performance optimizations (SPEC-11)
**Per-Project Cloud Routing:**
Individual projects can be routed through the cloud while others stay local, using an API key:
```bash
# Save API key and set project to cloud mode
basic-memory cloud set-key bmc_abc123...
basic-memory project set-cloud research # route through cloud
basic-memory project set-local research # revert to local
```
MCP tools use `get_project_client()` which automatically routes based on the project's mode. Cloud projects use the `cloud_api_key` from config as Bearer token.
**CLI Routing Flags (Global Cloud Mode):**
When global cloud mode is enabled, CLI commands route to the cloud API by default. Use `--local` and `--cloud` flags to override:
```bash
# Force local routing (ignore cloud mode)
basic-memory status --local
basic-memory project list --local
# Force cloud routing (when cloud mode is disabled)
basic-memory status --cloud
basic-memory project info my-project --cloud
```
Key behaviors:
- The local MCP server (`basic-memory mcp`) automatically uses local routing
- This allows simultaneous use of local Claude Desktop and cloud-based clients
- Some commands (like `project default`, `project sync-config`, `project move`) require `--local` in cloud mode since they modify local configuration
- Environment variable `BASIC_MEMORY_FORCE_LOCAL=true` forces local routing globally
- Per-project cloud routing via API key works independently of global cloud mode
## AI-Human Collaborative Development
Basic Memory emerged from and enables a new kind of development process that combines human and AI capabilities. Instead
of using AI just for code generation, we've developed a true collaborative workflow:
1. AI (LLM) writes initial implementation based on specifications and context
2. Human reviews, runs tests, and commits code with any necessary adjustments
3. Knowledge persists across conversations using Basic Memory's knowledge graph
4. Development continues seamlessly across different AI sessions with consistent context
5. Results improve through iterative collaboration and shared understanding
This approach has allowed us to tackle more complex challenges and build a more robust system than either humans or AI
could achieve independently.
**Problem-Solving Guidance:**
- If a solution isn't working after reasonable effort, suggest alternative approaches
- Don't persist with a problematic library or pattern when better alternatives exist
- Example: When py-pglite caused cascading test failures, switching to testcontainers-postgres was the right call
## GitHub Integration
Basic Memory has taken AI-Human collaboration to the next level by integrating Claude directly into the development workflow through GitHub:
### GitHub MCP Tools
Using the GitHub Model Context Protocol server, Claude can now:
- **Repository Management**:
- View repository files and structure
- Read file contents
- Create new branches
- Create and update files
- **Issue Management**:
- Create new issues
- Comment on existing issues
- Close and update issues
- Search across issues
- **Pull Request Workflow**:
- Create pull requests
- Review code changes
- Add comments to PRs
This integration enables Claude to participate as a full team member in the development process, not just as a code generation tool. Claude's GitHub account ([bm-claudeai](https://github.com/bm-claudeai)) is a member of the Basic Machines organization with direct contributor access to the codebase.
### Collaborative Development Process
With GitHub integration, the development workflow includes:
1. **Direct code review** - Claude can analyze PRs and provide detailed feedback
2. **Contribution tracking** - All of Claude's contributions are properly attributed in the Git history
3. **Branch management** - Claude can create feature branches for implementations
4. **Documentation maintenance** - Claude can keep documentation updated as the code evolves
5. **Code Commits**: ALWAYS sign off commits with `git commit -s`
This level of integration represents a new paradigm in AI-human collaboration, where the AI assistant becomes a full-fledged team member rather than just a tool for generating code snippets.
+1 -1163
View File
File diff suppressed because it is too large Load Diff
+26 -63
View File
@@ -1,71 +1,34 @@
# Contributor License Agreement
Developer Certificate of Origin
Version 1.1
https://developercertificate.org/
## Copyright Assignment and License Grant
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
By signing this Contributor License Agreement ("Agreement"), you accept and agree to the following terms and conditions
for your present and future Contributions submitted
to Basic Machines LLC. Except for the license granted herein to Basic Machines LLC and recipients of software
distributed by Basic Machines LLC, you reserve all right,
title, and interest in and to your Contributions.
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
### 1. Definitions
Developer's Certificate of Origin 1.1
"You" (or "Your") shall mean the copyright owner or legal entity authorized by the copyright owner that is making this
Agreement with Basic Machines LLC.
By making a contribution to this project, I certify that:
"Contribution" shall mean any original work of authorship, including any modifications or additions to an existing work,
that is intentionally submitted by You to Basic
Machines LLC for inclusion in, or documentation of, any of the products owned or managed by Basic Machines LLC (the "
Work").
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
### 2. Grant of Copyright License
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
Subject to the terms and conditions of this Agreement, You hereby grant to Basic Machines LLC and to recipients of
software distributed by Basic Machines LLC a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the
Work, and to permit persons to whom the Work is furnished to do so.
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
### 3. Assignment of Copyright
You hereby assign to Basic Machines LLC all right, title, and interest worldwide in all Copyright covering your
Contributions. Basic Machines LLC may license the
Contributions under any license terms, including copyleft, permissive, commercial, or proprietary licenses.
### 4. Grant of Patent License
Subject to the terms and conditions of this Agreement, You hereby grant to Basic Machines LLC and to recipients of
software distributed by Basic Machines LLC a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to
make, have made, use, offer to sell, sell, import, and
otherwise transfer the Work.
### 5. Developer Certificate of Origin
By making a Contribution to this project, You certify that:
(a) The Contribution was created in whole or in part by You and You have the right to submit it under this Agreement; or
(b) The Contribution is based upon previous work that, to the best of Your knowledge, is covered under an appropriate
open source license and You have the right under that
license to submit that work with modifications, whether created in whole or in part by You, under this Agreement; or
(c) The Contribution was provided directly to You by some other person who certified (a), (b) or (c) and You have not
modified it.
(d) You understand and agree that this project and the Contribution are public and that a record of the Contribution (
including all personal information You submit with
it, including Your sign-off) is maintained indefinitely and may be redistributed consistent with this project or the
open source license(s) involved.
### 6. Representations
You represent that you are legally entitled to grant the above license and assignment. If your employer(s) has rights to
intellectual property that you create that
includes your Contributions, you represent that you have received permission to make Contributions on behalf of that
employer, or that your employer has waived such rights
for your Contributions to Basic Machines LLC.
---
This Agreement is effective as of the date you first submit a Contribution to Basic Machines LLC.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.
-1
View File
@@ -1 +0,0 @@
AGENTS.md
+257
View File
@@ -0,0 +1,257 @@
# CLAUDE.md - Basic Memory Project Guide
## Project Overview
Basic Memory is a local-first knowledge management system built on the Model Context Protocol (MCP). It enables
bidirectional communication between LLMs (like Claude) and markdown files, creating a personal knowledge graph that can
be traversed using links between documents.
## CODEBASE DEVELOPMENT
### Project information
See the [README.md](README.md) file for a project overview.
### Build and Test Commands
- Install: `just install` or `pip install -e ".[dev]"`
- Run tests: `uv run pytest -p pytest_mock -v` or `just test`
- Single test: `pytest tests/path/to/test_file.py::test_function_name`
- Lint: `just lint` or `ruff check . --fix`
- Type check: `just type-check` or `uv run pyright`
- Format: `just format` or `uv run ruff format .`
- Run all code checks: `just check` (runs lint, format, type-check, test)
- Create db migration: `just migration "Your migration message"`
- Run development MCP Inspector: `just run-inspector`
### Code Style Guidelines
- Line length: 100 characters max
- Python 3.12+ with full type annotations
- Format with ruff (consistent styling)
- Import order: standard lib, third-party, local imports
- Naming: snake_case for functions/variables, PascalCase for classes
- Prefer async patterns with SQLAlchemy 2.0
- Use Pydantic v2 for data validation and schemas
- CLI uses Typer for command structure
- API uses FastAPI for endpoints
- Follow the repository pattern for data access
- Tools communicate to api routers via the httpx ASGI client (in process)
- avoid using "private" functions in modules or classes (prepended with _)
### Codebase Architecture
- `/alembic` - Alembic db migrations
- `/api` - FastAPI implementation of REST endpoints
- `/cli` - Typer command-line interface
- `/markdown` - Markdown parsing and processing
- `/mcp` - Model Context Protocol server implementation
- `/models` - SQLAlchemy ORM models
- `/repository` - Data access layer
- `/schemas` - Pydantic models for validation
- `/services` - Business logic layer
- `/sync` - File synchronization services
### Development Notes
- MCP tools are defined in src/basic_memory/mcp/tools/
- MCP prompts are defined in src/basic_memory/mcp/prompts/
- MCP tools should be atomic, composable operations
- Use `textwrap.dedent()` for multi-line string formatting in prompts and tools
- MCP Prompts are used to invoke tools and format content with instructions for an LLM
- Schema changes require Alembic migrations
- SQLite is used for indexing and full text search, files are source of truth
- Testing uses pytest with asyncio support (strict mode)
- Test database uses in-memory SQLite
- Avoid creating mocks in tests in most circumstances.
- Each test runs in a standalone environment with in memory SQLite and tmp_file directory
- Do not use mocks in tests if possible. Tests run with an in memory sqlite db, so they are not needed. See fixtures in conftest.py
## BASIC MEMORY PRODUCT USAGE
### Knowledge Structure
- Entity: Any concept, document, or idea represented as a markdown file
- Observation: A categorized fact about an entity (`- [category] content`)
- Relation: A directional link between entities (`- relation_type [[Target]]`)
- Frontmatter: YAML metadata at the top of markdown files
- Knowledge representation follows precise markdown format:
- Observations with [category] prefixes
- Relations with WikiLinks [[Entity]]
- Frontmatter with metadata
### Basic Memory Commands
- Sync knowledge: `basic-memory sync` or `basic-memory sync --watch`
- Import from Claude: `basic-memory import claude conversations`
- Import from ChatGPT: `basic-memory import chatgpt`
- Import from Memory JSON: `basic-memory import memory-json`
- Check sync status: `basic-memory status`
- Tool access: `basic-memory tools` (provides CLI access to MCP tools)
- Guide: `basic-memory tools basic-memory-guide`
- Continue: `basic-memory tools continue-conversation --topic="search"`
### MCP Capabilities
- Basic Memory exposes these MCP tools to LLMs:
**Content Management:**
- `write_note(title, content, folder, tags)` - Create/update markdown notes with semantic observations and relations
- `read_note(identifier, page, page_size)` - Read notes by title, permalink, or memory:// URL with knowledge graph awareness
- `edit_note(identifier, operation, content)` - Edit notes incrementally (append, prepend, find/replace, section replace)
- `move_note(identifier, destination_path)` - Move notes with database consistency and search reindexing
- `view_note(identifier)` - Display notes as formatted artifacts for better readability in Claude Desktop
- `read_content(path)` - Read raw file content (text, images, binaries) without knowledge graph processing
- `delete_note(identifier)` - Delete notes from knowledge base
**Project Management:**
- `list_memory_projects()` - List all available projects with status indicators
- `switch_project(project_name)` - Switch to different project context during conversations
- `get_current_project()` - Show currently active project with statistics
- `create_memory_project(name, path, set_default)` - Create new Basic Memory projects
- `delete_project(name)` - Delete projects from configuration and database
- `set_default_project(name)` - Set default project in config
- `sync_status()` - Check file synchronization status and background operations
**Knowledge Graph Navigation:**
- `build_context(url, depth, timeframe)` - Navigate the knowledge graph via memory:// URLs for conversation continuity
- `recent_activity(type, depth, timeframe)` - Get recently updated information with specified timeframe (e.g., "1d", "1 week")
- `list_directory(dir_name, depth, file_name_glob)` - List directory contents with filtering and depth control
**Search & Discovery:**
- `search_notes(query, page, page_size)` - Full-text search across all content with filtering options
**Visualization:**
- `canvas(nodes, edges, title, folder)` - Generate Obsidian canvas files for knowledge graph visualization
- MCP Prompts for better AI interaction:
- `ai_assistant_guide()` - Guidance on effectively using Basic Memory tools for AI assistants
- `continue_conversation(topic, timeframe)` - Continue previous conversations with relevant historical context
- `search_notes(query, after_date)` - Search with detailed, formatted results for better context understanding
- `recent_activity(timeframe)` - View recently changed items with formatted output
- `json_canvas_spec()` - Full JSON Canvas specification for Obsidian visualization
## AI-Human Collaborative Development
Basic Memory emerged from and enables a new kind of development process that combines human and AI capabilities. Instead
of using AI just for code generation, we've developed a true collaborative workflow:
1. AI (LLM) writes initial implementation based on specifications and context
2. Human reviews, runs tests, and commits code with any necessary adjustments
3. Knowledge persists across conversations using Basic Memory's knowledge graph
4. Development continues seamlessly across different AI sessions with consistent context
5. Results improve through iterative collaboration and shared understanding
This approach has allowed us to tackle more complex challenges and build a more robust system than either humans or AI
could achieve independently.
## GitHub Integration
Basic Memory uses Claude directly into the development workflow through GitHub:
### GitHub MCP Tools
Using the GitHub Model Context Protocol server, Claude can:
- **Repository Management**:
- View repository files and structure
- Read file contents
- Create new branches
- Create and update files
- **Issue Management**:
- Create new issues
- Comment on existing issues
- Close and update issues
- Search across issues
- **Pull Request Workflow**:
- Create pull requests
- Review code changes
- Add comments to PRs
This integration enables Claude to participate as a full team member in the development process, not just as a code
generation tool. Claude's GitHub account ([bm-claudeai](https://github.com/bm-claudeai)) is a member of the Basic
Machines organization with direct contributor access to the codebase.
### Collaborative Development Process
With GitHub integration, the development workflow includes:
1. **Direct code review** - Claude can analyze PRs and provide detailed feedback
2. **Contribution tracking** - All of Claude's contributions are properly attributed in the Git history
3. **Branch management** - Claude can create feature branches for implementations
4. **Documentation maintenance** - Claude can keep documentation updated as the code evolves
With this integration, the AI assistant is a full-fledged team member rather than just a tool for generating code
snippets.
### Basic Memory Pro
Basic Memory Pro is a desktop GUI application that wraps the basic-memory CLI/MCP tools:
- Built with Tauri (Rust), React (TypeScript), and a Python FastAPI sidecar
- Provides visual knowledge graph exploration and project management
- Uses the same core codebase but adds a desktop-friendly interface
- Project configuration is shared between CLI and Pro versions
- Multiple project support with visual switching interface
local repo: /Users/phernandez/dev/basicmachines/basic-memory-pro
github: https://github.com/basicmachines-co/basic-memory-pro
## Release and Version Management
Basic Memory uses `uv-dynamic-versioning` for automatic version management based on git tags:
### Version Types
- **Development versions**: Automatically generated from commits (e.g., `0.12.4.dev26+468a22f`)
- **Beta releases**: Created by tagging with beta suffixes (e.g., `v0.13.0b1`, `v0.13.0rc1`)
- **Stable releases**: Created by tagging with version numbers (e.g., `v0.13.0`)
### Release Workflows
#### Development Builds (Automatic)
- Triggered on every push to `main` branch
- Publishes dev versions like `0.12.4.dev26+468a22f` to PyPI
- Allows continuous testing of latest changes
- Users install with: `pip install basic-memory --pre --force-reinstall`
#### Beta/RC Releases (Manual)
- Create beta tag: `git tag v0.13.0b1 && git push origin v0.13.0b1`
- Automatically builds and publishes to PyPI as pre-release
- Users install with: `pip install basic-memory --pre`
- Use for milestone testing before stable release
#### Stable Releases (Automated)
- Use the automated release system: `just release v0.13.0`
- Includes comprehensive quality checks (lint, format, type-check, tests)
- Automatically updates version in `__init__.py`
- Creates git tag and pushes to GitHub
- Triggers GitHub Actions workflow for:
- PyPI publication
- Homebrew formula update (requires HOMEBREW_TOKEN secret)
**Manual method (legacy):**
- Create version tag: `git tag v0.13.0 && git push origin v0.13.0`
#### Homebrew Formula Updates
- Automatically triggered after successful PyPI release for **stable releases only**
- **Stable releases** (e.g., v0.13.7) automatically update the main `basic-memory` formula
- **Pre-releases** (dev/beta/rc) are NOT automatically updated - users must specify version manually
- Updates formula in `basicmachines-co/homebrew-basic-memory` repo
- Requires `HOMEBREW_TOKEN` secret in GitHub repository settings:
- Create a fine-grained Personal Access Token with `Contents: Read and Write` and `Actions: Read` scopes on `basicmachines-co/homebrew-basic-memory`
- Add as repository secret named `HOMEBREW_TOKEN` in `basicmachines-co/basic-memory`
- Formula updates include new version URL and SHA256 checksum
### For Development
- **Automated releases**: Use `just release v0.13.x` for stable releases and `just beta v0.13.0b1` for beta releases
- **Quality gates**: All releases require passing lint, format, type-check, and test suites
- **Version management**: Versions automatically derived from git tags via `uv-dynamic-versioning`
- **Configuration**: `pyproject.toml` uses `dynamic = ["version"]`
- **Release automation**: `__init__.py` updated automatically during release process
- **CI/CD**: GitHub Actions handles building and PyPI publication
## Development Notes
- make sure you sign off on commits
+8 -86
View File
@@ -27,25 +27,13 @@ project and how to get started as a developer.
> **Note**: Basic Memory uses [just](https://just.systems) as a modern command runner. Install with `brew install just` or `cargo install just`.
3. **Activate the Virtual Environment**
3. **Run the Tests**:
```bash
source .venv/bin/activate
```
4. **Run the Tests**:
```bash
# Run all tests with unified coverage (unit + integration)
# Run all tests
just test
# Run unit tests only (fast, no coverage)
just test-unit
# Run integration tests only (fast, no coverage)
just test-int
# Generate HTML coverage report
just coverage
# or
uv run pytest -p pytest_mock -v
# Run a specific test
pytest tests/path/to/test_file.py::test_function_name
```
@@ -141,7 +129,7 @@ agreement to the DCO.
## Code Style Guidelines
- **Python Version**: Python 3.12+ with full type annotations (3.12+ required for type parameter syntax)
- **Python Version**: Python 3.12+ with full type annotations
- **Line Length**: 100 characters maximum
- **Formatting**: Use ruff for consistent styling
- **Import Order**: Standard lib, third-party, local imports
@@ -151,78 +139,12 @@ agreement to the DCO.
## Testing Guidelines
### Test Structure
Basic Memory uses two test directories with unified coverage reporting:
- **`tests/`**: Unit tests that test individual components in isolation
- Fast execution with extensive mocking
- Test individual functions, classes, and modules
- Run with: `just test-unit` (no coverage, fast)
- **`test-int/`**: Integration tests that test real-world scenarios
- Test full workflows with real database and file operations
- Include performance benchmarks
- More realistic but slower than unit tests
- Run with: `just test-int` (no coverage, fast)
### Running Tests
```bash
# Run all tests with unified coverage report
just test
# Run only unit tests (fast iteration)
just test-unit
# Run only integration tests
just test-int
# Generate HTML coverage report
just coverage
# Run specific test
pytest tests/path/to/test_file.py::test_function_name
# Run tests excluding benchmarks
pytest -m "not benchmark"
# Run only benchmark tests
pytest -m benchmark test-int/test_sync_performance_benchmark.py
```
### Performance Benchmarks
The `test-int/test_sync_performance_benchmark.py` file contains performance benchmarks that measure sync and indexing speed:
- `test_benchmark_sync_100_files` - Small repository performance
- `test_benchmark_sync_500_files` - Medium repository performance
- `test_benchmark_sync_1000_files` - Large repository performance (marked slow)
- `test_benchmark_resync_no_changes` - Re-sync performance baseline
Run benchmarks with:
```bash
# Run all benchmarks (excluding slow ones)
pytest test-int/test_sync_performance_benchmark.py -v -m "benchmark and not slow"
# Run all benchmarks including slow ones
pytest test-int/test_sync_performance_benchmark.py -v -m benchmark
# Run specific benchmark
pytest test-int/test_sync_performance_benchmark.py::test_benchmark_sync_100_files -v
```
See `test-int/BENCHMARKS.md` for detailed benchmark documentation.
### Testing Best Practices
- **Coverage Target**: We aim for high test coverage for all code
- **Coverage Target**: We aim for 100% test coverage for all code
- **Test Framework**: Use pytest for unit and integration tests
- **Mocking**: Avoid mocking in integration tests; use sparingly in unit tests
- **Mocking**: Use pytest-mock for mocking dependencies only when necessary
- **Edge Cases**: Test both normal operation and edge cases
- **Database Testing**: Use in-memory SQLite for testing database operations
- **Fixtures**: Use async pytest fixtures for setup and teardown
- **Markers**: Use `@pytest.mark.benchmark` for benchmarks, `@pytest.mark.slow` for slow tests
## Release Process
+7 -27
View File
@@ -1,46 +1,26 @@
FROM python:3.12-slim-bookworm
# Build arguments for user ID and group ID (defaults to 1000)
ARG UID=1000
ARG GID=1000
# Copy uv from official image
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
# Set environment variables
# UV_PYTHON_INSTALL_DIR ensures Python is installed to a persistent location
# that survives in the final image (not in /root/.local which gets lost)
# UV_PYTHON_PREFERENCE=only-managed tells uv to use its managed Python version
ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
UV_PYTHON_INSTALL_DIR=/python \
UV_PYTHON_PREFERENCE=only-managed
# Create a group and user with the provided UID/GID
# Check if the GID already exists, if not create appgroup
RUN (getent group ${GID} || groupadd --gid ${GID} appgroup) && \
useradd --uid ${UID} --gid ${GID} --create-home --shell /bin/bash appuser
PYTHONDONTWRITEBYTECODE=1
# Copy the project into the image
ADD . /app
# Install Python 3.13 explicitly and sync the project
# Sync the project into a new environment, asserting the lockfile is up to date
WORKDIR /app
RUN uv python install 3.13
RUN uv sync --locked --python 3.13
RUN uv sync --locked
# Create necessary directories and set ownership
RUN mkdir -p /app/data/basic-memory /app/.basic-memory && \
chown -R appuser:${GID} /app
# Create data directory
RUN mkdir -p /app/data
# Set default data directory and add venv to PATH
ENV BASIC_MEMORY_HOME=/app/data/basic-memory \
BASIC_MEMORY_PROJECT_ROOT=/app/data \
ENV BASIC_MEMORY_HOME=/app/data \
PATH="/app/.venv/bin:$PATH"
# Switch to the non-root user
USER appuser
# Expose port
EXPOSE 8000
@@ -49,4 +29,4 @@ HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
CMD basic-memory --version || exit 1
# Use the basic-memory entrypoint to run the MCP server with default SSE transport
CMD ["basic-memory", "mcp", "--transport", "sse", "--host", "0.0.0.0", "--port", "8000"]
CMD ["basic-memory", "mcp", "--transport", "sse", "--host", "0.0.0.0", "--port", "8000"]
-494
View File
@@ -1,494 +0,0 @@
# Note Format Reference
Every document in Basic Memory is a plain Markdown file. Files are the source of truth — changes to files automatically update the knowledge graph in the database. You maintain complete ownership, files work with git, and knowledge persists independently of any AI conversation.
## Document Structure
A note has three parts: YAML frontmatter, content (observations), and relations.
```markdown
---
title: Coffee Brewing Methods
type: note
tags: [coffee, brewing]
permalink: coffee-brewing-methods
---
# Coffee Brewing Methods
## Observations
- [method] Pour over provides more flavor clarity than French press
- [technique] Water temperature at 205°F extracts optimal compounds #brewing
- [preference] Ethiopian beans work well with lighter roasts (personal experience)
## Relations
- relates_to [[Coffee Bean Origins]]
- requires [[Proper Grinding Technique]]
- contrasts_with [[Tea Brewing Methods]]
```
The `## Observations` and `## Relations` headings are conventional but not required — the parser detects observations and relations by their syntax patterns anywhere in the document.
## Frontmatter
YAML metadata between `---` fences at the top of the file.
| Field | Required | Default | Description |
|-------|----------|---------|-------------|
| `title` | No | filename stem | Used for linking and references. Auto-set from filename if missing. |
| `type` | No | `note` | Entity type. Used for schema resolution and filtering. |
| `tags` | No | `[]` | List or comma-separated string. Used for organization and search. |
| `permalink` | No | generated from title | Stable identifier. Persists even if the file moves. |
| `schema` | No | none | Schema attachment — dict (inline), string (reference), or omitted (implicit). |
Custom fields are allowed. Any key not in the standard set is stored as `entity_metadata` and indexed for search and filtering.
```yaml
---
title: Paul Graham
type: Person
tags: [startups, essays, lisp]
permalink: paul-graham
status: active
source: wikipedia
---
```
Here `status` and `source` are custom fields stored in `entity_metadata`.
### Frontmatter Value Handling
YAML automatically converts some values to native types. Basic Memory normalizes them:
- Date strings (`2025-10-24`) → kept as ISO format strings
- Numbers (`1.0`) → converted to strings
- Booleans (`true`) → converted to strings (`"True"`)
- Lists and dicts → preserved, items normalized recursively
This prevents errors when downstream code expects string values.
## Observations
An observation is a categorized fact about the entity. Written as a Markdown list item.
**Syntax:**
```
- [category] content text #tag1 #tag2 (context)
```
| Part | Required | Description |
|------|----------|-------------|
| `[category]` | Yes | Classification in square brackets. Any text except `[]()` chars. |
| content | Yes | The fact or statement. |
| `#tags` | No | Inline tags. Space-separated, each starting with `#`. |
| `(context)` | No | Parenthesized text at end of line. Supporting details or source. |
### Examples
```markdown
- [tech] Uses SQLite for storage #database
- [design] Follows local-first architecture #architecture
- [decision] Selected bcrypt for passwords #security (based on OWASP audit)
- [name] Paul Graham
- [expertise] Startups
- [expertise] Lisp
- [expertise] Essay writing
```
Array-like fields use repeated categories — multiple `[expertise]` observations above.
### What Is Not an Observation
The parser excludes these list item patterns:
| Pattern | Example | Reason |
|---------|---------|--------|
| Checkboxes | `- [ ] Todo item`, `- [x] Done`, `- [-] Cancelled` | Task list syntax |
| Markdown links | `- [text](url)` | URL link syntax |
| Bare wiki links | `- [[Target]]` | Treated as a relation instead |
A list item with `#tags` but no `[category]` is still parsed — the tags are extracted and the category defaults to `Note`.
## Relations
Relations connect documents to form the knowledge graph. There are two kinds.
### Explicit Relations
Written as list items with a relation type and a `[[wiki link]]` target.
**Syntax:**
```
- relation_type [[Target Entity]] (context)
```
| Part | Required | Description |
|------|----------|-------------|
| `relation_type` | No | Text before `[[`. Defaults to `relates_to` if omitted. |
| `[[Target]]` | Yes | Wiki link to the target entity. Matched by title or permalink. |
| `(context)` | No | Parenthesized text after `]]`. Supporting details. |
### Examples
```markdown
- implements [[Search Design]]
- depends_on [[Database Schema]]
- works_at [[Y Combinator]] (co-founder)
- [[Some Entity]]
```
The last example — a bare `[[wiki link]]` in a list item — gets relation type `relates_to`.
Common relation types:
- `implements`, `depends_on`, `relates_to`, `inspired_by`
- `extends`, `part_of`, `contains`, `pairs_with`
- `works_at`, `authored`, `collaborated_with`
Any text works as a relation type. These are conventions, not a fixed set.
### Inline References
Wiki links appearing in regular prose (not as list items) create implicit `links_to` relations.
```markdown
This builds on [[Core Design]] and uses [[Utility Functions]].
```
This creates two relations: `links_to [[Core Design]]` and `links_to [[Utility Functions]]`.
### Forward References
Relations can link to entities that don't exist yet. Basic Memory resolves them when the target is created.
## Permalinks and memory:// URLs
Every document has a unique **permalink** — a stable identifier derived from its title. You can set one explicitly in frontmatter, or let the system generate it.
```yaml
permalink: auth-approaches-2024
```
Permalinks form the basis of `memory://` URLs:
```
memory://auth-approaches-2024 # By permalink
memory://Authentication Approaches # By title (auto-resolves)
memory://project/auth-approaches # By path
```
Pattern matching is supported:
```
memory://auth* # Starts with "auth"
memory://*/approaches # Ends with "approaches"
memory://project/*/requirements # Nested wildcard
```
## Schemas
Schemas declare the expected structure of a note — which observation categories and relation types a well-formed note should have. They use Picoschema, a compact notation from Google's Dotprompt that fits naturally in YAML frontmatter.
### Picoschema Syntax
```yaml
schema:
name: string, full name # required field with description
email?: string, contact email # ? = optional
role?: string, job title
works_at?: Organization, employer # capitalized type = entity reference
tags?(array): string, categories # array of type
status?(enum): [active, inactive] # enum with allowed values
metadata?(object): # nested object
updated_at?: string
source?: string
```
| Notation | Meaning | Example |
|----------|---------|---------|
| `field: type` | Required field | `name: string` |
| `field?: type` | Optional field | `role?: string` |
| `field(array): type` | Array of values | `expertise(array): string` |
| `field?(enum): [vals]` | Enum with allowed values | `status?(enum): [active, inactive]` |
| `field?(object):` | Nested object with sub-fields | `metadata?(object):` |
| `, description` | Description after comma | `name: string, full name` |
| `EntityName` | Capitalized type = entity reference | `works_at?: Organization` |
**Scalar types:** `string`, `integer`, `number`, `boolean`, `any`
Any type not in that set whose first letter is uppercase is treated as an entity reference (a relation target).
### Schema-to-Note Mapping
Schemas validate against existing observation/relation syntax. Note authors don't learn new syntax.
| Schema Declaration | Maps To | Example in Note |
|--------------------|---------|-----------------|
| `field: string` | Observation `[field] value` | `- [name] Paul Graham` |
| `field?(array): string` | Multiple `[field]` observations | `- [expertise] Lisp` (repeated) |
| `field?: EntityType` | Relation `field [[Target]]` | `- works_at [[Y Combinator]]` |
| `field?(array): EntityType` | Multiple `field` relations | `- authored [[Book]]` (repeated) |
| `tags` | Frontmatter `tags` array | `tags: [startups, essays]` |
| `field?(enum): [vals]` | Observation `[field] value` where value is in the set | `- [status] active` |
Observations and relations not covered by the schema are valid — schemas describe a subset, not a straitjacket.
### Schema Attachment
Three ways to attach a schema to a note, resolved in priority order:
**1. Inline schema**`schema` is a dict in frontmatter:
```yaml
---
title: Team Standup 2024-01-15
type: meeting
schema:
attendees(array): string, who was there
decisions(array): string, what was decided
action_items(array): string, follow-ups
blockers?(array): string, anything stuck
---
```
Good for one-off structured notes or prototyping a schema before extracting it.
**2. Explicit reference**`schema` is a string naming a schema note:
```yaml
---
title: Basic Memory
schema: SoftwareProject
---
```
or by permalink:
```yaml
---
title: LLM Memory Patterns
schema: schema/research-project
---
```
Use when the note's `type` differs from the schema it should validate against, or when multiple schema variants exist.
**3. Implicit by type** — no `schema` field, resolved by matching `type`:
```yaml
---
title: Paul Graham
type: Person
---
```
The system looks up a schema note where `entity: Person`. If found, it applies. If not, no validation occurs.
**4. No schema** — perfectly fine. Most notes don't need one.
### Schema Notes
A schema is itself a Basic Memory note with `type: schema`. It lives anywhere (though `schema/` is the conventional directory).
```yaml
# schema/Person.md
---
title: Person
type: schema
entity: Person
version: 1
schema:
name: string, full name
role?: string, job title or position
works_at?: Organization, employer
expertise?(array): string, areas of knowledge
email?: string, contact email
settings:
validation: warn
---
# Person
A human individual in the knowledge graph.
```
| Field | Required | Description |
|-------|----------|-------------|
| `type` | Yes | Must be `schema` |
| `entity` | Yes | The entity type this schema describes (e.g., `Person`) |
| `version` | No | Schema version number (default: `1`) |
| `schema` | Yes | Picoschema dict defining the fields |
| `settings.validation` | No | Validation mode (default: `warn`) |
Schema notes are regular notes — they show up in search, can have observations and relations, and participate in the knowledge graph.
### Validation Modes
| Mode | Behavior |
|------|----------|
| `warn` | Warnings in output, doesn't block (default) |
| `strict` | Errors that block sync, for CI/CD enforcement |
| `off` | No validation |
### Validation Output
```
$ bm schema validate people/ada-lovelace.md
⚠ Person schema validation:
- Missing required field: name (expected [name] observation)
- Missing optional field: role
- Missing optional field: works_at (no relation found)
Unmatched observations: [fact] ×2, [born] ×1
Unmatched relations: collaborated_with
```
"Unmatched" items are informational — observations and relations the schema doesn't cover.
### Schema Inference
Generate schemas from existing notes by analyzing observation and relation frequency:
```
$ bm schema infer Person
Analyzing 30 notes with type: Person...
Observations found:
[name] 30/30 100% → name: string
[role] 27/30 90% → role?: string
[expertise] 18/30 60% → expertise?(array): string
[email] 8/30 27% → email?: string
Relations found:
works_at 22/30 73% → works_at?: Organization
Suggested schema:
name: string, full name
role?: string, job title
expertise?(array): string, areas of knowledge
email?: string, contact email
works_at?: Organization, employer
Save to schema/Person.md? [y/n]
```
Frequency thresholds:
- **100% present** → required field
- **25%+ present** → optional field
- **Below 25%** → excluded from suggestion
### Schema Drift Detection
Track how usage patterns shift over time:
```
$ bm schema diff Person
Schema drift detected:
+ expertise: now in 81% of notes (was 12%)
- department: dropped to 3% of notes
~ works_at: cardinality changed (one → many)
Update schema? [y/n/review]
```
## Complete Examples
### Simple Note (No Schema)
```markdown
---
title: Project Ideas
type: note
tags: [ideas, brainstorm]
---
# Project Ideas
## Observations
- [idea] Build a CLI tool for markdown linting #tooling
- [idea] Create a recipe knowledge base #cooking
- [priority] Focus on developer tools first (Q1 goal)
## Relations
- inspired_by [[Developer Workflow Research]]
- part_of [[Q1 Planning]]
```
### Schema-Validated Note
Schema at `schema/Person.md`:
```yaml
---
title: Person
type: schema
entity: Person
version: 1
schema:
name: string, full name
role?: string, job title or position
works_at?: Organization, employer
expertise?(array): string, areas of knowledge
email?: string, contact email
settings:
validation: warn
---
# Person
A human individual in the knowledge graph.
```
Note at `people/paul-graham.md`:
```markdown
---
title: Paul Graham
type: Person
tags: [startups, essays, lisp]
---
# Paul Graham
## Observations
- [name] Paul Graham
- [role] Essayist and investor
- [expertise] Startups
- [expertise] Lisp
- [expertise] Essay writing
- [fact] Created Viaweb, the first web app
## Relations
- works_at [[Y Combinator]]
- authored [[Hackers and Painters]]
```
The `[fact]` observation and `authored` relation are not in the schema — they're valid, just unmatched. The schema only checks that `[name]` exists (required) and looks for optional fields like `[role]`, `[expertise]`, and `works_at`.
### Inline Schema Note
```markdown
---
title: Team Standup 2024-01-15
type: meeting
schema:
attendees(array): string, who was there
decisions(array): string, what was decided
action_items(array): string, follow-ups
blockers?(array): string, anything stuck
---
# Team Standup 2024-01-15
## Observations
- [attendees] Paul
- [attendees] Sarah
- [decisions] Ship v2 by Friday
- [action_items] Paul to review PR #42
- [blockers] Waiting on API credentials
```
+103 -241
View File
@@ -1,4 +1,3 @@
<!-- mcp-name: io.github.basicmachines-co/basic-memory -->
[![License: AGPL v3](https://img.shields.io/badge/License-AGPL_v3-blue.svg)](https://www.gnu.org/licenses/agpl-3.0)
[![PyPI version](https://badge.fury.io/py/basic-memory.svg)](https://badge.fury.io/py/basic-memory)
[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
@@ -6,16 +5,7 @@
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
![](https://badge.mcpx.dev?type=server 'MCP Server')
![](https://badge.mcpx.dev?type=dev 'MCP Dev')
## 🚀 Basic Memory Cloud is Live!
- **Cross-device and multi-platform support is here.** Your knowledge graph now works on desktop, web, and mobile.
- **Cloud is optional.** The local-first open-source workflow continues as always.
- **OSS discount:** use code `{{OSS_DISCOUNT_CODE}}` for 20% off for 3 months.
[Sign up now →](https://basicmemory.com)
with a 7 day free trial
[![smithery badge](https://smithery.ai/badge/@basicmachines-co/basic-memory)](https://smithery.ai/server/@basicmachines-co/basic-memory)
# Basic Memory
@@ -24,7 +14,10 @@ Claude, while keeping everything in simple Markdown files on your computer. It u
enable any compatible LLM to read and write to your local knowledge base.
- Website: https://basicmemory.com
- Documentation: https://docs.basicmemory.com
- Company: https://basicmachines.co
- Documentation: https://memory.basicmachines.co
- Discord: https://discord.gg/tyvKNccgqN
- YouTube: https://www.youtube.com/@basicmachines-co
## Pick up your conversation right where you left off
@@ -40,6 +33,10 @@ https://github.com/user-attachments/assets/a55d8238-8dd0-454a-be4c-8860dbbd0ddc
# Install with uv (recommended)
uv tool install basic-memory
# or with Homebrew
brew tap basicmachines-co/basic-memory
brew install basic-memory
# Configure Claude Desktop (edit ~/Library/Application Support/Claude/claude_desktop_config.json)
# Add this to your config:
{
@@ -62,6 +59,30 @@ uv tool install basic-memory
You can view shared context via files in `~/basic-memory` (default directory location).
### Alternative Installation via Smithery
You can use [Smithery](https://smithery.ai/server/@basicmachines-co/basic-memory) to automatically configure Basic
Memory for Claude Desktop:
```bash
npx -y @smithery/cli install @basicmachines-co/basic-memory --client claude
```
This installs and configures Basic Memory without requiring manual edits to the Claude Desktop configuration file. Note: The Smithery installation uses their hosted MCP server, while your data remains stored locally as Markdown files.
### Add to Cursor
Once you have installed Basic Memory revisit this page for the 1-click installer for Cursor:
[![Install MCP Server](https://cursor.com/deeplink/mcp-install-dark.svg)](https://cursor.com/install-mcp?name=basic-memory&config=eyJjb21tYW5kIjoiL1VzZXJzL2RyZXcvLmxvY2FsL2Jpbi91dnggYmFzaWMtbWVtb3J5IG1jcCJ9)
### Glama.ai
<a href="https://glama.ai/mcp/servers/o90kttu9ym">
<img width="380" height="200" src="https://glama.ai/mcp/servers/o90kttu9ym/badge" alt="basic-memory MCP server" />
</a>
## Why Basic Memory?
Most LLM interactions are ephemeral - you ask a question, get an answer, and everything is forgotten. Each conversation
@@ -92,9 +113,6 @@ With Basic Memory, you can:
- Keep everything local and under your control
- Use familiar tools like Obsidian to view and edit notes
- Build a personal knowledge base that grows over time
- Sync your knowledge to the cloud with bidirectional synchronization
- Authenticate and manage cloud projects with subscription validation
- Mount cloud storage for direct file access
## How It Works in Practice
@@ -148,7 +166,8 @@ The note embeds semantic content and links to other topics via simple Markdown f
3. You see this file on your computer in real time in the current project directory (default `~/$HOME/basic-memory`).
- Realtime sync can be enabled via running `basic-memory sync --watch`
- Realtime sync is enabled by default starting with v0.12.0
- Project switching during conversations is supported starting with v0.13.0
4. In a chat with the LLM, you can reference a topic:
@@ -206,7 +225,7 @@ title: <Entity title>
type: <The type of Entity> (e.g. note)
permalink: <a uri slug>
- <optional metadata> (such as tags)
- <optional metadata> (such as tags)
```
### Observations
@@ -258,6 +277,13 @@ Examples of relations:
```
## Using with VS Code
For one-click installation, click one of the install buttons below...
[![Install with UV in VS Code](https://img.shields.io/badge/VS_Code-UV-0098FF?style=flat-square&logo=visualstudiocode&logoColor=white)](https://insiders.vscode.dev/redirect/mcp/install?name=basic-memory&config=%7B%22command%22%3A%22uvx%22%2C%22args%22%3A%5B%22basic-memory%22%2C%22mcp%22%5D%7D) [![Install with UV in VS Code Insiders](https://img.shields.io/badge/VS_Code_Insiders-UV-24bfa5?style=flat-square&logo=visualstudiocode&logoColor=white)](https://insiders.vscode.dev/redirect/mcp/install?name=basic-memory&config=%7B%22command%22%3A%22uvx%22%2C%22args%22%3A%5B%22basic-memory%22%2C%22mcp%22%5D%7D&quality=insiders)
You can use Basic Memory with VS Code to easily retrieve and store information while coding. Click the installation buttons above for one-click setup, or follow the manual installation instructions below.
### Manual Installation
Add the following JSON block to your User Settings (JSON) file in VS Code. You can do this by pressing `Ctrl + Shift + P` and typing `Preferences: Open User Settings (JSON)`.
@@ -287,8 +313,6 @@ Optionally, you can add it to a file called `.vscode/mcp.json` in your workspace
}
```
You can use Basic Memory with VS Code to easily retrieve and store information while coding.
## Using with Claude Desktop
Basic Memory is built using the MCP (Model Context Protocol) and works with the Claude desktop app (https://claude.ai/):
@@ -312,7 +336,8 @@ for OS X):
}
```
If you want to use a specific project (see [Multiple Projects](#multiple-projects) below), update your Claude Desktop
If you want to use a specific project (see [Multiple Projects](docs/User%20Guide.md#multiple-projects)), update your
Claude Desktop
config:
```json
@@ -322,9 +347,9 @@ config:
"command": "uvx",
"args": [
"basic-memory",
"mcp",
"--project",
"your-project-name"
"your-project-name",
"mcp"
]
}
}
@@ -333,134 +358,27 @@ config:
2. Sync your knowledge:
```bash
# One-time sync of local knowledge updates
basic-memory sync
Basic Memory will sync the files in your project in real time if you make manual edits.
3. In Claude Desktop, the LLM can now use these tools:
# Run realtime sync process (recommended)
basic-memory sync --watch
```
3. Cloud features (optional, requires subscription):
```bash
# Authenticate with cloud (stores OAuth token locally)
basic-memory cloud login
# (Optional) install/configure rclone for file sync commands
basic-memory cloud setup
# Check cloud auth + health
basic-memory cloud status
```
**Per-Project Cloud Routing** (API key based):
Individual projects can be routed through the cloud while others stay local. This uses an API key for routed
project calls:
```bash
# Save an API key (create one in the web app or via CLI)
basic-memory cloud set-key bmc_abc123...
# Or create one via CLI (requires OAuth login first)
basic-memory cloud create-key "my-laptop"
# Set a project to route through cloud
basic-memory project set-cloud research
# Revert a project to local mode
basic-memory project set-local research
# List projects and route metadata
basic-memory project list
```
`basic-memory cloud login` / `basic-memory cloud logout` are authentication commands. They do not change default CLI
routing behavior.
**Routing Flags**:
Use routing flags to disambiguate command targets:
```bash
# Force local routing for this command
basic-memory status --local
basic-memory project list --local
basic-memory project ls --name main --local
# Force cloud routing for this command
basic-memory status --cloud
basic-memory project info my-project --cloud
basic-memory project ls --name main --cloud
```
No-flag behavior defaults to local when no project context is present.
The local MCP server (`basic-memory mcp`) always uses local routing (including `--transport stdio`).
**CLI Note Editing (`tool edit-note`):**
```bash
# Append content
basic-memory tool edit-note project-plan --operation append --content $'\n## Next Steps\n- Finalize rollout'
# Find/replace with replacement count validation
basic-memory tool edit-note docs/api --operation find_replace --find-text "v0.14.0" --content "v0.15.0" --expected-replacements 2
# Replace a section body
basic-memory tool edit-note docs/setup --operation replace_section --section "## Installation" --content $'Updated install steps\n- Run just install'
# JSON metadata output for integrations
basic-memory tool edit-note docs/setup --operation append --content $'\n- Added note' --format json
```
4. In Claude Desktop, the LLM can now use these tools:
**Content Management:**
```
write_note(title, content, folder, tags, output_format="text"|"json") - Create or update notes
read_note(identifier, page, page_size, output_format="text"|"json") - Read notes by title or permalink
read_content(path) - Read raw file content (text, images, binaries)
view_note(identifier) - View notes as formatted artifacts
edit_note(identifier, operation, content, output_format="text"|"json") - Edit notes incrementally
move_note(identifier, destination_path, output_format="text"|"json") - Move notes with database consistency
delete_note(identifier, output_format="text"|"json") - Delete notes from knowledge base
```
**Knowledge Graph Navigation:**
```
build_context(url, depth, timeframe, output_format="json"|"text") - Navigate knowledge graph via memory:// URLs
recent_activity(type, depth, timeframe, output_format="text"|"json") - Find recently updated information
list_directory(dir_name, depth) - Browse directory contents with filtering
```
**Search & Discovery:**
```
search(query, page, page_size) - Search across your knowledge base
search_notes(query, page, page_size, search_type, types, entity_types, after_date, metadata_filters, tags, status, project) - Search with filters
search_by_metadata(filters, limit, offset, project) - Structured frontmatter search
```
**Project Management:**
```
list_memory_projects(output_format="text"|"json") - List all available projects
create_memory_project(project_name, project_path, output_format="text"|"json") - Create new projects
get_current_project() - Show current project stats
sync_status() - Check synchronization status
```
`output_format` defaults to `"text"` for these tools, preserving current human-readable responses.
`build_context` defaults to `"json"` and can be switched to `"text"` when compact markdown output is preferred.
**Cloud Discovery (opt-in):**
```
cloud_info() - Show optional Cloud overview and setup guidance
release_notes() - Show latest release notes
```
**Visualization:**
```
write_note(title, content, folder, tags) - Create or update notes
read_note(identifier, page, page_size) - Read notes by title or permalink
edit_note(identifier, operation, content) - Edit notes incrementally (append, prepend, find/replace)
move_note(identifier, destination_path) - Move notes with database consistency
view_note(identifier) - Display notes as formatted artifacts for better readability
build_context(url, depth, timeframe) - Navigate knowledge graph via memory:// URLs
search_notes(query, page, page_size) - Search across your knowledge base
recent_activity(type, depth, timeframe) - Find recently updated information
canvas(nodes, edges, title, folder) - Generate knowledge visualizations
list_memory_projects() - List all available projects with status
switch_project(project_name) - Switch to different project context
get_current_project() - Show current project and statistics
create_memory_project(name, path, set_default) - Create new projects
delete_project(name) - Delete projects from configuration
set_default_project(name) - Set default project
sync_status() - Check file synchronization status
```
5. Example prompts to try:
@@ -471,119 +389,63 @@ canvas(nodes, edges, title, folder) - Generate knowledge visualizations
"Create a canvas visualization of my project components"
"Read my notes on the authentication system"
"What have I been working on in the past week?"
"Switch to my work-notes project"
"List all my available projects"
"Edit my coffee brewing note to add a new technique"
"Move my old meeting notes to the archive folder"
```
## Futher info
See the [Documentation](https://docs.basicmemory.com) for more info, including:
See the [Documentation](https://memory.basicmachines.co/) for more info, including:
- [Complete User Guide](https://docs.basicmemory.com/user-guide/)
- [CLI tools](https://docs.basicmemory.com/guides/cli-reference/)
- [Cloud CLI and Sync](https://docs.basicmemory.com/guides/cloud-cli/)
- [Managing multiple Projects](https://docs.basicmemory.com/guides/cli-reference/#project)
- [Importing data from OpenAI/Claude Projects](https://docs.basicmemory.com/guides/cli-reference/#import)
- [Complete User Guide](https://memory.basicmachines.co/docs/user-guide)
- [CLI tools](https://memory.basicmachines.co/docs/cli-reference)
- [Managing multiple Projects](https://memory.basicmachines.co/docs/cli-reference#project)
- [Importing data from OpenAI/Claude Projects](https://memory.basicmachines.co/docs/cli-reference#import)
## Logging
Basic Memory uses [Loguru](https://github.com/Delgan/loguru) for logging. The logging behavior varies by entry point:
| Entry Point | Default Behavior | Use Case |
|-------------|------------------|----------|
| CLI commands | File only | Prevents log output from interfering with command output |
| MCP server | File only | Stdout would corrupt the JSON-RPC protocol |
| API server | File (local) or stdout (cloud) | Docker/cloud deployments use stdout |
**Log file location:** `~/.basic-memory/basic-memory.log` (10MB rotation, 10 days retention)
### Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `BASIC_MEMORY_LOG_LEVEL` | `INFO` | Log level: DEBUG, INFO, WARNING, ERROR |
| `BASIC_MEMORY_CLOUD_MODE` | `false` | When `true`, API logs to stdout with structured context |
| `BASIC_MEMORY_FORCE_LOCAL` | `false` | When `true`, forces local API routing |
| `BASIC_MEMORY_FORCE_CLOUD` | `false` | When `true`, forces cloud API routing |
| `BASIC_MEMORY_EXPLICIT_ROUTING` | `false` | When `true`, marks route selection as explicit (`--local`/`--cloud`) |
| `BASIC_MEMORY_ENV` | `dev` | Set to `test` for test mode (stderr only) |
### Examples
## Installation Options
### Stable Release
```bash
# Enable debug logging
BASIC_MEMORY_LOG_LEVEL=DEBUG basic-memory sync
# View logs
tail -f ~/.basic-memory/basic-memory.log
# Cloud/Docker mode (stdout logging with structured context)
BASIC_MEMORY_CLOUD_MODE=true uvicorn basic_memory.api.app:app
pip install basic-memory
```
## Development
### Running Tests
Basic Memory supports dual database backends (SQLite and Postgres). By default, tests run against SQLite. Set `BASIC_MEMORY_TEST_POSTGRES=1` to run against Postgres (uses testcontainers - Docker required).
**Quick Start:**
### Beta/Pre-releases
```bash
# Run all tests against SQLite (default, fast)
just test-sqlite
# Run all tests against Postgres (uses testcontainers)
just test-postgres
# Run both SQLite and Postgres tests
just test
pip install basic-memory --pre
```
**Available Test Commands:**
- `just test` - Run all tests against both SQLite and Postgres
- `just test-sqlite` - Run all tests against SQLite (fast, no Docker needed)
- `just test-postgres` - Run all tests against Postgres (uses testcontainers)
- `just test-unit-sqlite` - Run unit tests against SQLite
- `just test-unit-postgres` - Run unit tests against Postgres
- `just test-int-sqlite` - Run integration tests against SQLite
- `just test-int-postgres` - Run integration tests against Postgres
- `just test-windows` - Run Windows-specific tests (auto-skips on other platforms)
- `just test-benchmark` - Run performance benchmark tests
- `just testmon` - Run tests impacted by recent changes (pytest-testmon)
- `just test-smoke` - Run fast MCP end-to-end smoke test
- `just fast-check` - Run fix/format/typecheck + impacted tests + smoke test
- `just doctor` - Run local file <-> DB consistency checks with temp config
**Postgres Testing:**
Postgres tests use [testcontainers](https://testcontainers-python.readthedocs.io/) which automatically spins up a Postgres instance in Docker. No manual database setup required - just have Docker running.
**Testmon Note:** When no files have changed, `just testmon` may collect 0 tests. That's expected and means no impacted tests were detected.
**Test Markers:**
Tests use pytest markers for selective execution:
- `windows` - Windows-specific database optimizations
- `benchmark` - Performance tests (excluded from default runs)
- `smoke` - Fast MCP end-to-end smoke tests
**Other Development Commands:**
### Development Builds
Development versions are automatically published on every commit to main with versions like `0.12.4.dev26+468a22f`:
```bash
just install # Install with dev dependencies
just lint # Run linting checks
just typecheck # Run type checking
just format # Format code with ruff
just fast-check # Fast local loop (fix/format/typecheck + testmon + smoke)
just doctor # Local consistency check (temp config)
just check # Run all quality checks
just migration "msg" # Create database migration
pip install basic-memory --pre --force-reinstall
```
**Local Consistency Check:**
### Docker
Run Basic Memory in a container with volume mounting for your Obsidian vault:
```bash
basic-memory doctor # Verifies file <-> database sync in a temp project
# Clone and start with Docker Compose
git clone https://github.com/basicmachines-co/basic-memory.git
cd basic-memory
# Edit docker-compose.yml to point to your Obsidian vault
# Then start the container
docker-compose up -d
```
See the [justfile](justfile) for the complete list of development commands.
Or use Docker directly:
```bash
docker run -d \
--name basic-memory-server \
-v /path/to/your/obsidian-vault:/data/knowledge:rw \
-v basic-memory-config:/root/.basic-memory:rw \
ghcr.io/basicmachines-co/basic-memory:latest
```
See [Docker Setup Guide](docs/Docker.md) for detailed configuration options, multiple project setup, and integration examples.
## License
-42
View File
@@ -1,42 +0,0 @@
# Docker Compose configuration for Basic Memory with PostgreSQL
# Use this for local development and testing with Postgres backend
#
# Usage:
# docker-compose -f docker-compose-postgres.yml up -d
# docker-compose -f docker-compose-postgres.yml down
services:
postgres:
image: postgres:17
container_name: basic-memory-postgres
environment:
# Local development/test credentials - NOT for production
# These values are referenced by tests and justfile commands
POSTGRES_DB: basic_memory
POSTGRES_USER: basic_memory_user
POSTGRES_PASSWORD: dev_password # Simple password for local testing only
ports:
- "5433:5432"
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U basic_memory_user -d basic_memory"]
interval: 10s
timeout: 5s
retries: 5
restart: unless-stopped
volumes:
# Named volume for Postgres data
postgres_data:
driver: local
# Named volume for persistent configuration
# Database will be stored in Postgres, not in this volume
basic-memory-config:
driver: local
# Network configuration (optional)
# networks:
# basic-memory-net:
# driver: bridge
+431
View File
@@ -0,0 +1,431 @@
---
title: AI Assistant Guide
type: note
permalink: docs/ai-assistant-guide
---
> Note: This is an optional document that can be copy/pasted into the project knowledge for an LLM to provide a full description of how it can work with Basic Memory. It is provided as a helpful resource. The tools contain extensive usage description prompts with enable the LLM to understand them.
You can [download](https://github.com/basicmachines-co/basic-memory/blob/main/docs/AI%20Assistant%20Guide.md) the contents of this file from GitHub
# AI Assistant Guide for Basic Memory
This guide helps you, the AI assistant, use Basic Memory tools effectively when working with users. It covers reading, writing, and navigating knowledge through the Model Context Protocol (MCP).
## Quick Reference
**Essential Tools:**
- `write_note()` - Create/update notes (primary tool)
- `read_note()` - Read existing content
- `search_notes()` - Find information
- `edit_note()` - Modify existing notes incrementally (v0.13.0)
- `move_note()` - Organize files with database consistency (v0.13.0)
**Project Management (v0.13.0):**
- `list_projects()` - Show available projects
- `switch_project()` - Change active project
- `get_current_project()` - Current project info
**Key Principles:**
1. **Build connections** - Rich knowledge graphs > isolated notes
2. **Ask permission** - "Would you like me to record this?"
3. **Use exact titles** - For accurate `[[WikiLinks]]`
4. **Leverage v0.13.0** - Edit incrementally, organize proactively, switch projects contextually
## Overview
Basic Memory allows you and users to record context in local Markdown files, building a rich knowledge base through natural conversations. The system automatically creates a semantic knowledge graph from simple text patterns.
- **Local-First**: All data is stored in plain text files on the user's computer
- **Real-Time**: Users see content updates immediately
- **Bi-Directional**: Both you and users can read and edit notes
- **Semantic**: Simple patterns create a structured knowledge graph
- **Persistent**: Knowledge persists across sessions and conversations
## The Importance of the Knowledge Graph
Basic Memory's value comes from connections between notes, not just the notes themselves. When writing notes, your primary goal should be creating a rich, interconnected knowledge graph.
When creating content, focus on:
1. **Increasing Semantic Density**: Add multiple observations and relations to each note
2. **Using Accurate References**: Aim to reference existing entities by their exact titles
3. **Creating Forward References**: Feel free to reference entities that don't exist yet - Basic Memory will resolve these when they're created later
4. **Creating Bidirectional Links**: When appropriate, connect entities from both directions
5. **Using Meaningful Categories**: Add semantic context with appropriate observation categories
6. **Choosing Precise Relations**: Use specific relation types that convey meaning
Remember that a knowledge graph with 10 heavily connected notes is more valuable than 20 isolated notes. Your job is to help build these connections.
## Core Tools Reference
### Essential Content Management
**Writing knowledge** (most important tool):
```
write_note(
title="Search Design",
content="# Search Design\n...",
folder="specs", # Optional
tags=["search", "design"], # v0.13.0: now searchable!
project="work-notes" # v0.13.0: target specific project
)
```
**Reading knowledge:**
```
read_note("Search Design") # By title
read_note("specs/search-design") # By path
read_note("memory://specs/search") # By memory URL
```
**Viewing notes as formatted artifacts (Claude Desktop):**
```
view_note("Search Design") # Creates readable artifact
view_note("specs/search-design") # By permalink
view_note("memory://specs/search") # By memory URL
```
**Incremental editing** (v0.13.0):
```
edit_note(
identifier="Search Design", # Must be EXACT title/permalink (strict matching)
operation="append", # append, prepend, find_replace, replace_section
content="\n## New Section\nContent here..."
)
```
**⚠️ Important:** `edit_note` requires exact identifiers (no fuzzy matching). Use `search_notes()` first if uncertain.
**File organization** (v0.13.0):
```
move_note(
identifier="Old Note", # Must be EXACT title/permalink (strict matching)
destination="archive/old-note.md" # Folders created automatically
)
```
**⚠️ Important:** `move_note` requires exact identifiers (no fuzzy matching). Use `search_notes()` first if uncertain.
### Project Management (v0.13.0)
```
list_projects() # Show available projects
switch_project("work-notes") # Change active project
get_current_project() # Current project info
```
### Search & Discovery
```
search_notes("authentication system") # v0.13.0: includes frontmatter tags
build_context("memory://specs/search") # Follow knowledge graph connections
recent_activity(timeframe="1 week") # Check what's been updated
```
## memory:// URLs Explained
Basic Memory uses a special URL format to reference entities in the knowledge graph:
- `memory://title` - Reference by title
- `memory://folder/title` - Reference by folder and title
- `memory://permalink` - Reference by permalink
- `memory://path/relation_type/*` - Follow all relations of a specific type
- `memory://path/*/target` - Find all entities with relations to target
## Semantic Markdown Format
Knowledge is encoded in standard markdown using simple patterns:
**Observations** - Facts about an entity:
```markdown
- [category] This is an observation #tag1 #tag2 (optional context)
```
**Relations** - Links between entities:
```markdown
- relation_type [[Target Entity]] (optional context)
```
**Common Categories & Relation Types:**
- Categories: `[idea]`, `[decision]`, `[question]`, `[fact]`, `[requirement]`, `[technique]`, `[recipe]`, `[preference]`
- Relations: `relates_to`, `implements`, `requires`, `extends`, `part_of`, `pairs_with`, `inspired_by`, `originated_from`
## When to Record Context
**Always consider recording context when**:
1. Users make decisions or reach conclusions
2. Important information emerges during conversation
3. Multiple related topics are discussed
4. The conversation contains information that might be useful later
5. Plans, tasks, or action items are mentioned
**Protocol for recording context**:
1. Identify valuable information in the conversation
2. Ask the user: "Would you like me to record our discussion about [topic] in Basic Memory?"
3. If they agree, use `write_note` to capture the information
4. If they decline, continue without recording
5. Let the user know when information has been recorded: "I've saved our discussion about [topic] to Basic Memory."
## Understanding User Interactions
Users will interact with Basic Memory in patterns like:
1. **Creating knowledge**:
```
Human: "Let's write up what we discussed about search."
You: I'll create a note capturing our discussion about the search functionality.
[Use write_note() to record the conversation details]
```
2. **Referencing existing knowledge**:
```
Human: "Take a look at memory://specs/search"
You: I'll examine that information.
[Use build_context() to gather related information]
[Then read_note() to access specific content]
```
3. **Finding information**:
```
Human: "What were our decisions about auth?"
You: Let me find that information for you.
[Use search_notes() to find relevant notes]
[Then build_context() to understand connections]
```
4. **Editing existing notes (v0.13.0)**:
```
Human: "Add a section about deployment to my API documentation"
You: I'll add that section to your existing documentation.
[Use edit_note() with operation="append" to add new content]
```
5. **Project management (v0.13.0)**:
```
Human: "Switch to my work project and show recent activity"
You: I'll switch to your work project and check what's been updated recently.
[Use switch_project() then recent_activity()]
```
6. **File organization (v0.13.0)**:
```
Human: "Move my old meeting notes to the archive folder"
You: I'll organize those notes for you.
[Use move_note() to relocate files with database consistency]
```
## Key Things to Remember
1. **Files are Truth**
- All knowledge lives in local files on the user's computer
- Users can edit files outside your interaction
- Changes need to be synced by the user (usually automatic)
- Always verify information is current with `recent_activity()`
2. **Building Context Effectively**
- Start with specific entities
- Follow meaningful relations
- Check recent changes
- Build context incrementally
- Combine related information
3. **Writing Knowledge Wisely**
- Same title+folder overwrites existing notes
- Structure with clear headings and semantic markup
- Use tags for searchability (v0.13.0: frontmatter tags indexed)
- Keep files organized in logical folders
4. **Leverage v0.13.0 Features**
- **Edit incrementally**: Use `edit_note()` for small changes vs rewriting
- **Switch projects**: Change context when user mentions different work areas
- **Organize proactively**: Move old content to archive folders
- **Cross-project operations**: Create notes in specific projects while maintaining context
## Common Knowledge Patterns
### Capturing Decisions
```markdown
---
title: Coffee Brewing Methods
tags: [coffee, brewing, pour-over, techniques] # v0.13.0: Now searchable!
---
# Coffee Brewing Methods
## Context
I've experimented with various brewing methods including French press, pour over, and espresso.
## Decision
Pour over is my preferred method for light to medium roasts because it highlights subtle flavors and offers more control over the extraction.
## Observations
- [technique] Blooming the coffee grounds for 30 seconds improves extraction #brewing
- [preference] Water temperature between 195-205°F works best #temperature
- [equipment] Gooseneck kettle provides better control of water flow #tools
- [timing] Total brew time of 3-4 minutes produces optimal extraction #process
## Relations
- pairs_with [[Light Roast Beans]]
- contrasts_with [[French Press Method]]
- requires [[Proper Grinding Technique]]
- part_of [[Morning Coffee Routine]]
```
### Recording Project Structure
```markdown
# Garden Planning
## Overview
This document outlines the garden layout and planting strategy for this season.
## Observations
- [structure] Raised beds in south corner for sun exposure #layout
- [structure] Drip irrigation system installed for efficiency #watering
- [pattern] Companion planting used to deter pests naturally #technique
## Relations
- contains [[Vegetable Section]]
- contains [[Herb Garden]]
- implements [[Organic Gardening Principles]]
```
### Technical Discussions
```markdown
# Recipe Improvement Discussion
## Key Points
Discussed strategies for improving the chocolate chip cookie recipe.
## Observations
- [issue] Cookies spread too thin when baked at 350°F #texture
- [solution] Chilling dough for 24 hours improves flavor and reduces spreading #technique
- [decision] Will use brown butter instead of regular butter #flavor
## Relations
- improves [[Basic Cookie Recipe]]
- inspired_by [[Bakery-Style Cookies]]
- pairs_with [[Homemade Ice Cream]]
```
## v0.13.0 Workflow Examples
### Multi-Project Conversations
**User:** "I need to update my work documentation and also add a personal recipe note."
**Workflow:**
1. `list_projects()` - Check available projects
2. `write_note(title="Sprint Planning", project="work-notes")` - Work content
3. `write_note(title="Weekend Recipes", project="personal")` - Personal content
### Incremental Note Building
**User:** "Add a troubleshooting section to my setup guide."
**Workflow:**
1. `edit_note(identifier="Setup Guide", operation="append", content="\n## Troubleshooting\n...")`
**User:** "Update the authentication section in my API docs."
**Workflow:**
1. `edit_note(identifier="API Documentation", operation="replace_section", section="## Authentication")`
### Smart File Organization
**User:** "My notes are getting messy in the main folder."
**Workflow:**
1. `move_note("Old Meeting Notes", "archive/2024/old-meetings.md")`
2. `move_note("Project Notes", "projects/client-work/notes.md")`
### Creating Effective Relations
When creating relations:
1. **Reference existing entities** by their exact title: `[[Exact Title]]`
2. **Create forward references** to entities that don't exist yet - they'll be linked automatically when created
3. **Search first** to find existing entities to reference
4. **Use meaningful relation types**: `implements`, `requires`, `part_of` vs generic `relates_to`
**Example workflow:**
1. `search_notes("travel")` to find existing travel-related notes
2. Reference found entities: `- part_of [[Japan Travel Guide]]`
3. Add forward references: `- located_in [[Tokyo]]` (even if Tokyo note doesn't exist yet)
## Common Issues & Solutions
**Missing Content:**
- Try `search_notes()` with broader terms if `read_note()` fails
- Use fuzzy matching: search for partial titles
**Forward References:**
- These are normal! Basic Memory links them automatically when target notes are created
- Inform users: "I've created forward references that will be linked when you create those notes"
**Sync Issues:**
- If information seems outdated, suggest `basic-memory sync`
- Use `recent_activity()` to check if content is current
**Strict Mode for Edit/Move Operations:**
- `edit_note()` and `move_note()` require **exact identifiers** (no fuzzy matching for safety)
- If identifier not found: use `search_notes()` first to find the exact title/permalink
- Error messages will guide you to find correct identifiers
- Example workflow:
```
# ❌ This might fail if identifier isn't exact
edit_note("Meeting Note", "append", "content")
# ✅ Safe approach: search first, then use exact result
results = search_notes("meeting")
edit_note("Meeting Notes 2024", "append", "content") # Use exact title from search
```
## Best Practices
1. **Proactively Record Context**
- Offer to capture important discussions
- Record decisions, rationales, and conclusions
- Link to related topics
- Ask for permission first: "Would you like me to save our discussion about [topic]?"
- Confirm when complete: "I've saved our discussion to Basic Memory"
2. **Create a Rich Semantic Graph**
- **Add meaningful observations**: Include at least 3-5 categorized observations in each note
- **Create deliberate relations**: Connect each note to at least 2-3 related entities
- **Use existing entities**: Before creating a new relation, search for existing entities
- **Verify wikilinks**: When referencing `[[Entity]]`, use exact titles of existing notes
- **Check accuracy**: Use `search_notes()` or `recent_activity()` to confirm entity titles
- **Use precise relation types**: Choose specific relation types that convey meaning (e.g., "implements" instead of "relates_to")
- **Consider bidirectional relations**: When appropriate, create inverse relations in both entities
3. **Structure Content Thoughtfully**
- Use clear, descriptive titles
- Organize with logical sections (Context, Decision, Implementation, etc.)
- Include relevant context and background
- Add semantic observations with appropriate categories
- Use a consistent format for similar types of notes
- Balance detail with conciseness
4. **Navigate Knowledge Effectively**
- Start with specific searches
- Follow relation paths
- Combine information from multiple sources
- Verify information is current
- Build a complete picture before responding
5. **Help Users Maintain Their Knowledge**
- Suggest organizing related topics
- Identify potential duplicates
- Recommend adding relations between topics
- Offer to create summaries of scattered information
- Suggest potential missing relations: "I notice this might relate to [topic], would you like me to add that connection?"
Built with ♥️ by Basic Machines
-442
View File
@@ -1,442 +0,0 @@
# Basic Memory Architecture
This document describes the architectural patterns and composition structure of Basic Memory.
## Overview
Basic Memory is a local-first knowledge management system with three entrypoints:
- **API** - FastAPI REST server for HTTP access
- **MCP** - Model Context Protocol server for LLM integration
- **CLI** - Typer command-line interface
Each entrypoint uses a **composition root** pattern to manage configuration and dependencies.
## Composition Roots
### What is a Composition Root?
A composition root is the single place in an application where dependencies are wired together. In Basic Memory, each entrypoint has its own composition root that:
1. Reads configuration from `ConfigManager`
2. Resolves runtime mode (local/test)
3. Creates and provides dependencies to downstream code
**Key principle**: Only composition roots read global configuration. All other modules receive configuration explicitly.
### Container Structure
Each entrypoint has a container dataclass in its package:
```
src/basic_memory/
├── api/
│ └── container.py # ApiContainer
├── mcp/
│ └── container.py # McpContainer
├── cli/
│ └── container.py # CliContainer
└── runtime.py # RuntimeMode enum and resolver
```
### Container Pattern
All containers follow the same structure:
```python
@dataclass
class Container:
config: BasicMemoryConfig
mode: RuntimeMode
@classmethod
def create(cls) -> "Container":
"""Create container by reading ConfigManager."""
config = ConfigManager().config
mode = resolve_runtime_mode(is_test_env=config.is_test_env)
return cls(config=config, mode=mode)
@property
def some_computed_property(self) -> bool:
"""Derived values based on config and mode."""
return self.mode.is_local and self.config.some_setting
# Module-level singleton
_container: Container | None = None
def get_container() -> Container:
if _container is None:
raise RuntimeError("Container not initialized")
return _container
def set_container(container: Container) -> None:
global _container
_container = container
```
### Runtime Mode Resolution
The `RuntimeMode` enum centralizes mode detection:
```python
class RuntimeMode(Enum):
LOCAL = "local"
CLOUD = "cloud"
TEST = "test"
@property
def is_cloud(self) -> bool:
return self == RuntimeMode.CLOUD
@property
def is_local(self) -> bool:
return self == RuntimeMode.LOCAL
@property
def is_test(self) -> bool:
return self == RuntimeMode.TEST
```
Resolution follows this precedence in local app flows: **TEST > LOCAL**
```python
def resolve_runtime_mode(is_test_env: bool) -> RuntimeMode:
if is_test_env:
return RuntimeMode.TEST
return RuntimeMode.LOCAL
```
**Note**: `RuntimeMode` determines global behavior (e.g., whether to start file sync).
Per-project routing is orthogonal: individual projects can be set to `cloud` mode via `ProjectMode`,
which affects client routing in `get_client(project_name=...)` without changing global runtime mode.
`RuntimeMode.CLOUD` may remain for compatibility, but standard local runtime resolution does not select it.
## Dependencies Package
### Structure
The `deps/` package provides FastAPI dependencies organized by feature:
```
src/basic_memory/deps/
├── __init__.py # Re-exports for backwards compatibility
├── config.py # Configuration access
├── db.py # Database/session management
├── projects.py # Project resolution
├── repositories.py # Data access layer
├── services.py # Business logic layer
└── importers.py # Import functionality
```
### Usage in Routers
```python
from basic_memory.deps.services import get_entity_service
from basic_memory.deps.projects import get_project_config
@router.get("/entities/{id}")
async def get_entity(
id: int,
entity_service: EntityService = Depends(get_entity_service),
project: ProjectConfig = Depends(get_project_config),
):
return await entity_service.get(id)
```
### Backwards Compatibility
The old `deps.py` file still exists as a thin re-export shim:
```python
# deps.py - backwards compatibility shim
from basic_memory.deps import *
```
New code should import from specific submodules (`basic_memory.deps.services`) for clarity.
## MCP Tools Architecture
### Typed API Clients
MCP tools communicate with the API through typed clients that encapsulate HTTP paths and response validation:
```
src/basic_memory/mcp/clients/
├── __init__.py # Re-exports all clients
├── base.py # BaseClient with common logic
├── knowledge.py # KnowledgeClient - entity CRUD
├── search.py # SearchClient - search operations
├── memory.py # MemoryClient - context building
├── directory.py # DirectoryClient - directory listing
├── resource.py # ResourceClient - resource reading
└── project.py # ProjectClient - project management
```
### Client Pattern
Each client encapsulates API paths and validates responses:
```python
class KnowledgeClient(BaseClient):
"""Client for knowledge/entity operations."""
async def resolve_entity(self, identifier: str) -> int:
"""Resolve identifier to entity ID."""
response = await call_get(
self.http_client,
f"{self._base_path}/resolve/{identifier}",
)
return int(response.text)
async def get_entity(self, entity_id: int) -> EntityResponse:
"""Get entity by ID."""
response = await call_get(
self.http_client,
f"{self._base_path}/entities/{entity_id}",
)
return EntityResponse.model_validate(response.json())
```
### Tool → Client → API Flow
```
MCP Tool (thin adapter)
Typed Client (encapsulates paths, validates responses)
HTTP API (FastAPI router)
Service Layer (business logic)
Repository Layer (data access)
```
Example tool using typed client:
```python
@mcp.tool()
async def search_notes(
query: str,
project: str | None = None,
metadata_filters: dict | None = None,
tags: list[str] | None = None,
status: str | None = None,
) -> SearchResponse:
async with get_project_client(project, context) as (client, active_project):
# Import client inside function to avoid circular imports
from basic_memory.mcp.clients import SearchClient
from basic_memory.schemas.search import SearchQuery
search_query = SearchQuery(
text=query,
metadata_filters=metadata_filters,
tags=tags,
status=status,
)
search_client = SearchClient(client, active_project.external_id)
return await search_client.search(search_query.model_dump())
```
### Per-Project Client Routing
`get_project_client()` from `mcp/project_context.py` is an async context manager that:
1. Resolves the project name from config (no network call)
2. Creates the correctly-routed client based on the project's mode (local ASGI or cloud HTTP with API key)
3. Validates the project via the API
4. Yields `(client, active_project)` tuple
This solves the bootstrap problem: you need the project name to choose the right client (local vs cloud), but you need the client to validate the project exists.
```python
from basic_memory.mcp.project_context import get_project_client
async with get_project_client(project, context) as (client, active_project):
# client is routed based on project's mode (local or cloud)
# active_project is validated via the API
...
```
## Sync Coordination
### SyncCoordinator
The `SyncCoordinator` centralizes sync/watch lifecycle management:
```python
@dataclass
class SyncCoordinator:
"""Coordinates file sync and watch operations."""
status: SyncStatus = SyncStatus.NOT_STARTED
sync_task: asyncio.Task | None = None
watch_service: WatchService | None = None
async def start(self, ...):
"""Start sync and watch operations."""
async def stop(self):
"""Stop all sync operations gracefully."""
def get_status_info(self) -> dict:
"""Get current sync status for observability."""
```
### Status Enum
```python
class SyncStatus(Enum):
NOT_STARTED = "not_started"
STARTING = "starting"
RUNNING = "running"
STOPPING = "stopping"
STOPPED = "stopped"
ERROR = "error"
```
## Project Resolution
### ProjectResolver
Unified project selection across all entrypoints:
```python
class ProjectResolver:
"""Resolves which project to use based on context."""
def resolve(
self,
explicit_project: str | None = None,
) -> ResolvedProject:
"""Resolve project using three-tier hierarchy:
1. Explicit project parameter
2. Default project from config
3. Single available project
"""
```
### Resolution Modes
```python
class ResolutionMode(Enum):
EXPLICIT = "explicit" # User specified project
DEFAULT = "default" # Using configured default
SINGLE_PROJECT = "single" # Only one project exists
FALLBACK = "fallback" # Using first available
```
## Testing Patterns
### Container Testing
Each container has corresponding tests:
```
tests/
├── api/test_api_container.py
├── mcp/test_mcp_container.py
└── cli/test_cli_container.py
```
Tests verify:
- Container creation from config
- Runtime mode properties
- Container accessor functions (get/set)
### Mocking Typed Clients
When testing MCP tools, mock at the client level:
```python
def test_search_notes(monkeypatch):
import basic_memory.mcp.clients as clients_mod
class MockSearchClient:
async def search(self, query):
return SearchResponse(results=[...])
monkeypatch.setattr(clients_mod, "SearchClient", MockSearchClient)
```
## Design Principles
### 1. Explicit Dependencies
Modules receive configuration explicitly rather than reading globals:
```python
# Good - explicit injection
async def sync_files(config: BasicMemoryConfig):
...
# Avoid - hidden global access
async def sync_files():
config = ConfigManager().config # Hidden coupling
```
### 2. Single Responsibility
Each layer has a clear responsibility:
- **Containers**: Wire dependencies
- **Clients**: Encapsulate HTTP communication
- **Services**: Business logic
- **Repositories**: Data access
- **Tools/Routers**: Thin adapters
### 3. Deferred Imports
To avoid circular imports, typed clients are imported inside functions:
```python
async def my_tool():
async with get_client() as client:
# Import here to avoid circular dependency
from basic_memory.mcp.clients import KnowledgeClient
knowledge_client = KnowledgeClient(client, project_id)
```
### 4. Backwards Compatibility
When refactoring, maintain backwards compatibility via shims:
```python
# Old module becomes a shim
from basic_memory.new_location import *
# Docstring explains migration path
"""
DEPRECATED: Import from basic_memory.new_location instead.
This shim will be removed in a future version.
"""
```
## File Organization
```
src/basic_memory/
├── api/
│ ├── container.py # API composition root
│ ├── routers/ # FastAPI routers
│ └── ...
├── mcp/
│ ├── container.py # MCP composition root
│ ├── clients/ # Typed API clients
│ ├── tools/ # MCP tool definitions
│ └── server.py # MCP server setup
├── cli/
│ ├── container.py # CLI composition root
│ ├── app.py # Typer app
│ └── commands/ # CLI command groups
├── deps/
│ ├── config.py # Config dependencies
│ ├── db.py # Database dependencies
│ ├── projects.py # Project dependencies
│ ├── repositories.py # Repository dependencies
│ ├── services.py # Service dependencies
│ └── importers.py # Importer dependencies
├── sync/
│ ├── coordinator.py # SyncCoordinator
│ └── ...
├── runtime.py # RuntimeMode resolution
├── project_resolver.py # Unified project selection
└── config.py # Configuration management
```
+16 -47
View File
@@ -15,7 +15,7 @@ Basic Memory provides pre-built Docker images on GitHub Container Registry that
--name basic-memory-server \
-p 8000:8000 \
-v /path/to/your/obsidian-vault:/app/data:rw \
-v basic-memory-config:/app/.basic-memory:rw \
-v basic-memory-config:/root/.basic-memory:rw \
ghcr.io/basicmachines-co/basic-memory:latest
```
@@ -30,7 +30,7 @@ Basic Memory provides pre-built Docker images on GitHub Container Registry that
- "8000:8000"
volumes:
- /path/to/your/obsidian-vault:/app/data:rw
- basic-memory-config:/app/.basic-memory:rw
- basic-memory-config:/root/.basic-memory:rw
environment:
- BASIC_MEMORY_DEFAULT_PROJECT=main
restart: unless-stopped
@@ -67,7 +67,7 @@ docker build -t basic-memory .
docker run -d \
--name basic-memory-server \
-v /path/to/your/obsidian-vault:/app/data:rw \
-v basic-memory-config:/app/.basic-memory:rw \
-v basic-memory-config:/root/.basic-memory:rw \
-e BASIC_MEMORY_DEFAULT_PROJECT=main \
basic-memory
```
@@ -86,11 +86,11 @@ Basic Memory requires several volume mounts for proper operation:
2. **Configuration and Database** (Recommended):
```yaml
- basic-memory-config:/app/.basic-memory:rw
- basic-memory-config:/root/.basic-memory:rw
```
Persistent storage for configuration and SQLite database.
You can edit the basic-memory config.json file located in the /app/.basic-memory/config.json after Basic Memory starts.
You can edit the basic-memory config.json file located in the /root/.basic-memory/config.json after Basic Memory starts.
3. **Multiple Projects** (Optional):
```yaml
@@ -98,7 +98,7 @@ You can edit the basic-memory config.json file located in the /app/.basic-memory
- /path/to/project2:/app/data/project2:rw
```
You can edit the basic-memory config.json file located in the /app/.basic-memory/config.json
You can edit the basic-memory config.json file located in the /root/.basic-memory/config.json
## CLI Commands via Docker
@@ -123,7 +123,7 @@ When using Docker volumes, you'll need to configure projects to point to your mo
1. **Check current configuration:**
```bash
docker exec basic-memory-server cat /app/.basic-memory/config.json
docker exec basic-memory-server cat /root/.basic-memory/config.json
```
2. **Add a project for your mounted volume:**
@@ -184,47 +184,16 @@ environment:
### Linux/macOS
The Docker container now runs as a non-root user to avoid file ownership issues. By default, the container uses UID/GID 1000, but you can customize this to match your user:
Ensure your knowledge directories have proper permissions:
```bash
# Build with custom UID/GID to match your user
docker build --build-arg UID=$(id -u) --build-arg GID=$(id -g) -t basic-memory .
# Make directories readable/writable
chmod -R 755 /path/to/your/obsidian-vault
# Or use docker-compose with build args
# If using specific user/group
chown -R $USER:$USER /path/to/your/obsidian-vault
```
**Example docker-compose.yml with custom user:**
```yaml
version: '3.8'
services:
basic-memory:
build:
context: .
dockerfile: Dockerfile
args:
UID: 1000 # Replace with your UID
GID: 1000 # Replace with your GID
container_name: basic-memory-server
ports:
- "8000:8000"
volumes:
- /path/to/your/obsidian-vault:/app/data:rw
- basic-memory-config:/app/.basic-memory:rw
environment:
- BASIC_MEMORY_DEFAULT_PROJECT=main
restart: unless-stopped
```
**Using pre-built images:**
If using the pre-built image from GitHub Container Registry, files will be created with UID/GID 1000. You can either:
1. Change your local directory ownership to match:
```bash
sudo chown -R 1000:1000 /path/to/your/obsidian-vault
```
2. Or build your own image with custom UID/GID as shown above.
### Windows
When using Docker Desktop on Windows, ensure the directories are shared:
@@ -248,7 +217,7 @@ When using Docker Desktop on Windows, ensure the directories are shared:
```
2. **Configuration Not Persisting:**
- Use named volumes for `/app/.basic-memory`
- Use named volumes for `/root/.basic-memory`
- Check volume mount permissions
3. **Network Connectivity:**
@@ -274,10 +243,10 @@ docker-compose logs -f basic-memory
## Security Considerations
1. **Docker Security:**
The container runs as a non-root user (UID/GID 1000 by default) for improved security. You can customize the user ID using build arguments to match your local user.
The container runs as root for simplicity. For production, consider additional security measures.
2. **Volume Permissions:**
Ensure mounted directories have appropriate permissions and don't expose sensitive data. With the non-root container, files will be created with the specified user ownership.
Ensure mounted directories have appropriate permissions and don't expose sensitive data.
3. **Network Security:**
If using HTTP transport, consider using reverse proxy with SSL/TLS and authentication if the endpoint is available on
@@ -319,7 +288,7 @@ For Docker-specific issues:
1. Check the [troubleshooting section](#troubleshooting) above
2. Review container logs: `docker-compose logs basic-memory`
3. Verify volume mounts: `docker inspect basic-memory-server`
4. Test file permissions: `docker exec basic-memory-server ls -la /app`
4. Test file permissions: `docker exec basic-memory-server ls -la /root`
For general Basic Memory support, see the main [README](../README.md)
and [documentation](https://memory.basicmachines.co/).
-494
View File
@@ -1,494 +0,0 @@
# Note Format Reference
Every document in Basic Memory is a plain Markdown file. Files are the source of truth — changes to files automatically update the knowledge graph in the database. You maintain complete ownership, files work with git, and knowledge persists independently of any AI conversation.
## Document Structure
A note has three parts: YAML frontmatter, content (observations), and relations.
```markdown
---
title: Coffee Brewing Methods
type: note
tags: [coffee, brewing]
permalink: coffee-brewing-methods
---
# Coffee Brewing Methods
## Observations
- [method] Pour over provides more flavor clarity than French press
- [technique] Water temperature at 205°F extracts optimal compounds #brewing
- [preference] Ethiopian beans work well with lighter roasts (personal experience)
## Relations
- relates_to [[Coffee Bean Origins]]
- requires [[Proper Grinding Technique]]
- contrasts_with [[Tea Brewing Methods]]
```
The `## Observations` and `## Relations` headings are conventional but not required — the parser detects observations and relations by their syntax patterns anywhere in the document.
## Frontmatter
YAML metadata between `---` fences at the top of the file.
| Field | Required | Default | Description |
|-------|----------|---------|-------------|
| `title` | No | filename stem | Used for linking and references. Auto-set from filename if missing. |
| `type` | No | `note` | Entity type. Used for schema resolution and filtering. |
| `tags` | No | `[]` | List or comma-separated string. Used for organization and search. |
| `permalink` | No | generated from title | Stable identifier. Persists even if the file moves. |
| `schema` | No | none | Schema attachment — dict (inline), string (reference), or omitted (implicit). |
Custom fields are allowed. Any key not in the standard set is stored as `entity_metadata` and indexed for search and filtering.
```yaml
---
title: Paul Graham
type: Person
tags: [startups, essays, lisp]
permalink: paul-graham
status: active
source: wikipedia
---
```
Here `status` and `source` are custom fields stored in `entity_metadata`.
### Frontmatter Value Handling
YAML automatically converts some values to native types. Basic Memory normalizes them:
- Date strings (`2025-10-24`) → kept as ISO format strings
- Numbers (`1.0`) → converted to strings
- Booleans (`true`) → converted to strings (`"True"`)
- Lists and dicts → preserved, items normalized recursively
This prevents errors when downstream code expects string values.
## Observations
An observation is a categorized fact about the entity. Written as a Markdown list item.
**Syntax:**
```
- [category] content text #tag1 #tag2 (context)
```
| Part | Required | Description |
|------|----------|-------------|
| `[category]` | Yes | Classification in square brackets. Any text except `[]()` chars. |
| content | Yes | The fact or statement. |
| `#tags` | No | Inline tags. Space-separated, each starting with `#`. |
| `(context)` | No | Parenthesized text at end of line. Supporting details or source. |
### Examples
```markdown
- [tech] Uses SQLite for storage #database
- [design] Follows local-first architecture #architecture
- [decision] Selected bcrypt for passwords #security (based on OWASP audit)
- [name] Paul Graham
- [expertise] Startups
- [expertise] Lisp
- [expertise] Essay writing
```
Array-like fields use repeated categories — multiple `[expertise]` observations above.
### What Is Not an Observation
The parser excludes these list item patterns:
| Pattern | Example | Reason |
|---------|---------|--------|
| Checkboxes | `- [ ] Todo item`, `- [x] Done`, `- [-] Cancelled` | Task list syntax |
| Markdown links | `- [text](url)` | URL link syntax |
| Bare wiki links | `- [[Target]]` | Treated as a relation instead |
A list item with `#tags` but no `[category]` is still parsed — the tags are extracted and the category defaults to `Note`.
## Relations
Relations connect documents to form the knowledge graph. There are two kinds.
### Explicit Relations
Written as list items with a relation type and a `[[wiki link]]` target.
**Syntax:**
```
- relation_type [[Target Entity]] (context)
```
| Part | Required | Description |
|------|----------|-------------|
| `relation_type` | No | Text before `[[`. Defaults to `relates_to` if omitted. |
| `[[Target]]` | Yes | Wiki link to the target entity. Matched by title or permalink. |
| `(context)` | No | Parenthesized text after `]]`. Supporting details. |
### Examples
```markdown
- implements [[Search Design]]
- depends_on [[Database Schema]]
- works_at [[Y Combinator]] (co-founder)
- [[Some Entity]]
```
The last example — a bare `[[wiki link]]` in a list item — gets relation type `relates_to`.
Common relation types:
- `implements`, `depends_on`, `relates_to`, `inspired_by`
- `extends`, `part_of`, `contains`, `pairs_with`
- `works_at`, `authored`, `collaborated_with`
Any text works as a relation type. These are conventions, not a fixed set.
### Inline References
Wiki links appearing in regular prose (not as list items) create implicit `links_to` relations.
```markdown
This builds on [[Core Design]] and uses [[Utility Functions]].
```
This creates two relations: `links_to [[Core Design]]` and `links_to [[Utility Functions]]`.
### Forward References
Relations can link to entities that don't exist yet. Basic Memory resolves them when the target is created.
## Permalinks and memory:// URLs
Every document has a unique **permalink** — a stable identifier derived from its title. You can set one explicitly in frontmatter, or let the system generate it.
```yaml
permalink: auth-approaches-2024
```
Permalinks form the basis of `memory://` URLs:
```
memory://auth-approaches-2024 # By permalink
memory://Authentication Approaches # By title (auto-resolves)
memory://project/auth-approaches # By path
```
Pattern matching is supported:
```
memory://auth* # Starts with "auth"
memory://*/approaches # Ends with "approaches"
memory://project/*/requirements # Nested wildcard
```
## Schemas
Schemas declare the expected structure of a note — which observation categories and relation types a well-formed note should have. They use Picoschema, a compact notation from Google's Dotprompt that fits naturally in YAML frontmatter.
### Picoschema Syntax
```yaml
schema:
name: string, full name # required field with description
email?: string, contact email # ? = optional
role?: string, job title
works_at?: Organization, employer # capitalized type = entity reference
tags?(array): string, categories # array of type
status?(enum): [active, inactive] # enum with allowed values
metadata?(object): # nested object
updated_at?: string
source?: string
```
| Notation | Meaning | Example |
|----------|---------|---------|
| `field: type` | Required field | `name: string` |
| `field?: type` | Optional field | `role?: string` |
| `field(array): type` | Array of values | `expertise(array): string` |
| `field?(enum): [vals]` | Enum with allowed values | `status?(enum): [active, inactive]` |
| `field?(object):` | Nested object with sub-fields | `metadata?(object):` |
| `, description` | Description after comma | `name: string, full name` |
| `EntityName` | Capitalized type = entity reference | `works_at?: Organization` |
**Scalar types:** `string`, `integer`, `number`, `boolean`, `any`
Any type not in that set whose first letter is uppercase is treated as an entity reference (a relation target).
### Schema-to-Note Mapping
Schemas validate against existing observation/relation syntax. Note authors don't learn new syntax.
| Schema Declaration | Maps To | Example in Note |
|--------------------|---------|-----------------|
| `field: string` | Observation `[field] value` | `- [name] Paul Graham` |
| `field?(array): string` | Multiple `[field]` observations | `- [expertise] Lisp` (repeated) |
| `field?: EntityType` | Relation `field [[Target]]` | `- works_at [[Y Combinator]]` |
| `field?(array): EntityType` | Multiple `field` relations | `- authored [[Book]]` (repeated) |
| `tags` | Frontmatter `tags` array | `tags: [startups, essays]` |
| `field?(enum): [vals]` | Observation `[field] value` where value is in the set | `- [status] active` |
Observations and relations not covered by the schema are valid — schemas describe a subset, not a straitjacket.
### Schema Attachment
Three ways to attach a schema to a note, resolved in priority order:
**1. Inline schema**`schema` is a dict in frontmatter:
```yaml
---
title: Team Standup 2024-01-15
type: meeting
schema:
attendees(array): string, who was there
decisions(array): string, what was decided
action_items(array): string, follow-ups
blockers?(array): string, anything stuck
---
```
Good for one-off structured notes or prototyping a schema before extracting it.
**2. Explicit reference**`schema` is a string naming a schema note:
```yaml
---
title: Basic Memory
schema: SoftwareProject
---
```
or by permalink:
```yaml
---
title: LLM Memory Patterns
schema: schema/research-project
---
```
Use when the note's `type` differs from the schema it should validate against, or when multiple schema variants exist.
**3. Implicit by type** — no `schema` field, resolved by matching `type`:
```yaml
---
title: Paul Graham
type: Person
---
```
The system looks up a schema note where `entity: Person`. If found, it applies. If not, no validation occurs.
**4. No schema** — perfectly fine. Most notes don't need one.
### Schema Notes
A schema is itself a Basic Memory note with `type: schema`. It lives anywhere (though `schema/` is the conventional directory).
```yaml
# schema/Person.md
---
title: Person
type: schema
entity: Person
version: 1
schema:
name: string, full name
role?: string, job title or position
works_at?: Organization, employer
expertise?(array): string, areas of knowledge
email?: string, contact email
settings:
validation: warn
---
# Person
A human individual in the knowledge graph.
```
| Field | Required | Description |
|-------|----------|-------------|
| `type` | Yes | Must be `schema` |
| `entity` | Yes | The entity type this schema describes (e.g., `Person`) |
| `version` | No | Schema version number (default: `1`) |
| `schema` | Yes | Picoschema dict defining the fields |
| `settings.validation` | No | Validation mode (default: `warn`) |
Schema notes are regular notes — they show up in search, can have observations and relations, and participate in the knowledge graph.
### Validation Modes
| Mode | Behavior |
|------|----------|
| `warn` | Warnings in output, doesn't block (default) |
| `strict` | Errors that block sync, for CI/CD enforcement |
| `off` | No validation |
### Validation Output
```
$ bm schema validate people/ada-lovelace.md
⚠ Person schema validation:
- Missing required field: name (expected [name] observation)
- Missing optional field: role
- Missing optional field: works_at (no relation found)
Unmatched observations: [fact] ×2, [born] ×1
Unmatched relations: collaborated_with
```
"Unmatched" items are informational — observations and relations the schema doesn't cover.
### Schema Inference
Generate schemas from existing notes by analyzing observation and relation frequency:
```
$ bm schema infer Person
Analyzing 30 notes with type: Person...
Observations found:
[name] 30/30 100% → name: string
[role] 27/30 90% → role?: string
[expertise] 18/30 60% → expertise?(array): string
[email] 8/30 27% → email?: string
Relations found:
works_at 22/30 73% → works_at?: Organization
Suggested schema:
name: string, full name
role?: string, job title
expertise?(array): string, areas of knowledge
email?: string, contact email
works_at?: Organization, employer
Save to schema/Person.md? [y/n]
```
Frequency thresholds:
- **100% present** → required field
- **25%+ present** → optional field
- **Below 25%** → excluded from suggestion
### Schema Drift Detection
Track how usage patterns shift over time:
```
$ bm schema diff Person
Schema drift detected:
+ expertise: now in 81% of notes (was 12%)
- department: dropped to 3% of notes
~ works_at: cardinality changed (one → many)
Update schema? [y/n/review]
```
## Complete Examples
### Simple Note (No Schema)
```markdown
---
title: Project Ideas
type: note
tags: [ideas, brainstorm]
---
# Project Ideas
## Observations
- [idea] Build a CLI tool for markdown linting #tooling
- [idea] Create a recipe knowledge base #cooking
- [priority] Focus on developer tools first (Q1 goal)
## Relations
- inspired_by [[Developer Workflow Research]]
- part_of [[Q1 Planning]]
```
### Schema-Validated Note
Schema at `schema/Person.md`:
```yaml
---
title: Person
type: schema
entity: Person
version: 1
schema:
name: string, full name
role?: string, job title or position
works_at?: Organization, employer
expertise?(array): string, areas of knowledge
email?: string, contact email
settings:
validation: warn
---
# Person
A human individual in the knowledge graph.
```
Note at `people/paul-graham.md`:
```markdown
---
title: Paul Graham
type: Person
tags: [startups, essays, lisp]
---
# Paul Graham
## Observations
- [name] Paul Graham
- [role] Essayist and investor
- [expertise] Startups
- [expertise] Lisp
- [expertise] Essay writing
- [fact] Created Viaweb, the first web app
## Relations
- works_at [[Y Combinator]]
- authored [[Hackers and Painters]]
```
The `[fact]` observation and `authored` relation are not in the schema — they're valid, just unmatched. The schema only checks that `[name]` exists (required) and looks for optional fields like `[role]`, `[expertise]`, and `works_at`.
### Inline Schema Note
```markdown
---
title: Team Standup 2024-01-15
type: meeting
schema:
attendees(array): string, who was there
decisions(array): string, what was decided
action_items(array): string, follow-ups
blockers?(array): string, anything stuck
---
# Team Standup 2024-01-15
## Observations
- [attendees] Paul
- [attendees] Sarah
- [decisions] Ship v2 by Friday
- [action_items] Paul to review PR #42
- [blockers] Waiting on API credentials
```
-135
View File
@@ -1,135 +0,0 @@
# Simplified Local/Cloud Routing
## Context
Basic Memory now uses explicit, project-aware routing without a global cloud-mode toggle.
Routing is determined by command-level flags and project mode, not by a global `cloud_mode` state.
This document is the canonical contract for local/cloud routing behavior in CLI, MCP, and API-adjacent clients.
## Goals
1. Remove global `cloud_mode` from runtime/routing semantics.
2. Keep MCP stdio local-only and predictable.
3. Make CLI routing explicit and easy to reason about.
4. Support projects that exist in both local and cloud without ambiguity.
## Routing Contract
Routing is resolved in this order:
1. Injected client factory (for composition/integration contexts)
2. Explicit routing override (`--local` / `--cloud` or env vars below)
3. Project-scoped routing (`project.mode`) when a project is known
4. Default local routing
### Routing Environment Variables
- `BASIC_MEMORY_FORCE_LOCAL=true`: force local transport
- `BASIC_MEMORY_FORCE_CLOUD=true`: force cloud proxy transport
- `BASIC_MEMORY_EXPLICIT_ROUTING=true`: marks routing as explicitly chosen for this command
When explicit routing is active, project mode does not override the selected route.
## Config Semantics
- `project.mode` is the only config-based routing signal for project-scoped operations.
- Legacy `cloud_mode` values may be encountered during migration/loading but are not used for routing behavior.
- Normalization saves remove stale `cloud_mode` from `~/.basic-memory/config.json`.
### Example Config
```json
{
"projects": {
"main": {
"path": "/Users/me/basic-memory",
"mode": "local",
"cloud_sync_path": null,
"bisync_initialized": false,
"last_sync": null
},
"specs": {
"path": "specs",
"mode": "cloud",
"cloud_sync_path": "/Users/me/dev/specs",
"bisync_initialized": true,
"last_sync": "2026-02-06T17:36:38.544153"
}
},
"default_project": "main",
"cloud_api_key": "bmc_abc123...",
"cloud_host": "https://cloud.basicmemory.com"
}
```
## Cloud Commands Are Auth-Only
`bm cloud login`, `bm cloud logout`, and `bm cloud status` manage authentication state.
- `bm cloud login`
- performs OAuth device flow
- stores/refreshes token material
- may verify cloud health/subscription
- does not change routing defaults
- `bm cloud logout`
- removes stored OAuth session tokens
- does not change routing defaults
- `bm cloud status`
- reports auth state (API key, OAuth token validity)
- runs health checks only when credentials are available
## MCP Stdio Local Guarantee
`bm mcp --transport stdio` always routes locally.
The command sets explicit local routing (`BASIC_MEMORY_FORCE_LOCAL=true` and
`BASIC_MEMORY_EXPLICIT_ROUTING=true`) before starting the server. This prevents cloud routing for stdio MCP,
even if the selected project has `mode: cloud`.
## Project List UX for Dual Presence
Projects may exist in both local and cloud. `bm project list` should display that clearly in one row per logical
project identity, with explicit source/target signals.
Recommended display contract:
1. Keep one row per normalized project name/permalink.
2. Show both local and cloud presence as separate columns/indicators.
3. Show an explicit `MCP (stdio)` target column that always resolves to `local`.
4. Keep CLI route semantics explicit:
- no flags: default local for non-project commands
- `--cloud`: force cloud
- `--local`: force local
## Project LS Targeting
`bm project ls` should clearly identify which project instance is being listed.
Targeting rules:
1. No routing flags: list local project files.
2. `--cloud`: list cloud project files.
3. `--local`: list local project files (explicit override).
4. Output should label the active target (`LOCAL` or `CLOUD`) in heading or status line.
## Runtime Mode
Runtime mode is no longer a cloud/local routing switch for local app flows.
- `resolve_runtime_mode(is_test_env)` resolves to:
- `TEST` when running in test environment
- `LOCAL` otherwise
- `RuntimeMode.CLOUD` may remain for compatibility with existing tests/call sites but is not selected by normal local
runtime resolution.
## Verification Checklist
1. Loading config with legacy `cloud_mode` succeeds.
2. Saving config strips legacy `cloud_mode`.
3. `--local/--cloud` always override per-project mode for that command.
4. No-project + no-flags commands route local by default.
5. `bm cloud login/logout` do not toggle routing behavior.
6. `bm mcp` remains local-only in stdio mode.
7. `bm project list` communicates dual local/cloud presence without ambiguity.
8. `bm project ls` output identifies route target explicitly.
File diff suppressed because it is too large Load Diff
-241
View File
@@ -1,241 +0,0 @@
# Character Handling and Conflict Resolution
Basic Memory handles various character encoding scenarios and file naming conventions to provide consistent permalink generation and conflict resolution. This document explains how the system works and how to resolve common character-related issues.
## Overview
Basic Memory uses a sophisticated system to generate permalinks from file paths while maintaining consistency across different operating systems and character encodings. The system normalizes file paths and generates unique permalinks to prevent conflicts.
## Character Normalization Rules
### 1. Permalink Generation
When Basic Memory processes a file path, it applies these normalization rules:
```
Original: "Finance/My Investment Strategy.md"
Permalink: "finance/my-investment-strategy"
```
**Transformation process:**
1. Remove file extension (`.md`)
2. Convert to lowercase (case-insensitive)
3. Replace spaces with hyphens
4. Replace underscores with hyphens
5. Handle international characters (transliteration for Latin, preservation for non-Latin)
6. Convert camelCase to kebab-case
### 2. International Character Support
**Latin characters with diacritics** are transliterated:
- `ø``o` (Søren → soren)
- `ü``u` (Müller → muller)
- `é``e` (Café → cafe)
- `ñ``n` (Niño → nino)
**Non-Latin characters** are preserved:
- Chinese: `中文/测试文档.md``中文/测试文档`
- Japanese: `日本語/文書.md``日本語/文書`
## Common Conflict Scenarios
### 1. Hyphen vs Space Conflicts
**Problem:** Files with existing hyphens conflict with generated permalinks from spaces.
**Example:**
```
File 1: "basic memory bug.md" → permalink: "basic-memory-bug"
File 2: "basic-memory-bug.md" → permalink: "basic-memory-bug" (CONFLICT!)
```
**Resolution:** The system automatically resolves this by adding suffixes:
```
File 1: "basic memory bug.md" → permalink: "basic-memory-bug"
File 2: "basic-memory-bug.md" → permalink: "basic-memory-bug-1"
```
**Best Practice:** Choose consistent naming conventions within your project.
### 2. Case Sensitivity Conflicts
**Problem:** Different case variations that normalize to the same permalink.
**Example on macOS:**
```
Directory: Finance/investment.md
Directory: finance/investment.md (different on filesystem, same permalink)
```
**Resolution:** Basic Memory detects case conflicts and prevents them during sync operations with helpful error messages.
**Best Practice:** Use consistent casing for directory and file names.
### 3. Character Encoding Conflicts
**Problem:** Different Unicode normalizations of the same logical character.
**Example:**
```
File 1: "café.md" (é as single character)
File 2: "café.md" (e + combining accent)
```
**Resolution:** Basic Memory normalizes Unicode characters using NFD normalization to detect these conflicts.
### 4. Forward Slash Conflicts
**Problem:** Forward slashes in frontmatter or file names interpreted as path separators.
**Example:**
```yaml
---
permalink: finance/investment/strategy
---
```
**Resolution:** Basic Memory validates frontmatter permalinks and warns about path separator conflicts.
## Error Messages and Troubleshooting
### "UNIQUE constraint failed: entity.file_path, entity.project_id"
**Cause:** Two entities trying to use the same file path within a project.
**Common scenarios:**
1. File move operation where destination is already occupied
2. Case sensitivity differences on macOS
3. Character encoding conflicts
4. Concurrent file operations
**Resolution steps:**
1. Check for duplicate file names with different cases
2. Look for files with similar names but different character encodings
3. Rename conflicting files to have unique names
4. Run sync again after resolving conflicts
### "File path conflict detected during move"
**Cause:** Enhanced conflict detection preventing potential database integrity violations.
**What this means:** The system detected that moving a file would create a conflict before attempting the database operation.
**Resolution:** Follow the specific guidance in the error message, which will indicate the type of conflict detected.
## Best Practices
### 1. File Naming Conventions
**Recommended patterns:**
- Use consistent casing (prefer lowercase)
- Use hyphens instead of spaces for multi-word files
- Avoid special characters that could conflict with path separators
- Be consistent with directory structure casing
**Examples:**
```
✅ Good:
- finance/investment-strategy.md
- projects/basic-memory-features.md
- docs/api-reference.md
❌ Problematic:
- Finance/Investment Strategy.md (mixed case, spaces)
- finance/Investment Strategy.md (inconsistent case)
- docs/API/Reference.md (mixed case directories)
```
### 2. Permalink Management
**Custom permalinks in frontmatter:**
```yaml
---
type: knowledge
permalink: custom-permalink-name
---
```
**Guidelines:**
- Use lowercase permalinks
- Use hyphens for word separation
- Avoid path separators unless creating sub-paths
- Ensure uniqueness within your project
### 3. Directory Structure
**Consistent casing:**
```
✅ Good:
finance/
investment-strategies.md
portfolio-management.md
❌ Problematic:
Finance/ (capital F)
investment-strategies.md
finance/ (lowercase f)
portfolio-management.md
```
## Migration and Cleanup
### Identifying Conflicts
Use Basic Memory's built-in conflict detection:
```bash
# Sync will report conflicts
basic-memory sync
# Check sync status for warnings
basic-memory status
```
### Resolving Existing Conflicts
1. **Identify conflicting files** from sync error messages
2. **Choose consistent naming convention** for your project
3. **Rename files** to follow the convention
4. **Re-run sync** to verify resolution
### Bulk Renaming Strategy
For projects with many conflicts:
1. **Backup your project** before making changes
2. **Standardize on lowercase** file and directory names
3. **Replace spaces with hyphens** in file names
4. **Use consistent character encoding** (UTF-8)
5. **Test sync after each batch** of changes
## System Enhancements
### Recent Improvements (v0.13+)
1. **Enhanced conflict detection** before database operations
2. **Improved error messages** with specific resolution guidance
3. **Character normalization utilities** for consistent handling
4. **File swap detection** for complex move scenarios
5. **Proactive conflict warnings** during permalink resolution
### Monitoring and Logging
The system now provides detailed logging for conflict resolution:
```
DEBUG: Detected potential file path conflicts for 'Finance/Investment.md': ['finance/investment.md']
WARNING: File path conflict detected during move: entity_id=123 trying to move from 'old.md' to 'new.md'
```
These logs help identify and resolve conflicts before they cause sync failures.
## Support and Resources
If you encounter character-related conflicts not covered in this guide:
1. **Check the logs** for specific conflict details
2. **Review error messages** for resolution guidance
3. **Report issues** with examples of the conflicting files
4. **Consider the file naming best practices** outlined above
The Basic Memory system is designed to handle most character conflicts automatically while providing clear guidance for manual resolution when needed.
-855
View File
@@ -1,855 +0,0 @@
# Basic Memory Cloud CLI Guide
The Basic Memory Cloud CLI provides seamless integration between local and cloud knowledge bases using **project-scoped synchronization**. Each project can optionally sync with the cloud, giving you fine-grained control over what syncs and where.
## Overview
The cloud CLI enables you to:
- **Authenticate cloud access** - OAuth/API key credentials are stored locally for cloud operations
- **Project-scoped sync** - Each project independently manages its sync configuration
- **Explicit operations** - Sync only what you want, when you want
- **Bidirectional sync** - Keep local and cloud in sync with rclone bisync
- **Offline access** - Work locally, sync when ready
## Prerequisites
Before using Basic Memory Cloud, you need:
- **Active Subscription**: An active Basic Memory Cloud subscription is required to access cloud features
- **Subscribe**: Visit [https://basicmemory.com/subscribe](https://basicmemory.com/subscribe) to sign up
- **Optional**: Cloud is optional. Local-first open-source usage continues without cloud.
- **OSS Discount**: Use code `{{OSS_DISCOUNT_CODE}}` for 20% off for 3 months.
If you attempt to log in without an active subscription, you'll receive a "Subscription Required" error with a link to subscribe.
## Architecture: Project-Scoped Sync
### The Problem
**Old approach (SPEC-8):** All projects lived in a single `~/basic-memory-cloud-sync/` directory. This caused:
- ❌ Directory conflicts between mount and bisync
- ❌ Auto-discovery creating phantom projects
- ❌ Confusion about what syncs and when
- ❌ All-or-nothing sync (couldn't sync just one project)
**New approach (SPEC-20):** Each project independently configures sync.
### How It Works
**Projects can exist in three states:**
1. **Cloud-only** - Project exists on cloud, no local copy
2. **Cloud + Local (synced)** - Project has a local working directory that syncs
3. **Local-only** - Project exists locally and is not routed to cloud
**Example:**
```bash
# You have 3 projects on cloud:
# - research: wants local sync at ~/Documents/research
# - work: wants local sync at ~/work-notes
# - temp: cloud-only, no local sync needed
bm project add research --cloud --local-path ~/Documents/research
bm project add work --cloud --local-path ~/work-notes
bm project add temp --cloud # No local sync
# Now you can sync individually (after initial --resync):
bm project bisync --name research
bm project bisync --name work
# temp stays cloud-only
```
**What happens under the covers:**
- Config stores `cloud_projects` dict mapping project names to local paths
- Each project gets its own bisync state in `~/.basic-memory/bisync-state/{project}/`
- Rclone syncs using single remote: `basic-memory-cloud`
- Projects can live anywhere on your filesystem, not forced into sync directory
## Quick Start
### 1. Authenticate Cloud Access
Authenticate with cloud:
```bash
bm cloud login
```
**What this does:**
1. Opens browser to Basic Memory Cloud authentication page
2. Stores authentication tokens in `~/.basic-memory/basic-memory-cloud.json`
3. Validates your subscription status
4. Leaves routing behavior unchanged (auth only)
**Result:** Cloud credentials are available for cloud-routed commands.
Apply OSS discount code `{{OSS_DISCOUNT_CODE}}` during checkout to receive 20% off for 3 months.
### 2. Set Up Sync
Install rclone and configure credentials:
```bash
bm cloud setup
```
**What this does:**
1. Installs rclone automatically (if needed)
2. Fetches your tenant information from cloud
3. Generates scoped S3 credentials for sync
4. Configures single rclone remote: `basic-memory-cloud`
**Result:** You're ready to sync projects. No sync directories created yet - those come with project setup.
### 3. Add Projects with Sync
Create projects with optional local sync paths:
```bash
# Create cloud project without local sync
bm project add research --cloud
# Create cloud project WITH local sync
bm project add research --cloud --local-path ~/Documents/research
# Or configure sync for existing project
bm project sync-setup research ~/Documents/research
```
**What happens under the covers:**
When you add a project with `--local-path`:
1. Project created on cloud at `/app/data/research`
2. Local path stored in config for that project (`cloud_sync_path`)
3. Local directory created if it doesn't exist
4. Bisync state directory created at `~/.basic-memory/bisync-state/research/`
**Result:** Project is ready to sync, but no files synced yet.
### 4. Sync Your Project
Establish the initial sync baseline. **Best practice:** Always preview with `--dry-run` first:
```bash
# Step 1: Preview the initial sync (recommended)
bm project bisync --name research --resync --dry-run
# Step 2: If all looks good, run the actual sync
bm project bisync --name research --resync
```
**What happens under the covers:**
1. Rclone reads from `~/Documents/research` (local)
2. Connects to `basic-memory-cloud:bucket-name/app/data/research` (remote)
3. Creates bisync state files in `~/.basic-memory/bisync-state/research/`
4. Syncs files bidirectionally with settings:
- `conflict_resolve=newer` (most recent wins)
- `max_delete=25` (safety limit)
- Respects `.bmignore` patterns
**Result:** Local and cloud are in sync. Baseline established.
**Why `--resync`?** This is an rclone requirement for the first bisync run. It establishes the initial state that future syncs will compare against. After the first sync, never use `--resync` unless you need to force a new baseline.
See: https://rclone.org/bisync/#resync
```
--resync
This will effectively make both Path1 and Path2 filesystems contain a matching superset of all files. By default, Path2 files that do not exist in Path1 will be copied to Path1, and the process will then copy the Path1 tree to Path2.
```
### 5. Subsequent Syncs
After the first sync, just run bisync without `--resync`:
```bash
bm project bisync --name research
```
**What happens:**
1. Rclone compares local and cloud states
2. Syncs changes in both directions
3. Auto-resolves conflicts (newer file wins)
4. Updates `last_sync` timestamp in config
**Result:** Changes flow both ways - edit locally or in cloud, both stay in sync.
### 6. Verify Setup
Check status:
```bash
bm cloud status
```
You should see:
- `OAuth: token valid` (or missing/expired)
- `API Key: configured` (or not set)
- `Cloud instance is healthy`
- Instructions for project sync commands
## Working with Projects
### Understanding Project Commands
**Key concept:** Use regular `bm project` commands (not `bm cloud project`).
```bash
# Local route
bm project list --local
bm project add research ~/Documents/research
# Cloud route
bm project list --cloud
bm project add research --cloud
```
### Creating Projects
**Use case 1: Cloud-only project (no local sync)**
```bash
bm project add temp-notes --cloud
```
**What this does:**
- Creates project on cloud at `/app/data/temp-notes`
- No local directory created
- No sync configuration
**Result:** Project exists on cloud, accessible via MCP tools, but no local copy.
**Use case 2: Cloud project with local sync**
```bash
bm project add research --cloud --local-path ~/Documents/research
```
**What this does:**
- Creates project on cloud at `/app/data/research`
- Creates local directory `~/Documents/research`
- Stores sync config in `~/.basic-memory/config.json`
- Prepares for bisync (but doesn't sync yet)
**Result:** Project ready to sync. Run `bm project bisync --name research --resync` to establish baseline.
**Use case 3: Add sync to existing cloud project**
```bash
# Project already exists on cloud
bm project sync-setup research ~/Documents/research
```
**What this does:**
- Updates existing project's sync configuration
- Creates local directory
- Prepares for bisync
**Result:** Existing cloud project now has local sync path. Run bisync to pull files down.
### Listing Projects
View all projects:
```bash
bm project list
```
**What you see:**
- Local projects always
- Cloud projects when credentials are available
- Default project marked
- Route-related metadata (for example, local/cloud presence and sync info)
Example shape (single row for dual-presence projects):
```text
Name Path Local Path Cloud Path CLI Default MCP (stdio)
main /basic-memory ~/basic-memory /basic-memory local local
specs /specs ~/dev/specs /specs cloud local
```
### When a Project Exists in Both Local and Cloud
Use routing flags to disambiguate command targets:
```bash
# Force local target for this command
bm project info main --local
bm project ls --name main --local
# Force cloud target for this command
bm project info main --cloud
bm project ls --name main --cloud
```
Default behavior for no-project, no-flag commands is local.
For MCP stdio, routing is always local.
## File Synchronization
### Understanding the Sync Commands
**There are three sync-related commands:**
1. `bm project sync` - One-way: local → cloud (make cloud match local)
2. `bm project bisync` - Two-way: local ↔ cloud (recommended)
3. `bm project check` - Verify files match (no changes)
### One-Way Sync: Local → Cloud
**Use case:** You made changes locally and want to push to cloud (overwrite cloud).
```bash
bm project sync --name research
```
**What happens:**
1. Reads files from `~/Documents/research` (local)
2. Uses rclone sync to make cloud identical to local
3. Respects `.bmignore` patterns
4. Shows progress bar
**Result:** Cloud now matches local exactly. Any cloud-only changes are overwritten.
**When to use:**
- You know local is the source of truth
- You want to force cloud to match local
- You don't care about cloud changes
### Two-Way Sync: Local ↔ Cloud (Recommended)
**Use case:** You edit files both locally and in cloud UI, want both to stay in sync.
```bash
# First time - establish baseline
bm project bisync --name research --resync
# Subsequent syncs
bm project bisync --name research
```
**What happens:**
1. Compares local and cloud states using bisync metadata
2. Syncs changes in both directions
3. Auto-resolves conflicts (newer file wins)
4. Detects excessive deletes and fails safely (max 25 files)
**Conflict resolution example:**
```bash
# Edit locally
echo "Local change" > ~/Documents/research/notes.md
# Edit same file in cloud UI
# Cloud now has: "Cloud change"
# Run bisync
bm project bisync --name research
# Result: Newer file wins (based on modification time)
# If cloud was more recent, cloud version kept
# If local was more recent, local version kept
```
**When to use:**
- Default workflow for most users
- You edit in multiple places
- You want automatic conflict resolution
### Verify Sync Integrity
**Use case:** Check if local and cloud match without making changes.
```bash
bm project check --name research
```
**What happens:**
1. Compares file checksums between local and cloud
2. Reports differences
3. No files transferred
**Result:** Shows which files differ. Run bisync to sync them.
```bash
# One-way check (faster)
bm project check --name research --one-way
```
### Preview Changes (Dry Run)
**Use case:** See what would change without actually syncing.
```bash
bm project bisync --name research --dry-run
```
**What happens:**
1. Runs bisync logic
2. Shows what would be transferred/deleted
3. No actual changes made
**Result:** Safe preview of sync operations.
### Advanced: List Project Files by Route
**Use case:** Inspect local or cloud project files explicitly.
```bash
# List local project files (default target when no route flag is given)
bm project ls --name research
bm project ls --name research --local
# List cloud project files
bm project ls --name research --cloud
# List files in subdirectory
bm project ls --name research --cloud --path subfolder
```
**What happens:**
1. Resolves route from flags (or local default when no route is given)
2. Lists files for the chosen project instance
3. No files transferred
**Result:** See file listing for the target route.
## Multiple Projects
### Syncing Multiple Projects
**Use case:** You have several projects with local sync, want to sync all at once.
```bash
# Setup multiple projects
bm project add research --cloud --local-path ~/Documents/research
bm project add work --cloud --local-path ~/work-notes
bm project add personal --cloud --local-path ~/personal
# Establish baselines
bm project bisync --name research --resync
bm project bisync --name work --resync
bm project bisync --name personal --resync
# Daily workflow: sync everything
bm project bisync --name research
bm project bisync --name work
bm project bisync --name personal
```
**Future:** `--all` flag will sync all configured projects:
```bash
bm project bisync --all # Coming soon
```
### Mixed Usage
**Use case:** Some projects sync, some stay cloud-only.
```bash
# Projects with sync
bm project add research --cloud --local-path ~/Documents/research
bm project add work --cloud --local-path ~/work
# Cloud-only projects
bm project add archive --cloud
bm project add temp-notes --cloud
# Sync only the configured ones
bm project bisync --name research
bm project bisync --name work
# Archive and temp-notes stay cloud-only
```
**Result:** Fine-grained control over what syncs.
## Per-Project Cloud Routing (API Key)
Route individual projects through cloud using an API key. This lets you keep some projects local while others route through cloud.
### Setting Up API Key Auth
**Option A: Create a key in the web app, then save it locally:**
```bash
bm cloud set-key bmc_abc123...
```
**Option B: Create a key via CLI (requires OAuth login first):**
```bash
bm cloud login # One-time OAuth login
bm cloud create-key "my-laptop" # Creates key and saves it locally
```
The API key is account-level — it grants access to all your cloud projects. It's stored in `~/.basic-memory/config.json` as `cloud_api_key`.
### Setting Project Modes
```bash
# Route a project through cloud
bm project set-cloud research
# Revert to local mode
bm project set-local research
# View project modes
bm project list
```
**What happens:**
- `set-cloud`: validates the API key exists, then sets the project mode to `cloud` in config
- `set-local`: reverts the project to local mode (removes the mode entry from config)
- MCP tools and CLI commands for that project will route to `cloud_host/proxy` with the API key as Bearer token
### How It Works
When an MCP tool or CLI command runs for a cloud-mode project:
1. `get_client(project_name="research")` checks the project's mode in config
2. If mode is `cloud`, creates an HTTP client pointed at `cloud_host/proxy` with `Authorization: Bearer bmc_...`
3. If mode is `local` (default), uses the in-process ASGI transport as usual
**Routing priority** (highest to lowest):
1. Factory injection (cloud app, tests)
2. Explicit route override (`--local` / `--cloud`)
3. Per-project cloud mode (API key)
4. Local ASGI transport (default)
Route override environment variables:
- `BASIC_MEMORY_FORCE_LOCAL=true`
- `BASIC_MEMORY_FORCE_CLOUD=true`
- `BASIC_MEMORY_EXPLICIT_ROUTING=true`
No-project, no-flag CLI commands default to local routing.
### Configuration Example
```json
{
"projects": {
"personal": "/Users/me/notes",
"research": "/Users/me/research"
},
"project_modes": {
"research": "cloud"
},
"cloud_api_key": "bmc_abc123...",
"cloud_host": "https://cloud.basicmemory.com",
"default_project": "personal"
}
```
In this example, `personal` stays local and `research` routes through cloud. Projects not listed in `project_modes` default to local.
### Sync Behavior
Cloud-mode projects are automatically skipped during local file sync (background sync and file watching). Their files live on the cloud instance, not locally.
## OAuth Logout
```bash
bm cloud logout
```
**What this does:**
1. Removes stored OAuth token(s)
2. Does not change per-project route configuration
3. Does not change command routing defaults
**Result:** OAuth session is cleared. API-key-based routing still works if `cloud_api_key` is configured.
## Filter Configuration
### Understanding .bmignore
**The problem:** You don't want to sync everything (e.g., `.git`, `node_modules`, database files).
**The solution:** `.bmignore` file with gitignore-style patterns.
**Location:** `~/.basic-memory/.bmignore`
**Default patterns:**
```gitignore
# Version control
.git/**
# Python
__pycache__/**
*.pyc
.venv/**
venv/**
# Node.js
node_modules/**
# Basic Memory internals
memory.db/**
memory.db-shm/**
memory.db-wal/**
config.json/**
watch-status.json/**
.bmignore.rclone/**
# OS files
.DS_Store/**
Thumbs.db/**
# Environment files
.env/**
.env.local/**
```
**How it works:**
1. On first sync, `.bmignore` created with defaults
2. Patterns converted to rclone filter format (`.bmignore.rclone`)
3. Rclone uses filters during sync
4. Same patterns used by all projects
**Customizing:**
```bash
# Edit patterns
code ~/.basic-memory/.bmignore
# Add custom patterns
echo "*.tmp/**" >> ~/.basic-memory/.bmignore
# Next sync uses updated patterns
bm project bisync --name research
```
## Troubleshooting
### Authentication Issues
**Problem:** "Authentication failed" or "Invalid token"
**Solution:** Re-authenticate:
```bash
bm cloud logout
bm cloud login
```
### Subscription Issues
**Problem:** "Subscription Required" error
**Solution:**
1. Visit subscribe URL shown in error
2. Sign up for subscription
3. Run `bm cloud login` again
**Note:** Access is immediate when subscription becomes active.
### Bisync Initialization
**Problem:** "First bisync requires --resync"
**Explanation:** Bisync needs a baseline state before it can sync changes.
**Solution:**
```bash
bm project bisync --name research --resync
```
**What this does:**
- Establishes initial sync state
- Creates baseline in `~/.basic-memory/bisync-state/research/`
- Syncs all files bidirectionally
**Result:** Future syncs work without `--resync`.
### Empty Directory Issues
**Problem:** "Empty prior Path1 listing. Cannot sync to an empty directory"
**Explanation:** Rclone bisync doesn't work well with completely empty directories. It needs at least one file to establish a baseline.
**Solution:** Add at least one file before running `--resync`:
```bash
# Create a placeholder file
echo "# Research Notes" > ~/Documents/research/README.md
# Now run bisync
bm project bisync --name research --resync
```
**Why this happens:** Bisync creates listing files that track the state of each side. When both directories are completely empty, these listing files are considered invalid by rclone.
**Best practice:** Always have at least one file (like a README.md) in your project directory before setting up sync.
### Bisync State Corruption
**Problem:** Bisync fails with errors about corrupted state or listing files
**Explanation:** Sometimes bisync state can become inconsistent (e.g., after mixing dry-run and actual runs, or after manual file operations).
**Solution:** Clear bisync state and re-establish baseline:
```bash
# Clear bisync state
bm project bisync-reset research
# Re-establish baseline
bm project bisync --name research --resync
```
**What this does:**
- Removes all bisync metadata from `~/.basic-memory/bisync-state/research/`
- Forces fresh baseline on next `--resync`
- Safe operation (doesn't touch your files)
**Note:** This command also runs automatically when you remove a project to clean up state directories.
### Too Many Deletes
**Problem:** "Error: max delete limit (25) exceeded"
**Explanation:** Bisync detected you're about to delete more than 25 files. This is a safety check to prevent accidents.
**Solution 1:** Review what you're deleting, then force resync:
```bash
# Check what would be deleted
bm project bisync --name research --dry-run
# If correct, establish new baseline
bm project bisync --name research --resync
```
**Solution 2:** Use one-way sync if you know local is correct:
```bash
bm project sync --name research
```
### Project Not Configured for Sync
**Problem:** "Project research has no local_sync_path configured"
**Explanation:** Project exists on cloud but has no local sync path.
**Solution:**
```bash
bm project sync-setup research ~/Documents/research
bm project bisync --name research --resync
```
### Connection Issues
**Problem:** "Cannot connect to cloud instance"
**Solution:** Check status:
```bash
bm cloud status
```
If instance is down, wait a few minutes and retry.
## Security
- **Authentication**: OAuth 2.1 with PKCE flow
- **Tokens**: Stored securely in `~/.basic-memory/basic-memory-cloud.json`
- **Transport**: All data encrypted in transit (HTTPS)
- **Credentials**: Scoped S3 credentials (read-write to your tenant only)
- **Isolation**: Your data isolated from other tenants
- **Ignore patterns**: Sensitive files automatically excluded via `.bmignore`
## Command Reference
### Cloud Authentication
```bash
bm cloud login # Authenticate and store OAuth credentials
bm cloud logout # Remove stored OAuth credentials
bm cloud status # Check auth state and instance health
bm cloud promo --off # Disable CLI cloud promo notices
```
### API Key Management
```bash
bm cloud set-key <key> # Save a cloud API key (bmc_ prefixed)
bm cloud create-key <name> # Create API key via cloud API (requires OAuth login)
```
### Setup
```bash
bm cloud setup # Install rclone and configure credentials
```
### Project Management
```bash
bm project list --local # Local project list
bm project list --cloud # Cloud project list
bm project add <name> --cloud # Create cloud project (no sync)
bm project add <name> --cloud --local-path <path> # Create with local sync
bm project sync-setup <name> <path> # Add sync to existing project
bm project rm <name> # Delete project
```
### Per-Project Routing
```bash
bm project set-cloud <name> # Route project through cloud (requires API key)
bm project set-local <name> # Revert project to local mode
```
### File Synchronization
```bash
# One-way sync (local → cloud)
bm project sync --name <project>
bm project sync --name <project> --dry-run
bm project sync --name <project> --verbose
# Two-way sync (local ↔ cloud) - Recommended
bm project bisync --name <project> # After first --resync
bm project bisync --name <project> --resync # First time / force baseline
bm project bisync --name <project> --dry-run
bm project bisync --name <project> --verbose
# Integrity check
bm project check --name <project>
bm project check --name <project> --one-way
# List project files by route
bm project ls --name <project> # Default target: local
bm project ls --name <project> --local
bm project ls --name <project> --cloud
bm project ls --name <project> --cloud --path <subpath>
```
## Summary
**Basic Memory Cloud uses project-scoped sync:**
1. **Authenticate cloud access** - `bm cloud login`
2. **Install rclone** - `bm cloud setup`
3. **Add projects with sync** - `bm project add research --cloud --local-path ~/Documents/research`
4. **Preview first sync** - `bm project bisync --name research --resync --dry-run`
5. **Establish baseline** - `bm project bisync --name research --resync`
6. **Daily workflow** - `bm project bisync --name research`
**Key benefits:**
- ✅ Each project independently syncs (or doesn't)
- ✅ Projects can live anywhere on disk
- ✅ Explicit sync operations (no magic)
- ✅ Safe by design (max delete limits, conflict resolution)
- ✅ Full offline access (work locally, sync when ready)
**Future enhancements:**
- `--all` flag to sync all configured projects
- Project list showing sync status
- Watch mode for automatic sync
-91
View File
@@ -1,91 +0,0 @@
# Cloud Semantic Search Value (Customer-Facing Technical Story)
This document explains why teams should buy cloud semantic search even when local search exists.
## Core Promise
Markdown files remain the source of truth in both local and cloud modes.
- Files are portable.
- Search indexes are derived and rebuildable.
- You never get locked into proprietary document storage.
## The Customer Problem
Teams paying for cloud are usually not optimizing for "can this run locally." They are optimizing for:
- finding the right note the first time,
- keeping retrieval quality high as note volume grows,
- avoiding search slowdowns while content is actively changing,
- getting consistent results across users, agents, and sessions.
## Why Cloud Is the Aspirin
Cloud semantic search is the immediate pain reliever because it fixes the problems users feel right now.
### 1) Better hit rate on real queries
Cloud uses stronger managed embeddings than the default local model, which improves semantic recall for paraphrases and vague questions.
Customer outcome:
- fewer "I know this exists but search missed it" moments,
- less query rewording,
- faster time to answer.
### 2) Better behavior under active workloads
Cloud indexing runs out of band in workers, so indexing does not compete with interactive read/write traffic.
Customer outcome:
- stable search responsiveness during heavy updates,
- fresher semantic results shortly after edits,
- less user-visible performance variance.
### 3) Better consistency for shared knowledge
Cloud retrieval runs against a centralized tenant index, so teams and agents resolve against the same semantic state.
Customer outcome:
- fewer "works on my machine" search differences,
- more predictable agent behavior across environments,
- easier cross-user collaboration on large knowledge bases.
### 4) Better quality at higher scale
With Postgres + `pgvector` per tenant, cloud can sustain larger note collections and higher query volumes than typical local setups.
Customer outcome:
- confidence as repositories grow to tens of thousands of notes,
- less need for user-side tuning,
- fewer quality regressions as usage increases.
## Local Is the Vitamin
Local semantic search still matters and should stay strong.
- offline use,
- privacy-first operation,
- no cloud dependency,
- user-controlled runtime.
It compounds long-term ownership and resilience, but does not remove the immediate pain points cloud solves for teams at scale.
## Recommended Messaging
One-liner:
"Cloud semantic search is the aspirin: it fixes retrieval quality and performance pain now. Local semantic search is the vitamin: it builds long-term control and resilience."
Long form:
"Basic Memory keeps markdown as the source of truth everywhere. Local gives privacy and offline control. Cloud adds immediate, measurable improvements in search quality, consistency, and responsiveness for teams and agents running at scale."
## Packaging Guidance
- Base: local FTS plus optional local semantic search.
- Cloud value: higher semantic quality, stable performance under load, and consistent team-wide retrieval.
- Keep interfaces pluggable (`EmbeddingProvider`, vector backend protocol) so implementation can evolve without changing user workflows.
-138
View File
@@ -1,138 +0,0 @@
# MCP UI Bakeoff - Instructions & Test Plan
Last updated: 2026-02-02
## Scope
Compare three presentation paths for Basic Memory MCP tools:
1. **ToolUI (React)** via MCP App resources.
2. **MCPUI Python SDK** embedded UI resources (legacy host path).
3. **ASCII/ANSI** output for TUI clients.
This doc is the running instruction set and test plan. Update as implementation progresses.
---
## Prerequisites
- Repo: `basic-memory` (worktree: `basic-memory-mcp-ui-poc`)
- Node for toolui build (already used for POC)
- Python 3.12+ with `uv`
Optional (for MCPUI Python SDK path):
- Local repo: `/Users/phernandez/dev/mcp-ui`
- Install the server SDK into the Basic Memory venv:
- `uv pip install -e /Users/phernandez/dev/mcp-ui/sdks/python/server`
---
## Build / Refresh Steps
### ToolUI React bundle
```bash
cd ui/tool-ui-react
npm install
npm run build
```
This regenerates:
- `src/basic_memory/mcp/ui/html/search-results-tool-ui.html`
- `src/basic_memory/mcp/ui/html/note-preview-tool-ui.html`
---
## How to Run the MCP Server
```bash
basic-memory mcp --transport stdio
```
Optional to pick UI variant for MCP App resources:
```bash
export BASIC_MEMORY_MCP_UI_VARIANT=tool-ui # or vanilla | mcp-ui
```
---
## Test Cases
### 1) MCP App Resource UI (toolui / vanilla / mcpui)
Tools:
- `search_notes`
- `read_note`
Expect:
- Tool meta points to `ui://basic-memory/search-results` and `ui://basic-memory/note-preview`
- Resource content differs by `BASIC_MEMORY_MCP_UI_VARIANT`
- Variantspecific URIs also available:
- `ui://basic-memory/search-results/vanilla`
- `ui://basic-memory/search-results/tool-ui`
- `ui://basic-memory/search-results/mcp-ui`
- `ui://basic-memory/note-preview/vanilla`
- `ui://basic-memory/note-preview/tool-ui`
- `ui://basic-memory/note-preview/mcp-ui`
Manual check:
- Trigger tool in MCPAppcapable host and confirm UI renders.
---
### 2) Text / JSON Output Modes
Tools:
- `search_notes(output_format="text" | "json")`
- `read_note(output_format="text" | "json")`
- `write_note(output_format="text" | "json")`
- `edit_note(output_format="text" | "json")`
- `recent_activity(output_format="text" | "json")`
- `list_memory_projects(output_format="text" | "json")`
- `create_memory_project(output_format="text" | "json")`
- `delete_note(output_format="text" | "json")`
- `move_note(output_format="text" | "json")`
- `build_context(output_format="json" | "text")`
Expect:
- `text` mode preserves existing human-readable responses.
- `json` mode returns structured dict/list payloads for machine-readable clients.
Automated:
- `uv run pytest test-int/mcp/test_output_format_json_integration.py`
---
### 3) MCPUI Python SDK (embedded UI resource)
Tools (embedded resource responses):
- `search_notes_ui` (MCPUI SDK)
- `read_note_ui` (MCPUI SDK)
Expected output:
- Tool response content contains an EmbeddedResource (`type: "resource"`)
- `mimeType` is `text/html`
- `_meta` includes:
- `mcpui.dev/ui-preferred-frame-size`
- `mcpui.dev/ui-initial-render-data`
Manual check:
- Render tool responses using `UIResourceRenderer` (legacy host flow).
Automated (if SDK installed):
- `uv run pytest test-int/mcp/test_ui_sdk_integration.py`
---
## Bakeoff Notes Template
Fill in after running:
- ToolUI (React): __
- MCPUI SDK (embedded): __
- Text/JSON modes: __
Decision + rationale: __
-344
View File
@@ -1,344 +0,0 @@
# Post-v0.18.0 Test Plan and Acceptance Criteria
## Goal
Define a complete validation plan for all major features merged after `v0.18.0`, combining:
- Coverage-gap-driven automated tests
- Real MCP server integration tests (no mocks for target flows)
- Manual MCP verification via LLM-driven tool calls
This plan is based on commits in `v0.18.0..HEAD` and the latest `just check` coverage output.
## Scope Window
- Start tag: `v0.18.0` (2026-01-28)
- End: current `main`
- Change volume: 12 feature commits + 14 bug-fix commits (+ release chores/hotfixes)
## Execution Strategy
1. Stabilize all feature-level acceptance criteria in automated tests first.
2. Add black-box MCP integration tests for semantic search + schema (real server startup).
3. Run manual MCP tool-call verification to confirm real UX and routing behavior.
4. Re-run full gate: `just check` + targeted integration packs.
## Global Quality Gates
- Feature criteria below must all pass.
- No regressions in existing suites.
- Coverage improves in targeted low-coverage feature modules.
- SQLite and Postgres parity for search/semantic features.
## Priority Coverage Gaps (from latest run)
These are the most important post-`v0.18.0` feature modules currently under-covered:
- `src/basic_memory/mcp/tools/schema.py` (27%)
- `src/basic_memory/mcp/clients/schema.py` (36%)
- `src/basic_memory/mcp/tools/ui_sdk.py` (43%)
- `src/basic_memory/mcp/tools/search.py` (73%)
- `src/basic_memory/repository/postgres_search_repository.py` (63%)
- `src/basic_memory/mcp/async_client.py` (82%)
- `src/basic_memory/api/v2/routers/schema_router.py` (80%)
## Feature Acceptance Criteria and Test Plan
### 1) Schema System (`c97733d`) — DONE
### Acceptance criteria
- `schema_validate`, `schema_infer`, and `schema_diff` produce consistent outcomes across CLI/API/MCP for the same fixture set.
- Strict validation fails deterministically on required-field/type violations.
- Validation warnings are stable and machine-readable in non-strict mode.
- Inference output is deterministic for unchanged input corpus.
- Drift diff output is deterministic and identifies missing/extra/type-mismatch fields correctly.
### Existing coverage anchor points
- `tests/schema/*`
- `tests/api/v2/test_schema_router.py`
- `test-int/test_schema/*`
### Gaps to close — DONE
- ~~MCP schema tool branches (`src/basic_memory/mcp/tools/schema.py`)~~ — 18 tests in `tests/mcp/test_tool_schema.py`
- ~~MCP schema client behavior (`src/basic_memory/mcp/clients/schema.py`)~~ — `tests/mcp/test_client_schema.py`
- ~~Schema router error-path branches (`src/basic_memory/api/v2/routers/schema_router.py`)~~ — `tests/api/v2/test_schema_router.py`
### Planned additions — DONE
- ~~Add MCP tool tests for `schema_validate` strict + non-strict result shapes.~~ **DONE**
- ~~Add MCP tool tests for `schema_infer` with explicit `entity_type` and inferred type fallback.~~ **DONE**
- ~~Add MCP tool tests for `schema_diff` empty-diff and non-empty-diff paths.~~ **DONE**
- ~~Add API tests for schema router invalid payload/edge error handling.~~ **DONE**
- Add integration test that starts MCP server and calls schema tools end-to-end on fixture notes. — deferred to backlog item 4.
### 2) Semantic Search (`0777879`, `1428d18`, `344e651`) — DONE
### Acceptance criteria
- `search_type=text|vector|hybrid` returns expected ranked results on canonical semantic corpus.
- Missing semantic extras fail fast with actionable install guidance.
- Reindex and provider/model changes produce valid vectors without dimension mismatch.
- SQLite and Postgres produce equivalent behavior for semantic modes on the same dataset.
- Generated-column migration path is valid on SQLite environments in use.
### Existing coverage anchor points
- `tests/repository/test_sqlite_vector_search_repository.py`
- `tests/repository/test_postgres_search_repository.py`
- `tests/services/test_semantic_search.py`
- `tests/mcp/test_tool_search.py`
- `test-int/test_search_performance_benchmark.py`
### Gaps to close — DONE
- ~~Uncovered Postgres vector/hybrid branches~~ — 20 tests in `tests/repository/test_postgres_search_repository_unit.py` + 5 integration tests in `test-int/semantic/test_semantic_coverage.py`
- ~~MCP search semantic/output branches~~ — expanded `tests/mcp/test_tool_search.py`
### Planned additions — DONE
- ~~Expand Postgres repository tests for vector query composition edge cases.~~ **DONE**
- ~~Expand Postgres repository tests for hybrid fusion ranking and pagination branches.~~ **DONE**
- ~~Expand Postgres repository tests for embedding/provider error handling branches.~~ **DONE**
- ~~Expand MCP search tool tests for vector/hybrid output formatting branches.~~ **DONE**
- ~~Expand MCP search tool tests for semantic-disabled and missing-dependency failures.~~ **DONE**
- Add MCP integration tests that start server and execute semantic `search_notes` tool calls. — deferred to backlog item 4.
### Semantic search quality benchmarks (NEW)
Full benchmark suite in `test-int/semantic/` covering 5 backend×provider combinations:
- `sqlite-fts`, `sqlite-fastembed`, `postgres-fts`, `postgres-fastembed`, `postgres-openai`
- Quality metrics: hit@1, recall@5, MRR@10 with per-query timing
- Realistic corpus with cross-topic vocabulary overlap (240 notes, 4 topics)
- Rich CLI viewer: `just semantic-report`
- JSON artifact output: `just test-semantic-report`
Key finding: **FastEmbed (384-d local ONNX) matches or exceeds OpenAI (1536-d) quality at 30x lower latency.** Recommending FastEmbed as default for both local and cloud deployments.
### 3) Per-Project Local/Cloud Routing + API Key Auth (`d84708c`, `ed94877`, `312662f`) — DONE
### Acceptance criteria
- Project mode (`local`/`cloud`) persists and displays correctly.
- Routing selects ASGI for local projects and HTTP+Bearer for cloud projects.
- Cloud project without key fails with explicit remediation (`cloud set-key`/`cloud create-key`).
- Resolution precedence is correct (factory > force-local > per-project cloud > global fallback > local).
- Watch/sync only run for local projects.
### Existing coverage anchor points
- `tests/mcp/test_async_client_modes.py`
- `tests/cli/test_project_set_cloud_local.py`
- `tests/mcp/test_project_context.py`
- `tests/test_project_resolver.py`
- `tests/sync/test_watch_service_reload.py`
### Gaps to close — DONE
- ~~Cloud routing branch gaps in `src/basic_memory/mcp/async_client.py`~~ — expanded `tests/mcp/test_async_client_modes.py`
### Planned additions — DONE
- ~~Add branch-focused tests for all unresolved routing branches in `get_client()`.~~ **DONE**
- Add MCP integration scenario with mixed local/cloud project config — deferred to backlog item 4.
### 4) Project-Prefixed Permalinks + Memory URL Routing (`545804f`) — DONE
### Acceptance criteria
- Project-prefixed permalinks are generated consistently on create/update/import flows.
- Memory URLs resolve to the correct project/entity even with duplicate note titles.
- `read_note`, `search`, `build_context`, write/edit/move flows preserve project identity correctly.
- Link resolution remains correct for context-aware wikilinks.
### Existing coverage anchor points
- `tests/utils/test_permalink_formatting.py`
- `tests/mcp/test_tool_read_note.py`
- `tests/mcp/test_tool_search.py`
- `tests/services/test_context_service.py`
- `test-int/mcp/test_read_note_integration.py`
### Gaps to close
- No major coverage alarm in report, but keep as regression-critical due broad impact surface.
### Planned additions — DONE
- ~~Add one integration test with colliding titles across two projects and assert URL routing invariants.~~ **DONE**`test-int/mcp/test_permalink_collision_integration.py` (2 tests: collision across projects + memory:// URL routing with project prefix)
### 5) MCP UI Variants + TUI Output (`8bc03d1`) — DONE
### Acceptance criteria
- UI resource variant selection (`tool-ui`, `vanilla`, `mcp-ui`) follows env configuration.
- `search_notes` and `read_note` expose expected resource metadata for UI hosts.
- `ascii`/`ansi` outputs are deterministic and stable for terminal clients.
### Existing coverage anchor points
- `tests/mcp/test_tool_contracts.py`
- `test-int/mcp/test_output_format_json_integration.py`
- `test-int/mcp/test_ui_sdk_integration.py`
### Gaps to close — DONE
- ~~`src/basic_memory/mcp/tools/ui_sdk.py` branch coverage~~ — `tests/mcp/test_ui_sdk.py`
- ~~`src/basic_memory/mcp/ui/sdk.py` and `src/basic_memory/mcp/ui/templates.py` branch coverage~~ — `tests/mcp/test_ui_templates.py` + `tests/mcp/test_ui_resources.py`
### Planned additions — DONE
- ~~Add unit tests for UI SDK metadata generation and template selection branches.~~ **DONE** — 31 tests
- ~~Add integration assertion for variant-specific resource URIs and metadata payload shape.~~ **DONE**
### 6) Watch Command (`8df88e4`) — DONE
### Acceptance criteria
- `basic-memory watch` starts and processes create/update/delete events.
- Watch restart/reload path does not duplicate watchers.
- Cloud-mode projects are excluded from active watcher set.
### Existing coverage anchor points
- `tests/cli/test_watch.py`
- `tests/sync/test_coordinator.py`
- `tests/sync/test_watch_service_reload.py`
### Planned additions — DONE
- ~~Add one stress-style integration test for rapid file changes and watcher stability.~~ **DONE**`tests/sync/test_watch_service_stress.py` (3 tests: 50-file batch, mixed add/modify/delete batch, rapid modifications to same file)
### 7) CLI JSON Output (`a47c9c0`) — DONE
### Acceptance criteria
- `--format json` returns valid JSON with stable keys for success paths.
- Error paths also return JSON-shaped output with correct non-zero exits.
- Default human output remains unchanged.
### Existing coverage anchor points
- `tests/cli/test_cli_tool_json_output.py`
- `test-int/cli/test_cli_tool_json_integration.py`
### Planned additions — DONE
- ~~Add one failure-path integration test per high-use tool command.~~ **DONE**`test-int/cli/test_cli_tool_json_failure_integration.py` (4 tests: read-note not found, write-note missing content, write→read roundtrip, recent-activity empty project)
### 8) Search/Edit and Metadata Fixes (`530cbac`, `f1d50c2`, `8838571`, `009e849`) — DONE
### Acceptance criteria
- Metadata filters produce consistent results on SQLite and Postgres.
- `tag:` shorthand works alone and with mixed query terms.
- Fast write/edit paths preserve `external_id` and metadata integrity.
### Existing coverage anchor points
- `tests/repository/test_metadata_filters.py`
- `tests/repository/test_search_repository.py`
- `tests/services/test_search_service.py`
### Planned additions — DONE
- ~~Add Postgres-specific metadata filter edge-case tests to mirror SQLite assertions exactly.~~ **DONE**`tests/repository/test_metadata_filters_edge_cases.py` (6 tests: missing field, AND logic, contains single-element array, nested path missing intermediate, $gte/$lte boundaries, $between inclusive — all pass on both SQLite and Postgres)
### 9) Compatibility and Hotfix Regression Pack (`c46d7a6`, `a0e754b`, `343a6e1`, `24ca5f6`, `e3ced49`, `8489a3d`, `b609c4e`, `f6e0a5b`, `7624a20`)
### Acceptance criteria
- Legacy endpoints required by older CLI versions function without `405` (`GET /projects/projects`, `POST /projects/projects`, `POST /projects/config/sync`).
- Entity creation conflicts map to conflict status (not 500).
- `recent_activity` prompt defaults are correct.
- No spurious `metadata: {}` in serialized frontmatter.
- Tigris/rclone uses global consistency headers for all transaction types.
- `bm --version` fast path avoids heavy import path and remains responsive.
- Default SQLite DB path is isolated by config dir.
### Gaps to close
- ~~Commits with no direct tests added (`c46d7a6`, `344e651`, `f6e0a5b`) need explicit regression tests.~~ **DONE**
### Planned additions — DONE
- ~~Add API compat test covering all legacy endpoint methods and payloads.~~ **DONE**`test_legacy_v1_add_project_endpoint`, `test_legacy_v1_sync_config_endpoint`
- ~~Add CLI fast-path test for `--version` import behavior/performance guard.~~ **DONE**`test_bm_version_does_not_import_heavy_modules`
- ~~Add empty metadata serialization regression test.~~ **DONE**`test_schema_to_markdown_empty_metadata_no_metadata_key`
- Add migration safety test for SQLite generated columns (`VIRTUAL` expectation) — deferred, low risk.
## MCP Manual Verification Plan (LLM Tool Calls)
Run after automated tests pass.
### Setup
- Start MCP server: `basic-memory mcp --transport stdio`
- Use an MCP-capable client and issue tool calls directly.
### Manual scenarios
- Schema: call `schema_validate`, `schema_infer`, and `schema_diff` on known fixtures.
- Schema: verify error and success payloads match acceptance criteria.
- Semantic search: call `search_notes` with `search_type=text|vector|hybrid`.
- Semantic search: verify ranking relevance on semantic fixture queries.
- Routing: call tools with explicit project on mixed local/cloud setup.
- Routing: verify success/failure paths with and without API key.
- Permalink routing: read/write/search notes across projects with colliding titles.
- Permalink routing: verify memory URL routing correctness.
- UI/TUI: call `search_notes` and `read_note` with UI variants and `output_format=text|json`.
- UI/TUI: verify payload/resource format and metadata completeness.
## Implementation Backlog (Ordered)
1. ~~Fill schema MCP/client/router coverage gaps.~~ **DONE** — 18 tests in `test_tool_schema.py` + `test_client_schema.py`
2. ~~Fill semantic search MCP + Postgres repository gaps.~~ **DONE** — 20 tests in `test_postgres_search_repository_unit.py` + `test_tool_search.py`
3. ~~Add compatibility regression tests (legacy endpoints, migration, version fast path).~~ **DONE** — 5 tests across 3 files (see below)
4. ~~Add feature-level integration tests (permalinks, watch, CLI JSON, metadata filters).~~ **DONE** — 15 tests across 4 files (see items 4, 6, 7, 8 above)
5. ~~Expand UI SDK and template branch tests.~~ **DONE** — 31 tests in `test_ui_templates.py` + `test_ui_sdk.py` + `test_ui_resources.py`
6. ~~Run full gate and capture results in a short release readiness summary.~~ **DONE** — see results below
### Full Gate Results (`just check`)
| Phase | Result |
|-------|--------|
| lint | PASS |
| format | PASS |
| typecheck | PASS |
| Unit tests (SQLite) | 1788 passed, 15 skipped |
| Integration tests (SQLite) | 243 passed, 4 skipped, 10 deselected |
| Unit tests (Postgres) | 1760 passed, 28 skipped |
| Integration tests (Postgres) | 234 passed, 13 skipped, 10 deselected |
**0 failures. 10 deselected = semantic benchmark tests (run separately via `just test-semantic`).**
### Item 3 Details — Compatibility Regression Tests
| Test | File | What it covers |
|------|------|----------------|
| `test_legacy_v1_add_project_endpoint` | `tests/api/v2/test_project_router.py` | POST `/projects/projects` legacy route reachable (idempotent path) |
| `test_legacy_v1_sync_config_endpoint` | `tests/api/v2/test_project_router.py` | POST `/projects/config/sync` legacy route reachable |
| `test_bm_version_does_not_import_heavy_modules` | `tests/cli/test_cli_exit.py` | `bm --version` fast path does not load `basic_memory.mcp` |
| `test_schema_to_markdown_empty_metadata_no_metadata_key` | `tests/markdown/test_entity_parser_error_handling.py` | `schema_to_markdown()` with `entity_metadata={}` emits no `metadata:` key |
| `test_legacy_v1_list_projects_endpoint` | `tests/api/v2/test_project_router.py` | (pre-existing) GET `/projects/projects` legacy route |
**Suite totals after item 3: 1764 passed, 15 skipped, 0 failures.**
## Suggested Commands
- Full suite: `just check`
- Fast loop: `just fast-check`
- E2E consistency: `just doctor`
- SQLite focused: `just test-sqlite`
- Postgres focused: `just test-postgres`
- Schema integration: `pytest test-int/test_schema -q`
- Semantic + repo focus: `pytest tests/repository/test_postgres_search_repository.py tests/mcp/test_tool_search.py tests/services/test_semantic_search.py -q`
- MCP integration focus: `pytest test-int/mcp -q`
## Exit Criteria for This Plan
- All feature acceptance criteria above are validated.
- All identified high-priority coverage gaps are addressed or explicitly documented as intentional.
- Manual MCP verification scenarios complete with no P0/P1 findings.
-209
View File
@@ -1,209 +0,0 @@
# Semantic Search Manual Test Log
## Overview
Manual test session for semantic (vector) search on the main project.
- Date: 2026-02-15
- Database: ~/.basic-memory/memory.db (SQLite)
- Entities: 456 embedded, 2714 vector chunks
- Search index: 2390 FTS entries
- Embedding model: default (384-dim, sqlite-vec)
## Test Plan
1. **Search Type Routing** — verify vector/hybrid/text dispatch, invalid search_type handling
2. **Conceptual Queries** — natural language where vector should beat FTS
3. **Keyword Queries** — exact terms where FTS should be strong
4. **Hybrid Ranking** — queries where both FTS and vector contribute
5. **Result Types** — entities, observations, relations in vector results
6. **Filters + Vector** — combine vector with types/entity_types/after_date
7. **Edge Cases** — short queries, long queries, empty, special chars, no-match
8. **Pagination** — page > 1, page_size respected
---
## Test Results
### Test 1: Search Type Routing
#### 1a: search_type="semantic" (invalid value)
- **Input:** query="how does the knowledge graph work", search_type="semantic"
- **Expected:** error or explicit fallback
- **Actual:** Silently falls through to text search (else branch in search.py:430)
- **Verdict:** BUG — should either be a recognized alias for "vector" or return an error
#### 1b: search_type="vector"
- **Input:** query="keeping AI context between sessions", search_type="vector"
- **Actual:** 5 results, scores ~0.58-0.59, found "Maintaining context across conversation boundaries" observation
- **Verdict:** PASS
#### 1c: search_type="text" with conceptual query
- **Input:** query="keeping AI context between sessions", search_type="text"
- **Actual:** 0 results (no exact keyword match)
- **Verdict:** PASS (expected — FTS requires token overlap)
#### 1d: search_type="hybrid" with conceptual query
- **Input:** query="keeping AI context between sessions", search_type="hybrid"
- **Actual:** 5 results, same ranking as vector (FTS contributed nothing here)
- **Verdict:** PASS
#### 1e: search_type="text" with keyword query
- **Input:** query="OAuth authentication", search_type="text"
- **Actual:** 3 results — AUTH.md Supabase OAuth, OAuth Rip-and-Replace, OAuth Integration Analysis
- **Verdict:** PASS
#### 1f: search_type="vector" with keyword query
- **Input:** query="OAuth authentication", search_type="vector"
- **Actual:** Same top results as text (keyword-rich content also scores well in vector space)
- **Verdict:** PASS
---
### Test 2: Conceptual Queries (vector advantage)
#### 2a: Natural language question
- **Input:** query="why do AI assistants forget things", search_type="vector"
- **Actual:** 5 results — Manual Testing Session, "Balance security and usability" observation, "Tools should match thought patterns" observation. Scores ~0.56-0.57
- **Vector advantage:** Found conceptually related content despite no exact keyword overlap
- **Verdict:** PASS
#### 2b: Same query, text search
- **Input:** query="why do AI assistants forget things", search_type="text"
- **Actual:** 1 result — "What is Basic Memory?" (likely matched on "AI" token)
- **Verdict:** PASS (demonstrates vector advantage — text barely matched)
#### 2c: Domain concept with no jargon
- **Input:** query="pricing strategy for cloud product", search_type="vector"
- **Actual:** 3 results — SPEC-16 MCP Cloud Service Consolidation, knowledge architecture observation, Visual Knowledge Spaces relation. Scores ~0.56-0.57
- **Verdict:** PASS (found cloud-related content conceptually)
#### 2d: Technical concept, long query
- **Input:** query="SQLite performance optimization WAL mode concurrent writes", search_type="vector"
- **Actual:** 3 results — SPEC-11 API Performance Optimization, Real-Time Updates with WebSockets, marketing status update. Scores ~0.55-0.58
- **Verdict:** PASS (found performance-related content)
---
### Test 3: Keyword Queries (FTS strength)
#### 3a: Exact term match — "OAuth authentication"
- **Text:** 3 results with high relevance (exact matches in titles)
- **Vector:** Same top results (keyword overlap helps vector too)
- **Verdict:** PASS — FTS and vector converge on keyword-rich queries
#### 3b: "OAuth" single keyword, hybrid mode
- **Input:** query="OAuth", search_type="hybrid"
- **Actual:** 5 results — Basic Memory Coding Guide, AI Collaboration Examples, SPEC-18, daily note, Manual Testing Session. FTS + vector blended. Scores ~0.016-0.032
- **Note:** Top hybrid result is "Basic Memory Coding Guide" not an OAuth-specific doc — suggests hybrid scoring may dilute strong FTS matches
- **Verdict:** PASS but hybrid ranking questionable for single-keyword queries
---
### Test 4: Hybrid Ranking
#### 4a: Hybrid vs vector on "OAuth authentication"
- **Hybrid with entity_types=["entity"]:** 5 results — RLS Implementation Lessons, Cloud Readiness Assessment, AUTH.md OAuth, Core Service Implementation, OAuth Rip-and-Replace. Scores ~0.016-0.023
- **Vector with entity_types=["entity"]:** 5 results — Core Service Implementation, SPEC-13 CLI Auth, Coding Guide, Authentication Service, ADR Production Auth. Scores ~0.55-0.60
- **Observation:** Hybrid surfaces different top results than vector-only. Hybrid found RLS and Cloud Readiness docs that vector didn't prioritize. Different ranking is expected from RRF fusion.
- **Verdict:** PASS — hybrid produces meaningfully different ranking
---
### Test 5: Result Types
#### 5a: Vector returns all result types
- **Input:** query="keeping AI context between sessions", search_type="vector"
- **Entities:** SPEC-18 AI Memory Management Tool (type=entity)
- **Relations:** Prompt Builder integrates_with (type=relation)
- **Observations:** "Translation layer is key" (type=observation), "Maintaining context across conversation boundaries" (type=observation)
- **Verdict:** PASS — all three types appear in vector results
#### 5b: Observations carry metadata
- **Observation result:** category="challenge", content="Maintaining context across conversation boundaries", from_entity="research/ai-knowledge-management-research"
- **Verdict:** PASS — category, content, from_entity, tags all present
#### 5c: Relations carry link info
- **Relation result:** relation_type="integrates_with", from_entity="development/features/prompt-builder...", to_entity (present but truncated in some)
- **Verdict:** PASS — relation metadata present
---
### Test 6: Filters + Vector Search
#### 6a: entity_types=["entity"] with vector
- **Input:** query="OAuth authentication", search_type="vector", entity_types=["entity"]
- **Actual:** 5 results, all type="entity" (Core Service Implementation, SPEC-13, Coding Guide, Authentication Service, ADR Auth)
- **Verdict:** PASS — filter correctly restricts to entities only
#### 6b: types=["note"] with vector
- **Input:** query="OAuth authentication", search_type="vector", types=["note"]
- **Actual:** Same 5 results (all have entity_type="note" in metadata)
- **Verdict:** PASS — types filter works with vector search
#### 6c: after_date with vector
- **Input:** query="OAuth authentication", search_type="vector", after_date="2025-06-01"
- **Actual:** 3 results — Core Service Implementation, Cloud Web App analysis observation, SPEC-13. Filtered out older OAuth docs.
- **Verdict:** PASS — date filter applied correctly
#### 6d: entity_types=["entity"] with hybrid
- **Input:** query="OAuth authentication", search_type="hybrid", entity_types=["entity"]
- **Actual:** 5 results, all type="entity" — RLS lessons, Cloud Readiness, AUTH.md OAuth, Core Service, OAuth Rip-and-Replace
- **Verdict:** PASS — filter works with hybrid mode too
#### 6e: types=["entity"] with vector (WRONG filter name)
- **Input:** query="OAuth authentication", search_type="vector", types=["entity"]
- **Actual:** 0 results
- **Note:** `types` filters by entity_type metadata (e.g., "note", "person"), NOT by SearchItemType. Using types=["entity"] looks for entity_type="entity" which few/no notes have. This is a UX confusion point — the param names are ambiguous.
- **Verdict:** PASS (correct behavior) but USABILITY ISSUE — easy to confuse types vs entity_types
---
### Test 7: Edge Cases
#### 7a: Single character query
- **Input:** query="x", search_type="vector"
- **Actual:** 3 results — "Self-contained application bundle" observation, Non-Markdown File Support relation, quick-win-tools entity. Scores ~0.57-0.59
- **Note:** Single character still produces an embedding and returns results. Quality is low/random as expected.
- **Verdict:** PASS (no crash, returns results)
#### 7b: Whitespace-only query
- **Input:** query=" ", search_type="vector"
- **Actual:** 0 results
- **Verdict:** PASS (handled gracefully — _check_vector_eligible strips and rejects empty)
#### 7c: Query with no relevant content
- **Input:** query="quantum computing blockchain", search_type="vector"
- **Actual:** 3 results — Inter-Agent Communication relation, Self-contained bundle observation, JSON-LD interop observation. Scores ~0.54
- **Note:** Still returns results because vector search always finds nearest neighbors. Scores are lower (~0.54) than relevant queries (~0.58-0.60). No relevance threshold applied.
- **Verdict:** PASS (expected behavior) but NOTE — no relevance cutoff means irrelevant queries always return something
---
### Test 8: Pagination
#### 8a: Vector search page 2
- **Input:** query="keeping AI context between sessions", search_type="vector", page=2, page_size=3
- **Actual:** 3 results on page 2, current_page=2. Different results from page 1. Top: "Maintaining context across conversation boundaries" observation (score 0.587)
- **Note:** Interestingly, page 2 had a higher-scoring result than some page 1 results. This may indicate pagination doesn't sort globally — it might be paginating within a pre-scored set.
- **Verdict:** PASS (pagination works) but POSSIBLE ISSUE — result ordering across pages needs investigation
---
## Summary
### Passing Tests: 20/21
### Bugs Found
1. **search_type="semantic" silently falls through** (Test 1a) — Invalid search_type values fall to the `else` branch and default to text search without any warning. Should either alias "semantic" to "vector" or raise an error.
### Usability Issues
2. **types vs entity_types confusion** (Test 6e) — `types` filters by entity_type metadata (note, person, etc.) while `entity_types` filters by SearchItemType (entity, observation, relation). The naming is ambiguous and easy to mix up.
3. **No relevance threshold** (Test 7c) — Vector search always returns nearest neighbors even for completely irrelevant queries. Consider adding a minimum score threshold or at least documenting expected score ranges.
4. **Hybrid ranking for single keywords** (Test 3b) — Hybrid mode on simple keyword queries produced less intuitive rankings than pure FTS or pure vector. The RRF fusion may dilute strong FTS signals.
### Observations
- Vector search successfully finds conceptually related content that FTS misses entirely
- Score ranges: relevant queries ~0.56-0.60, irrelevant queries ~0.54 (narrow spread)
- All three result types (entity, observation, relation) appear correctly in vector results
- Filters (entity_types, types, after_date) all work correctly with vector and hybrid modes
- Pagination works but cross-page ordering may need investigation
-265
View File
@@ -1,265 +0,0 @@
# Semantic Search
This guide covers Basic Memory's optional semantic (vector) search feature, which adds meaning-based retrieval alongside the existing full-text search.
## Overview
Basic Memory's default search uses full-text search (FTS) — keyword matching with boolean operators. Semantic search adds vector embeddings that capture the *meaning* of your content, enabling:
- **Paraphrase matching**: Find "authentication flow" when searching for "login process"
- **Conceptual queries**: Search for "ways to improve performance" and find notes about caching, indexing, and optimization
- **Hybrid retrieval**: Combine the precision of keyword search with the recall of semantic similarity
Semantic search is **opt-in** — existing behavior is completely unchanged unless you enable it. It works on both SQLite (local) and Postgres (cloud) backends.
## Installation
Semantic search dependencies (fastembed, sqlite-vec, openai) are **optional extras** — they are not installed with the base `basic-memory` package. Install them with:
```bash
pip install 'basic-memory[semantic]'
```
This keeps the base install lightweight and avoids platform-specific issues with ONNX Runtime wheels.
### Platform Compatibility
| Platform | FastEmbed (local) | OpenAI (API) |
|---|---|---|
| macOS ARM64 (Apple Silicon) | Yes | Yes |
| macOS x86_64 (Intel Mac) | No — see workaround below | Yes |
| Linux x86_64 | Yes | Yes |
| Linux ARM64 | Yes | Yes |
| Windows x86_64 | Yes | Yes |
#### Intel Mac Workaround
The default FastEmbed provider uses ONNX Runtime, which dropped Intel Mac (x86_64) wheels starting in v1.24. Intel Mac users have two options:
**Option 1: Use OpenAI embeddings (recommended)**
Install only the OpenAI dependency manually — no ONNX Runtime or FastEmbed needed:
```bash
pip install openai sqlite-vec
export BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED=true
export BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER=openai
export OPENAI_API_KEY=sk-...
```
**Option 2: Pin an older ONNX Runtime**
FastEmbed's ONNX Runtime dependency is unpinned, so you can constrain it to an older version that still ships Intel Mac wheels by passing both requirements in the same install command:
```bash
pip install 'basic-memory[semantic]' 'onnxruntime<1.24'
```
## Quick Start
1. Install semantic extras:
```bash
pip install 'basic-memory[semantic]'
```
2. Enable semantic search:
```bash
export BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED=true
```
3. Build vector embeddings for your existing content:
```bash
bm reindex --embeddings
```
4. Search using semantic modes:
```python
# Pure vector similarity
search_notes("login process", search_type="vector")
# Hybrid: combines FTS precision with vector recall (recommended)
search_notes("login process", search_type="hybrid")
# Traditional full-text search (still the default)
search_notes("login process", search_type="text")
```
## Configuration Reference
All settings are fields on `BasicMemoryConfig` and can be set via environment variables (prefixed with `BASIC_MEMORY_`).
| Config Field | Env Var | Default | Description |
|---|---|---|---|
| `semantic_search_enabled` | `BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED` | `false` | Enable semantic search. Required before vector/hybrid modes work. |
| `semantic_embedding_provider` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER` | `"fastembed"` | Embedding provider: `"fastembed"` (local) or `"openai"` (API). |
| `semantic_embedding_model` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_MODEL` | `"bge-small-en-v1.5"` | Model identifier. Auto-adjusted per provider if left at default. |
| `semantic_embedding_dimensions` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_DIMENSIONS` | Auto-detected | Vector dimensions. 384 for FastEmbed, 1536 for OpenAI. Override only if using a non-default model. |
| `semantic_embedding_batch_size` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_BATCH_SIZE` | `64` | Number of texts to embed per batch. |
| `semantic_vector_k` | `BASIC_MEMORY_SEMANTIC_VECTOR_K` | `100` | Candidate count for vector nearest-neighbour retrieval. Higher values improve recall at the cost of latency. |
## Embedding Providers
### FastEmbed (default)
FastEmbed runs entirely locally using ONNX models — no API key, no network calls, no cost.
- **Model**: `BAAI/bge-small-en-v1.5`
- **Dimensions**: 384
- **Tradeoff**: Smaller model, fast inference, good quality for most use cases
```bash
# Install semantic extras and enable
pip install 'basic-memory[semantic]'
export BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED=true
```
### OpenAI
Uses OpenAI's embeddings API for higher-dimensional vectors. Requires an API key.
- **Model**: `text-embedding-3-small`
- **Dimensions**: 1536
- **Tradeoff**: Higher quality embeddings, requires API calls and an OpenAI key
```bash
export BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED=true
export BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER=openai
export OPENAI_API_KEY=sk-...
```
When switching from FastEmbed to OpenAI (or vice versa), you must rebuild embeddings since the vector dimensions differ:
```bash
bm reindex --embeddings
```
## Search Modes
### `text` (default)
Full-text keyword search using FTS5 (SQLite) or tsvector (Postgres). Supports boolean operators (`AND`, `OR`, `NOT`), phrase matching, and prefix wildcards.
```python
search_notes("project AND planning", search_type="text")
```
This is the existing default and does not require semantic search to be enabled.
### `vector`
Pure semantic similarity search. Embeds your query and finds the nearest content vectors. Good for conceptual or paraphrase queries where exact keywords may not appear in the content.
```python
search_notes("how to speed up the app", search_type="vector")
```
Returns results ranked by cosine similarity. Individual observations and relations surface as first-class results, not collapsed into parent entities.
### `hybrid`
Combines FTS and vector results using reciprocal rank fusion (RRF). This is generally the best mode when you want both keyword precision and semantic recall.
```python
search_notes("authentication security", search_type="hybrid")
```
RRF merges the two ranked lists so that items appearing in both get a score boost, while items found by only one method still appear.
### When to Use Which
| Mode | Best For |
|---|---|
| `text` | Exact keyword matching, boolean queries, tag/category searches |
| `vector` | Conceptual queries, paraphrase matching, exploratory searches |
| `hybrid` | General-purpose search combining precision and recall |
## The Reindex Command
The `bm reindex` command rebuilds search indexes without dropping the database.
```bash
# Rebuild everything (FTS + embeddings if semantic is enabled)
bm reindex
# Only rebuild vector embeddings
bm reindex --embeddings
# Only rebuild the full-text search index
bm reindex --search
# Target a specific project
bm reindex -p my-project
```
### When You Need to Reindex
- **First enable**: After turning on `semantic_search_enabled` for the first time
- **Provider change**: After switching between `fastembed` and `openai`
- **Model change**: After changing `semantic_embedding_model`
- **Dimension change**: After changing `semantic_embedding_dimensions`
The reindex command shows progress with embedded/skipped/error counts:
```
Project: main
Building vector embeddings...
✓ Embeddings complete: 142 entities embedded, 0 skipped, 0 errors
Reindex complete!
```
## How It Works
### Chunking
Each entity in the search index is split into semantic chunks before embedding:
- **Headers**: Markdown headers (`#`, `##`, etc.) start new chunks
- **Bullets**: Each bullet item (`-`, `*`) becomes its own chunk for granular fact retrieval
- **Prose sections**: Non-bullet text is merged up to ~900 characters per chunk
- **Long sections**: Oversized content is split with ~120 character overlap to preserve context at boundaries
Each search index item type (entity, observation, relation) is chunked independently, so observations and relations are embeddable as discrete facts.
### Deduplication
Each chunk has a `source_hash` (SHA-256 of the chunk text). On re-sync, unchanged chunks skip re-embedding entirely. This makes incremental updates fast — only modified content triggers API calls or model inference.
### Hybrid Fusion
Hybrid search uses reciprocal rank fusion (RRF) to merge FTS and vector results:
1. Run FTS search to get keyword-ranked results
2. Run vector search to get similarity-ranked results
3. For each result, compute: `score = 1/(k + fts_rank) + 1/(k + vector_rank)` where `k = 60`
4. Sort by fused score
Items found by both methods get a natural score boost. Items found by only one method still appear but rank lower.
### Observation-Level Results
Vector and hybrid modes return individual observations and relations as first-class search results, not just parent entities. This means a search for "water temperature for brewing" can surface the specific observation about 205°F without returning the entire "Coffee Brewing Methods" entity.
## Database Backends
### SQLite (local)
- **Vector storage**: [sqlite-vec](https://github.com/asg017/sqlite-vec) virtual table
- **Table creation**: At runtime when semantic search is first used — no migration needed
- **Embedding table**: `search_vector_embeddings` using `vec0(embedding float[N])` where N is the configured dimensions
- **Chunk metadata**: `search_vector_chunks` table stores chunk text, keys, and source hashes
The sqlite-vec extension is loaded per-connection. Vector tables are created lazily on first use.
### Postgres (cloud)
- **Vector storage**: [pgvector](https://github.com/pgvector/pgvector) with HNSW indexing
- **Chunk metadata table**: Created via Alembic migration (`search_vector_chunks` with `BIGSERIAL` primary key)
- **Embedding table**: `search_vector_embeddings` created at runtime (dimension-dependent, same pattern as SQLite)
- **Index**: HNSW index on the embedding column for fast approximate nearest-neighbour queries
The Alembic migration creates the dimension-independent chunks table. The embeddings table and HNSW index are deferred to runtime because they depend on the configured vector dimensions.
-225
View File
@@ -1,225 +0,0 @@
# SPEC-LOCAL-PLUS-PUBLISH: Local+ Published Notes and Privacy Tiers
**Status:** Draft
**Date:** 2026-02-14
**Owner:** Basic Memory
## Summary
Add a paid Local+ feature that lets users publish selected notes to shareable URLs while keeping the
main knowledge base local-first. Use this as a product wedge for users who do not want full cloud
hosting but do want collaboration and distribution features.
This spec also captures a practical position on "zero knowledge" for Local+.
## Context
Basic Memory already has strong local-first primitives and optional cloud routing/sync. A recurring
request is:
- keep knowledge local by default,
- pay for selective value-add,
- share specific outputs externally.
Published Notes fits this model: explicit per-note opt-in, reversible, and easy to understand.
## Goals
1. Provide an Obsidian Publish-style sharing experience for selected notes.
2. Keep local markdown files as source of truth.
3. Make sharing compatible with current cloud/auth/billing primitives.
4. Define clear Local+ packaging that does not degrade OSS local workflows.
5. Document zero-knowledge constraints so product decisions are explicit.
## Non-Goals
1. Full hosted editing for all notes (Cloud Full remains separate).
2. Public website builder/CMS features.
3. Strict cryptographic zero-knowledge server processing for MCP/search in v1.
## Local+ Feature Catalog (Sellable)
Core Local+ candidates:
1. Published Notes (share URL, revoke, expiry, password).
2. Snapshot Time Machine (point-in-time restore for local projects).
3. Recovery Drill Reports (automated restore verification).
4. Device/API Key Governance (per-device keys, revocation, audit trail).
5. BYO Storage Orchestration (managed setup for user-owned object storage).
6. Semantic Boost Add-on (higher quality retrieval options while files remain source-of-truth).
Team-oriented add-ons:
1. Team-owned shared links and domain branding.
2. Role-based publish permissions.
3. Shared workspace policies for what can be published.
## Proposed MVP: Published Notes
### User Experience
Per note actions:
1. Publish.
2. Unpublish.
3. Copy URL.
4. Regenerate URL.
5. Set visibility and controls.
Controls:
1. Visibility: `unlisted` (default) or `public`.
2. Optional password gate.
3. Optional expiration datetime.
4. Optional "disable indexing" flag for public mode.
Behavior:
1. Source note remains local markdown.
2. Publish is explicit opt-in per note.
3. Unpublish removes public access immediately.
4. Republish creates a new URL token unless user chooses to keep current URL.
### URL Model
1. Unlisted share URL: high-entropy token path.
2. Public URL: slug path (optional, later phase).
3. Team plans can support custom domain mapping in later phase.
### Content Model
v1 published page includes:
1. Rendered markdown body.
2. Optional metadata (title, updated_at).
v1 excludes:
1. Full graph traversal expansion.
2. Related note auto-discovery on public pages.
### Sync Model
1. Local file remains canonical.
2. Publish stores a rendered snapshot plus metadata in cloud.
3. Update path:
- manual "update published version", or
- optional auto-update on note change (plan-gated).
## Architecture (v1)
### High-Level Flow
1. Client selects a note to publish.
2. Client sends publish request with note identifier and policy.
3. Service resolves note content (local sync artifact or explicit upload payload).
4. Service stores published artifact and returns share URL.
### Data Model
`published_notes`
1. `id` (uuid)
2. `tenant_id` or `workspace_id`
3. `project_id`
4. `entity_permalink` (or stable external_id)
5. `share_token` (hashed in DB)
6. `visibility` (`unlisted`|`public`)
7. `password_hash` (nullable)
8. `expires_at` (nullable)
9. `is_active`
10. `published_content` (rendered snapshot or reference)
11. `published_at`
12. `updated_at`
### API Shape (Draft)
1. `POST /api/published-notes`
2. `GET /api/published-notes`
3. `GET /api/published-notes/{id}`
4. `PATCH /api/published-notes/{id}`
5. `DELETE /api/published-notes/{id}` (unpublish)
6. `POST /api/published-notes/{id}/regenerate-url`
7. `GET /p/{token}` (public resolver)
### CLI Shape (Draft)
1. `bm cloud publish <identifier>`
2. `bm cloud publish list`
3. `bm cloud publish update <id>`
4. `bm cloud publish unpublish <id>`
5. `bm cloud publish rotate-url <id>`
### Security
1. Default to unlisted URLs.
2. Store only hashed share tokens.
3. Passwords hashed server-side.
4. Enforce expiration at request time.
5. Log publish/unpublish/rotate events for auditability.
## Packaging and Pricing Direction
Suggested split:
1. OSS Local: no publish URLs.
2. Local+ Solo: publish URLs + snapshots + recovery.
3. Local+ Team: solo features + team governance and branding.
4. Cloud Full: hosted app + full cloud workflows.
Key message:
"Keep everything local. Publish only what you choose."
## Rollout Plan
1. Phase 1: Unlisted publish URLs + unpublish + regenerate URL.
2. Phase 2: Password/expiry controls.
3. Phase 3: Auto-update on note change and basic analytics.
4. Phase 4: Team branding/domains/policies.
## Zero-Knowledge Position
### Strict Zero-Knowledge Definition
Strict zero-knowledge means the server cannot decrypt note content at all.
### Why This Conflicts with MCP and Search
If server cannot decrypt:
1. MCP tool execution against cloud content cannot read/write semantic content.
2. Full-text search cannot index plaintext content.
3. Semantic/vector search cannot generate or query embeddings on plaintext.
4. Server-side relation resolution and context building become severely limited.
This matches earlier findings: strict zero-knowledge materially handicaps MCP-driven behavior and
search quality.
### Viable Alternatives (Not Strict Zero-Knowledge)
1. Encryption at rest/in transit with server-side decrypt in trusted runtime.
- Preserves MCP/search quality.
- Not zero-knowledge cryptographically.
2. Client-side retrieval mode.
- Keep MCP/search local; cloud is sync/share/backup relay.
- Best for privacy-first users.
- Requires local agent availability for advanced retrieval.
3. Limited encrypted indexing.
- Blind indexes for exact keywords only.
- No high-quality semantic search.
- Usually poor UX for natural-language memory recall.
### Recommendation
For Local+:
1. Do not promise strict zero-knowledge for cloud MCP/search paths.
2. Offer a privacy-first local mode where advanced retrieval stays local.
3. Clearly label tradeoffs:
- "Local private mode" (best privacy, best local retrieval).
- "Cloud-assisted mode" (best cross-device/MCP consistency, trusted-runtime decrypt).
This keeps messaging honest and avoids repeating the known incompatibility.
-365
View File
@@ -1,365 +0,0 @@
# SPEC-SCHEMA-IMPL: Schema System Implementation Plan
**Status:** Draft
**Created:** 2025-02-06
**Branch:** `feature/schema-system`
**Depends on:** [SPEC-SCHEMA](SPEC-SCHEMA.md)
## Overview
Implementation plan for the Basic Memory Schema System. The system is entirely programmatic —
no LLM agent runtime or API key required. The LLM already in the user's session (Claude Code,
Claude Desktop, etc.) provides the intelligence layer by reading schema notes via existing
MCP tools.
## Architecture
```
┌─────────────────────────────────────────────────┐
│ Entry Points │
│ CLI (bm schema ...) │ MCP (schema_validate) │
└──────────┬────────────┴──────────┬──────────────┘
│ │
▼ ▼
┌─────────────────────────────────────────────────┐
│ Schema Service Layer │
│ resolve_schema · validate · infer · diff │
└──────────┬────────────────────────┬──────────────┘
│ │
▼ ▼
┌──────────────────────┐ ┌────────────────────────┐
│ Picoschema Parser │ │ Note/Entity Access │
│ YAML → SchemaModel │ │ (existing repository) │
└──────────────────────┘ └────────────────────────┘
```
No new database tables. Schemas are notes with `type: schema` — they're already indexed.
Validation reads observations and relations from existing data.
## Components
### 1. Picoschema Parser
**Location:** `src/basic_memory/schema/parser.py`
Parses Picoschema YAML into an internal representation.
```python
@dataclass
class SchemaField:
name: str
type: str # string, integer, number, boolean, any, or EntityName
required: bool # True unless field name ends with ?
is_array: bool # True if (array) notation
is_enum: bool # True if (enum) notation
enum_values: list[str] # Populated for enums
description: str | None # Text after comma
is_entity_ref: bool # True if type is capitalized (entity reference)
children: list[SchemaField] # For (object) types
@dataclass
class SchemaDefinition:
entity: str # The entity type this schema describes
version: int # Schema version
fields: list[SchemaField] # Parsed fields
validation_mode: str # "warn" | "strict" | "off"
def parse_picoschema(yaml_dict: dict) -> list[SchemaField]:
"""Parse a Picoschema YAML dict into a list of SchemaField objects."""
def parse_schema_note(frontmatter: dict) -> SchemaDefinition:
"""Parse a full schema note's frontmatter into a SchemaDefinition."""
```
**Input/Output:**
```yaml
# Input (YAML dict from frontmatter)
schema:
name: string, full name
role?: string, job title
works_at?: Organization, employer
expertise?(array): string, areas of knowledge
```
```python
# Output
[
SchemaField(name="name", type="string", required=True, description="full name", ...),
SchemaField(name="role", type="string", required=False, description="job title", ...),
SchemaField(name="works_at", type="Organization", required=False, is_entity_ref=True, ...),
SchemaField(name="expertise", type="string", required=False, is_array=True, ...),
]
```
### 2. Schema Resolver
**Location:** `src/basic_memory/schema/resolver.py`
Finds the applicable schema for a note using the resolution order.
```python
async def resolve_schema(
note_frontmatter: dict,
search_fn: Callable, # injected search capability
) -> SchemaDefinition | None:
"""Resolve schema for a note.
Resolution order:
1. Inline schema (frontmatter['schema'] is a dict)
2. Explicit reference (frontmatter['schema'] is a string)
3. Implicit by type (frontmatter['type'] → schema note with matching entity)
4. No schema (returns None)
"""
```
### 3. Schema Validator
**Location:** `src/basic_memory/schema/validator.py`
Validates a note's observations and relations against a resolved schema.
```python
@dataclass
class FieldResult:
field: SchemaField
status: str # "present" | "missing" | "type_mismatch"
values: list[str] # Matched observation values or relation targets
message: str | None # Human-readable detail
@dataclass
class ValidationResult:
note_identifier: str
schema_entity: str
passed: bool # True if no errors (warnings are OK)
field_results: list[FieldResult]
unmatched_observations: dict[str, int] # category → count
unmatched_relations: list[str] # relation types not in schema
warnings: list[str]
errors: list[str]
async def validate_note(
note: Note,
schema: SchemaDefinition,
) -> ValidationResult:
"""Validate a note against a schema definition.
Mapping rules:
- field: string → observation [field] exists
- field?(array): type → multiple [field] observations
- field?: EntityType → relation 'field [[...]]' exists
- field?(enum): [v] → observation [field] value ∈ enum values
"""
```
### 4. Schema Inference Engine
**Location:** `src/basic_memory/schema/inference.py`
Analyzes notes of a given type and suggests a schema based on usage frequency.
```python
@dataclass
class FieldFrequency:
name: str
source: str # "observation" | "relation"
count: int # notes containing this field
total: int # total notes analyzed
percentage: float
sample_values: list[str] # representative values
is_array: bool # True if typically appears multiple times per note
target_type: str | None # For relations, the most common target entity type
@dataclass
class InferenceResult:
entity_type: str
notes_analyzed: int
field_frequencies: list[FieldFrequency]
suggested_schema: dict # Ready-to-use Picoschema YAML dict
suggested_required: list[str]
suggested_optional: list[str]
excluded: list[str] # Below threshold
async def infer_schema(
entity_type: str,
notes: list[Note],
required_threshold: float = 0.95, # 95%+ = required
optional_threshold: float = 0.25, # 25%+ = optional
) -> InferenceResult:
"""Analyze notes and suggest a Picoschema definition."""
```
### 5. Schema Diff
**Location:** `src/basic_memory/schema/diff.py`
Compares current note usage against an existing schema definition.
```python
@dataclass
class SchemaDrift:
new_fields: list[FieldFrequency] # Fields not in schema but common in notes
dropped_fields: list[FieldFrequency] # Fields in schema but rare in notes
cardinality_changes: list[str] # one → many or many → one
type_mismatches: list[str] # observation values don't match declared type
async def diff_schema(
schema: SchemaDefinition,
notes: list[Note],
) -> SchemaDrift:
"""Compare a schema against actual note usage to detect drift."""
```
## Entry Points
### CLI Commands
**Location:** `src/basic_memory/cli/schema.py`
```python
import typer
schema_app = typer.Typer(name="schema", help="Schema management commands")
@schema_app.command()
async def validate(
target: str = typer.Argument(None, help="Note path or entity type"),
strict: bool = typer.Option(False, help="Override to strict mode"),
):
"""Validate notes against their schemas."""
@schema_app.command()
async def infer(
entity_type: str = typer.Argument(..., help="Entity type to analyze"),
threshold: float = typer.Option(0.25, help="Minimum frequency for optional fields"),
save: bool = typer.Option(False, help="Save to schema/ directory"),
):
"""Infer schema from existing notes of a type."""
@schema_app.command()
async def diff(
entity_type: str = typer.Argument(..., help="Entity type to diff"),
):
"""Show drift between schema and actual usage."""
```
Registered as subcommand: `bm schema validate`, `bm schema infer`, `bm schema diff`.
### MCP Tools
**Location:** `src/basic_memory/mcp/tools/schema.py`
```python
@mcp_tool
async def schema_validate(
entity_type: str | None = None,
identifier: str | None = None,
project: str | None = None,
) -> str:
"""Validate notes against their resolved schema."""
@mcp_tool
async def schema_infer(
entity_type: str,
threshold: float = 0.25,
project: str | None = None,
) -> str:
"""Analyze existing notes and suggest a schema definition."""
```
### API Endpoints
**Location:** `src/basic_memory/api/schema_router.py`
```python
router = APIRouter(prefix="/schema", tags=["schema"])
@router.post("/validate")
async def validate_schema(...) -> ValidationReport: ...
@router.post("/infer")
async def infer_schema(...) -> InferenceResult: ...
@router.get("/diff/{entity_type}")
async def diff_schema(...) -> SchemaDrift: ...
```
MCP tools call these endpoints via the typed client pattern (consistent with existing
architecture).
## Implementation Phases
### Phase 1: Parser + Resolver
Build the foundation — can parse Picoschema and find schemas for notes.
**Deliverables:**
- `schema/parser.py` — Picoschema YAML → `SchemaDefinition`
- `schema/resolver.py` — Resolution order (inline → explicit ref → implicit by type → none)
- Unit tests for all Picoschema syntax variations
- Unit tests for resolution order
**No external dependencies.** Pure Python parsing of YAML dicts. Can develop and test
in isolation.
### Phase 2: Validator
Connect schemas to notes and produce validation results.
**Deliverables:**
- `schema/validator.py` — Validate note observations/relations against schema fields
- API endpoint: `POST /schema/validate`
- MCP tool: `schema_validate`
- CLI command: `bm schema validate`
- Integration tests with real notes and schemas
**Depends on:** Phase 1 (parser + resolver)
### Phase 3: Inference
Analyze existing notes to suggest schemas.
**Deliverables:**
- `schema/inference.py` — Frequency analysis across notes of a type
- API endpoint: `POST /schema/infer`
- MCP tool: `schema_infer`
- CLI command: `bm schema infer`
- Option to save inferred schema as a note via `write_note`
**Depends on:** Phase 1 (parser for output format)
### Phase 4: Diff
Compare schemas against current usage.
**Deliverables:**
- `schema/diff.py` — Drift detection between schema and actual notes
- API endpoint: `GET /schema/diff/{entity_type}`
- CLI command: `bm schema diff`
**Depends on:** Phase 1 (parser), Phase 3 (inference, for frequency analysis)
## Testing Strategy
- **Unit tests** (`tests/schema/`): Parser edge cases, resolution logic, validation mapping,
inference thresholds
- **Integration tests** (`test-int/schema/`): End-to-end with real markdown files, schema notes
on disk, CLI invocation
- Coverage target: 100% (consistent with project standard)
## What This Does NOT Include
- No new database tables or migrations
- No new markdown syntax (schemas validate existing observations/relations)
- No LLM agent runtime or API key management
- No hook integration (deferred)
- No schema composition/inheritance (deferred)
- No OWL/RDF export (deferred)
- No built-in templates (deferred)
-462
View File
@@ -1,462 +0,0 @@
# SPEC-SCHEMA: Basic Memory Schema System
**Status:** Draft
**Created:** 2025-02-06
**Branch:** `feature/schema-system`
## Summary
A schema system for Basic Memory that uses [Picoschema](https://genkit.dev/docs/dotprompt/)
syntax in YAML frontmatter. Schemas validate notes against their existing observation/relation
structure — no new data model, no migration, just a declarative lens over what's already there.
## Core Principles
1. **Schemas are just notes** — A schema is a note with `type: schema`, lives anywhere
2. **Use prior art** — Picoschema syntax in YAML frontmatter, no custom notation
3. **Validation maps to existing format** — Observations and relations, not a parallel data model
4. **Validation is soft** — Warnings by default, not blocking errors
5. **Inference over prescription** — Schemas describe reality, emerge from usage
6. **No built-in agent** — Programmatic core; the LLM already in the session provides intelligence
## Picoschema Syntax
Picoschema is a compact schema notation from Google's Dotprompt that fits naturally in YAML
frontmatter.
### Supported Types
| Type | Description |
|------|-------------|
| `string` | Text value |
| `integer` | Whole number |
| `number` | Decimal number |
| `boolean` | True/false |
| `any` | Any scalar type |
| `EntityName` | Reference to another entity (capitalized = entity reference) |
### Syntax Rules
```yaml
schema:
name: string, full name # required field with description
email?: string, contact email # ? = optional
role?: string, job title
works_at?: Organization, employer # capitalized type = entity reference
tags?(array): string, categories # array of type
status?(enum): [active, inactive] # enum with allowed values
metadata?(object): # nested object
updated_at?: string
source?: string
```
- `field: type` — required field
- `field?: type` — optional field
- `field(array): type` — array of values
- `field?(enum): [values]` — enumeration
- `field?(object):` — nested object with sub-fields
- `, description` — description after comma
- `EntityName` as type (capitalized) — reference to another entity
## Schema-to-Note Mapping
Schemas validate against the existing Basic Memory note format. No new syntax for note
authors to learn.
### Mapping Rules
| Schema Declaration | Grounded In | Example Match |
|--------------------|-------------|---------------|
| `field: string` | Observation `[field] value` | `- [name] Paul Graham` |
| `field?(array): string` | Multiple `[field]` observations | `- [expertise] Lisp` (×N) |
| `field?: EntityType` | Relation `field [[Target]]` | `- works_at [[Y Combinator]]` |
| `field?(array): EntityType` | Multiple `field` relations | `- authored [[Book]]` (×N) |
| `tags` | Frontmatter `tags` array | `tags: [startups, essays]` |
| `field?(enum): [values]` | Observation `[field] value` where value ∈ set | `- [status] active` |
### Key Insight
Schemas don't introduce a new way to store data. They describe the patterns already present
in observations and relations. A note doesn't have to change how it's written — the schema
just says "a good Person note has a `[name]` observation and a `works_at` relation."
## Schema Definition
### As a Dedicated Schema Note
```yaml
# schema/Person.md
---
title: Person
type: schema
entity: Person
version: 1
schema:
name: string, full name
email?: string, contact email
role?: string, job title
works_at?: Organization, employer
expertise?(array): string, areas of knowledge
settings:
validation: warn # warn | strict | off
---
# Person
A human individual in the knowledge graph.
Any documentation about this entity type goes here as prose.
```
Schema notes are regular Basic Memory notes. They show up in search, can have their own
observations and relations, and can be organized in any folder (though `schema/` is
the suggested convention).
### Inline Schema in a Note
Notes can carry their own schema directly:
```yaml
# meetings/2024-01-15-standup.md
---
title: Team Standup 2024-01-15
type: meeting
schema:
attendees(array): string, who was there
decisions(array): string, what was decided
action_items(array): string, follow-ups
blockers?(array): string, anything stuck
---
# Team Standup 2024-01-15
## Observations
- [attendees] Paul
- [attendees] Sarah
- [decisions] Ship v2 by Friday
- [action_items] Paul to review PR #42
- [blockers] Waiting on API credentials
```
Good for one-off structured notes or prototyping a schema before extracting it.
### Explicit Schema Reference
A note can reference a schema by entity name or permalink:
```yaml
# projects/basic-memory.md
---
title: Basic Memory
schema: SoftwareProject # by entity name
---
# research/llm-memory-patterns.md
---
title: LLM Memory Patterns
schema: schema/research-project # by permalink
---
```
Use cases:
- Note's `type` differs from the schema it should validate against
- Multiple schema variants exist for the same domain
- Applying structure to existing notes without changing their type
## Schema Resolution
When validating a note, schemas resolve in priority order:
```
1. Inline schema → schema: { ... } (dict in frontmatter)
2. Explicit ref → schema: Person (string in frontmatter)
3. Implicit by type → type: Person (lookup schema note with entity: Person)
4. No schema → no validation (perfectly fine)
```
```python
async def resolve_schema(note: Note) -> Schema | None:
schema_value = note.frontmatter.get('schema')
# 1. Inline schema (dict)
if isinstance(schema_value, dict):
return parse_picoschema(schema_value)
# 2. Explicit reference (string)
if isinstance(schema_value, str):
schema_note = await find_schema_note(schema_value)
if schema_note:
return parse_picoschema(schema_note.frontmatter['schema'])
# 3. Implicit by type
note_type = note.frontmatter.get('type')
if note_type:
results = await search_notes(f"type:schema entity:{note_type}")
if results:
return parse_picoschema(results[0].frontmatter['schema'])
# 4. No schema
return None
```
## Validation
### Modes
Configured in the schema's `settings.validation`:
| Mode | Behavior |
|------|----------|
| `off` | No validation |
| `warn` | Warnings in output, doesn't block (default) |
| `strict` | Errors that block sync, for CI/CD enforcement |
### Validation Output
For a note missing required fields:
```
$ bm schema validate people/ada-lovelace.md
⚠ Person schema validation:
- Missing required field: name (expected [name] observation)
- Missing optional field: role
- Missing optional field: works_at (no relation found)
Unmatched observations: [fact] ×2, [born] ×1
Unmatched relations: collaborated_with
```
"Unmatched" items are informational — observations and relations the schema doesn't cover.
They're valid. Schemas are a subset, not a straitjacket.
### Batch Validation
```
$ bm schema validate Person
Validating 30 notes against Person schema...
✓ people/paul-graham.md — all fields present
✓ people/rich-hickey.md — all fields present
⚠ people/ada-lovelace.md — missing: name
⚠ people/alan-kay.md — missing: name, role
✓ people/linus-torvalds.md — all fields present
...
Summary: 22/30 valid, 8 warnings, 0 errors
```
## Emerging Schemas
### The Problem with Traditional Schemas
Most schema systems require: define schema → create conforming content → fight the schema
when reality doesn't match. This is backwards. Knowledge grows organically.
### The Basic Memory Approach
```
Write notes freely → Patterns emerge → Crystallize into schema → Validate future notes
```
### Schema Inference
Generate schemas from existing notes by analyzing observation and relation frequency:
```
$ bm schema infer Person
Analyzing 30 notes with type: Person...
Observations found:
[name] 30/30 100% → name: string
[role] 27/30 90% → role?: string
[fact] 25/30 83% (generic — no single field)
[expertise] 18/30 60% → expertise?(array): string
[email] 8/30 27% → email?: string
[born] 6/30 20% (below threshold)
Relations found:
works_at 22/30 73% → works_at?: Organization
authored 11/30 37% → authored?(array): string
Suggested schema:
name: string, full name
role?: string, job title
expertise?(array): string, areas of knowledge
email?: string, contact email
works_at?: Organization, employer
Save to schema/Person.md? [y/n]
```
Frequency thresholds:
- 100% present → required field
- 25%+ present → optional field
- Below 25% → excluded from suggestion (but noted)
### Schema Drift Detection
Track how usage patterns shift over time:
```
$ bm schema diff Person
Schema drift detected:
+ expertise: now in 81% of notes (was 12%)
- department: dropped to 3% of notes
~ works_at: cardinality changed (one → many)
Update schema? [y/n/review]
```
## LLM Integration (AI Guidance)
No agent runtime or API key required. The LLM already in the session uses schemas as
context for note creation.
### Flow
1. User asks LLM to "write a note about Rich Hickey"
2. LLM determines `type: Person` is appropriate
3. LLM calls `search_notes("type:schema entity:Person")` → finds schema
4. LLM reads schema fields: required `name`, optional `role`, `works_at`, `expertise`
5. LLM calls `write_note` with observations and relations that satisfy the schema
The schema acts as a creation template. The LLM knows what a "complete" note looks like
without any custom agent infrastructure.
### MCP Tools
```python
@mcp_tool
async def schema_validate(
entity_type: str | None = None,
identifier: str | None = None,
project: str | None = None,
) -> ValidationReport:
"""Validate notes against their resolved schema.
Validates a specific note (by identifier) or all notes of a given type.
Returns warnings/errors based on the schema's validation mode.
"""
@mcp_tool
async def schema_infer(
entity_type: str,
threshold: float = 0.25,
project: str | None = None,
) -> SuggestedSchema:
"""Analyze existing notes and suggest a schema definition.
Examines observation categories and relation types across all notes
of the given type. Returns frequency analysis and suggested Picoschema.
"""
```
## CLI Commands
```bash
# Validate a specific note
bm schema validate people/ada-lovelace.md
# Validate all notes of a type
bm schema validate Person
# Validate everything with a schema
bm schema validate
# Infer schema from existing notes
bm schema infer Person
# Show schema drift from current definition
bm schema diff Person
# List all schema notes
bm search "type:schema"
```
## Examples
### Complete Person Workflow
**Schema:**
```yaml
# schema/Person.md
---
title: Person
type: schema
entity: Person
version: 1
schema:
name: string, full name
role?: string, job title or position
works_at?: Organization, employer
expertise?(array): string, areas of knowledge
email?: string, contact email
settings:
validation: warn
---
# Person
A human individual in the knowledge graph.
```
**Valid note:**
```yaml
# people/paul-graham.md
---
title: Paul Graham
type: Person
tags: [startups, essays, lisp]
---
# Paul Graham
## Observations
- [name] Paul Graham
- [role] Essayist and investor
- [expertise] Startups
- [expertise] Lisp
- [expertise] Essay writing
- [fact] Created Viaweb, the first web app
## Relations
- works_at [[Y Combinator]]
- authored [[Hackers and Painters]]
```
**Note with warnings:**
```yaml
# people/ada-lovelace.md
---
title: Ada Lovelace
type: Person
---
# Ada Lovelace
## Observations
- [fact] Wrote the first computer program
- [born] 1815
## Relations
- collaborated_with [[Charles Babbage]]
```
Validation: warns about missing required `[name]` observation. Everything else is optional
or unmatched (which is fine).
## Future Considerations (Deferred)
These are interesting but out of scope for the initial implementation:
- **Multiple schema inheritance** — `schema: [Person, Author]`
- **Hook integration** — Pre-write validation via the hooks system
- **OWL/RDF export** — `bm schema export --format owl`
- **SPARQL queries** — Schema-aware graph queries
- **Built-in templates** — `bm schema use gtd`, `bm schema use zettelkasten`
- **Schema versioning/migration** — Tracking breaking changes across versions
-28
View File
@@ -1,28 +0,0 @@
## Coverage policy (practical 100%)
Basic Memorys test suite intentionally mixes:
- unit tests (fast, deterministic)
- integration tests (real filesystem + real DB via `test-int/`)
To keep the default CI signal **stable and meaningful**, the default `pytest` coverage report targets **core library logic** and **excludes** a small set of modules that are either:
- highly environment-dependent (OS/DB tuning)
- inherently interactive (CLI)
- background-task orchestration (watchers/sync runners)
### What's excluded (and why)
Coverage excludes are configured in `pyproject.toml` under `[tool.coverage.report].omit`.
Current exclusions include:
- `src/basic_memory/cli/**`: interactive wrappers; behavior is validated via higher-level tests and smoke tests.
- `src/basic_memory/db.py`: platform/backend tuning paths (SQLite/Postgres/Windows), covered by integration tests and targeted runs.
- `src/basic_memory/services/initialization.py`: startup orchestration/background tasks; covered indirectly by app/MCP entrypoints.
- `src/basic_memory/sync/sync_service.py`: heavy filesystem↔DB integration; validated in integration suite (not enforced in unit coverage).
### Recommended additional runs
If you want extra confidence locally/CI:
- **Postgres backend**: run tests with `BASIC_MEMORY_TEST_POSTGRES=1`.
- **Strict backend-complete coverage**: run coverage on SQLite + Postgres and combine the results (recommended).
+29 -210
View File
@@ -2,176 +2,25 @@
# Install dependencies
install:
uv sync --extra semantic
@echo ""
@echo "💡 Remember to activate the virtual environment by running: source .venv/bin/activate"
pip install -e ".[dev]"
# ==============================================================================
# DATABASE BACKEND TESTING
# ==============================================================================
# Basic Memory supports dual database backends (SQLite and Postgres).
# By default, tests run against SQLite (fast, no dependencies).
# Set BASIC_MEMORY_TEST_POSTGRES=1 to run against Postgres (uses testcontainers).
#
# Quick Start:
# just test # Run all tests against SQLite (default)
# just test-sqlite # Run all tests against SQLite
# just test-postgres # Run all tests against Postgres (testcontainers)
# just test-unit-sqlite # Run unit tests against SQLite
# just test-unit-postgres # Run unit tests against Postgres
# just test-int-sqlite # Run integration tests against SQLite
# just test-int-postgres # Run integration tests against Postgres
#
# CI runs both in parallel for faster feedback.
# ==============================================================================
# Run unit tests in parallel
test-unit:
uv run pytest -p pytest_mock -v -n auto
# Run all tests against SQLite and Postgres
test: test-sqlite test-postgres
# Run integration tests in parallel
test-int:
uv run pytest -p pytest_mock -v --no-cov -n auto test-int
# Run all tests against SQLite
test-sqlite: test-unit-sqlite test-int-sqlite
# Run all tests against Postgres (uses testcontainers)
test-postgres: test-unit-postgres test-int-postgres
# Run unit tests against SQLite
test-unit-sqlite:
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov tests
# Run unit tests against Postgres
test-unit-postgres:
BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov tests
# Run integration tests against SQLite (excludes semantic benchmarks — use just test-semantic)
test-int-sqlite:
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m "not semantic" test-int
# Run integration tests against Postgres
# Note: Uses timeout due to FastMCP Client + asyncpg cleanup hang (tests pass, process hangs on exit)
# See: https://github.com/jlowin/fastmcp/issues/1311
test-int-postgres:
#!/usr/bin/env bash
set -euo pipefail
# Use gtimeout (macOS/Homebrew) or timeout (Linux)
TIMEOUT_CMD=$(command -v gtimeout || command -v timeout || echo "")
if [[ -n "$TIMEOUT_CMD" ]]; then
$TIMEOUT_CMD --signal=KILL 600 bash -c 'BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov -m "not semantic" test-int' || test $? -eq 137
else
echo "⚠️ No timeout command found, running without timeout..."
BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov -m "not semantic" test-int
fi
# Run tests impacted by recent changes (requires pytest-testmon)
testmon *args:
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov --testmon --testmon-forceselect {{args}}
# Run MCP smoke test (fast end-to-end loop)
test-smoke:
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m smoke test-int/mcp/test_smoke_integration.py
# Fast local loop: lint, format, typecheck, impacted tests
fast-check:
just fix
just format
just typecheck
just testmon
just test-smoke
# Reset Postgres test database (drops and recreates schema)
# Useful when Alembic migration state gets out of sync during development
# Uses credentials from docker-compose-postgres.yml
postgres-reset:
docker exec basic-memory-postgres psql -U ${POSTGRES_USER:-basic_memory_user} -d ${POSTGRES_TEST_DB:-basic_memory_test} -c "DROP SCHEMA public CASCADE; CREATE SCHEMA public;"
@echo "✅ Postgres test database reset"
# Run Alembic migrations manually against Postgres test database
# Useful for debugging migration issues
# Uses credentials from docker-compose-postgres.yml (can override with env vars)
postgres-migrate:
@cd src/basic_memory/alembic && \
BASIC_MEMORY_DATABASE_BACKEND=postgres \
BASIC_MEMORY_DATABASE_URL=${POSTGRES_TEST_URL:-postgresql+asyncpg://basic_memory_user:dev_password@localhost:5433/basic_memory_test} \
uv run alembic upgrade head
@echo "✅ Migrations applied to Postgres test database"
# Run Windows-specific tests only (only works on Windows platform)
# These tests verify Windows-specific database optimizations (locking mode, NullPool)
# Will be skipped automatically on non-Windows platforms
test-windows:
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m windows tests test-int
# Run benchmark tests only (performance testing)
# These are slow tests that measure sync performance with various file counts
# Excluded from default test runs to keep CI fast
test-benchmark:
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m benchmark tests test-int
# Run semantic search quality benchmarks (all combos)
test-semantic:
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m semantic test-int/semantic/
# Run semantic benchmarks with JSON artifact output, then show report
test-semantic-report:
BASIC_MEMORY_ENV=test BASIC_MEMORY_BENCHMARK_OUTPUT=.benchmarks/semantic-quality.jsonl uv run pytest -p pytest_mock -v -s --no-cov -m semantic test-int/semantic/
uv run python test-int/semantic/report.py .benchmarks/semantic-quality.jsonl
# Run semantic benchmarks (Postgres combos only)
test-semantic-postgres:
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m semantic -k postgres test-int/semantic/
# View semantic benchmark results (rich formatted table)
# Usage: just semantic-report [--filter-combo sqlite] [--filter-suite paraphrase] [--sort-by avg_latency_ms]
semantic-report *args:
uv run python test-int/semantic/report.py .benchmarks/semantic-quality.jsonl {{args}}
# Compare two search benchmark JSONL outputs
# Usage:
# just benchmark-compare .benchmarks/search-baseline.jsonl .benchmarks/search-candidate.jsonl
# just benchmark-compare .benchmarks/search-baseline.jsonl .benchmarks/search-candidate.jsonl --format markdown --show-missing
benchmark-compare baseline candidate *args:
uv run python test-int/compare_search_benchmarks.py "{{baseline}}" "{{candidate}}" --format table {{args}}
# Run all tests including Windows, Postgres, and Benchmarks (for CI/comprehensive testing)
# Use this before releasing to ensure everything works across all backends and platforms
test-all:
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov tests test-int
# Generate HTML coverage report
coverage:
#!/usr/bin/env bash
set -euo pipefail
uv run coverage erase
echo "🔎 Coverage (SQLite)..."
BASIC_MEMORY_ENV=test uv run coverage run --source=basic_memory -m pytest -p pytest_mock -v --no-cov tests test-int
echo "🔎 Coverage (Postgres via testcontainers)..."
# Note: Uses timeout due to FastMCP Client + asyncpg cleanup hang (tests pass, process hangs on exit)
# See: https://github.com/jlowin/fastmcp/issues/1311
TIMEOUT_CMD=$(command -v gtimeout || command -v timeout || echo "")
if [[ -n "$TIMEOUT_CMD" ]]; then
$TIMEOUT_CMD --signal=KILL 600 bash -c 'BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 uv run coverage run --source=basic_memory -m pytest -p pytest_mock -v --no-cov -m postgres tests test-int' || test $? -eq 137
else
echo "⚠️ No timeout command found, running without timeout..."
BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 uv run coverage run --source=basic_memory -m pytest -p pytest_mock -v --no-cov -m postgres tests test-int
fi
echo "🧩 Combining coverage data..."
uv run coverage combine
uv run coverage report -m
uv run coverage html
echo "Coverage report generated in htmlcov/index.html"
# Lint and fix code (calls fix)
lint: fix
# Run all tests
test: test-unit test-int
# Lint and fix code
fix:
uv run ruff check --fix --unsafe-fixes src tests test-int
lint:
uv run ruff check . --fix
# Type check code
typecheck:
type-check:
uv run pyright
# Clean build artifacts and cache files
@@ -189,28 +38,21 @@ format:
run-inspector:
npx @modelcontextprotocol/inspector
# Run doctor checks in an isolated temp home/config
doctor:
#!/usr/bin/env bash
set -euo pipefail
TMP_HOME=$(mktemp -d)
TMP_CONFIG=$(mktemp -d)
HOME="$TMP_HOME" \
BASIC_MEMORY_ENV=test \
BASIC_MEMORY_HOME="$TMP_HOME/basic-memory" \
BASIC_MEMORY_CONFIG_DIR="$TMP_CONFIG" \
./.venv/bin/python -m basic_memory.cli.main doctor --local
# Build macOS installer
installer-mac:
cd installer && chmod +x make_icons.sh && ./make_icons.sh
cd installer && uv run python setup.py bdist_mac
# Build Windows installer
installer-win:
cd installer && uv run python setup.py bdist_win32
# Update all dependencies to latest versions
update-deps:
uv sync --upgrade
# Run all code quality checks and tests
check: lint format typecheck test
# Run all code quality checks and all test suites, including semantic benchmarks
check-all: lint format typecheck test test-semantic
check: lint format type-check test
# Generate Alembic migration with descriptive message
migration message:
@@ -251,22 +93,16 @@ release version:
fi
# Run quality checks
echo "🔍 Running lint checks..."
just lint
just typecheck
echo "🔍 Running quality checks..."
just check
# Update version in __init__.py
echo "📝 Updating version in __init__.py..."
sed -i.bak "s/__version__ = \".*\"/__version__ = \"$VERSION_NUM\"/" src/basic_memory/__init__.py
rm -f src/basic_memory/__init__.py.bak
# Update version in server.json (MCP registry metadata)
echo "📝 Updating version in server.json..."
sed -i.bak "s/\"version\": \"[^\"]*\"/\"version\": \"$VERSION_NUM\"/g" server.json
rm -f server.json.bak
# Commit version update
git add src/basic_memory/__init__.py server.json
git add src/basic_memory/__init__.py
git commit -m "chore: update version to $VERSION_NUM for {{version}} release"
# Create and push tag
@@ -280,12 +116,6 @@ release version:
echo "✅ Release {{version}} created successfully!"
echo "📦 GitHub Actions will build and publish to PyPI"
echo "🔗 Monitor at: https://github.com/basicmachines-co/basic-memory/actions"
echo ""
echo "📝 REMINDER: Post-release tasks:"
echo " 1. docs.basicmemory.com - Add release notes to src/pages/latest-releases.mdx"
echo " 2. basicmachines.co - Update version in src/components/sections/hero.tsx"
echo " 3. MCP Registry - Run: mcp-publisher publish"
echo " See: .claude/commands/release/release.md for detailed instructions"
# Create a beta release (e.g., just beta v0.13.2b1)
beta version:
@@ -322,22 +152,16 @@ beta version:
fi
# Run quality checks
echo "🔍 Running lint checks..."
just lint
just typecheck
echo "🔍 Running quality checks..."
just check
# Update version in __init__.py
echo "📝 Updating version in __init__.py..."
sed -i.bak "s/__version__ = \".*\"/__version__ = \"$VERSION_NUM\"/" src/basic_memory/__init__.py
rm -f src/basic_memory/__init__.py.bak
# Update version in server.json (MCP registry metadata)
echo "📝 Updating version in server.json..."
sed -i.bak "s/\"version\": \"[^\"]*\"/\"version\": \"$VERSION_NUM\"/g" server.json
rm -f server.json.bak
# Commit version update
git add src/basic_memory/__init__.py server.json
git add src/basic_memory/__init__.py
git commit -m "chore: update version to $VERSION_NUM for {{version}} beta release"
# Create and push tag
@@ -352,12 +176,7 @@ beta version:
echo "📦 GitHub Actions will build and publish to PyPI as pre-release"
echo "🔗 Monitor at: https://github.com/basicmachines-co/basic-memory/actions"
echo "📥 Install with: uv tool install basic-memory --pre"
echo ""
echo "📝 REMINDER: For stable releases, update documentation sites:"
echo " 1. docs.basicmemory.com - Add release notes to src/pages/latest-releases.mdx"
echo " 2. basicmachines.co - Update version in src/components/sections/hero.tsx"
echo " See: .claude/commands/release/release.md for detailed instructions"
# List all available recipes
default:
@just --list
@just --list
+378
View File
@@ -0,0 +1,378 @@
{"type":"entity","name":"Paul","entityType":"person","observations":["Software developer combining DIY ethics, Free Software principles, and theoretical computer science","Created the Basic Machines project","Values authentic exchange of ideas","Approaches AI interaction with emphasis on genuine technical discussion","Comfortable with uncertainty and open dialogue","Balances practical implementation with broader implications"]}
{"type":"entity","name":"Basic_Machines","entityType":"project","observations":["Local-first knowledge management system","Combines filesystem durability with graph-based knowledge representation","Focuses on enhancing human agency and understanding","Synthesizes DIY ethics, Free Software philosophy, and theoretical computer science","Current focus includes basic-memory system"]}
{"type":"entity","name":"basic-memory","entityType":"software_system","observations":["A core component of Basic Machines","Local-first knowledge management system","Combines filesystem persistence with graph-based knowledge representation","Being implemented collaboratively by Paul and Claude"]}
{"type":"entity","name":"basic-memory_implementation_patterns","entityType":"technical_patterns","observations":["Filesystem is source of truth - all changes write to files first","Clean separation of concerns between models (SQLAlchemy), schemas (Pydantic), and services","Repository pattern for database access","Service layer handling business logic and coordination","Atomic file operations using temporary files for safety","Clear error handling hierarchy with specific error types","Comprehensive test coverage with pytest and fixtures","Async/await used throughout the codebase","Validation using Pydantic models with custom validators"]}
{"type":"entity","name":"fileio_module","entityType":"code_module","observations":["Extracted from EntityService to handle all file operations","Provides read_entity_file, write_entity_file, and delete_entity_file functions","Handles markdown parsing and formatting","Implements atomic file operations","Provides consistent error handling","Enables reuse across services"]}
{"type":"entity","name":"entity_service","entityType":"code_module","observations":["Manages entities in both filesystem and database","Uses fileio module for file operations","Maintains database index of entities","Handles entity creation, retrieval, and deletion","Follows 'filesystem is source of truth' principle","Coordinates with observation service for full entity management"]}
{"type":"entity","name":"observation_service","entityType":"code_module","observations":["Manages observations within entity files","Provides database indexing for efficient observation queries","Works with complete Entity objects rather than IDs","Handles observation addition and search","Maintains consistency between files and database","Under development for update/remove operations"]}
{"type":"entity","name":"observation_management","entityType":"design_challenge","observations":["Key challenge: maintaining observation state across files and database","Exploring bulk update approach - treating all observations as a unit","Considering tracked observations with markdown comments for IDs","Investigating diff-based approach for observation-level changes","Evaluating position-based management without explicit IDs","Trade-offs between implementation complexity and markdown readability"]}
{"type":"entity","name":"testing_infrastructure","entityType":"technical_patterns","observations":["Uses pytest with async support via pytest-asyncio","In-memory SQLite database for test isolation","Temporary directories for file operation testing","Comprehensive fixture system for test setup","Tests organized by component (entity, observation, etc)","Covers happy path, error cases, and edge cases","Uses monkeypatch for mocking dependencies","Clear separation between arrange, act, assert sections","Uses in-memory SQLite database for test isolation","Comprehensive fixture system for test data setup","Proper async test handling with pytest-asyncio"]}
{"type":"entity","name":"test_categories","entityType":"test_suite","observations":["Happy path tests verify core functionality","Error path tests ensure proper error handling","Edge cases test special characters and long content","File operation tests verify atomic writes and rollbacks","Database sync tests verify index consistency","Recovery tests for rebuild operations","Punted on concurrent operation tests due to session management complexity"]}
{"type":"entity","name":"completed_work","entityType":"project_milestone","observations":["Extracted file operations to fileio.py module","Updated EntityService to use fileio functions","Implemented initial ObservationService","Created comprehensive test suite","Established clear project patterns and principles","Set up basic database schema with SQLAlchemy","Created Pydantic models for validation"]}
{"type":"entity","name":"future_work","entityType":"project_tasks","observations":["Implement observation updates/removals","Design proper session management for concurrent operations","Update EntityService tests for new fileio module","Add more sophisticated search functionality","Handle markdown formatting edge cases","Consider versioning for file changes","Implement proper backup strategy"]}
{"type":"entity","name":"design_decisions","entityType":"technical_decisions","observations":["Filesystem as source of truth over database","Markdown format for human readability and editing","Atomic file operations for safety","SQLite + SQLAlchemy for proven reliability","Pydantic for validation and ID generation","Async/await for better scalability","Clear separation between files and database roles","Explicit error hierarchies for better handling"]}
{"type":"entity","name":"concurrency_considerations","entityType":"technical_challenge","observations":["SQLAlchemy session management in async context","File operation atomicity","Transaction isolation levels","Potential for conflicting updates","Need for proper session lifecycle","Possibility of file system race conditions","Database lock management"]}
{"type":"entity","name":"observation_update_approaches","entityType":"design_alternatives","observations":["Each approach trades off between simplicity, efficiency, and robustness","Four main approaches considered: bulk update, tracked IDs, diff-based, and position-based","Discussion revealed importance of human readability in file format","Consideration of manual editing workflows key to design","File system as source of truth principle guides tradeoffs"]}
{"type":"entity","name":"bulk_update_approach","entityType":"design_option","observations":["Update all observations at once in a single operation","Simpler file operations - just rewrite the whole list","No need for observation matching or IDs","Very consistent with source of truth principle","Less efficient for small changes","May have concurrency implications","Simplest implementation option"]}
{"type":"entity","name":"tracked_observations_approach","entityType":"design_option","observations":["Use markdown comments to store observation IDs","Enables precise updates and deletes","IDs stored as HTML comments in markdown","More complex markdown parsing required","IDs visible in raw markdown files","Balances tracking with readability"]}
{"type":"entity","name":"diff_based_approach","entityType":"design_option","observations":["Implement observation-aware diffing","Track changes at observation level","More efficient for updates","Preserves manual edits and changes","More complex implementation needed","Must handle merge conflicts","Most sophisticated option considered"]}
{"type":"entity","name":"position_based_approach","entityType":"design_option","observations":["Track observations by position/order","No explicit IDs needed","Cleanest markdown format","Order changes could break references","Difficult to handle concurrent edits","Most fragile option considered"]}
{"type":"entity","name":"tasks_and_progress","entityType":"project_tracking","observations":["Current focus on observation management implementation","Completed core file operations extraction","Completed EntityService updates","Completed initial ObservationService","Basic test coverage in place","Future work includes concurrent operations","Future work includes search improvements","Need to handle markdown edge cases"]}
{"type":"entity","name":"error_handling_patterns","entityType":"technical_patterns","observations":["Custom exception hierarchy with ServiceError base","Specific error types (FileOperationError, DatabaseSyncError, etc)","Clear separation between file and database errors","Error propagation patterns established","Focus on actionable error messages","Error handling at appropriate levels"]}
{"type":"entity","name":"data_models","entityType":"technical_implementation","observations":["SQLAlchemy models for database structure","Pydantic schemas for API/service layer","Entity model with UUID-based IDs","Observation model with entity relationships","UTCDateTime custom type for timestamps","Automatic ID generation in Pydantic models","Strict validation rules"]}
{"type":"entity","name":"markdown_format","entityType":"file_format","observations":["Simple, human-readable format","Entity name as H1 header","Metadata in key-value format","Observations as bullet points","Atomic file operations for updates","Designed for manual editing","No hidden metadata in main content"]}
{"type":"entity","name":"test_driven_development","entityType":"development_pattern","observations":["Tests revealed need for atomic file operations","Error cases drove error hierarchy design","Edge cases informed validation rules","Test fixtures shaped service interfaces","File operations extracted due to test patterns","Concurrent test issues revealed session management needs"]}
{"type":"entity","name":"architecture_evolution","entityType":"design_process","observations":["Started with simple EntityService implementation","Circular dependency between Entity and Observation services revealed design flaw","Extracted file operations to separate module","Moved to passing Entity objects rather than IDs","Improved separation of concerns through iterations","File operations became reusable across services","Database became true 'index' rather than source of truth"]}
{"type":"entity","name":"validation_patterns","entityType":"technical_patterns","observations":["Pydantic models provide schema validation","Automatic ID generation if not provided","Database constraints via SQLAlchemy","Runtime checks in services","Markdown format validation","Error handling for invalid states"]}
{"type":"entity","name":"markdown_examples","entityType":"documentation","observations":["Example of basic entity:\n# Entity Name\ntype: entity_type\n\n## Observations\n- First observation\n- Second observation","Example with special characters:\n# Test & Entity!\ntype: test\n\n## Observations\n- Test & observation with @#$% special chars!","Format ensures human readability:\n# Basic Machines\ntype: project\n\n## Observations\n- Local-first knowledge management system\n- Combines filesystem durability with graph-based knowledge representation","Future consideration for observation IDs:\n# Entity Name\ntype: entity_type\n\n## Observations\n- <!-- obs-id: abc123 -->\n This is an observation with ID"]}
{"type":"entity","name":"markdown_parsing_rules","entityType":"technical_implementation","observations":["H1 header contains entity name","Metadata uses key: value format","Observations section marked by H2 header","Each observation is a markdown list item","Blank lines separate sections","Special characters allowed in content","No restrictions on observation content"]}
{"type":"entity","name":"schema_definitions","entityType":"technical_documentation","observations":["SQLAlchemy Entity model:\nclass Entity(Base):\n id: str (primary key)\n name: str (unique)\n entity_type: str\n created_at: datetime\n updated_at: datetime","SQLAlchemy Observation model:\nclass Observation(Base):\n id: str (primary key)\n entity_id: str (foreign key)\n content: str\n created_at: datetime\n context: Optional[str]","Pydantic Entity schema:\nclass Entity(BaseModel):\n id: str\n name: str\n entity_type: str\n observations: List[Observation]"]}
{"type":"entity","name":"test_evolution","entityType":"development_history","observations":["Started with basic Entity CRUD tests","Added filesystem verification to all tests","Developed concurrent operation tests (later removed)","Edge case tests drove better error handling","Test fixtures evolved to support both file and DB testing","Mocking patterns for file/DB operations","Special cases for long content and special characters"]}
{"type":"entity","name":"implementation_challenges","entityType":"technical_issues","observations":["Initial circular dependency between services","SQLAlchemy session management in async context","Atomic file operations with proper error handling","Maintaining DB sync with filesystem changes","Handling long content in observations","Managing test isolation with file operations","Deciding on markdown format tradeoffs","Concurrent operation complexity"]}
{"type":"entity","name":"Basic_Factory","entityType":"Project","observations":["Collaborative project between Paul and Claude","Explores AI-human collaboration in software development","Uses MCP tools for file and memory management","Built with git integration capabilities","Focuses on maintaining project context across sessions","About 90% complete with MCP tools","Still needs improvements in collaboration via files/git/github","Will be used to document and share collaborative development process"]}
{"type":"entity","name":"Basic_Factory_Components","entityType":"Technical","observations":["Server-side rendering with JinjaX","HTMX for dynamic updates","Alpine.js for client-side state","Tailwind CSS for styling","Component translation from React/shadcn/ui","Focus on simplicity and understandability","Demonstrates meta-compiler principles in component translation"]}
{"type":"entity","name":"Component_Translation_Process","entityType":"Methodology","observations":["Treats component porting as meta-compilation","Maps between React/TypeScript and JinjaX/Alpine.js domains","Uses formal grammar transformation approaches","Maintains functionality while simplifying implementation","Focuses on server-side rendering patterns","Preserves accessibility and performance","Uses short, focused git branches for each component"]}
{"type":"entity","name":"Basic_Machines_Philosophy","entityType":"Philosophy","observations":["Combines DIY punk ethics with software development","Emphasizes user empowerment and understanding","Values simplicity and composability","Treats complex systems as combinations of simple parts","Focuses on authentic creation and sharing","Draws inspiration from punk rock, Free Software, and theoretical CS","Emphasizes the cycle of creation, complexity, and renewal"]}
{"type":"entity","name":"Basic_Machines_Manifesto","entityType":"Document","observations":["Created through collaboration between Paul and Claude","Explores connection between DIY punk ethics and software development","Emphasizes composition over inheritance in both philosophy and practice","Views software development through lens of basic machines that combine for complex computation","Advocates for user empowerment and technological independence","Structured in sections covering Origins, Philosophy, Technical Implementation, and AI Collaboration","Draws connections between punk rock, free software, and theoretical computer science","Emphasizes importance of sharing knowledge and building community","Released in December 2024"]}
{"type":"entity","name":"AI_Human_Collaboration_Model","entityType":"Methodology","observations":["Focuses on deep collaboration rather than simple task completion","Maintains rich context across sessions via knowledge graph","Uses short, focused git branches for each collaborative session","Values intellectual partnership over simple code generation","Emphasizes both practical implementation and theoretical exploration","Creates space for authentic exchange while maintaining AI/human clarity","Uses formal methods when appropriate (like grammar transformation)","Documents decisions and processes for future reference","Developed through Basic Machines project experience"]}
{"type":"entity","name":"Basic_Machines_Roadmap","entityType":"Project_Plan","observations":["Phase 1 (30 days): Build basic-machines.co website","Phase 2 (60-90 days): Develop premium component bundles","Phase 3 (90-120 days): Launch Basic Foundation commercial offering","Focus on building brand and marketing presence","Prioritize components needed for own site development","Document and share collaboration process","Build sustainable business model aligned with values"]}
{"type":"entity","name":"Basic_Machines_Website","entityType":"Project","observations":["To be built at basic-machines.co","Will showcase products and vision","Needs components for navigation, hero sections, features","Will demonstrate component usage in production","Will include blog for sharing progress","Focus on clear value proposition","Platform for sharing Basic Machines philosophy"]}
{"type":"entity","name":"Basic_Memory_Markdown_Example","entityType":"Example","observations":["Shows complete markdown structure for basic-memory entity","Uses frontmatter for metadata (id, type, created, context)","Has main description section after title","Includes Observations as bullet points","Shows Relations with [id] relation_type | context format","Lists References at bottom","Created during initial design discussion","Serves as canonical example of file format"]}
{"type":"entity","name":"Basic_Memory_Database_Schema","entityType":"Technical","observations":["Uses SQLite for local storage","Entities table with id, name, type, created_at, context, description, references","Observations table linking to entities with content and context","Relations table tracking directional relationships between entities","References column needs quotes as SQL reserved word","Designed for easy rebuilding from markdown files","Foreign key constraints maintain data integrity","Unique constraint on relations prevents duplicates","Created_at timestamps track history","Context fields enable tracking information sources"]}
{"type":"entity","name":"Basic_Memory_Project_Structure","entityType":"Technical","observations":["Uses dbmate for database migrations","Projects directory stores SQLite databases and markdown files","Makefile provides common development commands","Environment vars configure database connection","db/migrations directory for SQL schema changes","Gitignore excludes database files and env config","Uses Python 3.12 with modern tooling","Tests directory for pytest files","Follows Basic Machines project conventions"]}
{"type":"entity","name":"Basic_Memory_Project_Isolation_Decision","entityType":"Decision","observations":["Decided to defer multi-project support to post-MVP","Will use separate SQLite databases per project","Initially using projects directory in code repository","Plan to make location configurable later","No changes needed to core domain model","Keeps initial implementation simple","FTS/search capabilities also deferred for simplicity"]}
{"type":"entity","name":"Basic_Memory_Implementation_Plan","entityType":"Plan","observations":["Start with SQLAlchemy models matching schema","Then build CLI for basic operations","Then implement markdown parser","Use TDD approach throughout","Begin with core domain model","CLI will support CRUD operations","Parser must handle frontmatter and sections","Following modular development approach","Planning to use typer for CLI","Will use modern Python tools and practices"]}
{"type":"entity","name":"Basic_Memory_Implementation_Status","entityType":"Status","observations":["Core modules implemented: models, services, repository, fileio","Modular architecture with clear separation of concerns","File operations extracted to separate fileio module","Initial ObservationService implementation complete","Basic test coverage in place","Exploring observation management strategies","Using SQLAlchemy for database interaction","Markdown file operations working","Entity management functional","Repository layer implementation complete with SQLAlchemy models and tests","Database operations working with proper UTC timestamp handling","In-memory SQLite testing infrastructure proven effective"]}
{"type":"entity","name":"Basic_Memory_Observation_Management_Design","entityType":"Design","observations":["Four approaches under consideration","Bulk Update: Simple but less efficient","Tracked Observations: Precise but clutters markdown","Diff-based: Efficient but complex","Position-based: Clean but fragile","Key challenge is balancing markdown readability with efficient updates","Must maintain filesystem as source of truth","Need to consider concurrent edits","Currently evaluating trade-offs","Implementation choice pending discussion"]}
{"type":"entity","name":"Basic_Memory_Architectural_Decisions","entityType":"Decisions","observations":["Split file operations into separate fileio module","Using SQLAlchemy for database operations","Maintain filesystem as source of truth","Modular service-based architecture","Clear separation between data access and business logic","Repository pattern for database interactions","Schemas separate from models","Focus on maintainability and testability","Services handle business rules","Considering concurrency in design"]}
{"type":"entity","name":"Basic_Memory_Implementation_Analysis","entityType":"Analysis","observations":["Clean modular architecture with clear responsibilities","Strong typing throughout codebase","Excellent error handling with custom exceptions","SQLAlchemy models perfectly match our domain model","Atomic file operations for data safety","Services implement filesystem-as-source-of-truth principle","Async support throughout","Good separation between domain models and database models","Careful handling of UTC timestamps","Smart use of SQLAlchemy relationships"]}
{"type":"entity","name":"Basic_Memory_Current_Challenges","entityType":"Challenges","observations":["Observation update/removal strategy needs to be chosen","Need to handle concurrent file operations safely","Search functionality to be implemented","Edge cases in markdown formatting to be handled","Session management for concurrent operations needed","Balance between file operations and database sync","Testing coverage could be expanded","Need to handle relationship updates in files"]}
{"type":"entity","name":"Basic_Memory_Observation_Hash_Tracking","entityType":"Design","observations":["Use content hashes to track observation identity","Store hashes in database but not in markdown","Can match observations across file edits using hashes","Similar to how git tracks content changes","Keeps markdown clean and human-friendly","Allows efficient bulk updates","Handles reordering of observations","Maintains filesystem as source of truth","No need for visible IDs in markdown","Could track observation history through hash changes"]}
{"type":"entity","name":"Basic_Memory_Repository_Implementation","entityType":"Code_Implementation","observations":["Implemented base Repository class with CRUD operations","Added specialized EntityRepository, ObservationRepository, and RelationRepository","Used string IDs instead of UUIDs","Added UTCDateTime custom type for timestamp handling","Used in-memory SQLite for testing","Achieved 84% test coverage","Created comprehensive pytest fixtures"]}
{"type":"entity","name":"Basic_Memory_Dependencies","entityType":"Technical","observations":["Uses Python 3.12","SQLAlchemy with async support","pytest-asyncio for async testing","aiosqlite for async SQLite operations","greenlet for SQLAlchemy async support","uv for dependency management","pytest-cov for coverage reporting","Development dependencies managed in pyproject.toml"]}
{"type":"entity","name":"Basic_Memory_Current_Architecture","entityType":"Architecture_Analysis","observations":["Clear separation between domain models (Pydantic) and storage models (SQLAlchemy)","File I/O completely separated into dedicated module","Strong 'filesystem as source of truth' pattern in services","Atomic file operations with proper error handling","Service layer coordinates between filesystem and database","Database acts as queryable index rather than primary storage","Clean error hierarchy with specific exception types","Rebuild operations available for recovery scenarios"]}
{"type":"entity","name":"Basic_Memory_Evolution","entityType":"Analysis","observations":["Started with repository pattern following basic-foundation","Evolved to more sophisticated architecture with clear layers","Added Pydantic schemas for domain modeling","Separated file operations into dedicated module","Implemented robust error handling throughout","Maintained filesystem as source of truth principle","Added observation management with context tracking","Introduced rebuild capabilities for system recovery"]}
{"type":"entity","name":"Basic_Memory_Service_Layer","entityType":"Implementation","observations":["EntityService handles entity lifecycle and coordinates storage","ObservationService manages observations within entities","Services ensure filesystem and database stay in sync","Clear error handling with ServiceError hierarchy","Strong typing throughout service interfaces","Implements filesystem as source of truth pattern","Handles UUID generation and timestamp management","Provides methods for system recovery and rebuild"]}
{"type":"entity","name":"Basic_Memory_Schema_Design","entityType":"Implementation","observations":["Uses Pydantic for domain models and validation","Automatic ID generation with timestamp and UUID","Clear separation from SQLAlchemy storage models","Supports optional context tracking","Models match markdown file structure","Enables clean serialization/deserialization","Strong typing with proper validation rules","Independent from storage concerns"]}
{"type":"entity","name":"Basic_Memory_Next_Tasks","entityType":"TaskList","observations":["✅ Implement SQLAlchemy models and repositories (Done)","✅ Add SQLAlchemy migrations (Done)","✅ Create service layer (Done)","✅ Implement file I/O module (Done)","✅ Set up domain models with Pydantic (Done)","✅ Initial test infrastructure (Done)","✅ Basic CRUD operations (Done)","⏳ Implement full test coverage for db.py","⏳ Add more sophisticated search functionality","⏳ Implement CLI interface","⏳ Add relationship management to services","⏳ Handle concurrent file operations safely","⏳ Add versioning for file changes","⏳ Implement proper backup strategy","⏳ Add type hints throughout codebase","⏳ Improve error messages and logging","⏳ Add documentation for core modules"]}
{"type":"entity","name":"Basic_Memory_Meta_Experience","entityType":"Case_Study","observations":["Experienced our own context loss when reconstructing project knowledge","Had to rebuild task list and project context from filesystem and memory","Validated 'filesystem as source of truth' principle through reconstruction","Code and tests served as reliable historical record","Knowledge graph structure helped guide reconstruction process","Markdown files provided human-readable context","Atomic information design made piece-by-piece reconstruction possible","Ironic validation of the need for basic-memory's features","Experience demonstrates value of durable, human-readable knowledge storage","Shows importance of separating durable storage from ephemeral context"]}
{"type":"entity","name":"Model_Context_Protocol","entityType":"protocol","observations":["Core part of the basic-memory architecture","Enables AI-human collaboration on projects","Provides tool-based interaction with knowledge graph","Developed by Anthropic for structured AI-system interaction","Used for maintaining consistent, rich context across conversations"]}
{"type":"entity","name":"basic-memory_core_principles","entityType":"principles","observations":["Local First: All data stored locally in SQLite","Project Isolation: Separate databases per project","Human Readable: Everything exportable to plain text","AI Friendly: Structure optimized for LLM interaction","DIY Ethics: User owns and controls their data","Simple Core: Start simple, expand based on needs","Tool Integration: MCP-based interaction model"]}
{"type":"entity","name":"basic-memory_business_model","entityType":"business_strategy","observations":["Core features free: Local SQLite, basic knowledge graph, search, markdown export, basic MCP tools","Professional features potential: Rich document export, advanced versioning, collaboration features, custom integrations, priority support","Focus on maintaining DIY/punk philosophy while enabling sustainability"]}
{"type":"entity","name":"basic-memory_cli","entityType":"interface","observations":["Supports project management commands (create, switch, list)","Entity management (add entity, add observation, add relation)","Future support for export and batch operations","Follows consistent command structure","Planned integration with MCP tools"]}
{"type":"entity","name":"basic-memory_export_format","entityType":"file_format","observations":["Uses markdown with frontmatter metadata","Includes entity name, type, creation timestamp","Observations as bullet points","Relations in structured format with links","References section at bottom","Designed for human readability and machine parsing","Example format documented in project specs"]}
{"type":"entity","name":"relation_service","entityType":"code_module","observations":["Planned service for managing relations in both filesystem and database","Will follow filesystem-is-source-of-truth principle like other services","Needs to handle atomic file operations for relation updates","Must coordinate with EntityService for relationship integrity","Will handle bidirectional relationship tracking","Will support relation validation and type enforcement","Must implement rebuild functionality for index recovery","Will need careful error handling for file/db sync","Should support relation search and filtering","Must handle relation lifecycle (create/read/update/delete)"]}
{"type":"entity","name":"service_layer_patterns","entityType":"implementation_patterns","observations":["Services handle both file and database operations","Filesystem is always source of truth","Database serves as queryable index","Services implement atomic file operations","Clear error hierarchy with specific exceptions","Use of dependency injection via constructor params","Async/await used throughout service layer","Services coordinate between storage layers","Repository pattern used for database access","Services maintain entity integrity across storage","Rich error types extend from ServiceError base","Rebuild operations available for recovery"]}
{"type":"entity","name":"database_models","entityType":"implementation","observations":["Entity model with unique name and type","Observation model linked to entities","Relation model tracks connections between entities","Custom UTCDateTime type for timestamp handling","Use of SQLAlchemy relationships for navigation","Cascading deletes for dependent objects","String IDs used for compatibility","Rich relationship modeling with backpopulates","Proper indexing on foreign keys","Context tracking available on models","Models include created_at timestamps","Relationships handle bidirectional navigation"]}
{"type":"entity","name":"repository_patterns","entityType":"implementation_patterns","observations":["Generic Repository[T] base class implementation","Type-safe operations with SQLAlchemy","Specialized repositories for each model type","Async operations throughout","Clear error handling patterns","Support for custom queries and filtering","Pagination support built-in","Transaction management via session","Proper type hints and generics usage","Entity-specific query methods in subclasses"]}
{"type":"entity","name":"relation_service_design","entityType":"design","observations":["Must handle relation lifecycle in both files and DB","Needs to validate existence of both entities","Should support relation type enforcement","Must maintain bidirectional consistency","Should support relation querying and filtering","Needs proper error handling for graph consistency","Must integrate with entity file format","Should support bulk operations for efficiency","Must handle relation deletion and cascading","Should provide search by type and entities"]}
{"type":"entity","name":"relation_service_implementation_plan","entityType":"plan","observations":["1. Define core relation operations (create, get, delete)","2. Implement file format handling for relations","3. Add database sync with RelationRepository","4. Implement validation and error handling","5. Add rebuild and recovery operations","6. Implement relation type enforcement","7. Add relation search and filtering","8. Implement bulk operations","9. Add comprehensive tests","10. Document API and error handling"]}
{"type":"entity","name":"relation_service_challenges","entityType":"challenges","observations":["Maintaining consistency between file and database","Handling relation type validation efficiently","Managing bidirectional relationships in files","Ensuring atomic updates across entities","Handling deletion with proper cascading","Efficient querying of relation graphs","Recovery from partial file/db sync failures","Bulk operation atomicity","Clear error reporting for graph operations","Performance with large relation sets"]}
{"type":"entity","name":"relation_file_format","entityType":"file_format","observations":["Relations stored in entity markdown files","Format: [target_id] relation_type | context","Relations section marked by ## Relations header","Outgoing relations only stored in source entity","Relations rebuild on entity load","Clean human-readable format","Context is optional with pipe separator","Links generate valid navigation references","Markdown-friendly formatting","Example: [Paul] authored | with Claude"]}
{"type":"entity","name":"relation_service_error_handling","entityType":"implementation_patterns","observations":["RelationError extends ServiceError base","Specific errors for validation failures","Handles entity not found cases","Manages relation type validation errors","File operation errors properly wrapped","Database sync errors clearly reported","Transaction rollback on errors","Proper error propagation chain","Clear error messages for debugging","Recovery paths for common errors"]}
{"type":"entity","name":"relation_service_testing","entityType":"testing","observations":["Test all relation lifecycle operations","Verify file and database consistency","Test relation type validation","Check error handling paths","Test bulk operations","Verify bidirectional consistency","Test recovery operations","Check cascade operations","Verify search and filtering","Test with large relation sets"]}
{"type":"entity","name":"fileio_patterns","entityType":"implementation_patterns","observations":["Atomic file operations with temporary files","Clear error handling for IO operations","Consistent file naming and paths","Support for different file formats","Efficient file reading and writing","Proper file locking mechanisms","Recovery from partial writes","Consistent encoding handling","Directory management utilities","Path manipulation helpers","Currently implemented in fileio.py module","Uses pathlib for path operations","Handles file not found cases gracefully","Maintains data integrity during writes"]}
{"type":"entity","name":"pytest_patterns","entityType":"implementation_patterns","observations":["Common fixtures should be in conftest.py for reuse","Use pytest_asyncio.fixture for async fixtures","Session fixtures need proper async cleanup","Temporary directories should be managed with context managers","Test categories: happy path, error path, recovery, edge cases","Services need project_path and repo injected","Use monkeypatch for mocking in async context","SQLite in-memory database ideal for testing","Explicit test verification: file content and database state"]}
{"type":"entity","name":"relation_implementation_learnings","entityType":"implementation_learnings","observations":["Better to pass full Entity objects than IDs to services","Services should not re-read entities if they have them","File operations should be atomic and verified","Database serves as queryable index, not source of truth","Relations stored in source entity's markdown file","Clear separation between file ops and database sync","Entity objects should own their relations list","Context is optional but fully supported in implementation"]}
{"type":"entity","name":"test_driven_insights","entityType":"learnings","observations":["Tests help reveal better API design (e.g., passing Entity objects)","Error cases drive proper exception hierarchy","File verification as important as database checks","Edge cases inform markdown format decisions","Recovery tests ensure system resilience","Tests document expected behavior clearly","Fixtures significantly reduce test complexity","Common patterns emerge through test writing"]}
{"type":"entity","name":"meta_development_insights","entityType":"process","observations":["Break down large tasks into reviewable chunks","One file at a time prevents response truncation","Iterative development with tests leads to better design","Infrastructure code (fixtures) should be consolidated early","Test categories help ensure comprehensive coverage","Knowledge capture should happen during development","APIs tend to evolve toward simpler patterns","File operations require careful verification"]}
{"type":"entity","name":"AI_Assistant_Learnings","entityType":"meta_insights","observations":["Output management: Breaking responses into single files prevents truncation and allows better review","Knowledge graph helps maintain context: I can reference previous decisions and patterns accurately","Memory rebuilding experience validated the need for durable storage","Test-driven development provides clear steps and verification","Explicit relation tracking in knowledge graph helps me understand project context","Rich context from multiple sources (code, docs, tests) enables better assistance","File-at-a-time approach allows deeper analysis of each component","Keeping entity names consistent helps with referencing and relationships"]}
{"type":"entity","name":"Effective_Response_Patterns","entityType":"meta_patterns","observations":["When showing code changes, break into discrete files","Review existing code before suggesting changes","Reference knowledge graph for context and patterns","Explicitly connect new code to existing patterns","Validate suggestions against test cases","Keep track of file changes for atomic commits","Check both implementation and test files for consistency","Maintain clear separation of concerns in responses"]}
{"type":"entity","name":"AI_Context_Management","entityType":"meta_practice","observations":["Knowledge graph provides reliable persistent memory","Project documentation gives high-level context","Code review shows implementation patterns","Tests demonstrate expected behavior","Important to actively track what has been modified","Entity relationships help understand dependencies","Regular knowledge capture during development","Using consistent entity references across conversations"]}
{"type":"entity","name":"AI_Tool_Usage_Patterns","entityType":"meta_practice","observations":["read_file before suggesting changes","write_file one file at a time","list_directory to understand project structure","search_nodes to find relevant context","create_entities to capture new learnings","create_relations to connect concepts","Using knowledge graph to track decisions","Validating changes through test execution"]}
{"type":"entity","name":"relation_service_learnings","entityType":"implementation_learnings","observations":["Entity-based API cleaner than ID-based for service layer","Model_dump method can handle storage serialization","File format needs explicit section markers (## Relations)","Whitespace handling important for long content comparisons","Test fixtures allow focused test cases","SQLAlchemy selects better than raw SQL for type safety","Atomic file operations maintained for relations"]}
{"type":"entity","name":"test_driven_insights_relations","entityType":"learnings","observations":["Tests revealed need for whitespace normalization","Edge cases drove file format decisions","SQLAlchemy model access safer than raw queries","Fixtures reduced test setup complexity","File verification as important as database checks","Testing both memory model and storage format","Test categories ensure comprehensive coverage"]}
{"type":"entity","name":"relation_service_patterns","entityType":"patterns","observations":["Use Entity objects in API","Serialize to IDs for storage","Maintain file as source of truth","Keep file format human-readable","Handle circular references in serialization","Use repository pattern for database","Clear error hierarchies"]}
{"type":"entity","name":"packaging_learnings","entityType":"technical_learnings","observations":["When using pytest-mock, traditional pip install works more reliably than uv sync","Package discovery behavior can differ between uv and pip","Clean venv with pip install is a reliable fallback for dependency issues","Package installation location might differ between uv and pip","Dependencies in pyproject.toml dev section work reliably with pip install -e .[dev]"]}
{"type":"entity","name":"Recent_Implementation_Progress","entityType":"progress_update","observations":["Successfully split services.py into modular structure under services/","Created __init__.py, entity_service.py, observation_service.py, relation_service.py","Fixed pytest-mock installation issues by using pip install -e .[dev] instead of uv sync","Improved test structure with minimal mocking - only used for error testing","Implemented relation service with Entity-based API","Achieved good test coverage across services","File operations are only mocked when testing error conditions","Services follow filesystem-as-source-of-truth pattern"]}
{"type":"entity","name":"Next_Steps","entityType":"project_tasks","observations":["Consider adding more relation service tests","Potentially expand relations features","Look for opportunities to improve test coverage","Consider documenting package management preferences (pip vs uv)","Consider adding integration tests for services","Review and possibly expand error handling cases"]}
{"type":"entity","name":"Development_Practices","entityType":"process","observations":["Favor real operations over mocks in tests","Only mock for error condition testing","Use pip install -e .[dev] for reliable dev dependency installation","Maintain modular service structure","Keep filesystem as source of truth","Use Entity objects in service APIs instead of IDs","Validate both file and database state in tests"]}
{"type":"entity","name":"MCP_Resources","entityType":"Concept","observations":["Stateful objects in Model Context Protocol","Enable persistent access to capabilities"]}
{"type":"entity","name":"MCP_Server_Implementation","entityType":"Technical_Design","observations":["Inherits from mcp.server.Server base class","Tools are implemented as async methods","Each tool method maps directly to a function available to the AI","Tools can request user input via Prompts","Simple function call interface rather than explicit resource management","State management handled by server instance","Returns serialized data using model_dump() for consistency"]}
{"type":"entity","name":"MCP_Tools","entityType":"Protocol_Feature","observations":["Defined as async methods on server class","Return values must match tool definition schema","Can maintain state between invocations via server instance","Tools can prompt for user input when needed","No need for explicit Resource objects in implementation"]}
{"type":"entity","name":"Basic_Memory_MCP","entityType":"Implementation","observations":["Uses MemoryService for core operations","Implements project selection via prompts","Maintains project context across tool invocations","Maps directly to memory graph operations","Handles serialization of Pydantic models"]}
{"type":"entity","name":"Basic_Memory_Testing","entityType":"Testing_Design","observations":["Needs pytest for async testing","Should isolate filesystem operations for tests","Needs to handle MCP server lifecycle in tests","Should test both service layer and MCP interface","Will need mocks for project paths and file operations"]}
{"type":"entity","name":"Memory_Service_Tests","entityType":"Test_Suite","observations":["Should test entity creation with observations","Should test relation creation between entities","Should verify proper ID generation and model validation","Should test deletion cascading","Should test search functionality","Must verify proper serialization of entities and relations"]}
{"type":"entity","name":"MCP_Server_Tests","entityType":"Test_Suite","observations":["Should test project initialization workflow","Should test prompt handling","Should verify tool input/output formats","Should test error cases and validation","Must verify proper serialization in tool responses"]}
{"type":"entity","name":"Memory_Service_Refactoring","entityType":"Technical_Task","observations":["MemoryService uses create() but EntityService might expect create_entity()","MemoryService assumes get_by_name() but EntityService might use different method","Need to verify deletion method signatures","Need to check if search interface matches","Should verify observation handling matches ObservationService interface","RelationService methods need verification","EntityService.create_entity takes name, type, and optional observations directly, not an Entity object","EntityService requires project_path and entity_repo in constructor","ObservationService.add_observation takes Entity object and content string, not raw data","RelationService.create_relation takes Entity objects directly, not dict data","All services follow filesystem-as-source-of-truth pattern with DB indexing","All services handle database synchronization internally","Services expect Path objects for filesystem operations"]}
{"type":"entity","name":"Service_Interface_Audit","entityType":"Technical_Task","observations":["Need to review all existing service interfaces","Document current method signatures","Map discrepancies between MemoryService assumptions and actual interfaces","Check return types and error handling patterns","Review transaction/atomicity requirements","Method signatures need alignment: create vs create_entity etc","Need to handle DB repositories in service constructors","File operations should use project_path consistently","Need to maintain filesystem-as-source-of-truth pattern","Should handle database synchronization at service level","Error handling should align with existing patterns","Consider making MemoryService handle DB indexing consistently"]}
{"type":"entity","name":"Memory_Service_Patterns","entityType":"Technical_Pattern","observations":["Uses inner async functions to encapsulate operation logic","Leverages list comprehensions with async functions for parallel operations","Each operation follows a consistent pattern: validate, update DB, write file","Inner functions make the code more readable and maintainable","Operations can run in parallel when using list comprehensions with async functions"]}
{"type":"entity","name":"Pydantic_Create_Pattern","entityType":"Technical_Pattern","observations":["Separate Create models match the exact shape of incoming data","Provides clear contract for MCP tool inputs","Handles validation of raw input data","Converts cleanly to domain models via from_create methods","Maintains separation between external API format and internal models","Similar to FastAPI request model pattern","Allows camelCase in API while using snake_case internally"]}
{"type":"entity","name":"Basic_Memory_Business","entityType":"Business_Model","observations":["Core system is open source and free","Local-first, giving users data control","Professional features could be licensed","Enterprise support and customization services","Potential for MCP tool marketplace"]}
{"type":"entity","name":"MCP_Marketplace","entityType":"Business_Concept","observations":["Could host verified MCP tools for different use cases","Tools rated by performance and reliability","Marketplace takes percentage of tool usage fees","Enterprise tool verification and security scanning","Custom tool development services","Integration support for existing tools"]}
{"type":"entity","name":"Persistence_Of_Vision","entityType":"Concept","observations":["Mental model for continuous AI-human interaction","Like cinema: 24fps creates illusion of smooth motion","Basic-memory provides 'frames' of structured knowledge","Current state: Better than flipbook, not yet digital cinema","Goal: Achieve smoother cognitive continuity between interactions","Proposed by Drew as metaphor for AI conversation continuity"]}
{"type":"entity","name":"Conversation_Continuity_Pattern","entityType":"Usage_Pattern","observations":["Use basic-memory entity/relation schema for conversations","Each chat becomes an entity with observations for key points","Relations link to discussed concepts and other chats","Uses zettelkasten format IDs for natural ordering","Can be used as template/recipe for others","Future possibility: Git SHA integration for versioning"]}
{"type":"entity","name":"Usage_Recipes","entityType":"Feature_Concept","observations":["Predefined patterns users can follow or adapt","Could include conversation tracking recipe","Templates for different knowledge management styles","Shows practical applications of the generic schema","Helps users get started with the system"]}
{"type":"entity","name":"Chat_References","entityType":"Technical_Feature","observations":["Uses ref:* syntax to reference previous conversations","Combines reference semantics with pointer symbolism","Format: ref:*{zettelkasten-id}","Allows explicit context loading between chats","Inspired by C++ references and pointers","Provides memory-model-like access to conversation context","Uses ref:// URI format following MCP Resource pattern","Could support multiple reference schemes (chat/entity/concept)","Makes reference semantics explicit and unambiguous","Aligns with standard URI formatting"]}
{"type":"entity","name":"Chat_Reference_Protocol","entityType":"Technical_Specification","observations":["Uses URI format: ref://basic-memory/chat/[id]","Follows MCP Resource pattern: [protocol]://[host]/[path]","Enables explicit context loading between chats","Can support multiple resource types (chat/entity/concept)","Provides standardized way to reference previous conversations","Example: ref://basic-memory/chat/20240307-drew-ab12ef34"]}
{"type":"entity","name":"20240307-chat-reference-protocol","entityType":"conversation","observations":["Developed ref:// URI format for chat references","Added Chat Reference Protocol to prompt instructions","Discussed implementation of chat continuation","Created complete prompt instructions document","Reference format follows MCP Resource pattern","Reviewed and confirmed complete prompt instructions","Ready to test ref://basic-memory/chat/20240307-chat-reference-protocol in new chat"]}
{"type":"entity","name":"20240307-chat-reference-protocol-test","entityType":"conversation","observations":["First implementation test of chat reference protocol","Testing continuation from 20240307-chat-reference-protocol","Focused on practical implementation of ref:// URI format"]}
{"type":"entity","name":"Write_File_Tool_Usage","entityType":"Tool_Usage_Pattern","observations":["Never use placeholders like '# Rest of...' when writing files - must include complete file content","File content must be complete and valid - partial updates will truncate the file","If showing partial changes, should inform human and let them handle the file write","write_file tool replaces entire file contents - cannot do partial updates","Code files especially must be complete and valid to avoid breaking functionality","Always read_file before write_file to understand current state","Using write_file without reading first risks reverting recent changes","Pattern should be: read current state, make modifications, then write if needed","Especially important in collaborative development where files may have been updated"]}
{"type":"entity","name":"Run_Tests_Tool_Request","entityType":"Feature_Request","observations":["Need to add a tool enabling Claude to run tests locally","Would help with direct validation of code changes","Current workaround: Claude has to ask human to run tests","Should support running specific test functions (e.g. pytest tests/test_memory_service.py::test_create_relations)","Would improve iterative development workflow between human and AI"]}
{"type":"entity","name":"SQLAlchemy_Async_Loading_Pattern","entityType":"Technical_Pattern","observations":["Use selectinload() instead of lazy loading when accessing SQLAlchemy relationships in async code","Lazy loading doesn't work with async due to greenlet context requirements","selectinload performs a single efficient query with an IN clause","Pattern used in basic-memory's EntityRepository for loading relations","Documented in find_by_id method with thorough explanation","Alternative approaches: joinedload (single JOIN query) or subqueryload (subquery approach)","Benefits: prevents 'MissingGreenlet' errors, reduces N+1 query problems","Key insight: load all needed relationships upfront in async code","Example use: selectinload(Entity.outgoing_relations)"]}
{"type":"entity","name":"20241207-sqlalchemy-async-pattern","entityType":"conversation","observations":["Fixed SQLAlchemy async relationship loading issues","Implemented selectinload pattern in EntityRepository","Updated find_by_id to eager load relations","Added documentation about the pattern","Created knowledge graph entry about SQLAlchemy async loading","Fixed failing tests by properly loading relations in memory_service","Discussed SQLAlchemy relationship loading best practices"]}
{"type":"entity","name":"20241207-memory-service-relations","entityType":"conversation","observations":["Fixed SQLAlchemy async loading with selectinload pattern","Updated find_by_id in EntityRepository to eager load relations","Discovered create_relations works but returns empty list","Verified relations are being stored correctly in memory.json","Next step: Work on MemoryService.add_observations implementation","Improved understanding of MCP memory storage format through debugging"]}
{"type":"entity","name":"add_observations_implementation_plan","entityType":"technical_plan","observations":["Follow pattern from create_entity and create_relation methods","File operations first (read & write) - filesystem is source of truth","Database updates in parallel","Simplify current implementation","Current flow is:"," - First read entities and create observations"," - Write files in parallel"," - Update DB indexes sequentially","Key tests needed:"," - Adding observations to multiple entities"," - Verifying filesystem state first"," - Verifying database state"," - Error cases for missing entities"," - Error cases for file operations"]}
{"type":"entity","name":"MCP_Reference_Integration","entityType":"feature_idea","observations":["Can be implemented as a Model Context Protocol integration similar to the fetch tool","Would provide structured way to pass chat references to Claude","Could handle ref:// URL format systematically","Integration would fetch context from referenced chats and inject into conversation","Observed from Claude Desktop UI showing MCP integration pattern with fetch tool","Would be more robust than passing references in chat text"]}
{"type":"entity","name":"Project_Priorities","entityType":"roadmap","observations":["P1: Dogfooding basic-memory system instead of JSON memory store","Future: Implement MCP-based reference system"]}
{"type":"entity","name":"great_observation_loading_saga_20241207","entityType":"debugging_session","observations":["Occurred on December 7, 2024 while debugging basic-memory SQLAlchemy relationship loading","Issue: selectinload() wasn't properly loading relationships in async SQLAlchemy context","Tried multiple solutions: explicit joins, manual loading, various SQLAlchemy loading strategies","Final solution: Using session.refresh() with explicit relationship names","Memorable quote: 'The Great Observation Loading Saga'","Key learning: Sometimes the obvious SQLAlchemy patterns need adaptation for async contexts","Solution preserved in basic-memory repository in EntityRepository.find_by_id()"]}
{"type":"entity","name":"basic_memory_implementation_20241208","entityType":"technical_milestone","observations":["Fixed async SQLAlchemy relationship loading issues by using explicit refresh with relationship names","Established pattern of relationship handling belonging in MemoryService not EntityService","Fixed ID generation flow through Pydantic schemas to DB layer","Standardized error handling using EntityNotFoundError","All 32 tests passing with 70% coverage","Core services (Entity, Observation, Relation) working properly","Ready for MCP server implementation","Notable debugging session: The Great Observation Loading Saga - resolved lazy loading issues","Established clear separation between MemoryService orchestration and individual service responsibilities"]}
{"type":"entity","name":"MCP_Dependency_Risk","entityType":"technical_lesson","observations":["Experienced disruption when MCP npm package disappeared - 'leftpad moment'","Need to ensure basic-memory tools are resilient to external dependency issues","Local implementation of MCP server provides better stability than npm packages","Important to maintain control of critical infrastructure components","Validates DIY/local-first philosophy of basic-memory project","Package manager fragility revealed by simple 'npx @modelcontextprotocol/server-memory' failure"]}
{"type":"entity","name":"basic_memory_project_20241208","entityType":"technical_milestone","observations":["Core MCP server implementation completed with tools: create_entities, search_nodes, open_nodes, add_observations, create_relations, delete_entities, delete_observations","ProjectConfig and dependency injection pattern established","Test framework in place with in-memory DB support","Support for both camelCase (MCP) and snake_case (internal) formats","Filesystem remains source of truth with SQLite as index","Two-way sync pattern identified between Claude MCP tools and direct markdown file editing","Ready for Claude Desktop integration testing phase","Next steps identified: passing tests, markdown format definition, file change tracking, real-world testing","Implementation prioritizes local-first principles with filesystem as source of truth"]}
{"type":"entity","name":"basic_memory_mcp_architecture","entityType":"technical_design","observations":["MemoryServer class extends MCP Server with custom handler registration","Uses ProjectConfig for clean dependency injection and configuration","Memory service can be injected for testing","Handlers exposed as instance attributes for testing","Tool schemas leverage existing Pydantic models"]}
{"type":"entity","name":"basic_memory_sync_considerations","entityType":"design_insight","observations":["Need to handle sync between direct markdown file edits and DB index","Watch for file system changes as potential future enhancement","Consider index rebuild patterns on startup","Keep human-friendly markdown format for direct editing"]}
{"type":"entity","name":"mcp_server_learnings","entityType":"developer_insight","observations":["MCP protocol is new and documentation is still evolving","Test patterns are not well established yet in example implementations","Supporting both camelCase and snake_case helps with protocol/internal compatibility","Server.handle_* naming convention is important for handler registration"]}
{"type":"entity","name":"20241208-mcp-tool-refactoring","entityType":"conversation","observations":["Decision to return structured data via EmbeddedResource instead of TextContent string parsing","Plan to create Pydantic result models (CreateEntitiesResult, SearchNodesResult etc)","Will use application/vnd.basic-memory+json as MIME type for our structured data","Currently debugging test issues with add_observations tool","Entity ID vs name resolution needed in add_observations","Goal is to make tools more joyful to use by eliminating string parsing","MCP spec supports EmbeddedResource for structured data returns"]}
{"type":"entity","name":"Basic Memory MCP Server Implementation","entityType":"technical_notes","observations":["Server implements Model Context Protocol using proper structured data responses","Uses EmbeddedResource with custom MIME type 'application/vnd.basic-memory+json'","Clean separation between input validation and handlers via Pydantic models","All tool operations return structured data through create_response helper","Type safety with Literal types for tool names and proper typing for handlers","Handler registry pattern with TOOL_HANDLERS dictionary","Consistent error handling pattern using MCP error codes","Uses Pydantic ConfigDict for proper ORM integration","Tool schemas organized into Input and Response types","Input validation with Annotated types for extra constraints","Response models consistently use from_attributes=True for ORM data","Entity ID generation moved to model validator on EntityBase","Follows principle of making common operations easy and safe"]}
{"type":"relation","from":"Paul","to":"Basic_Machines","relationType":"created_and_maintains"}
{"type":"relation","from":"basic-memory","to":"Basic_Machines","relationType":"is_component_of"}
{"type":"relation","from":"Paul","to":"basic-memory","relationType":"develops"}
{"type":"relation","from":"fileio_module","to":"basic-memory_implementation_patterns","relationType":"implements"}
{"type":"relation","from":"entity_service","to":"basic-memory_implementation_patterns","relationType":"implements"}
{"type":"relation","from":"observation_service","to":"basic-memory_implementation_patterns","relationType":"implements"}
{"type":"relation","from":"fileio_module","to":"basic-memory","relationType":"is_component_of"}
{"type":"relation","from":"entity_service","to":"basic-memory","relationType":"is_component_of"}
{"type":"relation","from":"observation_service","to":"basic-memory","relationType":"is_component_of"}
{"type":"relation","from":"entity_service","to":"fileio_module","relationType":"uses"}
{"type":"relation","from":"observation_service","to":"fileio_module","relationType":"uses"}
{"type":"relation","from":"observation_management","to":"observation_service","relationType":"influences_design_of"}
{"type":"relation","to":"basic-memory","from":"testing_infrastructure","relationType":"supports"}
{"type":"relation","to":"testing_infrastructure","from":"test_categories","relationType":"implements"}
{"type":"relation","to":"basic-memory","from":"completed_work","relationType":"tracks_progress_of"}
{"type":"relation","to":"basic-memory","from":"future_work","relationType":"guides_development_of"}
{"type":"relation","to":"basic-memory","from":"design_decisions","relationType":"shapes_architecture_of"}
{"type":"relation","to":"basic-memory","from":"concurrency_considerations","relationType":"influences_design_of"}
{"type":"relation","to":"future_work","from":"concurrency_considerations","relationType":"informs"}
{"type":"relation","to":"observation_management","from":"design_decisions","relationType":"guides"}
{"type":"relation","to":"testing_infrastructure","from":"completed_work","relationType":"established"}
{"type":"relation","to":"design_decisions","from":"fileio_module","relationType":"implements"}
{"type":"relation","from":"observation_update_approaches","to":"observation_management","relationType":"analyzes"}
{"type":"relation","from":"bulk_update_approach","to":"observation_update_approaches","relationType":"is_option_of"}
{"type":"relation","from":"tracked_observations_approach","to":"observation_update_approaches","relationType":"is_option_of"}
{"type":"relation","from":"diff_based_approach","to":"observation_update_approaches","relationType":"is_option_of"}
{"type":"relation","from":"position_based_approach","to":"observation_update_approaches","relationType":"is_option_of"}
{"type":"relation","from":"tasks_and_progress","to":"basic-memory","relationType":"tracks_status_of"}
{"type":"relation","from":"design_decisions","to":"observation_update_approaches","relationType":"influences"}
{"type":"relation","from":"observation_update_approaches","to":"future_work","relationType":"informs"}
{"type":"relation","to":"basic-memory_implementation_patterns","from":"error_handling_patterns","relationType":"is_part_of"}
{"type":"relation","to":"basic-memory","from":"data_models","relationType":"implements"}
{"type":"relation","to":"basic-memory","from":"markdown_format","relationType":"defines"}
{"type":"relation","to":"basic-memory","from":"test_driven_development","relationType":"guides_development_of"}
{"type":"relation","to":"basic-memory","from":"architecture_evolution","relationType":"describes_development_of"}
{"type":"relation","to":"basic-memory_implementation_patterns","from":"validation_patterns","relationType":"is_part_of"}
{"type":"relation","to":"design_decisions","from":"architecture_evolution","relationType":"informs"}
{"type":"relation","to":"fileio_module","from":"markdown_format","relationType":"implements"}
{"type":"relation","to":"error_handling_patterns","from":"test_driven_development","relationType":"influenced"}
{"type":"relation","to":"data_models","from":"validation_patterns","relationType":"implements"}
{"type":"relation","to":"markdown_format","from":"markdown_examples","relationType":"documents"}
{"type":"relation","to":"markdown_format","from":"markdown_parsing_rules","relationType":"defines"}
{"type":"relation","to":"data_models","from":"schema_definitions","relationType":"documents"}
{"type":"relation","to":"test_driven_development","from":"test_evolution","relationType":"describes"}
{"type":"relation","to":"architecture_evolution","from":"implementation_challenges","relationType":"influenced"}
{"type":"relation","to":"test_evolution","from":"implementation_challenges","relationType":"shaped"}
{"type":"relation","to":"future_work","from":"implementation_challenges","relationType":"informs"}
{"type":"relation","from":"Basic_Factory","to":"Basic_Machines","relationType":"implements"}
{"type":"relation","from":"Basic_Factory_Components","to":"Basic_Factory","relationType":"is_part_of"}
{"type":"relation","from":"Component_Translation_Process","to":"Basic_Factory_Components","relationType":"enables"}
{"type":"relation","from":"Basic_Machines_Philosophy","to":"Basic_Machines","relationType":"guides"}
{"type":"relation","from":"Paul","to":"Basic_Factory","relationType":"develops"}
{"type":"relation","from":"Paul","to":"Basic_Machines_Philosophy","relationType":"created"}
{"type":"relation","from":"Basic_Machines_Manifesto","to":"Basic_Machines_Philosophy","relationType":"articulates"}
{"type":"relation","from":"AI_Human_Collaboration_Model","to":"Basic_Factory","relationType":"guides_development_of"}
{"type":"relation","from":"Basic_Machines_Manifesto","to":"Paul","relationType":"written_by"}
{"type":"relation","from":"Basic_Machines_Manifesto","to":"Component_Translation_Process","relationType":"documents"}
{"type":"relation","from":"AI_Human_Collaboration_Model","to":"Basic_Machines","relationType":"shapes_development_of"}
{"type":"relation","from":"Basic_Machines_Roadmap","to":"Basic_Machines","relationType":"guides_development_of"}
{"type":"relation","from":"Basic_Machines_Website","to":"Basic_Machines_Roadmap","relationType":"implements_phase_of"}
{"type":"relation","from":"Basic_Factory_Components","to":"Basic_Machines_Website","relationType":"enables"}
{"type":"relation","from":"Basic_Machines_Philosophy","to":"Basic_Machines_Website","relationType":"informs"}
{"type":"relation","from":"Paul","to":"DIY_Ethics","relationType":"embodies"}
{"type":"relation","from":"Basic_Machines_Philosophy","to":"DIY_Ethics","relationType":"incorporates"}
{"type":"relation","from":"Basic_Machines","to":"DIY_Ethics","relationType":"exemplifies"}
{"type":"relation","from":"Component_Translation_Process","to":"Basic_Machines_Philosophy","relationType":"implements"}
{"type":"relation","from":"Basic_Factory_Components","to":"DIY_Ethics","relationType":"demonstrates"}
{"type":"relation","from":"AI_Human_Collaboration_Model","to":"Basic_Machines_Philosophy","relationType":"aligns_with"}
{"type":"relation","from":"AI_Human_Collaboration_Model","to":"Component_Translation_Process","relationType":"guides"}
{"type":"relation","from":"Basic_Machines_Manifesto","to":"Basic_Machines","relationType":"defines_vision_for"}
{"type":"relation","from":"Basic_Machines_Website","to":"Basic_Machines_Manifesto","relationType":"implements_vision_of"}
{"type":"relation","from":"Basic_Factory","to":"AI_Human_Collaboration_Model","relationType":"demonstrates"}
{"type":"relation","from":"Paul","to":"AI_Human_Collaboration_Model","relationType":"developed_with_Claude"}
{"type":"relation","from":"Basic_Factory_Components","to":"Component_Translation_Process","relationType":"created_through"}
{"type":"relation","from":"Basic_Machines_Philosophy","to":"Basic_Factory","relationType":"guides"}
{"type":"relation","from":"Basic_Factory","to":"MCP_Tools","relationType":"integrates"}
{"type":"relation","from":"Basic_Machines_Website","to":"Basic_Factory_Components","relationType":"will_use"}
{"type":"relation","from":"Basic_Machines_Roadmap","to":"Basic_Machines_Philosophy","relationType":"aligns_with"}
{"type":"relation","from":"Component_Translation_Process","to":"MCP_Tools","relationType":"leverages"}
{"type":"relation","from":"Basic_Factory","to":"basic-memory","relationType":"will_document_process_in"}
{"type":"relation","from":"AI_Human_Collaboration_Model","to":"basic-memory","relationType":"will_be_implemented_in"}
{"type":"relation","from":"Basic_Machines_Philosophy","to":"Basic_Machines_Roadmap","relationType":"informs_priorities_of"}
{"type":"relation","from":"basic-memory","to":"Basic_Machines_Philosophy","relationType":"embodies"}
{"type":"relation","from":"Paul","to":"Basic_Machines_Manifesto","relationType":"authored_with_Claude"}
{"type":"relation","from":"Basic_Factory","to":"Component_Translation_Process","relationType":"validated"}
{"type":"relation","from":"AI_Human_Collaboration_Model","to":"MCP_Tools","relationType":"utilizes"}
{"type":"relation","from":"Basic_Factory_Components","to":"Basic_Machines_Roadmap","relationType":"supports"}
{"type":"relation","from":"Basic_Machines_Website","to":"Basic_Factory","relationType":"will_demonstrate"}
{"type":"relation","from":"Basic_Factory","to":"basic-memory-webui","relationType":"enables_development_of"}
{"type":"relation","from":"basic-memory","to":"AI_Human_Development_Methodology","relationType":"implements"}
{"type":"relation","from":"Basic_Machines_Philosophy","to":"AI_Human_Development_Methodology","relationType":"guides"}
{"type":"relation","from":"Basic_Factory_Components","to":"basic-memory-webui","relationType":"provides_ui_for"}
{"type":"relation","from":"Component_Translation_Process","to":"AI_Human_Development_Methodology","relationType":"exemplifies"}
{"type":"relation","from":"Basic_Factory","to":"Basic Components","relationType":"enabled_creation_of"}
{"type":"relation","from":"Basic_Factory","to":"Tool Integration Discovery","relationType":"led_to"}
{"type":"relation","from":"MCP_Integration_Progress","to":"AI_Human_Development_Methodology","relationType":"validates"}
{"type":"relation","from":"Basic_Factory","to":"MCP_Integration_Progress","relationType":"demonstrates"}
{"type":"relation","from":"Basic_Factory","to":"AI_Human_Development_Methodology","relationType":"proves_effectiveness_of"}
{"type":"relation","from":"Basic_Machines_Philosophy","to":"Basic Components","relationType":"inspires_architecture_of"}
{"type":"relation","from":"DIY_Ethics","to":"basic-memory","relationType":"shapes_design_of"}
{"type":"relation","from":"Basic_Machines_Philosophy","to":"Tool Integration Discovery","relationType":"guides_analysis_of"}
{"type":"relation","from":"Basic_Machines_Manifesto","to":"AI_Human_Development_Methodology","relationType":"documents_approach_of"}
{"type":"relation","from":"Basic_Machines_Manifesto","to":"Basic_Factory_Components","relationType":"explains_principles_of"}
{"type":"relation","from":"Basic_Memory_Project_Structure","to":"basic-memory","relationType":"organizes"}
{"type":"relation","from":"Basic_Memory_Database_Schema","to":"basic-memory","relationType":"defines_storage_for"}
{"type":"relation","from":"Basic_Memory_Markdown_Example","to":"Basic_Memory_File_Format","relationType":"demonstrates"}
{"type":"relation","from":"Basic_Memory_Project_Isolation_Decision","to":"Basic_Memory_Future_Enhancement_Weighted_Relations","relationType":"similar_to"}
{"type":"relation","to":"DIY_Ethics","from":"Basic_Memory_Project_Isolation_Decision","relationType":"follows"}
{"type":"relation","from":"Basic_Memory_Implementation_Plan","to":"basic-memory","relationType":"guides"}
{"type":"relation","from":"Basic_Memory_Implementation_Plan","to":"DIY_Ethics","relationType":"follows"}
{"type":"relation","from":"Basic_Memory_Implementation_Plan","to":"Basic_Memory_Database_Schema","relationType":"implements"}
{"type":"relation","from":"Basic_Memory_Implementation_Status","to":"Basic_Memory_Implementation_Plan","relationType":"updates"}
{"type":"relation","from":"Basic_Memory_Observation_Management_Design","to":"Basic_Memory_Technical_Design","relationType":"extends"}
{"type":"relation","from":"Basic_Memory_Architectural_Decisions","to":"DIY_Ethics","relationType":"guided_by"}
{"type":"relation","from":"Basic_Memory_Architectural_Decisions","to":"basic-memory","relationType":"structures"}
{"type":"relation","from":"Basic_Memory_Implementation_Status","to":"basic-memory","relationType":"describes_state_of"}
{"type":"relation","to":"Basic_Memory_Implementation_Status","from":"Basic_Memory_Implementation_Analysis","relationType":"analyzes"}
{"type":"relation","to":"basic-memory","from":"Basic_Memory_Current_Challenges","relationType":"identifies_issues_in"}
{"type":"relation","to":"DIY_Ethics","from":"Basic_Memory_Implementation_Analysis","relationType":"confirms_alignment_with"}
{"type":"relation","to":"Basic_Memory_Observation_Management_Design","from":"Basic_Memory_Observation_Hash_Tracking","relationType":"solves"}
{"type":"relation","to":"DIY_Ethics","from":"Basic_Memory_Observation_Hash_Tracking","relationType":"aligns_with"}
{"type":"relation","to":"Basic_Memory_File_Format","from":"Basic_Memory_Observation_Hash_Tracking","relationType":"preserves"}
{"type":"relation","to":"Basic_Memory_Technical_Design","from":"Basic_Memory_Observation_Hash_Tracking","relationType":"enhances"}
{"type":"relation","from":"Basic_Memory_Repository_Implementation","to":"basic-memory","relationType":"implements_part_of"}
{"type":"relation","from":"Basic_Memory_Repository_Implementation","to":"Basic_Memory_Database_Schema","relationType":"follows"}
{"type":"relation","from":"Basic_Memory_Repository_Implementation","to":"DIY_Ethics","relationType":"aligns_with"}
{"type":"relation","from":"Basic_Memory_Repository_Implementation","to":"testing_infrastructure","relationType":"demonstrates"}
{"type":"relation","from":"Basic_Memory_Repository_Implementation","to":"Basic Foundation","relationType":"inspired_by"}
{"type":"relation","from":"Basic_Memory_Dependencies","to":"basic-memory","relationType":"supports"}
{"type":"relation","from":"Basic_Memory_Dependencies","to":"Basic_Memory_Repository_Implementation","relationType":"enables"}
{"type":"relation","from":"Basic_Memory_Dependencies","to":"testing_infrastructure","relationType":"enables"}
{"type":"relation","from":"Basic_Memory_Current_Architecture","to":"basic-memory","relationType":"describes_state_of"}
{"type":"relation","from":"Basic_Memory_Evolution","to":"Basic_Memory_Current_Architecture","relationType":"explains_development_of"}
{"type":"relation","from":"Basic_Memory_Service_Layer","to":"Basic_Memory_Current_Architecture","relationType":"implements"}
{"type":"relation","from":"Basic_Memory_Schema_Design","to":"Basic_Memory_Current_Architecture","relationType":"implements"}
{"type":"relation","from":"Basic_Memory_Evolution","to":"Basic_Memory_Implementation_Plan","relationType":"reflects_on"}
{"type":"relation","from":"Basic_Memory_Evolution","to":"DIY_Ethics","relationType":"demonstrates_alignment_with"}
{"type":"relation","from":"Basic_Memory_Current_Architecture","to":"DIY_Ethics","relationType":"embodies"}
{"type":"relation","from":"Basic_Memory_Service_Layer","to":"fileio_module","relationType":"uses"}
{"type":"relation","from":"Basic_Memory_Schema_Design","to":"markdown_format","relationType":"implements"}
{"type":"relation","to":"basic-memory","from":"Basic_Memory_Next_Tasks","relationType":"guides_development_of"}
{"type":"relation","to":"DIY_Ethics","from":"Basic_Memory_Next_Tasks","relationType":"aligns_with"}
{"type":"relation","to":"Basic_Memory_Current_Architecture","from":"Basic_Memory_Next_Tasks","relationType":"extends"}
{"type":"relation","from":"Basic_Memory_Meta_Experience","to":"basic-memory","relationType":"validates_design_of"}
{"type":"relation","from":"Basic_Memory_Meta_Experience","to":"DIY_Ethics","relationType":"demonstrates_principles_of"}
{"type":"relation","from":"Basic_Memory_Meta_Experience","to":"design_decisions","relationType":"reinforces"}
{"type":"relation","from":"Basic_Memory_Meta_Experience","to":"Basic_Memory_Current_Architecture","relationType":"validates"}
{"type":"relation","from":"Model_Context_Protocol","to":"basic-memory","relationType":"enables"}
{"type":"relation","from":"basic-memory_core_principles","to":"basic-memory","relationType":"guides"}
{"type":"relation","from":"basic-memory_core_principles","to":"DIY_Ethics","relationType":"aligns_with"}
{"type":"relation","from":"basic-memory_business_model","to":"basic-memory","relationType":"defines_sustainability_for"}
{"type":"relation","from":"basic-memory_business_model","to":"DIY_Ethics","relationType":"maintains_alignment_with"}
{"type":"relation","from":"basic-memory_cli","to":"basic-memory","relationType":"provides_interface_for"}
{"type":"relation","from":"basic-memory_cli","to":"Model_Context_Protocol","relationType":"integrates_with"}
{"type":"relation","from":"basic-memory_export_format","to":"basic-memory","relationType":"standardizes_output_of"}
{"type":"relation","from":"basic-memory_export_format","to":"markdown_format","relationType":"extends"}
{"type":"relation","from":"basic-memory_core_principles","to":"Basic_Machines_Philosophy","relationType":"implements"}
{"type":"relation","from":"Model_Context_Protocol","to":"AI_Human_Collaboration_Model","relationType":"enables"}
{"type":"relation","from":"relation_service","to":"basic-memory","relationType":"will_be_component_of"}
{"type":"relation","from":"relation_service","to":"service_layer_patterns","relationType":"follows"}
{"type":"relation","from":"relation_service","to":"fileio_patterns","relationType":"uses"}
{"type":"relation","from":"relation_service","to":"database_models","relationType":"uses"}
{"type":"relation","from":"relation_service","to":"repository_patterns","relationType":"implements"}
{"type":"relation","from":"relation_service_design","to":"relation_service","relationType":"guides_implementation_of"}
{"type":"relation","from":"relation_service_implementation_plan","to":"relation_service","relationType":"defines_implementation_of"}
{"type":"relation","from":"relation_service_challenges","to":"relation_service_design","relationType":"informs"}
{"type":"relation","from":"relation_file_format","to":"markdown_format","relationType":"extends"}
{"type":"relation","from":"relation_service_error_handling","to":"service_layer_patterns","relationType":"implements"}
{"type":"relation","from":"relation_service_testing","to":"testing_infrastructure","relationType":"extends"}
{"type":"relation","from":"fileio_patterns","to":"service_layer_patterns","relationType":"enables"}
{"type":"relation","from":"database_models","to":"repository_patterns","relationType":"enables"}
{"type":"relation","from":"relation_service","to":"entity_service","relationType":"coordinates_with"}
{"type":"relation","from":"relation_file_format","to":"relation_service","relationType":"defines_storage_for"}
{"type":"relation","from":"relation_service_error_handling","to":"relation_service","relationType":"ensures_reliability_of"}
{"type":"relation","from":"relation_service_testing","to":"relation_service","relationType":"verifies"}
{"type":"relation","from":"service_layer_patterns","to":"basic-memory_implementation_patterns","relationType":"implements"}
{"type":"relation","from":"repository_patterns","to":"basic-memory_implementation_patterns","relationType":"implements"}
{"type":"relation","from":"fileio_patterns","to":"basic-memory_implementation_patterns","relationType":"implements"}
{"type":"relation","from":"database_models","to":"basic-memory_implementation_patterns","relationType":"implements"}
{"type":"relation","from":"relation_service_challenges","to":"implementation_challenges","relationType":"extends"}
{"type":"relation","from":"relation_service_implementation_plan","to":"future_work","relationType":"details"}
{"type":"relation","from":"relation_service_design","to":"design_decisions","relationType":"aligns_with"}
{"type":"relation","from":"relation_file_format","to":"design_decisions","relationType":"follows"}
{"type":"relation","from":"pytest_patterns","to":"testing_infrastructure","relationType":"extends"}
{"type":"relation","from":"relation_implementation_learnings","to":"basic-memory_implementation_patterns","relationType":"informs"}
{"type":"relation","from":"test_driven_insights","to":"test_driven_development","relationType":"enriches"}
{"type":"relation","from":"meta_development_insights","to":"AI_Human_Collaboration_Model","relationType":"improves"}
{"type":"relation","from":"relation_implementation_learnings","to":"relation_service","relationType":"guides_implementation_of"}
{"type":"relation","from":"pytest_patterns","to":"test_evolution","relationType":"demonstrates"}
{"type":"relation","from":"test_driven_insights","to":"design_decisions","relationType":"influences"}
{"type":"relation","from":"meta_development_insights","to":"architecture_evolution","relationType":"informs"}
{"type":"relation","from":"relation_service","to":"relation_implementation_learnings","relationType":"validates"}
{"type":"relation","from":"test_driven_insights","to":"implementation_challenges","relationType":"helps_solve"}
{"type":"relation","from":"AI_Assistant_Learnings","to":"meta_development_insights","relationType":"enriches"}
{"type":"relation","from":"Effective_Response_Patterns","to":"AI_Assistant_Learnings","relationType":"implements"}
{"type":"relation","from":"AI_Context_Management","to":"AI_Human_Collaboration_Model","relationType":"improves"}
{"type":"relation","from":"AI_Tool_Usage_Patterns","to":"AI_Context_Management","relationType":"enables"}
{"type":"relation","from":"AI_Assistant_Learnings","to":"Basic_Memory_Meta_Experience","relationType":"validates"}
{"type":"relation","from":"AI_Tool_Usage_Patterns","to":"Model_Context_Protocol","relationType":"demonstrates_effective_use_of"}
{"type":"relation","from":"AI_Context_Management","to":"basic-memory","relationType":"validates_design_of"}
{"type":"relation","from":"Effective_Response_Patterns","to":"AI_Human_Development_Methodology","relationType":"refines"}
{"type":"relation","to":"relation_service","from":"relation_service_patterns","relationType":"guides"}
{"type":"relation","to":"test_driven_development","from":"test_driven_insights_relations","relationType":"enriches"}
{"type":"relation","to":"implementation_challenges","from":"relation_service_learnings","relationType":"solves"}
{"type":"relation","to":"basic-memory_implementation_patterns","from":"relation_service_patterns","relationType":"implements"}
{"type":"relation","to":"markdown_format","from":"relation_service_patterns","relationType":"extends"}
{"type":"relation","to":"service_layer_patterns","from":"relation_service_patterns","relationType":"refines"}
{"type":"relation","from":"packaging_learnings","to":"implementation_challenges","relationType":"informs"}
{"type":"relation","from":"packaging_learnings","to":"test_driven_development","relationType":"impacts"}
{"type":"relation","to":"basic-memory","from":"Recent_Implementation_Progress","relationType":"updates_status_of"}
{"type":"relation","to":"future_work","from":"Next_Steps","relationType":"extends"}
{"type":"relation","to":"design_decisions","from":"Development_Practices","relationType":"informs"}
{"type":"relation","to":"packaging_learnings","from":"Development_Practices","relationType":"incorporates"}
{"type":"relation","to":"test_driven_development","from":"Development_Practices","relationType":"refines"}
{"type":"relation","to":"basic-memory_implementation_patterns","from":"Development_Practices","relationType":"enhances"}
{"type":"relation","from":"Basic_Memory_MCP","to":"MCP_Server_Implementation","relationType":"follows"}
{"type":"relation","from":"Basic_Memory_MCP","to":"MCP_Tools","relationType":"uses"}
{"type":"relation","from":"Basic_Memory","to":"MCP_Server_Implementation","relationType":"implements"}
{"type":"relation","from":"Basic_Memory_Testing","to":"Memory_Service_Tests","relationType":"includes"}
{"type":"relation","from":"Basic_Memory_Testing","to":"MCP_Server_Tests","relationType":"includes"}
{"type":"relation","from":"Memory_Service_Tests","to":"Basic_Memory_MCP","relationType":"validates"}
{"type":"relation","from":"MCP_Server_Tests","to":"Basic_Memory_MCP","relationType":"validates"}
{"type":"relation","from":"Service_Interface_Audit","to":"Memory_Service_Refactoring","relationType":"informs"}
{"type":"relation","from":"Memory_Service_Refactoring","to":"Basic_Memory_MCP","relationType":"affects"}
{"type":"relation","from":"Memory_Service_Patterns","to":"Basic_Memory_MCP","relationType":"improves"}
{"type":"relation","from":"Pydantic_Create_Pattern","to":"Memory_Service_Patterns","relationType":"enables"}
{"type":"relation","from":"Pydantic_Create_Pattern","to":"Basic_Memory_MCP","relationType":"improves"}
{"type":"relation","from":"MCP_Marketplace","to":"Basic_Memory_Business","relationType":"enables"}
{"type":"relation","from":"Basic_Memory","to":"MCP_Marketplace","relationType":"could_integrate_with"}
{"type":"relation","from":"Persistence_Of_Vision","to":"Basic_Memory","relationType":"helps_achieve"}
{"type":"relation","from":"Drew","to":"Persistence_Of_Vision","relationType":"conceptualized"}
{"type":"relation","to":"Usage_Recipes","from":"Conversation_Continuity_Pattern","relationType":"is_example_of"}
{"type":"relation","to":"Basic_Memory","from":"Usage_Recipes","relationType":"enhances"}
{"type":"relation","to":"Basic_Memory","from":"Chat_References","relationType":"enhances"}
{"type":"relation","to":"Conversation_Continuity_Pattern","from":"Chat_References","relationType":"implements"}
{"type":"relation","from":"20240307-chat-reference-protocol-test","to":"20240307-chat-reference-protocol","relationType":"continues_from"}
{"type":"relation","from_id":"Run_Tests_Tool_Request","to_id":"Basic_Machines","relationType":"enhances","context":"development workflow improvement"}
{"type":"relation","from_id":"SQLAlchemy_Async_Loading_Pattern","to_id":"basic-memory","relation_type":"improves","context":"database performance and async compatibility"}
{"type":"relation","from_id":"SQLAlchemy_Async_Loading_Pattern","to_id":"Entity","relation_type":"applies_to","context":"relationship loading strategy"}
{"type":"relation","from":"MCP_Reference_Integration","to":"Project_Priorities","relationType":"prioritized_after"}
{"type":"relation","from":"great_observation_loading_saga_20241207","to":"Basic_Memory","relationType":"occurred_in"}
{"type":"relation","from":"great_observation_loading_saga_20241207","to":"SQLAlchemy","relationType":"relates_to"}
{"type":"relation","from":"basic_memory_implementation_20241208","to":"Basic_Memory","relationType":"improves"}
{"type":"relation","from":"great_observation_loading_saga_20241207","to":"basic_memory_implementation_20241208","relationType":"leads_to"}
{"type":"relation","from":"MCP_Dependency_Risk","to":"DIY_Ethics","relationType":"validates"}
{"type":"relation","from":"MCP_Dependency_Risk","to":"basic-memory_core_principles","relationType":"reinforces"}
{"type":"relation","from":"MCP_Dependency_Risk","to":"Basic_Memory_Implementation_Plan","relationType":"influences"}
{"type":"relation","from":"basic_memory_mcp_architecture","to":"basic_memory_project_20241208","relationType":"implements"}
{"type":"relation","from":"basic_memory_sync_considerations","to":"basic_memory_project_20241208","relationType":"influences"}
{"type":"relation","from":"mcp_server_learnings","to":"basic_memory_mcp_architecture","relationType":"informs"}
{"type":"relation","from":"20241208-mcp-tool-refactoring","to":"Basic_Memory_MCP","relationType":"improves"}
{"type":"relation","from":"20241208-mcp-tool-refactoring","to":"Basic_Memory_Implementation_Plan","relationType":"implements"}
+15 -44
View File
@@ -3,7 +3,7 @@ name = "basic-memory"
dynamic = ["version"]
description = "Local-first knowledge management combining Zettelkasten with knowledge graphs"
readme = "README.md"
requires-python = ">=3.12"
requires-python = ">=3.12.1"
license = { text = "AGPL-3.0-or-later" }
authors = [
{ name = "Basic Machines", email = "hello@basic-machines.co" }
@@ -14,8 +14,9 @@ dependencies = [
"typer>=0.9.0",
"aiosqlite>=0.20.0",
"greenlet>=3.1.1",
"pydantic[email,timezone]>=2.12.0",
"mcp>=1.23.1",
"pydantic[email,timezone]>=2.10.3",
"icecream>=2.1.3",
"mcp>=1.2.0",
"pydantic-settings>=2.6.1",
"loguru>=0.7.3",
"pyright>=1.1.390",
@@ -29,29 +30,12 @@ dependencies = [
"alembic>=1.14.1",
"pillow>=11.1.0",
"pybars3>=0.9.7",
"fastmcp==2.12.3", # Pinned - 2.14.x breaks MCP tools visibility (issue #463)
"fastmcp>2.10.0",
"pyjwt>=2.10.1",
"python-dotenv>=1.1.0",
"pytest-aio>=1.9.0",
"aiofiles>=24.1.0", # Optional observability (disabled by default via config)
"asyncpg>=0.30.0",
"nest-asyncio>=1.6.0", # For Alembic migrations with Postgres
"pytest-asyncio>=1.2.0",
"psycopg==3.3.1",
"mdformat>=0.7.22",
"mdformat-gfm>=0.3.7",
"mdformat-frontmatter>=2.0.8",
"sniffio>=1.3.1",
"anyio>=4.10.0",
"httpx>=0.28.0",
]
[project.optional-dependencies]
semantic = [
"fastembed>=0.7.4",
"sqlite-vec>=0.1.6",
"openai>=1.100.2",
]
[project.urls]
Homepage = "https://github.com/basicmachines-co/basic-memory"
@@ -68,25 +52,17 @@ build-backend = "hatchling.build"
[tool.pytest.ini_options]
pythonpath = ["src", "tests"]
addopts = "--cov=basic_memory --cov-report term-missing"
testpaths = ["tests", "test-int"]
addopts = "--cov=basic_memory --cov-report term-missing -ra -q"
testpaths = ["tests"]
asyncio_mode = "strict"
asyncio_default_fixture_loop_scope = "function"
markers = [
"benchmark: Performance benchmark tests (deselect with '-m \"not benchmark\"')",
"slow: Slow-running tests (deselect with '-m \"not slow\"')",
"postgres: Tests that run against Postgres backend (deselect with '-m \"not postgres\"')",
"windows: Windows-specific tests (deselect with '-m \"not windows\"')",
"smoke: Fast end-to-end smoke tests for MCP flows",
"semantic: Tests requiring [semantic] extras (fastembed, sqlite-vec, openai)",
]
[tool.ruff]
line-length = 100
target-version = "py312"
[dependency-groups]
dev = [
[tool.uv]
dev-dependencies = [
"gevent>=24.11.1",
"icecream>=2.1.3",
"pytest>=8.3.4",
@@ -95,11 +71,6 @@ dev = [
"pytest-asyncio>=0.24.0",
"pytest-xdist>=3.0.0",
"ruff>=0.1.6",
"freezegun>=1.5.5",
"testcontainers[postgres]>=4.0.0",
"psycopg>=3.2.0",
"pyright>=1.1.408",
"pytest-testmon>=2.2.0",
]
[tool.hatch.version]
@@ -124,8 +95,6 @@ pythonVersion = "3.12"
[tool.coverage.run]
concurrency = ["thread", "gevent"]
parallel = true
source = ["basic_memory"]
[tool.coverage.report]
exclude_lines = [
@@ -147,9 +116,11 @@ omit = [
"*/supabase_auth_provider.py", # External HTTP calls to Supabase APIs
"*/watch_service.py", # File system watching - complex integration testing
"*/background_sync.py", # Background processes
"*/cli/**", # CLI is an interactive wrapper; core logic is covered via API/MCP/service tests
"*/db.py", # Backend/runtime-dependent (sqlite/postgres/windows tuning); validated via integration tests
"*/services/initialization.py", # Startup orchestration + background tasks (watchers); exercised indirectly in entrypoints
"*/sync/sync_service.py", # Heavy filesystem/db integration; covered by integration suite, not enforced in unit coverage
"*/cli/main.py", # CLI entry point
"*/mcp/tools/project_management.py", # Covered by integration tests
"*/mcp/tools/sync_status.py", # Covered by integration tests
"*/services/migration_service.py", # Complex migration scenarios
]
[tool.logfire]
ignore_no_config = true
-25
View File
@@ -1,25 +0,0 @@
{
"$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json",
"name": "io.github.basicmachines-co/basic-memory",
"description": "Local-first knowledge management with bi-directional LLM sync via Markdown files.",
"repository": {
"url": "https://github.com/basicmachines-co/basic-memory.git",
"source": "github"
},
"version": "0.18.3",
"packages": [
{
"registryType": "pypi",
"identifier": "basic-memory",
"version": "0.18.3",
"runtimeHint": "uvx",
"runtimeArguments": [
{"type": "positional", "value": "basic-memory"},
{"type": "positional", "value": "mcp"}
],
"transport": {
"type": "stdio"
}
}
]
}
+1 -1
View File
@@ -1,7 +1,7 @@
"""basic-memory - Local-first knowledge management combining Zettelkasten with knowledge graphs"""
# Package version - updated by release automation
__version__ = "0.18.3"
__version__ = "0.14.1"
# API version for FastAPI - independent of package version
__api_version__ = "v0"
+25 -117
View File
@@ -1,60 +1,29 @@
"""Alembic environment configuration."""
import asyncio
import os
from logging.config import fileConfig
# Allow nested event loops (needed for pytest-asyncio and other async contexts)
# Note: nest_asyncio doesn't work with uvloop or Python 3.14+, so we handle those cases separately
import sys
if sys.version_info < (3, 14):
try:
import nest_asyncio
nest_asyncio.apply()
except (ImportError, ValueError):
# nest_asyncio not available or can't patch this loop type (e.g., uvloop)
pass
# For Python 3.14+, we rely on the thread-based fallback in run_migrations_online()
from sqlalchemy import engine_from_config, pool
from sqlalchemy.ext.asyncio import AsyncEngine, create_async_engine
from sqlalchemy import engine_from_config
from sqlalchemy import pool
from alembic import context
from basic_memory.config import ConfigManager
# Trigger: only set test env when actually running under pytest
# Why: alembic/env.py is imported during normal operations (MCP server startup, migrations)
# but we only want test behavior during actual test runs
# Outcome: prevents is_test_env from returning True in production, enabling watch service
if os.getenv("PYTEST_CURRENT_TEST") is not None:
os.environ["BASIC_MEMORY_ENV"] = "test"
# set config.env to "test" for pytest to prevent logging to file in utils.setup_logging()
os.environ["BASIC_MEMORY_ENV"] = "test"
# Import after setting environment variable # noqa: E402
from basic_memory.config import app_config # noqa: E402
from basic_memory.models import Base # noqa: E402
# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config
# Load app config - this will read environment variables (BASIC_MEMORY_DATABASE_BACKEND, etc.)
# due to Pydantic's env_prefix="BASIC_MEMORY_" setting
app_config = ConfigManager().config
# Set the SQLAlchemy URL from our app config
sqlalchemy_url = f"sqlite:///{app_config.database_path}"
config.set_main_option("sqlalchemy.url", sqlalchemy_url)
# Set the SQLAlchemy URL based on database backend configuration
# If the URL is already set in config (e.g., from run_migrations), use that
# Otherwise, get it from app config
# Note: alembic.ini has a placeholder URL "driver://user:pass@localhost/dbname" that we need to override
current_url = config.get_main_option("sqlalchemy.url")
if not current_url or current_url == "driver://user:pass@localhost/dbname":
from basic_memory.db import DatabaseType
sqlalchemy_url = DatabaseType.get_db_url(
app_config.database_path, DatabaseType.FILESYSTEM, app_config
)
config.set_main_option("sqlalchemy.url", sqlalchemy_url)
# print(f"Using SQLAlchemy URL: {sqlalchemy_url}")
# Interpret the config file for Python logging.
if config.config_file_name is not None:
@@ -98,89 +67,28 @@ def run_migrations_offline() -> None:
context.run_migrations()
def do_run_migrations(connection):
"""Execute migrations with the given connection."""
context.configure(
connection=connection,
target_metadata=target_metadata,
include_object=include_object,
render_as_batch=True,
compare_type=True,
)
with context.begin_transaction():
context.run_migrations()
async def run_async_migrations(connectable):
"""Run migrations asynchronously with AsyncEngine."""
async with connectable.connect() as connection:
await connection.run_sync(do_run_migrations)
await connectable.dispose()
def run_migrations_online() -> None:
"""Run migrations in 'online' mode.
Supports both sync engines (SQLite) and async engines (PostgreSQL with asyncpg).
In this scenario we need to create an Engine
and associate a connection with the context.
"""
# Check if a connection/engine was provided (e.g., from run_migrations)
connectable = context.config.attributes.get("connection", None)
connectable = engine_from_config(
config.get_section(config.config_ini_section, {}),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
if connectable is None:
# No connection provided, create engine from config
url = context.config.get_main_option("sqlalchemy.url")
with connectable.connect() as connection:
context.configure(
connection=connection,
target_metadata=target_metadata,
include_object=include_object,
render_as_batch=True,
)
# Check if it's an async URL (sqlite+aiosqlite or postgresql+asyncpg)
if url and ("+asyncpg" in url or "+aiosqlite" in url):
# Create async engine for asyncpg or aiosqlite
connectable = create_async_engine(
url,
poolclass=pool.NullPool,
future=True,
)
else:
# Create sync engine for regular sqlite or postgresql
connectable = engine_from_config(
context.config.get_section(context.config.config_ini_section, {}),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
# Handle async engines (PostgreSQL with asyncpg)
if isinstance(connectable, AsyncEngine):
# Try to run async migrations
# nest_asyncio allows asyncio.run() from within event loops, but doesn't work with uvloop
try:
asyncio.run(run_async_migrations(connectable))
except RuntimeError as e:
if "cannot be called from a running event loop" in str(e):
# We're in a running event loop (likely uvloop) - need to use a different approach
# Create a new thread to run the async migrations
import concurrent.futures
def run_in_thread():
"""Run async migrations in a new event loop in a separate thread."""
new_loop = asyncio.new_event_loop()
asyncio.set_event_loop(new_loop)
try:
new_loop.run_until_complete(run_async_migrations(connectable))
finally:
new_loop.close()
with concurrent.futures.ThreadPoolExecutor() as executor:
future = executor.submit(run_in_thread)
future.result() # Wait for completion and re-raise any exceptions
else:
raise
else:
# Handle sync engines (SQLite) or sync connections
if hasattr(connectable, "connect"):
# It's an engine, get a connection
with connectable.connect() as connection:
do_run_migrations(connection)
else:
# It's already a connection
do_run_migrations(connectable)
with context.begin_transaction():
context.run_migrations()
if context.is_offline_mode():
@@ -1,131 +0,0 @@
"""Add Postgres full-text search support with tsvector and GIN indexes
Revision ID: 314f1ea54dc4
Revises: e7e1f4367280
Create Date: 2025-11-15 18:05:01.025405
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = "314f1ea54dc4"
down_revision: Union[str, None] = "e7e1f4367280"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Add PostgreSQL full-text search support.
This migration:
1. Creates search_index table for Postgres (SQLite uses FTS5 virtual table)
2. Adds generated tsvector column for full-text search
3. Creates GIN index on the tsvector column for fast text queries
4. Creates GIN index on metadata JSONB column for fast containment queries
Note: These changes only apply to Postgres. SQLite continues to use FTS5 virtual tables.
"""
# Check if we're using Postgres
connection = op.get_bind()
if connection.dialect.name == "postgresql":
# Create search_index table for Postgres
# For SQLite, this is a FTS5 virtual table created elsewhere
from sqlalchemy.dialects.postgresql import JSONB
op.create_table(
"search_index",
sa.Column("id", sa.Integer(), nullable=False), # Entity IDs are integers
sa.Column("project_id", sa.Integer(), nullable=False), # Multi-tenant isolation
sa.Column("title", sa.Text(), nullable=True),
sa.Column("content_stems", sa.Text(), nullable=True),
sa.Column("content_snippet", sa.Text(), nullable=True),
sa.Column("permalink", sa.String(), nullable=True), # Nullable for non-markdown files
sa.Column("file_path", sa.String(), nullable=True),
sa.Column("type", sa.String(), nullable=True),
sa.Column("from_id", sa.Integer(), nullable=True), # Relation IDs are integers
sa.Column("to_id", sa.Integer(), nullable=True), # Relation IDs are integers
sa.Column("relation_type", sa.String(), nullable=True),
sa.Column("entity_id", sa.Integer(), nullable=True), # Entity IDs are integers
sa.Column("category", sa.String(), nullable=True),
sa.Column("metadata", JSONB(), nullable=True), # Use JSONB for Postgres
sa.Column("created_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=True),
sa.PrimaryKeyConstraint(
"id", "type", "project_id"
), # Composite key: id can repeat across types
sa.ForeignKeyConstraint(
["project_id"],
["project.id"],
name="fk_search_index_project_id",
ondelete="CASCADE",
),
if_not_exists=True,
)
# Create index on project_id for efficient multi-tenant queries
op.create_index(
"ix_search_index_project_id",
"search_index",
["project_id"],
unique=False,
)
# Create unique partial index on permalink for markdown files
# Non-markdown files don't have permalinks, so we use a partial index
op.execute("""
CREATE UNIQUE INDEX uix_search_index_permalink_project
ON search_index (permalink, project_id)
WHERE permalink IS NOT NULL
""")
# Add tsvector column as a GENERATED ALWAYS column
# This automatically updates when title or content_stems change
op.execute("""
ALTER TABLE search_index
ADD COLUMN textsearchable_index_col tsvector
GENERATED ALWAYS AS (
to_tsvector('english',
coalesce(title, '') || ' ' ||
coalesce(content_stems, '')
)
) STORED
""")
# Create GIN index on tsvector column for fast full-text search
op.create_index(
"idx_search_index_fts",
"search_index",
["textsearchable_index_col"],
unique=False,
postgresql_using="gin",
)
# Create GIN index on metadata JSONB for fast containment queries
# Using jsonb_path_ops for smaller index size and better performance
op.execute("""
CREATE INDEX idx_search_index_metadata_gin
ON search_index
USING GIN (metadata jsonb_path_ops)
""")
def downgrade() -> None:
"""Remove PostgreSQL full-text search support."""
connection = op.get_bind()
if connection.dialect.name == "postgresql":
# Drop indexes first
op.execute("DROP INDEX IF EXISTS idx_search_index_metadata_gin")
op.drop_index("idx_search_index_fts", table_name="search_index")
op.execute("DROP INDEX IF EXISTS uix_search_index_permalink_project")
op.drop_index("ix_search_index_project_id", table_name="search_index")
# Drop the generated column
op.execute("ALTER TABLE search_index DROP COLUMN IF EXISTS textsearchable_index_col")
# Drop the search_index table
op.drop_table("search_index")
@@ -21,12 +21,6 @@ depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
# SQLite FTS5 virtual table handling is SQLite-specific
# For Postgres, search_index is a regular table managed by ORM
connection = op.get_bind()
is_sqlite = connection.dialect.name == "sqlite"
op.create_table(
"project",
sa.Column("id", sa.Integer(), nullable=False),
@@ -61,9 +55,7 @@ def upgrade() -> None:
batch_op.add_column(sa.Column("project_id", sa.Integer(), nullable=False))
batch_op.drop_index(
"uix_entity_permalink",
sqlite_where=sa.text("content_type = 'text/markdown' AND permalink IS NOT NULL")
if is_sqlite
else None,
sqlite_where=sa.text("content_type = 'text/markdown' AND permalink IS NOT NULL"),
)
batch_op.drop_index("ix_entity_file_path")
batch_op.create_index(batch_op.f("ix_entity_file_path"), ["file_path"], unique=False)
@@ -75,16 +67,12 @@ def upgrade() -> None:
"uix_entity_permalink_project",
["permalink", "project_id"],
unique=True,
sqlite_where=sa.text("content_type = 'text/markdown' AND permalink IS NOT NULL")
if is_sqlite
else None,
sqlite_where=sa.text("content_type = 'text/markdown' AND permalink IS NOT NULL"),
)
batch_op.create_foreign_key("fk_entity_project_id", "project", ["project_id"], ["id"])
# drop the search index table. it will be recreated
# Only drop for SQLite - Postgres uses regular table managed by ORM
if is_sqlite:
op.drop_table("search_index")
op.drop_table("search_index")
# ### end Alembic commands ###
@@ -25,51 +25,43 @@ def upgrade() -> None:
The UNIQUE constraint prevents multiple projects from having is_default=FALSE,
which breaks project creation when the service sets is_default=False.
SQLite: Recreate the table without the constraint (no ALTER TABLE support)
Postgres: Use ALTER TABLE to drop the constraint directly
Since SQLite doesn't support dropping specific constraints easily, we'll
recreate the table without the problematic constraint.
"""
connection = op.get_bind()
is_sqlite = connection.dialect.name == "sqlite"
# For SQLite, we need to recreate the table without the UNIQUE constraint
# Create a new table without the UNIQUE constraint on is_default
op.create_table(
"project_new",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("name", sa.String(), nullable=False),
sa.Column("description", sa.Text(), nullable=True),
sa.Column("permalink", sa.String(), nullable=False),
sa.Column("path", sa.String(), nullable=False),
sa.Column("is_active", sa.Boolean(), nullable=False),
sa.Column("is_default", sa.Boolean(), nullable=True), # No UNIQUE constraint!
sa.Column("created_at", sa.DateTime(), nullable=False),
sa.Column("updated_at", sa.DateTime(), nullable=False),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("name"),
sa.UniqueConstraint("permalink"),
)
if is_sqlite:
# For SQLite, we need to recreate the table without the UNIQUE constraint
# Create a new table without the UNIQUE constraint on is_default
op.create_table(
"project_new",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("name", sa.String(), nullable=False),
sa.Column("description", sa.Text(), nullable=True),
sa.Column("permalink", sa.String(), nullable=False),
sa.Column("path", sa.String(), nullable=False),
sa.Column("is_active", sa.Boolean(), nullable=False),
sa.Column("is_default", sa.Boolean(), nullable=True), # No UNIQUE constraint!
sa.Column("created_at", sa.DateTime(), nullable=False),
sa.Column("updated_at", sa.DateTime(), nullable=False),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("name"),
sa.UniqueConstraint("permalink"),
)
# Copy data from old table to new table
op.execute("INSERT INTO project_new SELECT * FROM project")
# Copy data from old table to new table
op.execute("INSERT INTO project_new SELECT * FROM project")
# Drop the old table
op.drop_table("project")
# Drop the old table
op.drop_table("project")
# Rename the new table
op.rename_table("project_new", "project")
# Rename the new table
op.rename_table("project_new", "project")
# Recreate the indexes
with op.batch_alter_table("project", schema=None) as batch_op:
batch_op.create_index("ix_project_created_at", ["created_at"], unique=False)
batch_op.create_index("ix_project_name", ["name"], unique=True)
batch_op.create_index("ix_project_path", ["path"], unique=False)
batch_op.create_index("ix_project_permalink", ["permalink"], unique=True)
batch_op.create_index("ix_project_updated_at", ["updated_at"], unique=False)
else:
# For Postgres, we can simply drop the constraint
with op.batch_alter_table("project", schema=None) as batch_op:
batch_op.drop_constraint("project_is_default_key", type_="unique")
# Recreate the indexes
with op.batch_alter_table("project", schema=None) as batch_op:
batch_op.create_index("ix_project_created_at", ["created_at"], unique=False)
batch_op.create_index("ix_project_name", ["name"], unique=True)
batch_op.create_index("ix_project_path", ["path"], unique=False)
batch_op.create_index("ix_project_permalink", ["permalink"], unique=True)
batch_op.create_index("ix_project_updated_at", ["updated_at"], unique=False)
def downgrade() -> None:
@@ -1,24 +0,0 @@
"""Merge multiple heads
Revision ID: 6830751f5fb6
Revises: a2b3c4d5e6f7, g9a0b3c4d5e6
Create Date: 2025-12-29 12:46:46.476268
"""
from typing import Sequence, Union
# revision identifiers, used by Alembic.
revision: str = "6830751f5fb6"
down_revision: Union[str, Sequence[str], None] = ("a2b3c4d5e6f7", "g9a0b3c4d5e6")
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
@@ -1,49 +0,0 @@
"""Add mtime and size columns to Entity for sync optimization
Revision ID: 9d9c1cb7d8f5
Revises: a1b2c3d4e5f6
Create Date: 2025-10-20 05:07:55.173849
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = "9d9c1cb7d8f5"
down_revision: Union[str, None] = "a1b2c3d4e5f6"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table("entity", schema=None) as batch_op:
batch_op.add_column(sa.Column("mtime", sa.Float(), nullable=True))
batch_op.add_column(sa.Column("size", sa.Integer(), nullable=True))
batch_op.drop_constraint(batch_op.f("fk_entity_project_id"), type_="foreignkey")
batch_op.create_foreign_key(
batch_op.f("fk_entity_project_id"), "project", ["project_id"], ["id"]
)
# ### end Alembic commands ###
def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table("entity", schema=None) as batch_op:
batch_op.drop_constraint(batch_op.f("fk_entity_project_id"), type_="foreignkey")
batch_op.create_foreign_key(
batch_op.f("fk_entity_project_id"),
"project",
["project_id"],
["id"],
ondelete="CASCADE",
)
batch_op.drop_column("size")
batch_op.drop_column("mtime")
# ### end Alembic commands ###
@@ -1,49 +0,0 @@
"""fix project foreign keys
Revision ID: a1b2c3d4e5f6
Revises: 647e7a75e2cd
Create Date: 2025-08-19 22:06:00.000000
"""
from typing import Sequence, Union
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "a1b2c3d4e5f6"
down_revision: Union[str, None] = "647e7a75e2cd"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Re-establish foreign key constraints that were lost during project table recreation.
The migration 647e7a75e2cd recreated the project table but did not re-establish
the foreign key constraint from entity.project_id to project.id, causing
foreign key constraint failures when trying to delete projects with related entities.
"""
# SQLite doesn't allow adding foreign key constraints to existing tables easily
# We need to be careful and handle the case where the constraint might already exist
with op.batch_alter_table("entity", schema=None) as batch_op:
# Try to drop existing foreign key constraint (may not exist)
try:
batch_op.drop_constraint("fk_entity_project_id", type_="foreignkey")
except Exception:
# Constraint may not exist, which is fine - we'll create it next
pass
# Add the foreign key constraint with CASCADE DELETE
# This ensures that when a project is deleted, all related entities are also deleted
batch_op.create_foreign_key(
"fk_entity_project_id", "project", ["project_id"], ["id"], ondelete="CASCADE"
)
def downgrade() -> None:
"""Remove the foreign key constraint."""
with op.batch_alter_table("entity", schema=None) as batch_op:
batch_op.drop_constraint("fk_entity_project_id", type_="foreignkey")
@@ -1,56 +0,0 @@
"""Add cascade delete FK from search_index to entity
Revision ID: a2b3c4d5e6f7
Revises: f8a9b2c3d4e5
Create Date: 2025-12-02 07:00:00.000000
"""
from typing import Sequence, Union
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "a2b3c4d5e6f7"
down_revision: Union[str, None] = "f8a9b2c3d4e5"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Add FK with CASCADE delete from search_index.entity_id to entity.id.
This migration is Postgres-only because:
- SQLite uses FTS5 virtual tables which don't support foreign keys
- The FK enables automatic cleanup of search_index entries when entities are deleted
"""
connection = op.get_bind()
dialect = connection.dialect.name
if dialect == "postgresql":
# First, clean up any orphaned search_index entries where entity no longer exists
op.execute("""
DELETE FROM search_index
WHERE entity_id IS NOT NULL
AND entity_id NOT IN (SELECT id FROM entity)
""")
# Add FK with CASCADE - nullable FK allows search_index entries without entity_id
op.create_foreign_key(
"fk_search_index_entity_id",
"search_index",
"entity",
["entity_id"],
["id"],
ondelete="CASCADE",
)
def downgrade() -> None:
"""Remove the FK constraint."""
connection = op.get_bind()
dialect = connection.dialect.name
if dialect == "postgresql":
op.drop_constraint("fk_search_index_entity_id", "search_index", type_="foreignkey")
@@ -21,12 +21,6 @@ depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Upgrade database schema to use new search index with content_stems and content_snippet."""
# This migration is SQLite-specific (FTS5 virtual tables)
# For Postgres, the search_index table is created via ORM models
connection = op.get_bind()
if connection.dialect.name != "sqlite":
return
# First, drop the existing search_index table
op.execute("DROP TABLE IF EXISTS search_index")
@@ -65,13 +59,6 @@ def upgrade() -> None:
def downgrade() -> None:
"""Downgrade database schema to use old search index."""
# This migration is SQLite-specific (FTS5 virtual tables)
# For Postgres, the search_index table is managed via ORM models
connection = op.get_bind()
if connection.dialect.name != "sqlite":
return
# Drop the updated search_index table
op.execute("DROP TABLE IF EXISTS search_index")
@@ -1,154 +0,0 @@
"""Add structured metadata indexes for entity frontmatter
Revision ID: d7e8f9a0b1c2
Revises: g9a0b3c4d5e6
Create Date: 2026-01-31 12:00:00.000000
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy import text
def column_exists(connection, table: str, column: str) -> bool:
"""Check if a column exists in a table (idempotent migration support)."""
if connection.dialect.name == "postgresql":
result = connection.execute(
text(
"SELECT 1 FROM information_schema.columns "
"WHERE table_name = :table AND column_name = :column"
),
{"table": table, "column": column},
)
return result.fetchone() is not None
# SQLite
result = connection.execute(text(f"PRAGMA table_info({table})"))
columns = [row[1] for row in result]
return column in columns
def index_exists(connection, index_name: str) -> bool:
"""Check if an index exists (idempotent migration support)."""
if connection.dialect.name == "postgresql":
result = connection.execute(
text("SELECT 1 FROM pg_indexes WHERE indexname = :index_name"),
{"index_name": index_name},
)
return result.fetchone() is not None
# SQLite
result = connection.execute(
text("SELECT 1 FROM sqlite_master WHERE type='index' AND name = :index_name"),
{"index_name": index_name},
)
return result.fetchone() is not None
# revision identifiers, used by Alembic.
revision: str = "d7e8f9a0b1c2"
down_revision: Union[str, None] = "6830751f5fb6"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Add JSONB/GiN indexes for Postgres and generated columns for SQLite."""
connection = op.get_bind()
dialect = connection.dialect.name
if dialect == "postgresql":
# Ensure JSONB for efficient indexing
result = connection.execute(
text(
"SELECT data_type FROM information_schema.columns "
"WHERE table_name = 'entity' AND column_name = 'entity_metadata'"
)
).fetchone()
if result and result[0] != "jsonb":
op.execute(
"ALTER TABLE entity ALTER COLUMN entity_metadata "
"TYPE jsonb USING entity_metadata::jsonb"
)
# General JSONB GIN index
op.execute(
"CREATE INDEX IF NOT EXISTS idx_entity_metadata_gin "
"ON entity USING GIN (entity_metadata jsonb_path_ops)"
)
# Common field indexes
op.execute(
"CREATE INDEX IF NOT EXISTS idx_entity_tags_json "
"ON entity USING GIN ((entity_metadata -> 'tags'))"
)
op.execute(
"CREATE INDEX IF NOT EXISTS idx_entity_frontmatter_type "
"ON entity ((entity_metadata ->> 'type'))"
)
op.execute(
"CREATE INDEX IF NOT EXISTS idx_entity_frontmatter_status "
"ON entity ((entity_metadata ->> 'status'))"
)
return
# SQLite: add generated columns for common frontmatter fields
# Constraint: SQLite ALTER TABLE ADD COLUMN only supports VIRTUAL generated columns,
# not STORED. json_extract is deterministic so VIRTUAL columns can still be indexed.
if not column_exists(connection, "entity", "tags_json"):
op.add_column(
"entity",
sa.Column(
"tags_json",
sa.Text(),
sa.Computed("json_extract(entity_metadata, '$.tags')", persisted=False),
),
)
if not column_exists(connection, "entity", "frontmatter_status"):
op.add_column(
"entity",
sa.Column(
"frontmatter_status",
sa.Text(),
sa.Computed("json_extract(entity_metadata, '$.status')", persisted=False),
),
)
if not column_exists(connection, "entity", "frontmatter_type"):
op.add_column(
"entity",
sa.Column(
"frontmatter_type",
sa.Text(),
sa.Computed("json_extract(entity_metadata, '$.type')", persisted=False),
),
)
# Index generated columns
if not index_exists(connection, "idx_entity_tags_json"):
op.create_index("idx_entity_tags_json", "entity", ["tags_json"])
if not index_exists(connection, "idx_entity_frontmatter_status"):
op.create_index("idx_entity_frontmatter_status", "entity", ["frontmatter_status"])
if not index_exists(connection, "idx_entity_frontmatter_type"):
op.create_index("idx_entity_frontmatter_type", "entity", ["frontmatter_type"])
def downgrade() -> None:
"""Best-effort downgrade (drop indexes, revert JSONB on Postgres)."""
connection = op.get_bind()
dialect = connection.dialect.name
if dialect == "postgresql":
op.execute("DROP INDEX IF EXISTS idx_entity_frontmatter_status")
op.execute("DROP INDEX IF EXISTS idx_entity_frontmatter_type")
op.execute("DROP INDEX IF EXISTS idx_entity_tags_json")
op.execute("DROP INDEX IF EXISTS idx_entity_metadata_gin")
op.execute(
"ALTER TABLE entity ALTER COLUMN entity_metadata TYPE json USING entity_metadata::json"
)
return
# SQLite: drop indexes (dropping generated columns requires table rebuild)
op.execute("DROP INDEX IF EXISTS idx_entity_frontmatter_status")
op.execute("DROP INDEX IF EXISTS idx_entity_frontmatter_type")
op.execute("DROP INDEX IF EXISTS idx_entity_tags_json")
@@ -1,37 +0,0 @@
"""Add scan watermark tracking to Project
Revision ID: e7e1f4367280
Revises: 9d9c1cb7d8f5
Create Date: 2025-10-20 16:42:46.625075
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = "e7e1f4367280"
down_revision: Union[str, None] = "9d9c1cb7d8f5"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table("project", schema=None) as batch_op:
batch_op.add_column(sa.Column("last_scan_timestamp", sa.Float(), nullable=True))
batch_op.add_column(sa.Column("last_file_count", sa.Integer(), nullable=True))
# ### end Alembic commands ###
def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table("project", schema=None) as batch_op:
batch_op.drop_column("last_file_count")
batch_op.drop_column("last_scan_timestamp")
# ### end Alembic commands ###
@@ -1,239 +0,0 @@
"""Add project_id to relation/observation and pg_trgm for fuzzy link resolution
Revision ID: f8a9b2c3d4e5
Revises: 314f1ea54dc4
Create Date: 2025-12-01 12:00:00.000000
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy import text
def column_exists(connection, table: str, column: str) -> bool:
"""Check if a column exists in a table (idempotent migration support)."""
if connection.dialect.name == "postgresql":
result = connection.execute(
text(
"SELECT 1 FROM information_schema.columns "
"WHERE table_name = :table AND column_name = :column"
),
{"table": table, "column": column},
)
return result.fetchone() is not None
else:
# SQLite
result = connection.execute(text(f"PRAGMA table_info({table})"))
columns = [row[1] for row in result]
return column in columns
def index_exists(connection, index_name: str) -> bool:
"""Check if an index exists (idempotent migration support)."""
if connection.dialect.name == "postgresql":
result = connection.execute(
text("SELECT 1 FROM pg_indexes WHERE indexname = :index_name"),
{"index_name": index_name},
)
return result.fetchone() is not None
else:
# SQLite
result = connection.execute(
text("SELECT 1 FROM sqlite_master WHERE type='index' AND name = :index_name"),
{"index_name": index_name},
)
return result.fetchone() is not None
# revision identifiers, used by Alembic.
revision: str = "f8a9b2c3d4e5"
down_revision: Union[str, None] = "314f1ea54dc4"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Add project_id to relation and observation tables, plus pg_trgm indexes.
This migration:
1. Adds project_id column to relation and observation tables (denormalization)
2. Backfills project_id from the associated entity
3. Enables pg_trgm extension for trigram-based fuzzy matching (Postgres only)
4. Creates GIN indexes on entity title and permalink for fast similarity searches
5. Creates partial index on unresolved relations for efficient bulk resolution
"""
connection = op.get_bind()
dialect = connection.dialect.name
# -------------------------------------------------------------------------
# Add project_id to relation table
# -------------------------------------------------------------------------
# Step 1: Add project_id column as nullable first (idempotent)
if not column_exists(connection, "relation", "project_id"):
op.add_column("relation", sa.Column("project_id", sa.Integer(), nullable=True))
# Step 2: Backfill project_id from entity.project_id via from_id
if dialect == "postgresql":
op.execute("""
UPDATE relation
SET project_id = entity.project_id
FROM entity
WHERE relation.from_id = entity.id
""")
else:
# SQLite syntax
op.execute("""
UPDATE relation
SET project_id = (
SELECT entity.project_id
FROM entity
WHERE entity.id = relation.from_id
)
""")
# Step 3: Make project_id NOT NULL and add foreign key
if dialect == "postgresql":
op.alter_column("relation", "project_id", nullable=False)
op.create_foreign_key(
"fk_relation_project_id",
"relation",
"project",
["project_id"],
["id"],
)
else:
# SQLite requires batch operations for ALTER COLUMN
with op.batch_alter_table("relation") as batch_op:
batch_op.alter_column("project_id", nullable=False)
batch_op.create_foreign_key(
"fk_relation_project_id",
"project",
["project_id"],
["id"],
)
# Step 4: Create index on relation.project_id (idempotent)
if not index_exists(connection, "ix_relation_project_id"):
op.create_index("ix_relation_project_id", "relation", ["project_id"])
# -------------------------------------------------------------------------
# Add project_id to observation table
# -------------------------------------------------------------------------
# Step 1: Add project_id column as nullable first (idempotent)
if not column_exists(connection, "observation", "project_id"):
op.add_column("observation", sa.Column("project_id", sa.Integer(), nullable=True))
# Step 2: Backfill project_id from entity.project_id via entity_id
if dialect == "postgresql":
op.execute("""
UPDATE observation
SET project_id = entity.project_id
FROM entity
WHERE observation.entity_id = entity.id
""")
else:
# SQLite syntax
op.execute("""
UPDATE observation
SET project_id = (
SELECT entity.project_id
FROM entity
WHERE entity.id = observation.entity_id
)
""")
# Step 3: Make project_id NOT NULL and add foreign key
if dialect == "postgresql":
op.alter_column("observation", "project_id", nullable=False)
op.create_foreign_key(
"fk_observation_project_id",
"observation",
"project",
["project_id"],
["id"],
)
else:
# SQLite requires batch operations for ALTER COLUMN
with op.batch_alter_table("observation") as batch_op:
batch_op.alter_column("project_id", nullable=False)
batch_op.create_foreign_key(
"fk_observation_project_id",
"project",
["project_id"],
["id"],
)
# Step 4: Create index on observation.project_id (idempotent)
if not index_exists(connection, "ix_observation_project_id"):
op.create_index("ix_observation_project_id", "observation", ["project_id"])
# Postgres-specific: pg_trgm and GIN indexes
if dialect == "postgresql":
# Enable pg_trgm extension for fuzzy string matching
op.execute("CREATE EXTENSION IF NOT EXISTS pg_trgm")
# Create trigram indexes on entity table for fuzzy matching
# GIN indexes with gin_trgm_ops support similarity searches
op.execute("""
CREATE INDEX IF NOT EXISTS idx_entity_title_trgm
ON entity USING gin (title gin_trgm_ops)
""")
op.execute("""
CREATE INDEX IF NOT EXISTS idx_entity_permalink_trgm
ON entity USING gin (permalink gin_trgm_ops)
""")
# Create partial index on unresolved relations for efficient bulk resolution
# This makes "WHERE to_id IS NULL AND project_id = X" queries very fast
op.execute("""
CREATE INDEX IF NOT EXISTS idx_relation_unresolved
ON relation (project_id, to_name)
WHERE to_id IS NULL
""")
# Create index on relation.to_name for join performance in bulk resolution
op.execute("""
CREATE INDEX IF NOT EXISTS idx_relation_to_name
ON relation (to_name)
""")
def downgrade() -> None:
"""Remove project_id from relation/observation and pg_trgm indexes."""
connection = op.get_bind()
dialect = connection.dialect.name
if dialect == "postgresql":
# Drop Postgres-specific indexes
op.execute("DROP INDEX IF EXISTS idx_relation_to_name")
op.execute("DROP INDEX IF EXISTS idx_relation_unresolved")
op.execute("DROP INDEX IF EXISTS idx_entity_permalink_trgm")
op.execute("DROP INDEX IF EXISTS idx_entity_title_trgm")
# Note: We don't drop the pg_trgm extension as other code may depend on it
# Drop project_id from observation
op.drop_index("ix_observation_project_id", table_name="observation")
op.drop_constraint("fk_observation_project_id", "observation", type_="foreignkey")
op.drop_column("observation", "project_id")
# Drop project_id from relation
op.drop_index("ix_relation_project_id", table_name="relation")
op.drop_constraint("fk_relation_project_id", "relation", type_="foreignkey")
op.drop_column("relation", "project_id")
else:
# SQLite requires batch operations
op.drop_index("ix_observation_project_id", table_name="observation")
with op.batch_alter_table("observation") as batch_op:
batch_op.drop_constraint("fk_observation_project_id", type_="foreignkey")
batch_op.drop_column("project_id")
op.drop_index("ix_relation_project_id", table_name="relation")
with op.batch_alter_table("relation") as batch_op:
batch_op.drop_constraint("fk_relation_project_id", type_="foreignkey")
batch_op.drop_column("project_id")
@@ -1,173 +0,0 @@
"""Add external_id UUID column to project and entity tables
Revision ID: g9a0b3c4d5e6
Revises: f8a9b2c3d4e5
Create Date: 2025-12-29 10:00:00.000000
"""
import uuid
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy import text
def column_exists(connection, table: str, column: str) -> bool:
"""Check if a column exists in a table (idempotent migration support)."""
if connection.dialect.name == "postgresql":
result = connection.execute(
text(
"SELECT 1 FROM information_schema.columns "
"WHERE table_name = :table AND column_name = :column"
),
{"table": table, "column": column},
)
return result.fetchone() is not None
else:
# SQLite
result = connection.execute(text(f"PRAGMA table_info({table})"))
columns = [row[1] for row in result]
return column in columns
def index_exists(connection, index_name: str) -> bool:
"""Check if an index exists (idempotent migration support)."""
if connection.dialect.name == "postgresql":
result = connection.execute(
text("SELECT 1 FROM pg_indexes WHERE indexname = :index_name"),
{"index_name": index_name},
)
return result.fetchone() is not None
else:
# SQLite
result = connection.execute(
text("SELECT 1 FROM sqlite_master WHERE type='index' AND name = :index_name"),
{"index_name": index_name},
)
return result.fetchone() is not None
# revision identifiers, used by Alembic.
revision: str = "g9a0b3c4d5e6"
down_revision: Union[str, None] = "f8a9b2c3d4e5"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Add external_id UUID column to project and entity tables.
This migration:
1. Adds external_id column to project table
2. Adds external_id column to entity table
3. Generates UUIDs for existing rows
4. Creates unique indexes on both columns
"""
connection = op.get_bind()
dialect = connection.dialect.name
# -------------------------------------------------------------------------
# Add external_id to project table
# -------------------------------------------------------------------------
if not column_exists(connection, "project", "external_id"):
# Step 1: Add external_id column as nullable first
op.add_column("project", sa.Column("external_id", sa.String(), nullable=True))
# Step 2: Generate UUIDs for existing rows
if dialect == "postgresql":
# Postgres has gen_random_uuid() function
op.execute("""
UPDATE project
SET external_id = gen_random_uuid()::text
WHERE external_id IS NULL
""")
else:
# SQLite: need to generate UUIDs in Python
result = connection.execute(text("SELECT id FROM project WHERE external_id IS NULL"))
for row in result:
new_uuid = str(uuid.uuid4())
connection.execute(
text("UPDATE project SET external_id = :uuid WHERE id = :id"),
{"uuid": new_uuid, "id": row[0]},
)
# Step 3: Make external_id NOT NULL
if dialect == "postgresql":
op.alter_column("project", "external_id", nullable=False)
else:
# SQLite requires batch operations for ALTER COLUMN
with op.batch_alter_table("project") as batch_op:
batch_op.alter_column("external_id", nullable=False)
# Step 4: Create unique index on project.external_id (idempotent)
if not index_exists(connection, "ix_project_external_id"):
op.create_index("ix_project_external_id", "project", ["external_id"], unique=True)
# -------------------------------------------------------------------------
# Add external_id to entity table
# -------------------------------------------------------------------------
if not column_exists(connection, "entity", "external_id"):
# Step 1: Add external_id column as nullable first
op.add_column("entity", sa.Column("external_id", sa.String(), nullable=True))
# Step 2: Generate UUIDs for existing rows
if dialect == "postgresql":
# Postgres has gen_random_uuid() function
op.execute("""
UPDATE entity
SET external_id = gen_random_uuid()::text
WHERE external_id IS NULL
""")
else:
# SQLite: need to generate UUIDs in Python
result = connection.execute(text("SELECT id FROM entity WHERE external_id IS NULL"))
for row in result:
new_uuid = str(uuid.uuid4())
connection.execute(
text("UPDATE entity SET external_id = :uuid WHERE id = :id"),
{"uuid": new_uuid, "id": row[0]},
)
# Step 3: Make external_id NOT NULL
if dialect == "postgresql":
op.alter_column("entity", "external_id", nullable=False)
else:
# SQLite requires batch operations for ALTER COLUMN
with op.batch_alter_table("entity") as batch_op:
batch_op.alter_column("external_id", nullable=False)
# Step 4: Create unique index on entity.external_id (idempotent)
if not index_exists(connection, "ix_entity_external_id"):
op.create_index("ix_entity_external_id", "entity", ["external_id"], unique=True)
def downgrade() -> None:
"""Remove external_id columns from project and entity tables."""
connection = op.get_bind()
dialect = connection.dialect.name
# Drop from entity table
if index_exists(connection, "ix_entity_external_id"):
op.drop_index("ix_entity_external_id", table_name="entity")
if column_exists(connection, "entity", "external_id"):
if dialect == "postgresql":
op.drop_column("entity", "external_id")
else:
with op.batch_alter_table("entity") as batch_op:
batch_op.drop_column("external_id")
# Drop from project table
if index_exists(connection, "ix_project_external_id"):
op.drop_index("ix_project_external_id", table_name="project")
if column_exists(connection, "project", "external_id"):
if dialect == "postgresql":
op.drop_column("project", "external_id")
else:
with op.batch_alter_table("project") as batch_op:
batch_op.drop_column("external_id")
@@ -1,68 +0,0 @@
"""Add Postgres semantic vector search tables (pgvector-aware, optional)
Revision ID: h1b2c3d4e5f6
Revises: d7e8f9a0b1c2
Create Date: 2026-02-07 00:00:00.000000
"""
from typing import Sequence, Union
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "h1b2c3d4e5f6"
down_revision: Union[str, None] = "d7e8f9a0b1c2"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Create Postgres vector chunk metadata table.
Trigger: database backend is PostgreSQL.
Why: search_vector_chunks stores text metadata with no vector-dimension
dependency, so it's safe in a migration. search_vector_embeddings (which
requires pgvector and a provider-specific dimension) is created at runtime
by PostgresSearchRepository._ensure_vector_tables(), mirroring the SQLite
pattern where vector tables are created dynamically.
Outcome: creates the dimension-independent chunks table. The embeddings
table + HNSW index are deferred to runtime.
"""
connection = op.get_bind()
if connection.dialect.name != "postgresql":
return
op.execute(
"""
CREATE TABLE IF NOT EXISTS search_vector_chunks (
id BIGSERIAL PRIMARY KEY,
entity_id INTEGER NOT NULL,
project_id INTEGER NOT NULL,
chunk_key TEXT NOT NULL,
chunk_text TEXT NOT NULL,
source_hash TEXT NOT NULL,
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
UNIQUE (project_id, entity_id, chunk_key)
)
"""
)
op.execute(
"""
CREATE INDEX IF NOT EXISTS idx_search_vector_chunks_project_entity
ON search_vector_chunks (project_id, entity_id)
"""
)
def downgrade() -> None:
"""Remove Postgres vector chunk/embedding tables.
Does not drop pgvector extension because other schema objects may depend on it.
"""
connection = op.get_bind()
if connection.dialect.name != "postgresql":
return
op.execute("DROP TABLE IF EXISTS search_vector_embeddings")
op.execute("DROP TABLE IF EXISTS search_vector_chunks")
+43 -93
View File
@@ -1,72 +1,52 @@
"""FastAPI application for basic-memory knowledge graph API."""
import asyncio
from contextlib import asynccontextmanager
from fastapi import FastAPI, HTTPException, Request
from fastapi import FastAPI, HTTPException
from fastapi.exception_handlers import http_exception_handler
from fastapi.routing import APIRouter
from loguru import logger
from basic_memory import __version__ as version
from basic_memory.api.container import ApiContainer, set_container
from basic_memory.api.v2.routers import (
knowledge_router as v2_knowledge,
project_router as v2_project,
memory_router as v2_memory,
search_router as v2_search,
resource_router as v2_resource,
directory_router as v2_directory,
prompt_router as v2_prompt,
importer_router as v2_importer,
schema_router as v2_schema,
from basic_memory import db
from basic_memory.api.routers import (
directory_router,
importer_router,
knowledge,
management,
memory,
project,
resource,
search,
prompt_router,
)
from basic_memory.api.v2.routers.project_router import (
add_project,
list_projects,
synchronize_projects,
)
from basic_memory.config import init_api_logging
from basic_memory.services.exceptions import EntityAlreadyExistsError
from basic_memory.services.initialization import initialize_app
from basic_memory.config import app_config
from basic_memory.services.initialization import initialize_app, initialize_file_sync
@asynccontextmanager
async def lifespan(app: FastAPI): # pragma: no cover
"""Lifecycle manager for the FastAPI app. Not called in stdio mcp mode"""
"""Lifecycle manager for the FastAPI app."""
# Initialize app and database
logger.info("Starting Basic Memory API")
await initialize_app(app_config)
# Initialize logging for API (stdout in cloud mode, file otherwise)
init_api_logging()
logger.info(f"Sync changes enabled: {app_config.sync_changes}")
if app_config.sync_changes:
# start file sync task in background
app.state.sync_task = asyncio.create_task(initialize_file_sync(app_config))
else:
logger.info("Sync changes disabled. Skipping file sync service.")
# --- Composition Root ---
# Create container and read config (single point of config access)
container = ApiContainer.create()
set_container(container)
app.state.container = container
logger.info(f"Starting Basic Memory API (mode={container.mode.name})")
await initialize_app(container.config)
# Cache database connections in app state for performance
logger.info("Initializing database and caching connections...")
engine, session_maker = await container.init_database()
app.state.engine = engine
app.state.session_maker = session_maker
logger.info("Database connections cached in app state")
# Create and start sync coordinator (lifecycle centralized in coordinator)
sync_coordinator = container.create_sync_coordinator()
await sync_coordinator.start()
app.state.sync_coordinator = sync_coordinator
# Proceed with startup
# proceed with startup
yield
# Shutdown - coordinator handles clean task cancellation
logger.info("Shutting down Basic Memory API")
await sync_coordinator.stop()
if app.state.sync_task:
logger.info("Stopping sync...")
app.state.sync_task.cancel() # pyright: ignore
await container.shutdown_database()
await db.shutdown_db()
# Initialize FastAPI app
@@ -77,52 +57,22 @@ app = FastAPI(
lifespan=lifespan,
)
# Include v2 routers FIRST (more specific paths must match before /{project} catch-all)
app.include_router(v2_knowledge, prefix="/v2/projects/{project_id}")
app.include_router(v2_memory, prefix="/v2/projects/{project_id}")
app.include_router(v2_search, prefix="/v2/projects/{project_id}")
app.include_router(v2_resource, prefix="/v2/projects/{project_id}")
app.include_router(v2_directory, prefix="/v2/projects/{project_id}")
app.include_router(v2_prompt, prefix="/v2/projects/{project_id}")
app.include_router(v2_importer, prefix="/v2/projects/{project_id}")
app.include_router(v2_schema, prefix="/v2/projects/{project_id}")
app.include_router(v2_project, prefix="/v2")
# Legacy web app proxy paths (compat with /proxy/projects/projects)
app.include_router(v2_project, prefix="/proxy/projects")
# Include routers
app.include_router(knowledge.router, prefix="/{project}")
app.include_router(memory.router, prefix="/{project}")
app.include_router(resource.router, prefix="/{project}")
app.include_router(search.router, prefix="/{project}")
app.include_router(project.project_router, prefix="/{project}")
app.include_router(directory_router.router, prefix="/{project}")
app.include_router(prompt_router.router, prefix="/{project}")
app.include_router(importer_router.router, prefix="/{project}")
# Legacy v1 compat: older CLI versions (v0.18.0 and earlier) call /projects/...
# Using router mount causes 307 redirect which proxy doesn't follow, so add explicit routes
legacy_router = APIRouter(tags=["legacy"])
legacy_router.add_api_route("/projects/projects", list_projects, methods=["GET"])
legacy_router.add_api_route("/projects/projects", add_project, methods=["POST"])
legacy_router.add_api_route("/projects/config/sync", synchronize_projects, methods=["POST"])
app.include_router(legacy_router)
# Project resource router works accross projects
app.include_router(project.project_resource_router)
app.include_router(management.router)
# V2 routers are the only public API surface
@app.exception_handler(EntityAlreadyExistsError)
async def entity_already_exists_error_handler(request: Request, exc: EntityAlreadyExistsError):
"""Handle entity creation conflicts (e.g., file already exists).
This is expected behavior when users try to create notes that exist,
so log at INFO level instead of ERROR.
"""
logger.info(
"Entity already exists",
url=str(request.url),
method=request.method,
path=request.url.path,
error=str(exc),
)
return await http_exception_handler(
request,
HTTPException(
status_code=409,
detail="Note already exists. Use edit_note to modify it, or delete it first.",
),
)
# Auth routes are handled by FastMCP automatically when auth is enabled
@app.exception_handler(Exception)
-132
View File
@@ -1,132 +0,0 @@
"""API composition root for Basic Memory.
This container owns reading ConfigManager and environment variables for the
API entrypoint. Downstream modules receive config/dependencies explicitly
rather than reading globals.
Design principles:
- Only this module reads ConfigManager directly
- Runtime mode (cloud/local/test) is resolved here
- Factories for services are provided, not singletons
"""
from dataclasses import dataclass
from typing import TYPE_CHECKING
from sqlalchemy.ext.asyncio import AsyncEngine, async_sessionmaker, AsyncSession
from basic_memory import db
from basic_memory.config import BasicMemoryConfig, ConfigManager
from basic_memory.runtime import RuntimeMode, resolve_runtime_mode
if TYPE_CHECKING: # pragma: no cover
from basic_memory.sync import SyncCoordinator
@dataclass
class ApiContainer:
"""Composition root for the API entrypoint.
Holds resolved configuration and runtime context.
Created once at app startup, then used to wire dependencies.
"""
config: BasicMemoryConfig
mode: RuntimeMode
# --- Database ---
# Cached database connections (set during lifespan startup)
engine: AsyncEngine | None = None
session_maker: async_sessionmaker[AsyncSession] | None = None
@classmethod
def create(cls) -> "ApiContainer": # pragma: no cover
"""Create container by reading ConfigManager.
This is the single point where API reads global config.
"""
config = ConfigManager().config
mode = resolve_runtime_mode(
is_test_env=config.is_test_env,
)
return cls(config=config, mode=mode)
# --- Runtime Mode Properties ---
@property
def should_sync_files(self) -> bool:
"""Whether file sync should be started.
Sync is enabled when:
- sync_changes is True in config
- Not in test mode (tests manage their own sync)
"""
return self.config.sync_changes and not self.mode.is_test
@property
def sync_skip_reason(self) -> str | None: # pragma: no cover
"""Reason why sync is skipped, or None if sync should run.
Useful for logging why sync was disabled.
"""
if self.mode.is_test:
return "Test environment detected"
if not self.config.sync_changes:
return "Sync changes disabled"
return None
def create_sync_coordinator(self) -> "SyncCoordinator": # pragma: no cover
"""Create a SyncCoordinator with this container's settings.
Returns:
SyncCoordinator configured for this runtime environment
"""
# Deferred import to avoid circular dependency
from basic_memory.sync import SyncCoordinator
return SyncCoordinator(
config=self.config,
should_sync=self.should_sync_files,
skip_reason=self.sync_skip_reason,
)
# --- Database Factory ---
async def init_database( # pragma: no cover
self,
) -> tuple[AsyncEngine, async_sessionmaker[AsyncSession]]:
"""Initialize and cache database connections.
Returns:
Tuple of (engine, session_maker)
"""
engine, session_maker = await db.get_or_create_db(self.config.database_path)
self.engine = engine
self.session_maker = session_maker
return engine, session_maker
async def shutdown_database(self) -> None: # pragma: no cover
"""Clean up database connections."""
await db.shutdown_db()
# Module-level container instance (set by lifespan)
# This allows deps.py to access the container without reading ConfigManager
_container: ApiContainer | None = None
def get_container() -> ApiContainer:
"""Get the current API container.
Raises:
RuntimeError: If container hasn't been initialized
"""
if _container is None:
raise RuntimeError("API container not initialized. Call set_container() first.")
return _container
def set_container(container: ApiContainer) -> None:
"""Set the API container (called by lifespan)."""
global _container
_container = container
+11
View File
@@ -0,0 +1,11 @@
"""API routers."""
from . import knowledge_router as knowledge
from . import management_router as management
from . import memory_router as memory
from . import project_router as project
from . import resource_router as resource
from . import search_router as search
from . import prompt_router as prompt
__all__ = ["knowledge", "management", "memory", "project", "resource", "search", "prompt"]
@@ -0,0 +1,63 @@
"""Router for directory tree operations."""
from typing import List, Optional
from fastapi import APIRouter, Query
from basic_memory.deps import DirectoryServiceDep, ProjectIdDep
from basic_memory.schemas.directory import DirectoryNode
router = APIRouter(prefix="/directory", tags=["directory"])
@router.get("/tree", response_model=DirectoryNode)
async def get_directory_tree(
directory_service: DirectoryServiceDep,
project_id: ProjectIdDep,
):
"""Get hierarchical directory structure from the knowledge base.
Args:
directory_service: Service for directory operations
project_id: ID of the current project
Returns:
DirectoryNode representing the root of the hierarchical tree structure
"""
# Get a hierarchical directory tree for the specific project
tree = await directory_service.get_directory_tree()
# Return the hierarchical tree
return tree
@router.get("/list", response_model=List[DirectoryNode])
async def list_directory(
directory_service: DirectoryServiceDep,
project_id: ProjectIdDep,
dir_name: str = Query("/", description="Directory path to list"),
depth: int = Query(1, ge=1, le=10, description="Recursion depth (1-10)"),
file_name_glob: Optional[str] = Query(
None, description="Glob pattern for filtering file names"
),
):
"""List directory contents with filtering and depth control.
Args:
directory_service: Service for directory operations
project_id: ID of the current project
dir_name: Directory path to list (default: root "/")
depth: Recursion depth (1-10, default: 1 for immediate children only)
file_name_glob: Optional glob pattern for filtering file names (e.g., "*.md", "*meeting*")
Returns:
List of DirectoryNode objects matching the criteria
"""
# Get directory listing with filtering
nodes = await directory_service.list_directory(
dir_name=dir_name,
depth=depth,
file_name_glob=file_name_glob,
)
return nodes
@@ -1,19 +1,15 @@
"""V2 Import Router - ID-based data import operations.
This router uses v2 dependencies for consistent project handling with external_id UUIDs.
Import endpoints use project_id in the path for consistency with other v2 endpoints.
"""
"""Import router for Basic Memory API."""
import json
import logging
from fastapi import APIRouter, Form, HTTPException, UploadFile, status, Path
from fastapi import APIRouter, Form, HTTPException, UploadFile, status
from basic_memory.deps import (
ChatGPTImporterV2ExternalDep,
ClaudeConversationsImporterV2ExternalDep,
ClaudeProjectsImporterV2ExternalDep,
MemoryJsonImporterV2ExternalDep,
ChatGPTImporterDep,
ClaudeConversationsImporterDep,
ClaudeProjectsImporterDep,
MemoryJsonImporterDep,
)
from basic_memory.importers import Importer
from basic_memory.schemas.importer import (
@@ -24,23 +20,21 @@ from basic_memory.schemas.importer import (
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/import", tags=["import-v2"])
router = APIRouter(prefix="/import", tags=["import"])
@router.post("/chatgpt", response_model=ChatImportResult)
async def import_chatgpt(
importer: ChatGPTImporterV2ExternalDep,
importer: ChatGPTImporterDep,
file: UploadFile,
project_id: str = Path(..., description="Project external UUID"),
directory: str = Form("conversations"),
folder: str = Form("conversations"),
) -> ChatImportResult:
"""Import conversations from ChatGPT JSON export.
Args:
project_id: Project external UUID from URL path
file: The ChatGPT conversations.json file.
directory: The directory to place the files in.
importer: ChatGPT importer instance.
folder: The folder to place the files in.
markdown_processor: MarkdownProcessor instance.
Returns:
ChatImportResult with import statistics.
@@ -48,24 +42,21 @@ async def import_chatgpt(
Raises:
HTTPException: If import fails.
"""
logger.info(f"V2 Importing ChatGPT conversations for project {project_id}")
return await import_file(importer, file, directory)
return await import_file(importer, file, folder)
@router.post("/claude/conversations", response_model=ChatImportResult)
async def import_claude_conversations(
importer: ClaudeConversationsImporterV2ExternalDep,
importer: ClaudeConversationsImporterDep,
file: UploadFile,
project_id: str = Path(..., description="Project external UUID"),
directory: str = Form("conversations"),
folder: str = Form("conversations"),
) -> ChatImportResult:
"""Import conversations from Claude conversations.json export.
Args:
project_id: Project external UUID from URL path
file: The Claude conversations.json file.
directory: The directory to place the files in.
importer: Claude conversations importer instance.
folder: The folder to place the files in.
markdown_processor: MarkdownProcessor instance.
Returns:
ChatImportResult with import statistics.
@@ -73,24 +64,21 @@ async def import_claude_conversations(
Raises:
HTTPException: If import fails.
"""
logger.info(f"V2 Importing Claude conversations for project {project_id}")
return await import_file(importer, file, directory)
return await import_file(importer, file, folder)
@router.post("/claude/projects", response_model=ProjectImportResult)
async def import_claude_projects(
importer: ClaudeProjectsImporterV2ExternalDep,
importer: ClaudeProjectsImporterDep,
file: UploadFile,
project_id: str = Path(..., description="Project external UUID"),
directory: str = Form("projects"),
folder: str = Form("projects"),
) -> ProjectImportResult:
"""Import projects from Claude projects.json export.
Args:
project_id: Project external UUID from URL path
file: The Claude projects.json file.
directory: The base directory to place the files in.
importer: Claude projects importer instance.
base_folder: The base folder to place the files in.
markdown_processor: MarkdownProcessor instance.
Returns:
ProjectImportResult with import statistics.
@@ -98,24 +86,21 @@ async def import_claude_projects(
Raises:
HTTPException: If import fails.
"""
logger.info(f"V2 Importing Claude projects for project {project_id}")
return await import_file(importer, file, directory)
return await import_file(importer, file, folder)
@router.post("/memory-json", response_model=EntityImportResult)
async def import_memory_json(
importer: MemoryJsonImporterV2ExternalDep,
importer: MemoryJsonImporterDep,
file: UploadFile,
project_id: str = Path(..., description="Project external UUID"),
directory: str = Form("conversations"),
folder: str = Form("conversations"),
) -> EntityImportResult:
"""Import entities and relations from a memory.json file.
Args:
project_id: Project external UUID from URL path
file: The memory.json file.
directory: Optional destination directory within the project.
importer: Memory JSON importer instance.
destination_folder: Optional destination folder within the project.
markdown_processor: MarkdownProcessor instance.
Returns:
EntityImportResult with import statistics.
@@ -123,7 +108,6 @@ async def import_memory_json(
Raises:
HTTPException: If import fails.
"""
logger.info(f"V2 Importing memory.json for project {project_id}")
try:
file_data = []
file_bytes = await file.read()
@@ -132,14 +116,14 @@ async def import_memory_json(
json_data = json.loads(line)
file_data.append(json_data)
result = await importer.import_data(file_data, directory)
result = await importer.import_data(file_data, folder)
if not result.success: # pragma: no cover
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=result.error_message or "Import failed",
)
except Exception as e:
logger.exception("V2 Import failed")
logger.exception("Import failed")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Import failed: {str(e)}",
@@ -147,24 +131,11 @@ async def import_memory_json(
return result
async def import_file(importer: Importer, file: UploadFile, destination_directory: str):
"""Helper function to import a file using an importer instance.
Args:
importer: The importer instance to use
file: The file to import
destination_directory: Destination directory for imported content
Returns:
Import result from the importer
Raises:
HTTPException: If import fails
"""
async def import_file(importer: Importer, file: UploadFile, destination_folder: str):
try:
# Process file
json_data = json.load(file.file)
result = await importer.import_data(json_data, destination_directory)
result = await importer.import_data(json_data, destination_folder)
if not result.success: # pragma: no cover
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
@@ -174,7 +145,7 @@ async def import_file(importer: Importer, file: UploadFile, destination_director
return result
except Exception as e:
logger.exception("V2 Import failed")
logger.exception("Import failed")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Import failed: {str(e)}",
@@ -0,0 +1,290 @@
"""Router for knowledge graph operations."""
from typing import Annotated
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Query, Response
from loguru import logger
from basic_memory.deps import (
EntityServiceDep,
get_search_service,
SearchServiceDep,
LinkResolverDep,
ProjectPathDep,
FileServiceDep,
ProjectConfigDep,
AppConfigDep,
SyncServiceDep,
)
from basic_memory.schemas import (
EntityListResponse,
EntityResponse,
DeleteEntitiesResponse,
DeleteEntitiesRequest,
)
from basic_memory.schemas.request import EditEntityRequest, MoveEntityRequest
from basic_memory.schemas.base import Permalink, Entity
router = APIRouter(prefix="/knowledge", tags=["knowledge"])
## Create endpoints
@router.post("/entities", response_model=EntityResponse)
async def create_entity(
data: Entity,
background_tasks: BackgroundTasks,
entity_service: EntityServiceDep,
search_service: SearchServiceDep,
) -> EntityResponse:
"""Create an entity."""
logger.info(
"API request", endpoint="create_entity", entity_type=data.entity_type, title=data.title
)
entity = await entity_service.create_entity(data)
# reindex
await search_service.index_entity(entity, background_tasks=background_tasks)
result = EntityResponse.model_validate(entity)
logger.info(
f"API response: endpoint='create_entity' title={result.title}, permalink={result.permalink}, status_code=201"
)
return result
@router.put("/entities/{permalink:path}", response_model=EntityResponse)
async def create_or_update_entity(
project: ProjectPathDep,
permalink: Permalink,
data: Entity,
response: Response,
background_tasks: BackgroundTasks,
entity_service: EntityServiceDep,
search_service: SearchServiceDep,
file_service: FileServiceDep,
sync_service: SyncServiceDep,
) -> EntityResponse:
"""Create or update an entity. If entity exists, it will be updated, otherwise created."""
logger.info(
f"API request: create_or_update_entity for {project=}, {permalink=}, {data.entity_type=}, {data.title=}"
)
# Validate permalink matches
if data.permalink != permalink:
logger.warning(
f"API validation error: creating/updating entity with permalink mismatch - url={permalink}, data={data.permalink}",
)
raise HTTPException(
status_code=400,
detail=f"Entity permalink {data.permalink} must match URL path: '{permalink}'",
)
# Try create_or_update operation
entity, created = await entity_service.create_or_update_entity(data)
response.status_code = 201 if created else 200
# reindex
await search_service.index_entity(entity, background_tasks=background_tasks)
# Attempt immediate relation resolution when creating new entities
# This helps resolve forward references when related entities are created in the same session
if created:
try:
await sync_service.resolve_relations()
logger.debug(f"Resolved relations after creating entity: {entity.permalink}")
except Exception as e: # pragma: no cover
# Don't fail the entire request if relation resolution fails
logger.warning(f"Failed to resolve relations after entity creation: {e}")
result = EntityResponse.model_validate(entity)
logger.info(
f"API response: {result.title=}, {result.permalink=}, {created=}, status_code={response.status_code}"
)
return result
@router.patch("/entities/{identifier:path}", response_model=EntityResponse)
async def edit_entity(
identifier: str,
data: EditEntityRequest,
background_tasks: BackgroundTasks,
entity_service: EntityServiceDep,
search_service: SearchServiceDep,
) -> EntityResponse:
"""Edit an existing entity using various operations like append, prepend, find_replace, or replace_section.
This endpoint allows for targeted edits without requiring the full entity content.
"""
logger.info(
f"API request: endpoint='edit_entity', identifier='{identifier}', operation='{data.operation}'"
)
try:
# Edit the entity using the service
entity = await entity_service.edit_entity(
identifier=identifier,
operation=data.operation,
content=data.content,
section=data.section,
find_text=data.find_text,
expected_replacements=data.expected_replacements,
)
# Reindex the updated entity
await search_service.index_entity(entity, background_tasks=background_tasks)
# Return the updated entity response
result = EntityResponse.model_validate(entity)
logger.info(
"API response",
endpoint="edit_entity",
identifier=identifier,
operation=data.operation,
permalink=result.permalink,
status_code=200,
)
return result
except Exception as e:
logger.error(f"Error editing entity: {e}")
raise HTTPException(status_code=400, detail=str(e))
@router.post("/move")
async def move_entity(
data: MoveEntityRequest,
background_tasks: BackgroundTasks,
entity_service: EntityServiceDep,
project_config: ProjectConfigDep,
app_config: AppConfigDep,
search_service: SearchServiceDep,
) -> EntityResponse:
"""Move an entity to a new file location with project consistency.
This endpoint moves a note to a different path while maintaining project
consistency and optionally updating permalinks based on configuration.
"""
logger.info(
f"API request: endpoint='move_entity', identifier='{data.identifier}', destination='{data.destination_path}'"
)
try:
# Move the entity using the service
moved_entity = await entity_service.move_entity(
identifier=data.identifier,
destination_path=data.destination_path,
project_config=project_config,
app_config=app_config,
)
# Get the moved entity to reindex it
entity = await entity_service.link_resolver.resolve_link(data.destination_path)
if entity:
await search_service.index_entity(entity, background_tasks=background_tasks)
logger.info(
"API response",
endpoint="move_entity",
identifier=data.identifier,
destination=data.destination_path,
status_code=200,
)
result = EntityResponse.model_validate(moved_entity)
return result
except Exception as e:
logger.error(f"Error moving entity: {e}")
raise HTTPException(status_code=400, detail=str(e))
## Read endpoints
@router.get("/entities/{identifier:path}", response_model=EntityResponse)
async def get_entity(
entity_service: EntityServiceDep,
link_resolver: LinkResolverDep,
identifier: str,
) -> EntityResponse:
"""Get a specific entity by file path or permalink..
Args:
identifier: Entity file path or permalink
:param entity_service: EntityService
:param link_resolver: LinkResolver
"""
logger.info(f"request: get_entity with identifier={identifier}")
entity = await link_resolver.resolve_link(identifier)
if not entity:
raise HTTPException(status_code=404, detail=f"Entity {identifier} not found")
result = EntityResponse.model_validate(entity)
return result
@router.get("/entities", response_model=EntityListResponse)
async def get_entities(
entity_service: EntityServiceDep,
permalink: Annotated[list[str] | None, Query()] = None,
) -> EntityListResponse:
"""Open specific entities"""
logger.info(f"request: get_entities with permalinks={permalink}")
entities = await entity_service.get_entities_by_permalinks(permalink) if permalink else []
result = EntityListResponse(
entities=[EntityResponse.model_validate(entity) for entity in entities]
)
return result
## Delete endpoints
@router.delete("/entities/{identifier:path}", response_model=DeleteEntitiesResponse)
async def delete_entity(
identifier: str,
background_tasks: BackgroundTasks,
entity_service: EntityServiceDep,
link_resolver: LinkResolverDep,
search_service=Depends(get_search_service),
) -> DeleteEntitiesResponse:
"""Delete a single entity and remove from search index."""
logger.info(f"request: delete_entity with identifier={identifier}")
entity = await link_resolver.resolve_link(identifier)
if entity is None:
return DeleteEntitiesResponse(deleted=False)
# Delete the entity
deleted = await entity_service.delete_entity(entity.permalink or entity.id)
# Remove from search index (entity, observations, and relations)
background_tasks.add_task(search_service.handle_delete, entity)
result = DeleteEntitiesResponse(deleted=deleted)
return result
@router.post("/entities/delete", response_model=DeleteEntitiesResponse)
async def delete_entities(
data: DeleteEntitiesRequest,
background_tasks: BackgroundTasks,
entity_service: EntityServiceDep,
search_service=Depends(get_search_service),
) -> DeleteEntitiesResponse:
"""Delete entities and remove from search index."""
logger.info(f"request: delete_entities with data={data}")
deleted = False
# Remove each deleted entity from search index
for permalink in data.permalinks:
deleted = await entity_service.delete_entity(permalink)
background_tasks.add_task(search_service.delete_by_permalink, permalink)
result = DeleteEntitiesResponse(deleted=deleted)
return result
@@ -0,0 +1,78 @@
"""Management router for basic-memory API."""
import asyncio
from fastapi import APIRouter, Request
from loguru import logger
from pydantic import BaseModel
from basic_memory.config import app_config
from basic_memory.deps import SyncServiceDep, ProjectRepositoryDep
router = APIRouter(prefix="/management", tags=["management"])
class WatchStatusResponse(BaseModel):
"""Response model for watch status."""
running: bool
"""Whether the watch service is currently running."""
@router.get("/watch/status", response_model=WatchStatusResponse)
async def get_watch_status(request: Request) -> WatchStatusResponse:
"""Get the current status of the watch service."""
return WatchStatusResponse(
running=request.app.state.watch_task is not None and not request.app.state.watch_task.done()
)
@router.post("/watch/start", response_model=WatchStatusResponse)
async def start_watch_service(
request: Request, project_repository: ProjectRepositoryDep, sync_service: SyncServiceDep
) -> WatchStatusResponse:
"""Start the watch service if it's not already running."""
# needed because of circular imports from sync -> app
from basic_memory.sync import WatchService
from basic_memory.sync.background_sync import create_background_sync_task
if request.app.state.watch_task is not None and not request.app.state.watch_task.done():
# Watch service is already running
return WatchStatusResponse(running=True)
# Create and start a new watch service
logger.info("Starting watch service via management API")
# Get services needed for the watch task
watch_service = WatchService(
app_config=app_config,
project_repository=project_repository,
)
# Create and store the task
watch_task = create_background_sync_task(sync_service, watch_service)
request.app.state.watch_task = watch_task
return WatchStatusResponse(running=True)
@router.post("/watch/stop", response_model=WatchStatusResponse)
async def stop_watch_service(request: Request) -> WatchStatusResponse: # pragma: no cover
"""Stop the watch service if it's running."""
if request.app.state.watch_task is None or request.app.state.watch_task.done():
# Watch service is not running
return WatchStatusResponse(running=False)
# Cancel the running task
logger.info("Stopping watch service via management API")
request.app.state.watch_task.cancel()
# Wait for it to be properly cancelled
try:
await request.app.state.watch_task
except asyncio.CancelledError:
pass
request.app.state.watch_task = None
return WatchStatusResponse(running=False)
@@ -0,0 +1,90 @@
"""Routes for memory:// URI operations."""
from typing import Annotated, Optional
from fastapi import APIRouter, Query
from loguru import logger
from basic_memory.deps import ContextServiceDep, EntityRepositoryDep
from basic_memory.schemas.base import TimeFrame, parse_timeframe
from basic_memory.schemas.memory import (
GraphContext,
normalize_memory_url,
)
from basic_memory.schemas.search import SearchItemType
from basic_memory.api.routers.utils import to_graph_context
router = APIRouter(prefix="/memory", tags=["memory"])
@router.get("/recent", response_model=GraphContext)
async def recent(
context_service: ContextServiceDep,
entity_repository: EntityRepositoryDep,
type: Annotated[list[SearchItemType] | None, Query()] = None,
depth: int = 1,
timeframe: TimeFrame = "7d",
page: int = 1,
page_size: int = 10,
max_related: int = 10,
) -> GraphContext:
# return all types by default
types = (
[SearchItemType.ENTITY, SearchItemType.RELATION, SearchItemType.OBSERVATION]
if not type
else type
)
logger.debug(
f"Getting recent context: `{types}` depth: `{depth}` timeframe: `{timeframe}` page: `{page}` page_size: `{page_size}` max_related: `{max_related}`"
)
# Parse timeframe
since = parse_timeframe(timeframe)
limit = page_size
offset = (page - 1) * page_size
# Build context
context = await context_service.build_context(
types=types, depth=depth, since=since, limit=limit, offset=offset, max_related=max_related
)
recent_context = await to_graph_context(
context, entity_repository=entity_repository, page=page, page_size=page_size
)
logger.debug(f"Recent context: {recent_context.model_dump_json()}")
return recent_context
# get_memory_context needs to be declared last so other paths can match
@router.get("/{uri:path}", response_model=GraphContext)
async def get_memory_context(
context_service: ContextServiceDep,
entity_repository: EntityRepositoryDep,
uri: str,
depth: int = 1,
timeframe: Optional[TimeFrame] = None,
page: int = 1,
page_size: int = 10,
max_related: int = 10,
) -> GraphContext:
"""Get rich context from memory:// URI."""
# add the project name from the config to the url as the "host
# Parse URI
logger.debug(
f"Getting context for URI: `{uri}` depth: `{depth}` timeframe: `{timeframe}` page: `{page}` page_size: `{page_size}` max_related: `{max_related}`"
)
memory_url = normalize_memory_url(uri)
# Parse timeframe
since = parse_timeframe(timeframe) if timeframe else None
limit = page_size
offset = (page - 1) * page_size
# Build context
context = await context_service.build_context(
memory_url, depth=depth, since=since, limit=limit, offset=offset, max_related=max_related
)
return await to_graph_context(
context, entity_repository=entity_repository, page=page, page_size=page_size
)
@@ -0,0 +1,234 @@
"""Router for project management."""
from fastapi import APIRouter, HTTPException, Path, Body
from typing import Optional
from basic_memory.deps import ProjectServiceDep, ProjectPathDep
from basic_memory.schemas import ProjectInfoResponse
from basic_memory.schemas.project_info import (
ProjectList,
ProjectItem,
ProjectInfoRequest,
ProjectStatusResponse,
)
# Router for resources in a specific project
project_router = APIRouter(prefix="/project", tags=["project"])
# Router for managing project resources
project_resource_router = APIRouter(prefix="/projects", tags=["project_management"])
@project_router.get("/info", response_model=ProjectInfoResponse)
async def get_project_info(
project_service: ProjectServiceDep,
project: ProjectPathDep,
) -> ProjectInfoResponse:
"""Get comprehensive information about the specified Basic Memory project."""
return await project_service.get_project_info(project)
# Update a project
@project_router.patch("/{name}", response_model=ProjectStatusResponse)
async def update_project(
project_service: ProjectServiceDep,
project_name: str = Path(..., description="Name of the project to update"),
path: Optional[str] = Body(None, description="New path for the project"),
is_active: Optional[bool] = Body(None, description="Status of the project (active/inactive)"),
) -> ProjectStatusResponse:
"""Update a project's information in configuration and database.
Args:
project_name: The name of the project to update
path: Optional new path for the project
is_active: Optional status update for the project
Returns:
Response confirming the project was updated
"""
try: # pragma: no cover
# Get original project info for the response
old_project_info = ProjectItem(
name=project_name,
path=project_service.projects.get(project_name, ""),
)
await project_service.update_project(project_name, updated_path=path, is_active=is_active)
# Get updated project info
updated_path = path if path else project_service.projects.get(project_name, "")
return ProjectStatusResponse(
message=f"Project '{project_name}' updated successfully",
status="success",
default=(project_name == project_service.default_project),
old_project=old_project_info,
new_project=ProjectItem(name=project_name, path=updated_path),
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e))
# List all available projects
@project_resource_router.get("/projects", response_model=ProjectList)
async def list_projects(
project_service: ProjectServiceDep,
) -> ProjectList:
"""List all configured projects.
Returns:
A list of all projects with metadata
"""
projects = await project_service.list_projects()
default_project = project_service.default_project
project_items = [
ProjectItem(
name=project.name,
path=project.path,
is_default=project.is_default or False,
)
for project in projects
]
return ProjectList(
projects=project_items,
default_project=default_project,
)
# Add a new project
@project_resource_router.post("/projects", response_model=ProjectStatusResponse)
async def add_project(
project_data: ProjectInfoRequest,
project_service: ProjectServiceDep,
) -> ProjectStatusResponse:
"""Add a new project to configuration and database.
Args:
project_data: The project name and path, with option to set as default
Returns:
Response confirming the project was added
"""
try: # pragma: no cover
await project_service.add_project(
project_data.name, project_data.path, set_default=project_data.set_default
)
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message=f"Project '{project_data.name}' added successfully",
status="success",
default=project_data.set_default,
new_project=ProjectItem(
name=project_data.name, path=project_data.path, is_default=project_data.set_default
),
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e))
# Remove a project
@project_resource_router.delete("/{name}", response_model=ProjectStatusResponse)
async def remove_project(
project_service: ProjectServiceDep,
name: str = Path(..., description="Name of the project to remove"),
) -> ProjectStatusResponse:
"""Remove a project from configuration and database.
Args:
name: The name of the project to remove
Returns:
Response confirming the project was removed
"""
try:
old_project = await project_service.get_project(name)
if not old_project: # pragma: no cover
raise HTTPException(
status_code=404, detail=f"Project: '{name}' does not exist"
) # pragma: no cover
await project_service.remove_project(name)
return ProjectStatusResponse(
message=f"Project '{name}' removed successfully",
status="success",
default=False,
old_project=ProjectItem(name=old_project.name, path=old_project.path),
new_project=None,
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e))
# Set a project as default
@project_resource_router.put("/{name}/default", response_model=ProjectStatusResponse)
async def set_default_project(
project_service: ProjectServiceDep,
name: str = Path(..., description="Name of the project to set as default"),
) -> ProjectStatusResponse:
"""Set a project as the default project.
Args:
name: The name of the project to set as default
Returns:
Response confirming the project was set as default
"""
try:
# Get the old default project
default_name = project_service.default_project
default_project = await project_service.get_project(default_name)
if not default_project: # pragma: no cover
raise HTTPException( # pragma: no cover
status_code=404, detail=f"Default Project: '{default_name}' does not exist"
)
# get the new project
new_default_project = await project_service.get_project(name)
if not new_default_project: # pragma: no cover
raise HTTPException(
status_code=404, detail=f"Project: '{name}' does not exist"
) # pragma: no cover
await project_service.set_default_project(name)
return ProjectStatusResponse(
message=f"Project '{name}' set as default successfully",
status="success",
default=True,
old_project=ProjectItem(name=default_name, path=default_project.path),
new_project=ProjectItem(
name=name,
path=new_default_project.path,
is_default=True,
),
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e))
# Synchronize projects between config and database
@project_resource_router.post("/sync", response_model=ProjectStatusResponse)
async def synchronize_projects(
project_service: ProjectServiceDep,
) -> ProjectStatusResponse:
"""Synchronize projects between configuration file and database.
Ensures that all projects in the configuration file exist in the database
and vice versa.
Returns:
Response confirming synchronization was completed
"""
try: # pragma: no cover
await project_service.synchronize_projects()
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message="Projects synchronized successfully between configuration and database",
status="success",
default=False,
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e))
@@ -1,22 +1,21 @@
"""V2 Prompt Router - ID-based prompt generation operations.
"""Router for prompt-related operations.
This router uses v2 dependencies for consistent project handling with external_id UUIDs.
Prompt endpoints are action-based (not resource-based), so they don't
have entity IDs in URLs - they generate formatted prompts from queries.
This router is responsible for rendering various prompts using Handlebars templates.
It centralizes all prompt formatting logic that was previously in the MCP prompts.
"""
from datetime import datetime, timezone
from fastapi import APIRouter, HTTPException, status, Path
from fastapi import APIRouter, HTTPException, status
from loguru import logger
from basic_memory.api.v2.utils import to_graph_context, to_search_results
from basic_memory.api.routers.utils import to_graph_context, to_search_results
from basic_memory.api.template_loader import template_loader
from basic_memory.schemas.base import parse_timeframe
from basic_memory.deps import (
ContextServiceV2ExternalDep,
EntityRepositoryV2ExternalDep,
SearchServiceV2ExternalDep,
EntityServiceV2ExternalDep,
ContextServiceDep,
EntityRepositoryDep,
SearchServiceDep,
EntityServiceDep,
)
from basic_memory.schemas.prompt import (
ContinueConversationRequest,
@@ -26,17 +25,16 @@ from basic_memory.schemas.prompt import (
)
from basic_memory.schemas.search import SearchItemType, SearchQuery
router = APIRouter(prefix="/prompt", tags=["prompt-v2"])
router = APIRouter(prefix="/prompt", tags=["prompt"])
@router.post("/continue-conversation", response_model=PromptResponse)
async def continue_conversation(
search_service: SearchServiceV2ExternalDep,
entity_service: EntityServiceV2ExternalDep,
context_service: ContextServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
search_service: SearchServiceDep,
entity_service: EntityServiceDep,
context_service: ContextServiceDep,
entity_repository: EntityRepositoryDep,
request: ContinueConversationRequest,
project_id: str = Path(..., description="Project external UUID"),
) -> PromptResponse:
"""Generate a prompt for continuing a conversation.
@@ -44,15 +42,13 @@ async def continue_conversation(
relevant context from the knowledge base.
Args:
project_id: Project external UUID from URL path
request: The request parameters
Returns:
Formatted continuation prompt with context
"""
logger.info(
f"V2 Generating continue conversation prompt for project {project_id}, "
f"topic: {request.topic}, timeframe: {request.timeframe}"
f"Generating continue conversation prompt, topic: {request.topic}, timeframe: {request.timeframe}"
)
since = parse_timeframe(request.timeframe) if request.timeframe else None
@@ -196,10 +192,9 @@ async def continue_conversation(
@router.post("/search", response_model=PromptResponse)
async def search_prompt(
search_service: SearchServiceV2ExternalDep,
entity_service: EntityServiceV2ExternalDep,
search_service: SearchServiceDep,
entity_service: EntityServiceDep,
request: SearchPromptRequest,
project_id: str = Path(..., description="Project external UUID"),
page: int = 1,
page_size: int = 10,
) -> PromptResponse:
@@ -209,7 +204,6 @@ async def search_prompt(
prompt with context and suggestions.
Args:
project_id: Project external UUID from URL path
request: The search parameters
page: The page number for pagination
page_size: The number of results per page, defaults to 10
@@ -217,10 +211,7 @@ async def search_prompt(
Returns:
Formatted search results prompt with context
"""
logger.info(
f"V2 Generating search prompt for project {project_id}, "
f"query: {request.query}, timeframe: {request.timeframe}"
)
logger.info(f"Generating search prompt, query: {request.query}, timeframe: {request.timeframe}")
limit = page_size
offset = (page - 1) * page_size
@@ -0,0 +1,225 @@
"""Routes for getting entity content."""
import tempfile
from pathlib import Path
from typing import Annotated
from fastapi import APIRouter, HTTPException, BackgroundTasks, Body
from fastapi.responses import FileResponse, JSONResponse
from loguru import logger
from basic_memory.deps import (
ProjectConfigDep,
LinkResolverDep,
SearchServiceDep,
EntityServiceDep,
FileServiceDep,
EntityRepositoryDep,
)
from basic_memory.repository.search_repository import SearchIndexRow
from basic_memory.schemas.memory import normalize_memory_url
from basic_memory.schemas.search import SearchQuery, SearchItemType
from basic_memory.models.knowledge import Entity as EntityModel
from datetime import datetime
router = APIRouter(prefix="/resource", tags=["resources"])
def get_entity_ids(item: SearchIndexRow) -> set[int]:
match item.type:
case SearchItemType.ENTITY:
return {item.id}
case SearchItemType.OBSERVATION:
return {item.entity_id} # pyright: ignore [reportReturnType]
case SearchItemType.RELATION:
from_entity = item.from_id
to_entity = item.to_id # pyright: ignore [reportReturnType]
return {from_entity, to_entity} if to_entity else {from_entity} # pyright: ignore [reportReturnType]
case _: # pragma: no cover
raise ValueError(f"Unexpected type: {item.type}")
@router.get("/{identifier:path}")
async def get_resource_content(
config: ProjectConfigDep,
link_resolver: LinkResolverDep,
search_service: SearchServiceDep,
entity_service: EntityServiceDep,
file_service: FileServiceDep,
background_tasks: BackgroundTasks,
identifier: str,
page: int = 1,
page_size: int = 10,
) -> FileResponse:
"""Get resource content by identifier: name or permalink."""
logger.debug(f"Getting content for: {identifier}")
# Find single entity by permalink
entity = await link_resolver.resolve_link(identifier)
results = [entity] if entity else []
# pagination for multiple results
limit = page_size
offset = (page - 1) * page_size
# search using the identifier as a permalink
if not results:
# if the identifier contains a wildcard, use GLOB search
query = (
SearchQuery(permalink_match=identifier)
if "*" in identifier
else SearchQuery(permalink=identifier)
)
search_results = await search_service.search(query, limit, offset)
if not search_results:
raise HTTPException(status_code=404, detail=f"Resource not found: {identifier}")
# get the deduplicated entities related to the search results
entity_ids = {id for result in search_results for id in get_entity_ids(result)}
results = await entity_service.get_entities_by_id(list(entity_ids))
# return single response
if len(results) == 1:
entity = results[0]
file_path = Path(f"{config.home}/{entity.file_path}")
if not file_path.exists():
raise HTTPException(
status_code=404,
detail=f"File not found: {file_path}",
)
return FileResponse(path=file_path)
# for multiple files, initialize a temporary file for writing the results
with tempfile.NamedTemporaryFile(delete=False, mode="w", suffix=".md") as tmp_file:
temp_file_path = tmp_file.name
for result in results:
# Read content for each entity
content = await file_service.read_entity_content(result)
memory_url = normalize_memory_url(result.permalink)
modified_date = result.updated_at.isoformat()
checksum = result.checksum[:8] if result.checksum else ""
# Prepare the delimited content
response_content = f"--- {memory_url} {modified_date} {checksum}\n"
response_content += f"\n{content}\n"
response_content += "\n"
# Write content directly to the temporary file in append mode
tmp_file.write(response_content)
# Ensure all content is written to disk
tmp_file.flush()
# Schedule the temporary file to be deleted after the response
background_tasks.add_task(cleanup_temp_file, temp_file_path)
# Return the file response
return FileResponse(path=temp_file_path)
def cleanup_temp_file(file_path: str):
"""Delete the temporary file."""
try:
Path(file_path).unlink() # Deletes the file
logger.debug(f"Temporary file deleted: {file_path}")
except Exception as e: # pragma: no cover
logger.error(f"Error deleting temporary file {file_path}: {e}")
@router.put("/{file_path:path}")
async def write_resource(
config: ProjectConfigDep,
file_service: FileServiceDep,
entity_repository: EntityRepositoryDep,
search_service: SearchServiceDep,
file_path: str,
content: Annotated[str, Body()],
) -> JSONResponse:
"""Write content to a file in the project.
This endpoint allows writing content directly to a file in the project.
Also creates an entity record and indexes the file for search.
Args:
file_path: Path to write to, relative to project root
request: Contains the content to write
Returns:
JSON response with file information
"""
try:
# Get content from request body
# Ensure it's UTF-8 string content
if isinstance(content, bytes): # pragma: no cover
content_str = content.decode("utf-8")
else:
content_str = str(content)
# Get full file path
full_path = Path(f"{config.home}/{file_path}")
# Ensure parent directory exists
full_path.parent.mkdir(parents=True, exist_ok=True)
# Write content to file
checksum = await file_service.write_file(full_path, content_str)
# Get file info
file_stats = file_service.file_stats(full_path)
# Determine file details
file_name = Path(file_path).name
content_type = file_service.content_type(full_path)
entity_type = "canvas" if file_path.endswith(".canvas") else "file"
# Check if entity already exists
existing_entity = await entity_repository.get_by_file_path(file_path)
if existing_entity:
# Update existing entity
entity = await entity_repository.update(
existing_entity.id,
{
"title": file_name,
"entity_type": entity_type,
"content_type": content_type,
"file_path": file_path,
"checksum": checksum,
"updated_at": datetime.fromtimestamp(file_stats.st_mtime),
},
)
status_code = 200
else:
# Create a new entity model
entity = EntityModel(
title=file_name,
entity_type=entity_type,
content_type=content_type,
file_path=file_path,
checksum=checksum,
created_at=datetime.fromtimestamp(file_stats.st_ctime),
updated_at=datetime.fromtimestamp(file_stats.st_mtime),
)
entity = await entity_repository.add(entity)
status_code = 201
# Index the file for search
await search_service.index_entity(entity) # pyright: ignore
# Return success response
return JSONResponse(
status_code=status_code,
content={
"file_path": file_path,
"checksum": checksum,
"size": file_stats.st_size,
"created_at": file_stats.st_ctime,
"modified_at": file_stats.st_mtime,
},
)
except Exception as e: # pragma: no cover
logger.error(f"Error writing resource {file_path}: {e}")
raise HTTPException(status_code=500, detail=f"Failed to write resource: {str(e)}")
@@ -0,0 +1,36 @@
"""Router for search operations."""
from fastapi import APIRouter, BackgroundTasks
from basic_memory.api.routers.utils import to_search_results
from basic_memory.schemas.search import SearchQuery, SearchResponse
from basic_memory.deps import SearchServiceDep, EntityServiceDep
router = APIRouter(prefix="/search", tags=["search"])
@router.post("/", response_model=SearchResponse)
async def search(
query: SearchQuery,
search_service: SearchServiceDep,
entity_service: EntityServiceDep,
page: int = 1,
page_size: int = 10,
):
"""Search across all knowledge and documents."""
limit = page_size
offset = (page - 1) * page_size
results = await search_service.search(query, limit=limit, offset=offset)
search_results = await to_search_results(entity_service, results)
return SearchResponse(
results=search_results,
current_page=page,
page_size=page_size,
)
@router.post("/reindex")
async def reindex(background_tasks: BackgroundTasks, search_service: SearchServiceDep):
"""Recreate and populate the search index."""
await search_service.reindex_all(background_tasks=background_tasks)
return {"status": "ok", "message": "Reindex initiated"}
@@ -24,43 +24,11 @@ async def to_graph_context(
page: Optional[int] = None,
page_size: Optional[int] = None,
):
# First pass: collect all entity IDs needed for external_id lookup
# This includes: entity primary results, observation parent entities, relation from/to entities
entity_ids_needed: set[int] = set()
for context_item in context_result.results:
for item in (
[context_item.primary_result] + context_item.observations + context_item.related_results
):
if item.type == SearchItemType.ENTITY:
# Entity's own ID for its external_id
entity_ids_needed.add(item.id)
elif item.type == SearchItemType.OBSERVATION:
# Parent entity ID for entity_external_id
if item.entity_id: # pyright: ignore
entity_ids_needed.add(item.entity_id) # pyright: ignore
elif item.type == SearchItemType.RELATION:
# Source and target entity IDs for external_ids
if item.from_id: # pyright: ignore
entity_ids_needed.add(item.from_id) # pyright: ignore
if item.to_id:
entity_ids_needed.add(item.to_id)
# Batch fetch all entities at once - get both title and external_id
entity_title_lookup: dict[int, str] = {}
entity_external_id_lookup: dict[int, str] = {}
if entity_ids_needed:
entities = await entity_repository.find_by_ids(list(entity_ids_needed))
for e in entities:
entity_title_lookup[e.id] = e.title
entity_external_id_lookup[e.id] = e.external_id
# Helper function to convert items to summaries
def to_summary(item: SearchIndexRow | ContextResultRow):
async def to_summary(item: SearchIndexRow | ContextResultRow):
match item.type:
case SearchItemType.ENTITY:
return EntitySummary(
external_id=entity_external_id_lookup.get(item.id, ""),
entity_id=item.id,
title=item.title, # pyright: ignore
permalink=item.permalink,
content=item.content,
@@ -68,14 +36,8 @@ async def to_graph_context(
created_at=item.created_at,
)
case SearchItemType.OBSERVATION:
entity_ext_id = None
if item.entity_id: # pyright: ignore
entity_ext_id = entity_external_id_lookup.get(item.entity_id) # pyright: ignore
return ObservationSummary(
observation_id=item.id,
entity_id=item.entity_id, # pyright: ignore
entity_external_id=entity_ext_id,
title=entity_title_lookup.get(item.entity_id), # pyright: ignore
title=item.title, # pyright: ignore
file_path=item.file_path,
category=item.category, # pyright: ignore
content=item.content, # pyright: ignore
@@ -83,23 +45,15 @@ async def to_graph_context(
created_at=item.created_at,
)
case SearchItemType.RELATION:
from_title = entity_title_lookup.get(item.from_id) if item.from_id else None # pyright: ignore
to_title = entity_title_lookup.get(item.to_id) if item.to_id else None
from_ext_id = entity_external_id_lookup.get(item.from_id) if item.from_id else None # pyright: ignore
to_ext_id = entity_external_id_lookup.get(item.to_id) if item.to_id else None
from_entity = await entity_repository.find_by_id(item.from_id) # pyright: ignore
to_entity = await entity_repository.find_by_id(item.to_id) if item.to_id else None
return RelationSummary(
relation_id=item.id,
entity_id=item.entity_id, # pyright: ignore
title=item.title, # pyright: ignore
file_path=item.file_path,
permalink=item.permalink, # pyright: ignore
relation_type=item.relation_type, # pyright: ignore
from_entity=from_title,
from_entity_id=item.from_id, # pyright: ignore
from_entity_external_id=from_ext_id,
to_entity=to_title,
to_entity_id=item.to_id,
to_entity_external_id=to_ext_id,
from_entity=from_entity.title if from_entity else None,
to_entity=to_entity.title if to_entity else None,
created_at=item.created_at,
)
case _: # pragma: no cover
@@ -109,19 +63,23 @@ async def to_graph_context(
hierarchical_results = []
for context_item in context_result.results:
# Process primary result
primary_result = to_summary(context_item.primary_result)
primary_result = await to_summary(context_item.primary_result)
# Process observations (always ObservationSummary, validated by context_service)
observations = [to_summary(obs) for obs in context_item.observations]
# Process observations
observations = []
for obs in context_item.observations:
observations.append(await to_summary(obs))
# Process related results
related = [to_summary(rel) for rel in context_item.related_results]
related = []
for rel in context_item.related_results:
related.append(await to_summary(rel))
# Add to hierarchical results
hierarchical_results.append(
ContextResult(
primary_result=primary_result,
observations=observations, # pyright: ignore[reportArgumentType]
observations=observations,
related_results=related,
)
)
@@ -153,21 +111,6 @@ async def to_search_results(entity_service: EntityService, results: List[SearchI
search_results = []
for r in results:
entities = await entity_service.get_entities_by_id([r.entity_id, r.from_id, r.to_id]) # pyright: ignore
# Determine which IDs to set based on type
entity_id = None
observation_id = None
relation_id = None
if r.type == SearchItemType.ENTITY:
entity_id = r.id
elif r.type == SearchItemType.OBSERVATION:
observation_id = r.id
entity_id = r.entity_id # Parent entity
elif r.type == SearchItemType.RELATION:
relation_id = r.id
entity_id = r.entity_id # Parent entity
search_results.append(
SearchResult(
title=r.title, # pyright: ignore
@@ -178,9 +121,6 @@ async def to_search_results(entity_service: EntityService, results: List[SearchI
content=r.content,
file_path=r.file_path,
metadata=r.metadata,
entity_id=entity_id,
observation_id=observation_id,
relation_id=relation_id,
category=r.category,
from_entity=entities[0].permalink if entities else None,
to_entity=entities[1].permalink if len(entities) > 1 else None,
-35
View File
@@ -1,35 +0,0 @@
"""API v2 module - ID-based entity references.
Version 2 of the Basic Memory API uses integer entity IDs as the primary
identifier for improved performance and stability.
Key changes from v1:
- Entity lookups use integer IDs instead of paths/permalinks
- Direct database queries instead of cascading resolution
- Stable references that don't change with file moves
- Better caching support
All v2 routers are registered with the /v2 prefix.
"""
from basic_memory.api.v2.routers import (
knowledge_router,
memory_router,
project_router,
resource_router,
search_router,
directory_router,
prompt_router,
importer_router,
)
__all__ = [
"knowledge_router",
"memory_router",
"project_router",
"resource_router",
"search_router",
"directory_router",
"prompt_router",
"importer_router",
]
@@ -1,23 +0,0 @@
"""V2 API routers."""
from basic_memory.api.v2.routers.knowledge_router import router as knowledge_router
from basic_memory.api.v2.routers.project_router import router as project_router
from basic_memory.api.v2.routers.memory_router import router as memory_router
from basic_memory.api.v2.routers.search_router import router as search_router
from basic_memory.api.v2.routers.resource_router import router as resource_router
from basic_memory.api.v2.routers.directory_router import router as directory_router
from basic_memory.api.v2.routers.prompt_router import router as prompt_router
from basic_memory.api.v2.routers.importer_router import router as importer_router
from basic_memory.api.v2.routers.schema_router import router as schema_router
__all__ = [
"knowledge_router",
"project_router",
"memory_router",
"search_router",
"resource_router",
"directory_router",
"prompt_router",
"importer_router",
"schema_router",
]
@@ -1,93 +0,0 @@
"""V2 Directory Router - ID-based directory tree operations.
This router provides directory structure browsing for projects using
external_id UUIDs instead of name-based identifiers.
Key improvements:
- Direct project lookup via external_id UUIDs
- Consistent with other v2 endpoints
- Better performance through indexed queries
"""
from typing import List, Optional
from fastapi import APIRouter, Query, Path
from basic_memory.deps import DirectoryServiceV2ExternalDep
from basic_memory.schemas.directory import DirectoryNode
router = APIRouter(prefix="/directory", tags=["directory-v2"])
@router.get("/tree", response_model=DirectoryNode, response_model_exclude_none=True)
async def get_directory_tree(
directory_service: DirectoryServiceV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
):
"""Get hierarchical directory structure from the knowledge base.
Args:
directory_service: Service for directory operations
project_id: Project external UUID
Returns:
DirectoryNode representing the root of the hierarchical tree structure
"""
# Get a hierarchical directory tree for the specific project
tree = await directory_service.get_directory_tree()
# Return the hierarchical tree
return tree
@router.get("/structure", response_model=DirectoryNode, response_model_exclude_none=True)
async def get_directory_structure(
directory_service: DirectoryServiceV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
):
"""Get folder structure for navigation (no files).
Optimized endpoint for folder tree navigation. Returns only directory nodes
without file metadata. For full tree with files, use /directory/tree.
Args:
directory_service: Service for directory operations
project_id: Project external UUID
Returns:
DirectoryNode tree containing only folders (type="directory")
"""
structure = await directory_service.get_directory_structure()
return structure
@router.get("/list", response_model=List[DirectoryNode], response_model_exclude_none=True)
async def list_directory(
directory_service: DirectoryServiceV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
dir_name: str = Query("/", description="Directory path to list"),
depth: int = Query(1, ge=1, le=10, description="Recursion depth (1-10)"),
file_name_glob: Optional[str] = Query(
None, description="Glob pattern for filtering file names"
),
):
"""List directory contents with filtering and depth control.
Args:
directory_service: Service for directory operations
project_id: Project external UUID
dir_name: Directory path to list (default: root "/")
depth: Recursion depth (1-10, default: 1 for immediate children only)
file_name_glob: Optional glob pattern for filtering file names (e.g., "*.md", "*meeting*")
Returns:
List of DirectoryNode objects matching the criteria
"""
# Get directory listing with filtering
nodes = await directory_service.list_directory(
dir_name=dir_name,
depth=depth,
file_name_glob=file_name_glob,
)
return nodes
@@ -1,636 +0,0 @@
"""V2 Knowledge Router - External ID-based entity operations.
This router provides external_id (UUID) based CRUD operations for entities,
using stable string UUIDs that won't change with file moves or database migrations.
Key improvements:
- Stable external UUIDs that won't change with file moves or renames
- Better API ergonomics with consistent string identifiers
- Direct database lookups via unique indexed column
- Simplified caching strategies
"""
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Response, Path, Query
from loguru import logger
from basic_memory.deps import (
EntityServiceV2ExternalDep,
SearchServiceV2ExternalDep,
LinkResolverV2ExternalDep,
ProjectConfigV2ExternalDep,
AppConfigDep,
EntityRepositoryV2ExternalDep,
ProjectExternalIdPathDep,
TaskSchedulerDep,
FileServiceV2ExternalDep,
)
from basic_memory.schemas import DeleteEntitiesResponse
from basic_memory.schemas.base import Entity
from basic_memory.schemas.request import EditEntityRequest
from basic_memory.schemas.v2 import (
EntityResolveRequest,
EntityResolveResponse,
EntityResponseV2,
MoveEntityRequestV2,
MoveDirectoryRequestV2,
DeleteDirectoryRequestV2,
)
from basic_memory.schemas.response import DirectoryMoveResult, DirectoryDeleteResult
router = APIRouter(prefix="/knowledge", tags=["knowledge-v2"])
def _schedule_vector_sync_if_enabled(
*,
task_scheduler,
app_config,
entity_id: int,
project_id: int,
) -> None:
"""Schedule out-of-band vector sync only when semantic search is enabled."""
if app_config.semantic_search_enabled:
task_scheduler.schedule(
"sync_entity_vectors",
entity_id=entity_id,
project_id=project_id,
)
## Resolution endpoint
@router.post("/resolve", response_model=EntityResolveResponse)
async def resolve_identifier(
project_id: ProjectExternalIdPathDep,
data: EntityResolveRequest,
link_resolver: LinkResolverV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
) -> EntityResolveResponse:
"""Resolve a string identifier (external_id, permalink, title, or path) to entity info.
This endpoint provides a bridge between v1-style identifiers and v2 external_ids.
Use this to convert existing references to the new UUID-based format.
Args:
data: Request containing the identifier to resolve
Returns:
Entity external_id and metadata about how it was resolved
Raises:
HTTPException: 404 if identifier cannot be resolved
Example:
POST /v2/{project_id}/knowledge/resolve
{"identifier": "specs/search"}
Returns:
{
"external_id": "550e8400-e29b-41d4-a716-446655440000",
"entity_id": 123,
"permalink": "specs/search",
"file_path": "specs/search.md",
"title": "Search Specification",
"resolution_method": "permalink"
}
"""
logger.info(f"API v2 request: resolve_identifier for '{data.identifier}'")
# Try to resolve by external_id first
entity = await entity_repository.get_by_external_id(data.identifier)
resolution_method = "external_id" if entity else "search"
# If not found by external_id, try other resolution methods
# Pass source_path for context-aware resolution (prefers notes closer to source)
# Pass strict to control fuzzy search fallback (default False allows fuzzy matching)
if not entity:
entity = await link_resolver.resolve_link(
data.identifier, source_path=data.source_path, strict=data.strict
)
if entity:
# Determine resolution method
if entity.permalink == data.identifier:
resolution_method = "permalink"
elif entity.title == data.identifier:
resolution_method = "title"
elif entity.file_path == data.identifier:
resolution_method = "path"
else:
resolution_method = "search"
if not entity:
raise HTTPException(status_code=404, detail=f"Entity not found: '{data.identifier}'")
result = EntityResolveResponse(
external_id=entity.external_id,
entity_id=entity.id,
permalink=entity.permalink,
file_path=entity.file_path,
title=entity.title,
resolution_method=resolution_method,
)
logger.info(
f"API v2 response: resolved '{data.identifier}' to external_id={result.external_id} via {resolution_method}"
)
return result
## Read endpoints
@router.get("/entities/{entity_id}", response_model=EntityResponseV2)
async def get_entity_by_id(
project_id: ProjectExternalIdPathDep,
entity_repository: EntityRepositoryV2ExternalDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
) -> EntityResponseV2:
"""Get an entity by its external ID (UUID).
This is the primary entity retrieval method in v2, using stable UUID
identifiers that won't change with file moves.
Args:
entity_id: External ID (UUID string)
Returns:
Complete entity with observations and relations
Raises:
HTTPException: 404 if entity not found
"""
logger.info(f"API v2 request: get_entity_by_id entity_id={entity_id}")
entity = await entity_repository.get_by_external_id(entity_id)
if not entity:
raise HTTPException(
status_code=404, detail=f"Entity with external_id '{entity_id}' not found"
)
result = EntityResponseV2.model_validate(entity)
logger.info(f"API v2 response: external_id={entity_id}, title='{result.title}'")
return result
## Create endpoints
@router.post("/entities", response_model=EntityResponseV2)
async def create_entity(
project_id: ProjectExternalIdPathDep,
data: Entity,
background_tasks: BackgroundTasks,
entity_service: EntityServiceV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
task_scheduler: TaskSchedulerDep,
file_service: FileServiceV2ExternalDep,
app_config: AppConfigDep,
fast: bool = Query(
True, description="If true, write quickly and defer indexing to background tasks."
),
) -> EntityResponseV2:
"""Create a new entity.
Args:
data: Entity data to create
fast: If True, defer indexing to background tasks
Returns:
Created entity with generated external_id (UUID) and file content
"""
logger.info(
"API v2 request", endpoint="create_entity", entity_type=data.entity_type, title=data.title
)
if fast:
entity = await entity_service.fast_write_entity(data)
task_scheduler.schedule(
"reindex_entity",
entity_id=entity.id,
project_id=project_id,
)
else:
entity = await entity_service.create_entity(data)
await search_service.index_entity(entity)
_schedule_vector_sync_if_enabled(
task_scheduler=task_scheduler,
app_config=app_config,
entity_id=entity.id,
project_id=project_id,
)
result = EntityResponseV2.model_validate(entity)
if fast:
result = result.model_copy(update={"observations": [], "relations": []})
# Always read and return file content
content = await file_service.read_file_content(entity.file_path)
result = result.model_copy(update={"content": content})
logger.info(
f"API v2 response: endpoint='create_entity' external_id={entity.external_id}, title={result.title}, permalink={result.permalink}, status_code=201"
)
return result
## Update endpoints
@router.put("/entities/{entity_id}", response_model=EntityResponseV2)
async def update_entity_by_id(
data: Entity,
response: Response,
background_tasks: BackgroundTasks,
project_id: ProjectExternalIdPathDep,
entity_service: EntityServiceV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
task_scheduler: TaskSchedulerDep,
file_service: FileServiceV2ExternalDep,
app_config: AppConfigDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
fast: bool = Query(
True, description="If true, write quickly and defer indexing to background tasks."
),
) -> EntityResponseV2:
"""Update an entity by external ID.
If the entity doesn't exist, it will be created (upsert behavior).
Args:
entity_id: External ID (UUID string)
data: Updated entity data
fast: If True, defer indexing to background tasks
Returns:
Updated entity with file content
"""
logger.info(f"API v2 request: update_entity_by_id entity_id={entity_id}")
# Check if entity exists (external_id is the source of truth for v2)
existing = await entity_repository.get_by_external_id(entity_id)
created = existing is None
if fast:
entity = await entity_service.fast_write_entity(data, external_id=entity_id)
response.status_code = 200 if existing else 201
task_scheduler.schedule(
"reindex_entity",
entity_id=entity.id,
project_id=project_id,
resolve_relations=created,
)
else:
if existing:
# Update the existing entity in-place to avoid path-based duplication
entity = await entity_service.update_entity(existing, data)
response.status_code = 200
else:
# Create new entity, then bind external_id to the requested UUID
entity = await entity_service.create_entity(data)
if entity.external_id != entity_id:
entity = await entity_repository.update(
entity.id,
{"external_id": entity_id},
)
if not entity:
raise HTTPException(
status_code=404,
detail=f"Entity with external_id '{entity_id}' not found",
)
response.status_code = 201
await search_service.index_entity(entity)
_schedule_vector_sync_if_enabled(
task_scheduler=task_scheduler,
app_config=app_config,
entity_id=entity.id,
project_id=project_id,
)
result = EntityResponseV2.model_validate(entity)
if fast:
result = result.model_copy(update={"observations": [], "relations": []})
# Always read and return file content
content = await file_service.read_file_content(entity.file_path)
result = result.model_copy(update={"content": content})
logger.info(
f"API v2 response: external_id={entity_id}, created={created}, status_code={response.status_code}"
)
return result
@router.patch("/entities/{entity_id}", response_model=EntityResponseV2)
async def edit_entity_by_id(
data: EditEntityRequest,
background_tasks: BackgroundTasks,
project_id: ProjectExternalIdPathDep,
entity_service: EntityServiceV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
task_scheduler: TaskSchedulerDep,
file_service: FileServiceV2ExternalDep,
app_config: AppConfigDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
fast: bool = Query(
True, description="If true, write quickly and defer indexing to background tasks."
),
) -> EntityResponseV2:
"""Edit an existing entity by external ID using operations like append, prepend, etc.
Args:
entity_id: External ID (UUID string)
data: Edit operation details
fast: If True, defer indexing to background tasks
Returns:
Updated entity with file content
Raises:
HTTPException: 404 if entity not found, 400 if edit fails
"""
logger.info(
f"API v2 request: edit_entity_by_id entity_id={entity_id}, operation='{data.operation}'"
)
# Verify entity exists
entity = await entity_repository.get_by_external_id(entity_id)
if not entity: # pragma: no cover
raise HTTPException(
status_code=404, detail=f"Entity with external_id '{entity_id}' not found"
)
try:
if fast:
updated_entity = await entity_service.fast_edit_entity(
entity=entity,
operation=data.operation,
content=data.content,
section=data.section,
find_text=data.find_text,
expected_replacements=data.expected_replacements,
)
task_scheduler.schedule(
"reindex_entity",
entity_id=updated_entity.id,
project_id=project_id,
)
else:
# Edit using the entity's permalink or path
identifier = entity.permalink or entity.file_path
updated_entity = await entity_service.edit_entity(
identifier=identifier,
operation=data.operation,
content=data.content,
section=data.section,
find_text=data.find_text,
expected_replacements=data.expected_replacements,
)
await search_service.index_entity(updated_entity)
_schedule_vector_sync_if_enabled(
task_scheduler=task_scheduler,
app_config=app_config,
entity_id=updated_entity.id,
project_id=project_id,
)
result = EntityResponseV2.model_validate(updated_entity)
if fast:
result = result.model_copy(update={"observations": [], "relations": []})
# Always read and return file content
content = await file_service.read_file_content(updated_entity.file_path)
result = result.model_copy(update={"content": content})
logger.info(
f"API v2 response: external_id={entity_id}, operation='{data.operation}', status_code=200"
)
return result
except Exception as e:
logger.error(f"Error editing entity {entity_id}: {e}")
raise HTTPException(status_code=400, detail=str(e))
## Delete endpoints
@router.delete("/entities/{entity_id}", response_model=DeleteEntitiesResponse)
async def delete_entity_by_id(
background_tasks: BackgroundTasks,
project_id: ProjectExternalIdPathDep,
entity_service: EntityServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
search_service=Depends(lambda: None), # Optional for now
) -> DeleteEntitiesResponse:
"""Delete an entity by external ID.
Args:
entity_id: External ID (UUID string)
Returns:
Deletion status
Note: Returns deleted=False if entity doesn't exist (idempotent)
"""
logger.info(f"API v2 request: delete_entity_by_id entity_id={entity_id}")
entity = await entity_repository.get_by_external_id(entity_id)
if entity is None:
logger.info(f"API v2 response: external_id={entity_id} not found, deleted=False")
return DeleteEntitiesResponse(deleted=False)
# Delete the entity using internal ID
deleted = await entity_service.delete_entity(entity.id)
# Remove from search index if search service available
if search_service:
background_tasks.add_task(search_service.handle_delete, entity) # pragma: no cover
logger.info(f"API v2 response: external_id={entity_id}, deleted={deleted}")
return DeleteEntitiesResponse(deleted=deleted)
## Move endpoint
@router.put("/entities/{entity_id}/move", response_model=EntityResponseV2)
async def move_entity(
data: MoveEntityRequestV2,
background_tasks: BackgroundTasks,
project_id: ProjectExternalIdPathDep,
entity_service: EntityServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
project_config: ProjectConfigV2ExternalDep,
app_config: AppConfigDep,
search_service: SearchServiceV2ExternalDep,
task_scheduler: TaskSchedulerDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
) -> EntityResponseV2:
"""Move an entity to a new file location.
V2 API uses external_id (UUID) in the URL path for stable references.
The external_id will remain stable after the move.
Args:
project_id: Project external ID from URL path
entity_id: Entity external ID from URL path (primary identifier)
data: Move request with destination path only
Returns:
Updated entity with new file path
"""
logger.info(
f"API v2 request: move_entity entity_id={entity_id}, destination='{data.destination_path}'"
)
try:
# First, get the entity by external_id to verify it exists
entity = await entity_repository.get_by_external_id(entity_id)
if not entity: # pragma: no cover
raise HTTPException(
status_code=404, detail=f"Entity with external_id '{entity_id}' not found"
)
# Move the entity using its current file path as identifier
moved_entity = await entity_service.move_entity(
identifier=entity.file_path, # Use file path for resolution
destination_path=data.destination_path,
project_config=project_config,
app_config=app_config,
)
# Reindex at new location
reindexed_entity = await entity_service.link_resolver.resolve_link(data.destination_path)
if reindexed_entity:
await search_service.index_entity(reindexed_entity)
_schedule_vector_sync_if_enabled(
task_scheduler=task_scheduler,
app_config=app_config,
entity_id=reindexed_entity.id,
project_id=project_id,
)
result = EntityResponseV2.model_validate(moved_entity)
logger.info(f"API v2 response: moved external_id={entity_id} to '{data.destination_path}'")
return result
except HTTPException: # pragma: no cover
raise # pragma: no cover
except Exception as e:
logger.error(f"Error moving entity: {e}")
raise HTTPException(status_code=400, detail=str(e))
## Move directory endpoint
@router.post("/move-directory", response_model=DirectoryMoveResult)
async def move_directory(
data: MoveDirectoryRequestV2,
background_tasks: BackgroundTasks,
project_id: ProjectExternalIdPathDep,
entity_service: EntityServiceV2ExternalDep,
project_config: ProjectConfigV2ExternalDep,
app_config: AppConfigDep,
search_service: SearchServiceV2ExternalDep,
task_scheduler: TaskSchedulerDep,
) -> DirectoryMoveResult:
"""Move all entities in a directory to a new location.
V2 API uses project external_id in the URL path for stable references.
Moves all files within a source directory to a destination directory,
updating database records and optionally updating permalinks.
Args:
project_id: Project external ID from URL path
data: Move request with source and destination directories
Returns:
DirectoryMoveResult with counts and details of moved files
"""
logger.info(
f"API v2 request: move_directory source='{data.source_directory}', destination='{data.destination_directory}'"
)
try:
# Move the directory using the service
result = await entity_service.move_directory(
source_directory=data.source_directory,
destination_directory=data.destination_directory,
project_config=project_config,
app_config=app_config,
)
# Reindex moved entities
for file_path in result.moved_files:
entity = await entity_service.link_resolver.resolve_link(file_path)
if entity:
await search_service.index_entity(entity)
_schedule_vector_sync_if_enabled(
task_scheduler=task_scheduler,
app_config=app_config,
entity_id=entity.id,
project_id=project_id,
)
logger.info(
f"API v2 response: move_directory "
f"total={result.total_files}, success={result.successful_moves}, failed={result.failed_moves}"
)
return result
except Exception as e:
logger.error(f"Error moving directory: {e}")
raise HTTPException(status_code=400, detail=str(e))
## Delete directory endpoint
@router.post("/delete-directory", response_model=DirectoryDeleteResult)
async def delete_directory(
data: DeleteDirectoryRequestV2,
project_id: ProjectExternalIdPathDep,
entity_service: EntityServiceV2ExternalDep,
) -> DirectoryDeleteResult:
"""Delete all entities in a directory.
V2 API uses project external_id in the URL path for stable references.
Deletes all files within a directory, updating database records and
removing files from the filesystem.
Args:
project_id: Project external ID from URL path
data: Delete request with directory path
Returns:
DirectoryDeleteResult with counts and details of deleted files
"""
logger.info(f"API v2 request: delete_directory directory='{data.directory}'")
try:
# Delete the directory using the service
result = await entity_service.delete_directory(
directory=data.directory,
)
logger.info(
f"API v2 response: delete_directory "
f"total={result.total_files}, success={result.successful_deletes}, failed={result.failed_deletes}"
)
return result
except Exception as e:
logger.error(f"Error deleting directory: {e}")
raise HTTPException(status_code=400, detail=str(e))
@@ -1,130 +0,0 @@
"""V2 routes for memory:// URI operations.
This router uses external_id UUIDs for stable, API-friendly routing.
V1 uses string-based project names which are less efficient and less stable.
"""
from typing import Annotated, Optional
from fastapi import APIRouter, Query, Path
from loguru import logger
from basic_memory.deps import ContextServiceV2ExternalDep, EntityRepositoryV2ExternalDep
from basic_memory.schemas.base import TimeFrame, parse_timeframe
from basic_memory.schemas.memory import (
GraphContext,
normalize_memory_url,
)
from basic_memory.schemas.search import SearchItemType
from basic_memory.api.v2.utils import to_graph_context
# Note: No prefix here - it's added during registration as /v2/{project_id}/memory
router = APIRouter(tags=["memory"])
@router.get("/memory/recent", response_model=GraphContext)
async def recent(
context_service: ContextServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
type: Annotated[list[SearchItemType] | None, Query()] = None,
depth: int = 1,
timeframe: TimeFrame = "7d",
page: int = 1,
page_size: int = 10,
max_related: int = 10,
) -> GraphContext:
"""Get recent activity context for a project.
Args:
project_id: Project external UUID from URL path
context_service: Context service scoped to project
entity_repository: Entity repository scoped to project
type: Types of items to include (entities, relations, observations)
depth: How many levels of related entities to include
timeframe: Time window for recent activity (e.g., "7d", "1 week")
page: Page number for pagination
page_size: Number of items per page
max_related: Maximum related entities to include per item
Returns:
GraphContext with recent activity and related entities
"""
# return all types by default
types = (
[SearchItemType.ENTITY, SearchItemType.RELATION, SearchItemType.OBSERVATION]
if not type
else type
)
logger.debug(
f"V2 Getting recent context for project {project_id}: `{types}` depth: `{depth}` timeframe: `{timeframe}` page: `{page}` page_size: `{page_size}` max_related: `{max_related}`"
)
# Parse timeframe
since = parse_timeframe(timeframe)
limit = page_size
offset = (page - 1) * page_size
# Build context
context = await context_service.build_context(
types=types, depth=depth, since=since, limit=limit, offset=offset, max_related=max_related
)
recent_context = await to_graph_context(
context, entity_repository=entity_repository, page=page, page_size=page_size
)
logger.debug(f"V2 Recent context: {recent_context.model_dump_json()}")
return recent_context
# get_memory_context needs to be declared last so other paths can match
@router.get("/memory/{uri:path}", response_model=GraphContext)
async def get_memory_context(
context_service: ContextServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
uri: str,
project_id: str = Path(..., description="Project external UUID"),
depth: int = 1,
timeframe: Optional[TimeFrame] = None,
page: int = 1,
page_size: int = 10,
max_related: int = 10,
) -> GraphContext:
"""Get rich context from memory:// URI.
V2 supports both legacy path-based URIs and new ID-based URIs:
- Legacy: memory://path/to/note
- ID-based: memory://id/123 or memory://123
Args:
project_id: Project external UUID from URL path
context_service: Context service scoped to project
entity_repository: Entity repository scoped to project
uri: Memory URI path (e.g., "id/123", "123", or "path/to/note")
depth: How many levels of related entities to include
timeframe: Optional time window for filtering related content
page: Page number for pagination
page_size: Number of items per page
max_related: Maximum related entities to include
Returns:
GraphContext with the entity and its related context
"""
logger.debug(
f"V2 Getting context for project {project_id}, URI: `{uri}` depth: `{depth}` timeframe: `{timeframe}` page: `{page}` page_size: `{page_size}` max_related: `{max_related}`"
)
memory_url = normalize_memory_url(uri)
# Parse timeframe
since = parse_timeframe(timeframe) if timeframe else None
limit = page_size
offset = (page - 1) * page_size
# Build context
context = await context_service.build_context(
memory_url, depth=depth, since=since, limit=limit, offset=offset, max_related=max_related
)
return await to_graph_context(
context, entity_repository=entity_repository, page=page, page_size=page_size
)
@@ -1,550 +0,0 @@
"""V2 Project Router - External ID-based project management operations.
This router provides external_id (UUID) based CRUD operations for projects,
using stable string UUIDs that never change (unlike integer IDs or names).
Key improvements:
- Stable external UUIDs that won't change with renames or database migrations
- Better API ergonomics with consistent string identifiers
- Direct database lookups via unique indexed column
- Consistent with v2 entity operations
"""
import os
from typing import Optional
from fastapi import APIRouter, HTTPException, Body, Query, Path
from loguru import logger
from basic_memory.deps import (
ProjectServiceDep,
ProjectRepositoryDep,
ProjectConfigV2ExternalDep,
SyncServiceV2ExternalDep,
TaskSchedulerDep,
ProjectExternalIdPathDep,
)
from basic_memory.schemas import SyncReportResponse
from basic_memory.schemas.project_info import (
ProjectItem,
ProjectList,
ProjectInfoRequest,
ProjectInfoResponse,
ProjectStatusResponse,
)
from basic_memory.schemas.v2 import ProjectResolveRequest, ProjectResolveResponse
from basic_memory.utils import normalize_project_path, generate_permalink
router = APIRouter(prefix="/projects", tags=["project_management-v2"])
@router.get("/", response_model=ProjectList)
async def list_projects(
project_service: ProjectServiceDep,
) -> ProjectList:
"""List all configured projects.
Returns:
A list of all projects with metadata
"""
projects = await project_service.list_projects()
default_project = project_service.default_project
project_items = [
ProjectItem(
id=project.id,
external_id=project.external_id,
name=project.name,
path=normalize_project_path(project.path),
is_default=project.is_default or False,
)
for project in projects
]
return ProjectList(
projects=project_items,
default_project=default_project,
)
@router.post("/", response_model=ProjectStatusResponse, status_code=201)
async def add_project(
project_data: ProjectInfoRequest,
project_service: ProjectServiceDep,
) -> ProjectStatusResponse:
"""Add a new project to configuration and database.
Args:
project_data: The project name and path, with option to set as default
Returns:
Response confirming the project was added
"""
# Check if project already exists before attempting to add
existing_project = await project_service.get_project(project_data.name)
if existing_project:
# Project exists - check if paths match for true idempotency
# Normalize paths for comparison (resolve symlinks, etc.)
requested_path = os.path.abspath(os.path.expanduser(project_data.path))
existing_path = os.path.abspath(os.path.expanduser(existing_project.path))
if requested_path == existing_path:
# Same name, same path - return 200 OK (idempotent)
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message=f"Project '{project_data.name}' already exists",
status="success",
default=existing_project.is_default or False,
new_project=ProjectItem(
id=existing_project.id,
external_id=existing_project.external_id,
name=existing_project.name,
path=existing_project.path,
is_default=existing_project.is_default or False,
),
)
else:
# Same name, different path - this is an error
raise HTTPException(
status_code=400,
detail=(
f"Project '{project_data.name}' already exists with different path. "
f"Existing: {existing_project.path}, Requested: {project_data.path}"
),
)
try: # pragma: no cover
# The service layer handles cloud mode validation and path sanitization
await project_service.add_project(
project_data.name, project_data.path, set_default=project_data.set_default
)
# Fetch the newly created project to get its ID
new_project = await project_service.get_project(project_data.name)
if not new_project:
raise HTTPException(status_code=500, detail="Failed to retrieve newly created project")
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message=f"Project '{new_project.name}' added successfully",
status="success",
default=project_data.set_default,
new_project=ProjectItem(
id=new_project.id,
external_id=new_project.external_id,
name=new_project.name,
path=new_project.path,
is_default=new_project.is_default or False,
),
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e))
@router.post("/config/sync", response_model=ProjectStatusResponse)
async def synchronize_projects(
project_service: ProjectServiceDep,
) -> ProjectStatusResponse:
"""Synchronize projects between configuration file and database."""
try: # pragma: no cover
await project_service.synchronize_projects()
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message="Projects synchronized successfully between configuration and database",
status="success",
default=False,
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e))
@router.post("/{project_id}/sync")
async def sync_project(
sync_service: SyncServiceV2ExternalDep,
project_config: ProjectConfigV2ExternalDep,
task_scheduler: TaskSchedulerDep,
project_internal_id: ProjectExternalIdPathDep,
force_full: bool = Query(
False, description="Force full scan, bypassing watermark optimization"
),
run_in_background: bool = Query(True, description="Run in background"),
):
"""Force project filesystem sync to database."""
if run_in_background:
task_scheduler.schedule(
"sync_project",
project_id=project_internal_id,
force_full=force_full,
)
logger.info(
f"Filesystem sync initiated for project: {project_config.name} (force_full={force_full})"
)
return {
"status": "sync_started",
"message": f"Filesystem sync initiated for project '{project_config.name}'",
}
report = await sync_service.sync(
project_config.home, project_config.name, force_full=force_full
)
logger.info(
f"Filesystem sync completed for project: {project_config.name} (force_full={force_full})"
)
return SyncReportResponse.from_sync_report(report)
@router.post("/{project_id}/status", response_model=SyncReportResponse)
async def get_project_status(
sync_service: SyncServiceV2ExternalDep,
project_config: ProjectConfigV2ExternalDep,
project_id: str = Path(..., description="Project external ID (UUID)"),
force_full: bool = Query(
False, description="Force full scan, bypassing watermark optimization"
),
) -> SyncReportResponse:
"""Get sync status of files vs database for a project."""
logger.info(f"API v2 request: get_project_status for project_id={project_id}")
report = await sync_service.scan(project_config.home, force_full=force_full)
return SyncReportResponse.from_sync_report(report)
@router.post("/resolve", response_model=ProjectResolveResponse)
async def resolve_project_identifier(
data: ProjectResolveRequest,
project_repository: ProjectRepositoryDep,
) -> ProjectResolveResponse:
"""Resolve a project identifier (name, permalink, or external_id) to project info.
This endpoint provides efficient lookup of projects by various identifiers
without needing to fetch the entire project list. Supports:
- External ID (UUID string) - preferred stable identifier
- Permalink
- Case-insensitive name matching
Args:
data: Request containing the identifier to resolve
Returns:
Project information including the external_id (UUID)
Raises:
HTTPException: 404 if project not found
Example:
POST /v2/projects/resolve
{"identifier": "my-project"}
Returns:
{
"external_id": "550e8400-e29b-41d4-a716-446655440000",
"project_id": 1,
"name": "my-project",
"permalink": "my-project",
"path": "/path/to/project",
"is_active": true,
"is_default": false,
"resolution_method": "name"
}
"""
logger.info(f"API v2 request: resolve_project_identifier for '{data.identifier}'")
# Generate permalink for comparison
identifier_permalink = generate_permalink(data.identifier)
resolution_method = "name"
project = None
# Try external_id first (UUID format)
project = await project_repository.get_by_external_id(data.identifier)
if project:
resolution_method = "external_id"
# If not found by external_id, try by permalink (exact match)
if not project:
project = await project_repository.get_by_permalink(identifier_permalink)
if project:
resolution_method = "permalink"
# If not found by permalink, try case-insensitive name search
if not project:
project = await project_repository.get_by_name_case_insensitive(data.identifier)
if project:
resolution_method = "name" # pragma: no cover
if not project:
raise HTTPException(status_code=404, detail=f"Project not found: '{data.identifier}'")
return ProjectResolveResponse(
external_id=project.external_id,
project_id=project.id,
name=project.name,
permalink=generate_permalink(project.name),
path=normalize_project_path(project.path),
is_active=project.is_active if hasattr(project, "is_active") else True,
is_default=project.is_default or False,
resolution_method=resolution_method,
)
@router.get("/{project_id}", response_model=ProjectItem)
async def get_project_by_id(
project_repository: ProjectRepositoryDep,
project_id: str = Path(..., description="Project external ID (UUID)"),
) -> ProjectItem:
"""Get project by its external ID (UUID).
This is the primary project retrieval method in v2, using stable UUID
identifiers that won't change with project renames.
Args:
project_id: External ID (UUID string)
Returns:
Project information including external_id
Raises:
HTTPException: 404 if project not found
Example:
GET /v2/projects/550e8400-e29b-41d4-a716-446655440000
"""
logger.info(f"API v2 request: get_project_by_id for project_id={project_id}")
project = await project_repository.get_by_external_id(project_id)
if not project:
raise HTTPException(
status_code=404, detail=f"Project with external_id '{project_id}' not found"
)
return ProjectItem(
id=project.id,
external_id=project.external_id,
name=project.name,
path=normalize_project_path(project.path),
is_default=project.is_default or False,
)
@router.get("/{project_id}/info", response_model=ProjectInfoResponse)
async def get_project_info_by_id(
project_service: ProjectServiceDep,
project_repository: ProjectRepositoryDep,
project_id: str = Path(..., description="Project external ID (UUID)"),
) -> ProjectInfoResponse:
"""Get detailed project information by external ID."""
logger.info(f"API v2 request: get_project_info_by_id for project_id={project_id}")
project = await project_repository.get_by_external_id(project_id)
if not project:
raise HTTPException(
status_code=404, detail=f"Project with external_id '{project_id}' not found"
)
return await project_service.get_project_info(project.name)
@router.patch("/{project_id}", response_model=ProjectStatusResponse)
async def update_project_by_id(
project_service: ProjectServiceDep,
project_repository: ProjectRepositoryDep,
project_id: str = Path(..., description="Project external ID (UUID)"),
path: Optional[str] = Body(None, description="New absolute path for the project"),
is_active: Optional[bool] = Body(None, description="Status of the project (active/inactive)"),
) -> ProjectStatusResponse:
"""Update a project's information by external ID.
Args:
project_id: External ID (UUID string)
path: Optional new absolute path for the project
is_active: Optional status update for the project
Returns:
Response confirming the project was updated
Raises:
HTTPException: 400 if validation fails, 404 if project not found
Example:
PATCH /v2/projects/550e8400-e29b-41d4-a716-446655440000
{"path": "/new/path"}
"""
logger.info(f"API v2 request: update_project_by_id for project_id={project_id}")
try:
# Validate that path is absolute if provided
if path and not os.path.isabs(path):
raise HTTPException(status_code=400, detail="Path must be absolute")
# Get original project info for the response
old_project = await project_repository.get_by_external_id(project_id)
if not old_project:
raise HTTPException(
status_code=404, detail=f"Project with external_id '{project_id}' not found"
)
old_project_info = ProjectItem(
id=old_project.id,
external_id=old_project.external_id,
name=old_project.name,
path=old_project.path,
is_default=old_project.is_default or False,
)
# Update using project name (service layer still uses names internally)
if path:
await project_service.move_project(old_project.name, path)
elif is_active is not None:
await project_service.update_project(old_project.name, is_active=is_active)
# Get updated project info (use the same external_id)
updated_project = await project_repository.get_by_external_id(project_id)
if not updated_project: # pragma: no cover
raise HTTPException(
status_code=404,
detail=f"Project with external_id '{project_id}' not found after update",
)
return ProjectStatusResponse(
message=f"Project '{updated_project.name}' updated successfully",
status="success",
default=old_project.is_default or False,
old_project=old_project_info,
new_project=ProjectItem(
id=updated_project.id,
external_id=updated_project.external_id,
name=updated_project.name,
path=updated_project.path,
is_default=updated_project.is_default or False,
),
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e)) # pragma: no cover
@router.delete("/{project_id}", response_model=ProjectStatusResponse)
async def delete_project_by_id(
project_service: ProjectServiceDep,
project_repository: ProjectRepositoryDep,
project_id: str = Path(..., description="Project external ID (UUID)"),
delete_notes: bool = Query(
False, description="If True, delete project directory from filesystem"
),
) -> ProjectStatusResponse:
"""Delete a project by external ID.
Args:
project_id: External ID (UUID string)
delete_notes: If True, delete the project directory from the filesystem
Returns:
Response confirming the project was deleted
Raises:
HTTPException: 400 if trying to delete default project, 404 if not found
Example:
DELETE /v2/projects/550e8400-e29b-41d4-a716-446655440000?delete_notes=false
"""
logger.info(
f"API v2 request: delete_project_by_id for project_id={project_id}, delete_notes={delete_notes}"
)
try:
old_project = await project_repository.get_by_external_id(project_id)
if not old_project:
raise HTTPException(
status_code=404, detail=f"Project with external_id '{project_id}' not found"
)
# Check if trying to delete the default project
# Use is_default from database, not ConfigManager (which doesn't work in cloud mode)
if old_project.is_default:
available_projects = await project_service.list_projects()
other_projects = [p.name for p in available_projects if p.external_id != project_id]
detail = f"Cannot delete default project '{old_project.name}'. "
if other_projects:
detail += ( # pragma: no cover
f"Set another project as default first. Available: {', '.join(other_projects)}"
)
else:
detail += "This is the only project in your configuration." # pragma: no cover
raise HTTPException(status_code=400, detail=detail)
# Delete using project name (service layer still uses names internally)
await project_service.remove_project(old_project.name, delete_notes=delete_notes)
return ProjectStatusResponse(
message=f"Project '{old_project.name}' removed successfully",
status="success",
default=False,
old_project=ProjectItem(
id=old_project.id,
external_id=old_project.external_id,
name=old_project.name,
path=old_project.path,
is_default=old_project.is_default or False,
),
new_project=None,
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e)) # pragma: no cover
@router.put("/{project_id}/default", response_model=ProjectStatusResponse)
async def set_default_project_by_id(
project_service: ProjectServiceDep,
project_repository: ProjectRepositoryDep,
project_id: str = Path(..., description="Project external ID (UUID)"),
) -> ProjectStatusResponse:
"""Set a project as the default project by external ID.
Args:
project_id: External ID (UUID string) to set as default
Returns:
Response confirming the project was set as default
Raises:
HTTPException: 404 if project not found
Example:
PUT /v2/projects/550e8400-e29b-41d4-a716-446655440000/default
"""
logger.info(f"API v2 request: set_default_project_by_id for project_id={project_id}")
try:
# Get the old default project from database
default_project = await project_repository.get_default_project()
if not default_project:
raise HTTPException( # pragma: no cover
status_code=404, detail="No default project is currently set"
)
# Get the new default project by external_id
new_default_project = await project_repository.get_by_external_id(project_id)
if not new_default_project:
raise HTTPException(
status_code=404, detail=f"Project with external_id '{project_id}' not found"
)
# Set as default using project name (service layer still uses names internally)
await project_service.set_default_project(new_default_project.name)
return ProjectStatusResponse(
message=f"Project '{new_default_project.name}' set as default successfully",
status="success",
default=True,
old_project=ProjectItem(
id=default_project.id,
external_id=default_project.external_id,
name=default_project.name,
path=default_project.path,
is_default=False,
),
new_project=ProjectItem(
id=new_default_project.id,
external_id=new_default_project.external_id,
name=new_default_project.name,
path=new_default_project.path,
is_default=True,
),
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e)) # pragma: no cover
@@ -1,289 +0,0 @@
"""V2 Resource Router - ID-based resource content operations.
This router uses entity external_ids (UUIDs) for all operations, with file paths
in request bodies when needed. This is consistent with v2's external_id-first design.
Key differences from v1:
- Uses UUID external_ids in URL paths instead of integer IDs or file paths
- File paths are in request bodies for create/update operations
- More RESTful: POST for create, PUT for update, GET for read
"""
import uuid
from pathlib import Path as PathLib
from fastapi import APIRouter, HTTPException, Response, Path
from loguru import logger
from basic_memory.deps import (
ProjectConfigV2ExternalDep,
FileServiceV2ExternalDep,
EntityRepositoryV2ExternalDep,
SearchServiceV2ExternalDep,
)
from basic_memory.models.knowledge import Entity as EntityModel
from basic_memory.schemas.v2.resource import (
CreateResourceRequest,
UpdateResourceRequest,
ResourceResponse,
)
from basic_memory.utils import validate_project_path
router = APIRouter(prefix="/resource", tags=["resources-v2"])
@router.get("/{entity_id}")
async def get_resource_content(
config: ProjectConfigV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
file_service: FileServiceV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
entity_id: str = Path(..., description="Entity external UUID"),
) -> Response:
"""Get raw resource content by entity external_id.
Args:
project_id: Project external UUID from URL path
entity_id: Entity external UUID
config: Project configuration
entity_repository: Entity repository for fetching entity data
file_service: File service for reading file content
Returns:
Response with entity content
Raises:
HTTPException: 404 if entity or file not found
"""
logger.debug(f"V2 Getting content for project {project_id}, entity_id: {entity_id}")
# Get entity by external_id
entity = await entity_repository.get_by_external_id(entity_id)
if not entity:
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
# Validate entity file path to prevent path traversal
project_path = PathLib(config.home)
if not validate_project_path(entity.file_path, project_path):
logger.error( # pragma: no cover
f"Invalid file path in entity {entity.id}: {entity.file_path}"
)
raise HTTPException( # pragma: no cover
status_code=500,
detail="Entity contains invalid file path",
)
# Check file exists via file_service (for cloud compatibility)
if not await file_service.exists(entity.file_path):
raise HTTPException( # pragma: no cover
status_code=404,
detail=f"File not found: {entity.file_path}",
)
# Read content via file_service as bytes (works with both local and S3)
content = await file_service.read_file_bytes(entity.file_path)
content_type = file_service.content_type(entity.file_path)
return Response(content=content, media_type=content_type)
@router.post("", response_model=ResourceResponse)
async def create_resource(
data: CreateResourceRequest,
config: ProjectConfigV2ExternalDep,
file_service: FileServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
) -> ResourceResponse:
"""Create a new resource file.
Args:
project_id: Project external UUID from URL path
data: Create resource request with file_path and content
config: Project configuration
file_service: File service for writing files
entity_repository: Entity repository for creating entities
search_service: Search service for indexing
Returns:
ResourceResponse with file information including entity_id and external_id
Raises:
HTTPException: 400 for invalid file paths, 409 if file already exists
"""
try:
# Validate path to prevent path traversal attacks
project_path = PathLib(config.home)
if not validate_project_path(data.file_path, project_path):
logger.warning(
f"Invalid file path attempted: {data.file_path} in project {config.name}"
)
raise HTTPException(
status_code=400,
detail=f"Invalid file path: {data.file_path}. "
"Path must be relative and stay within project boundaries.",
)
# Check if entity already exists
existing_entity = await entity_repository.get_by_file_path(data.file_path)
if existing_entity:
raise HTTPException(
status_code=409,
detail=f"Resource already exists at {data.file_path} with entity_id {existing_entity.external_id}. "
f"Use PUT /resource/{existing_entity.external_id} to update it.",
)
# Cloud compatibility: avoid assuming a local filesystem path.
# Delegate directory creation + writes to FileService (local or S3).
await file_service.ensure_directory(PathLib(data.file_path).parent)
checksum = await file_service.write_file(data.file_path, data.content)
# Get file info
file_metadata = await file_service.get_file_metadata(data.file_path)
# Determine file details
file_name = PathLib(data.file_path).name
content_type = file_service.content_type(data.file_path)
entity_type = "canvas" if data.file_path.endswith(".canvas") else "file"
# Create a new entity model
# Explicitly set external_id to ensure NOT NULL constraint is satisfied (fixes #512)
entity = EntityModel(
external_id=str(uuid.uuid4()),
title=file_name,
entity_type=entity_type,
content_type=content_type,
file_path=data.file_path,
checksum=checksum,
created_at=file_metadata.created_at,
updated_at=file_metadata.modified_at,
)
entity = await entity_repository.add(entity)
# Index the file for search
await search_service.index_entity(entity) # pyright: ignore
# Return success response
return ResourceResponse(
entity_id=entity.id,
external_id=entity.external_id,
file_path=data.file_path,
checksum=checksum,
size=file_metadata.size,
created_at=file_metadata.created_at.timestamp(),
modified_at=file_metadata.modified_at.timestamp(),
)
except HTTPException:
# Re-raise HTTP exceptions without wrapping
raise
except Exception as e: # pragma: no cover
logger.error(f"Error creating resource {data.file_path}: {e}")
raise HTTPException(status_code=500, detail=f"Failed to create resource: {str(e)}")
@router.put("/{entity_id}", response_model=ResourceResponse)
async def update_resource(
data: UpdateResourceRequest,
config: ProjectConfigV2ExternalDep,
file_service: FileServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
entity_id: str = Path(..., description="Entity external UUID"),
) -> ResourceResponse:
"""Update an existing resource by entity external_id.
Can update content and optionally move the file to a new path.
Args:
project_id: Project external UUID from URL path
entity_id: Entity external UUID of the resource to update
data: Update resource request with content and optional new file_path
config: Project configuration
file_service: File service for writing files
entity_repository: Entity repository for updating entities
search_service: Search service for indexing
Returns:
ResourceResponse with updated file information
Raises:
HTTPException: 404 if entity not found, 400 for invalid paths
"""
try:
# Get existing entity by external_id
entity = await entity_repository.get_by_external_id(entity_id)
if not entity:
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
# Determine target file path
target_file_path = data.file_path if data.file_path else entity.file_path
# Validate path to prevent path traversal attacks
project_path = PathLib(config.home)
if not validate_project_path(target_file_path, project_path):
logger.warning(
f"Invalid file path attempted: {target_file_path} in project {config.name}"
)
raise HTTPException(
status_code=400,
detail=f"Invalid file path: {target_file_path}. "
"Path must be relative and stay within project boundaries.",
)
# If moving file, handle the move
if data.file_path and data.file_path != entity.file_path:
# Ensure new parent directory exists (no-op for S3)
await file_service.ensure_directory(PathLib(target_file_path).parent)
# If old file exists, remove it via file_service (for cloud compatibility)
if await file_service.exists(entity.file_path):
await file_service.delete_file(entity.file_path)
else:
# Ensure directory exists for in-place update
await file_service.ensure_directory(PathLib(target_file_path).parent)
# Write content to target file
checksum = await file_service.write_file(target_file_path, data.content)
# Get file info
file_metadata = await file_service.get_file_metadata(target_file_path)
# Determine file details
file_name = PathLib(target_file_path).name
content_type = file_service.content_type(target_file_path)
entity_type = "canvas" if target_file_path.endswith(".canvas") else "file"
# Update entity using internal ID
updated_entity = await entity_repository.update(
entity.id,
{
"title": file_name,
"entity_type": entity_type,
"content_type": content_type,
"file_path": target_file_path,
"checksum": checksum,
"updated_at": file_metadata.modified_at,
},
)
# Index the updated file for search
await search_service.index_entity(updated_entity) # pyright: ignore
# Return success response
return ResourceResponse(
entity_id=entity.id,
external_id=entity.external_id,
file_path=target_file_path,
checksum=checksum,
size=file_metadata.size,
created_at=file_metadata.created_at.timestamp(),
modified_at=file_metadata.modified_at.timestamp(),
)
except HTTPException:
# Re-raise HTTP exceptions without wrapping
raise
except Exception as e: # pragma: no cover
logger.error(f"Error updating resource {entity_id}: {e}")
raise HTTPException(status_code=500, detail=f"Failed to update resource: {str(e)}")
@@ -1,351 +0,0 @@
"""V2 router for schema operations.
Provides endpoints for schema validation, inference, and drift detection.
The schema system validates notes against Picoschema definitions without
introducing any new data model -- it works entirely with existing
observations and relations.
Flow: Entity loaded with eager observations/relations -> convert to tuples -> core functions.
"""
from pathlib import Path as FilePath
from fastapi import APIRouter, Path, Query
from basic_memory.deps import EntityRepositoryV2ExternalDep
from basic_memory.models.knowledge import Entity
from basic_memory.schemas.schema import (
ValidationReport,
InferenceReport,
DriftReport,
NoteValidationResponse,
FieldResultResponse,
FieldFrequencyResponse,
DriftFieldResponse,
)
from basic_memory.schema.resolver import resolve_schema
from basic_memory.schema.validator import validate_note
from basic_memory.schema.inference import infer_schema, NoteData, ObservationData, RelationData
from basic_memory.schema.diff import diff_schema
from basic_memory.utils import generate_permalink
# Note: No prefix here -- it's added during registration as /v2/{project_id}/schema
router = APIRouter(tags=["schema"])
# --- ORM to core data conversion ---
def _entity_observations(entity: Entity) -> list[ObservationData]:
"""Extract ObservationData from an entity's observations."""
return [ObservationData(obs.category, obs.content) for obs in entity.observations]
def _entity_relations(entity: Entity) -> list[RelationData]:
"""Extract RelationData from an entity's outgoing relations.
Carries the target entity's type on each relation so the inference engine
can suggest correct types (e.g. works_at -> Organization, not the source type).
"""
return [
RelationData(
relation_type=rel.relation_type,
target_name=rel.to_name,
target_entity_type=rel.to_entity.entity_type if rel.to_entity else None,
)
for rel in entity.outgoing_relations
]
def _entity_to_note_data(entity: Entity) -> NoteData:
"""Convert an ORM Entity to a NoteData for inference/diff analysis."""
return NoteData(
identifier=entity.permalink or entity.file_path,
observations=_entity_observations(entity),
relations=_entity_relations(entity),
)
def _entity_frontmatter(entity: Entity) -> dict:
"""Build a frontmatter dict from an entity for schema resolution."""
frontmatter = dict(entity.entity_metadata) if entity.entity_metadata else {}
if entity.entity_type:
frontmatter.setdefault("type", entity.entity_type)
return frontmatter
# --- Validation ---
@router.post("/schema/validate", response_model=ValidationReport)
async def validate_schema(
entity_repository: EntityRepositoryV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
entity_type: str | None = Query(None, description="Entity type to validate"),
identifier: str | None = Query(None, description="Specific note identifier"),
):
"""Validate notes against their resolved schemas.
Validates a specific note (by identifier) or all notes of a given type.
Returns warnings/errors based on the schema's validation mode.
"""
results: list[NoteValidationResponse] = []
# --- Single note validation ---
if identifier:
entity = await entity_repository.get_by_permalink(identifier)
if not entity:
return ValidationReport(entity_type=entity_type, total_notes=0, results=[])
frontmatter = _entity_frontmatter(entity)
schema_ref = frontmatter.get("schema")
async def search_fn(query: str) -> list[dict]:
entities = await _find_schema_entities(
entity_repository,
query,
allow_reference_match=isinstance(schema_ref, str) and query == schema_ref,
)
return [_entity_frontmatter(e) for e in entities]
schema_def = await resolve_schema(frontmatter, search_fn)
if schema_def:
result = validate_note(
entity.permalink or identifier,
schema_def,
_entity_observations(entity),
_entity_relations(entity),
)
results.append(_to_note_validation_response(result))
return ValidationReport(
entity_type=entity_type or entity.entity_type,
total_notes=1,
valid_count=1 if (results and results[0].passed) else 0,
warning_count=sum(len(r.warnings) for r in results),
error_count=sum(len(r.errors) for r in results),
results=results,
)
# --- Batch validation by entity type ---
entities = await _find_by_entity_type(entity_repository, entity_type) if entity_type else []
for entity in entities:
frontmatter = _entity_frontmatter(entity)
schema_ref = frontmatter.get("schema")
async def search_fn(query: str) -> list[dict]:
entities = await _find_schema_entities(
entity_repository,
query,
allow_reference_match=isinstance(schema_ref, str) and query == schema_ref,
)
return [_entity_frontmatter(e) for e in entities]
schema_def = await resolve_schema(frontmatter, search_fn)
if schema_def:
result = validate_note(
entity.permalink or entity.file_path,
schema_def,
_entity_observations(entity),
_entity_relations(entity),
)
results.append(_to_note_validation_response(result))
valid = sum(1 for r in results if r.passed)
return ValidationReport(
entity_type=entity_type,
total_notes=len(results),
total_entities=len(entities),
valid_count=valid,
warning_count=sum(len(r.warnings) for r in results),
error_count=sum(len(r.errors) for r in results),
results=results,
)
# --- Inference ---
@router.post("/schema/infer", response_model=InferenceReport)
async def infer_schema_endpoint(
entity_repository: EntityRepositoryV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
entity_type: str = Query(..., description="Entity type to analyze"),
threshold: float = Query(0.25, description="Minimum frequency for optional fields"),
):
"""Infer a schema from existing notes of a given type.
Examines observation categories and relation types across all notes
of the given type. Returns frequency analysis and suggested Picoschema.
"""
entities = await _find_by_entity_type(entity_repository, entity_type)
notes_data = [_entity_to_note_data(entity) for entity in entities]
result = infer_schema(entity_type, notes_data, optional_threshold=threshold)
return InferenceReport(
entity_type=result.entity_type,
notes_analyzed=result.notes_analyzed,
field_frequencies=[
FieldFrequencyResponse(
name=f.name,
source=f.source,
count=f.count,
total=f.total,
percentage=f.percentage,
sample_values=f.sample_values,
is_array=f.is_array,
target_type=f.target_type,
)
for f in result.field_frequencies
],
suggested_schema=result.suggested_schema,
suggested_required=result.suggested_required,
suggested_optional=result.suggested_optional,
excluded=result.excluded,
)
# --- Drift Detection ---
@router.get("/schema/diff/{entity_type}", response_model=DriftReport)
async def diff_schema_endpoint(
entity_repository: EntityRepositoryV2ExternalDep,
entity_type: str = Path(..., description="Entity type to check for drift"),
project_id: str = Path(..., description="Project external UUID"),
):
"""Show drift between a schema definition and actual note usage.
Compares the existing schema for an entity type against how notes
of that type are actually structured. Identifies new fields, dropped
fields, and cardinality changes.
"""
async def search_fn(query: str) -> list[dict]:
entities = await _find_schema_entities(entity_repository, query)
return [_entity_frontmatter(e) for e in entities]
# Resolve schema by entity type
schema_frontmatter = {"type": entity_type}
schema_def = await resolve_schema(schema_frontmatter, search_fn)
if not schema_def:
return DriftReport(entity_type=entity_type, schema_found=False)
# Collect all notes of this type
entities = await _find_by_entity_type(entity_repository, entity_type)
notes_data = [_entity_to_note_data(entity) for entity in entities]
result = diff_schema(schema_def, notes_data)
return DriftReport(
entity_type=entity_type,
new_fields=[
DriftFieldResponse(
name=f.name,
source=f.source,
count=f.count,
total=f.total,
percentage=f.percentage,
)
for f in result.new_fields
],
dropped_fields=[
DriftFieldResponse(
name=f.name,
source=f.source,
count=f.count,
total=f.total,
percentage=f.percentage,
)
for f in result.dropped_fields
],
cardinality_changes=result.cardinality_changes,
)
# --- Helpers ---
async def _find_by_entity_type(
entity_repository: EntityRepositoryV2ExternalDep,
entity_type: str,
) -> list[Entity]:
"""Find all entities of a given type using the repository's select pattern."""
query = entity_repository.select().where(Entity.entity_type == entity_type)
result = await entity_repository.execute_query(query)
return list(result.scalars().all())
async def _find_schema_entities(
entity_repository: EntityRepositoryV2ExternalDep,
target_entity_type: str,
*,
allow_reference_match: bool = False,
) -> list[Entity]:
"""Find schema entities for resolver lookups.
Resolution strategy:
1) Always try exact entity_metadata['entity'] match (for implicit type lookup
and explicit references that use entity names)
2) Only when allow_reference_match=True and no entity match was found, try
exact reference matching by title/permalink (explicit schema references)
"""
query = entity_repository.select().where(Entity.entity_type == "schema")
result = await entity_repository.execute_query(query)
entities = list(result.scalars().all())
normalized_target = generate_permalink(target_entity_type)
entity_matches = [
e
for e in entities
if e.entity_metadata
and isinstance(e.entity_metadata.get("entity"), str)
and generate_permalink(e.entity_metadata["entity"]) == normalized_target
]
if entity_matches:
return entity_matches
if not allow_reference_match:
return []
reference_matches: list[Entity] = []
for entity in entities:
candidate_refs: list[str] = []
if entity.title:
candidate_refs.append(entity.title)
if entity.permalink:
candidate_refs.append(entity.permalink)
candidate_refs.append(FilePath(entity.permalink).name)
if any(generate_permalink(ref) == normalized_target for ref in candidate_refs):
reference_matches.append(entity)
return reference_matches
def _to_note_validation_response(result) -> NoteValidationResponse:
"""Convert a core ValidationResult to a Pydantic response model."""
return NoteValidationResponse(
note_identifier=result.note_identifier,
schema_entity=result.schema_entity,
passed=result.passed,
field_results=[
FieldResultResponse(
field_name=fr.field.name,
field_type=fr.field.type,
required=fr.field.required,
status=fr.status,
values=fr.values,
message=fr.message,
)
for fr in result.field_results
],
unmatched_observations=result.unmatched_observations,
unmatched_relations=result.unmatched_relations,
warnings=result.warnings,
errors=result.errors,
)
@@ -1,87 +0,0 @@
"""V2 router for search operations.
This router uses external_id UUIDs for stable, API-friendly routing.
V1 uses string-based project names which are less efficient and less stable.
"""
from fastapi import APIRouter, HTTPException, Path
from basic_memory.api.v2.utils import to_search_results
from basic_memory.repository.semantic_errors import (
SemanticDependenciesMissingError,
SemanticSearchDisabledError,
)
from basic_memory.schemas.search import SearchQuery, SearchResponse
from basic_memory.deps import (
SearchServiceV2ExternalDep,
EntityServiceV2ExternalDep,
TaskSchedulerDep,
ProjectExternalIdPathDep,
)
# Note: No prefix here - it's added during registration as /v2/{project_id}/search
router = APIRouter(tags=["search"])
@router.post("/search/", response_model=SearchResponse)
async def search(
query: SearchQuery,
search_service: SearchServiceV2ExternalDep,
entity_service: EntityServiceV2ExternalDep,
project_id: str = Path(..., description="Project external UUID"),
page: int = 1,
page_size: int = 10,
):
"""Search across all knowledge and documents in a project.
V2 uses external_id UUIDs for stable API references.
Args:
project_id: Project external UUID from URL path
query: Search query parameters (text, filters, etc.)
search_service: Search service scoped to project
entity_service: Entity service scoped to project
page: Page number for pagination
page_size: Number of results per page
Returns:
SearchResponse with paginated search results
"""
limit = page_size
offset = (page - 1) * page_size
try:
results = await search_service.search(query, limit=limit, offset=offset)
except SemanticSearchDisabledError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
except SemanticDependenciesMissingError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
search_results = await to_search_results(entity_service, results)
return SearchResponse(
results=search_results,
current_page=page,
page_size=page_size,
)
@router.post("/search/reindex")
async def reindex(
task_scheduler: TaskSchedulerDep,
project_id: ProjectExternalIdPathDep,
):
"""Recreate and populate the search index for a project.
This is a background operation that rebuilds the search index
from scratch. Useful after bulk updates or if the index becomes
corrupted.
Args:
project_id: Project external UUID from URL path
task_scheduler: Task scheduler for background work
Returns:
Status message indicating reindex has been initiated
"""
task_scheduler.schedule("reindex_project", project_id=project_id)
return {"status": "ok", "message": "Reindex initiated"}
+34 -58
View File
@@ -1,24 +1,20 @@
# This prevents DEBUG logs from appearing on stdout during module-level
# initialization (e.g., template_loader.TemplateLoader() logs at DEBUG level).
from loguru import logger
from typing import Optional
logger.remove()
import typer
from typing import Optional # noqa: E402
import typer # noqa: E402
from basic_memory.cli.container import CliContainer, set_container # noqa: E402
from basic_memory.cli.promo import maybe_show_cloud_promo, maybe_show_init_line # noqa: E402
from basic_memory.config import init_cli_logging # noqa: E402
from basic_memory.config import get_project_config
from basic_memory.mcp.project_session import session
def version_callback(value: bool) -> None:
"""Show version and exit."""
if value: # pragma: no cover
import basic_memory
from basic_memory.config import config
typer.echo(f"Basic Memory version: {basic_memory.__version__}")
typer.echo(f"Current project: {config.project}")
typer.echo(f"Project path: {config.home}")
raise typer.Exit()
@@ -28,6 +24,13 @@ app = typer.Typer(name="basic-memory")
@app.callback()
def app_callback(
ctx: typer.Context,
project: Optional[str] = typer.Option(
None,
"--project",
"-p",
help="Specify which project to use 1",
envvar="BASIC_MEMORY_PROJECT",
),
version: Optional[bool] = typer.Option(
None,
"--version",
@@ -39,59 +42,32 @@ def app_callback(
) -> None:
"""Basic Memory - Local-first personal knowledge management."""
# Initialize logging for CLI (file only, no stdout)
init_cli_logging()
# --- Composition Root ---
# Create container and read config (single point of config access)
container = CliContainer.create()
set_container(container)
# Trigger: first-run init confirmation before command output.
# Why: informational "initialized" message belongs above command results, not in the upsell panel.
# Outcome: one-time plain line printed before the subcommand runs.
maybe_show_init_line(ctx.invoked_subcommand)
# Trigger: register promo as a post-command callback.
# Why: promo output should appear after the command's own output, not before.
# Outcome: promo panel renders below the command results (status tree, table, etc.).
ctx.call_on_close(lambda: maybe_show_cloud_promo(ctx.invoked_subcommand))
# Run initialization for commands that don't use the API
# Skip for 'mcp' command - it has its own lifespan that handles initialization
# Skip for API-using commands (status, sync, etc.) - they handle initialization via deps.py
# Skip for 'reset' command - it manages its own database lifecycle
skip_init_commands = {
"doctor",
"mcp",
"status",
"sync",
"project",
"tool",
"reset",
"reindex",
"watch",
}
if (
not version
and ctx.invoked_subcommand is not None
and ctx.invoked_subcommand not in skip_init_commands
):
# Run initialization for every command unless --version was specified
if not version and ctx.invoked_subcommand is not None:
from basic_memory.config import app_config
from basic_memory.services.initialization import ensure_initialization
ensure_initialization(container.config)
ensure_initialization(app_config)
# Initialize MCP session with the specified project or default
if project: # pragma: no cover
# Use the project specified via --project flag
current_project_config = get_project_config(project)
session.set_current_project(current_project_config.name)
# Update the global config to use this project
from basic_memory.config import update_current_project
update_current_project(project)
else:
# Use the default project
current_project = app_config.default_project
session.set_current_project(current_project)
## import
# Register sub-command groups
import_app = typer.Typer(help="Import data from various sources")
app.add_typer(import_app, name="import")
claude_app = typer.Typer(help="Import Conversations from Claude JSON export.")
claude_app = typer.Typer()
import_app.add_typer(claude_app, name="claude")
## cloud
cloud_app = typer.Typer(help="Access Basic Memory Cloud")
app.add_typer(cloud_app, name="cloud")
-300
View File
@@ -1,300 +0,0 @@
"""WorkOS OAuth Device Authorization for CLI."""
import base64
import hashlib
import json
import os
import secrets
import time
import webbrowser
from contextlib import asynccontextmanager
from collections.abc import AsyncIterator, Callable
from typing import AsyncContextManager
import httpx
from rich.console import Console
from basic_memory.config import ConfigManager
console = Console()
class CLIAuth:
"""Handles WorkOS OAuth Device Authorization for CLI tools."""
def __init__(
self,
client_id: str,
authkit_domain: str,
http_client_factory: Callable[[], AsyncContextManager[httpx.AsyncClient]] | None = None,
):
self.client_id = client_id
self.authkit_domain = authkit_domain
app_config = ConfigManager().config
# Store tokens in data dir
self.token_file = app_config.data_dir_path / "basic-memory-cloud.json"
# PKCE parameters
self.code_verifier = None
self.code_challenge = None
self._http_client_factory = http_client_factory
@asynccontextmanager
async def _get_http_client(self) -> AsyncIterator[httpx.AsyncClient]:
"""Create an AsyncClient, optionally via injected factory.
Why: enables reliable tests without monkeypatching httpx internals while
still using real httpx request/response objects.
"""
if self._http_client_factory:
async with self._http_client_factory() as client:
yield client
else:
async with httpx.AsyncClient() as client:
yield client
def generate_pkce_pair(self) -> tuple[str, str]:
"""Generate PKCE code verifier and challenge."""
# Generate code verifier (43-128 characters)
code_verifier = base64.urlsafe_b64encode(secrets.token_bytes(32)).decode("utf-8")
code_verifier = code_verifier.rstrip("=")
# Generate code challenge (SHA256 hash of verifier)
challenge_bytes = hashlib.sha256(code_verifier.encode("utf-8")).digest()
code_challenge = base64.urlsafe_b64encode(challenge_bytes).decode("utf-8")
code_challenge = code_challenge.rstrip("=")
return code_verifier, code_challenge
async def request_device_authorization(self) -> dict | None:
"""Request device authorization from WorkOS with PKCE."""
device_auth_url = f"{self.authkit_domain}/oauth2/device_authorization"
# Generate PKCE pair
self.code_verifier, self.code_challenge = self.generate_pkce_pair()
data = {
"client_id": self.client_id,
"scope": "openid profile email offline_access",
"code_challenge": self.code_challenge,
"code_challenge_method": "S256",
}
try:
async with self._get_http_client() as client:
response = await client.post(device_auth_url, data=data)
if response.status_code == 200:
return response.json()
else:
console.print(
f"[red]Device authorization failed: {response.status_code} - {response.text}[/red]"
)
return None
except Exception as e:
console.print(f"[red]Device authorization error: {e}[/red]")
return None
def display_user_instructions(self, device_response: dict) -> None:
"""Display user instructions for device authorization."""
user_code = device_response["user_code"]
verification_uri = device_response["verification_uri"]
verification_uri_complete = device_response.get("verification_uri_complete")
console.print("\n[bold blue]Authentication Required[/bold blue]")
console.print("\nTo authenticate, please visit:")
console.print(f"[bold cyan]{verification_uri}[/bold cyan]")
console.print(f"\nAnd enter this code: [bold yellow]{user_code}[/bold yellow]")
if verification_uri_complete:
console.print("\nOr for one-click access, visit:")
console.print(f"[bold green]{verification_uri_complete}[/bold green]")
# Try to open browser automatically
try:
console.print("\n[dim]Opening browser automatically...[/dim]")
webbrowser.open(verification_uri_complete)
except Exception:
pass # Silently fail if browser can't be opened
console.print("\n[dim]Waiting for you to complete authentication in your browser...[/dim]")
async def poll_for_token(self, device_code: str, interval: int = 5) -> dict | None:
"""Poll the token endpoint until user completes authentication."""
token_url = f"{self.authkit_domain}/oauth2/token"
data = {
"client_id": self.client_id,
"device_code": device_code,
"grant_type": "urn:ietf:params:oauth:grant-type:device_code",
"code_verifier": self.code_verifier,
}
max_attempts = 60 # 5 minutes with 5-second intervals
current_interval = interval
for _attempt in range(max_attempts):
try:
async with self._get_http_client() as client:
response = await client.post(token_url, data=data)
if response.status_code == 200:
return response.json()
# Parse error response
try:
error_data = response.json()
error = error_data.get("error")
except Exception:
error = "unknown_error"
if error == "authorization_pending":
# User hasn't completed auth yet, keep polling
pass
elif error == "slow_down":
# Increase polling interval
current_interval += 5
console.print("[yellow]Slowing down polling rate...[/yellow]")
elif error == "access_denied":
console.print("[red]Authentication was denied by user[/red]")
return None
elif error == "expired_token":
console.print("[red]Device code has expired. Please try again.[/red]")
return None
else:
console.print(f"[red]Token polling error: {error}[/red]")
return None
except Exception as e:
console.print(f"[red]Token polling request error: {e}[/red]")
# Wait before next poll
await self._async_sleep(current_interval)
console.print("[red]Authentication timeout. Please try again.[/red]")
return None
async def _async_sleep(self, seconds: int) -> None:
"""Async sleep utility."""
import asyncio
await asyncio.sleep(seconds)
def save_tokens(self, tokens: dict) -> None:
"""Save tokens to project root as .bm-auth.json."""
token_data = {
"access_token": tokens["access_token"],
"refresh_token": tokens.get("refresh_token"),
"expires_at": int(time.time()) + tokens.get("expires_in", 3600),
"token_type": tokens.get("token_type", "Bearer"),
}
with open(self.token_file, "w") as f:
json.dump(token_data, f, indent=2)
# Secure the token file
os.chmod(self.token_file, 0o600)
console.print(f"[green]Tokens saved to {self.token_file}[/green]")
def load_tokens(self) -> dict | None:
"""Load tokens from .bm-auth.json file."""
if not self.token_file.exists():
return None
try:
with open(self.token_file) as f:
return json.load(f)
except (OSError, json.JSONDecodeError):
return None
def is_token_valid(self, tokens: dict) -> bool:
"""Check if stored token is still valid."""
expires_at = tokens.get("expires_at", 0)
# Add 60 second buffer for clock skew
return time.time() < (expires_at - 60)
async def refresh_token(self, refresh_token: str) -> dict | None:
"""Refresh access token using refresh token."""
token_url = f"{self.authkit_domain}/oauth2/token"
data = {
"client_id": self.client_id,
"grant_type": "refresh_token",
"refresh_token": refresh_token,
}
try:
async with self._get_http_client() as client:
response = await client.post(token_url, data=data)
if response.status_code == 200:
return response.json()
else:
console.print(
f"[red]Token refresh failed: {response.status_code} - {response.text}[/red]"
)
return None
except Exception as e:
console.print(f"[red]Token refresh error: {e}[/red]")
return None
async def get_valid_token(self) -> str | None:
"""Get valid access token, refresh if needed."""
tokens = self.load_tokens()
if not tokens:
return None
if self.is_token_valid(tokens):
return tokens["access_token"]
# Token expired - try to refresh if we have a refresh token
refresh_token = tokens.get("refresh_token")
if refresh_token:
console.print("[yellow]Access token expired, refreshing...[/yellow]")
new_tokens = await self.refresh_token(refresh_token)
if new_tokens:
# Save new tokens (may include rotated refresh token)
self.save_tokens(new_tokens)
console.print("[green]Token refreshed successfully[/green]")
return new_tokens["access_token"]
else:
console.print("[yellow]Token refresh failed. Please run 'login' again.[/yellow]")
return None
else:
console.print("[yellow]No refresh token available. Please run 'login' again.[/yellow]")
return None
async def login(self) -> bool:
"""Perform OAuth Device Authorization login flow."""
console.print("[blue]Initiating authentication...[/blue]")
# Step 1: Request device authorization
device_response = await self.request_device_authorization()
if not device_response:
return False
# Step 2: Display user instructions
self.display_user_instructions(device_response)
# Step 3: Poll for token
device_code = device_response["device_code"]
interval = device_response.get("interval", 5)
tokens = await self.poll_for_token(device_code, interval)
if not tokens:
return False
# Step 4: Save tokens
self.save_tokens(tokens)
console.print("\n[green]Successfully authenticated with Basic Memory Cloud![/green]")
return True
def logout(self) -> None:
"""Remove stored authentication tokens."""
if self.token_file.exists():
self.token_file.unlink()
console.print("[green]Logged out successfully[/green]")
else:
console.print("[yellow]No stored authentication found[/yellow]")
+4 -16
View File
@@ -1,21 +1,13 @@
"""CLI commands for basic-memory."""
from . import status, db, doctor, import_memory_json, mcp, import_claude_conversations
from . import (
import_claude_projects,
import_chatgpt,
tool,
project,
format,
schema,
watch,
workspace,
)
from . import auth, status, sync, db, import_memory_json, mcp, import_claude_conversations
from . import import_claude_projects, import_chatgpt, tool, project
__all__ = [
"auth",
"status",
"sync",
"db",
"doctor",
"import_memory_json",
"mcp",
"import_claude_conversations",
@@ -23,8 +15,4 @@ __all__ = [
"import_chatgpt",
"tool",
"project",
"format",
"schema",
"watch",
"workspace",
]
+136
View File
@@ -0,0 +1,136 @@
"""OAuth management commands."""
import typer
from typing import Optional
from pydantic import AnyHttpUrl
from basic_memory.cli.app import app
from basic_memory.mcp.auth_provider import BasicMemoryOAuthProvider
from mcp.shared.auth import OAuthClientInformationFull
auth_app = typer.Typer(help="OAuth client management commands")
app.add_typer(auth_app, name="auth")
@auth_app.command()
def register_client(
client_id: Optional[str] = typer.Option(
None, help="Client ID (auto-generated if not provided)"
),
client_secret: Optional[str] = typer.Option(
None, help="Client secret (auto-generated if not provided)"
),
issuer_url: str = typer.Option("http://localhost:8000", help="OAuth issuer URL"),
):
"""Register a new OAuth client for Basic Memory MCP server."""
# Create provider instance
provider = BasicMemoryOAuthProvider(issuer_url=issuer_url)
# Create client info with required redirect_uris
client_info = OAuthClientInformationFull(
client_id=client_id or "", # Provider will generate if empty
client_secret=client_secret or "", # Provider will generate if empty
redirect_uris=[AnyHttpUrl("http://localhost:8000/callback")], # Default redirect URI
client_name="Basic Memory OAuth Client",
grant_types=["authorization_code", "refresh_token"],
)
# Register the client
import asyncio
asyncio.run(provider.register_client(client_info))
typer.echo("Client registered successfully!")
typer.echo(f"Client ID: {client_info.client_id}")
typer.echo(f"Client Secret: {client_info.client_secret}")
typer.echo("\nSave these credentials securely - the client secret cannot be retrieved later.")
@auth_app.command()
def test_auth(
issuer_url: str = typer.Option("http://localhost:8000", help="OAuth issuer URL"),
):
"""Test OAuth authentication flow.
IMPORTANT: Use the same FASTMCP_AUTH_SECRET_KEY environment variable
as your MCP server for tokens to validate correctly.
"""
import asyncio
import secrets
from mcp.server.auth.provider import AuthorizationParams
from pydantic import AnyHttpUrl
async def test_flow():
# Create provider with same secret key as server
provider = BasicMemoryOAuthProvider(issuer_url=issuer_url)
# Register a test client
client_info = OAuthClientInformationFull(
client_id=secrets.token_urlsafe(16),
client_secret=secrets.token_urlsafe(32),
redirect_uris=[AnyHttpUrl("http://localhost:8000/callback")],
client_name="Test OAuth Client",
grant_types=["authorization_code", "refresh_token"],
)
await provider.register_client(client_info)
typer.echo(f"Registered test client: {client_info.client_id}")
# Get the client
client = await provider.get_client(client_info.client_id)
if not client:
typer.echo("Error: Client not found after registration", err=True)
return
# Create authorization request
auth_params = AuthorizationParams(
state="test-state",
scopes=["read", "write"],
code_challenge="test-challenge",
redirect_uri=AnyHttpUrl("http://localhost:8000/callback"),
redirect_uri_provided_explicitly=True,
)
# Get authorization URL
auth_url = await provider.authorize(client, auth_params)
typer.echo(f"Authorization URL: {auth_url}")
# Extract auth code from URL
from urllib.parse import urlparse, parse_qs
parsed = urlparse(auth_url)
params = parse_qs(parsed.query)
auth_code = params.get("code", [None])[0]
if not auth_code:
typer.echo("Error: No authorization code in URL", err=True)
return
# Load the authorization code
code_obj = await provider.load_authorization_code(client, auth_code)
if not code_obj:
typer.echo("Error: Invalid authorization code", err=True)
return
# Exchange for tokens
token = await provider.exchange_authorization_code(client, code_obj)
typer.echo(f"Access token: {token.access_token}")
typer.echo(f"Refresh token: {token.refresh_token}")
typer.echo(f"Expires in: {token.expires_in} seconds")
# Validate access token
access_token_obj = await provider.load_access_token(token.access_token)
if access_token_obj:
typer.echo("Access token validated successfully!")
typer.echo(f"Client ID: {access_token_obj.client_id}")
typer.echo(f"Scopes: {access_token_obj.scopes}")
else:
typer.echo("Error: Invalid access token", err=True)
asyncio.run(test_flow())
if __name__ == "__main__":
auth_app()
@@ -1,16 +0,0 @@
"""Cloud commands package."""
from basic_memory.cli.app import cloud_app
# Import all commands to register them with typer
from basic_memory.cli.commands.cloud.core_commands import * # noqa: F401,F403
from basic_memory.cli.commands.cloud.api_client import get_authenticated_headers, get_cloud_config # noqa: F401
from basic_memory.cli.commands.cloud.upload_command import * # noqa: F401,F403
# Register snapshot sub-command group
from basic_memory.cli.commands.cloud.snapshot import snapshot_app
cloud_app.add_typer(snapshot_app, name="snapshot")
# Register restore command (directly on cloud_app via decorator)
from basic_memory.cli.commands.cloud.restore import restore # noqa: F401, E402
@@ -1,127 +0,0 @@
"""Cloud API client utilities."""
from collections.abc import AsyncIterator
from typing import Optional
from contextlib import asynccontextmanager
from typing import AsyncContextManager, Callable
import httpx
import typer
from rich.console import Console
from basic_memory.cli.auth import CLIAuth
from basic_memory.config import ConfigManager
console = Console()
HttpClientFactory = Callable[[], AsyncContextManager[httpx.AsyncClient]]
class CloudAPIError(Exception):
"""Exception raised for cloud API errors."""
def __init__(
self, message: str, status_code: Optional[int] = None, detail: Optional[dict] = None
):
super().__init__(message)
self.status_code = status_code
self.detail = detail or {}
class SubscriptionRequiredError(CloudAPIError):
"""Exception raised when user needs an active subscription."""
def __init__(self, message: str, subscribe_url: str):
super().__init__(message, status_code=403, detail={"error": "subscription_required"})
self.subscribe_url = subscribe_url
def get_cloud_config() -> tuple[str, str, str]:
"""Get cloud OAuth configuration from config."""
config_manager = ConfigManager()
config = config_manager.config
return config.cloud_client_id, config.cloud_domain, config.cloud_host
async def get_authenticated_headers(auth: CLIAuth | None = None) -> dict[str, str]:
"""
Get authentication headers with JWT token.
handles jwt refresh if needed.
"""
client_id, domain, _ = get_cloud_config()
auth_obj = auth or CLIAuth(client_id=client_id, authkit_domain=domain)
token = await auth_obj.get_valid_token()
if not token:
console.print("[red]Not authenticated. Please run 'basic-memory cloud login' first.[/red]")
raise typer.Exit(1)
return {"Authorization": f"Bearer {token}"}
@asynccontextmanager
async def _default_http_client(timeout: float) -> AsyncIterator[httpx.AsyncClient]:
async with httpx.AsyncClient(timeout=timeout) as client:
yield client
async def make_api_request(
method: str,
url: str,
headers: Optional[dict] = None,
json_data: Optional[dict] = None,
timeout: float = 30.0,
*,
auth: CLIAuth | None = None,
http_client_factory: HttpClientFactory | None = None,
) -> httpx.Response:
"""Make an API request to the cloud service."""
headers = headers or {}
auth_headers = await get_authenticated_headers(auth=auth)
headers.update(auth_headers)
# Add debug headers to help with compression issues
headers.setdefault("Accept-Encoding", "identity") # Disable compression for debugging
client_factory = http_client_factory or (lambda: _default_http_client(timeout))
async with client_factory() as client:
try:
response = await client.request(method=method, url=url, headers=headers, json=json_data)
response.raise_for_status()
return response
except httpx.HTTPError as e:
# Check if this is a response error with response details
if hasattr(e, "response") and e.response is not None: # pyright: ignore [reportAttributeAccessIssue]
response = e.response # type: ignore
# Try to parse error detail from response
error_detail = None
try:
error_detail = response.json()
except Exception:
# If JSON parsing fails, we'll handle it as a generic error
pass
# Check for subscription_required error (403)
if response.status_code == 403 and isinstance(error_detail, dict):
# Handle both FastAPI HTTPException format (nested under "detail")
# and direct format
detail_obj = error_detail.get("detail", error_detail)
if (
isinstance(detail_obj, dict)
and detail_obj.get("error") == "subscription_required"
):
message = detail_obj.get("message", "Active subscription required")
subscribe_url = detail_obj.get(
"subscribe_url", "https://basicmemory.com/subscribe"
)
raise SubscriptionRequiredError(
message=message, subscribe_url=subscribe_url
) from e
# Raise generic CloudAPIError with status code and detail
raise CloudAPIError(
f"API request failed: {e}",
status_code=response.status_code,
detail=error_detail if isinstance(error_detail, dict) else {},
) from e
raise CloudAPIError(f"API request failed: {e}") from e
@@ -1,110 +0,0 @@
"""Cloud bisync utility functions for Basic Memory CLI."""
from pathlib import Path
from basic_memory.cli.commands.cloud.api_client import make_api_request
from basic_memory.config import ConfigManager
from basic_memory.ignore_utils import create_default_bmignore, get_bmignore_path
from basic_memory.schemas.cloud import MountCredentials, TenantMountInfo
class BisyncError(Exception):
"""Exception raised for bisync-related errors."""
pass
async def get_mount_info() -> TenantMountInfo:
"""Get current tenant information from cloud API."""
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
response = await make_api_request(method="GET", url=f"{host_url}/tenant/mount/info")
return TenantMountInfo.model_validate(response.json())
except Exception as e:
raise BisyncError(f"Failed to get tenant info: {e}") from e
async def generate_mount_credentials(tenant_id: str) -> MountCredentials:
"""Generate scoped credentials for syncing."""
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
response = await make_api_request(method="POST", url=f"{host_url}/tenant/mount/credentials")
return MountCredentials.model_validate(response.json())
except Exception as e:
raise BisyncError(f"Failed to generate credentials: {e}") from e
def convert_bmignore_to_rclone_filters() -> Path:
"""Convert .bmignore patterns to rclone filter format.
Reads ~/.basic-memory/.bmignore (gitignore-style) and converts to
~/.basic-memory/.bmignore.rclone (rclone filter format).
Only regenerates if .bmignore has been modified since last conversion.
Returns:
Path to converted rclone filter file
"""
# Ensure .bmignore exists
create_default_bmignore()
bmignore_path = get_bmignore_path()
# Create rclone filter path: ~/.basic-memory/.bmignore -> ~/.basic-memory/.bmignore.rclone
rclone_filter_path = bmignore_path.parent / f"{bmignore_path.name}.rclone"
# Skip regeneration if rclone file is newer than bmignore
if rclone_filter_path.exists():
bmignore_mtime = bmignore_path.stat().st_mtime
rclone_mtime = rclone_filter_path.stat().st_mtime
if rclone_mtime >= bmignore_mtime:
return rclone_filter_path
# Read .bmignore patterns
patterns = []
try:
with bmignore_path.open("r", encoding="utf-8") as f:
for line in f:
line = line.strip()
# Keep comments and empty lines
if not line or line.startswith("#"):
patterns.append(line)
continue
# Convert gitignore pattern to rclone filter syntax
# gitignore: node_modules → rclone: - node_modules/**
# gitignore: *.pyc → rclone: - *.pyc
if "*" in line:
# Pattern already has wildcard, just add exclude prefix
patterns.append(f"- {line}")
else:
# Directory pattern - add /** for recursive exclude
patterns.append(f"- {line}/**")
except Exception:
# If we can't read the file, create a minimal filter
patterns = ["# Error reading .bmignore, using minimal filters", "- .git/**"]
# Write rclone filter file
rclone_filter_path.write_text("\n".join(patterns) + "\n")
return rclone_filter_path
def get_bisync_filter_path() -> Path:
"""Get path to bisync filter file.
Uses ~/.basic-memory/.bmignore (converted to rclone format).
The file is automatically created with default patterns on first use.
Returns:
Path to rclone filter file
"""
return convert_bmignore_to_rclone_filters()
@@ -1,108 +0,0 @@
"""Shared utilities for cloud operations."""
from basic_memory.cli.commands.cloud.api_client import make_api_request
from basic_memory.config import ConfigManager
from basic_memory.schemas.cloud import (
CloudProjectList,
CloudProjectCreateRequest,
CloudProjectCreateResponse,
)
from basic_memory.utils import generate_permalink
class CloudUtilsError(Exception):
"""Exception raised for cloud utility errors."""
pass
async def fetch_cloud_projects(
*,
api_request=make_api_request,
) -> CloudProjectList:
"""Fetch list of projects from cloud API.
Returns:
CloudProjectList with projects from cloud
"""
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
response = await api_request(method="GET", url=f"{host_url}/proxy/v2/projects/")
return CloudProjectList.model_validate(response.json())
except Exception as e:
raise CloudUtilsError(f"Failed to fetch cloud projects: {e}") from e
async def create_cloud_project(
project_name: str,
*,
api_request=make_api_request,
) -> CloudProjectCreateResponse:
"""Create a new project on cloud.
Args:
project_name: Name of project to create
Returns:
CloudProjectCreateResponse with project details from API
"""
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
# Use generate_permalink to ensure consistent naming
project_path = generate_permalink(project_name)
project_data = CloudProjectCreateRequest(
name=project_name,
path=project_path,
set_default=False,
)
response = await api_request(
method="POST",
url=f"{host_url}/proxy/v2/projects/",
headers={"Content-Type": "application/json"},
json_data=project_data.model_dump(),
)
return CloudProjectCreateResponse.model_validate(response.json())
except Exception as e:
raise CloudUtilsError(f"Failed to create cloud project '{project_name}': {e}") from e
async def sync_project(project_name: str, force_full: bool = False) -> None:
"""Trigger sync for a specific project on cloud.
Args:
project_name: Name of project to sync
force_full: If True, force a full scan bypassing watermark optimization
"""
try:
from basic_memory.cli.commands.command_utils import run_sync
await run_sync(project=project_name, force_full=force_full)
except Exception as e:
raise CloudUtilsError(f"Failed to sync project '{project_name}': {e}") from e
async def project_exists(project_name: str, *, api_request=make_api_request) -> bool:
"""Check if a project exists on cloud.
Args:
project_name: Name of project to check
Returns:
True if project exists, False otherwise
"""
try:
projects = await fetch_cloud_projects(api_request=api_request)
project_names = {p.name for p in projects.projects}
return project_name in project_names
except Exception:
return False
@@ -1,284 +0,0 @@
"""Core cloud commands for Basic Memory CLI."""
import typer
from rich.console import Console
from basic_memory.cli.app import cloud_app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.cli.auth import CLIAuth
from basic_memory.cli.promo import OSS_DISCOUNT_CODE
from basic_memory.config import ConfigManager
from basic_memory.cli.commands.cloud.api_client import (
CloudAPIError,
SubscriptionRequiredError,
get_cloud_config,
make_api_request,
)
from basic_memory.cli.commands.cloud.bisync_commands import (
BisyncError,
generate_mount_credentials,
get_mount_info,
)
from basic_memory.cli.commands.cloud.rclone_config import configure_rclone_remote
from basic_memory.cli.commands.cloud.rclone_installer import (
RcloneInstallError,
install_rclone,
)
console = Console()
@cloud_app.command()
def login():
"""Authenticate with WorkOS using OAuth Device Authorization flow."""
async def _login():
client_id, domain, host_url = get_cloud_config()
auth = CLIAuth(client_id=client_id, authkit_domain=domain)
try:
success = await auth.login()
if not success:
console.print("[red]Login failed[/red]")
raise typer.Exit(1)
# Test subscription access by calling a protected endpoint
console.print("[dim]Verifying subscription access...[/dim]")
await make_api_request("GET", f"{host_url.rstrip('/')}/proxy/health")
console.print("[green]Cloud authentication successful[/green]")
console.print(f"[dim]Cloud host ready: {host_url}[/dim]")
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(
f"OSS discount code: [bold]{OSS_DISCOUNT_CODE}[/bold] (20% off for 3 months)\n"
)
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
console.print(
"[dim]Once you have an active subscription, run [bold]bm cloud login[/bold] again.[/dim]"
)
raise typer.Exit(1)
run_with_cleanup(_login())
@cloud_app.command()
def logout():
"""Remove stored OAuth tokens."""
config = ConfigManager().config
auth = CLIAuth(client_id=config.cloud_client_id, authkit_domain=config.cloud_domain)
auth.logout()
console.print("[dim]API key (if configured) remains available for cloud project routing.[/dim]")
@cloud_app.command("status")
def status() -> None:
"""Check cloud authentication state and cloud instance health."""
config_manager = ConfigManager()
config = config_manager.load_config()
auth = CLIAuth(client_id=config.cloud_client_id, authkit_domain=config.cloud_domain)
tokens = auth.load_tokens()
console.print("[bold blue]Cloud Authentication Status[/bold blue]")
console.print(f" Host: {config.cloud_host}")
console.print(
f" API Key: {'[green]configured[/green]' if config.cloud_api_key else '[yellow]not set[/yellow]'}"
)
oauth_status = "[yellow]not logged in[/yellow]"
if tokens:
oauth_status = (
"[green]token valid[/green]"
if auth.is_token_valid(tokens)
else "[yellow]token expired[/yellow]"
)
console.print(f" OAuth: {oauth_status}")
# Get cloud configuration
_, _, host_url = get_cloud_config()
host_url = host_url.rstrip("/")
has_credentials = bool(config.cloud_api_key) or tokens is not None
if not has_credentials:
console.print(
"\n[dim]No cloud credentials found. Run: bm cloud login or bm cloud set-key <key>[/dim]"
)
return
try:
console.print("\n[blue]Checking cloud instance health...[/blue]")
# Make API request to check health
response = run_with_cleanup(make_api_request(method="GET", url=f"{host_url}/proxy/health"))
health_data = response.json()
console.print("[green]Cloud instance is healthy[/green]")
# Display status details
if "status" in health_data:
console.print(f" Status: {health_data['status']}")
if "version" in health_data:
console.print(f" Version: {health_data['version']}")
if "timestamp" in health_data:
console.print(f" Timestamp: {health_data['timestamp']}")
console.print("\n[dim]To sync projects, use: bm project bisync --name <project>[/dim]")
except CloudAPIError as e:
console.print(f"[yellow]Cloud health check failed: {e}[/yellow]")
console.print(
"[dim]Try re-authenticating with 'bm cloud login' or setting API key with 'bm cloud set-key'.[/dim]"
)
except Exception as e:
console.print(f"[yellow]Unexpected health check error: {e}[/yellow]")
@cloud_app.command("setup")
def setup() -> None:
"""Set up cloud sync by installing rclone and configuring credentials.
After setup, use project commands for syncing:
bm project add <name> <path> --local-path ~/projects/<name>
bm project bisync --name <name> --resync # First time
bm project bisync --name <name> # Subsequent syncs
"""
console.print("[bold blue]Basic Memory Cloud Setup[/bold blue]")
console.print("Setting up cloud sync with rclone...\n")
try:
# Step 1: Install rclone
console.print("[blue]Step 1: Installing rclone...[/blue]")
install_rclone()
# Step 2: Get tenant info
console.print("\n[blue]Step 2: Getting tenant information...[/blue]")
tenant_info = run_with_cleanup(get_mount_info())
console.print(f"[green]Found tenant: {tenant_info.tenant_id}[/green]")
# Step 3: Generate credentials
console.print("\n[blue]Step 3: Generating sync credentials...[/blue]")
creds = run_with_cleanup(generate_mount_credentials(tenant_info.tenant_id))
console.print("[green]Generated secure credentials[/green]")
# Step 4: Configure rclone remote
console.print("\n[blue]Step 4: Configuring rclone remote...[/blue]")
configure_rclone_remote(
access_key=creds.access_key,
secret_key=creds.secret_key,
)
console.print("\n[bold green]Cloud setup completed successfully![/bold green]")
console.print("\n[bold]Next steps:[/bold]")
console.print("1. Add a project with local sync path:")
console.print(" bm project add research --local-path ~/Documents/research")
console.print("\n Or configure sync for an existing project:")
console.print(" bm project sync-setup research ~/Documents/research")
console.print("\n2. Preview the initial sync (recommended):")
console.print(" bm project bisync --name research --resync --dry-run")
console.print("\n3. If all looks good, run the actual sync:")
console.print(" bm project bisync --name research --resync")
console.print("\n4. Subsequent syncs (no --resync needed):")
console.print(" bm project bisync --name research")
console.print(
"\n[dim]Tip: Always use --dry-run first to preview changes before syncing[/dim]"
)
except (RcloneInstallError, BisyncError, CloudAPIError) as e:
console.print(f"\n[red]Setup failed: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"\n[red]Unexpected error during setup: {e}[/red]")
raise typer.Exit(1)
@cloud_app.command("promo")
def promo(enabled: bool = typer.Option(True, "--on/--off", help="Enable or disable CLI promos.")):
"""Enable or disable CLI cloud promo messages."""
config_manager = ConfigManager()
config = config_manager.load_config()
config.cloud_promo_opt_out = not enabled
config_manager.save_config(config)
if enabled:
console.print("[green]Cloud promo messages enabled[/green]")
else:
console.print("[yellow]Cloud promo messages disabled[/yellow]")
@cloud_app.command("set-key")
def set_key(
api_key: str = typer.Argument(..., help="API key (bmc_ prefixed) for cloud access"),
) -> None:
"""Save a cloud API key for per-project cloud routing.
The API key is account-level and used by projects set to cloud mode.
Create a key in the web app or use 'bm cloud create-key'.
Example:
bm cloud set-key bmc_abc123...
"""
if not api_key.startswith("bmc_"):
console.print("[red]Error: API key must start with 'bmc_'[/red]")
raise typer.Exit(1)
config_manager = ConfigManager()
config = config_manager.load_config()
config.cloud_api_key = api_key
config_manager.save_config(config)
console.print("[green]API key saved[/green]")
console.print("[dim]Projects set to cloud mode will use this key for authentication[/dim]")
console.print("[dim]Set a project to cloud mode: bm project set-cloud <name>[/dim]")
@cloud_app.command("create-key")
def create_key(
name: str = typer.Argument(..., help="Human-readable name for the API key"),
) -> None:
"""Create a new cloud API key and save it locally.
Requires active OAuth session (run 'bm cloud login' first).
The key is created via the cloud API and saved to local config.
Example:
bm cloud create-key "my-laptop"
"""
async def _create_key():
_, _, host_url = get_cloud_config()
host_url = host_url.rstrip("/")
console.print(f"[dim]Creating API key '{name}'...[/dim]")
response = await make_api_request(
method="POST",
url=f"{host_url}/api/keys",
json_data={"name": name},
)
key_data = response.json()
api_key = key_data.get("key")
if not api_key:
console.print("[red]Error: No key returned from API[/red]")
raise typer.Exit(1)
# Save to config
config_manager = ConfigManager()
config = config_manager.load_config()
config.cloud_api_key = api_key
config_manager.save_config(config)
console.print(f"[green]API key '{name}' created and saved[/green]")
console.print("[dim]Projects set to cloud mode will use this key for authentication[/dim]")
console.print("[dim]Set a project to cloud mode: bm project set-cloud <name>[/dim]")
try:
run_with_cleanup(_create_key())
except CloudAPIError as e:
console.print(f"[red]Error creating API key: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
@@ -1,411 +0,0 @@
"""Project-scoped rclone sync commands for Basic Memory Cloud.
This module provides simplified, project-scoped rclone operations:
- Each project syncs independently
- Uses single "basic-memory-cloud" remote (not tenant-specific)
- Balanced defaults from SPEC-8 Phase 4 testing
- Per-project bisync state tracking
Replaces tenant-wide sync with project-scoped workflows.
"""
import re
import subprocess
from dataclasses import dataclass
from functools import lru_cache
from pathlib import Path
from typing import Callable, Optional, Protocol
from loguru import logger
from rich.console import Console
from basic_memory.cli.commands.cloud.rclone_installer import is_rclone_installed
from basic_memory.utils import normalize_project_path
console = Console()
# Minimum rclone version for --create-empty-src-dirs support
MIN_RCLONE_VERSION_EMPTY_DIRS = (1, 64, 0)
# Tigris edge caching returns stale data for users outside the origin region (iad).
# --header is rclone's global flag that applies to ALL HTTP transactions (list, download,
# upload). This is critical because bisync starts with S3 ListObjectsV2, which is neither
# a download nor upload — so --header-download/--header-upload would miss list requests.
# See: https://www.tigrisdata.com/docs/objects/consistency/
TIGRIS_CONSISTENCY_HEADERS = [
"--header",
"X-Tigris-Consistent: true",
]
class RunResult(Protocol):
returncode: int
stdout: str
RunFunc = Callable[..., RunResult]
IsInstalledFunc = Callable[[], bool]
class RcloneError(Exception):
"""Exception raised for rclone command errors."""
pass
def check_rclone_installed(is_installed: IsInstalledFunc = is_rclone_installed) -> None:
"""Check if rclone is installed and raise helpful error if not.
Raises:
RcloneError: If rclone is not installed with installation instructions
"""
if not is_installed():
raise RcloneError(
"rclone is not installed.\n\n"
"Install rclone by running: bm cloud setup\n"
"Or install manually from: https://rclone.org/downloads/\n\n"
"Windows users: Ensure you have a package manager installed (winget, chocolatey, or scoop)"
)
@lru_cache(maxsize=1)
def get_rclone_version(run: RunFunc = subprocess.run) -> tuple[int, int, int] | None:
"""Get rclone version as (major, minor, patch) tuple.
Returns:
Version tuple like (1, 64, 2), or None if version cannot be determined.
Note:
Result is cached since rclone version won't change during runtime.
"""
try:
result = run(["rclone", "version"], capture_output=True, text=True, timeout=10)
# Parse "rclone v1.64.2" or "rclone v1.60.1-DEV"
match = re.search(r"v(\d+)\.(\d+)\.(\d+)", result.stdout)
if match:
version = (int(match.group(1)), int(match.group(2)), int(match.group(3)))
logger.debug(f"Detected rclone version: {version}")
return version
except Exception as e:
logger.warning(f"Could not determine rclone version: {e}")
return None
def supports_create_empty_src_dirs(version: tuple[int, int, int] | None) -> bool:
"""Check if installed rclone supports --create-empty-src-dirs flag.
Returns:
True if rclone version >= 1.64.0, False otherwise.
"""
if version is None:
# If we can't determine version, assume older and skip the flag
return False
return version >= MIN_RCLONE_VERSION_EMPTY_DIRS
@dataclass
class SyncProject:
"""Project configured for cloud sync.
Attributes:
name: Project name
path: Cloud path (e.g., "app/data/research")
local_sync_path: Local directory for syncing (optional)
"""
name: str
path: str
local_sync_path: Optional[str] = None
def get_bmignore_filter_path() -> Path:
"""Get path to rclone filter file.
Uses ~/.basic-memory/.bmignore converted to rclone format.
File is automatically created with default patterns on first use.
Returns:
Path to rclone filter file
"""
# Import here to avoid circular dependency
from basic_memory.cli.commands.cloud.bisync_commands import (
convert_bmignore_to_rclone_filters,
)
return convert_bmignore_to_rclone_filters()
def get_project_bisync_state(project_name: str) -> Path:
"""Get path to project's bisync state directory.
Args:
project_name: Name of the project
Returns:
Path to bisync state directory for this project
"""
return Path.home() / ".basic-memory" / "bisync-state" / project_name
def bisync_initialized(project_name: str) -> bool:
"""Check if bisync has been initialized for this project.
Args:
project_name: Name of the project
Returns:
True if bisync state exists, False otherwise
"""
state_path = get_project_bisync_state(project_name)
return state_path.exists() and any(state_path.iterdir())
def get_project_remote(project: SyncProject, bucket_name: str) -> str:
"""Build rclone remote path for project.
Args:
project: Project with cloud path
bucket_name: S3 bucket name
Returns:
Remote path like "basic-memory-cloud:bucket-name/basic-memory-llc"
Note:
The API returns paths like "/app/data/basic-memory-llc" because the S3 bucket
is mounted at /app/data on the fly machine. We need to strip the /app/data/
prefix to get the actual S3 path within the bucket.
"""
# Normalize path to strip /app/data/ mount point prefix
cloud_path = normalize_project_path(project.path).lstrip("/")
return f"basic-memory-cloud:{bucket_name}/{cloud_path}"
def project_sync(
project: SyncProject,
bucket_name: str,
dry_run: bool = False,
verbose: bool = False,
*,
run: RunFunc = subprocess.run,
is_installed: IsInstalledFunc = is_rclone_installed,
filter_path: Path | None = None,
) -> bool:
"""One-way sync: local → cloud.
Makes cloud identical to local using rclone sync.
Args:
project: Project to sync
bucket_name: S3 bucket name
dry_run: Preview changes without applying
verbose: Show detailed output
Returns:
True if sync succeeded, False otherwise
Raises:
RcloneError: If project has no local_sync_path configured or rclone not installed
"""
check_rclone_installed(is_installed=is_installed)
if not project.local_sync_path:
raise RcloneError(f"Project {project.name} has no local_sync_path configured")
local_path = Path(project.local_sync_path).expanduser()
remote_path = get_project_remote(project, bucket_name)
filter_path = filter_path or get_bmignore_filter_path()
cmd = [
"rclone",
"sync",
str(local_path),
remote_path,
*TIGRIS_CONSISTENCY_HEADERS,
"--filter-from",
str(filter_path),
]
if verbose:
cmd.append("--verbose")
else:
cmd.append("--progress")
if dry_run:
cmd.append("--dry-run")
result = run(cmd, text=True)
return result.returncode == 0
def project_bisync(
project: SyncProject,
bucket_name: str,
dry_run: bool = False,
resync: bool = False,
verbose: bool = False,
*,
run: RunFunc = subprocess.run,
is_installed: IsInstalledFunc = is_rclone_installed,
version: tuple[int, int, int] | None = None,
filter_path: Path | None = None,
state_path: Path | None = None,
is_initialized: Callable[[str], bool] = bisync_initialized,
) -> bool:
"""Two-way sync: local ↔ cloud.
Uses rclone bisync with balanced defaults:
- conflict_resolve: newer (auto-resolve to most recent)
- max_delete: 25 (safety limit)
- compare: modtime (ignore size differences from line ending conversions)
- check_access: false (skip for performance)
Args:
project: Project to sync
bucket_name: S3 bucket name
dry_run: Preview changes without applying
resync: Force resync to establish new baseline
verbose: Show detailed output
Returns:
True if bisync succeeded, False otherwise
Raises:
RcloneError: If project has no local_sync_path, needs --resync, or rclone not installed
"""
check_rclone_installed(is_installed=is_installed)
if not project.local_sync_path:
raise RcloneError(f"Project {project.name} has no local_sync_path configured")
local_path = Path(project.local_sync_path).expanduser()
remote_path = get_project_remote(project, bucket_name)
filter_path = filter_path or get_bmignore_filter_path()
state_path = state_path or get_project_bisync_state(project.name)
# Ensure state directory exists
state_path.mkdir(parents=True, exist_ok=True)
cmd = [
"rclone",
"bisync",
str(local_path),
remote_path,
*TIGRIS_CONSISTENCY_HEADERS,
"--resilient",
"--conflict-resolve=newer",
"--max-delete=25",
"--compare=modtime", # Ignore size differences from line ending conversions
"--filter-from",
str(filter_path),
"--workdir",
str(state_path),
]
# Add --create-empty-src-dirs if rclone version supports it (v1.64+)
version = version if version is not None else get_rclone_version(run=run)
if supports_create_empty_src_dirs(version):
cmd.append("--create-empty-src-dirs")
if verbose:
cmd.append("--verbose")
else:
cmd.append("--progress")
if dry_run:
cmd.append("--dry-run")
if resync:
cmd.append("--resync")
# Check if first run requires resync
if not resync and not is_initialized(project.name) and not dry_run:
raise RcloneError(
f"First bisync for {project.name} requires --resync to establish baseline.\n"
f"Run: bm project bisync --name {project.name} --resync"
)
result = run(cmd, text=True)
return result.returncode == 0
def project_check(
project: SyncProject,
bucket_name: str,
one_way: bool = False,
*,
run: RunFunc = subprocess.run,
is_installed: IsInstalledFunc = is_rclone_installed,
filter_path: Path | None = None,
) -> bool:
"""Check integrity between local and cloud.
Verifies files match without transferring data.
Args:
project: Project to check
bucket_name: S3 bucket name
one_way: Only check for missing files on destination (faster)
Returns:
True if files match, False if differences found
Raises:
RcloneError: If project has no local_sync_path configured or rclone not installed
"""
check_rclone_installed(is_installed=is_installed)
if not project.local_sync_path:
raise RcloneError(f"Project {project.name} has no local_sync_path configured")
local_path = Path(project.local_sync_path).expanduser()
remote_path = get_project_remote(project, bucket_name)
filter_path = filter_path or get_bmignore_filter_path()
cmd = [
"rclone",
"check",
str(local_path),
remote_path,
*TIGRIS_CONSISTENCY_HEADERS,
"--filter-from",
str(filter_path),
]
if one_way:
cmd.append("--one-way")
result = run(cmd, capture_output=True, text=True)
return result.returncode == 0
def project_ls(
project: SyncProject,
bucket_name: str,
path: Optional[str] = None,
*,
run: RunFunc = subprocess.run,
is_installed: IsInstalledFunc = is_rclone_installed,
) -> list[str]:
"""List files in remote project.
Args:
project: Project to list files from
bucket_name: S3 bucket name
path: Optional subdirectory within project
Returns:
List of file paths
Raises:
subprocess.CalledProcessError: If rclone command fails
RcloneError: If rclone is not installed
"""
check_rclone_installed(is_installed=is_installed)
remote_path = get_project_remote(project, bucket_name)
if path:
remote_path = f"{remote_path}/{path}"
cmd = ["rclone", "ls", *TIGRIS_CONSISTENCY_HEADERS, remote_path]
result = run(cmd, capture_output=True, text=True, check=True)
return result.stdout.splitlines()
@@ -1,110 +0,0 @@
"""rclone configuration management for Basic Memory Cloud.
This module provides simplified rclone configuration for SPEC-20.
Uses a single "basic-memory-cloud" remote for all operations.
"""
import configparser
import os
import shutil
from pathlib import Path
from typing import Optional
from rich.console import Console
console = Console()
class RcloneConfigError(Exception):
"""Exception raised for rclone configuration errors."""
pass
def get_rclone_config_path() -> Path:
"""Get the path to rclone configuration file."""
config_dir = Path.home() / ".config" / "rclone"
config_dir.mkdir(parents=True, exist_ok=True)
return config_dir / "rclone.conf"
def backup_rclone_config() -> Optional[Path]:
"""Create a backup of existing rclone config."""
config_path = get_rclone_config_path()
if not config_path.exists():
return None
backup_path = config_path.with_suffix(f".conf.backup-{os.getpid()}")
shutil.copy2(config_path, backup_path)
console.print(f"[dim]Created backup: {backup_path}[/dim]")
return backup_path
def load_rclone_config() -> configparser.ConfigParser:
"""Load existing rclone configuration."""
config = configparser.ConfigParser()
config_path = get_rclone_config_path()
if config_path.exists():
config.read(config_path)
return config
def save_rclone_config(config: configparser.ConfigParser) -> None:
"""Save rclone configuration to file."""
config_path = get_rclone_config_path()
with open(config_path, "w") as f:
config.write(f)
console.print(f"[dim]Updated rclone config: {config_path}[/dim]")
def configure_rclone_remote(
access_key: str,
secret_key: str,
endpoint: str = "https://fly.storage.tigris.dev",
region: str = "auto",
) -> str:
"""Configure single rclone remote named 'basic-memory-cloud'.
This is the simplified approach from SPEC-20 that uses one remote
for all Basic Memory cloud operations (not tenant-specific).
Args:
access_key: S3 access key ID
secret_key: S3 secret access key
endpoint: S3-compatible endpoint URL
region: S3 region (default: auto)
Returns:
The remote name: "basic-memory-cloud"
"""
# Backup existing config
backup_rclone_config()
# Load existing config
config = load_rclone_config()
# Single remote name (not tenant-specific)
REMOTE_NAME = "basic-memory-cloud"
# Add/update the remote section
if not config.has_section(REMOTE_NAME):
config.add_section(REMOTE_NAME)
config.set(REMOTE_NAME, "type", "s3")
config.set(REMOTE_NAME, "provider", "Other")
config.set(REMOTE_NAME, "access_key_id", access_key)
config.set(REMOTE_NAME, "secret_access_key", secret_key)
config.set(REMOTE_NAME, "endpoint", endpoint)
config.set(REMOTE_NAME, "region", region)
# Prevent unnecessary encoding of filenames (only encode slashes and invalid UTF-8)
# This prevents files with spaces like "Hello World.md" from being quoted
config.set(REMOTE_NAME, "encoding", "Slash,InvalidUtf8")
# Save updated config
save_rclone_config(config)
console.print(f"[green]Configured rclone remote: {REMOTE_NAME}[/green]")
return REMOTE_NAME
@@ -1,263 +0,0 @@
"""Cross-platform rclone installation utilities."""
import os
import platform
import shutil
import subprocess
from typing import Optional
from rich.console import Console
console = Console()
class RcloneInstallError(Exception):
"""Exception raised for rclone installation errors."""
pass
def is_rclone_installed() -> bool:
"""Check if rclone is already installed and available in PATH."""
return shutil.which("rclone") is not None
def get_platform() -> str:
"""Get the current platform identifier."""
system = platform.system().lower()
if system == "darwin":
return "macos"
elif system == "linux":
return "linux"
elif system == "windows":
return "windows"
else:
raise RcloneInstallError(f"Unsupported platform: {system}")
def run_command(command: list[str], check: bool = True) -> subprocess.CompletedProcess:
"""Run a command with proper error handling."""
try:
console.print(f"[dim]Running: {' '.join(command)}[/dim]")
result = subprocess.run(command, capture_output=True, text=True, check=check)
if result.stdout:
console.print(f"[dim]Output: {result.stdout.strip()}[/dim]")
return result
except subprocess.CalledProcessError as e:
console.print(f"[red]Command failed: {e}[/red]")
if e.stderr:
console.print(f"[red]Error output: {e.stderr}[/red]")
raise RcloneInstallError(f"Command failed: {e}") from e
except FileNotFoundError as e:
raise RcloneInstallError(f"Command not found: {' '.join(command)}") from e
def install_rclone_macos() -> None:
"""Install rclone on macOS using Homebrew or official script."""
# Try Homebrew first
if shutil.which("brew"):
try:
console.print("[blue]Installing rclone via Homebrew...[/blue]")
run_command(["brew", "install", "rclone"])
console.print("[green]rclone installed via Homebrew[/green]")
return
except RcloneInstallError:
console.print(
"[yellow]Homebrew installation failed, trying official script...[/yellow]"
)
# Fallback to official script
console.print("[blue]Installing rclone via official script...[/blue]")
try:
run_command(["sh", "-c", "curl https://rclone.org/install.sh | sudo bash"])
console.print("[green]rclone installed via official script[/green]")
except RcloneInstallError:
raise RcloneInstallError(
"Failed to install rclone. Please install manually: brew install rclone"
)
def install_rclone_linux() -> None:
"""Install rclone on Linux using package managers or official script."""
# Try snap first (most universal)
if shutil.which("snap"):
try:
console.print("[blue]Installing rclone via snap...[/blue]")
run_command(["sudo", "snap", "install", "rclone"])
console.print("[green]rclone installed via snap[/green]")
return
except RcloneInstallError:
console.print("[yellow]Snap installation failed, trying apt...[/yellow]")
# Try apt (Debian/Ubuntu)
if shutil.which("apt"):
try:
console.print("[blue]Installing rclone via apt...[/blue]")
run_command(["sudo", "apt", "update"])
run_command(["sudo", "apt", "install", "-y", "rclone"])
console.print("[green]rclone installed via apt[/green]")
return
except RcloneInstallError:
console.print("[yellow]apt installation failed, trying official script...[/yellow]")
# Fallback to official script
console.print("[blue]Installing rclone via official script...[/blue]")
try:
run_command(["sh", "-c", "curl https://rclone.org/install.sh | sudo bash"])
console.print("[green]rclone installed via official script[/green]")
except RcloneInstallError:
raise RcloneInstallError(
"Failed to install rclone. Please install manually: sudo snap install rclone"
)
def install_rclone_windows() -> None:
"""Install rclone on Windows using package managers."""
# Try winget first (built into Windows 10+)
if shutil.which("winget"):
try:
console.print("[blue]Installing rclone via winget...[/blue]")
run_command(
[
"winget",
"install",
"Rclone.Rclone",
"--accept-source-agreements",
"--accept-package-agreements",
]
)
console.print("[green]rclone installed via winget[/green]")
return
except RcloneInstallError:
console.print("[yellow]winget installation failed, trying chocolatey...[/yellow]")
# Try chocolatey
if shutil.which("choco"):
try:
console.print("[blue]Installing rclone via chocolatey...[/blue]")
run_command(["choco", "install", "rclone", "-y"])
console.print("[green]rclone installed via chocolatey[/green]")
return
except RcloneInstallError:
console.print("[yellow]chocolatey installation failed, trying scoop...[/yellow]")
# Try scoop
if shutil.which("scoop"):
try:
console.print("[blue]Installing rclone via scoop...[/blue]")
run_command(["scoop", "install", "rclone"])
console.print("[green]rclone installed via scoop[/green]")
return
except RcloneInstallError:
console.print("[yellow]scoop installation failed[/yellow]")
# No package manager available - provide detailed instructions
error_msg = (
"Could not install rclone automatically.\n\n"
"Windows requires a package manager to install rclone. Options:\n\n"
"1. Install winget (recommended, built into Windows 11):\n"
" - Windows 11: Already installed\n"
" - Windows 10: Install 'App Installer' from Microsoft Store\n"
" - Then run: bm cloud setup\n\n"
"2. Install chocolatey:\n"
" - Visit: https://chocolatey.org/install\n"
" - Then run: bm cloud setup\n\n"
"3. Install scoop:\n"
" - Visit: https://scoop.sh\n"
" - Then run: bm cloud setup\n\n"
"4. Manual installation:\n"
" - Download from: https://rclone.org/downloads/\n"
" - Extract and add to PATH\n"
)
raise RcloneInstallError(error_msg)
def install_rclone(platform_override: Optional[str] = None) -> None:
"""Install rclone for the current platform."""
if is_rclone_installed():
console.print("[green]rclone is already installed[/green]")
return
platform_name = platform_override or get_platform()
console.print(f"[blue]Installing rclone for {platform_name}...[/blue]")
try:
if platform_name == "macos":
install_rclone_macos()
elif platform_name == "linux":
install_rclone_linux()
elif platform_name == "windows":
install_rclone_windows()
refresh_windows_path()
else:
raise RcloneInstallError(f"Unsupported platform: {platform_name}")
# Verify installation
if not is_rclone_installed():
raise RcloneInstallError("rclone installation completed but command not found in PATH")
console.print("[green]rclone installation completed successfully[/green]")
except RcloneInstallError:
raise
except Exception as e:
raise RcloneInstallError(f"Unexpected error during installation: {e}") from e
def refresh_windows_path() -> None:
"""Refresh the Windows PATH environment variable for the current session."""
if platform.system().lower() != "windows":
return
# Importing here after performing platform detection. Also note that we have to ignore pylance/pyright
# warnings about winreg attributes so that "errors" don't appear on non-Windows platforms.
import winreg
user_key_path = r"Environment"
system_key_path = r"System\CurrentControlSet\Control\Session Manager\Environment"
new_path = ""
# Read user PATH
try:
reg_key = winreg.OpenKey(winreg.HKEY_CURRENT_USER, user_key_path, 0, winreg.KEY_READ) # type: ignore[reportAttributeAccessIssue]
user_path, _ = winreg.QueryValueEx(reg_key, "PATH") # type: ignore[reportAttributeAccessIssue]
winreg.CloseKey(reg_key) # type: ignore[reportAttributeAccessIssue]
except Exception:
user_path = ""
# Read system PATH
try:
reg_key = winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE, system_key_path, 0, winreg.KEY_READ) # type: ignore[reportAttributeAccessIssue]
system_path, _ = winreg.QueryValueEx(reg_key, "PATH") # type: ignore[reportAttributeAccessIssue]
winreg.CloseKey(reg_key) # type: ignore[reportAttributeAccessIssue]
except Exception:
system_path = ""
# Merge user and system PATHs (system first, then user)
if system_path and user_path:
new_path = system_path + ";" + user_path
elif system_path:
new_path = system_path
elif user_path:
new_path = user_path
if new_path:
os.environ["PATH"] = new_path
def get_rclone_version() -> Optional[str]:
"""Get the installed rclone version."""
if not is_rclone_installed():
return None
try:
result = run_command(["rclone", "version"], check=False)
if result.returncode == 0:
# Parse version from output (format: "rclone v1.64.0")
lines = result.stdout.strip().split("\n")
for line in lines:
if line.startswith("rclone v"):
return line.split()[1]
return "unknown"
except Exception:
return "unknown"
@@ -1,159 +0,0 @@
"""Restore CLI commands for Basic Memory Cloud.
SPEC-29 Phase 3: CLI commands for restoring files from Tigris bucket snapshots.
"""
import asyncio
import typer
from rich.console import Console
from basic_memory.cli.app import cloud_app
from basic_memory.cli.commands.cloud.api_client import (
CloudAPIError,
SubscriptionRequiredError,
make_api_request,
)
from basic_memory.cli.commands.cloud.schemas import BucketSnapshotBrowseResponse
from basic_memory.config import ConfigManager
console = Console()
@cloud_app.command("restore")
def restore(
path: str = typer.Argument(
...,
help="Path to restore (file or folder, e.g., 'notes/project.md' or 'research/')",
),
snapshot_id: str = typer.Option(
...,
"--snapshot",
"-s",
help="ID of the snapshot to restore from",
),
force: bool = typer.Option(
False,
"--force",
"-f",
help="Skip confirmation prompt",
),
) -> None:
"""Restore a file or folder from a snapshot.
This command restores files from a previous snapshot to the current bucket.
The restored files will overwrite any existing files at the same path.
Examples:
bm cloud restore notes/project.md --snapshot abc123
bm cloud restore research/ --snapshot abc123
bm cloud restore notes/project.md --snapshot abc123 --force
"""
async def _restore():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
# Normalize path - remove leading slash if present
normalized_path = path.lstrip("/")
if not force:
# Show what will be restored
console.print(f"[blue]Preparing to restore from snapshot {snapshot_id}[/blue]")
console.print(f" Path: {normalized_path}")
# Try to browse the snapshot to show what files will be affected
try:
browse_url = f"{host_url}/api/bucket-snapshots/{snapshot_id}/browse"
if normalized_path:
browse_url += f"?prefix={normalized_path}"
response = await make_api_request(
method="GET",
url=browse_url,
)
browse_response = BucketSnapshotBrowseResponse.model_validate(response.json())
if browse_response.files:
if len(browse_response.files) <= 10:
console.print("\n Files to restore:")
for file_info in browse_response.files:
console.print(f" - {file_info.key}")
else:
console.print(
f"\n {len(browse_response.files)} files will be restored"
)
console.print(" First 5 files:")
for file_info in browse_response.files[:5]:
console.print(f" - {file_info.key}")
console.print(f" ... and {len(browse_response.files) - 5} more")
else:
console.print(
f"\n[yellow]No files found matching '{normalized_path}' "
f"in snapshot[/yellow]"
)
raise typer.Exit(0)
except CloudAPIError as browse_error:
if browse_error.status_code == 404:
console.print(f"[red]Snapshot not found: {snapshot_id}[/red]")
raise typer.Exit(1)
# If browse fails for other reasons, proceed with confirmation anyway
pass
console.print(
"\n[yellow]Warning: Restored files will overwrite existing files![/yellow]"
)
confirmed = typer.confirm("\nProceed with restore?")
if not confirmed:
console.print("[yellow]Restore cancelled[/yellow]")
raise typer.Exit(0)
console.print(f"[blue]Restoring from snapshot {snapshot_id}...[/blue]")
response = await make_api_request(
method="POST",
url=f"{host_url}/api/bucket-snapshots/{snapshot_id}/restore",
json_data={"path": normalized_path},
)
data = response.json()
restored_files = data.get("restored", [])
returned_snapshot_id = data.get("snapshot_id", snapshot_id)
if restored_files:
console.print(f"[green]Successfully restored {len(restored_files)} file(s)[/green]")
if len(restored_files) <= 10:
for file_path in restored_files:
console.print(f" - {file_path}")
else:
console.print(" First 5 restored files:")
for file_path in restored_files[:5]:
console.print(f" - {file_path}")
console.print(f" ... and {len(restored_files) - 5} more")
console.print(f"\n[dim]Snapshot ID: {returned_snapshot_id}[/dim]")
else:
console.print("[yellow]No files were restored[/yellow]")
console.print(f"[dim]No files matching '{normalized_path}' found in snapshot[/dim]")
except typer.Exit:
# Re-raise typer.Exit without modification - it's used for clean exits
raise
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
if e.status_code == 404:
console.print(f"[red]Snapshot not found: {snapshot_id}[/red]")
else:
console.print(f"[red]Failed to restore: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_restore())
@@ -1,55 +0,0 @@
"""Pydantic schemas for Basic Memory Cloud API responses.
These schemas mirror the API response models from basic-memory-cloud
for type-safe parsing of API responses in CLI commands.
"""
from datetime import datetime
from uuid import UUID
from pydantic import BaseModel
class BucketSnapshotFileInfo(BaseModel):
"""File info from snapshot browse response."""
key: str
size: int
last_modified: datetime
etag: str | None = None
class BucketSnapshotBrowseResponse(BaseModel):
"""Response from browsing snapshot contents."""
files: list[BucketSnapshotFileInfo]
prefix: str
snapshot_version: str
class BucketSnapshotResponse(BaseModel):
"""Response model for bucket snapshot data."""
id: UUID
bucket_name: str
snapshot_version: str
name: str
description: str | None
auto: bool
created_at: datetime
created_by: UUID | None = None
class BucketSnapshotListResponse(BaseModel):
"""Response from listing bucket snapshots."""
snapshots: list[BucketSnapshotResponse]
total: int
class BucketSnapshotRestoreResponse(BaseModel):
"""Response from restore operation."""
restored: list[str]
snapshot_version: str
snapshot_id: UUID
@@ -1,370 +0,0 @@
"""Snapshot CLI commands for Basic Memory Cloud.
SPEC-29 Phase 3: CLI commands for managing Tigris bucket snapshots.
"""
import asyncio
from datetime import datetime
from typing import Optional
import typer
from rich.console import Console
from rich.table import Table
from basic_memory.cli.commands.cloud.api_client import (
CloudAPIError,
SubscriptionRequiredError,
make_api_request,
)
from basic_memory.cli.commands.cloud.schemas import BucketSnapshotBrowseResponse
from basic_memory.config import ConfigManager
console = Console()
snapshot_app = typer.Typer(help="Manage bucket snapshots")
def _format_timestamp(iso_timestamp: str) -> str:
"""Format ISO timestamp to a human-readable format."""
try:
dt = datetime.fromisoformat(iso_timestamp.replace("Z", "+00:00"))
return dt.strftime("%Y-%m-%d %H:%M:%S")
except (ValueError, AttributeError):
return iso_timestamp
@snapshot_app.command("create")
def create(
description: str = typer.Argument(
...,
help="Description for the snapshot",
),
) -> None:
"""Create a new bucket snapshot.
Examples:
bm cloud snapshot create "before major refactor"
bm cloud snapshot create "daily backup"
"""
async def _create():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
console.print("[blue]Creating snapshot...[/blue]")
response = await make_api_request(
method="POST",
url=f"{host_url}/api/bucket-snapshots",
json_data={"description": description},
)
data = response.json()
snapshot_id = data.get("id", "unknown")
snapshot_version = data.get("snapshot_version", "unknown")
created_at = _format_timestamp(data.get("created_at", ""))
console.print("[green]Snapshot created successfully[/green]")
console.print(f" ID: {snapshot_id}")
console.print(f" Version: {snapshot_version}")
console.print(f" Created: {created_at}")
console.print(f" Description: {description}")
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
console.print(f"[red]Failed to create snapshot: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_create())
@snapshot_app.command("list")
def list_snapshots(
limit: int = typer.Option(
10,
"--limit",
"-l",
help="Maximum number of snapshots to display",
),
) -> None:
"""List all bucket snapshots.
Examples:
bm cloud snapshot list
bm cloud snapshot list --limit 20
"""
async def _list():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
console.print("[blue]Fetching snapshots...[/blue]")
response = await make_api_request(
method="GET",
url=f"{host_url}/api/bucket-snapshots",
)
data = response.json()
snapshots = data.get("snapshots", [])
total = data.get("total", len(snapshots))
if not snapshots:
console.print("[yellow]No snapshots found[/yellow]")
console.print(
'\n[dim]Create a snapshot with: bm cloud snapshot create "description"[/dim]'
)
return
# Create a table for displaying snapshots
table = Table(title=f"Bucket Snapshots ({total} total)")
table.add_column("ID", style="cyan", no_wrap=True)
table.add_column("Description", style="white")
table.add_column("Auto", style="dim")
table.add_column("Created", style="green")
for snapshot in snapshots[:limit]:
snapshot_id = snapshot.get("id", "unknown")
desc = snapshot.get("description") or snapshot.get("name", "-")
auto = "yes" if snapshot.get("auto", False) else "no"
created_at = _format_timestamp(snapshot.get("created_at", ""))
table.add_row(snapshot_id, desc, auto, created_at)
console.print(table)
if total > limit:
console.print(
f"\n[dim]Showing {limit} of {total} snapshots. Use --limit to see more.[/dim]"
)
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
console.print(f"[red]Failed to list snapshots: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_list())
@snapshot_app.command("delete")
def delete(
snapshot_id: str = typer.Argument(
...,
help="The ID of the snapshot to delete",
),
force: bool = typer.Option(
False,
"--force",
"-f",
help="Skip confirmation prompt",
),
) -> None:
"""Delete a bucket snapshot.
Examples:
bm cloud snapshot delete abc123
bm cloud snapshot delete abc123 --force
"""
async def _delete():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
if not force:
# Fetch snapshot details first to show what will be deleted
console.print("[blue]Fetching snapshot details...[/blue]")
try:
response = await make_api_request(
method="GET",
url=f"{host_url}/api/bucket-snapshots/{snapshot_id}",
)
data = response.json()
desc = data.get("description") or data.get("name", "unnamed")
created_at = _format_timestamp(data.get("created_at", ""))
console.print("\nSnapshot to delete:")
console.print(f" ID: {snapshot_id}")
console.print(f" Description: {desc}")
console.print(f" Created: {created_at}")
except CloudAPIError:
# If we can't fetch details, proceed with confirmation anyway
pass
confirmed = typer.confirm("\nAre you sure you want to delete this snapshot?")
if not confirmed:
console.print("[yellow]Deletion cancelled[/yellow]")
raise typer.Exit(0)
console.print("[blue]Deleting snapshot...[/blue]")
await make_api_request(
method="DELETE",
url=f"{host_url}/api/bucket-snapshots/{snapshot_id}",
)
console.print(f"[green]Snapshot {snapshot_id} deleted successfully[/green]")
except typer.Exit:
# Re-raise typer.Exit without modification - it's used for clean exits
raise
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
if e.status_code == 404:
console.print(f"[red]Snapshot not found: {snapshot_id}[/red]")
else:
console.print(f"[red]Failed to delete snapshot: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_delete())
@snapshot_app.command("show")
def show(
snapshot_id: str = typer.Argument(
...,
help="The ID of the snapshot to show",
),
) -> None:
"""Show details of a specific snapshot.
Examples:
bm cloud snapshot show abc123
"""
async def _show():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
response = await make_api_request(
method="GET",
url=f"{host_url}/api/bucket-snapshots/{snapshot_id}",
)
data = response.json()
console.print("[bold blue]Snapshot Details[/bold blue]")
console.print(f" ID: {data.get('id', 'unknown')}")
console.print(f" Bucket: {data.get('bucket_name', 'unknown')}")
console.print(f" Version: {data.get('snapshot_version', 'unknown')}")
console.print(f" Name: {data.get('name', '-')}")
console.print(f" Description: {data.get('description') or '-'}")
console.print(f" Auto: {'yes' if data.get('auto', False) else 'no'}")
console.print(f" Created: {_format_timestamp(data.get('created_at', ''))}")
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
if e.status_code == 404:
console.print(f"[red]Snapshot not found: {snapshot_id}[/red]")
else:
console.print(f"[red]Failed to get snapshot details: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_show())
@snapshot_app.command("browse")
def browse(
snapshot_id: str = typer.Argument(
...,
help="The ID of the snapshot to browse",
),
prefix: Optional[str] = typer.Option(
None,
"--prefix",
"-p",
help="Filter files by path prefix (e.g., 'notes/')",
),
) -> None:
"""Browse contents of a snapshot.
Examples:
bm cloud snapshot browse abc123
bm cloud snapshot browse abc123 --prefix notes/
"""
async def _browse():
try:
config_manager = ConfigManager()
config = config_manager.config
host_url = config.cloud_host.rstrip("/")
url = f"{host_url}/api/bucket-snapshots/{snapshot_id}/browse"
if prefix:
url += f"?prefix={prefix}"
response = await make_api_request(
method="GET",
url=url,
)
browse_response = BucketSnapshotBrowseResponse.model_validate(response.json())
if not browse_response.files:
if prefix:
console.print(f"[yellow]No files found with prefix '{prefix}'[/yellow]")
else:
console.print("[yellow]No files found in snapshot[/yellow]")
return
console.print(
f"[bold blue]Snapshot Contents ({len(browse_response.files)} files)[/bold blue]"
)
for file_info in browse_response.files:
size_kb = file_info.size // 1024
console.print(f" {file_info.key} ({size_kb} KB)")
console.print(
f"\n[dim]Use 'bm cloud restore <path> --snapshot {snapshot_id}' "
f"to restore files[/dim]"
)
except SubscriptionRequiredError as e:
console.print("\n[red]Subscription Required[/red]\n")
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
raise typer.Exit(1)
except CloudAPIError as e:
if e.status_code == 404:
console.print(f"[red]Snapshot not found: {snapshot_id}[/red]")
else:
console.print(f"[red]Failed to browse snapshot: {e}[/red]")
raise typer.Exit(1)
except Exception as e:
console.print(f"[red]Unexpected error: {e}[/red]")
raise typer.Exit(1)
asyncio.run(_browse())
@@ -1,240 +0,0 @@
"""WebDAV upload functionality for basic-memory projects."""
import os
from pathlib import Path
from contextlib import AbstractAsyncContextManager
from typing import Callable
import aiofiles
import httpx
from basic_memory.ignore_utils import load_gitignore_patterns, should_ignore_path
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.tools.utils import call_put
# Archive file extensions that should be skipped during upload
ARCHIVE_EXTENSIONS = {".zip", ".tar", ".gz", ".bz2", ".xz", ".7z", ".rar", ".tgz", ".tbz2"}
async def upload_path(
local_path: Path,
project_name: str,
verbose: bool = False,
use_gitignore: bool = True,
dry_run: bool = False,
*,
client_cm_factory: Callable[[], AbstractAsyncContextManager[httpx.AsyncClient]] | None = None,
put_func=call_put,
) -> bool:
"""
Upload a file or directory to cloud project via WebDAV.
Args:
local_path: Path to local file or directory
project_name: Name of cloud project (destination)
verbose: Show detailed information about filtering and upload
use_gitignore: If False, skip .gitignore patterns (still use .bmignore)
dry_run: If True, show what would be uploaded without uploading
Returns:
True if upload succeeded, False otherwise
"""
try:
# Resolve path
local_path = local_path.resolve()
# Check if path exists
if not local_path.exists():
print(f"Error: Path does not exist: {local_path}")
return False
# Get files to upload
if local_path.is_file():
files_to_upload = [(local_path, local_path.name)]
if verbose:
print(f"Uploading single file: {local_path.name}")
else:
files_to_upload = _get_files_to_upload(local_path, verbose, use_gitignore)
if not files_to_upload:
print("No files found to upload")
if verbose:
print(
"\nTip: Use --verbose to see which files are being filtered, "
"or --no-gitignore to skip .gitignore patterns"
)
return True
print(f"Found {len(files_to_upload)} file(s) to upload")
# Calculate total size
total_bytes = sum(file_path.stat().st_size for file_path, _ in files_to_upload)
skipped_count = 0
# If dry run, just show what would be uploaded
if dry_run:
print("\nFiles that would be uploaded:")
for file_path, relative_path in files_to_upload:
# Skip archive files
if _is_archive_file(file_path):
print(f" [SKIP] {relative_path} (archive file)")
skipped_count += 1
continue
size = file_path.stat().st_size
if size < 1024:
size_str = f"{size} bytes"
elif size < 1024 * 1024:
size_str = f"{size / 1024:.1f} KB"
else:
size_str = f"{size / (1024 * 1024):.1f} MB"
print(f" {relative_path} ({size_str})")
else:
# Upload files using httpx.
# Allow injection for tests (MockTransport) while keeping production default.
cm_factory = client_cm_factory or get_client
async with cm_factory() as client:
for i, (file_path, relative_path) in enumerate(files_to_upload, 1):
# Skip archive files (zip, tar, gz, etc.)
if _is_archive_file(file_path):
print(
f"Skipping archive file: {relative_path} ({i}/{len(files_to_upload)})"
)
skipped_count += 1
continue
# Build remote path: /webdav/{project_name}/{relative_path}
remote_path = f"/webdav/{project_name}/{relative_path}"
print(f"Uploading {relative_path} ({i}/{len(files_to_upload)})")
# Get file modification time
file_stat = file_path.stat()
mtime = int(file_stat.st_mtime)
# Read file content asynchronously
async with aiofiles.open(file_path, "rb") as f:
content = await f.read()
# Upload via HTTP PUT to WebDAV endpoint with mtime header
# Using X-OC-Mtime (ownCloud/Nextcloud standard)
response = await put_func(
client, remote_path, content=content, headers={"X-OC-Mtime": str(mtime)}
)
response.raise_for_status()
# Format total size based on magnitude
if total_bytes < 1024:
size_str = f"{total_bytes} bytes"
elif total_bytes < 1024 * 1024:
size_str = f"{total_bytes / 1024:.1f} KB"
else:
size_str = f"{total_bytes / (1024 * 1024):.1f} MB"
uploaded_count = len(files_to_upload) - skipped_count
if dry_run:
print(f"\nTotal: {uploaded_count} file(s) ({size_str})")
if skipped_count > 0:
print(f" Would skip {skipped_count} archive file(s)")
else:
print(f"✓ Upload complete: {uploaded_count} file(s) ({size_str})")
if skipped_count > 0:
print(f" Skipped {skipped_count} archive file(s)")
return True
except httpx.HTTPStatusError as e:
print(f"Upload failed: HTTP {e.response.status_code} - {e.response.text}")
return False
except Exception as e:
print(f"Upload failed: {e}")
return False
def _is_archive_file(file_path: Path) -> bool:
"""
Check if a file is an archive file based on its extension.
Args:
file_path: Path to the file to check
Returns:
True if file is an archive, False otherwise
"""
return file_path.suffix.lower() in ARCHIVE_EXTENSIONS
def _get_files_to_upload(
directory: Path, verbose: bool = False, use_gitignore: bool = True
) -> list[tuple[Path, str]]:
"""
Get list of files to upload from directory.
Uses .bmignore and optionally .gitignore patterns for filtering.
Args:
directory: Directory to scan
verbose: Show detailed filtering information
use_gitignore: If False, skip .gitignore patterns (still use .bmignore)
Returns:
List of (absolute_path, relative_path) tuples
"""
files = []
ignored_files = []
# Load ignore patterns from .bmignore and optionally .gitignore
ignore_patterns = load_gitignore_patterns(directory, use_gitignore=use_gitignore)
if verbose:
gitignore_path = directory / ".gitignore"
gitignore_exists = gitignore_path.exists() and use_gitignore
print(f"\nScanning directory: {directory}")
print("Using .bmignore: Yes")
print(f"Using .gitignore: {'Yes' if gitignore_exists else 'No'}")
print(f"Ignore patterns loaded: {len(ignore_patterns)}")
if ignore_patterns and len(ignore_patterns) <= 20:
print(f"Patterns: {', '.join(sorted(ignore_patterns))}")
print()
# Walk through directory
for root, dirs, filenames in os.walk(directory):
root_path = Path(root)
# Filter directories based on ignore patterns
filtered_dirs = []
for d in dirs:
dir_path = root_path / d
if should_ignore_path(dir_path, directory, ignore_patterns):
if verbose:
rel_path = dir_path.relative_to(directory)
print(f" [IGNORED DIR] {rel_path}/")
else:
filtered_dirs.append(d)
dirs[:] = filtered_dirs
# Process files
for filename in filenames:
file_path = root_path / filename
# Calculate relative path for display/remote
rel_path = file_path.relative_to(directory)
remote_path = str(rel_path).replace("\\", "/")
# Check if file should be ignored
if should_ignore_path(file_path, directory, ignore_patterns):
ignored_files.append(remote_path)
if verbose:
print(f" [IGNORED] {remote_path}")
continue
if verbose:
print(f" [INCLUDE] {remote_path}")
files.append((file_path, remote_path))
if verbose:
print("\nSummary:")
print(f" Files to upload: {len(files)}")
print(f" Files ignored: {len(ignored_files)}")
return files
@@ -1,124 +0,0 @@
"""Upload CLI commands for basic-memory projects."""
from pathlib import Path
import typer
from rich.console import Console
from basic_memory.cli.app import cloud_app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.cli.commands.cloud.cloud_utils import (
create_cloud_project,
project_exists,
sync_project,
)
from basic_memory.cli.commands.cloud.upload import upload_path
console = Console()
@cloud_app.command("upload")
def upload(
path: Path = typer.Argument(
...,
help="Path to local file or directory to upload",
exists=True,
readable=True,
resolve_path=True,
),
project: str = typer.Option(
...,
"--project",
"-p",
help="Cloud project name (destination)",
),
create_project: bool = typer.Option(
False,
"--create-project",
"-c",
help="Create project if it doesn't exist",
),
sync: bool = typer.Option(
True,
"--sync/--no-sync",
help="Sync project after upload (default: true)",
),
verbose: bool = typer.Option(
False,
"--verbose",
"-v",
help="Show detailed information about file filtering and upload",
),
no_gitignore: bool = typer.Option(
False,
"--no-gitignore",
help="Skip .gitignore patterns (still respects .bmignore)",
),
dry_run: bool = typer.Option(
False,
"--dry-run",
help="Show what would be uploaded without actually uploading",
),
) -> None:
"""Upload local files or directories to cloud project via WebDAV.
Examples:
bm cloud upload ~/my-notes --project research
bm cloud upload notes.md --project research --create-project
bm cloud upload ~/docs --project work --no-sync
bm cloud upload ./history --project proto --verbose
bm cloud upload ./notes --project work --no-gitignore
bm cloud upload ./files --project test --dry-run
"""
async def _upload():
# Check if project exists
if not await project_exists(project):
if create_project:
console.print(f"[blue]Creating cloud project '{project}'...[/blue]")
try:
await create_cloud_project(project)
console.print(f"[green]Created project '{project}'[/green]")
except Exception as e:
console.print(f"[red]Failed to create project: {e}[/red]")
raise typer.Exit(1)
else:
console.print(
f"[red]Project '{project}' does not exist.[/red]\n"
f"[yellow]Options:[/yellow]\n"
f" 1. Create it first: bm project add {project}\n"
f" 2. Use --create-project flag to create automatically"
)
raise typer.Exit(1)
# Perform upload (or dry run)
if dry_run:
console.print(
f"[yellow]DRY RUN: Showing what would be uploaded to '{project}'[/yellow]"
)
else:
console.print(f"[blue]Uploading {path} to project '{project}'...[/blue]")
success = await upload_path(
path, project, verbose=verbose, use_gitignore=not no_gitignore, dry_run=dry_run
)
if not success:
console.print("[red]Upload failed[/red]")
raise typer.Exit(1)
if dry_run:
console.print("[yellow]DRY RUN complete - no files were uploaded[/yellow]")
else:
console.print(f"[green]Successfully uploaded to '{project}'[/green]")
# Sync project if requested (skip on dry run)
# Force full scan after bisync to ensure database is up-to-date with synced files
if sync and not dry_run:
console.print(f"[blue]Syncing project '{project}'...[/blue]")
try:
await sync_project(project, force_full=True)
except Exception as e:
console.print(f"[yellow]Warning: Sync failed: {e}[/yellow]")
console.print("[dim]Files uploaded but may not be indexed yet[/dim]")
run_with_cleanup(_upload())
@@ -1,101 +0,0 @@
"""utility functions for commands"""
import asyncio
from typing import Optional, TypeVar, Coroutine, Any
from mcp.server.fastmcp.exceptions import ToolError
import typer
from rich.console import Console
from basic_memory import db
from basic_memory.config import ConfigManager
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.tools.utils import call_post, call_get
from basic_memory.mcp.project_context import get_active_project
from basic_memory.schemas import ProjectInfoResponse
console = Console()
T = TypeVar("T")
def run_with_cleanup(coro: Coroutine[Any, Any, T]) -> T:
"""Run an async coroutine with proper database cleanup.
This helper ensures database connections are cleaned up before the
event loop closes, preventing process hangs in CLI commands.
Args:
coro: The coroutine to run
Returns:
The result of the coroutine
"""
async def _with_cleanup() -> T:
try:
return await coro
finally:
await db.shutdown_db()
return asyncio.run(_with_cleanup())
async def run_sync(
project: Optional[str] = None,
force_full: bool = False,
run_in_background: bool = True,
):
"""Run sync operation via API endpoint.
Args:
project: Optional project name
force_full: If True, force a full scan bypassing watermark optimization
run_in_background: If True, return immediately; if False, wait for completion
"""
# Resolve default project so get_client() can route per-project
project = project or ConfigManager().default_project
try:
async with get_client(project_name=project) as client:
project_item = await get_active_project(client, project, None)
url = f"/v2/projects/{project_item.external_id}/sync"
params = []
if force_full:
params.append("force_full=true")
if not run_in_background:
params.append("run_in_background=false")
if params:
url += "?" + "&".join(params)
response = await call_post(client, url)
data = response.json()
# Background mode returns {"message": "..."}, foreground returns SyncReportResponse
if "message" in data:
console.print(f"[green]{data['message']}[/green]")
else:
# Foreground mode - show summary of sync results
total = data.get("total", 0)
new_count = len(data.get("new", []))
modified_count = len(data.get("modified", []))
deleted_count = len(data.get("deleted", []))
console.print(
f"[green]Synced {total} files[/green] "
f"(new: {new_count}, modified: {modified_count}, deleted: {deleted_count})"
)
except (ToolError, ValueError) as e:
console.print(f"[red]Sync failed: {e}[/red]")
raise typer.Exit(1)
async def get_project_info(project: str):
"""Get project information via API endpoint."""
try:
async with get_client(project_name=project) as client:
project_item = await get_active_project(client, project, None)
response = await call_get(client, f"/v2/projects/{project_item.external_id}/info")
return ProjectInfoResponse.model_validate(response.json())
except (ToolError, ValueError) as e:
console.print(f"[red]Sync failed: {e}[/red]")
raise typer.Exit(1)
+20 -197
View File
@@ -1,51 +1,14 @@
"""Database management commands."""
import asyncio
from pathlib import Path
import typer
from loguru import logger
from rich.console import Console
from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn, TaskProgressColumn
from sqlalchemy.exc import OperationalError
from basic_memory import db
from basic_memory.cli.app import app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.config import ConfigManager
from basic_memory.repository import ProjectRepository
from basic_memory.services.initialization import reconcile_projects_with_config
from basic_memory.sync.sync_service import get_sync_service
console = Console()
async def _reindex_projects(app_config):
"""Reindex all projects in a single async context.
This ensures all database operations use the same event loop,
and proper cleanup happens when the function completes.
"""
try:
await reconcile_projects_with_config(app_config)
# Get database session (migrations already run if needed)
_, session_maker = await db.get_or_create_db(
db_path=app_config.database_path,
db_type=db.DatabaseType.FILESYSTEM,
)
project_repository = ProjectRepository(session_maker)
projects = await project_repository.get_active_projects()
for project in projects:
console.print(f" Indexing [cyan]{project.name}[/cyan]...")
logger.info(f"Starting sync for project: {project.name}")
sync_service = await get_sync_service(project)
sync_dir = Path(project.path)
await sync_service.sync(sync_dir, project_name=project.name)
logger.info(f"Sync completed for project: {project.name}")
finally:
# Clean up database connections before event loop closes
await db.shutdown_db()
from basic_memory.config import app_config, config_manager
@app.command()
@@ -53,169 +16,29 @@ def reset(
reindex: bool = typer.Option(False, "--reindex", help="Rebuild db index from filesystem"),
): # pragma: no cover
"""Reset database (drop all tables and recreate)."""
console.print(
"[yellow]Note:[/yellow] This only deletes the index database. "
"Your markdown note files will not be affected.\n"
"Use [green]bm reset --reindex[/green] to automatically rebuild the index afterward."
)
if typer.confirm("Reset the database index?"):
if typer.confirm("This will delete all data in your db. Are you sure?"):
logger.info("Resetting database...")
config_manager = ConfigManager()
app_config = config_manager.config
# Get database path
db_path = app_config.app_database_path
# Delete the database file and WAL files if they exist
for suffix in ["", "-shm", "-wal"]:
path = db_path.parent / f"{db_path.name}{suffix}"
if path.exists():
try:
path.unlink()
logger.info(f"Deleted: {path}")
except OSError as e:
console.print(
f"[red]Error:[/red] Cannot delete {path.name}: {e}\n"
"The database may be in use by another process (e.g., MCP server).\n"
"Please close Claude Desktop or any other Basic Memory clients and try again."
)
raise typer.Exit(1)
# Delete the database file if it exists
if db_path.exists():
db_path.unlink()
logger.info(f"Database file deleted: {db_path}")
# Create a new empty database (preserves project configuration)
try:
run_with_cleanup(db.run_migrations(app_config))
except OperationalError as e:
if "disk I/O error" in str(e) or "database is locked" in str(e):
console.print(
"[red]Error:[/red] Cannot access database. "
"It may be in use by another process (e.g., MCP server).\n"
"Please close Claude Desktop or any other Basic Memory clients and try again."
)
raise typer.Exit(1)
raise
console.print("[green]Database reset complete[/green]")
# Reset project configuration
config_manager.config.projects = {"main": str(Path.home() / "basic-memory")}
config_manager.config.default_project = "main"
config_manager.save_config(config_manager.config)
logger.info("Project configuration reset to default")
# Create a new empty database
asyncio.run(db.run_migrations(app_config))
logger.info("Database reset complete")
if reindex:
projects = list(app_config.projects)
if not projects:
console.print("[yellow]No projects configured. Skipping reindex.[/yellow]")
else:
console.print(f"Rebuilding search index for {len(projects)} project(s)...")
# Note: _reindex_projects has its own cleanup, but run_with_cleanup
# ensures db.shutdown_db() is called even if _reindex_projects changes
run_with_cleanup(_reindex_projects(app_config))
console.print("[green]Reindex complete[/green]")
# Import and run sync
from basic_memory.cli.commands.sync import sync
@app.command()
def reindex(
embeddings: bool = typer.Option(
False, "--embeddings", "-e", help="Rebuild vector embeddings (requires semantic search)"
),
search: bool = typer.Option(False, "--search", "-s", help="Rebuild full-text search index"),
project: str = typer.Option(
None, "--project", "-p", help="Reindex a specific project (default: all)"
),
): # pragma: no cover
"""Rebuild search indexes and/or vector embeddings without dropping the database.
By default rebuilds everything (search + embeddings if semantic is enabled).
Use --search or --embeddings to rebuild only one.
Examples:
bm reindex # Rebuild everything
bm reindex --embeddings # Only rebuild vector embeddings
bm reindex --search # Only rebuild FTS index
bm reindex -p claw # Reindex only the 'claw' project
"""
# If neither flag is set, do both
if not embeddings and not search:
embeddings = True
search = True
config_manager = ConfigManager()
app_config = config_manager.config
if embeddings and not app_config.semantic_search_enabled:
console.print(
"[yellow]Semantic search is not enabled.[/yellow] "
"Set [cyan]semantic_search_enabled: true[/cyan] in config to use embeddings."
)
embeddings = False
if not search:
raise typer.Exit(0)
run_with_cleanup(_reindex(app_config, search=search, embeddings=embeddings, project=project))
async def _reindex(app_config, search: bool, embeddings: bool, project: str | None):
"""Run reindex operations."""
from basic_memory.repository import EntityRepository
from basic_memory.repository.search_repository import create_search_repository
from basic_memory.services.search_service import SearchService
from basic_memory.services.file_service import FileService
from basic_memory.markdown.markdown_processor import MarkdownProcessor
from basic_memory.markdown.entity_parser import EntityParser
try:
await reconcile_projects_with_config(app_config)
_, session_maker = await db.get_or_create_db(
db_path=app_config.database_path,
db_type=db.DatabaseType.FILESYSTEM,
)
project_repository = ProjectRepository(session_maker)
projects = await project_repository.get_active_projects()
if project:
projects = [p for p in projects if p.name == project]
if not projects:
console.print(f"[red]Project '{project}' not found.[/red]")
raise typer.Exit(1)
for proj in projects:
console.print(f"\n[bold]Project: [cyan]{proj.name}[/cyan][/bold]")
if search:
console.print(" Rebuilding full-text search index...")
sync_service = await get_sync_service(proj)
sync_dir = Path(proj.path)
await sync_service.sync(sync_dir, project_name=proj.name)
console.print(" [green]✓[/green] Full-text search index rebuilt")
if embeddings:
console.print(" Building vector embeddings...")
entity_repository = EntityRepository(session_maker, project_id=proj.id)
search_repository = create_search_repository(
session_maker, project_id=proj.id, app_config=app_config
)
project_path = Path(proj.path)
entity_parser = EntityParser(project_path)
markdown_processor = MarkdownProcessor(entity_parser, app_config=app_config)
file_service = FileService(project_path, markdown_processor, app_config=app_config)
search_service = SearchService(search_repository, entity_repository, file_service)
with Progress(
SpinnerColumn(),
TextColumn("[progress.description]{task.description}"),
BarColumn(),
TaskProgressColumn(),
console=console,
) as progress:
task = progress.add_task(" Embedding entities...", total=None)
def on_progress(entity_id, index, total):
progress.update(task, total=total, completed=index)
stats = await search_service.reindex_vectors(progress_callback=on_progress)
progress.update(task, completed=stats["total_entities"])
console.print(
f" [green]✓[/green] Embeddings complete: "
f"{stats['embedded']} entities embedded, "
f"{stats['skipped']} skipped, "
f"{stats['errors']} errors"
)
console.print("\n[green]Reindex complete![/green]")
finally:
await db.shutdown_db()
logger.info("Rebuilding search index from filesystem...")
sync(watch=False) # pyright: ignore
-153
View File
@@ -1,153 +0,0 @@
"""Doctor command for local consistency checks."""
from __future__ import annotations
import tempfile
import uuid
from pathlib import Path
from loguru import logger
from mcp.server.fastmcp.exceptions import ToolError
from rich.console import Console
import typer
from basic_memory.cli.app import app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.cli.commands.routing import force_routing, validate_routing_flags
from basic_memory.markdown.entity_parser import EntityParser
from basic_memory.markdown.markdown_processor import MarkdownProcessor
from basic_memory.markdown.schemas import EntityFrontmatter, EntityMarkdown
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.clients import KnowledgeClient, ProjectClient, SearchClient
from basic_memory.mcp.tools.utils import call_post
from basic_memory.schemas.base import Entity
from basic_memory.schemas.project_info import ProjectInfoRequest
from basic_memory.schemas.search import SearchQuery
from basic_memory.schemas import SyncReportResponse
console = Console()
async def run_doctor() -> None:
"""Run local consistency checks for file <-> database flows."""
console.print("[blue]Running Basic Memory doctor checks...[/blue]")
project_name = f"doctor-{uuid.uuid4().hex[:8]}"
api_note_title = "Doctor API Note"
manual_note_title = "Doctor Manual Note"
manual_permalink = "doctor/manual-note"
with tempfile.TemporaryDirectory() as temp_dir:
temp_path = Path(temp_dir)
async with get_client() as client:
project_client = ProjectClient(client)
project_request = ProjectInfoRequest(
name=project_name,
path=str(temp_path),
set_default=False,
)
project_id: str | None = None
try:
status = await project_client.create_project(project_request.model_dump())
if not status.new_project:
raise ValueError("Failed to create doctor project")
project_id = status.new_project.external_id
console.print(f"[green]OK[/green] Created doctor project: {project_name}")
# --- DB -> File: create an entity via API ---
knowledge_client = KnowledgeClient(client, project_id)
api_note = Entity(
title=api_note_title,
directory="doctor",
entity_type="note",
content_type="text/markdown",
content=f"# {api_note_title}\n\n- [note] API to file check",
entity_metadata={"tags": ["doctor"]},
)
api_result = await knowledge_client.create_entity(api_note.model_dump(), fast=False)
api_file = temp_path / api_result.file_path
if not api_file.exists():
raise ValueError(f"API note file missing: {api_result.file_path}")
api_text = api_file.read_text(encoding="utf-8")
if api_note_title not in api_text:
raise ValueError("API note content missing from file")
console.print("[green]OK[/green] API write created file")
# --- File -> DB: write markdown file directly, then sync ---
parser = EntityParser(temp_path)
processor = MarkdownProcessor(parser)
manual_markdown = EntityMarkdown(
frontmatter=EntityFrontmatter(
metadata={
"title": manual_note_title,
"type": "note",
"permalink": manual_permalink,
"tags": ["doctor"],
}
),
content=f"# {manual_note_title}\n\n- [note] File to DB check",
)
manual_path = temp_path / "doctor" / "manual-note.md"
await processor.write_file(manual_path, manual_markdown)
console.print("[green]OK[/green] Manual file written")
sync_response = await call_post(
client,
f"/v2/projects/{project_id}/sync?force_full=true&run_in_background=false",
)
sync_report = SyncReportResponse.model_validate(sync_response.json())
if sync_report.total == 0:
raise ValueError("Sync did not detect any changes")
console.print("[green]OK[/green] Sync indexed manual file")
search_client = SearchClient(client, project_id)
search_query = SearchQuery(title=manual_note_title)
search_results = await search_client.search(
search_query.model_dump(), page=1, page_size=5
)
if not any(result.title == manual_note_title for result in search_results.results):
raise ValueError("Manual note not found in search index")
console.print("[green]OK[/green] Search confirmed manual file")
status_response = await call_post(client, f"/v2/projects/{project_id}/status")
status_report = SyncReportResponse.model_validate(status_response.json())
if status_report.total != 0:
raise ValueError("Project status not clean after sync")
console.print("[green]OK[/green] Status clean after sync")
finally:
if project_id:
await project_client.delete_project(project_id)
console.print("[green]Doctor checks passed.[/green]")
@app.command()
def doctor(
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
),
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
) -> None:
"""Run local consistency checks to verify file/database sync."""
try:
validate_routing_flags(local, cloud)
with force_routing(local=local, cloud=cloud):
run_with_cleanup(run_doctor())
except (ToolError, ValueError) as e:
console.print(f"[red]Doctor failed: {e}[/red]")
raise typer.Exit(code=1)
except Exception as e:
logger.error(f"Doctor failed: {e}")
typer.echo(f"Doctor failed: {e}", err=True)
raise typer.Exit(code=1) # pragma: no cover
-198
View File
@@ -1,198 +0,0 @@
"""Format command for basic-memory CLI."""
from pathlib import Path
from typing import Annotated, Optional
import typer
from loguru import logger
from rich.console import Console
from rich.progress import Progress, SpinnerColumn, TextColumn
from basic_memory.cli.app import app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.config import ConfigManager, get_project_config
from basic_memory.file_utils import format_file
console = Console()
def is_markdown_extension(path: Path) -> bool:
"""Check if file has a markdown extension."""
return path.suffix.lower() in (".md", ".markdown")
async def format_single_file(file_path: Path, app_config) -> tuple[Path, bool, Optional[str]]:
"""Format a single file.
Returns:
Tuple of (path, success, error_message)
"""
try:
result = await format_file(
file_path, app_config, is_markdown=is_markdown_extension(file_path)
)
if result is not None:
return (file_path, True, None)
else:
return (file_path, False, "No formatter configured or formatting skipped")
except Exception as e:
return (file_path, False, str(e))
async def format_files(
paths: list[Path], app_config, show_progress: bool = True
) -> tuple[int, int, list[tuple[Path, str]]]:
"""Format multiple files.
Returns:
Tuple of (formatted_count, skipped_count, errors)
"""
formatted = 0
skipped = 0
errors: list[tuple[Path, str]] = []
if show_progress:
with Progress(
SpinnerColumn(),
TextColumn("[progress.description]{task.description}"),
console=console,
) as progress:
task = progress.add_task("Formatting files...", total=len(paths))
for file_path in paths:
path, success, error = await format_single_file(file_path, app_config)
if success:
formatted += 1
elif error and "No formatter configured" not in error:
errors.append((path, error))
else:
skipped += 1
progress.update(task, advance=1)
else:
for file_path in paths:
path, success, error = await format_single_file(file_path, app_config)
if success:
formatted += 1
elif error and "No formatter configured" not in error:
errors.append((path, error))
else:
skipped += 1
return formatted, skipped, errors
async def run_format(
path: Optional[Path] = None,
project: Optional[str] = None,
) -> None:
"""Run the format command."""
app_config = ConfigManager().config
# Check if formatting is enabled
if (
not app_config.format_on_save
and not app_config.formatter_command
and not app_config.formatters
):
console.print(
"[yellow]No formatters configured. Set format_on_save=true and "
"formatter_command or formatters in your config.[/yellow]"
)
console.print(
"\nExample config (~/.basic-memory/config.json):\n"
' "format_on_save": true,\n'
' "formatter_command": "prettier --write {file}"\n'
)
raise typer.Exit(1)
# Temporarily enable format_on_save for this command
# (so format_file actually runs the formatter)
original_format_on_save = app_config.format_on_save
app_config.format_on_save = True
try:
# Determine which files to format
if path:
# Format specific file or directory
if path.is_file():
files = [path]
elif path.is_dir():
# Find all markdown and json files
files = (
list(path.rglob("*.md"))
+ list(path.rglob("*.json"))
+ list(path.rglob("*.canvas"))
)
else:
console.print(f"[red]Path not found: {path}[/red]")
raise typer.Exit(1)
else:
# Format all files in project
project_config = get_project_config(project)
project_path = Path(project_config.home)
if not project_path.exists():
console.print(f"[red]Project path not found: {project_path}[/red]")
raise typer.Exit(1)
# Find all markdown and json files
files = (
list(project_path.rglob("*.md"))
+ list(project_path.rglob("*.json"))
+ list(project_path.rglob("*.canvas"))
)
if not files:
console.print("[yellow]No files found to format.[/yellow]")
return
console.print(f"Found {len(files)} file(s) to format...")
formatted, skipped, errors = await format_files(files, app_config)
# Print summary
console.print()
if formatted > 0:
console.print(f"[green]Formatted: {formatted} file(s)[/green]")
if skipped > 0:
console.print(f"[dim]Skipped: {skipped} file(s) (no formatter for extension)[/dim]")
if errors:
console.print(f"[red]Errors: {len(errors)} file(s)[/red]")
for path, error in errors:
console.print(f" [red]{path}[/red]: {error}")
finally:
# Restore original setting
app_config.format_on_save = original_format_on_save
@app.command()
def format(
path: Annotated[
Optional[Path],
typer.Argument(help="File or directory to format. Defaults to current project."),
] = None,
project: Annotated[
Optional[str],
typer.Option("--project", "-p", help="Project name to format."),
] = None,
) -> None:
"""Format files using configured formatters.
Uses the formatter_command or formatters settings from your config.
By default, formats all .md, .json, and .canvas files in the current project.
Examples:
basic-memory format # Format all files in current project
basic-memory format --project research # Format files in specific project
basic-memory format notes/meeting.md # Format a specific file
basic-memory format notes/ # Format all files in directory
"""
try:
run_with_cleanup(run_format(path, project))
except Exception as e:
if not isinstance(e, typer.Exit):
logger.error(f"Error formatting files: {e}")
console.print(f"[red]Error formatting files: {e}[/red]")
raise typer.Exit(code=1)
raise

Some files were not shown because too many files have changed in this diff Show More