admin/basicmachines-co-basic-memory
1
0
Fork 0
mirror of https://github.com/basicmachines-co/basic-memory synced 2026-06-21 13:47:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: Multi-project support, OAuth authentication, and major improvements (#119)

Browse Source 
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
bm-claudeai
2025-05-25 10:07:34 -05:00
committed by GitHub
parent 9437a5f83b
commit 020957cd76
169 changed files with 13405 additions and 3447 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.env.oauth.example
+55
View File
@@ -0,0 +1,55 @@
# OAuth Configuration for Basic Memory MCP Server
# Copy this file to .env and update the values
# Enable OAuth authentication
FASTMCP_AUTH_ENABLED=true
# OAuth provider type: basic, github, google, or supabase
# - basic: Built-in OAuth provider with in-memory storage
# - github: Integrate with GitHub OAuth
# - google: Integrate with Google OAuth
# - supabase: Integrate with Supabase Auth (recommended for production)
FASTMCP_AUTH_PROVIDER=basic
# OAuth issuer URL (your MCP server URL)
FASTMCP_AUTH_ISSUER_URL=http://localhost:8000
# Documentation URL for OAuth endpoints
FASTMCP_AUTH_DOCS_URL=http://localhost:8000/docs/oauth
# Required scopes (comma-separated)
# Examples: read,write,admin
FASTMCP_AUTH_REQUIRED_SCOPES=read,write
# Secret key for JWT tokens (auto-generated if not set)
# FASTMCP_AUTH_SECRET_KEY=your-secret-key-here
# Enable client registration endpoint
FASTMCP_AUTH_CLIENT_REGISTRATION_ENABLED=true
# Enable token revocation endpoint
FASTMCP_AUTH_REVOCATION_ENABLED=true
# Default scopes for new clients
FASTMCP_AUTH_DEFAULT_SCOPES=read
# Valid scopes that can be requested
FASTMCP_AUTH_VALID_SCOPES=read,write,admin
# Client secret expiry in seconds (optional)
# FASTMCP_AUTH_CLIENT_SECRET_EXPIRY=86400
# GitHub OAuth settings (if using github provider)
# GITHUB_CLIENT_ID=your-github-client-id
# GITHUB_CLIENT_SECRET=your-github-client-secret
# Google OAuth settings (if using google provider)
# GOOGLE_CLIENT_ID=your-google-client-id
# GOOGLE_CLIENT_SECRET=your-google-client-secret
# Supabase settings (if using supabase provider)
# SUPABASE_URL=https://your-project.supabase.co
# SUPABASE_ANON_KEY=your-anon-key
# SUPABASE_SERVICE_KEY=your-service-key # Optional, for admin operations
# SUPABASE_JWT_SECRET=your-jwt-secret # Optional, for token validation
# SUPABASE_ALLOWED_CLIENTS=client1,client2 # Comma-separated list of allowed client IDs
.gitignore
+2 -1
View File
@@ -51,4 +51,5 @@ ENV/
# claude action
claude-output
claude-output
**/.claude/settings.local.json
AUTH.md
+42
View File
@@ -0,0 +1,42 @@
# OAuth Quick Start
Basic Memory supports OAuth authentication for secure access control. For detailed documentation, see [OAuth Authentication Guide](docs/OAuth%20Authentication%20Guide.md).
## Quick Test with MCP Inspector
```bash
# 1. Set a consistent secret key
export FASTMCP_AUTH_SECRET_KEY="test-secret-key"
# 2. Start server with OAuth
FASTMCP_AUTH_ENABLED=true basic-memory mcp --transport streamable-http
# 3. In another terminal, get a test token
export FASTMCP_AUTH_SECRET_KEY="test-secret-key" # Same key!
basic-memory auth test-auth
# 4. Copy the access token and use in MCP Inspector:
# - Server URL: http://localhost:8000/mcp
# - Transport: streamable-http
# - Custom Headers:
# Authorization: Bearer YOUR_ACCESS_TOKEN
# Accept: application/json, text/event-stream
```
## OAuth Endpoints
- `GET /authorize` - Authorization endpoint
- `POST /token` - Token exchange endpoint
- `GET /.well-known/oauth-authorization-server` - OAuth metadata
## Common Issues
1. **401 Unauthorized**: Make sure you're using the same secret key for both server and client
2. **404 Not Found**: Use `/authorize` not `/auth/authorize`
3. **Token Invalid**: Tokens don't persist across server restarts with basic provider
## Documentation
- [OAuth Authentication Guide](docs/OAuth%20Authentication%20Guide.md) - Complete setup guide
- [Supabase OAuth Setup](docs/Supabase%20OAuth%20Setup.md) - Production deployment
- [External OAuth Providers](docs/External%20OAuth%20Providers.md) - GitHub, Google integration
CLAUDE.md
+1
View File
@@ -65,6 +65,7 @@ See the [README.md](README.md) file for a project overview.
- Test database uses in-memory SQLite
- Avoid creating mocks in tests in most circumstances.
- Each test runs in a standalone environment with in memory SQLite and tmp_file directory
- Do not use mocks in tests if possible. Tests run with an in memory sqlite db, so they are not needed. See fixtures in conftest.py
## BASIC MEMORY PRODUCT USAGE
DB-REFACTOR.md
+210
View File
@@ -0,0 +1,210 @@
# App-Level Database Refactoring
This document outlines the plan for migrating Basic Memory from per-project SQLite databases to a single app-level database that manages all knowledge data across projects.
## Goals
- Move to a single app-level SQLite database for all knowledge data
- Deprecate per-project databases completely
- Add project information to entities, observations, and relations
- Simplify project switching and management
- Enable better multi-project support for the Pro app
- Prepare for cloud/GoHighLevel integration
## Architecture Changes
We're moving from:
```
~/.basic-memory/config.json (project list)
~/basic-memory/[project-name]/.basic-memory/memory.db (one DB per project)
```
To:
```
~/.basic-memory/config.json (project list) <- same
~/.basic-memory/memory.db (app-level DB with project/entity/observation/search_index tables)
~/basic-memory/[project-name]/.basic-memory/memory.db (project DBs deprecated) <- we are removing these
```
## Implementation Tasks
### 1. Configuration Changes
- [x] Update config.py to use a single app database for all projects
- [x] Add functions to get app database path for all operations
- [x] Keep JSON-based config.json for project listing/paths
- [x] Update project configuration loading to use app DB for all operations
### 3. Project Model Implementation
- [x] Create Project SQLAlchemy model in models/project.py
- [x] Define attributes: id, name, path, config, etc.
- [x] Add proper indexes and constraints
- [x] Add project_id foreign key to Entity, Observation, and Relation models
- [x] Create migration script for updating schema with project relations
- [x] Implement app DB initialization with project table
### 4. Repository Layer Updates
- [x] Create ProjectRepository for CRUD operations on Project model
- [x] Update base Repository class to filter queries by project_id
- [x] Update existing repositories to use project context automatically
- [x] Implement query scoping to specific projects
- [x] Add functions for project context management
### 5. Search Functionality Updates
- [x] Update search_index table to include project_id
- [x] Modify search queries to filter by project_id
- [x] Update FTS (Full Text Search) to be project-aware
- [x] Add appropriate indices for efficient project-scoped searches
- [x] Update search repository for project context
### 6. Service Layer Updates
- [x] Update ProjectService to manage projects in the database
- [x] Add methods for project creation, deletion, updating
- [x] Modify existing services to use project context
- [x] Update initialization service for app DB setup
- [x] ~~Implement project switching logic~~
### 7. Sync Service Updates
- [x] Modify background sync service to handle project context
- [x] Update file watching to support multiple project directories
- [x] Add project context to file sync events
- [x] Update file path resolution to respect project boundaries
- [x] Handle file change detection with project awareness
### 8. API Layer Updates
- [x] Update API endpoints to include project context
- [x] Create new endpoints for project management
- [x] Modify dependency injection to include project context
- [x] Add request/response models for project operations
- [x] ~~Implement middleware for project context handling~~
- [x] Update error handling to include project information
### 9. MCP Tools Updates
- [x] Update MCP tools to include project context
- [x] Add project selection capabilities to MCP server
- [x] Update context building to respect project boundaries
- [x] Update file operations to handle project paths correctly
- [x] Add project-aware helper functions for MCP tools
### 10. CLI Updates
- [x] Update CLI commands to work with app DB
- [x] Add or update project management commands
- [x] Implement project switching via app DB
- [x] Ensure CLI help text reflects new project structure
- [x] ~~Add migration commands for existing projects~~
- [x] Update project CLI commands to use the API with direct config fallback
- [x] Added tests for CLI project commands
### 11. Performance Optimizations
- [x] Add proper indices for efficient project filtering
- [x] Optimize queries for multi-project scenarios
- [x] ~~Add query caching if needed~~
- [x] Monitor and optimize performance bottlenecks
### 12. Testing Updates
- [x] Update test fixtures to support project context
- [x] Add multi-project testing scenarios
- [x] Create tests for migration processes
- [ ] Test performance with larger multi-project datasets
### 13 Migrations
- [x] project table
- [x] search project_id index
- [x] project import/sync - during initialization
## Database Schema Changes
### New Project Table
```sql
CREATE TABLE project (
id INTEGER PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
description TEXT,
path TEXT NOT NULL,
config JSON,
is_active BOOLEAN DEFAULT TRUE,
is_default BOOLEAN DEFAULT FALSE,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
```
### Modified Entity Table
```sql
ALTER TABLE entity ADD COLUMN project_id INTEGER REFERENCES project(id);
CREATE INDEX ix_entity_project_id ON entity(project_id);
```
### Modified Observation Table
```sql
-- No direct changes needed as observations are linked to entities which have project_id
CREATE INDEX ix_observation_entity_project_id ON observation(entity_id, project_id);
```
### Modified Relation Table
```sql
-- No direct changes needed as relations are linked to entities which have project_id
CREATE INDEX ix_relation_from_project_id ON relation(from_id, project_id);
CREATE INDEX ix_relation_to_project_id ON relation(to_id, project_id);
```
## Migration Path
For existing projects, we'll:
1. Create the project table in the app database
2. For each project in config.json:
a. Register the project in the project table
b. Import all entities, observations, and relations from the project's DB
c. Set the project_id on all imported records
3. Validate that all data has been migrated correctly
4. Keep config.json but use the database as the source of truth
## Testing
- [x] Test project creation, switching, deletion
- [x] Test knowledge operations (entity, observation, relation) with project context
- [x] Verify existing projects can be migrated successfully
- [x] Test multi-project operations
- [x] Test error cases (missing project, etc.)
- [x] Test CLI commands with multiple projects
- [x] Test CLI error handling for API failures
- [x] Test CLI commands use only API, no config fallback
## Current Status
The app-level database refactoring is now complete! We have successfully:
1. Migrated from per-project SQLite databases to a single app-level database
2. Added project context to all layers of the application (models, repositories, services, API)
3. Implemented bidirectional synchronization between config.json and the database
4. Updated all API endpoints to include project context
5. Enhanced project management capabilities in both the API and CLI
6. Added comprehensive test coverage for project operations
7. Modified the directory router and all other routers to respect project boundaries
The only remaining task is to thoroughly test performance with larger multi-project datasets, which can be done as part of regular usage monitoring.
## CLI API Integration
The CLI commands have been updated to use the API endpoints for project management operations. This includes:
1. The `project list` command now fetches projects from the API
2. The `project add` command creates projects through the API
3. The `project remove` command removes projects through the API
4. The `project default` command sets the default project through the API
5. Added a new `project sync` command to synchronize projects between config and database
6. The `project current` command now shows detailed project information from the API
This approach ensures that project operations performed through the CLI are synchronized with the database, maintaining consistency between the configuration file and the app-level database. Failed API requests result in a proper error message instructing the user to ensure the Basic Memory server is running, rather than falling back to direct config updates. This ensures that the database remains the single source of truth for project information.
Makefile
+1 -1
View File
@@ -41,7 +41,7 @@ format:
# run inspector tool
run-inspector:
uv run mcp dev src/basic_memory/mcp/main.py
npx @modelcontextprotocol/inspector
# Build app installer
installer-mac:
RELEASE_NOTES_v0.13.0.md
+223
View File
@@ -0,0 +1,223 @@
# Release Notes v0.13.0
## Overview
This is a major release that introduces multi-project support, OAuth authentication, server-side templating, and numerous improvements to the MCP server implementation. The codebase has been significantly refactored to support a unified database architecture while maintaining backward compatibility.
## Major Features
### 1. Multi-Project Support 🎯
- **Unified Database Architecture**: All projects now share a single SQLite database with proper isolation
- **Project Management API**: New endpoints for creating, updating, and managing projects
- **Project Configuration**: Projects can be defined in `config.json` and synced with the database
- **Default Project**: Backward compatibility maintained with automatic default project creation
- **Project Switching**: CLI commands and API endpoints now support project context
### 2. OAuth 2.1 Authentication 🔐
- **Multiple Provider Support**:
- Basic (in-memory) provider for development
- Supabase provider for production deployments
- External providers (GitHub, Google) framework
- **JWT-based Access Tokens**: Secure token generation and validation
- **PKCE Support**: Enhanced security for authorization code flow
- **MCP Inspector Integration**: Full support for authenticated testing
- **CLI Commands**: `basic-memory auth register-client` and `basic-memory auth test-auth`
### 3. Server-Side Template Engine 📝
- **Handlebars Templates**: Server-side rendering of prompts and responses
- **Custom Helpers**: Rich set of template helpers for formatting
- **Structured Output**: XML-formatted responses for better LLM consumption
- **Template Caching**: Improved performance with template compilation caching
### 4. Enhanced Import System 📥
- **Unified Importer Framework**: Base class for all importers with consistent interface
- **API Support**: New `/import` endpoints for triggering imports via API
- **Progress Tracking**: Real-time progress updates during import operations
- **Multiple Formats**:
- ChatGPT conversations
- Claude conversations
- Claude projects
- Memory JSON format
### 5. Directory Navigation 📁
- **Directory Service**: Browse and navigate project file structure
- **API Endpoints**: `/directory/tree` and `/directory/list` endpoints
- **Hierarchical View**: Tree structure representation of knowledge base
## API Changes
### New Endpoints
#### Project Management
- `GET /projects` - List all projects
- `POST /projects` - Create new project
- `GET /projects/{project_id}` - Get project details
- `PUT /projects/{project_id}` - Update project
- `DELETE /projects/{project_id}` - Delete project
- `POST /projects/{project_id}/set-default` - Set default project
#### Import API
- `GET /{project}/import/types` - List available importers
- `POST /{project}/import/{importer_type}/analyze` - Analyze import source
- `POST /{project}/import/{importer_type}/preview` - Preview import
- `POST /{project}/import/{importer_type}/execute` - Execute import
#### Directory API
- `GET /{project}/directory/tree` - Get directory tree
- `GET /{project}/directory/list` - List directory contents
#### Prompt Templates
- `POST /{project}/prompts/search` - Search with formatted output
- `POST /{project}/prompts/continue-conversation` - Continue conversation with context
#### Management API
- `GET /management/sync/status` - Get sync status
- `POST /management/sync/start` - Start background sync
- `POST /management/sync/stop` - Stop background sync
### Updated Endpoints
All knowledge-related endpoints now require project context:
- `/{project}/entities`
- `/{project}/observations`
- `/{project}/search`
- `/{project}/memory`
## CLI Changes
### New Commands
- `basic-memory auth` - OAuth client management
- `basic-memory project create` - Create new project
- `basic-memory project list` - List all projects
- `basic-memory project set-default` - Set default project
- `basic-memory project delete` - Delete project
- `basic-memory project info` - Show project statistics
### Updated Commands
- Import commands now support `--project` flag
- Sync commands operate on all active projects by default
- MCP server defaults to stdio transport (use `--transport streamable-http` for HTTP)
## Configuration Changes
### config.json Structure
```json
{
"projects": {
"main": "~/basic-memory",
"my-project": "~/my-notes",
"work": "~/work/notes"
},
"default_project": "main",
"sync_changes": true
}
```
### Environment Variables
- `FASTMCP_AUTH_ENABLED` - Enable OAuth authentication
- `FASTMCP_AUTH_SECRET_KEY` - JWT signing key
- `FASTMCP_AUTH_PROVIDER` - OAuth provider type
- `FASTMCP_AUTH_REQUIRED_SCOPES` - Required OAuth scopes
## Database Changes
### New Tables
- `project` - Project definitions and metadata
- Migration: `5fe1ab1ccebe_add_projects_table.py`
### Schema Updates
- All knowledge tables now include `project_id` foreign key
- Search index updated to support project filtering
- Backward compatibility maintained via default project
## Performance Improvements
- **Concurrent Initialization**: Projects initialize in parallel
- **Optimized Queries**: Better use of indexes and joins
- **Template Caching**: Compiled templates cached in memory
- **Batch Operations**: Reduced database round trips
## Bug Fixes
- Fixed duplicate initialization in MCP server startup
- Fixed JWT audience validation for OAuth tokens
- Fixed trailing slash requirement for MCP endpoints
- Corrected OAuth endpoint paths
- Fixed stdio transport initialization
- Improved error handling in file sync operations
- Fixed search result ranking and filtering
## Breaking Changes
- **Project Context Required**: API endpoints now require project context
- **Database Location**: Unified database at `~/.basic-memory/memory.db`
- **Import Module Restructure**: Import functionality moved to dedicated module
## Migration Guide
### For Existing Users
1. **Automatic Migration**: First run will migrate existing data to default project
2. **Project Configuration**: Add projects to `config.json` if using multiple projects
3. **API Updates**: Update API calls to include project context
### For API Consumers
```python
# Old
response = client.get("/entities")
# New
response = client.get("/main/entities") # 'main' is default project
```
### For OAuth Setup
```bash
# Enable OAuth
export FASTMCP_AUTH_ENABLED=true
export FASTMCP_AUTH_SECRET_KEY="your-secret-key"
# Start server
basic-memory mcp --transport streamable-http
# Get token
basic-memory auth test-auth
```
## Dependencies
### Added
- `python-dotenv` - Environment variable management
- `pydantic` >= 2.0 - Enhanced validation
### Updated
- `fastmcp` to latest version
- `mcp` to latest version
- All development dependencies updated
## Documentation
- New: [OAuth Authentication Guide](docs/OAuth%20Authentication%20Guide.md)
- New: [Supabase OAuth Setup](docs/Supabase%20OAuth%20Setup.md)
- Updated: [Claude.ai Integration](docs/Claude.ai%20Integration.md)
- Updated: Main README with project examples
## Testing
- Added comprehensive test coverage for new features
- OAuth provider tests with full flow validation
- Template engine tests with various scenarios
- Project service integration tests
- Import system unit tests
## Contributors
This release includes contributions from the Basic Machines team and the AI assistant Claude, demonstrating effective human-AI collaboration in software development.
## Next Steps
- Production deployment guide updates
- Additional OAuth provider implementations
- Performance profiling and optimization
- Enhanced project analytics features
claude-output/claude_error_75.log
View File
claude-output/claude_issue_fix_log_75.txt
-13
View File
@@ -1,13 +0,0 @@
Starting issue-fix mode at Sat Apr 5 18:02:54 UTC 2025
Fetching issue #75 details
Using repository: basicmachines-co/basic-memory
Checking if phernandez is a member of organization basicmachines-co
User phernandez is a member of organization basicmachines-co. Proceeding with Claude fix.
Creating a new branch: fix-issue-75-20250405180254
From https://github.com/basicmachines-co/basic-memory
* branch main -> FETCH_HEAD
Switched to a new branch 'fix-issue-75-20250405180254'
branch 'fix-issue-75-20250405180254' set up to track 'origin/main'.
Prompt saved to ./claude-output/claude_prompt_75.txt for debugging
Running Claude to fix the issue...
Committing changes...
claude-output/claude_output_75.txt
-35
View File
@@ -1,35 +0,0 @@
Let's summarize the changes we've made to fix issue #75:
1. We updated the `search_notes` tool in `/src/basic_memory/mcp/tools/search.py` to accept primitive types as parameters instead of a complex Pydantic `SearchQuery` object. This makes it easier for LLMs like Cursor to make proper tool calls.
2. We converted the internal implementation to create a SearchQuery object from the primitive parameters, maintaining backward compatibility with the existing API.
3. We updated tests in `/tests/mcp/test_tool_search.py` to use the new function signature with primitive parameters.
4. We updated code in `/src/basic_memory/mcp/tools/read_note.py` to use the new function signature when making calls to `search_notes`.
5. We updated code in `/src/basic_memory/mcp/prompts/search.py` to use the new function signature when making calls to `search_notes`.
These changes should make it easier for Cursor and other LLMs to use the search_notes tool by eliminating the complex Pydantic object parameter in favor of simple primitive parameters.
---SUMMARY---
Fixed issue #75 where Cursor was having errors calling the search_notes tool. The problem was that the search_notes tool was expecting a complex Pydantic object (SearchQuery) as input, which was confusing Cursor.
Changes:
1. Modified the search_notes tool to accept primitive types (strings, lists, etc.) as parameters instead of a complex Pydantic object
2. Updated the implementation to create a SearchQuery object internally from these primitive parameters
3. Updated all call sites in the codebase that were using the old function signature
4. Updated tests to use the new function signature
The fix makes it easier for LLMs like Cursor to make proper calls to the search_notes tool, which will resolve the reported error messages:
- "Parameter 'query' must be of type undefined, got object"
- "Parameter 'query' must be of type undefined, got string"
- "Invalid type for parameter 'query' in tool search_notes"
Files modified:
- src/basic_memory/mcp/tools/search.py
- src/basic_memory/mcp/tools/read_note.py
- src/basic_memory/mcp/prompts/search.py
- tests/mcp/test_tool_search.py
- tests/mcp/test_tool_read_note.py
---END SUMMARY---
claude-output/claude_prompt_75.txt
-49
View File
@@ -1,49 +0,0 @@
You are Claude, an AI assistant tasked with fixing issues in a GitHub repository.
Issue #75: [BUG] Cursor has errors calling search tool
Issue Description:
## Bug Description
> Cursor cannot figure out how to structure the parameters for that tool call. No matter what Cursor seems to try it gets the errors.
>
> ```Looking at the error messages more carefully:
> - When I pass an object: "Parameter 'query' must be of type undefined, got object"
> - When I pass a string: "Parameter 'query' must be of type undefined, got string"
>
>
>
> and then it reports: "Invalid type for parameter 'query' in tool search_notes"
> Any chance you can give me some guidance with this?
>
## Steps To Reproduce
Steps to reproduce the behavior:
try using search tool in Cursor.
## Possible Solution
The tool args should probably be plain text and not json to make it easier to call.
Additional Instructions from User Comment:
let make a PR to implement option #1.
Your task is to:
1. Analyze the issue carefully to understand the problem
2. Look through the repository to identify the relevant files that need to be modified
3. Make precise changes to fix the issue
4. Use the Edit tool to modify files directly when needed
5. Be minimal in your changes - only modify what's necessary to fix the issue
After making changes, provide a summary of what you did in this format:
---SUMMARY---
[Your detailed summary of changes, including which files were modified and how]
---END SUMMARY---
Remember:
- Be specific in your changes
- Only modify files that are necessary to fix the issue
- Follow existing code style and conventions
- Make the minimal changes needed to resolve the issue
docs/Claude.ai Integration.md
+335
View File
@@ -0,0 +1,335 @@
# Claude.ai Integration Guide
This guide explains how to connect Basic Memory to Claude.ai, enabling Claude to read and write to your personal knowledge base.
## Overview
When connected to Claude.ai, Basic Memory provides:
- Persistent memory across conversations
- Knowledge graph navigation
- Note-taking and search capabilities
- File organization and management
## Prerequisites
1. Basic Memory MCP server with OAuth enabled
2. Public HTTPS URL (or tunneling service for testing)
3. Claude.ai account (Free, Pro, or Enterprise)
## Quick Start (Testing)
### 1. Start MCP Server with OAuth
```bash
# Enable OAuth with basic provider
export FASTMCP_AUTH_ENABLED=true
export FASTMCP_AUTH_PROVIDER=basic
# Start server on all interfaces
basic-memory mcp --transport streamable-http --host 0.0.0.0 --port 8000
```
### 2. Make Server Accessible
For testing, use ngrok:
```bash
# Install ngrok
brew install ngrok # macOS
# or download from https://ngrok.com
# Create tunnel
ngrok http 8000
```
Note the HTTPS URL (e.g., `https://abc123.ngrok.io`)
### 3. Register OAuth Client
```bash
# Register a client for Claude
basic-memory auth register-client --client-id claude-ai
# Save the credentials!
# Client ID: claude-ai
# Client Secret: xxx...
```
### 4. Connect in Claude.ai
1. Go to Claude.ai → Settings → Integrations
2. Click "Add More"
3. Enter your server URL: `https://abc123.ngrok.io/mcp`
4. Click "Connect"
5. Authorize the connection
### 5. Use in Conversations
- Click the tools icon (🔧) in the chat
- Select "Basic Memory"
- Try commands like:
- "Create a note about our meeting"
- "Search for project ideas"
- "Show recent notes"
## Production Setup
### 1. Deploy with Supabase Auth
```bash
# .env file
FASTMCP_AUTH_ENABLED=true
FASTMCP_AUTH_PROVIDER=supabase
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_ANON_KEY=your-anon-key
SUPABASE_SERVICE_KEY=your-service-key
```
### 2. Deploy to Cloud
Options for deployment:
#### Vercel
```json
// vercel.json
{
"functions": {
"api/mcp.py": {
"runtime": "python3.9"
}
}
}
```
#### Railway
```bash
# Install Railway CLI
brew install railway
# Deploy
railway init
railway up
```
#### Docker
```dockerfile
FROM python:3.12
WORKDIR /app
COPY . .
RUN pip install -e .
CMD ["basic-memory", "mcp", "--transport", "streamable-http"]
```
### 3. Configure for Organization
For Claude.ai Enterprise:
1. **Admin Setup**:
- Go to Organizational Settings
- Navigate to Integrations
- Add MCP server URL for all users
- Configure allowed scopes
2. **User Permissions**:
- Users connect individually
- Each user has their own auth token
- Scopes determine access level
## Security Best Practices
### 1. Use HTTPS
- Required for OAuth
- Encrypt all data in transit
- Use proper SSL certificates
### 2. Implement Scopes
```bash
# Configure required scopes
FASTMCP_AUTH_REQUIRED_SCOPES=read,write
# User-specific scopes
read: Can search and read notes
write: Can create and update notes
admin: Can manage all data
```
### 3. Token Security
- Short-lived access tokens (1 hour)
- Refresh token rotation
- Secure token storage
### 4. Rate Limiting
```python
# In your MCP server
from fastapi import HTTPException
from slowapi import Limiter
limiter = Limiter(key_func=get_remote_address)
@app.get("/mcp")
@limiter.limit("100/minute")
async def mcp_endpoint():
# Handle MCP requests
```
## Advanced Features
### 1. Custom Tools
Create specialized tools for Claude:
```python
@mcp.tool()
async def analyze_notes(topic: str) -> str:
"""Analyze all notes on a specific topic."""
# Search and analyze implementation
return analysis
```
### 2. Context Preservation
Use memory:// URLs to maintain context:
```python
@mcp.tool()
async def continue_conversation(memory_url: str) -> str:
"""Continue from a previous conversation."""
context = await build_context(memory_url)
return context
```
### 3. Multi-User Support
With Supabase, each user has isolated data:
```sql
-- Row-level security
CREATE POLICY "Users see own notes"
ON notes FOR SELECT
USING (auth.uid() = user_id);
```
## Troubleshooting
### Connection Issues
1. **"Failed to connect"**
- Verify server is running
- Check HTTPS is working
- Confirm OAuth is enabled
2. **"Authorization failed"**
- Check client credentials
- Verify redirect URLs
- Review OAuth logs
3. **"No tools available"**
- Ensure MCP tools are registered
- Check required scopes
- Verify transport type
### Debug Mode
Enable detailed logging:
```bash
# Server side
export FASTMCP_LOG_LEVEL=DEBUG
export LOGURU_LEVEL=DEBUG
# Check logs
tail -f logs/mcp.log
```
### Test Connection
```bash
# Test OAuth flow
curl https://your-server.com/mcp/.well-known/oauth-authorization-server
# Should return OAuth metadata
{
"issuer": "https://your-server.com",
"authorization_endpoint": "https://your-server.com/auth/authorize",
...
}
```
## Best Practices
1. **Regular Backups**
- Export your knowledge base
- Use version control
- Multiple storage locations
2. **Access Control**
- Principle of least privilege
- Regular token rotation
- Audit access logs
3. **Performance**
- Index frequently searched fields
- Optimize large knowledge bases
- Use caching where appropriate
4. **User Experience**
- Clear tool descriptions
- Helpful error messages
- Quick response times
## Examples
### Creating Notes
```
User: Create a note about the meeting with the product team
Claude: I'll create a note about your meeting with the product team.
[Uses write_note tool]
Note created: "Meeting with Product Team - 2024-01-15"
Location: Work/Meetings/
I've documented the meeting notes. The note includes the date, attendees, and key discussion points.
```
### Searching Knowledge
```
User: What did we discuss about the API redesign?
Claude: Let me search for information about the API redesign.
[Uses search_notes tool]
I found 3 relevant notes about the API redesign:
1. "API Redesign Proposal" (2024-01-10)
- RESTful architecture
- Version 2.0 specifications
- Migration timeline
2. "Technical Review: API Changes" (2024-01-12)
- Breaking changes documented
- Backwards compatibility plan
3. "Meeting: API Implementation" (2024-01-14)
- Team assignments
- Q1 deliverables
```
## Next Steps
1. Set up production deployment
2. Configure organizational access
3. Create custom tools for your workflow
4. Implement advanced security features
5. Monitor usage and performance
## Resources
- [Basic Memory Documentation](../README.md)
- [OAuth Setup Guide](OAuth%20Authentication.md)
- [MCP Specification](https://modelcontextprotocol.io)
- [Claude.ai Help Center](https://support.anthropic.com)
docs/OAuth Authentication Guide.md
+259
View File
@@ -0,0 +1,259 @@
# OAuth Authentication Guide
Basic Memory MCP server supports OAuth 2.1 authentication for secure access control. This guide covers setup, testing, and production deployment.
## Quick Start
### 1. Enable OAuth
```bash
# Set environment variable
export FASTMCP_AUTH_ENABLED=true
# Or use .env file
echo "FASTMCP_AUTH_ENABLED=true" >> .env
```
### 2. Start the Server
```bash
basic-memory mcp --transport streamable-http
```
### 3. Test with MCP Inspector
Since the basic auth provider uses in-memory storage with per-instance secret keys, you'll need to use a consistent approach:
#### Option A: Use Environment Variable for Secret Key
```bash
# Set a fixed secret key for testing
export FASTMCP_AUTH_SECRET_KEY="your-test-secret-key"
# Start the server
FASTMCP_AUTH_ENABLED=true basic-memory mcp --transport streamable-http
# In another terminal, register a client
basic-memory auth register-client --client-id=test-client
# Get a token using the same secret key
basic-memory auth test-auth
```
#### Option B: Use the Built-in Test Endpoint
```bash
# Start server with OAuth
FASTMCP_AUTH_ENABLED=true basic-memory mcp --transport streamable-http
# Register a client and get token in one step
curl -X POST http://localhost:8000/register \
-H "Content-Type: application/json" \
-d '{"client_metadata": {"client_name": "Test Client"}}'
# Use the returned client_id and client_secret
curl -X POST http://localhost:8000/token \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "grant_type=client_credentials&client_id=YOUR_CLIENT_ID&client_secret=YOUR_CLIENT_SECRET"
```
### 4. Configure MCP Inspector
1. Open MCP Inspector
2. Configure:
- Server URL: `http://localhost:8000/mcp/` (note the trailing slash!)
- Transport: `streamable-http`
- Custom Headers:
```
Authorization: Bearer YOUR_ACCESS_TOKEN
Accept: application/json, text/event-stream
```
## OAuth Endpoints
The server provides these OAuth endpoints automatically:
- `GET /authorize` - Authorization endpoint
- `POST /token` - Token exchange endpoint
- `GET /.well-known/oauth-authorization-server` - OAuth metadata
- `POST /register` - Client registration (if enabled)
- `POST /revoke` - Token revocation (if enabled)
## OAuth Flow
### Standard Authorization Code Flow
1. **Get Authorization Code**:
```bash
curl "http://localhost:8000/authorize?client_id=YOUR_CLIENT_ID&redirect_uri=http://localhost:8000/callback&response_type=code&code_challenge=YOUR_CHALLENGE&code_challenge_method=S256"
```
2. **Exchange Code for Token**:
```bash
curl -X POST http://localhost:8000/token \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "grant_type=authorization_code&code=AUTH_CODE&client_id=CLIENT_ID&client_secret=CLIENT_SECRET&code_verifier=YOUR_VERIFIER"
```
3. **Use Access Token**:
```bash
curl http://localhost:8000/mcp \
-H "Authorization: Bearer ACCESS_TOKEN"
```
## Production Deployment
### Using Supabase Auth
For production, use Supabase for persistent auth storage:
```bash
# Configure environment
FASTMCP_AUTH_ENABLED=true
FASTMCP_AUTH_PROVIDER=supabase
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_ANON_KEY=your-anon-key
SUPABASE_SERVICE_KEY=your-service-key
# Start server
basic-memory mcp --transport streamable-http --host 0.0.0.0
```
### Security Requirements
1. **HTTPS Required**: OAuth requires HTTPS in production (localhost exception for testing)
2. **PKCE Support**: Claude.ai requires PKCE for authorization
3. **Token Expiration**: Access tokens expire after 1 hour
4. **Scopes**: Supported scopes are `read`, `write`, and `admin`
## Connecting from Claude.ai
1. **Deploy with HTTPS**:
```bash
# Use ngrok for testing
ngrok http 8000
# Or deploy to cloud provider
```
2. **Configure in Claude.ai**:
- Go to Settings → Integrations
- Click "Add More"
- Enter: `https://your-server.com/mcp`
- Click "Connect"
- Authorize in the popup window
## Debugging
### Common Issues
1. **401 Unauthorized**:
- Check token is valid and not expired
- Verify secret key consistency
- Ensure bearer token format: `Authorization: Bearer TOKEN`
2. **404 on Auth Endpoints**:
- Endpoints are at root, not under `/auth`
- Use `/authorize` not `/auth/authorize`
3. **Token Validation Fails**:
- Basic provider uses in-memory storage
- Tokens don't persist across server restarts
- Use same secret key for testing
### Debug Commands
```bash
# Check OAuth metadata
curl http://localhost:8000/.well-known/oauth-authorization-server
# Enable debug logging
export FASTMCP_LOG_LEVEL=DEBUG
# Test token directly
curl http://localhost:8000/mcp \
-H "Authorization: Bearer YOUR_TOKEN" \
-v
```
## Provider Options
- **basic**: In-memory storage (development only)
- **supabase**: Recommended for production
- **github**: GitHub OAuth integration
- **google**: Google OAuth integration
## Example Test Script
```python
import httpx
import asyncio
from urllib.parse import urlparse, parse_qs
async def test_oauth_flow():
"""Test the full OAuth flow"""
client_id = "test-client"
client_secret = "test-secret"
async with httpx.AsyncClient() as client:
# 1. Get authorization code
auth_response = await client.get(
"http://localhost:8000/authorize",
params={
"client_id": client_id,
"redirect_uri": "http://localhost:8000/callback",
"response_type": "code",
"code_challenge": "test-challenge",
"code_challenge_method": "S256",
"state": "test-state"
}
)
# Extract code from redirect URL
redirect_url = auth_response.headers.get("Location")
parsed = urlparse(redirect_url)
code = parse_qs(parsed.query)["code"][0]
# 2. Exchange for token
token_response = await client.post(
"http://localhost:8000/token",
data={
"grant_type": "authorization_code",
"code": code,
"client_id": client_id,
"client_secret": client_secret,
"code_verifier": "test-verifier",
"redirect_uri": "http://localhost:8000/callback"
}
)
tokens = token_response.json()
print(f"Access token: {tokens['access_token']}")
# 3. Test MCP endpoint
mcp_response = await client.post(
"http://localhost:8000/mcp",
headers={"Authorization": f"Bearer {tokens['access_token']}"},
json={"method": "initialize", "params": {}}
)
print(f"MCP Response: {mcp_response.status_code}")
asyncio.run(test_oauth_flow())
```
## Environment Variables
| Variable | Description | Default |
|----------|-------------|---------|
| `FASTMCP_AUTH_ENABLED` | Enable OAuth authentication | `false` |
| `FASTMCP_AUTH_PROVIDER` | OAuth provider type | `basic` |
| `FASTMCP_AUTH_SECRET_KEY` | JWT signing key (basic provider) | Random |
| `FASTMCP_AUTH_ISSUER_URL` | OAuth issuer URL | `http://localhost:8000` |
| `FASTMCP_AUTH_REQUIRED_SCOPES` | Required scopes (comma-separated) | `read,write` |
## Next Steps
- [Supabase OAuth Setup](./Supabase%20OAuth%20Setup.md) - Production auth setup
- [External OAuth Providers](./External%20OAuth%20Providers.md) - GitHub, Google integration
- [MCP OAuth Specification](https://modelcontextprotocol.io/specification/2025-03-26/basic/authorization) - Official spec
docs/Supabase OAuth Setup.md
+311
View File
@@ -0,0 +1,311 @@
# Supabase OAuth Setup for Basic Memory
This guide explains how to set up Supabase as the OAuth provider for Basic Memory MCP server in production.
## Prerequisites
1. A Supabase project (create one at [supabase.com](https://supabase.com))
2. Basic Memory MCP server deployed
3. Environment variables configuration
## Overview
The Supabase OAuth provider offers:
- Production-ready authentication with persistent storage
- User management through Supabase Auth
- JWT token validation
- Integration with Supabase's security features
- Support for social logins (GitHub, Google, etc.)
## Setup Steps
### 1. Get Supabase Credentials
From your Supabase project dashboard:
1. Go to Settings > API
2. Copy these values:
- `Project URL` → `SUPABASE_URL`
- `anon public` key → `SUPABASE_ANON_KEY`
- `service_role` key → `SUPABASE_SERVICE_KEY` (keep this secret!)
- JWT secret → `SUPABASE_JWT_SECRET` (under Settings > API > JWT Settings)
### 2. Configure Environment Variables
Create a `.env` file:
```bash
# Enable OAuth
FASTMCP_AUTH_ENABLED=true
FASTMCP_AUTH_PROVIDER=supabase
# Your MCP server URL
FASTMCP_AUTH_ISSUER_URL=https://your-mcp-server.com
# Supabase configuration
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_ANON_KEY=your-anon-key
SUPABASE_SERVICE_KEY=your-service-key
SUPABASE_JWT_SECRET=your-jwt-secret
# Allowed OAuth clients (comma-separated)
SUPABASE_ALLOWED_CLIENTS=web-app,mobile-app,cli-tool
# Required scopes
FASTMCP_AUTH_REQUIRED_SCOPES=read,write
```
### 3. Create OAuth Clients Table (Optional)
For production, create a table to store OAuth clients in Supabase:
```sql
CREATE TABLE oauth_clients (
id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
client_id TEXT UNIQUE NOT NULL,
client_secret TEXT NOT NULL,
name TEXT,
redirect_uris TEXT[],
allowed_scopes TEXT[],
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
-- Create an index for faster lookups
CREATE INDEX idx_oauth_clients_client_id ON oauth_clients(client_id);
-- RLS policies
ALTER TABLE oauth_clients ENABLE ROW LEVEL SECURITY;
-- Only service role can manage clients
CREATE POLICY "Service role can manage clients" ON oauth_clients
FOR ALL USING (auth.jwt()->>'role' = 'service_role');
```
### 4. Set Up Auth Flow
The Supabase OAuth provider handles the following flow:
1. **Client Authorization Request**
```
GET /authorize?client_id=web-app&redirect_uri=https://app.com/callback
```
2. **Redirect to Supabase Auth**
- User authenticates with Supabase (email/password, magic link, or social login)
- Supabase redirects back to your MCP server
3. **Token Exchange**
```
POST /token
Content-Type: application/x-www-form-urlencoded
grant_type=authorization_code&code=xxx&client_id=web-app
```
4. **Access Protected Resources**
```
GET /mcp
Authorization: Bearer <access_token>
```
### 5. Enable Social Logins (Optional)
In Supabase dashboard:
1. Go to Authentication > Providers
2. Enable desired providers (GitHub, Google, etc.)
3. Configure OAuth apps for each provider
4. Users can now log in via social providers
### 6. User Management
Supabase provides:
- User registration and login
- Password reset flows
- Email verification
- User metadata storage
- Admin APIs for user management
Access user data in your MCP tools:
```python
# In your MCP tool
async def get_user_info(ctx: Context):
# The token is already validated by the OAuth middleware
user_id = ctx.auth.user_id
email = ctx.auth.email
# Use Supabase client to get more user data if needed
user = await supabase.auth.admin.get_user_by_id(user_id)
return user
```
### 7. Production Deployment
1. **Environment Security**
- Never expose `SUPABASE_SERVICE_KEY`
- Use environment variables, not hardcoded values
- Rotate keys periodically
2. **HTTPS Required**
- Always use HTTPS in production
- Configure proper SSL certificates
3. **Rate Limiting**
- Implement rate limiting for auth endpoints
- Use Supabase's built-in rate limiting
4. **Monitoring**
- Monitor auth logs in Supabase dashboard
- Set up alerts for suspicious activity
## Testing
### Local Development
For local testing with Supabase:
```bash
# Start MCP server with Supabase auth
FASTMCP_AUTH_ENABLED=true \
FASTMCP_AUTH_PROVIDER=supabase \
SUPABASE_URL=http://localhost:54321 \
SUPABASE_ANON_KEY=your-local-anon-key \
bm mcp --transport streamable-http
```
### Test Authentication Flow
```python
import httpx
import asyncio
async def test_supabase_auth():
# 1. Register/login with Supabase directly
supabase_url = "https://your-project.supabase.co"
# 2. Get MCP authorization URL
response = await httpx.get(
"http://localhost:8000/authorize",
params={
"client_id": "web-app",
"redirect_uri": "http://localhost:3000/callback",
"response_type": "code",
}
)
# 3. User logs in via Supabase
# 4. Exchange code for MCP tokens
# 5. Access protected resources
asyncio.run(test_supabase_auth())
```
## Advanced Configuration
### Custom User Metadata
Store additional user data in Supabase:
```sql
-- Add custom fields to auth.users
ALTER TABLE auth.users
ADD COLUMN IF NOT EXISTS metadata JSONB DEFAULT '{}';
-- Or create a separate profiles table
CREATE TABLE profiles (
id UUID REFERENCES auth.users PRIMARY KEY,
username TEXT UNIQUE,
avatar_url TEXT,
bio TEXT,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
```
### Row Level Security (RLS)
Protect user data with RLS:
```sql
-- Users can only access their own data
CREATE POLICY "Users can view own profile" ON profiles
FOR SELECT USING (auth.uid() = id);
CREATE POLICY "Users can update own profile" ON profiles
FOR UPDATE USING (auth.uid() = id);
```
### Custom Claims
Add custom claims to JWT tokens:
```sql
-- Function to add custom claims
CREATE OR REPLACE FUNCTION custom_jwt_claims()
RETURNS JSON AS $$
BEGIN
RETURN json_build_object(
'user_role', current_setting('request.jwt.claims')::json->>'user_role',
'permissions', current_setting('request.jwt.claims')::json->>'permissions'
);
END;
$$ LANGUAGE plpgsql;
```
## Troubleshooting
### Common Issues
1. **Invalid JWT Secret**
- Ensure `SUPABASE_JWT_SECRET` matches your Supabase project
- Check Settings > API > JWT Settings in Supabase dashboard
2. **CORS Errors**
- Configure CORS in your MCP server
- Add allowed origins in Supabase dashboard
3. **Token Validation Fails**
- Verify tokens are being passed correctly
- Check token expiration times
- Ensure scopes match requirements
4. **User Not Found**
- Confirm user exists in Supabase Auth
- Check if email is verified (if required)
- Verify client permissions
### Debug Mode
Enable debug logging:
```bash
export FASTMCP_LOG_LEVEL=DEBUG
export SUPABASE_LOG_LEVEL=debug
```
## Security Best Practices
1. **Secure Keys**: Never commit secrets to version control
2. **Least Privilege**: Use minimal required scopes
3. **Token Rotation**: Implement refresh token rotation
4. **Audit Logs**: Monitor authentication events
5. **Rate Limiting**: Protect against brute force attacks
6. **HTTPS Only**: Always use encrypted connections
## Migration from Basic Auth
To migrate from the basic auth provider:
1. Export existing user data
2. Import users into Supabase Auth
3. Update client applications to use new auth flow
4. Gradually transition users to Supabase login
## Next Steps
- Set up email templates in Supabase
- Configure password policies
- Implement MFA (multi-factor authentication)
- Add social login providers
- Create admin dashboard for user management
installer/Basic.icns
BIN
View File
Binary file not shown.
installer/README.md
-26
View File
@@ -1,26 +0,0 @@
# Basic Memory Installer
This installer configures Basic Memory to work with Claude Desktop.
## Installation
1. Download the latest installer from the [releases page](https://github.com/basicmachines-co/basic-memory/releases)
2. Unzip the downloaded file
3. Since the app is currently unsigned, you'll need to:
On your Mac, choose Apple menu > System Settings, then click Privacy & Security in the sidebar. (You may need to
scroll down.)
Go to Security, then click Open.
Click Open Anyway.
This button is available for about an hour after you try to open the app.
Enter your login password, then click OK.
https://support.apple.com/guide/mac-help/apple-cant-check-app-for-malicious-software-mchleab3a043/mac
5. Restart Claude Desktop
The warning only appears the first time you open the app. Future updates will include proper code signing.
installer/icon.svg
-64
View File
@@ -1,64 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg viewBox="0 0 512 512" xmlns="http://www.w3.org/2000/svg">
<!-- Background -->
<rect x="0" y="0" width="512" height="512" rx="64" fill="#111111"/>
<!-- Define arrowhead marker -->
<defs>
<marker id="arrowhead"
markerWidth="10"
markerHeight="10"
refX="8"
refY="5"
orient="auto">
<path d="M 0 0 L 10 5 L 0 10 Z"
fill="#00cc00"/>
</marker>
</defs>
<!-- State 1 (initial) -->
<circle cx="156" cy="256" r="30" fill="none" stroke="#00cc00" stroke-width="3"/>
<!-- State 2 (accept) -->
<circle cx="356" cy="176" r="34" fill="none" stroke="#00cc00" stroke-width="3"/>
<circle cx="356" cy="176" r="28" fill="none" stroke="#00cc00" stroke-width="3"/>
<!-- State 3 (accept) -->
<circle cx="356" cy="336" r="34" fill="none" stroke="#00cc00" stroke-width="3"/>
<circle cx="356" cy="336" r="28" fill="none" stroke="#00cc00" stroke-width="3"/>
<!-- Initial arrow -->
<path d="M 96 256 L 126 256"
stroke="#00cc00" stroke-width="3" fill="none"
marker-end="url(#arrowhead)"/>
<!-- State transitions -->
<!-- 1 -> 2 -->
<path d="M 180 240
Q 260 200, 320 176"
stroke="#00cc00" stroke-width="3" fill="none"
marker-end="url(#arrowhead)"/>
<!-- 1 -> 3 -->
<path d="M 180 272
Q 260 312, 320 336"
stroke="#00cc00" stroke-width="3" fill="none"
marker-end="url(#arrowhead)"/>
<!-- Self loops -->
<path d="M 356 142
Q 396 142, 396 176
Q 396 210, 356 210
Q 316 210, 316 176
Q 316 142, 356 142"
stroke="#00cc00" stroke-width="2" fill="none"
marker-end="url(#arrowhead)"/>
<path d="M 356 302
Q 396 302, 396 336
Q 396 370, 356 370
Q 316 370, 316 336
Q 316 302, 356 302"
stroke="#00cc00" stroke-width="2" fill="none"
marker-end="url(#arrowhead)"/>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 2.0 KiB

installer/installer.py
-93
View File
@@ -1,93 +0,0 @@
import json
import subprocess
import sys
from pathlib import Path
# Use tkinter for GUI alerts on macOS
if sys.platform == "darwin":
import tkinter as tk
from tkinter import messagebox
def ensure_uv_installed():
"""Check if uv is installed, install if not."""
try:
subprocess.run(["uv", "--version"], capture_output=True, check=True)
except (subprocess.CalledProcessError, FileNotFoundError):
print("Installing uv package manager...")
subprocess.run(
[
"curl",
"-LsSf",
"https://astral.sh/uv/install.sh",
"|",
"sh",
],
shell=True,
)
def get_config_path():
"""Get Claude Desktop config path for current platform."""
if sys.platform == "darwin":
return Path.home() / "Library/Application Support/Claude/claude_desktop_config.json"
elif sys.platform == "win32":
return Path.home() / "AppData/Roaming/Claude/claude_desktop_config.json"
else:
raise RuntimeError(f"Unsupported platform: {sys.platform}")
def update_claude_config():
"""Update Claude Desktop config to include basic-memory."""
config_path = get_config_path()
config_path.parent.mkdir(parents=True, exist_ok=True)
# Load existing config or create new
if config_path.exists():
config = json.loads(config_path.read_text(encoding="utf-8"))
else:
config = {"mcpServers": {}}
# Add/update basic-memory config
config["mcpServers"]["basic-memory"] = {
"command": "uvx",
"args": ["basic-memory@latest", "mcp"],
}
# Write back config
config_path.write_text(json.dumps(config, indent=2))
def print_completion_message():
"""Show completion message with helpful tips."""
message = """Installation complete! Basic Memory is now available in Claude Desktop.
Please restart Claude Desktop for changes to take effect.
Quick Start:
1. You can run sync directly using: uvx basic-memory sync
2. Optionally, install globally with: uv pip install basic-memory
Built with ♥️ by Basic Machines."""
if sys.platform == "darwin":
# Show GUI message on macOS
root = tk.Tk()
root.withdraw() # Hide the main window
messagebox.showinfo("Basic Memory", message)
root.destroy()
else:
# Fallback to console output
print(message)
def main():
print("Welcome to Basic Memory installer")
ensure_uv_installed()
print("Configuring Claude Desktop...")
update_claude_config()
print_completion_message()
if __name__ == "__main__":
main()
installer/make_icons.sh
-27
View File
@@ -1,27 +0,0 @@
#!/bin/bash
# Convert SVG to PNG at various required sizes
rsvg-convert -h 16 -w 16 icon.svg > icon_16x16.png
rsvg-convert -h 32 -w 32 icon.svg > icon_32x32.png
rsvg-convert -h 128 -w 128 icon.svg > icon_128x128.png
rsvg-convert -h 256 -w 256 icon.svg > icon_256x256.png
rsvg-convert -h 512 -w 512 icon.svg > icon_512x512.png
# Create iconset directory
mkdir -p Basic.iconset
# Move files into iconset with Mac-specific names
cp icon_16x16.png Basic.iconset/icon_16x16.png
cp icon_32x32.png Basic.iconset/icon_16x16@2x.png
cp icon_32x32.png Basic.iconset/icon_32x32.png
cp icon_128x128.png Basic.iconset/icon_32x32@2x.png
cp icon_256x256.png Basic.iconset/icon_128x128.png
cp icon_512x512.png Basic.iconset/icon_256x256.png
cp icon_512x512.png Basic.iconset/icon_512x512.png
# Convert iconset to icns
iconutil -c icns Basic.iconset
# Clean up
rm -rf Basic.iconset
rm icon_*.png
installer/setup.py
-40
View File
@@ -1,40 +0,0 @@
from cx_Freeze import setup, Executable
import sys
# Build options for all platforms
build_exe_options = {
"packages": ["json", "pathlib"],
"excludes": ["unittest", "pydoc", "test"],
}
# Platform-specific options
if sys.platform == "win32":
base = "Win32GUI" # Use GUI base for Windows
build_exe_options.update(
{
"include_msvcr": True,
}
)
target_name = "Basic Memory Installer.exe"
else: # darwin
base = None # Don't use GUI base for macOS
target_name = "Basic Memory Installer"
executables = [
Executable(script="installer.py", target_name=target_name, base=base, icon="Basic.icns")
]
setup(
name="basic-memory",
version=open("../pyproject.toml").read().split('version = "', 1)[1].split('"', 1)[0],
description="Basic Memory - Local-first knowledge management",
options={
"build_exe": build_exe_options,
"bdist_mac": {
"bundle_name": "Basic Memory Installer",
"iconfile": "Basic.icns",
"codesign_identity": "-", # Force ad-hoc signing
},
},
executables=executables,
)
pyproject.toml
+27
View File
@@ -30,6 +30,10 @@ dependencies = [
"alembic>=1.14.1",
"qasync>=0.27.1",
"pillow>=11.1.0",
"pybars3>=0.9.7",
"fastmcp>=2.3.4",
"pyjwt>=2.10.1",
"python-dotenv>=1.1.0",
]
@@ -103,5 +107,28 @@ commit_message = "chore(release): {version} [skip ci]"
[tool.coverage.run]
concurrency = ["thread", "gevent"]
[tool.coverage.report]
exclude_lines = [
"pragma: no cover",
"def __repr__",
"if self.debug:",
"if settings.DEBUG",
"raise AssertionError",
"raise NotImplementedError",
"if 0:",
"if __name__ == .__main__.:",
"class .*\\bProtocol\\):",
"@(abc\\.)?abstractmethod",
]
# Exclude specific modules that are difficult to test comprehensively
omit = [
"*/external_auth_provider.py", # External HTTP calls to OAuth providers
"*/supabase_auth_provider.py", # External HTTP calls to Supabase APIs
"*/watch_service.py", # File system watching - complex integration testing
"*/background_sync.py", # Background processes
"*/cli/main.py", # CLI entry point
]
[tool.logfire]
ignore_no_config = true
src/basic_memory/alembic/env.py
+1 -1
View File
@@ -13,7 +13,7 @@ from basic_memory.models import Base
# set config.env to "test" for pytest to prevent logging to file in utils.setup_logging()
os.environ["BASIC_MEMORY_ENV"] = "test"
from basic_memory.config import config as app_config
from basic_memory.config import app_config
# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
src/basic_memory/alembic/versions/5fe1ab1ccebe_add_projects_table.py
+108
View File
@@ -0,0 +1,108 @@
"""add projects table
Revision ID: 5fe1ab1ccebe
Revises: cc7172b46608
Create Date: 2025-05-14 09:05:18.214357
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = "5fe1ab1ccebe"
down_revision: Union[str, None] = "cc7172b46608"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.create_table(
"project",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("name", sa.String(), nullable=False),
sa.Column("description", sa.Text(), nullable=True),
sa.Column("permalink", sa.String(), nullable=False),
sa.Column("path", sa.String(), nullable=False),
sa.Column("is_active", sa.Boolean(), nullable=False),
sa.Column("is_default", sa.Boolean(), nullable=True),
sa.Column("created_at", sa.DateTime(), nullable=False),
sa.Column("updated_at", sa.DateTime(), nullable=False),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("is_default"),
sa.UniqueConstraint("name"),
sa.UniqueConstraint("permalink"),
if_not_exists=True,
)
with op.batch_alter_table("project", schema=None) as batch_op:
batch_op.create_index(
"ix_project_created_at", ["created_at"], unique=False, if_not_exists=True
)
batch_op.create_index("ix_project_name", ["name"], unique=True, if_not_exists=True)
batch_op.create_index("ix_project_path", ["path"], unique=False, if_not_exists=True)
batch_op.create_index(
"ix_project_permalink", ["permalink"], unique=True, if_not_exists=True
)
batch_op.create_index(
"ix_project_updated_at", ["updated_at"], unique=False, if_not_exists=True
)
with op.batch_alter_table("entity", schema=None) as batch_op:
batch_op.add_column(sa.Column("project_id", sa.Integer(), nullable=False))
batch_op.drop_index(
"uix_entity_permalink",
sqlite_where=sa.text("content_type = 'text/markdown' AND permalink IS NOT NULL"),
)
batch_op.drop_index("ix_entity_file_path")
batch_op.create_index(batch_op.f("ix_entity_file_path"), ["file_path"], unique=False)
batch_op.create_index("ix_entity_project_id", ["project_id"], unique=False)
batch_op.create_index(
"uix_entity_file_path_project", ["file_path", "project_id"], unique=True
)
batch_op.create_index(
"uix_entity_permalink_project",
["permalink", "project_id"],
unique=True,
sqlite_where=sa.text("content_type = 'text/markdown' AND permalink IS NOT NULL"),
)
batch_op.create_foreign_key("fk_entity_project_id", "project", ["project_id"], ["id"])
# drop the search index table. it will be recreated
op.drop_table("search_index")
# ### end Alembic commands ###
def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table("entity", schema=None) as batch_op:
batch_op.drop_constraint("fk_entity_project_id", type_="foreignkey")
batch_op.drop_index(
"uix_entity_permalink_project",
sqlite_where=sa.text("content_type = 'text/markdown' AND permalink IS NOT NULL"),
)
batch_op.drop_index("uix_entity_file_path_project")
batch_op.drop_index("ix_entity_project_id")
batch_op.drop_index(batch_op.f("ix_entity_file_path"))
batch_op.create_index("ix_entity_file_path", ["file_path"], unique=1)
batch_op.create_index(
"uix_entity_permalink",
["permalink"],
unique=1,
sqlite_where=sa.text("content_type = 'text/markdown' AND permalink IS NOT NULL"),
)
batch_op.drop_column("project_id")
with op.batch_alter_table("project", schema=None) as batch_op:
batch_op.drop_index("ix_project_updated_at")
batch_op.drop_index("ix_project_permalink")
batch_op.drop_index("ix_project_path")
batch_op.drop_index("ix_project_name")
batch_op.drop_index("ix_project_created_at")
op.drop_table("project")
# ### end Alembic commands ###
src/basic_memory/api/app.py
+40 -13
View File
@@ -1,29 +1,50 @@
"""FastAPI application for basic-memory knowledge graph API."""
import asyncio
from contextlib import asynccontextmanager
from fastapi import FastAPI, HTTPException
from fastapi.exception_handlers import http_exception_handler
from loguru import logger
from basic_memory import __version__ as version
from basic_memory import db
from basic_memory.api.routers import knowledge, memory, project_info, resource, search
from basic_memory.config import config as project_config
from basic_memory.services.initialization import initialize_app
from basic_memory.api.routers import (
directory_router,
importer_router,
knowledge,
management,
memory,
project,
resource,
search,
prompt_router,
)
from basic_memory.config import app_config
from basic_memory.services.initialization import initialize_app, initialize_file_sync
@asynccontextmanager
async def lifespan(app: FastAPI): # pragma: no cover
"""Lifecycle manager for the FastAPI app."""
# Initialize database and file sync services
watch_task = await initialize_app(project_config)
# Initialize app and database
logger.info("Starting Basic Memory API")
await initialize_app(app_config)
logger.info(f"Sync changes enabled: {app_config.sync_changes}")
if app_config.sync_changes:
# start file sync task in background
app.state.sync_task = asyncio.create_task(initialize_file_sync(app_config))
else:
logger.info("Sync changes disabled. Skipping file sync service.")
# proceed with startup
yield
logger.info("Shutting down Basic Memory API")
if watch_task:
watch_task.cancel()
if app.state.sync_task:
logger.info("Stopping sync...")
app.state.sync_task.cancel() # pyright: ignore
await db.shutdown_db()
@@ -32,17 +53,23 @@ async def lifespan(app: FastAPI): # pragma: no cover
app = FastAPI(
title="Basic Memory API",
description="Knowledge graph API for basic-memory",
version="0.1.0",
version=version,
lifespan=lifespan,
)
# Include routers
app.include_router(knowledge.router)
app.include_router(search.router)
app.include_router(memory.router)
app.include_router(resource.router)
app.include_router(project_info.router)
app.include_router(knowledge.router, prefix="/{project}")
app.include_router(management.router, prefix="/{project}")
app.include_router(memory.router, prefix="/{project}")
app.include_router(resource.router, prefix="/{project}")
app.include_router(search.router, prefix="/{project}")
app.include_router(project.router, prefix="/{project}")
app.include_router(directory_router.router, prefix="/{project}")
app.include_router(prompt_router.router, prefix="/{project}")
app.include_router(importer_router.router, prefix="/{project}")
# Auth routes are handled by FastMCP automatically when auth is enabled
@app.exception_handler(Exception)
src/basic_memory/api/routers/__init__.py
+4 -2
View File
@@ -1,9 +1,11 @@
"""API routers."""
from . import knowledge_router as knowledge
from . import management_router as management
from . import memory_router as memory
from . import project_router as project
from . import resource_router as resource
from . import search_router as search
from . import project_info_router as project_info
from . import prompt_router as prompt
__all__ = ["knowledge", "memory", "resource", "search", "project_info"]
__all__ = ["knowledge", "management", "memory", "project", "resource", "search", "prompt"]
src/basic_memory/api/routers/directory_router.py
+29
View File
@@ -0,0 +1,29 @@
"""Router for directory tree operations."""
from fastapi import APIRouter
from basic_memory.deps import DirectoryServiceDep, ProjectIdDep
from basic_memory.schemas.directory import DirectoryNode
router = APIRouter(prefix="/directory", tags=["directory"])
@router.get("/tree", response_model=DirectoryNode)
async def get_directory_tree(
directory_service: DirectoryServiceDep,
project_id: ProjectIdDep,
):
"""Get hierarchical directory structure from the knowledge base.
Args:
directory_service: Service for directory operations
project_id: ID of the current project
Returns:
DirectoryNode representing the root of the hierarchical tree structure
"""
# Get a hierarchical directory tree for the specific project
tree = await directory_service.get_directory_tree()
# Return the hierarchical tree
return tree
src/basic_memory/api/routers/importer_router.py
+152
View File
@@ -0,0 +1,152 @@
"""Import router for Basic Memory API."""
import json
import logging
from fastapi import APIRouter, Form, HTTPException, UploadFile, status
from basic_memory.deps import (
ChatGPTImporterDep,
ClaudeConversationsImporterDep,
ClaudeProjectsImporterDep,
MemoryJsonImporterDep,
)
from basic_memory.importers import Importer
from basic_memory.schemas.importer import (
ChatImportResult,
EntityImportResult,
ProjectImportResult,
)
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/import", tags=["import"])
@router.post("/chatgpt", response_model=ChatImportResult)
async def import_chatgpt(
importer: ChatGPTImporterDep,
file: UploadFile,
folder: str = Form("conversations"),
) -> ChatImportResult:
"""Import conversations from ChatGPT JSON export.
Args:
file: The ChatGPT conversations.json file.
folder: The folder to place the files in.
markdown_processor: MarkdownProcessor instance.
Returns:
ChatImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
return await import_file(importer, file, folder)
@router.post("/claude/conversations", response_model=ChatImportResult)
async def import_claude_conversations(
importer: ClaudeConversationsImporterDep,
file: UploadFile,
folder: str = Form("conversations"),
) -> ChatImportResult:
"""Import conversations from Claude conversations.json export.
Args:
file: The Claude conversations.json file.
folder: The folder to place the files in.
markdown_processor: MarkdownProcessor instance.
Returns:
ChatImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
return await import_file(importer, file, folder)
@router.post("/claude/projects", response_model=ProjectImportResult)
async def import_claude_projects(
importer: ClaudeProjectsImporterDep,
file: UploadFile,
folder: str = Form("projects"),
) -> ProjectImportResult:
"""Import projects from Claude projects.json export.
Args:
file: The Claude projects.json file.
base_folder: The base folder to place the files in.
markdown_processor: MarkdownProcessor instance.
Returns:
ProjectImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
return await import_file(importer, file, folder)
@router.post("/memory-json", response_model=EntityImportResult)
async def import_memory_json(
importer: MemoryJsonImporterDep,
file: UploadFile,
folder: str = Form("conversations"),
) -> EntityImportResult:
"""Import entities and relations from a memory.json file.
Args:
file: The memory.json file.
destination_folder: Optional destination folder within the project.
markdown_processor: MarkdownProcessor instance.
Returns:
EntityImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
try:
file_data = []
file_bytes = await file.read()
file_str = file_bytes.decode("utf-8")
for line in file_str.splitlines():
json_data = json.loads(line)
file_data.append(json_data)
result = await importer.import_data(file_data, folder)
if not result.success: # pragma: no cover
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=result.error_message or "Import failed",
)
except Exception as e:
logger.exception("Import failed")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Import failed: {str(e)}",
)
return result
async def import_file(importer: Importer, file: UploadFile, destination_folder: str):
try:
# Process file
json_data = json.load(file.file)
result = await importer.import_data(json_data, destination_folder)
if not result.success: # pragma: no cover
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=result.error_message or "Import failed",
)
return result
except Exception as e:
logger.exception("Import failed")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Import failed: {str(e)}",
)
src/basic_memory/api/routers/knowledge_router.py
+17 -14
View File
@@ -18,7 +18,6 @@ from basic_memory.schemas import (
DeleteEntitiesRequest,
)
from basic_memory.schemas.base import Permalink, Entity
from basic_memory.services.exceptions import EntityNotFoundError
router = APIRouter(prefix="/knowledge", tags=["knowledge"])
@@ -80,7 +79,10 @@ async def create_or_update_entity(
data_permalink=data.permalink,
error="Permalink mismatch",
)
raise HTTPException(status_code=400, detail="Entity permalink must match URL path")
raise HTTPException(
status_code=400,
detail=f"Entity permalink {data.permalink} must match URL path: '{permalink}'",
)
# Try create_or_update operation
entity, created = await entity_service.create_or_update_entity(data)
@@ -104,25 +106,26 @@ async def create_or_update_entity(
## Read endpoints
@router.get("/entities/{permalink:path}", response_model=EntityResponse)
@router.get("/entities/{identifier:path}", response_model=EntityResponse)
async def get_entity(
entity_service: EntityServiceDep,
permalink: str,
link_resolver: LinkResolverDep,
identifier: str,
) -> EntityResponse:
"""Get a specific entity by ID.
"""Get a specific entity by file path or permalink..
Args:
permalink: Entity path ID
content: If True, include full file content
identifier: Entity file path or permalink
:param entity_service: EntityService
:param link_resolver: LinkResolver
"""
logger.info(f"request: get_entity with permalink={permalink}")
try:
entity = await entity_service.get_by_permalink(permalink)
result = EntityResponse.model_validate(entity)
return result
except EntityNotFoundError:
raise HTTPException(status_code=404, detail=f"Entity with {permalink} not found")
logger.info(f"request: get_entity with identifier={identifier}")
entity = await link_resolver.resolve_link(identifier)
if not entity:
raise HTTPException(status_code=404, detail=f"Entity {identifier} not found")
result = EntityResponse.model_validate(entity)
return result
@router.get("/entities", response_model=EntityListResponse)
src/basic_memory/api/routers/management_router.py
+78
View File
@@ -0,0 +1,78 @@
"""Management router for basic-memory API."""
import asyncio
from fastapi import APIRouter, Request
from loguru import logger
from pydantic import BaseModel
from basic_memory.config import app_config
from basic_memory.deps import SyncServiceDep, ProjectRepositoryDep
router = APIRouter(prefix="/management", tags=["management"])
class WatchStatusResponse(BaseModel):
"""Response model for watch status."""
running: bool
"""Whether the watch service is currently running."""
@router.get("/watch/status", response_model=WatchStatusResponse)
async def get_watch_status(request: Request) -> WatchStatusResponse:
"""Get the current status of the watch service."""
return WatchStatusResponse(
running=request.app.state.watch_task is not None and not request.app.state.watch_task.done()
)
@router.post("/watch/start", response_model=WatchStatusResponse)
async def start_watch_service(
request: Request, project_repository: ProjectRepositoryDep, sync_service: SyncServiceDep
) -> WatchStatusResponse:
"""Start the watch service if it's not already running."""
# needed because of circular imports from sync -> app
from basic_memory.sync import WatchService
from basic_memory.sync.background_sync import create_background_sync_task
if request.app.state.watch_task is not None and not request.app.state.watch_task.done():
# Watch service is already running
return WatchStatusResponse(running=True)
# Create and start a new watch service
logger.info("Starting watch service via management API")
# Get services needed for the watch task
watch_service = WatchService(
app_config=app_config,
project_repository=project_repository,
)
# Create and store the task
watch_task = create_background_sync_task(sync_service, watch_service)
request.app.state.watch_task = watch_task
return WatchStatusResponse(running=True)
@router.post("/watch/stop", response_model=WatchStatusResponse)
async def stop_watch_service(request: Request) -> WatchStatusResponse: # pragma: no cover
"""Stop the watch service if it's running."""
if request.app.state.watch_task is None or request.app.state.watch_task.done():
# Watch service is not running
return WatchStatusResponse(running=False)
# Cancel the running task
logger.info("Stopping watch service via management API")
request.app.state.watch_task.cancel()
# Wait for it to be properly cancelled
try:
await request.app.state.watch_task
except asyncio.CancelledError:
pass
request.app.state.watch_task = None
return WatchStatusResponse(running=False)
src/basic_memory/api/routers/memory_router.py
+4 -59
View File
@@ -1,78 +1,23 @@
"""Routes for memory:// URI operations."""
from typing import Annotated
from typing import Annotated, Optional
from dateparser import parse
from fastapi import APIRouter, Query
from loguru import logger
from basic_memory.deps import ContextServiceDep, EntityRepositoryDep
from basic_memory.repository import EntityRepository
from basic_memory.repository.search_repository import SearchIndexRow
from basic_memory.schemas.base import TimeFrame
from basic_memory.schemas.memory import (
GraphContext,
RelationSummary,
EntitySummary,
ObservationSummary,
MemoryMetadata,
normalize_memory_url,
)
from basic_memory.schemas.search import SearchItemType
from basic_memory.services.context_service import ContextResultRow
from basic_memory.api.routers.utils import to_graph_context
router = APIRouter(prefix="/memory", tags=["memory"])
async def to_graph_context(context, entity_repository: EntityRepository, page: int, page_size: int):
# return results
async def to_summary(item: SearchIndexRow | ContextResultRow):
match item.type:
case SearchItemType.ENTITY:
return EntitySummary(
title=item.title, # pyright: ignore
permalink=item.permalink,
content=item.content,
file_path=item.file_path,
created_at=item.created_at,
)
case SearchItemType.OBSERVATION:
return ObservationSummary(
title=item.title, # pyright: ignore
file_path=item.file_path,
category=item.category, # pyright: ignore
content=item.content, # pyright: ignore
permalink=item.permalink, # pyright: ignore
created_at=item.created_at,
)
case SearchItemType.RELATION:
from_entity = await entity_repository.find_by_id(item.from_id) # pyright: ignore
to_entity = await entity_repository.find_by_id(item.to_id) if item.to_id else None
return RelationSummary(
title=item.title, # pyright: ignore
file_path=item.file_path,
permalink=item.permalink, # pyright: ignore
relation_type=item.type,
from_entity=from_entity.permalink, # pyright: ignore
to_entity=to_entity.permalink if to_entity else None,
created_at=item.created_at,
)
case _: # pragma: no cover
raise ValueError(f"Unexpected type: {item.type}")
primary_results = [await to_summary(r) for r in context["primary_results"]]
related_results = [await to_summary(r) for r in context["related_results"]]
metadata = MemoryMetadata.model_validate(context["metadata"])
# Transform to GraphContext
return GraphContext(
primary_results=primary_results,
related_results=related_results,
metadata=metadata,
page=page,
page_size=page_size,
)
@router.get("/recent", response_model=GraphContext)
async def recent(
context_service: ContextServiceDep,
@@ -119,7 +64,7 @@ async def get_memory_context(
entity_repository: EntityRepositoryDep,
uri: str,
depth: int = 1,
timeframe: TimeFrame = "7d",
timeframe: Optional[TimeFrame] = None,
page: int = 1,
page_size: int = 10,
max_related: int = 10,
@@ -133,7 +78,7 @@ async def get_memory_context(
memory_url = normalize_memory_url(uri)
# Parse timeframe
since = parse(timeframe)
since = parse(timeframe) if timeframe else None
limit = page_size
offset = (page - 1) * page_size
src/basic_memory/api/routers/project_info_router.py
-274
View File
@@ -1,274 +0,0 @@
"""Router for statistics and system information."""
import json
from datetime import datetime
from basic_memory.config import config, config_manager
from basic_memory.deps import (
ProjectInfoRepositoryDep,
)
from basic_memory.repository.project_info_repository import ProjectInfoRepository
from basic_memory.schemas import (
ProjectInfoResponse,
ProjectStatistics,
ActivityMetrics,
SystemStatus,
)
from basic_memory.sync.watch_service import WATCH_STATUS_JSON
from fastapi import APIRouter
from sqlalchemy import text
router = APIRouter(prefix="/stats", tags=["statistics"])
@router.get("/project-info", response_model=ProjectInfoResponse)
async def get_project_info(
repository: ProjectInfoRepositoryDep,
) -> ProjectInfoResponse:
"""Get comprehensive information about the current Basic Memory project."""
# Get statistics
statistics = await get_statistics(repository)
# Get activity metrics
activity = await get_activity_metrics(repository)
# Get system status
system = await get_system_status()
# Get project configuration information
project_name = config.project
project_path = str(config.home)
available_projects = config_manager.projects
default_project = config_manager.default_project
# Construct the response
return ProjectInfoResponse(
project_name=project_name,
project_path=project_path,
available_projects=available_projects,
default_project=default_project,
statistics=statistics,
activity=activity,
system=system,
)
async def get_statistics(repository: ProjectInfoRepository) -> ProjectStatistics:
"""Get statistics about the current project."""
# Get basic counts
entity_count_result = await repository.execute_query(text("SELECT COUNT(*) FROM entity"))
total_entities = entity_count_result.scalar() or 0
observation_count_result = await repository.execute_query(
text("SELECT COUNT(*) FROM observation")
)
total_observations = observation_count_result.scalar() or 0
relation_count_result = await repository.execute_query(text("SELECT COUNT(*) FROM relation"))
total_relations = relation_count_result.scalar() or 0
unresolved_count_result = await repository.execute_query(
text("SELECT COUNT(*) FROM relation WHERE to_id IS NULL")
)
total_unresolved = unresolved_count_result.scalar() or 0
# Get entity counts by type
entity_types_result = await repository.execute_query(
text("SELECT entity_type, COUNT(*) FROM entity GROUP BY entity_type")
)
entity_types = {row[0]: row[1] for row in entity_types_result.fetchall()}
# Get observation counts by category
category_result = await repository.execute_query(
text("SELECT category, COUNT(*) FROM observation GROUP BY category")
)
observation_categories = {row[0]: row[1] for row in category_result.fetchall()}
# Get relation counts by type
relation_types_result = await repository.execute_query(
text("SELECT relation_type, COUNT(*) FROM relation GROUP BY relation_type")
)
relation_types = {row[0]: row[1] for row in relation_types_result.fetchall()}
# Find most connected entities (most outgoing relations)
connected_result = await repository.execute_query(
text("""
SELECT e.id, e.title, e.permalink, COUNT(r.id) AS relation_count
FROM entity e
JOIN relation r ON e.id = r.from_id
GROUP BY e.id
ORDER BY relation_count DESC
LIMIT 10
""")
)
most_connected = [
{"id": row[0], "title": row[1], "permalink": row[2], "relation_count": row[3]}
for row in connected_result.fetchall()
]
# Count isolated entities (no relations)
isolated_result = await repository.execute_query(
text("""
SELECT COUNT(e.id)
FROM entity e
LEFT JOIN relation r1 ON e.id = r1.from_id
LEFT JOIN relation r2 ON e.id = r2.to_id
WHERE r1.id IS NULL AND r2.id IS NULL
""")
)
isolated_count = isolated_result.scalar() or 0
return ProjectStatistics(
total_entities=total_entities,
total_observations=total_observations,
total_relations=total_relations,
total_unresolved_relations=total_unresolved,
entity_types=entity_types,
observation_categories=observation_categories,
relation_types=relation_types,
most_connected_entities=most_connected,
isolated_entities=isolated_count,
)
async def get_activity_metrics(repository: ProjectInfoRepository) -> ActivityMetrics:
"""Get activity metrics for the current project."""
# Get recently created entities
created_result = await repository.execute_query(
text("""
SELECT id, title, permalink, entity_type, created_at
FROM entity
ORDER BY created_at DESC
LIMIT 10
""")
)
recently_created = [
{
"id": row[0],
"title": row[1],
"permalink": row[2],
"entity_type": row[3],
"created_at": row[4],
}
for row in created_result.fetchall()
]
# Get recently updated entities
updated_result = await repository.execute_query(
text("""
SELECT id, title, permalink, entity_type, updated_at
FROM entity
ORDER BY updated_at DESC
LIMIT 10
""")
)
recently_updated = [
{
"id": row[0],
"title": row[1],
"permalink": row[2],
"entity_type": row[3],
"updated_at": row[4],
}
for row in updated_result.fetchall()
]
# Get monthly growth over the last 6 months
# Calculate the start of 6 months ago
now = datetime.now()
six_months_ago = datetime(
now.year - (1 if now.month <= 6 else 0), ((now.month - 6) % 12) or 12, 1
)
# Query for monthly entity creation
entity_growth_result = await repository.execute_query(
text(f"""
SELECT
strftime('%Y-%m', created_at) AS month,
COUNT(*) AS count
FROM entity
WHERE created_at >= '{six_months_ago.isoformat()}'
GROUP BY month
ORDER BY month
""")
)
entity_growth = {row[0]: row[1] for row in entity_growth_result.fetchall()}
# Query for monthly observation creation
observation_growth_result = await repository.execute_query(
text(f"""
SELECT
strftime('%Y-%m', created_at) AS month,
COUNT(*) AS count
FROM observation
INNER JOIN entity ON observation.entity_id = entity.id
WHERE entity.created_at >= '{six_months_ago.isoformat()}'
GROUP BY month
ORDER BY month
""")
)
observation_growth = {row[0]: row[1] for row in observation_growth_result.fetchall()}
# Query for monthly relation creation
relation_growth_result = await repository.execute_query(
text(f"""
SELECT
strftime('%Y-%m', created_at) AS month,
COUNT(*) AS count
FROM relation
INNER JOIN entity ON relation.from_id = entity.id
WHERE entity.created_at >= '{six_months_ago.isoformat()}'
GROUP BY month
ORDER BY month
""")
)
relation_growth = {row[0]: row[1] for row in relation_growth_result.fetchall()}
# Combine all monthly growth data
monthly_growth = {}
for month in set(
list(entity_growth.keys()) + list(observation_growth.keys()) + list(relation_growth.keys())
):
monthly_growth[month] = {
"entities": entity_growth.get(month, 0),
"observations": observation_growth.get(month, 0),
"relations": relation_growth.get(month, 0),
"total": (
entity_growth.get(month, 0)
+ observation_growth.get(month, 0)
+ relation_growth.get(month, 0)
),
}
return ActivityMetrics(
recently_created=recently_created,
recently_updated=recently_updated,
monthly_growth=monthly_growth,
)
async def get_system_status() -> SystemStatus:
"""Get system status information."""
import basic_memory
# Get database information
db_path = config.database_path
db_size = db_path.stat().st_size if db_path.exists() else 0
db_size_readable = f"{db_size / (1024 * 1024):.2f} MB"
# Get watch service status if available
watch_status = None
watch_status_path = config.home / ".basic-memory" / WATCH_STATUS_JSON
if watch_status_path.exists():
try:
watch_status = json.loads(watch_status_path.read_text(encoding="utf-8"))
except Exception: # pragma: no cover
pass
return SystemStatus(
version=basic_memory.__version__,
database_path=str(db_path),
database_size=db_size_readable,
watch_status=watch_status,
timestamp=datetime.now(),
)
src/basic_memory/api/routers/project_router.py
+235
View File
@@ -0,0 +1,235 @@
"""Router for project management."""
from fastapi import APIRouter, HTTPException, Path, Body
from typing import Optional
from basic_memory.deps import ProjectServiceDep
from basic_memory.schemas import ProjectInfoResponse
from basic_memory.schemas.project_info import (
ProjectList,
ProjectItem,
ProjectSwitchRequest,
ProjectStatusResponse,
ProjectWatchStatus,
)
# Define the router - we'll combine stats and project operations
router = APIRouter(prefix="/project", tags=["project"])
# Get project information (moved from project_info_router.py)
@router.get("/info", response_model=ProjectInfoResponse)
async def get_project_info(
project_service: ProjectServiceDep,
) -> ProjectInfoResponse:
"""Get comprehensive information about the current Basic Memory project."""
return await project_service.get_project_info()
# List all available projects
@router.get("/projects", response_model=ProjectList)
async def list_projects(
project_service: ProjectServiceDep,
) -> ProjectList:
"""List all configured projects.
Returns:
A list of all projects with metadata
"""
projects_dict = project_service.projects
default_project = project_service.default_project
current_project = project_service.current_project
project_items = []
for name, path in projects_dict.items():
project_items.append(
ProjectItem(
name=name,
path=path,
is_default=(name == default_project),
is_current=(name == current_project),
)
)
return ProjectList(
projects=project_items,
default_project=default_project,
current_project=current_project,
)
# Add a new project
@router.post("/projects", response_model=ProjectStatusResponse)
async def add_project(
project_data: ProjectSwitchRequest,
project_service: ProjectServiceDep,
) -> ProjectStatusResponse:
"""Add a new project to configuration and database.
Args:
project_data: The project name and path, with option to set as default
Returns:
Response confirming the project was added
"""
try: # pragma: no cover
await project_service.add_project(project_data.name, project_data.path)
if project_data.set_default: # pragma: no cover
await project_service.set_default_project(project_data.name)
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message=f"Project '{project_data.name}' added successfully",
status="success",
default=project_data.set_default,
new_project=ProjectWatchStatus(
name=project_data.name,
path=project_data.path,
watch_status=None,
),
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e))
# Remove a project
@router.delete("/projects/{name}", response_model=ProjectStatusResponse)
async def remove_project(
project_service: ProjectServiceDep,
name: str = Path(..., description="Name of the project to remove"),
) -> ProjectStatusResponse:
"""Remove a project from configuration and database.
Args:
name: The name of the project to remove
Returns:
Response confirming the project was removed
"""
try: # pragma: no cover
# Get project info before removal for the response
old_project = ProjectWatchStatus(
name=name,
path=project_service.projects.get(name, ""),
watch_status=None,
)
await project_service.remove_project(name)
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message=f"Project '{name}' removed successfully",
status="success",
default=False,
old_project=old_project,
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e))
# Set a project as default
@router.put("/projects/{name}/default", response_model=ProjectStatusResponse)
async def set_default_project(
project_service: ProjectServiceDep,
name: str = Path(..., description="Name of the project to set as default"),
) -> ProjectStatusResponse:
"""Set a project as the default project.
Args:
name: The name of the project to set as default
Returns:
Response confirming the project was set as default
"""
try: # pragma: no cover
# Get the old default project
old_default = project_service.default_project
old_project = None
if old_default != name:
old_project = ProjectWatchStatus(
name=old_default,
path=project_service.projects.get(old_default, ""),
watch_status=None,
)
await project_service.set_default_project(name)
return ProjectStatusResponse(
message=f"Project '{name}' set as default successfully",
status="success",
default=True,
old_project=old_project,
new_project=ProjectWatchStatus(
name=name,
path=project_service.projects.get(name, ""),
watch_status=None,
),
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e))
# Update a project
@router.patch("/projects/{name}", response_model=ProjectStatusResponse)
async def update_project(
project_service: ProjectServiceDep,
name: str = Path(..., description="Name of the project to update"),
path: Optional[str] = Body(None, description="New path for the project"),
is_active: Optional[bool] = Body(None, description="Status of the project (active/inactive)"),
) -> ProjectStatusResponse:
"""Update a project's information in configuration and database.
Args:
name: The name of the project to update
path: Optional new path for the project
is_active: Optional status update for the project
Returns:
Response confirming the project was updated
"""
try: # pragma: no cover
# Get original project info for the response
old_project = ProjectWatchStatus(
name=name,
path=project_service.projects.get(name, ""),
watch_status=None,
)
await project_service.update_project(name, updated_path=path, is_active=is_active)
# Get updated project info
updated_path = path if path else project_service.projects.get(name, "")
return ProjectStatusResponse(
message=f"Project '{name}' updated successfully",
status="success",
default=(name == project_service.default_project),
old_project=old_project,
new_project=ProjectWatchStatus(name=name, path=updated_path, watch_status=None),
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e))
# Synchronize projects between config and database
@router.post("/sync", response_model=ProjectStatusResponse)
async def synchronize_projects(
project_service: ProjectServiceDep,
) -> ProjectStatusResponse:
"""Synchronize projects between configuration file and database.
Ensures that all projects in the configuration file exist in the database
and vice versa.
Returns:
Response confirming synchronization was completed
"""
try: # pragma: no cover
await project_service.synchronize_projects()
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message="Projects synchronized successfully between configuration and database",
status="success",
default=False,
)
except ValueError as e: # pragma: no cover
raise HTTPException(status_code=400, detail=str(e))
src/basic_memory/api/routers/prompt_router.py
+260
View File
@@ -0,0 +1,260 @@
"""Router for prompt-related operations.
This router is responsible for rendering various prompts using Handlebars templates.
It centralizes all prompt formatting logic that was previously in the MCP prompts.
"""
from datetime import datetime, timezone
from dateparser import parse
from fastapi import APIRouter, HTTPException, status
from loguru import logger
from basic_memory.api.routers.utils import to_graph_context, to_search_results
from basic_memory.api.template_loader import template_loader
from basic_memory.deps import (
ContextServiceDep,
EntityRepositoryDep,
SearchServiceDep,
EntityServiceDep,
)
from basic_memory.schemas.prompt import (
ContinueConversationRequest,
SearchPromptRequest,
PromptResponse,
PromptMetadata,
)
from basic_memory.schemas.search import SearchItemType, SearchQuery
router = APIRouter(prefix="/prompt", tags=["prompt"])
@router.post("/continue-conversation", response_model=PromptResponse)
async def continue_conversation(
search_service: SearchServiceDep,
entity_service: EntityServiceDep,
context_service: ContextServiceDep,
entity_repository: EntityRepositoryDep,
request: ContinueConversationRequest,
) -> PromptResponse:
"""Generate a prompt for continuing a conversation.
This endpoint takes a topic and/or timeframe and generates a prompt with
relevant context from the knowledge base.
Args:
request: The request parameters
Returns:
Formatted continuation prompt with context
"""
logger.info(
f"Generating continue conversation prompt, topic: {request.topic}, timeframe: {request.timeframe}"
)
since = parse(request.timeframe) if request.timeframe else None
# Initialize search results
search_results = []
# Get data needed for template
if request.topic:
query = SearchQuery(text=request.topic, after_date=request.timeframe)
results = await search_service.search(query, limit=request.search_items_limit)
search_results = await to_search_results(entity_service, results)
# Build context from results
all_hierarchical_results = []
for result in search_results:
if hasattr(result, "permalink") and result.permalink:
# Get hierarchical context using the new dataclass-based approach
context_result = await context_service.build_context(
result.permalink,
depth=request.depth,
since=since,
max_related=request.related_items_limit,
include_observations=True, # Include observations for entities
)
# Process results into the schema format
graph_context = await to_graph_context(
context_result, entity_repository=entity_repository
)
# Add results to our collection (limit to top results for each permalink)
if graph_context.results:
all_hierarchical_results.extend(graph_context.results[:3])
# Limit to a reasonable number of total results
all_hierarchical_results = all_hierarchical_results[:10]
template_context = {
"topic": request.topic,
"timeframe": request.timeframe,
"hierarchical_results": all_hierarchical_results,
"has_results": len(all_hierarchical_results) > 0,
}
else:
# If no topic, get recent activity
context_result = await context_service.build_context(
types=[SearchItemType.ENTITY],
depth=request.depth,
since=since,
max_related=request.related_items_limit,
include_observations=True,
)
recent_context = await to_graph_context(context_result, entity_repository=entity_repository)
hierarchical_results = recent_context.results[:5] # Limit to top 5 recent items
template_context = {
"topic": f"Recent Activity from ({request.timeframe})",
"timeframe": request.timeframe,
"hierarchical_results": hierarchical_results,
"has_results": len(hierarchical_results) > 0,
}
try:
# Render template
rendered_prompt = await template_loader.render(
"prompts/continue_conversation.hbs", template_context
)
# Calculate metadata
# Count items of different types
observation_count = 0
relation_count = 0
entity_count = 0
# Get the hierarchical results from the template context
hierarchical_results_for_count = template_context.get("hierarchical_results", [])
# For topic-based search
if request.topic:
for item in hierarchical_results_for_count:
if hasattr(item, "observations"):
observation_count += len(item.observations) if item.observations else 0
if hasattr(item, "related_results"):
for related in item.related_results or []:
if hasattr(related, "type"):
if related.type == "relation":
relation_count += 1
elif related.type == "entity": # pragma: no cover
entity_count += 1 # pragma: no cover
# For recent activity
else:
for item in hierarchical_results_for_count:
if hasattr(item, "observations"):
observation_count += len(item.observations) if item.observations else 0
if hasattr(item, "related_results"):
for related in item.related_results or []:
if hasattr(related, "type"):
if related.type == "relation":
relation_count += 1
elif related.type == "entity": # pragma: no cover
entity_count += 1 # pragma: no cover
# Build metadata
metadata = {
"query": request.topic,
"timeframe": request.timeframe,
"search_count": len(search_results)
if request.topic
else 0, # Original search results count
"context_count": len(hierarchical_results_for_count),
"observation_count": observation_count,
"relation_count": relation_count,
"total_items": (
len(hierarchical_results_for_count)
+ observation_count
+ relation_count
+ entity_count
),
"search_limit": request.search_items_limit,
"context_depth": request.depth,
"related_limit": request.related_items_limit,
"generated_at": datetime.now(timezone.utc).isoformat(),
}
prompt_metadata = PromptMetadata(**metadata)
return PromptResponse(
prompt=rendered_prompt, context=template_context, metadata=prompt_metadata
)
except Exception as e:
logger.error(f"Error rendering continue conversation template: {e}")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Error rendering prompt template: {str(e)}",
)
@router.post("/search", response_model=PromptResponse)
async def search_prompt(
search_service: SearchServiceDep,
entity_service: EntityServiceDep,
request: SearchPromptRequest,
page: int = 1,
page_size: int = 10,
) -> PromptResponse:
"""Generate a prompt for search results.
This endpoint takes a search query and formats the results into a helpful
prompt with context and suggestions.
Args:
request: The search parameters
page: The page number for pagination
page_size: The number of results per page, defaults to 10
Returns:
Formatted search results prompt with context
"""
logger.info(f"Generating search prompt, query: {request.query}, timeframe: {request.timeframe}")
limit = page_size
offset = (page - 1) * page_size
query = SearchQuery(text=request.query, after_date=request.timeframe)
results = await search_service.search(query, limit=limit, offset=offset)
search_results = await to_search_results(entity_service, results)
template_context = {
"query": request.query,
"timeframe": request.timeframe,
"results": search_results,
"has_results": len(search_results) > 0,
"result_count": len(search_results),
}
try:
# Render template
rendered_prompt = await template_loader.render("prompts/search.hbs", template_context)
# Build metadata
metadata = {
"query": request.query,
"timeframe": request.timeframe,
"search_count": len(search_results),
"context_count": len(search_results),
"observation_count": 0, # Search results don't include observations
"relation_count": 0, # Search results don't include relations
"total_items": len(search_results),
"search_limit": limit,
"context_depth": 0, # No context depth for basic search
"related_limit": 0, # No related items for basic search
"generated_at": datetime.now(timezone.utc).isoformat(),
}
prompt_metadata = PromptMetadata(**metadata)
return PromptResponse(
prompt=rendered_prompt, context=template_context, metadata=prompt_metadata
)
except Exception as e:
logger.error(f"Error rendering search template: {e}")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Error rendering prompt template: {str(e)}",
)
src/basic_memory/api/routers/search_router.py
+3 -21
View File
@@ -2,7 +2,8 @@
from fastapi import APIRouter, BackgroundTasks
from basic_memory.schemas.search import SearchQuery, SearchResult, SearchResponse
from basic_memory.api.routers.utils import to_search_results
from basic_memory.schemas.search import SearchQuery, SearchResponse
from basic_memory.deps import SearchServiceDep, EntityServiceDep
router = APIRouter(prefix="/search", tags=["search"])
@@ -20,26 +21,7 @@ async def search(
limit = page_size
offset = (page - 1) * page_size
results = await search_service.search(query, limit=limit, offset=offset)
search_results = []
for r in results:
entities = await entity_service.get_entities_by_id([r.entity_id, r.from_id, r.to_id]) # pyright: ignore
search_results.append(
SearchResult(
title=r.title, # pyright: ignore
type=r.type, # pyright: ignore
permalink=r.permalink,
score=r.score, # pyright: ignore
entity=entities[0].permalink if entities else None,
content=r.content,
file_path=r.file_path,
metadata=r.metadata,
category=r.category,
from_entity=entities[0].permalink if entities else None,
to_entity=entities[1].permalink if len(entities) > 1 else None,
relation_type=r.relation_type,
)
)
search_results = await to_search_results(entity_service, results)
return SearchResponse(
results=search_results,
current_page=page,
src/basic_memory/api/routers/utils.py
+130
View File
@@ -0,0 +1,130 @@
from typing import Optional, List
from basic_memory.repository import EntityRepository
from basic_memory.repository.search_repository import SearchIndexRow
from basic_memory.schemas.memory import (
EntitySummary,
ObservationSummary,
RelationSummary,
MemoryMetadata,
GraphContext,
ContextResult,
)
from basic_memory.schemas.search import SearchItemType, SearchResult
from basic_memory.services import EntityService
from basic_memory.services.context_service import (
ContextResultRow,
ContextResult as ServiceContextResult,
)
async def to_graph_context(
context_result: ServiceContextResult,
entity_repository: EntityRepository,
page: Optional[int] = None,
page_size: Optional[int] = None,
):
# Helper function to convert items to summaries
async def to_summary(item: SearchIndexRow | ContextResultRow):
match item.type:
case SearchItemType.ENTITY:
return EntitySummary(
title=item.title, # pyright: ignore
permalink=item.permalink,
content=item.content,
file_path=item.file_path,
created_at=item.created_at,
)
case SearchItemType.OBSERVATION:
return ObservationSummary(
title=item.title, # pyright: ignore
file_path=item.file_path,
category=item.category, # pyright: ignore
content=item.content, # pyright: ignore
permalink=item.permalink, # pyright: ignore
created_at=item.created_at,
)
case SearchItemType.RELATION:
from_entity = await entity_repository.find_by_id(item.from_id) # pyright: ignore
to_entity = await entity_repository.find_by_id(item.to_id) if item.to_id else None
return RelationSummary(
title=item.title, # pyright: ignore
file_path=item.file_path,
permalink=item.permalink, # pyright: ignore
relation_type=item.relation_type, # pyright: ignore
from_entity=from_entity.title, # pyright: ignore
to_entity=to_entity.title if to_entity else None,
created_at=item.created_at,
)
case _: # pragma: no cover
raise ValueError(f"Unexpected type: {item.type}")
# Process the hierarchical results
hierarchical_results = []
for context_item in context_result.results:
# Process primary result
primary_result = await to_summary(context_item.primary_result)
# Process observations
observations = []
for obs in context_item.observations:
observations.append(await to_summary(obs))
# Process related results
related = []
for rel in context_item.related_results:
related.append(await to_summary(rel))
# Add to hierarchical results
hierarchical_results.append(
ContextResult(
primary_result=primary_result,
observations=observations,
related_results=related,
)
)
# Create schema metadata from service metadata
metadata = MemoryMetadata(
uri=context_result.metadata.uri,
types=context_result.metadata.types,
depth=context_result.metadata.depth,
timeframe=context_result.metadata.timeframe,
generated_at=context_result.metadata.generated_at,
primary_count=context_result.metadata.primary_count,
related_count=context_result.metadata.related_count,
total_results=context_result.metadata.primary_count + context_result.metadata.related_count,
total_relations=context_result.metadata.total_relations,
total_observations=context_result.metadata.total_observations,
)
# Return new GraphContext with just hierarchical results
return GraphContext(
results=hierarchical_results,
metadata=metadata,
page=page,
page_size=page_size,
)
async def to_search_results(entity_service: EntityService, results: List[SearchIndexRow]):
search_results = []
for r in results:
entities = await entity_service.get_entities_by_id([r.entity_id, r.from_id, r.to_id]) # pyright: ignore
search_results.append(
SearchResult(
title=r.title, # pyright: ignore
type=r.type, # pyright: ignore
permalink=r.permalink,
score=r.score, # pyright: ignore
entity=entities[0].permalink if entities else None,
content=r.content,
file_path=r.file_path,
metadata=r.metadata,
category=r.category,
from_entity=entities[0].permalink if entities else None,
to_entity=entities[1].permalink if len(entities) > 1 else None,
relation_type=r.relation_type,
)
)
return search_results
src/basic_memory/api/template_loader.py
+292
View File
@@ -0,0 +1,292 @@
"""Template loading and rendering utilities for the Basic Memory API.
This module handles the loading and rendering of Handlebars templates from the
templates directory, providing a consistent interface for all prompt-related
formatting needs.
"""
import textwrap
from typing import Dict, Any, Optional, Callable
from pathlib import Path
import json
import datetime
import pybars
from loguru import logger
# Get the base path of the templates directory
TEMPLATES_DIR = Path(__file__).parent.parent / "templates"
# Custom helpers for Handlebars
def _date_helper(this, *args):
"""Format a date using the given format string."""
if len(args) < 1: # pragma: no cover
return ""
timestamp = args[0]
format_str = args[1] if len(args) > 1 else "%Y-%m-%d %H:%M"
if hasattr(timestamp, "strftime"):
result = timestamp.strftime(format_str)
elif isinstance(timestamp, str):
try:
dt = datetime.datetime.fromisoformat(timestamp)
result = dt.strftime(format_str)
except ValueError:
result = timestamp
else:
result = str(timestamp) # pragma: no cover
return pybars.strlist([result])
def _default_helper(this, *args):
"""Return a default value if the given value is None or empty."""
if len(args) < 2: # pragma: no cover
return ""
value = args[0]
default_value = args[1]
result = default_value if value is None or value == "" else value
# Use strlist for consistent handling of HTML escaping
return pybars.strlist([str(result)])
def _capitalize_helper(this, *args):
"""Capitalize the first letter of a string."""
if len(args) < 1: # pragma: no cover
return ""
text = args[0]
if not text or not isinstance(text, str): # pragma: no cover
result = ""
else:
result = text.capitalize()
return pybars.strlist([result])
def _round_helper(this, *args):
"""Round a number to the specified number of decimal places."""
if len(args) < 1:
return ""
value = args[0]
decimal_places = args[1] if len(args) > 1 else 2
try:
result = str(round(float(value), int(decimal_places)))
except (ValueError, TypeError):
result = str(value)
return pybars.strlist([result])
def _size_helper(this, *args):
"""Return the size/length of a collection."""
if len(args) < 1:
return 0
value = args[0]
if value is None:
result = "0"
elif isinstance(value, (list, tuple, dict, str)):
result = str(len(value)) # pragma: no cover
else: # pragma: no cover
result = "0"
return pybars.strlist([result])
def _json_helper(this, *args):
"""Convert a value to a JSON string."""
if len(args) < 1: # pragma: no cover
return "{}"
value = args[0]
# For pybars, we need to return a SafeString to prevent HTML escaping
result = json.dumps(value) # pragma: no cover
# Safe string implementation to prevent HTML escaping
return pybars.strlist([result])
def _math_helper(this, *args):
"""Perform basic math operations."""
if len(args) < 3:
return pybars.strlist(["Math error: Insufficient arguments"])
lhs = args[0]
operator = args[1]
rhs = args[2]
try:
lhs = float(lhs)
rhs = float(rhs)
if operator == "+":
result = str(lhs + rhs)
elif operator == "-":
result = str(lhs - rhs)
elif operator == "*":
result = str(lhs * rhs)
elif operator == "/":
result = str(lhs / rhs)
else:
result = f"Unsupported operator: {operator}"
except (ValueError, TypeError) as e:
result = f"Math error: {e}"
return pybars.strlist([result])
def _lt_helper(this, *args):
"""Check if left hand side is less than right hand side."""
if len(args) < 2:
return False
lhs = args[0]
rhs = args[1]
try:
return float(lhs) < float(rhs)
except (ValueError, TypeError):
# Fall back to string comparison for non-numeric values
return str(lhs) < str(rhs)
def _if_cond_helper(this, options, condition):
"""Block helper for custom if conditionals."""
if condition:
return options["fn"](this)
elif "inverse" in options:
return options["inverse"](this)
return "" # pragma: no cover
def _dedent_helper(this, options):
"""Dedent a block of text to remove common leading whitespace.
Usage:
{{#dedent}}
This text will have its
common leading whitespace removed
while preserving relative indentation.
{{/dedent}}
"""
if "fn" not in options: # pragma: no cover
return ""
# Get the content from the block
content = options["fn"](this)
# Convert to string if it's a strlist
if (
isinstance(content, list)
or hasattr(content, "__iter__")
and not isinstance(content, (str, bytes))
):
content_str = "".join(str(item) for item in content) # pragma: no cover
else:
content_str = str(content) # pragma: no cover
# Add trailing and leading newlines to ensure proper dedenting
# This is critical for textwrap.dedent to work correctly with mixed content
content_str = "\n" + content_str + "\n"
# Use textwrap to dedent the content and remove the extra newlines we added
dedented = textwrap.dedent(content_str)[1:-1]
# Return as a SafeString to prevent HTML escaping
return pybars.strlist([dedented]) # pragma: no cover
class TemplateLoader:
"""Loader for Handlebars templates.
This class is responsible for loading templates from disk and rendering
them with the provided context data.
"""
def __init__(self, template_dir: Optional[str] = None):
"""Initialize the template loader.
Args:
template_dir: Optional custom template directory path
"""
self.template_dir = Path(template_dir) if template_dir else TEMPLATES_DIR
self.template_cache: Dict[str, Callable] = {}
self.compiler = pybars.Compiler()
# Set up standard helpers
self.helpers = {
"date": _date_helper,
"default": _default_helper,
"capitalize": _capitalize_helper,
"round": _round_helper,
"size": _size_helper,
"json": _json_helper,
"math": _math_helper,
"lt": _lt_helper,
"if_cond": _if_cond_helper,
"dedent": _dedent_helper,
}
logger.debug(f"Initialized template loader with directory: {self.template_dir}")
def get_template(self, template_path: str) -> Callable:
"""Get a template by path, using cache if available.
Args:
template_path: The path to the template, relative to the templates directory
Returns:
The compiled Handlebars template
Raises:
FileNotFoundError: If the template doesn't exist
"""
if template_path in self.template_cache:
return self.template_cache[template_path]
# Convert from Liquid-style path to Handlebars extension
if template_path.endswith(".liquid"):
template_path = template_path.replace(".liquid", ".hbs")
elif not template_path.endswith(".hbs"):
template_path = f"{template_path}.hbs"
full_path = self.template_dir / template_path
if not full_path.exists():
raise FileNotFoundError(f"Template not found: {full_path}")
with open(full_path, "r", encoding="utf-8") as f:
template_str = f.read()
template = self.compiler.compile(template_str)
self.template_cache[template_path] = template
logger.debug(f"Loaded template: {template_path}")
return template
async def render(self, template_path: str, context: Dict[str, Any]) -> str:
"""Render a template with the given context.
Args:
template_path: The path to the template, relative to the templates directory
context: The context data to pass to the template
Returns:
The rendered template as a string
"""
template = self.get_template(template_path)
return template(context, helpers=self.helpers)
def clear_cache(self) -> None:
"""Clear the template cache."""
self.template_cache.clear()
logger.debug("Template cache cleared")
# Global template loader instance
template_loader = TemplateLoader()
src/basic_memory/cli/app.py
+7 -7
View File
@@ -53,17 +53,17 @@ def app_callback(
importlib.reload(config_module)
# Update the local reference
global config
from basic_memory.config import config as new_config
global app_config
from basic_memory.config import app_config as new_config
config = new_config
app_config = new_config
# Run migrations for every command unless --version was specified
# Run initialization for every command unless --version was specified
if not version and ctx.invoked_subcommand is not None:
from basic_memory.config import config
from basic_memory.services.initialization import ensure_initialize_database
from basic_memory.config import app_config
from basic_memory.services.initialization import ensure_initialization
ensure_initialize_database(config)
ensure_initialization(app_config)
# Register sub-command groups
src/basic_memory/cli/commands/__init__.py
+2 -1
View File
@@ -1,9 +1,10 @@
"""CLI commands for basic-memory."""
from . import status, sync, db, import_memory_json, mcp, import_claude_conversations
from . import auth, status, sync, db, import_memory_json, mcp, import_claude_conversations
from . import import_claude_projects, import_chatgpt, tool, project
__all__ = [
"auth",
"status",
"sync",
"db",
src/basic_memory/cli/commands/auth.py
+136
View File
@@ -0,0 +1,136 @@
"""OAuth management commands."""
import typer
from typing import Optional
from pydantic import AnyHttpUrl
from basic_memory.cli.app import app
from basic_memory.mcp.auth_provider import BasicMemoryOAuthProvider
from mcp.shared.auth import OAuthClientInformationFull
auth_app = typer.Typer(help="OAuth client management commands")
app.add_typer(auth_app, name="auth")
@auth_app.command()
def register_client(
client_id: Optional[str] = typer.Option(
None, help="Client ID (auto-generated if not provided)"
),
client_secret: Optional[str] = typer.Option(
None, help="Client secret (auto-generated if not provided)"
),
issuer_url: str = typer.Option("http://localhost:8000", help="OAuth issuer URL"),
):
"""Register a new OAuth client for Basic Memory MCP server."""
# Create provider instance
provider = BasicMemoryOAuthProvider(issuer_url=issuer_url)
# Create client info with required redirect_uris
client_info = OAuthClientInformationFull(
client_id=client_id or "", # Provider will generate if empty
client_secret=client_secret or "", # Provider will generate if empty
redirect_uris=[AnyHttpUrl("http://localhost:8000/callback")], # Default redirect URI
client_name="Basic Memory OAuth Client",
grant_types=["authorization_code", "refresh_token"],
)
# Register the client
import asyncio
asyncio.run(provider.register_client(client_info))
typer.echo("Client registered successfully!")
typer.echo(f"Client ID: {client_info.client_id}")
typer.echo(f"Client Secret: {client_info.client_secret}")
typer.echo("\nSave these credentials securely - the client secret cannot be retrieved later.")
@auth_app.command()
def test_auth(
issuer_url: str = typer.Option("http://localhost:8000", help="OAuth issuer URL"),
):
"""Test OAuth authentication flow.
IMPORTANT: Use the same FASTMCP_AUTH_SECRET_KEY environment variable
as your MCP server for tokens to validate correctly.
"""
import asyncio
import secrets
from mcp.server.auth.provider import AuthorizationParams
from pydantic import AnyHttpUrl
async def test_flow():
# Create provider with same secret key as server
provider = BasicMemoryOAuthProvider(issuer_url=issuer_url)
# Register a test client
client_info = OAuthClientInformationFull(
client_id=secrets.token_urlsafe(16),
client_secret=secrets.token_urlsafe(32),
redirect_uris=[AnyHttpUrl("http://localhost:8000/callback")],
client_name="Test OAuth Client",
grant_types=["authorization_code", "refresh_token"],
)
await provider.register_client(client_info)
typer.echo(f"Registered test client: {client_info.client_id}")
# Get the client
client = await provider.get_client(client_info.client_id)
if not client:
typer.echo("Error: Client not found after registration", err=True)
return
# Create authorization request
auth_params = AuthorizationParams(
state="test-state",
scopes=["read", "write"],
code_challenge="test-challenge",
redirect_uri=AnyHttpUrl("http://localhost:8000/callback"),
redirect_uri_provided_explicitly=True,
)
# Get authorization URL
auth_url = await provider.authorize(client, auth_params)
typer.echo(f"Authorization URL: {auth_url}")
# Extract auth code from URL
from urllib.parse import urlparse, parse_qs
parsed = urlparse(auth_url)
params = parse_qs(parsed.query)
auth_code = params.get("code", [None])[0]
if not auth_code:
typer.echo("Error: No authorization code in URL", err=True)
return
# Load the authorization code
code_obj = await provider.load_authorization_code(client, auth_code)
if not code_obj:
typer.echo("Error: Invalid authorization code", err=True)
return
# Exchange for tokens
token = await provider.exchange_authorization_code(client, code_obj)
typer.echo(f"Access token: {token.access_token}")
typer.echo(f"Refresh token: {token.refresh_token}")
typer.echo(f"Expires in: {token.expires_in} seconds")
# Validate access token
access_token_obj = await provider.load_access_token(token.access_token)
if access_token_obj:
typer.echo("Access token validated successfully!")
typer.echo(f"Client ID: {access_token_obj.client_id}")
typer.echo(f"Scopes: {access_token_obj.scopes}")
else:
typer.echo("Error: Invalid access token", err=True)
asyncio.run(test_flow())
if __name__ == "__main__":
auth_app()
src/basic_memory/cli/commands/db.py
+3 -3
View File
@@ -7,7 +7,7 @@ from loguru import logger
from basic_memory import db
from basic_memory.cli.app import app
from basic_memory.config import config
from basic_memory.config import app_config
@app.command()
@@ -18,7 +18,7 @@ def reset(
if typer.confirm("This will delete all data in your db. Are you sure?"):
logger.info("Resetting database...")
# Get database path
db_path = config.database_path
db_path = app_config.app_database_path
# Delete the database file if it exists
if db_path.exists():
@@ -26,7 +26,7 @@ def reset(
logger.info(f"Database file deleted: {db_path}")
# Create a new empty database
asyncio.run(db.run_migrations(config))
asyncio.run(db.run_migrations(app_config))
logger.info("Database reset complete")
if reindex:
src/basic_memory/cli/commands/import_chatgpt.py
+29 -205
View File
@@ -2,203 +2,21 @@
import asyncio
import json
from datetime import datetime
from pathlib import Path
from typing import Dict, Any, List, Annotated, Set, Optional
from typing import Annotated
import typer
from basic_memory.cli.app import import_app
from basic_memory.config import config
from basic_memory.importers import ChatGPTImporter
from basic_memory.markdown import EntityParser, MarkdownProcessor
from basic_memory.markdown.schemas import EntityMarkdown, EntityFrontmatter
from loguru import logger
from rich.console import Console
from rich.panel import Panel
from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn
console = Console()
def clean_filename(text: str) -> str:
"""Convert text to safe filename."""
clean = "".join(c if c.isalnum() else "-" for c in text.lower()).strip("-")
return clean
def format_timestamp(ts: float) -> str:
"""Format Unix timestamp for display."""
dt = datetime.fromtimestamp(ts)
return dt.strftime("%Y-%m-%d %H:%M:%S")
def get_message_content(message: Dict[str, Any]) -> str:
"""Extract clean message content."""
if not message or "content" not in message:
return "" # pragma: no cover
content = message["content"]
if content.get("content_type") == "text":
return "\n".join(content.get("parts", []))
elif content.get("content_type") == "code":
return f"```{content.get('language', '')}\n{content.get('text', '')}\n```"
return "" # pragma: no cover
def traverse_messages(
mapping: Dict[str, Any], root_id: Optional[str], seen: Set[str]
) -> List[Dict[str, Any]]:
"""Traverse message tree and return messages in order."""
messages = []
node = mapping.get(root_id) if root_id else None
while node:
if node["id"] not in seen and node.get("message"):
seen.add(node["id"])
messages.append(node["message"])
# Follow children
children = node.get("children", [])
for child_id in children:
child_msgs = traverse_messages(mapping, child_id, seen)
messages.extend(child_msgs)
break # Don't follow siblings
return messages
def format_chat_markdown(
title: str,
mapping: Dict[str, Any],
root_id: Optional[str],
created_at: float,
modified_at: float,
) -> str:
"""Format chat as clean markdown."""
# Start with title
lines = [f"# {title}\n"]
# Traverse message tree
seen_msgs = set()
messages = traverse_messages(mapping, root_id, seen_msgs)
# Format each message
for msg in messages:
# Skip hidden messages
if msg.get("metadata", {}).get("is_visually_hidden_from_conversation"):
continue
# Get author and timestamp
author = msg["author"]["role"].title()
ts = format_timestamp(msg["create_time"]) if msg.get("create_time") else ""
# Add message header
lines.append(f"### {author} ({ts})")
# Add message content
content = get_message_content(msg)
if content:
lines.append(content)
# Add spacing
lines.append("")
return "\n".join(lines)
def format_chat_content(folder: str, conversation: Dict[str, Any]) -> EntityMarkdown:
"""Convert chat conversation to Basic Memory entity."""
# Extract timestamps
created_at = conversation["create_time"]
modified_at = conversation["update_time"]
root_id = None
# Find root message
for node_id, node in conversation["mapping"].items():
if node.get("parent") is None:
root_id = node_id
break
# Generate permalink
date_prefix = datetime.fromtimestamp(created_at).strftime("%Y%m%d")
clean_title = clean_filename(conversation["title"])
# Format content
content = format_chat_markdown(
title=conversation["title"],
mapping=conversation["mapping"],
root_id=root_id,
created_at=created_at,
modified_at=modified_at,
)
# Create entity
entity = EntityMarkdown(
frontmatter=EntityFrontmatter(
metadata={
"type": "conversation",
"title": conversation["title"],
"created": format_timestamp(created_at),
"modified": format_timestamp(modified_at),
"permalink": f"{folder}/{date_prefix}-{clean_title}",
}
),
content=content,
)
return entity
async def process_chatgpt_json(
json_path: Path, folder: str, markdown_processor: MarkdownProcessor
) -> Dict[str, int]:
"""Import conversations from ChatGPT JSON format."""
with Progress(
SpinnerColumn(),
TextColumn("[progress.description]{task.description}"),
BarColumn(),
TextColumn("[progress.percentage]{task.percentage:>3.0f}%"),
console=console,
) as progress:
read_task = progress.add_task("Reading chat data...", total=None)
# Read conversations
conversations = json.loads(json_path.read_text(encoding="utf-8"))
progress.update(read_task, total=len(conversations))
# Process each conversation
messages_imported = 0
chats_imported = 0
for chat in conversations:
# Convert to entity
entity = format_chat_content(folder, chat)
# Write file
file_path = config.home / f"{entity.frontmatter.metadata['permalink']}.md"
# logger.info(f"Writing file: {file_path.absolute()}")
await markdown_processor.write_file(file_path, entity)
# Count messages
msg_count = sum(
1
for node in chat["mapping"].values()
if node.get("message")
and not node.get("message", {})
.get("metadata", {})
.get("is_visually_hidden_from_conversation")
)
chats_imported += 1
messages_imported += msg_count
progress.update(read_task, advance=1)
return {"conversations": chats_imported, "messages": messages_imported}
async def get_markdown_processor() -> MarkdownProcessor:
"""Get MarkdownProcessor instance."""
entity_parser = EntityParser(config.home)
@@ -225,30 +43,36 @@ def import_chatgpt(
"""
try:
if conversations_json:
if not conversations_json.exists():
typer.echo(f"Error: File not found: {conversations_json}", err=True)
raise typer.Exit(1)
if not conversations_json.exists(): # pragma: no cover
typer.echo(f"Error: File not found: {conversations_json}", err=True)
raise typer.Exit(1)
# Get markdown processor
markdown_processor = asyncio.run(get_markdown_processor())
# Get markdown processor
markdown_processor = asyncio.run(get_markdown_processor())
# Process the file
base_path = config.home / folder
console.print(f"\nImporting chats from {conversations_json}...writing to {base_path}")
results = asyncio.run(
process_chatgpt_json(conversations_json, folder, markdown_processor)
)
# Show results
console.print(
Panel(
f"[green]Import complete![/green]\n\n"
f"Imported {results['conversations']} conversations\n"
f"Containing {results['messages']} messages",
expand=False,
)
# Process the file
base_path = config.home / folder
console.print(f"\nImporting chats from {conversations_json}...writing to {base_path}")
# Create importer and run import
importer = ChatGPTImporter(config.home, markdown_processor)
with conversations_json.open("r", encoding="utf-8") as file:
json_data = json.load(file)
result = asyncio.run(importer.import_data(json_data, folder))
if not result.success: # pragma: no cover
typer.echo(f"Error during import: {result.error_message}", err=True)
raise typer.Exit(1)
# Show results
console.print(
Panel(
f"[green]Import complete![/green]\n\n"
f"Imported {result.conversations} conversations\n"
f"Containing {result.messages} messages",
expand=False,
)
)
console.print("\nRun 'basic-memory sync' to index the new files.")
src/basic_memory/cli/commands/import_claude_conversations.py
+16 -142
View File
@@ -2,156 +2,21 @@
import asyncio
import json
from datetime import datetime
from pathlib import Path
from typing import Dict, Any, List, Annotated
from typing import Annotated
import typer
from basic_memory.cli.app import claude_app
from basic_memory.config import config
from basic_memory.importers.claude_conversations_importer import ClaudeConversationsImporter
from basic_memory.markdown import EntityParser, MarkdownProcessor
from basic_memory.markdown.schemas import EntityMarkdown, EntityFrontmatter
from loguru import logger
from rich.console import Console
from rich.panel import Panel
from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn
console = Console()
def clean_filename(text: str) -> str:
"""Convert text to safe filename."""
# Remove invalid characters and convert spaces
clean = "".join(c if c.isalnum() else "-" for c in text.lower()).strip("-")
return clean
def format_timestamp(ts: str) -> str:
"""Format ISO timestamp for display."""
dt = datetime.fromisoformat(ts.replace("Z", "+00:00"))
return dt.strftime("%Y-%m-%d %H:%M:%S")
def format_chat_markdown(
name: str, messages: List[Dict[str, Any]], created_at: str, modified_at: str, permalink: str
) -> str:
"""Format chat as clean markdown."""
# Start with frontmatter and title
lines = [
f"# {name}\n",
]
# Add messages
for msg in messages:
# Format timestamp
ts = format_timestamp(msg["created_at"])
# Add message header
lines.append(f"### {msg['sender'].title()} ({ts})")
# Handle message content
content = msg.get("text", "")
if msg.get("content"):
content = " ".join(c.get("text", "") for c in msg["content"])
lines.append(content)
# Handle attachments
attachments = msg.get("attachments", [])
for attachment in attachments:
if "file_name" in attachment:
lines.append(f"\n**Attachment: {attachment['file_name']}**")
if "extracted_content" in attachment:
lines.append("```")
lines.append(attachment["extracted_content"])
lines.append("```")
# Add spacing between messages
lines.append("")
return "\n".join(lines)
def format_chat_content(
base_path: Path, name: str, messages: List[Dict[str, Any]], created_at: str, modified_at: str
) -> EntityMarkdown:
"""Convert chat messages to Basic Memory entity format."""
# Generate permalink
date_prefix = datetime.fromisoformat(created_at.replace("Z", "+00:00")).strftime("%Y%m%d")
clean_title = clean_filename(name)
permalink = f"{base_path}/{date_prefix}-{clean_title}"
# Format content
content = format_chat_markdown(
name=name,
messages=messages,
created_at=created_at,
modified_at=modified_at,
permalink=permalink,
)
# Create entity
entity = EntityMarkdown(
frontmatter=EntityFrontmatter(
metadata={
"type": "conversation",
"title": name,
"created": created_at,
"modified": modified_at,
"permalink": permalink,
}
),
content=content,
)
return entity
async def process_conversations_json(
json_path: Path, base_path: Path, markdown_processor: MarkdownProcessor
) -> Dict[str, int]:
"""Import chat data from conversations2.json format."""
with Progress(
SpinnerColumn(),
TextColumn("[progress.description]{task.description}"),
BarColumn(),
TextColumn("[progress.percentage]{task.percentage:>3.0f}%"),
console=console,
) as progress:
read_task = progress.add_task("Reading chat data...", total=None)
# Read chat data - handle array of arrays format
data = json.loads(json_path.read_text(encoding="utf-8"))
conversations = [chat for chat in data]
progress.update(read_task, total=len(conversations))
# Process each conversation
messages_imported = 0
chats_imported = 0
for chat in conversations:
# Convert to entity
entity = format_chat_content(
base_path=base_path,
name=chat["name"],
messages=chat["chat_messages"],
created_at=chat["created_at"],
modified_at=chat["updated_at"],
)
# Write file
file_path = Path(f"{entity.frontmatter.metadata['permalink']}.md")
await markdown_processor.write_file(file_path, entity)
chats_imported += 1
messages_imported += len(chat["chat_messages"])
progress.update(read_task, advance=1)
return {"conversations": chats_imported, "messages": messages_imported}
async def get_markdown_processor() -> MarkdownProcessor:
"""Get MarkdownProcessor instance."""
entity_parser = EntityParser(config.home)
@@ -185,19 +50,28 @@ def import_claude(
# Get markdown processor
markdown_processor = asyncio.run(get_markdown_processor())
# Create the importer
importer = ClaudeConversationsImporter(config.home, markdown_processor)
# Process the file
base_path = config.home / folder
console.print(f"\nImporting chats from {conversations_json}...writing to {base_path}")
results = asyncio.run(
process_conversations_json(conversations_json, base_path, markdown_processor)
)
# Run the import
with conversations_json.open("r", encoding="utf-8") as file:
json_data = json.load(file)
result = asyncio.run(importer.import_data(json_data, folder))
if not result.success: # pragma: no cover
typer.echo(f"Error during import: {result.error_message}", err=True)
raise typer.Exit(1)
# Show results
console.print(
Panel(
f"[green]Import complete![/green]\n\n"
f"Imported {results['conversations']} conversations\n"
f"Containing {results['messages']} messages",
f"Imported {result.conversations} conversations\n"
f"Containing {result.messages} messages",
expand=False,
)
)
src/basic_memory/cli/commands/import_claude_projects.py
+31 -141
View File
@@ -3,138 +3,20 @@
import asyncio
import json
from pathlib import Path
from typing import Dict, Any, Annotated, Optional
from typing import Annotated
import typer
from basic_memory.cli.app import claude_app
from basic_memory.config import config
from basic_memory.importers.claude_projects_importer import ClaudeProjectsImporter
from basic_memory.markdown import EntityParser, MarkdownProcessor
from basic_memory.markdown.schemas import EntityMarkdown, EntityFrontmatter
from loguru import logger
from rich.console import Console
from rich.panel import Panel
from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn
console = Console()
def clean_filename(text: str) -> str:
"""Convert text to safe filename."""
clean = "".join(c if c.isalnum() else "-" for c in text.lower()).strip("-")
return clean
def format_project_markdown(project: Dict[str, Any], doc: Dict[str, Any]) -> EntityMarkdown:
"""Format a project document as a Basic Memory entity."""
# Extract timestamps
created_at = doc.get("created_at") or project["created_at"]
modified_at = project["updated_at"]
# Generate clean names for organization
project_dir = clean_filename(project["name"])
doc_file = clean_filename(doc["filename"])
# Create entity
entity = EntityMarkdown(
frontmatter=EntityFrontmatter(
metadata={
"type": "project_doc",
"title": doc["filename"],
"created": created_at,
"modified": modified_at,
"permalink": f"{project_dir}/docs/{doc_file}",
"project_name": project["name"],
"project_uuid": project["uuid"],
"doc_uuid": doc["uuid"],
}
),
content=doc["content"],
)
return entity
def format_prompt_markdown(project: Dict[str, Any]) -> Optional[EntityMarkdown]:
"""Format project prompt template as a Basic Memory entity."""
if not project.get("prompt_template"):
return None
# Extract timestamps
created_at = project["created_at"]
modified_at = project["updated_at"]
# Generate clean project directory name
project_dir = clean_filename(project["name"])
# Create entity
entity = EntityMarkdown(
frontmatter=EntityFrontmatter(
metadata={
"type": "prompt_template",
"title": f"Prompt Template: {project['name']}",
"created": created_at,
"modified": modified_at,
"permalink": f"{project_dir}/prompt-template",
"project_name": project["name"],
"project_uuid": project["uuid"],
}
),
content=f"# Prompt Template: {project['name']}\n\n{project['prompt_template']}",
)
return entity
async def process_projects_json(
json_path: Path, base_path: Path, markdown_processor: MarkdownProcessor
) -> Dict[str, int]:
"""Import project data from Claude.ai projects.json format."""
with Progress(
SpinnerColumn(),
TextColumn("[progress.description]{task.description}"),
BarColumn(),
TextColumn("[progress.percentage]{task.percentage:>3.0f}%"),
console=console,
) as progress:
read_task = progress.add_task("Reading project data...", total=None)
# Read project data
data = json.loads(json_path.read_text(encoding="utf-8"))
progress.update(read_task, total=len(data))
# Track import counts
docs_imported = 0
prompts_imported = 0
# Process each project
for project in data:
project_dir = clean_filename(project["name"])
# Create project directories
docs_dir = base_path / project_dir / "docs"
docs_dir.mkdir(parents=True, exist_ok=True)
# Import prompt template if it exists
if prompt_entity := format_prompt_markdown(project):
file_path = base_path / f"{prompt_entity.frontmatter.metadata['permalink']}.md"
await markdown_processor.write_file(file_path, prompt_entity)
prompts_imported += 1
# Import project documents
for doc in project.get("docs", []):
entity = format_project_markdown(project, doc)
file_path = base_path / f"{entity.frontmatter.metadata['permalink']}.md"
await markdown_processor.write_file(file_path, entity)
docs_imported += 1
progress.update(read_task, advance=1)
return {"documents": docs_imported, "prompts": prompts_imported}
async def get_markdown_processor() -> MarkdownProcessor:
"""Get MarkdownProcessor instance."""
entity_parser = EntityParser(config.home)
@@ -160,30 +42,38 @@ def import_projects(
After importing, run 'basic-memory sync' to index the new files.
"""
try:
if projects_json:
if not projects_json.exists():
typer.echo(f"Error: File not found: {projects_json}", err=True)
raise typer.Exit(1)
if not projects_json.exists():
typer.echo(f"Error: File not found: {projects_json}", err=True)
raise typer.Exit(1)
# Get markdown processor
markdown_processor = asyncio.run(get_markdown_processor())
# Get markdown processor
markdown_processor = asyncio.run(get_markdown_processor())
# Process the file
base_path = config.home / base_folder if base_folder else config.home
console.print(f"\nImporting projects from {projects_json}...writing to {base_path}")
results = asyncio.run(
process_projects_json(projects_json, base_path, markdown_processor)
)
# Show results
console.print(
Panel(
f"[green]Import complete![/green]\n\n"
f"Imported {results['documents']} project documents\n"
f"Imported {results['prompts']} prompt templates",
expand=False,
)
# Create the importer
importer = ClaudeProjectsImporter(config.home, markdown_processor)
# Process the file
base_path = config.home / base_folder if base_folder else config.home
console.print(f"\nImporting projects from {projects_json}...writing to {base_path}")
# Run the import
with projects_json.open("r", encoding="utf-8") as file:
json_data = json.load(file)
result = asyncio.run(importer.import_data(json_data, base_folder))
if not result.success: # pragma: no cover
typer.echo(f"Error during import: {result.error_message}", err=True)
raise typer.Exit(1)
# Show results
console.print(
Panel(
f"[green]Import complete![/green]\n\n"
f"Imported {result.documents} project documents\n"
f"Imported {result.prompts} prompt templates",
expand=False,
)
)
console.print("\nRun 'basic-memory sync' to index the new files.")
src/basic_memory/cli/commands/import_memory_json.py
+26 -83
View File
@@ -3,94 +3,20 @@
import asyncio
import json
from pathlib import Path
from typing import Dict, Any, List, Annotated
from typing import Annotated
import typer
from basic_memory.cli.app import import_app
from basic_memory.config import config
from basic_memory.importers.memory_json_importer import MemoryJsonImporter
from basic_memory.markdown import EntityParser, MarkdownProcessor
from loguru import logger
from rich.console import Console
from rich.panel import Panel
from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn
from basic_memory.cli.app import import_app
from basic_memory.config import config
from basic_memory.markdown import EntityParser, MarkdownProcessor
from basic_memory.markdown.schemas import EntityMarkdown, EntityFrontmatter, Observation, Relation
console = Console()
async def process_memory_json(
json_path: Path, base_path: Path, markdown_processor: MarkdownProcessor
):
"""Import entities from memory.json using markdown processor."""
# First pass - collect all relations by source entity
entity_relations: Dict[str, List[Relation]] = {}
entities: Dict[str, Dict[str, Any]] = {}
with Progress(
SpinnerColumn(),
TextColumn("[progress.description]{task.description}"),
BarColumn(),
TextColumn("[progress.percentage]{task.percentage:>3.0f}%"),
console=console,
) as progress:
read_task = progress.add_task("Reading memory.json...", total=None)
# First pass - collect entities and relations
with open(json_path, encoding="utf-8") as f:
lines = f.readlines()
progress.update(read_task, total=len(lines))
for line in lines:
data = json.loads(line)
if data["type"] == "entity":
entities[data["name"]] = data
elif data["type"] == "relation":
# Store relation with its source entity
source = data.get("from") or data.get("from_id")
if source not in entity_relations:
entity_relations[source] = []
entity_relations[source].append(
Relation(
type=data.get("relationType") or data.get("relation_type"),
target=data.get("to") or data.get("to_id"),
)
)
progress.update(read_task, advance=1)
# Second pass - create and write entities
write_task = progress.add_task("Creating entities...", total=len(entities))
entities_created = 0
for name, entity_data in entities.items():
entity = EntityMarkdown(
frontmatter=EntityFrontmatter(
metadata={
"type": entity_data["entityType"],
"title": name,
"permalink": f"{entity_data['entityType']}/{name}",
}
),
content=f"# {name}\n",
observations=[Observation(content=obs) for obs in entity_data["observations"]],
relations=entity_relations.get(
name, []
), # Add any relations where this entity is the source
)
# Let markdown processor handle writing
file_path = base_path / f"{entity_data['entityType']}/{name}.md"
await markdown_processor.write_file(file_path, entity)
entities_created += 1
progress.update(write_task, advance=1)
return {
"entities": entities_created,
"relations": sum(len(rels) for rels in entity_relations.values()),
}
async def get_markdown_processor() -> MarkdownProcessor:
"""Get MarkdownProcessor instance."""
entity_parser = EntityParser(config.home)
@@ -102,6 +28,9 @@ def memory_json(
json_path: Annotated[Path, typer.Argument(..., help="Path to memory.json file")] = Path(
"memory.json"
),
destination_folder: Annotated[
str, typer.Option(help="Optional destination folder within the project")
] = "",
):
"""Import entities and relations from a memory.json file.
@@ -121,17 +50,31 @@ def memory_json(
# Get markdown processor
markdown_processor = asyncio.run(get_markdown_processor())
# Create the importer
importer = MemoryJsonImporter(config.home, markdown_processor)
# Process the file
base_path = config.home
base_path = config.home if not destination_folder else config.home / destination_folder
console.print(f"\nImporting from {json_path}...writing to {base_path}")
results = asyncio.run(process_memory_json(json_path, base_path, markdown_processor))
# Run the import for json log format
file_data = []
with json_path.open("r", encoding="utf-8") as file:
for line in file:
json_data = json.loads(line)
file_data.append(json_data)
result = asyncio.run(importer.import_data(file_data, destination_folder))
if not result.success: # pragma: no cover
typer.echo(f"Error during import: {result.error_message}", err=True)
raise typer.Exit(1)
# Show results
console.print(
Panel(
f"[green]Import complete![/green]\n\n"
f"Created {results['entities']} entities\n"
f"Added {results['relations']} relations",
f"Created {result.entities} entities\n"
f"Added {result.relations} relations",
expand=False,
)
)
src/basic_memory/cli/commands/mcp.py
+71 -18
View File
@@ -1,6 +1,8 @@
"""MCP server command."""
"""MCP server command with streamable HTTP transport."""
import asyncio
import typer
import basic_memory
from basic_memory.cli.app import app
# Import mcp instance
@@ -9,27 +11,78 @@ from basic_memory.mcp.server import mcp as mcp_server # pragma: no cover
# Import mcp tools to register them
import basic_memory.mcp.tools # noqa: F401 # pragma: no cover
# Import prompts to register them
import basic_memory.mcp.prompts # noqa: F401 # pragma: no cover
from loguru import logger
@app.command()
def mcp(): # pragma: no cover
"""Run the MCP server"""
from basic_memory.config import config
import asyncio
from basic_memory.services.initialization import initialize_database
def mcp(
transport: str = typer.Option("stdio", help="Transport type: stdio, streamable-http, or sse"),
host: str = typer.Option(
"0.0.0.0", help="Host for HTTP transports (use 0.0.0.0 to allow external connections)"
),
port: int = typer.Option(8000, help="Port for HTTP transports"),
path: str = typer.Option("/mcp", help="Path prefix for streamable-http transport"),
): # pragma: no cover
"""Run the MCP server with configurable transport options.
# First, run just the database migrations synchronously
asyncio.run(initialize_database(config))
This command starts an MCP server using one of three transport options:
# Load config to check if sync is enabled
from basic_memory.config import config_manager
- stdio: Standard I/O (good for local usage)
- streamable-http: Recommended for web deployments (default)
- sse: Server-Sent Events (for compatibility with existing clients)
"""
basic_memory_config = config_manager.load_config()
# Check if OAuth is enabled
import os
if basic_memory_config.sync_changes:
# For now, we'll just log that sync will be handled by the MCP server
from loguru import logger
auth_enabled = os.getenv("FASTMCP_AUTH_ENABLED", "false").lower() == "true"
if auth_enabled:
logger.info("OAuth authentication is ENABLED")
logger.info(f"Issuer URL: {os.getenv('FASTMCP_AUTH_ISSUER_URL', 'http://localhost:8000')}")
if os.getenv("FASTMCP_AUTH_REQUIRED_SCOPES"):
logger.info(f"Required scopes: {os.getenv('FASTMCP_AUTH_REQUIRED_SCOPES')}")
else:
logger.info("OAuth authentication is DISABLED")
logger.info("File sync will be handled by the MCP server")
from basic_memory.config import app_config
from basic_memory.services.initialization import initialize_file_sync
# Start the MCP server
mcp_server.run()
# Start the MCP server with the specified transport
# Use unified thread-based sync approach for both transports
import threading
def run_file_sync():
"""Run file sync in a separate thread with its own event loop."""
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
loop.run_until_complete(initialize_file_sync(app_config))
except Exception as e:
logger.error(f"File sync error: {e}", err=True)
finally:
loop.close()
logger.info(f"Sync changes enabled: {app_config.sync_changes}")
if app_config.sync_changes:
# Start the sync thread
sync_thread = threading.Thread(target=run_file_sync, daemon=True)
sync_thread.start()
logger.info("Started file sync in background")
# Now run the MCP server (blocks)
logger.info(f"Starting MCP server with {transport.upper()} transport")
if transport == "stdio":
mcp_server.run(
transport=transport,
)
elif transport == "streamable-http" or transport == "sse":
mcp_server.run(
transport=transport,
host=host,
port=port,
path=path,
)
src/basic_memory/cli/commands/project.py
+124 -59
View File
@@ -9,13 +9,20 @@ from rich.console import Console
from rich.table import Table
from basic_memory.cli.app import app
from basic_memory.config import ConfigManager, config
from basic_memory.config import config
from basic_memory.mcp.tools.project_info import project_info
import json
from datetime import datetime
from rich.panel import Panel
from rich.tree import Tree
from basic_memory.mcp.async_client import client
from basic_memory.mcp.tools.utils import call_get
from basic_memory.schemas.project_info import ProjectList
from basic_memory.mcp.tools.utils import call_post
from basic_memory.schemas.project_info import ProjectStatusResponse
from basic_memory.mcp.tools.utils import call_delete
from basic_memory.mcp.tools.utils import call_put
console = Console()
@@ -35,105 +42,162 @@ def format_path(path: str) -> str:
@project_app.command("list")
def list_projects() -> None:
"""List all configured projects."""
config_manager = ConfigManager()
projects = config_manager.projects
# Use API to list projects
table = Table(title="Basic Memory Projects")
table.add_column("Name", style="cyan")
table.add_column("Path", style="green")
table.add_column("Default", style="yellow")
table.add_column("Active", style="magenta")
project_url = config.project_url
default_project = config_manager.default_project
active_project = config.project
try:
response = asyncio.run(call_get(client, f"{project_url}/project/projects"))
result = ProjectList.model_validate(response.json())
for name, path in projects.items():
is_default = "✓" if name == default_project else ""
is_active = "✓" if name == active_project else ""
table.add_row(name, format_path(path), is_default, is_active)
table = Table(title="Basic Memory Projects")
table.add_column("Name", style="cyan")
table.add_column("Path", style="green")
table.add_column("Default", style="yellow")
table.add_column("Active", style="magenta")
console.print(table)
for project in result.projects:
is_default = "✓" if project.is_default else ""
is_active = "✓" if project.is_current else ""
table.add_row(project.name, format_path(project.path), is_default, is_active)
console.print(table)
except Exception as e:
console.print(f"[red]Error listing projects: {str(e)}[/red]")
console.print("[yellow]Note: Make sure the Basic Memory server is running.[/yellow]")
raise typer.Exit(1)
@project_app.command("add")
def add_project(
name: str = typer.Argument(..., help="Name of the project"),
path: str = typer.Argument(..., help="Path to the project directory"),
set_default: bool = typer.Option(False, "--default", help="Set as default project"),
) -> None:
"""Add a new project."""
config_manager = ConfigManager()
# Resolve to absolute path
resolved_path = os.path.abspath(os.path.expanduser(path))
try:
# Resolve to absolute path
resolved_path = os.path.abspath(os.path.expanduser(path))
config_manager.add_project(name, resolved_path)
console.print(f"[green]Project '{name}' added at {format_path(resolved_path)}[/green]")
project_url = config.project_url
data = {"name": name, "path": resolved_path, "set_default": set_default}
# Display usage hint
console.print("\nTo use this project:")
console.print(f" basic-memory --project={name} <command>")
console.print(" # or")
console.print(f" basic-memory project default {name}")
except ValueError as e:
console.print(f"[red]Error: {e}[/red]")
response = asyncio.run(call_post(client, f"{project_url}/project/projects", json=data))
result = ProjectStatusResponse.model_validate(response.json())
console.print(f"[green]{result.message}[/green]")
except Exception as e:
console.print(f"[red]Error adding project: {str(e)}[/red]")
console.print("[yellow]Note: Make sure the Basic Memory server is running.[/yellow]")
raise typer.Exit(1)
# Display usage hint
console.print("\nTo use this project:")
console.print(f" basic-memory --project={name} <command>")
console.print(" # or")
console.print(f" basic-memory project default {name}")
@project_app.command("remove")
def remove_project(
name: str = typer.Argument(..., help="Name of the project to remove"),
) -> None:
"""Remove a project from configuration."""
config_manager = ConfigManager()
try:
config_manager.remove_project(name)
console.print(f"[green]Project '{name}' removed from configuration[/green]")
console.print("[yellow]Note: The project files have not been deleted from disk.[/yellow]")
except ValueError as e: # pragma: no cover
console.print(f"[red]Error: {e}[/red]")
project_url = config.project_url
response = asyncio.run(call_delete(client, f"{project_url}/project/projects/{name}"))
result = ProjectStatusResponse.model_validate(response.json())
console.print(f"[green]{result.message}[/green]")
except Exception as e:
console.print(f"[red]Error removing project: {str(e)}[/red]")
console.print("[yellow]Note: Make sure the Basic Memory server is running.[/yellow]")
raise typer.Exit(1)
# Show this message regardless of method used
console.print("[yellow]Note: The project files have not been deleted from disk.[/yellow]")
@project_app.command("default")
def set_default_project(
name: str = typer.Argument(..., help="Name of the project to set as default"),
) -> None:
"""Set the default project and activate it for the current session."""
config_manager = ConfigManager()
try:
# Set the default project
config_manager.set_default_project(name)
project_url = config.project_url
# Also activate it for the current session by setting the environment variable
os.environ["BASIC_MEMORY_PROJECT"] = name
response = asyncio.run(call_put(client, f"{project_url}/project/projects/{name}/default"))
result = ProjectStatusResponse.model_validate(response.json())
# Reload configuration to apply the change
from importlib import reload
from basic_memory import config as config_module
reload(config_module)
console.print(f"[green]Project '{name}' set as default and activated[/green]")
except ValueError as e: # pragma: no cover
console.print(f"[red]Error: {e}[/red]")
console.print(f"[green]{result.message}[/green]")
except Exception as e:
console.print(f"[red]Error setting default project: {str(e)}[/red]")
console.print("[yellow]Note: Make sure the Basic Memory server is running.[/yellow]")
raise typer.Exit(1)
# Always activate it for the current session
os.environ["BASIC_MEMORY_PROJECT"] = name
# Reload configuration to apply the change
from importlib import reload
from basic_memory import config as config_module
reload(config_module)
console.print("[green]Project activated for current session[/green]")
@project_app.command("current")
def show_current_project() -> None:
"""Show the current project."""
config_manager = ConfigManager()
current = os.environ.get("BASIC_MEMORY_PROJECT", config_manager.default_project)
# Use API to get current project
project_url = config.project_url
try:
path = config_manager.get_project_path(current)
console.print(f"Current project: [cyan]{current}[/cyan]")
console.print(f"Path: [green]{format_path(str(path))}[/green]")
console.print(f"Database: [blue]{format_path(str(config.database_path))}[/blue]")
except ValueError: # pragma: no cover
console.print(f"[yellow]Warning: Project '{current}' not found in configuration[/yellow]")
console.print(f"Using default project: [cyan]{config_manager.default_project}[/cyan]")
response = asyncio.run(call_get(client, f"{project_url}/project/projects"))
result = ProjectList.model_validate(response.json())
# Find the current project from the API response
current_project = result.current_project
default_project = result.default_project
# Find the project details in the list
for project in result.projects:
if project.name == current_project:
console.print(f"Current project: [cyan]{project.name}[/cyan]")
console.print(f"Path: [green]{format_path(project.path)}[/green]")
# Use app_config for database_path, not project config
from basic_memory.config import app_config
console.print(
f"Database: [blue]{format_path(str(app_config.app_database_path))}[/blue]"
)
console.print(f"Default project: [yellow]{default_project}[/yellow]")
break
except Exception as e:
console.print(f"[red]Error getting current project: {str(e)}[/red]")
console.print("[yellow]Note: Make sure the Basic Memory server is running.[/yellow]")
raise typer.Exit(1)
@project_app.command("sync")
def synchronize_projects() -> None:
"""Synchronize projects between configuration file and database."""
# Call the API to synchronize projects
project_url = config.project_url
try:
response = asyncio.run(call_post(client, f"{project_url}/project/sync"))
result = ProjectStatusResponse.model_validate(response.json())
console.print(f"[green]{result.message}[/green]")
except Exception as e: # pragma: no cover
console.print(f"[red]Error synchronizing projects: {str(e)}[/red]")
console.print("[yellow]Note: Make sure the Basic Memory server is running.[/yellow]")
raise typer.Exit(1)
@project_app.command("info")
@@ -266,9 +330,10 @@ def display_project_info(
projects_table.add_column("Path", style="cyan")
projects_table.add_column("Default", style="green")
for name, path in info.available_projects.items():
for name, proj_info in info.available_projects.items():
is_default = name == info.default_project
projects_table.add_row(name, path, "✓" if is_default else "")
project_path = proj_info["path"]
projects_table.add_row(name, project_path, "✓" if is_default else "")
console.print(projects_table)
src/basic_memory/cli/commands/status.py
+19 -9
View File
@@ -9,10 +9,11 @@ from rich.console import Console
from rich.panel import Panel
from rich.tree import Tree
from basic_memory import db
from basic_memory.cli.app import app
from basic_memory.cli.commands.sync import get_sync_service
from basic_memory.config import config
from basic_memory.sync import SyncService
from basic_memory.config import config, app_config
from basic_memory.repository import ProjectRepository
from basic_memory.sync.sync_service import SyncReport
# Create rich console
@@ -86,9 +87,9 @@ def build_directory_summary(counts: Dict[str, int]) -> str:
return " ".join(parts)
def display_changes(title: str, changes: SyncReport, verbose: bool = False):
def display_changes(project_name: str, title: str, changes: SyncReport, verbose: bool = False):
"""Display changes using Rich for better visualization."""
tree = Tree(title)
tree = Tree(f"{project_name}: {title}")
if changes.total == 0:
tree.add("No changes")
@@ -121,11 +122,21 @@ def display_changes(title: str, changes: SyncReport, verbose: bool = False):
console.print(Panel(tree, expand=False))
async def run_status(sync_service: SyncService, verbose: bool = False):
async def run_status(verbose: bool = False):
"""Check sync status of files vs database."""
# Check knowledge/ directory
_, session_maker = await db.get_or_create_db(
db_path=app_config.database_path, db_type=db.DatabaseType.FILESYSTEM
)
project_repository = ProjectRepository(session_maker)
project = await project_repository.get_by_name(config.project)
if not project: # pragma: no cover
raise Exception(f"Project '{config.project}' not found")
sync_service = await get_sync_service(project)
knowledge_changes = await sync_service.scan(config.home)
display_changes("Status", knowledge_changes, verbose)
display_changes(project.name, "Status", knowledge_changes, verbose)
@app.command()
@@ -134,9 +145,8 @@ def status(
):
"""Show sync status between files and database."""
try:
sync_service = asyncio.run(get_sync_service())
asyncio.run(run_status(sync_service, verbose)) # pragma: no cover
asyncio.run(run_status(verbose)) # pragma: no cover
except Exception as e:
logger.exception(f"Error checking status: {e}")
logger.error(f"Error checking status: {e}")
typer.echo(f"Error checking status: {e}", err=True)
raise typer.Exit(code=1) # pragma: no cover
src/basic_memory/cli/commands/sync.py
+44 -58
View File
@@ -16,10 +16,12 @@ from basic_memory.cli.app import app
from basic_memory.config import config
from basic_memory.markdown import EntityParser
from basic_memory.markdown.markdown_processor import MarkdownProcessor
from basic_memory.models import Project
from basic_memory.repository import (
EntityRepository,
ObservationRepository,
RelationRepository,
ProjectRepository,
)
from basic_memory.repository.search_repository import SearchRepository
from basic_memory.services import EntityService, FileService
@@ -27,7 +29,7 @@ from basic_memory.services.link_resolver import LinkResolver
from basic_memory.services.search_service import SearchService
from basic_memory.sync import SyncService
from basic_memory.sync.sync_service import SyncReport
from basic_memory.sync.watch_service import WatchService
from basic_memory.config import app_config
console = Console()
@@ -38,21 +40,22 @@ class ValidationIssue:
error: str
async def get_sync_service(): # pragma: no cover
async def get_sync_service(project: Project) -> SyncService: # pragma: no cover
"""Get sync service instance with all dependencies."""
_, session_maker = await db.get_or_create_db(
db_path=config.database_path, db_type=db.DatabaseType.FILESYSTEM
db_path=app_config.database_path, db_type=db.DatabaseType.FILESYSTEM
)
entity_parser = EntityParser(config.home)
project_path = Path(project.path)
entity_parser = EntityParser(project_path)
markdown_processor = MarkdownProcessor(entity_parser)
file_service = FileService(config.home, markdown_processor)
file_service = FileService(project_path, markdown_processor)
# Initialize repositories
entity_repository = EntityRepository(session_maker)
observation_repository = ObservationRepository(session_maker)
relation_repository = RelationRepository(session_maker)
search_repository = SearchRepository(session_maker)
entity_repository = EntityRepository(session_maker, project_id=project.id)
observation_repository = ObservationRepository(session_maker, project_id=project.id)
relation_repository = RelationRepository(session_maker, project_id=project.id)
search_repository = SearchRepository(session_maker, project_id=project.id)
# Initialize services
search_service = SearchService(search_repository, entity_repository, file_service)
@@ -70,7 +73,7 @@ async def get_sync_service(): # pragma: no cover
# Create sync service
sync_service = SyncService(
config=config,
app_config=app_config,
entity_service=entity_service,
entity_parser=entity_parser,
entity_repository=entity_repository,
@@ -153,8 +156,16 @@ def display_detailed_sync_results(knowledge: SyncReport):
console.print(knowledge_tree)
async def run_sync(verbose: bool = False, watch: bool = False, console_status: bool = False):
async def run_sync(verbose: bool = False):
"""Run sync operation."""
_, session_maker = await db.get_or_create_db(
db_path=app_config.database_path, db_type=db.DatabaseType.FILESYSTEM
)
project_repository = ProjectRepository(session_maker)
project = await project_repository.get_by_name(config.project)
if not project: # pragma: no cover
raise Exception(f"Project '{config.project}' not found")
import time
start_time = time.time()
@@ -162,50 +173,33 @@ async def run_sync(verbose: bool = False, watch: bool = False, console_status: b
logger.info(
"Sync command started",
project=config.project,
watch_mode=watch,
verbose=verbose,
directory=str(config.home),
)
sync_service = await get_sync_service()
sync_service = await get_sync_service(project)
# Start watching if requested
if watch:
logger.info("Starting watch service after initial sync")
watch_service = WatchService(
sync_service=sync_service,
file_service=sync_service.entity_service.file_service,
config=config,
)
logger.info("Running one-time sync")
knowledge_changes = await sync_service.sync(config.home)
# full sync - no progress bars in watch mode
await sync_service.sync(config.home)
# Log results
duration_ms = int((time.time() - start_time) * 1000)
logger.info(
"Sync command completed",
project=config.project,
total_changes=knowledge_changes.total,
new_files=len(knowledge_changes.new),
modified_files=len(knowledge_changes.modified),
deleted_files=len(knowledge_changes.deleted),
moved_files=len(knowledge_changes.moves),
duration_ms=duration_ms,
)
# watch changes
await watch_service.run() # pragma: no cover
# Display results
if verbose:
display_detailed_sync_results(knowledge_changes)
else:
# one time sync
logger.info("Running one-time sync")
knowledge_changes = await sync_service.sync(config.home)
# Log results
duration_ms = int((time.time() - start_time) * 1000)
logger.info(
"Sync command completed",
project=config.project,
total_changes=knowledge_changes.total,
new_files=len(knowledge_changes.new),
modified_files=len(knowledge_changes.modified),
deleted_files=len(knowledge_changes.deleted),
moved_files=len(knowledge_changes.moves),
duration_ms=duration_ms,
)
# Display results
if verbose:
display_detailed_sync_results(knowledge_changes)
else:
display_sync_summary(knowledge_changes) # pragma: no cover
display_sync_summary(knowledge_changes) # pragma: no cover
@app.command()
@@ -216,22 +210,15 @@ def sync(
"-v",
help="Show detailed sync information.",
),
watch: bool = typer.Option(
False,
"--watch",
"-w",
help="Start watching for changes after sync.",
),
) -> None:
"""Sync knowledge files with the database."""
try:
# Show which project we're syncing
if not watch: # Don't show in watch mode as it would break the UI
typer.echo(f"Syncing project: {config.project}")
typer.echo(f"Project path: {config.home}")
typer.echo(f"Syncing project: {config.project}")
typer.echo(f"Project path: {config.home}")
# Run sync
asyncio.run(run_sync(verbose=verbose, watch=watch))
asyncio.run(run_sync(verbose=verbose))
except Exception as e: # pragma: no cover
if not isinstance(e, typer.Exit):
@@ -240,7 +227,6 @@ def sync(
f"project={config.project},"
f"error={str(e)},"
f"error_type={type(e).__name__},"
f"watch_mode={watch},"
f"directory={str(config.home)}",
)
typer.echo(f"Error during sync: {e}", err=True)
src/basic_memory/cli/main.py
+1 -5
View File
@@ -4,6 +4,7 @@ from basic_memory.cli.app import app # pragma: no cover
# Register commands
from basic_memory.cli.commands import ( # noqa: F401 # pragma: no cover
auth,
db,
import_chatgpt,
import_claude_conversations,
@@ -15,12 +16,7 @@ from basic_memory.cli.commands import ( # noqa: F401 # pragma: no cover
sync,
tool,
)
from basic_memory.config import config
from basic_memory.services.initialization import ensure_initialization
if __name__ == "__main__": # pragma: no cover
# Run initialization if we are starting as a module
ensure_initialization(config)
# start the app
app()
src/basic_memory/config.py
+109 -86
View File
@@ -2,76 +2,47 @@
import json
import os
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Dict, Literal, Optional
from typing import Any, Dict, Literal, Optional, List
from loguru import logger
from pydantic import Field, field_validator
from pydantic_settings import BaseSettings, SettingsConfigDict
import basic_memory
from basic_memory.utils import setup_logging
from basic_memory.utils import setup_logging, generate_permalink
DATABASE_NAME = "memory.db"
APP_DATABASE_NAME = "memory.db" # Using the same name but in the app directory
DATA_DIR_NAME = ".basic-memory"
CONFIG_FILE_NAME = "config.json"
WATCH_STATUS_JSON = "watch-status.json"
Environment = Literal["test", "dev", "user"]
class ProjectConfig(BaseSettings):
@dataclass
class ProjectConfig:
"""Configuration for a specific basic-memory project."""
env: Environment = Field(default="dev", description="Environment name")
# Default to ~/basic-memory but allow override with env var: BASIC_MEMORY_HOME
home: Path = Field(
default_factory=lambda: Path.home() / "basic-memory",
description="Base path for basic-memory files",
)
# Name of the project
project: str = Field(default="default", description="Project name")
# Watch service configuration
sync_delay: int = Field(
default=1000, description="Milliseconds to wait after changes before syncing", gt=0
)
# update permalinks on move
update_permalinks_on_move: bool = Field(
default=False,
description="Whether to update permalinks when files are moved or renamed. default (False)",
)
model_config = SettingsConfigDict(
env_prefix="BASIC_MEMORY_",
extra="ignore",
env_file=".env",
env_file_encoding="utf-8",
)
name: str
home: Path
@property
def database_path(self) -> Path:
"""Get SQLite database path."""
database_path = self.home / DATA_DIR_NAME / DATABASE_NAME
if not database_path.exists():
database_path.parent.mkdir(parents=True, exist_ok=True)
database_path.touch()
return database_path
def project(self):
return self.name
@field_validator("home")
@classmethod
def ensure_path_exists(cls, v: Path) -> Path: # pragma: no cover
"""Ensure project path exists."""
if not v.exists():
v.mkdir(parents=True)
return v
@property
def project_url(self) -> str: # pragma: no cover
return f"/{generate_permalink(self.name)}"
class BasicMemoryConfig(BaseSettings):
"""Pydantic model for Basic Memory global configuration."""
env: Environment = Field(default="dev", description="Environment name")
projects: Dict[str, str] = Field(
default_factory=lambda: {"main": str(Path.home() / "basic-memory")},
description="Mapping of project names to their filesystem paths",
@@ -81,8 +52,15 @@ class BasicMemoryConfig(BaseSettings):
description="Name of the default project to use",
)
# overridden by ~/.basic-memory/config.json
log_level: str = "INFO"
# Watch service configuration
sync_delay: int = Field(
default=1000, description="Milliseconds to wait after changes before syncing", gt=0
)
# update permalinks on move
update_permalinks_on_move: bool = Field(
default=False,
description="Whether to update permalinks when files are moved or renamed. default (False)",
@@ -96,18 +74,73 @@ class BasicMemoryConfig(BaseSettings):
model_config = SettingsConfigDict(
env_prefix="BASIC_MEMORY_",
extra="ignore",
env_file=".env",
env_file_encoding="utf-8",
)
def get_project_path(self, project_name: Optional[str] = None) -> Path: # pragma: no cover
"""Get the path for a specific project or the default project."""
name = project_name or self.default_project
if name not in self.projects:
raise ValueError(f"Project '{name}' not found in configuration")
return Path(self.projects[name])
def model_post_init(self, __context: Any) -> None:
"""Ensure configuration is valid after initialization."""
# Ensure main project exists
if "main" not in self.projects:
if "main" not in self.projects: # pragma: no cover
self.projects["main"] = str(Path.home() / "basic-memory")
# Ensure default project is valid
if self.default_project not in self.projects:
if self.default_project not in self.projects: # pragma: no cover
self.default_project = "main"
@property
def app_database_path(self) -> Path:
"""Get the path to the app-level database.
This is the single database that will store all knowledge data
across all projects.
"""
database_path = Path.home() / DATA_DIR_NAME / APP_DATABASE_NAME
if not database_path.exists(): # pragma: no cover
database_path.parent.mkdir(parents=True, exist_ok=True)
database_path.touch()
return database_path
@property
def database_path(self) -> Path:
"""Get SQLite database path.
Rreturns the app-level database path
for backward compatibility in the codebase.
"""
# Load the app-level database path from the global config
config = config_manager.load_config() # pragma: no cover
return config.app_database_path # pragma: no cover
@property
def project_list(self) -> List[ProjectConfig]: # pragma: no cover
"""Get all configured projects as ProjectConfig objects."""
return [ProjectConfig(name=name, home=Path(path)) for name, path in self.projects.items()]
@field_validator("projects")
@classmethod
def ensure_project_paths_exists(cls, v: Dict[str, str]) -> Dict[str, str]: # pragma: no cover
"""Ensure project path exists."""
for name, path_value in v.items():
path = Path(path_value)
if not Path(path).exists():
try:
path.mkdir(parents=True)
except Exception as e:
logger.error(f"Failed to create project path: {e}")
raise e
return v
class ConfigManager:
"""Manages Basic Memory configuration."""
@@ -123,13 +156,16 @@ class ConfigManager:
# Load or create configuration
self.config = self.load_config()
# Current project context for the session
self.current_project_id: int
def load_config(self) -> BasicMemoryConfig:
"""Load configuration from file or create default."""
if self.config_file.exists():
try:
data = json.loads(self.config_file.read_text(encoding="utf-8"))
return BasicMemoryConfig(**data)
except Exception as e:
except Exception as e: # pragma: no cover
logger.error(f"Failed to load config: {e}")
config = BasicMemoryConfig()
self.save_config(config)
@@ -156,37 +192,24 @@ class ConfigManager:
"""Get the default project name."""
return self.config.default_project
def get_project_path(self, project_name: Optional[str] = None) -> Path:
"""Get the path for a specific project or the default project."""
name = project_name or self.config.default_project
# Check if specified in environment variable
if not project_name and "BASIC_MEMORY_PROJECT" in os.environ:
name = os.environ["BASIC_MEMORY_PROJECT"]
if name not in self.config.projects:
raise ValueError(f"Project '{name}' not found in configuration")
return Path(self.config.projects[name])
def add_project(self, name: str, path: str) -> None:
"""Add a new project to the configuration."""
if name in self.config.projects:
if name in self.config.projects: # pragma: no cover
raise ValueError(f"Project '{name}' already exists")
# Ensure the path exists
project_path = Path(path)
project_path.mkdir(parents=True, exist_ok=True)
project_path.mkdir(parents=True, exist_ok=True) # pragma: no cover
self.config.projects[name] = str(project_path)
self.save_config(self.config)
def remove_project(self, name: str) -> None:
"""Remove a project from the configuration."""
if name not in self.config.projects:
if name not in self.config.projects: # pragma: no cover
raise ValueError(f"Project '{name}' not found")
if name == self.config.default_project:
if name == self.config.default_project: # pragma: no cover
raise ValueError(f"Cannot remove the default project '{name}'")
del self.config.projects[name]
@@ -201,34 +224,30 @@ class ConfigManager:
self.save_config(self.config)
def get_project_config(project_name: Optional[str] = None) -> ProjectConfig:
"""Get a project configuration for the specified project."""
config_manager = ConfigManager()
def get_project_config() -> ProjectConfig:
"""Get the project configuration for the current session."""
# Get project name from environment variable or use provided name or default
actual_project_name = os.environ.get(
"BASIC_MEMORY_PROJECT", project_name or config_manager.default_project
)
env_project_name = os.environ.get("BASIC_MEMORY_PROJECT", None)
actual_project_name = env_project_name or config_manager.default_project
update_permalinks_on_move = config_manager.load_config().update_permalinks_on_move
try:
project_path = config_manager.get_project_path(actual_project_name)
return ProjectConfig(
home=project_path,
project=actual_project_name,
update_permalinks_on_move=update_permalinks_on_move,
)
except ValueError: # pragma: no cover
logger.warning(f"Project '{actual_project_name}' not found, using default")
project_path = config_manager.get_project_path(config_manager.default_project)
return ProjectConfig(home=project_path, project=config_manager.default_project)
# the config contains a dict[str,str] of project names and absolute paths
project_path = config_manager.projects.get(actual_project_name)
if not project_path: # pragma: no cover
raise ValueError(f"Project '{actual_project_name}' not found")
return ProjectConfig(name=actual_project_name, home=Path(project_path))
# Create config manager
config_manager = ConfigManager()
# Load project config for current context
config = get_project_config()
# Export the app-level config
app_config: BasicMemoryConfig = config_manager.config
# Load project config for the default project (backward compatibility)
config: ProjectConfig = get_project_config()
# setup logging to a single log file in user home directory
user_home = Path.home()
@@ -236,6 +255,7 @@ log_dir = user_home / DATA_DIR_NAME
log_dir.mkdir(parents=True, exist_ok=True)
# Process info for logging
def get_process_name(): # pragma: no cover
"""
get the type of process for logging
@@ -258,6 +278,9 @@ process_name = get_process_name()
_LOGGING_SETUP = False
# Logging
def setup_basic_memory_logging(): # pragma: no cover
"""Set up logging for basic-memory, ensuring it only happens once."""
global _LOGGING_SETUP
@@ -267,7 +290,7 @@ def setup_basic_memory_logging(): # pragma: no cover
return
setup_logging(
env=config.env,
env=config_manager.config.env,
home_dir=user_home, # Use user home for logs
log_level=config_manager.load_config().log_level,
log_file=f"{DATA_DIR_NAME}/basic-memory-{process_name}.log",
src/basic_memory/db.py
+6 -4
View File
@@ -4,8 +4,7 @@ from enum import Enum, auto
from pathlib import Path
from typing import AsyncGenerator, Optional
from basic_memory.config import ProjectConfig
from basic_memory.config import BasicMemoryConfig
from alembic import command
from alembic.config import Config
@@ -147,7 +146,7 @@ async def engine_session_factory(
async def run_migrations(
app_config: ProjectConfig, database_type=DatabaseType.FILESYSTEM
app_config: BasicMemoryConfig, database_type=DatabaseType.FILESYSTEM
): # pragma: no cover
"""Run any pending alembic migrations."""
logger.info("Running database migrations...")
@@ -172,7 +171,10 @@ async def run_migrations(
logger.info("Migrations completed successfully")
_, session_maker = await get_or_create_db(app_config.database_path, database_type)
await SearchRepository(session_maker).init_search_index()
# initialize the search Index schema
# the project_id is not used for init_search_index, so we pass a dummy value
await SearchRepository(session_maker, 1).init_search_index()
except Exception as e: # pragma: no cover
logger.error(f"Error running migrations: {e}")
raise
src/basic_memory/deps.py
+195 -27
View File
@@ -2,7 +2,7 @@
from typing import Annotated
from fastapi import Depends
from fastapi import Depends, HTTPException, Path, status
from sqlalchemy.ext.asyncio import (
AsyncSession,
AsyncEngine,
@@ -10,21 +10,35 @@ from sqlalchemy.ext.asyncio import (
)
from basic_memory import db
from basic_memory.config import ProjectConfig, config
from basic_memory.config import ProjectConfig, config, BasicMemoryConfig
from basic_memory.importers import (
ChatGPTImporter,
ClaudeConversationsImporter,
ClaudeProjectsImporter,
MemoryJsonImporter,
)
from basic_memory.markdown import EntityParser
from basic_memory.markdown.markdown_processor import MarkdownProcessor
from basic_memory.repository.entity_repository import EntityRepository
from basic_memory.repository.observation_repository import ObservationRepository
from basic_memory.repository.project_info_repository import ProjectInfoRepository
from basic_memory.repository.project_repository import ProjectRepository
from basic_memory.repository.relation_repository import RelationRepository
from basic_memory.repository.search_repository import SearchRepository
from basic_memory.services import (
EntityService,
)
from basic_memory.services import EntityService, ProjectService
from basic_memory.services.context_service import ContextService
from basic_memory.services.directory_service import DirectoryService
from basic_memory.services.file_service import FileService
from basic_memory.services.link_resolver import LinkResolver
from basic_memory.services.search_service import SearchService
from basic_memory.sync import SyncService
from basic_memory.config import app_config
def get_app_config() -> BasicMemoryConfig: # pragma: no cover
return app_config
AppConfigDep = Annotated[BasicMemoryConfig, Depends(get_app_config)] # pragma: no cover
## project
@@ -36,15 +50,14 @@ def get_project_config() -> ProjectConfig: # pragma: no cover
ProjectConfigDep = Annotated[ProjectConfig, Depends(get_project_config)] # pragma: no cover
## sqlalchemy
async def get_engine_factory(
project_config: ProjectConfigDep,
app_config: AppConfigDep,
) -> tuple[AsyncEngine, async_sessionmaker[AsyncSession]]: # pragma: no cover
"""Get engine and session maker."""
engine, session_maker = await db.get_or_create_db(project_config.database_path)
engine, session_maker = await db.get_or_create_db(app_config.database_path)
return engine, session_maker
@@ -65,11 +78,70 @@ SessionMakerDep = Annotated[async_sessionmaker, Depends(get_session_maker)]
## repositories
async def get_project_repository(
session_maker: SessionMakerDep,
) -> ProjectRepository:
"""Get the project repository."""
return ProjectRepository(session_maker)
ProjectRepositoryDep = Annotated[ProjectRepository, Depends(get_project_repository)]
ProjectPathDep = Annotated[str, Path()] # Use Path dependency to extract from URL
async def get_project_id(
project_repository: ProjectRepositoryDep,
project: ProjectPathDep,
) -> int:
"""Get the current project ID from request state.
When using sub-applications with /{project} mounting, the project value
is stored in request.state by middleware.
Args:
request: The current request object
project_repository: Repository for project operations
Returns:
The resolved project ID
Raises:
HTTPException: If project is not found
"""
# Try by permalink first (most common case with URL paths)
project_obj = await project_repository.get_by_permalink(str(project))
if project_obj:
return project_obj.id
# Try by name if permalink lookup fails
project_obj = await project_repository.get_by_name(str(project)) # pragma: no cover
if project_obj: # pragma: no cover
return project_obj.id
# Not found
raise HTTPException( # pragma: no cover
status_code=status.HTTP_404_NOT_FOUND, detail=f"Project '{project}' not found."
)
"""
The project_id dependency is used in the following:
- EntityRepository
- ObservationRepository
- RelationRepository
- SearchRepository
- ProjectInfoRepository
"""
ProjectIdDep = Annotated[int, Depends(get_project_id)]
async def get_entity_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
) -> EntityRepository:
"""Create an EntityRepository instance."""
return EntityRepository(session_maker)
"""Create an EntityRepository instance for the current project."""
return EntityRepository(session_maker, project_id=project_id)
EntityRepositoryDep = Annotated[EntityRepository, Depends(get_entity_repository)]
@@ -77,9 +149,10 @@ EntityRepositoryDep = Annotated[EntityRepository, Depends(get_entity_repository)
async def get_observation_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
) -> ObservationRepository:
"""Create an ObservationRepository instance."""
return ObservationRepository(session_maker)
"""Create an ObservationRepository instance for the current project."""
return ObservationRepository(session_maker, project_id=project_id)
ObservationRepositoryDep = Annotated[ObservationRepository, Depends(get_observation_repository)]
@@ -87,9 +160,10 @@ ObservationRepositoryDep = Annotated[ObservationRepository, Depends(get_observat
async def get_relation_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
) -> RelationRepository:
"""Create a RelationRepository instance."""
return RelationRepository(session_maker)
"""Create a RelationRepository instance for the current project."""
return RelationRepository(session_maker, project_id=project_id)
RelationRepositoryDep = Annotated[RelationRepository, Depends(get_relation_repository)]
@@ -97,22 +171,17 @@ RelationRepositoryDep = Annotated[RelationRepository, Depends(get_relation_repos
async def get_search_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
) -> SearchRepository:
"""Create a SearchRepository instance."""
return SearchRepository(session_maker)
"""Create a SearchRepository instance for the current project."""
return SearchRepository(session_maker, project_id=project_id)
SearchRepositoryDep = Annotated[SearchRepository, Depends(get_search_repository)]
def get_project_info_repository(
session_maker: SessionMakerDep,
):
"""Dependency for StatsRepository."""
return ProjectInfoRepository(session_maker)
ProjectInfoRepositoryDep = Annotated[ProjectInfoRepository, Depends(get_project_info_repository)]
# ProjectInfoRepository is deprecated and will be removed in a future version.
# Use ProjectRepository instead, which has the same functionality plus more project-specific operations.
## services
@@ -184,9 +253,108 @@ LinkResolverDep = Annotated[LinkResolver, Depends(get_link_resolver)]
async def get_context_service(
search_repository: SearchRepositoryDep, entity_repository: EntityRepositoryDep
search_repository: SearchRepositoryDep,
entity_repository: EntityRepositoryDep,
observation_repository: ObservationRepositoryDep,
) -> ContextService:
return ContextService(search_repository, entity_repository)
return ContextService(
search_repository=search_repository,
entity_repository=entity_repository,
observation_repository=observation_repository,
)
ContextServiceDep = Annotated[ContextService, Depends(get_context_service)]
async def get_sync_service(
entity_service: EntityServiceDep,
entity_parser: EntityParserDep,
entity_repository: EntityRepositoryDep,
relation_repository: RelationRepositoryDep,
search_service: SearchServiceDep,
file_service: FileServiceDep,
) -> SyncService: # pragma: no cover
"""
:rtype: object
"""
return SyncService(
app_config=app_config,
entity_service=entity_service,
entity_parser=entity_parser,
entity_repository=entity_repository,
relation_repository=relation_repository,
search_service=search_service,
file_service=file_service,
)
SyncServiceDep = Annotated[SyncService, Depends(get_sync_service)]
async def get_project_service(
project_repository: ProjectRepositoryDep,
) -> ProjectService:
"""Create ProjectService with repository."""
return ProjectService(repository=project_repository)
ProjectServiceDep = Annotated[ProjectService, Depends(get_project_service)]
async def get_directory_service(
entity_repository: EntityRepositoryDep,
) -> DirectoryService:
"""Create DirectoryService with dependencies."""
return DirectoryService(
entity_repository=entity_repository,
)
DirectoryServiceDep = Annotated[DirectoryService, Depends(get_directory_service)]
# Import
async def get_chatgpt_importer(
project_config: ProjectConfigDep, markdown_processor: MarkdownProcessorDep
) -> ChatGPTImporter:
"""Create ChatGPTImporter with dependencies."""
return ChatGPTImporter(project_config.home, markdown_processor)
ChatGPTImporterDep = Annotated[ChatGPTImporter, Depends(get_chatgpt_importer)]
async def get_claude_conversations_importer(
project_config: ProjectConfigDep, markdown_processor: MarkdownProcessorDep
) -> ClaudeConversationsImporter:
"""Create ChatGPTImporter with dependencies."""
return ClaudeConversationsImporter(project_config.home, markdown_processor)
ClaudeConversationsImporterDep = Annotated[
ClaudeConversationsImporter, Depends(get_claude_conversations_importer)
]
async def get_claude_projects_importer(
project_config: ProjectConfigDep, markdown_processor: MarkdownProcessorDep
) -> ClaudeProjectsImporter:
"""Create ChatGPTImporter with dependencies."""
return ClaudeProjectsImporter(project_config.home, markdown_processor)
ClaudeProjectsImporterDep = Annotated[ClaudeProjectsImporter, Depends(get_claude_projects_importer)]
async def get_memory_json_importer(
project_config: ProjectConfigDep, markdown_processor: MarkdownProcessorDep
) -> MemoryJsonImporter:
"""Create ChatGPTImporter with dependencies."""
return MemoryJsonImporter(project_config.home, markdown_processor)
MemoryJsonImporterDep = Annotated[MemoryJsonImporter, Depends(get_memory_json_importer)]
src/basic_memory/importers/__init__.py
+27
View File
@@ -0,0 +1,27 @@
"""Import services for Basic Memory."""
from basic_memory.importers.base import Importer
from basic_memory.importers.chatgpt_importer import ChatGPTImporter
from basic_memory.importers.claude_conversations_importer import (
ClaudeConversationsImporter,
)
from basic_memory.importers.claude_projects_importer import ClaudeProjectsImporter
from basic_memory.importers.memory_json_importer import MemoryJsonImporter
from basic_memory.schemas.importer import (
ChatImportResult,
EntityImportResult,
ImportResult,
ProjectImportResult,
)
__all__ = [
"Importer",
"ChatGPTImporter",
"ClaudeConversationsImporter",
"ClaudeProjectsImporter",
"MemoryJsonImporter",
"ImportResult",
"ChatImportResult",
"EntityImportResult",
"ProjectImportResult",
]
src/basic_memory/importers/base.py
+79
View File
@@ -0,0 +1,79 @@
"""Base import service for Basic Memory."""
import logging
from abc import abstractmethod
from pathlib import Path
from typing import Any, Optional, TypeVar
from basic_memory.markdown.markdown_processor import MarkdownProcessor
from basic_memory.markdown.schemas import EntityMarkdown
from basic_memory.schemas.importer import ImportResult
logger = logging.getLogger(__name__)
T = TypeVar("T", bound=ImportResult)
class Importer[T: ImportResult]:
"""Base class for all import services."""
def __init__(self, base_path: Path, markdown_processor: MarkdownProcessor):
"""Initialize the import service.
Args:
markdown_processor: MarkdownProcessor instance for writing markdown files.
"""
self.base_path = base_path.resolve() # Get absolute path
self.markdown_processor = markdown_processor
@abstractmethod
async def import_data(self, source_data, destination_folder: str, **kwargs: Any) -> T:
"""Import data from source file to destination folder.
Args:
source_path: Path to the source file.
destination_folder: Destination folder within the project.
**kwargs: Additional keyword arguments for specific import types.
Returns:
ImportResult containing statistics and status of the import.
"""
pass # pragma: no cover
async def write_entity(self, entity: EntityMarkdown, file_path: Path) -> None:
"""Write entity to file using markdown processor.
Args:
entity: EntityMarkdown instance to write.
file_path: Path to write the entity to.
"""
await self.markdown_processor.write_file(file_path, entity)
def ensure_folder_exists(self, folder: str) -> Path:
"""Ensure folder exists, create if it doesn't.
Args:
base_path: Base path of the project.
folder: Folder name or path within the project.
Returns:
Path to the folder.
"""
folder_path = self.base_path / folder
folder_path.mkdir(parents=True, exist_ok=True)
return folder_path
@abstractmethod
def handle_error(
self, message: str, error: Optional[Exception] = None
) -> T: # pragma: no cover
"""Handle errors during import.
Args:
message: Error message.
error: Optional exception that caused the error.
Returns:
ImportResult with error information.
"""
pass
src/basic_memory/importers/chatgpt_importer.py
+222
View File
@@ -0,0 +1,222 @@
"""ChatGPT import service for Basic Memory."""
import logging
from datetime import datetime
from typing import Any, Dict, List, Optional, Set
from basic_memory.markdown.schemas import EntityFrontmatter, EntityMarkdown
from basic_memory.importers.base import Importer
from basic_memory.schemas.importer import ChatImportResult
from basic_memory.importers.utils import clean_filename, format_timestamp
logger = logging.getLogger(__name__)
class ChatGPTImporter(Importer[ChatImportResult]):
"""Service for importing ChatGPT conversations."""
async def import_data(
self, source_data, destination_folder: str, **kwargs: Any
) -> ChatImportResult:
"""Import conversations from ChatGPT JSON export.
Args:
source_path: Path to the ChatGPT conversations.json file.
destination_folder: Destination folder within the project.
**kwargs: Additional keyword arguments.
Returns:
ChatImportResult containing statistics and status of the import.
"""
try: # pragma: no cover
# Ensure the destination folder exists
self.ensure_folder_exists(destination_folder)
conversations = source_data
# Process each conversation
messages_imported = 0
chats_imported = 0
for chat in conversations:
# Convert to entity
entity = self._format_chat_content(destination_folder, chat)
# Write file
file_path = self.base_path / f"{entity.frontmatter.metadata['permalink']}.md"
await self.write_entity(entity, file_path)
# Count messages
msg_count = sum(
1
for node in chat["mapping"].values()
if node.get("message")
and not node.get("message", {})
.get("metadata", {})
.get("is_visually_hidden_from_conversation")
)
chats_imported += 1
messages_imported += msg_count
return ChatImportResult(
import_count={"conversations": chats_imported, "messages": messages_imported},
success=True,
conversations=chats_imported,
messages=messages_imported,
)
except Exception as e: # pragma: no cover
logger.exception("Failed to import ChatGPT conversations")
return self.handle_error("Failed to import ChatGPT conversations", e) # pyright: ignore [reportReturnType]
def _format_chat_content(
self, folder: str, conversation: Dict[str, Any]
) -> EntityMarkdown: # pragma: no cover
"""Convert chat conversation to Basic Memory entity.
Args:
folder: Destination folder name.
conversation: ChatGPT conversation data.
Returns:
EntityMarkdown instance representing the conversation.
"""
# Extract timestamps
created_at = conversation["create_time"]
modified_at = conversation["update_time"]
root_id = None
# Find root message
for node_id, node in conversation["mapping"].items():
if node.get("parent") is None:
root_id = node_id
break
# Generate permalink
date_prefix = datetime.fromtimestamp(created_at).strftime("%Y%m%d")
clean_title = clean_filename(conversation["title"])
# Format content
content = self._format_chat_markdown(
title=conversation["title"],
mapping=conversation["mapping"],
root_id=root_id,
created_at=created_at,
modified_at=modified_at,
)
# Create entity
entity = EntityMarkdown(
frontmatter=EntityFrontmatter(
metadata={
"type": "conversation",
"title": conversation["title"],
"created": format_timestamp(created_at),
"modified": format_timestamp(modified_at),
"permalink": f"{folder}/{date_prefix}-{clean_title}",
}
),
content=content,
)
return entity
def _format_chat_markdown(
self,
title: str,
mapping: Dict[str, Any],
root_id: Optional[str],
created_at: float,
modified_at: float,
) -> str: # pragma: no cover
"""Format chat as clean markdown.
Args:
title: Chat title.
mapping: Message mapping.
root_id: Root message ID.
created_at: Creation timestamp.
modified_at: Modification timestamp.
Returns:
Formatted markdown content.
"""
# Start with title
lines = [f"# {title}\n"]
# Traverse message tree
seen_msgs: Set[str] = set()
messages = self._traverse_messages(mapping, root_id, seen_msgs)
# Format each message
for msg in messages:
# Skip hidden messages
if msg.get("metadata", {}).get("is_visually_hidden_from_conversation"):
continue
# Get author and timestamp
author = msg["author"]["role"].title()
ts = format_timestamp(msg["create_time"]) if msg.get("create_time") else ""
# Add message header
lines.append(f"### {author} ({ts})")
# Add message content
content = self._get_message_content(msg)
if content:
lines.append(content)
# Add spacing
lines.append("")
return "\n".join(lines)
def _get_message_content(self, message: Dict[str, Any]) -> str: # pragma: no cover
"""Extract clean message content.
Args:
message: Message data.
Returns:
Cleaned message content.
"""
if not message or "content" not in message:
return ""
content = message["content"]
if content.get("content_type") == "text":
return "\n".join(content.get("parts", []))
elif content.get("content_type") == "code":
return f"```{content.get('language', '')}\n{content.get('text', '')}\n```"
return ""
def _traverse_messages(
self, mapping: Dict[str, Any], root_id: Optional[str], seen: Set[str]
) -> List[Dict[str, Any]]: # pragma: no cover
"""Traverse message tree and return messages in order.
Args:
mapping: Message mapping.
root_id: Root message ID.
seen: Set of seen message IDs.
Returns:
List of message data.
"""
messages = []
node = mapping.get(root_id) if root_id else None
while node:
if node["id"] not in seen and node.get("message"):
seen.add(node["id"])
messages.append(node["message"])
# Follow children
children = node.get("children", [])
for child_id in children:
child_msgs = self._traverse_messages(mapping, child_id, seen)
messages.extend(child_msgs)
break # Don't follow siblings
return messages
src/basic_memory/importers/claude_conversations_importer.py
+172
View File
@@ -0,0 +1,172 @@
"""Claude conversations import service for Basic Memory."""
import logging
from datetime import datetime
from pathlib import Path
from typing import Any, Dict, List
from basic_memory.markdown.schemas import EntityFrontmatter, EntityMarkdown
from basic_memory.importers.base import Importer
from basic_memory.schemas.importer import ChatImportResult
from basic_memory.importers.utils import clean_filename, format_timestamp
logger = logging.getLogger(__name__)
class ClaudeConversationsImporter(Importer[ChatImportResult]):
"""Service for importing Claude conversations."""
async def import_data(
self, source_data, destination_folder: str, **kwargs: Any
) -> ChatImportResult:
"""Import conversations from Claude JSON export.
Args:
source_data: Path to the Claude conversations.json file.
destination_folder: Destination folder within the project.
**kwargs: Additional keyword arguments.
Returns:
ChatImportResult containing statistics and status of the import.
"""
try:
# Ensure the destination folder exists
folder_path = self.ensure_folder_exists(destination_folder)
conversations = source_data
# Process each conversation
messages_imported = 0
chats_imported = 0
for chat in conversations:
# Convert to entity
entity = self._format_chat_content(
base_path=folder_path,
name=chat["name"],
messages=chat["chat_messages"],
created_at=chat["created_at"],
modified_at=chat["updated_at"],
)
# Write file
file_path = self.base_path / Path(f"{entity.frontmatter.metadata['permalink']}.md")
await self.write_entity(entity, file_path)
chats_imported += 1
messages_imported += len(chat["chat_messages"])
return ChatImportResult(
import_count={"conversations": chats_imported, "messages": messages_imported},
success=True,
conversations=chats_imported,
messages=messages_imported,
)
except Exception as e: # pragma: no cover
logger.exception("Failed to import Claude conversations")
return self.handle_error("Failed to import Claude conversations", e) # pyright: ignore [reportReturnType]
def _format_chat_content(
self,
base_path: Path,
name: str,
messages: List[Dict[str, Any]],
created_at: str,
modified_at: str,
) -> EntityMarkdown:
"""Convert chat messages to Basic Memory entity format.
Args:
base_path: Base path for the entity.
name: Chat name.
messages: List of chat messages.
created_at: Creation timestamp.
modified_at: Modification timestamp.
Returns:
EntityMarkdown instance representing the conversation.
"""
# Generate permalink
date_prefix = datetime.fromisoformat(created_at.replace("Z", "+00:00")).strftime("%Y%m%d")
clean_title = clean_filename(name)
permalink = f"{base_path.name}/{date_prefix}-{clean_title}"
# Format content
content = self._format_chat_markdown(
name=name,
messages=messages,
created_at=created_at,
modified_at=modified_at,
permalink=permalink,
)
# Create entity
entity = EntityMarkdown(
frontmatter=EntityFrontmatter(
metadata={
"type": "conversation",
"title": name,
"created": created_at,
"modified": modified_at,
"permalink": permalink,
}
),
content=content,
)
return entity
def _format_chat_markdown(
self,
name: str,
messages: List[Dict[str, Any]],
created_at: str,
modified_at: str,
permalink: str,
) -> str:
"""Format chat as clean markdown.
Args:
name: Chat name.
messages: List of chat messages.
created_at: Creation timestamp.
modified_at: Modification timestamp.
permalink: Permalink for the entity.
Returns:
Formatted markdown content.
"""
# Start with frontmatter and title
lines = [
f"# {name}\n",
]
# Add messages
for msg in messages:
# Format timestamp
ts = format_timestamp(msg["created_at"])
# Add message header
lines.append(f"### {msg['sender'].title()} ({ts})")
# Handle message content
content = msg.get("text", "")
if msg.get("content"):
content = " ".join(c.get("text", "") for c in msg["content"])
lines.append(content)
# Handle attachments
attachments = msg.get("attachments", [])
for attachment in attachments:
if "file_name" in attachment:
lines.append(f"\n**Attachment: {attachment['file_name']}**")
if "extracted_content" in attachment:
lines.append("```")
lines.append(attachment["extracted_content"])
lines.append("```")
# Add spacing between messages
lines.append("")
return "\n".join(lines)
src/basic_memory/importers/claude_projects_importer.py
+148
View File
@@ -0,0 +1,148 @@
"""Claude projects import service for Basic Memory."""
import logging
from typing import Any, Dict, Optional
from basic_memory.markdown.schemas import EntityFrontmatter, EntityMarkdown
from basic_memory.importers.base import Importer
from basic_memory.schemas.importer import ProjectImportResult
from basic_memory.importers.utils import clean_filename
logger = logging.getLogger(__name__)
class ClaudeProjectsImporter(Importer[ProjectImportResult]):
"""Service for importing Claude projects."""
async def import_data(
self, source_data, destination_folder: str, **kwargs: Any
) -> ProjectImportResult:
"""Import projects from Claude JSON export.
Args:
source_path: Path to the Claude projects.json file.
destination_folder: Base folder for projects within the project.
**kwargs: Additional keyword arguments.
Returns:
ProjectImportResult containing statistics and status of the import.
"""
try:
# Ensure the base folder exists
base_path = self.base_path
if destination_folder:
base_path = self.ensure_folder_exists(destination_folder)
projects = source_data
# Process each project
docs_imported = 0
prompts_imported = 0
for project in projects:
project_dir = clean_filename(project["name"])
# Create project directories
docs_dir = base_path / project_dir / "docs"
docs_dir.mkdir(parents=True, exist_ok=True)
# Import prompt template if it exists
if prompt_entity := self._format_prompt_markdown(project):
file_path = base_path / f"{prompt_entity.frontmatter.metadata['permalink']}.md"
await self.write_entity(prompt_entity, file_path)
prompts_imported += 1
# Import project documents
for doc in project.get("docs", []):
entity = self._format_project_markdown(project, doc)
file_path = base_path / f"{entity.frontmatter.metadata['permalink']}.md"
await self.write_entity(entity, file_path)
docs_imported += 1
return ProjectImportResult(
import_count={"documents": docs_imported, "prompts": prompts_imported},
success=True,
documents=docs_imported,
prompts=prompts_imported,
)
except Exception as e: # pragma: no cover
logger.exception("Failed to import Claude projects")
return self.handle_error("Failed to import Claude projects", e) # pyright: ignore [reportReturnType]
def _format_project_markdown(
self, project: Dict[str, Any], doc: Dict[str, Any]
) -> EntityMarkdown:
"""Format a project document as a Basic Memory entity.
Args:
project: Project data.
doc: Document data.
Returns:
EntityMarkdown instance representing the document.
"""
# Extract timestamps
created_at = doc.get("created_at") or project["created_at"]
modified_at = project["updated_at"]
# Generate clean names for organization
project_dir = clean_filename(project["name"])
doc_file = clean_filename(doc["filename"])
# Create entity
entity = EntityMarkdown(
frontmatter=EntityFrontmatter(
metadata={
"type": "project_doc",
"title": doc["filename"],
"created": created_at,
"modified": modified_at,
"permalink": f"{project_dir}/docs/{doc_file}",
"project_name": project["name"],
"project_uuid": project["uuid"],
"doc_uuid": doc["uuid"],
}
),
content=doc["content"],
)
return entity
def _format_prompt_markdown(self, project: Dict[str, Any]) -> Optional[EntityMarkdown]:
"""Format project prompt template as a Basic Memory entity.
Args:
project: Project data.
Returns:
EntityMarkdown instance representing the prompt template, or None if
no prompt template exists.
"""
if not project.get("prompt_template"):
return None
# Extract timestamps
created_at = project["created_at"]
modified_at = project["updated_at"]
# Generate clean project directory name
project_dir = clean_filename(project["name"])
# Create entity
entity = EntityMarkdown(
frontmatter=EntityFrontmatter(
metadata={
"type": "prompt_template",
"title": f"Prompt Template: {project['name']}",
"created": created_at,
"modified": modified_at,
"permalink": f"{project_dir}/prompt-template",
"project_name": project["name"],
"project_uuid": project["uuid"],
}
),
content=f"# Prompt Template: {project['name']}\n\n{project['prompt_template']}",
)
return entity
src/basic_memory/importers/memory_json_importer.py
+93
View File
@@ -0,0 +1,93 @@
"""Memory JSON import service for Basic Memory."""
import logging
from typing import Any, Dict, List
from basic_memory.config import config
from basic_memory.markdown.schemas import EntityFrontmatter, EntityMarkdown, Observation, Relation
from basic_memory.importers.base import Importer
from basic_memory.schemas.importer import EntityImportResult
logger = logging.getLogger(__name__)
class MemoryJsonImporter(Importer[EntityImportResult]):
"""Service for importing memory.json format data."""
async def import_data(
self, source_data, destination_folder: str = "", **kwargs: Any
) -> EntityImportResult:
"""Import entities and relations from a memory.json file.
Args:
source_data: Path to the memory.json file.
destination_folder: Optional destination folder within the project.
**kwargs: Additional keyword arguments.
Returns:
EntityImportResult containing statistics and status of the import.
"""
try:
# First pass - collect all relations by source entity
entity_relations: Dict[str, List[Relation]] = {}
entities: Dict[str, Dict[str, Any]] = {}
# Ensure the base path exists
base_path = config.home # pragma: no cover
if destination_folder: # pragma: no cover
base_path = self.ensure_folder_exists(destination_folder)
# First pass - collect entities and relations
for line in source_data:
data = line
if data["type"] == "entity":
entities[data["name"]] = data
elif data["type"] == "relation":
# Store relation with its source entity
source = data.get("from") or data.get("from_id")
if source not in entity_relations:
entity_relations[source] = []
entity_relations[source].append(
Relation(
type=data.get("relationType") or data.get("relation_type"),
target=data.get("to") or data.get("to_id"),
)
)
# Second pass - create and write entities
entities_created = 0
for name, entity_data in entities.items():
# Ensure entity type directory exists
entity_type_dir = base_path / entity_data["entityType"]
entity_type_dir.mkdir(parents=True, exist_ok=True)
entity = EntityMarkdown(
frontmatter=EntityFrontmatter(
metadata={
"type": entity_data["entityType"],
"title": name,
"permalink": f"{entity_data['entityType']}/{name}",
}
),
content=f"# {name}\n",
observations=[Observation(content=obs) for obs in entity_data["observations"]],
relations=entity_relations.get(name, []),
)
# Write entity file
file_path = base_path / f"{entity_data['entityType']}/{name}.md"
await self.write_entity(entity, file_path)
entities_created += 1
relations_count = sum(len(rels) for rels in entity_relations.values())
return EntityImportResult(
import_count={"entities": entities_created, "relations": relations_count},
success=True,
entities=entities_created,
relations=relations_count,
)
except Exception as e: # pragma: no cover
logger.exception("Failed to import memory.json")
return self.handle_error("Failed to import memory.json", e) # pyright: ignore [reportReturnType]
src/basic_memory/importers/utils.py
+58
View File
@@ -0,0 +1,58 @@
"""Utility functions for import services."""
import re
from datetime import datetime
from typing import Any
def clean_filename(name: str) -> str: # pragma: no cover
"""Clean a string to be used as a filename.
Args:
name: The string to clean.
Returns:
A cleaned string suitable for use as a filename.
"""
# Replace common punctuation and whitespace with underscores
name = re.sub(r"[\s\-,.:/\\\[\]\(\)]+", "_", name)
# Remove any non-alphanumeric or underscore characters
name = re.sub(r"[^\w]+", "", name)
# Ensure the name isn't too long
if len(name) > 100: # pragma: no cover
name = name[:100]
# Ensure the name isn't empty
if not name: # pragma: no cover
name = "untitled"
return name
def format_timestamp(timestamp: Any) -> str: # pragma: no cover
"""Format a timestamp for use in a filename or title.
Args:
timestamp: A timestamp in various formats.
Returns:
A formatted string representation of the timestamp.
"""
if isinstance(timestamp, str):
try:
# Try ISO format
timestamp = datetime.fromisoformat(timestamp.replace("Z", "+00:00"))
except ValueError:
try:
# Try unix timestamp as string
timestamp = datetime.fromtimestamp(float(timestamp))
except ValueError:
# Return as is if we can't parse it
return timestamp
elif isinstance(timestamp, (int, float)):
# Unix timestamp
timestamp = datetime.fromtimestamp(timestamp)
if isinstance(timestamp, datetime):
return timestamp.strftime("%Y-%m-%d %H:%M:%S")
# Return as is if we can't format it
return str(timestamp) # pragma: no cover
src/basic_memory/mcp/auth_provider.py
+270
View File
@@ -0,0 +1,270 @@
"""OAuth authentication provider for Basic Memory MCP server."""
import secrets
from datetime import datetime, timedelta
from typing import Dict, Optional
import jwt
from mcp.server.auth.provider import (
OAuthAuthorizationServerProvider,
AuthorizationParams,
AuthorizationCode,
RefreshToken,
AccessToken,
)
from mcp.shared.auth import OAuthClientInformationFull, OAuthToken
from loguru import logger
class BasicMemoryAuthorizationCode(AuthorizationCode):
"""Extended authorization code with additional metadata."""
issuer_state: Optional[str] = None
class BasicMemoryRefreshToken(RefreshToken):
"""Extended refresh token with additional metadata."""
pass
class BasicMemoryAccessToken(AccessToken):
"""Extended access token with additional metadata."""
pass
class BasicMemoryOAuthProvider(
OAuthAuthorizationServerProvider[
BasicMemoryAuthorizationCode, BasicMemoryRefreshToken, BasicMemoryAccessToken
]
):
"""OAuth provider for Basic Memory MCP server.
This is a simple in-memory implementation that can be extended
to integrate with external OAuth providers or use persistent storage.
"""
def __init__(self, issuer_url: str = "http://localhost:8000", secret_key: Optional[str] = None):
self.issuer_url = issuer_url
# Use environment variable for secret key if available, otherwise generate
import os
self.secret_key = (
secret_key or os.getenv("FASTMCP_AUTH_SECRET_KEY") or secrets.token_urlsafe(32)
)
# In-memory storage - in production, use a proper database
self.clients: Dict[str, OAuthClientInformationFull] = {}
self.authorization_codes: Dict[str, BasicMemoryAuthorizationCode] = {}
self.refresh_tokens: Dict[str, BasicMemoryRefreshToken] = {}
self.access_tokens: Dict[str, BasicMemoryAccessToken] = {}
async def get_client(self, client_id: str) -> Optional[OAuthClientInformationFull]:
"""Get a client by ID."""
return self.clients.get(client_id)
async def register_client(self, client_info: OAuthClientInformationFull) -> None:
"""Register a new OAuth client."""
# Generate client ID if not provided
if not client_info.client_id:
client_info.client_id = secrets.token_urlsafe(16)
# Generate client secret if not provided
if not client_info.client_secret:
client_info.client_secret = secrets.token_urlsafe(32)
self.clients[client_info.client_id] = client_info
logger.info(f"Registered OAuth client: {client_info.client_id}")
async def authorize(
self, client: OAuthClientInformationFull, params: AuthorizationParams
) -> str:
"""Create an authorization URL for the OAuth flow.
For basic-memory, we'll implement a simple authorization flow.
In production, this might redirect to an external provider.
"""
# Generate authorization code
auth_code = secrets.token_urlsafe(32)
# Store authorization code with metadata
self.authorization_codes[auth_code] = BasicMemoryAuthorizationCode(
code=auth_code,
scopes=params.scopes or [],
expires_at=(datetime.utcnow() + timedelta(minutes=10)).timestamp(),
client_id=client.client_id,
code_challenge=params.code_challenge,
redirect_uri=params.redirect_uri,
redirect_uri_provided_explicitly=params.redirect_uri_provided_explicitly,
issuer_state=params.state,
)
# In a real implementation, we'd redirect to an authorization page
# For now, we'll just return the redirect URL with the code
redirect_uri = str(params.redirect_uri)
separator = "&" if "?" in redirect_uri else "?"
auth_url = f"{redirect_uri}{separator}code={auth_code}"
if params.state:
auth_url += f"&state={params.state}"
return auth_url
async def load_authorization_code(
self, client: OAuthClientInformationFull, authorization_code: str
) -> Optional[BasicMemoryAuthorizationCode]:
"""Load an authorization code."""
code = self.authorization_codes.get(authorization_code)
if code and code.client_id == client.client_id:
# Check if expired
if datetime.utcnow().timestamp() > code.expires_at:
del self.authorization_codes[authorization_code]
return None
return code
return None
async def exchange_authorization_code(
self, client: OAuthClientInformationFull, authorization_code: BasicMemoryAuthorizationCode
) -> OAuthToken:
"""Exchange an authorization code for tokens."""
# Generate tokens
access_token = self._generate_access_token(client.client_id, authorization_code.scopes)
refresh_token = secrets.token_urlsafe(32)
# Store tokens
expires_at = (datetime.utcnow() + timedelta(hours=1)).timestamp()
self.access_tokens[access_token] = BasicMemoryAccessToken(
token=access_token,
client_id=client.client_id,
scopes=authorization_code.scopes,
expires_at=int(expires_at),
)
self.refresh_tokens[refresh_token] = BasicMemoryRefreshToken(
token=refresh_token,
client_id=client.client_id,
scopes=authorization_code.scopes,
)
# Remove used authorization code
del self.authorization_codes[authorization_code.code]
return OAuthToken(
access_token=access_token,
token_type="bearer",
expires_in=3600, # 1 hour
refresh_token=refresh_token,
scope=" ".join(authorization_code.scopes) if authorization_code.scopes else None,
)
async def load_refresh_token(
self, client: OAuthClientInformationFull, refresh_token: str
) -> Optional[BasicMemoryRefreshToken]:
"""Load a refresh token."""
token = self.refresh_tokens.get(refresh_token)
if token and token.client_id == client.client_id:
return token
return None
async def exchange_refresh_token(
self,
client: OAuthClientInformationFull,
refresh_token: BasicMemoryRefreshToken,
scopes: list[str],
) -> OAuthToken:
"""Exchange a refresh token for new tokens."""
# Use requested scopes or original scopes
token_scopes = scopes if scopes else refresh_token.scopes
# Generate new tokens
new_access_token = self._generate_access_token(client.client_id, token_scopes)
new_refresh_token = secrets.token_urlsafe(32)
# Store new tokens
expires_at = (datetime.utcnow() + timedelta(hours=1)).timestamp()
self.access_tokens[new_access_token] = BasicMemoryAccessToken(
token=new_access_token,
client_id=client.client_id,
scopes=token_scopes,
expires_at=int(expires_at),
)
self.refresh_tokens[new_refresh_token] = BasicMemoryRefreshToken(
token=new_refresh_token,
client_id=client.client_id,
scopes=token_scopes,
)
# Remove old tokens
del self.refresh_tokens[refresh_token.token]
return OAuthToken(
access_token=new_access_token,
token_type="bearer",
expires_in=3600, # 1 hour
refresh_token=new_refresh_token,
scope=" ".join(token_scopes) if token_scopes else None,
)
async def load_access_token(self, token: str) -> Optional[BasicMemoryAccessToken]:
"""Load and validate an access token."""
logger.debug("Loading access token, checking in-memory store first")
access_token = self.access_tokens.get(token)
if access_token:
# Check if expired
if access_token.expires_at and datetime.utcnow().timestamp() > access_token.expires_at:
logger.debug("Token found in memory but expired, removing")
del self.access_tokens[token]
return None
logger.debug("Token found in memory and valid")
return access_token
# Try to decode as JWT
logger.debug("Token not in memory, attempting JWT decode with secret key")
try:
# Decode with audience verification - PyJWT expects the audience to match
payload = jwt.decode(
token,
self.secret_key,
algorithms=["HS256"],
audience="basic-memory", # Expecting this audience
issuer=self.issuer_url, # And this issuer
)
logger.debug(f"JWT decoded successfully: {payload}")
return BasicMemoryAccessToken(
token=token,
client_id=payload.get("sub", ""),
scopes=payload.get("scopes", []),
expires_at=payload.get("exp"),
)
except jwt.InvalidTokenError as e:
logger.error(f"JWT decode failed: {e}")
return None
async def revoke_token(self, token: BasicMemoryAccessToken | BasicMemoryRefreshToken) -> None:
"""Revoke an access or refresh token."""
if isinstance(token, BasicMemoryAccessToken):
self.access_tokens.pop(token.token, None)
else:
self.refresh_tokens.pop(token.token, None)
def _generate_access_token(self, client_id: str, scopes: list[str]) -> str:
"""Generate a JWT access token."""
payload = {
"iss": self.issuer_url,
"sub": client_id,
"aud": "basic-memory",
"exp": datetime.utcnow() + timedelta(hours=1),
"iat": datetime.utcnow(),
"scopes": scopes,
}
return jwt.encode(payload, self.secret_key, algorithm="HS256")
src/basic_memory/mcp/external_auth_provider.py
+321
View File
@@ -0,0 +1,321 @@
"""External OAuth provider integration for Basic Memory MCP server."""
import os
from typing import Optional, Dict, Any
from dataclasses import dataclass
import httpx
from loguru import logger
from mcp.server.auth.provider import (
OAuthAuthorizationServerProvider,
AuthorizationParams,
AuthorizationCode,
RefreshToken,
AccessToken,
construct_redirect_uri,
)
from mcp.shared.auth import OAuthClientInformationFull, OAuthToken
@dataclass
class ExternalAuthorizationCode(AuthorizationCode):
"""Authorization code with external provider metadata."""
external_code: Optional[str] = None
state: Optional[str] = None
@dataclass
class ExternalRefreshToken(RefreshToken):
"""Refresh token with external provider metadata."""
external_token: Optional[str] = None
@dataclass
class ExternalAccessToken(AccessToken):
"""Access token with external provider metadata."""
external_token: Optional[str] = None
class ExternalOAuthProvider(
OAuthAuthorizationServerProvider[
ExternalAuthorizationCode, ExternalRefreshToken, ExternalAccessToken
]
):
"""OAuth provider that delegates to external OAuth providers.
This provider can integrate with services like:
- GitHub OAuth
- Google OAuth
- Auth0
- Okta
"""
def __init__(
self,
issuer_url: str,
external_provider: str,
external_client_id: str,
external_client_secret: str,
external_authorize_url: str,
external_token_url: str,
external_userinfo_url: Optional[str] = None,
):
self.issuer_url = issuer_url
self.external_provider = external_provider
self.external_client_id = external_client_id
self.external_client_secret = external_client_secret
self.external_authorize_url = external_authorize_url
self.external_token_url = external_token_url
self.external_userinfo_url = external_userinfo_url
# In-memory storage - in production, use a database
self.clients: Dict[str, OAuthClientInformationFull] = {}
self.codes: Dict[str, ExternalAuthorizationCode] = {}
self.tokens: Dict[str, Any] = {}
self.http_client = httpx.AsyncClient()
async def get_client(self, client_id: str) -> Optional[OAuthClientInformationFull]:
"""Get a client by ID."""
return self.clients.get(client_id)
async def register_client(self, client_info: OAuthClientInformationFull) -> None:
"""Register a new OAuth client."""
self.clients[client_info.client_id] = client_info
logger.info(f"Registered external OAuth client: {client_info.client_id}")
async def authorize(
self, client: OAuthClientInformationFull, params: AuthorizationParams
) -> str:
"""Create authorization URL redirecting to external provider."""
# Store authorization request
import secrets
state = secrets.token_urlsafe(32)
self.codes[state] = ExternalAuthorizationCode(
code=state,
scopes=params.scopes or [],
expires_at=0, # Will be set by external provider
client_id=client.client_id,
code_challenge=params.code_challenge,
redirect_uri=params.redirect_uri,
redirect_uri_provided_explicitly=params.redirect_uri_provided_explicitly,
state=params.state,
)
# Build external provider URL
external_params = {
"client_id": self.external_client_id,
"redirect_uri": f"{self.issuer_url}/callback",
"response_type": "code",
"state": state,
"scope": " ".join(params.scopes or []),
}
return construct_redirect_uri(self.external_authorize_url, **external_params)
async def handle_callback(self, code: str, state: str) -> str:
"""Handle callback from external provider."""
# Get original authorization request
auth_code = self.codes.get(state)
if not auth_code:
raise ValueError("Invalid state parameter")
# Exchange code with external provider
token_data = {
"grant_type": "authorization_code",
"code": code,
"redirect_uri": f"{self.issuer_url}/callback",
"client_id": self.external_client_id,
"client_secret": self.external_client_secret,
}
response = await self.http_client.post(
self.external_token_url,
data=token_data,
)
response.raise_for_status()
external_tokens = response.json()
# Store external tokens
import secrets
internal_code = secrets.token_urlsafe(32)
self.codes[internal_code] = ExternalAuthorizationCode(
code=internal_code,
scopes=auth_code.scopes,
expires_at=0,
client_id=auth_code.client_id,
code_challenge=auth_code.code_challenge,
redirect_uri=auth_code.redirect_uri,
redirect_uri_provided_explicitly=auth_code.redirect_uri_provided_explicitly,
external_code=code,
state=auth_code.state,
)
self.tokens[internal_code] = external_tokens
# Redirect to original client
return construct_redirect_uri(
str(auth_code.redirect_uri),
code=internal_code,
state=auth_code.state,
)
async def load_authorization_code(
self, client: OAuthClientInformationFull, authorization_code: str
) -> Optional[ExternalAuthorizationCode]:
"""Load an authorization code."""
code = self.codes.get(authorization_code)
if code and code.client_id == client.client_id:
return code
return None
async def exchange_authorization_code(
self, client: OAuthClientInformationFull, authorization_code: ExternalAuthorizationCode
) -> OAuthToken:
"""Exchange authorization code for tokens."""
# Get stored external tokens
external_tokens = self.tokens.get(authorization_code.code)
if not external_tokens:
raise ValueError("No tokens found for authorization code")
# Map external tokens to MCP tokens
access_token = external_tokens.get("access_token")
refresh_token = external_tokens.get("refresh_token")
expires_in = external_tokens.get("expires_in", 3600)
# Store the mapping
self.tokens[access_token] = {
"client_id": client.client_id,
"external_token": access_token,
"scopes": authorization_code.scopes,
}
if refresh_token:
self.tokens[refresh_token] = {
"client_id": client.client_id,
"external_token": refresh_token,
"scopes": authorization_code.scopes,
}
# Clean up authorization code
del self.codes[authorization_code.code]
return OAuthToken(
access_token=access_token,
token_type="bearer",
expires_in=expires_in,
refresh_token=refresh_token,
scope=" ".join(authorization_code.scopes) if authorization_code.scopes else None,
)
async def load_refresh_token(
self, client: OAuthClientInformationFull, refresh_token: str
) -> Optional[ExternalRefreshToken]:
"""Load a refresh token."""
token_info = self.tokens.get(refresh_token)
if token_info and token_info["client_id"] == client.client_id:
return ExternalRefreshToken(
token=refresh_token,
client_id=client.client_id,
scopes=token_info["scopes"],
external_token=token_info.get("external_token"),
)
return None
async def exchange_refresh_token(
self,
client: OAuthClientInformationFull,
refresh_token: ExternalRefreshToken,
scopes: list[str],
) -> OAuthToken:
"""Exchange refresh token for new tokens."""
# Exchange with external provider
token_data = {
"grant_type": "refresh_token",
"refresh_token": refresh_token.external_token or refresh_token.token,
"client_id": self.external_client_id,
"client_secret": self.external_client_secret,
}
response = await self.http_client.post(
self.external_token_url,
data=token_data,
)
response.raise_for_status()
external_tokens = response.json()
# Update stored tokens
new_access_token = external_tokens.get("access_token")
new_refresh_token = external_tokens.get("refresh_token", refresh_token.token)
expires_in = external_tokens.get("expires_in", 3600)
self.tokens[new_access_token] = {
"client_id": client.client_id,
"external_token": new_access_token,
"scopes": scopes or refresh_token.scopes,
}
if new_refresh_token != refresh_token.token:
self.tokens[new_refresh_token] = {
"client_id": client.client_id,
"external_token": new_refresh_token,
"scopes": scopes or refresh_token.scopes,
}
del self.tokens[refresh_token.token]
return OAuthToken(
access_token=new_access_token,
token_type="bearer",
expires_in=expires_in,
refresh_token=new_refresh_token,
scope=" ".join(scopes or refresh_token.scopes),
)
async def load_access_token(self, token: str) -> Optional[ExternalAccessToken]:
"""Load and validate an access token."""
token_info = self.tokens.get(token)
if token_info:
return ExternalAccessToken(
token=token,
client_id=token_info["client_id"],
scopes=token_info["scopes"],
external_token=token_info.get("external_token"),
)
return None
async def revoke_token(self, token: ExternalAccessToken | ExternalRefreshToken) -> None:
"""Revoke a token."""
self.tokens.pop(token.token, None)
def create_github_provider() -> ExternalOAuthProvider:
"""Create an OAuth provider for GitHub integration."""
return ExternalOAuthProvider(
issuer_url=os.getenv("FASTMCP_AUTH_ISSUER_URL", "http://localhost:8000"),
external_provider="github",
external_client_id=os.getenv("GITHUB_CLIENT_ID", ""),
external_client_secret=os.getenv("GITHUB_CLIENT_SECRET", ""),
external_authorize_url="https://github.com/login/oauth/authorize",
external_token_url="https://github.com/login/oauth/access_token",
external_userinfo_url="https://api.github.com/user",
)
def create_google_provider() -> ExternalOAuthProvider:
"""Create an OAuth provider for Google integration."""
return ExternalOAuthProvider(
issuer_url=os.getenv("FASTMCP_AUTH_ISSUER_URL", "http://localhost:8000"),
external_provider="google",
external_client_id=os.getenv("GOOGLE_CLIENT_ID", ""),
external_client_secret=os.getenv("GOOGLE_CLIENT_SECRET", ""),
external_authorize_url="https://accounts.google.com/o/oauth2/v2/auth",
external_token_url="https://oauth2.googleapis.com/token",
external_userinfo_url="https://www.googleapis.com/oauth2/v1/userinfo",
)
src/basic_memory/mcp/main.py
-24
View File
@@ -1,24 +0,0 @@
"""Main MCP entrypoint for Basic Memory.
Creates and configures the shared MCP instance and handles server startup.
"""
from loguru import logger # pragma: no cover
from basic_memory.config import config # pragma: no cover
# Import shared mcp instance
from basic_memory.mcp.server import mcp # pragma: no cover
# Import tools to register them
import basic_memory.mcp.tools # noqa: F401 # pragma: no cover
# Import prompts to register them
import basic_memory.mcp.prompts # noqa: F401 # pragma: no cover
if __name__ == "__main__": # pragma: no cover
home_dir = config.home
logger.info("Starting Basic Memory MCP server")
logger.info(f"Home directory: {home_dir}")
mcp.run()
src/basic_memory/mcp/prompts/continue_conversation.py
+18 -68
View File
@@ -4,20 +4,17 @@ These prompts help users continue conversations and work across sessions,
providing context from previous interactions to maintain continuity.
"""
from textwrap import dedent
from typing import Annotated, Optional
from loguru import logger
from pydantic import Field
from basic_memory.mcp.prompts.utils import PromptContext, PromptContextItem, format_prompt_context
from basic_memory.config import get_project_config
from basic_memory.mcp.async_client import client
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.build_context import build_context
from basic_memory.mcp.tools.recent_activity import recent_activity
from basic_memory.mcp.tools.search import search_notes
from basic_memory.mcp.tools.utils import call_post
from basic_memory.schemas.base import TimeFrame
from basic_memory.schemas.memory import GraphContext
from basic_memory.schemas.search import SearchItemType
from basic_memory.schemas.prompt import ContinueConversationRequest
@mcp.prompt(
@@ -45,67 +42,20 @@ async def continue_conversation(
"""
logger.info(f"Continuing session, topic: {topic}, timeframe: {timeframe}")
# If topic provided, search for it
if topic:
search_results = await search_notes(
query=topic, after_date=timeframe, entity_types=[SearchItemType.ENTITY]
)
# Create request model
request = ContinueConversationRequest( # pyright: ignore [reportCallIssue]
topic=topic, timeframe=timeframe
)
# Build context from results
contexts = []
for result in search_results.results:
if hasattr(result, "permalink") and result.permalink:
context: GraphContext = await build_context(f"memory://{result.permalink}")
if context.primary_results:
contexts.append(
PromptContextItem(
primary_results=context.primary_results[:1], # pyright: ignore
related_results=context.related_results[:3], # pyright: ignore
)
)
project_url = get_project_config().project_url
# get context for the top 3 results
prompt_context = format_prompt_context(
PromptContext(topic=topic, timeframe=timeframe, results=contexts) # pyright: ignore
)
# Call the prompt API endpoint
response = await call_post(
client,
f"{project_url}/prompt/continue-conversation",
json=request.model_dump(exclude_none=True),
)
else:
# If no topic, get recent activity
timeframe = timeframe or "7d"
recent: GraphContext = await recent_activity(
timeframe=timeframe, type=[SearchItemType.ENTITY]
)
prompt_context = format_prompt_context(
PromptContext(
topic=f"Recent Activity from ({timeframe})",
timeframe=timeframe,
results=[
PromptContextItem(
primary_results=recent.primary_results[:5], # pyright: ignore
related_results=recent.related_results[:2], # pyright: ignore
)
],
)
)
# Add next steps with strong encouragement to write
next_steps = dedent(f"""
## Next Steps
You can:
- Explore more with: `search_notes({{"text": "{topic}"}})`
- See what's changed: `recent_activity(timeframe="{timeframe or "7d"}")`
- **Record new learnings or decisions from this conversation:** `write_note(title="[Create a meaningful title]", content="[Content with observations and relations]")`
## Knowledge Capture Recommendation
As you continue this conversation, **actively look for opportunities to:**
1. Record key information, decisions, or insights that emerge
2. Link new knowledge to existing topics
3. Suggest capturing important context when appropriate
4. Create forward references to topics that might be created later
Remember that capturing knowledge during conversations is one of the most valuable aspects of Basic Memory.
""")
return prompt_context + next_steps
# Extract the rendered prompt from the response
result = response.json()
return result["prompt"]
src/basic_memory/mcp/prompts/recent_activity.py
+19 -3
View File
@@ -40,20 +40,36 @@ async def recent_activity_prompt(
recent = await recent_activity(timeframe=timeframe, type=[SearchItemType.ENTITY])
# Extract primary results from the hierarchical structure
primary_results = []
related_results = []
if recent.results:
# Take up to 5 primary results
for item in recent.results[:5]:
primary_results.append(item.primary_result)
# Add up to 2 related results per primary item
if item.related_results:
related_results.extend(item.related_results[:2])
prompt_context = format_prompt_context(
PromptContext(
topic=f"Recent Activity from ({timeframe})",
timeframe=timeframe,
results=[
PromptContextItem(
primary_results=recent.primary_results[:5],
related_results=recent.related_results[:2],
primary_results=primary_results,
related_results=related_results[:10], # Limit total related results
)
],
)
)
# Add suggestions for summarizing recent activity
first_title = "Recent Topic"
if primary_results and len(primary_results) > 0:
first_title = primary_results[0].title
capture_suggestions = f"""
## Opportunity to Capture Activity Summary
@@ -76,7 +92,7 @@ async def recent_activity_prompt(
- [insight] [Connection between different activities]
## Relations
- summarizes [[{recent.primary_results[0].title if recent.primary_results else "Recent Topic"}]]
- summarizes [[{first_title}]]
- relates_to [[Project Overview]]
'''
)
src/basic_memory/mcp/prompts/search.py
+14 -140
View File
@@ -3,16 +3,17 @@
These prompts help users search and explore their knowledge base.
"""
from textwrap import dedent
from typing import Annotated, Optional
from loguru import logger
from pydantic import Field
from basic_memory.config import get_project_config
from basic_memory.mcp.async_client import client
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.search import search_notes as search_tool
from basic_memory.mcp.tools.utils import call_post
from basic_memory.schemas.base import TimeFrame
from basic_memory.schemas.search import SearchResponse
from basic_memory.schemas.prompt import SearchPromptRequest
@mcp.prompt(
@@ -40,143 +41,16 @@ async def search_prompt(
"""
logger.info(f"Searching knowledge base, query: {query}, timeframe: {timeframe}")
search_results = await search_tool(query=query, after_date=timeframe)
return format_search_results(query, search_results, timeframe)
# Create request model
request = SearchPromptRequest(query=query, timeframe=timeframe)
project_url = get_project_config().project_url
def format_search_results(
query: str, results: SearchResponse, timeframe: Optional[TimeFrame] = None
) -> str:
"""Format search results into a helpful summary.
# Call the prompt API endpoint
response = await call_post(
client, f"{project_url}/prompt/search", json=request.model_dump(exclude_none=True)
)
Args:
query: The search query
results: Search results object
timeframe: How far back results were searched
Returns:
Formatted search results summary
"""
if not results.results:
return dedent(f"""
# Search Results for: "{query}"
I couldn't find any results for this query.
## Opportunity to Capture Knowledge!
This is an excellent opportunity to create new knowledge on this topic. Consider:
```python
await write_note(
title="{query.capitalize()}",
content=f'''
# {query.capitalize()}
## Overview
[Summary of what we've discussed about {query}]
## Observations
- [category] [First observation about {query}]
- [category] [Second observation about {query}]
## Relations
- relates_to [[Other Relevant Topic]]
'''
)
```
## Other Suggestions
- Try a different search term
- Broaden your search criteria
- Check recent activity with `recent_activity(timeframe="1w")`
""")
# Start building our summary with header
time_info = f" (after {timeframe})" if timeframe else ""
summary = dedent(f"""
# Search Results for: "{query}"{time_info}
This is a memory search session.
Please use the available basic-memory tools to gather relevant context before responding.
I found {len(results.results)} results that match your query.
Here are the most relevant results:
""")
# Add each search result
for i, result in enumerate(results.results[:5]): # Limit to top 5 results
summary += dedent(f"""
## {i + 1}. {result.title}
- **Type**: {result.type.value}
""")
# Add creation date if available in metadata
if result.metadata and "created_at" in result.metadata:
created_at = result.metadata["created_at"]
if hasattr(created_at, "strftime"):
summary += (
f"- **Created**: {created_at.strftime('%Y-%m-%d %H:%M')}\n" # pragma: no cover
)
elif isinstance(created_at, str):
summary += f"- **Created**: {created_at}\n"
# Add score and excerpt
summary += f"- **Relevance Score**: {result.score:.2f}\n"
# Add excerpt if available in metadata
if result.content:
summary += f"- **Excerpt**:\n{result.content}\n"
# Add permalink for retrieving content
if result.permalink:
summary += dedent(f"""
You can view this content with: `read_note("{result.permalink}")`
Or explore its context with: `build_context("memory://{result.permalink}")`
""")
else:
summary += dedent(f"""
You can view this file with: `read_file("{result.file_path}")`
""") # pragma: no cover
# Add next steps with strong write encouragement
summary += dedent(f"""
## Next Steps
You can:
- Refine your search: `search_notes("{query} AND additional_term")`
- Exclude terms: `search_notes("{query} NOT exclude_term")`
- View more results: `search_notes("{query}", after_date=None)`
- Check recent activity: `recent_activity()`
## Synthesize and Capture Knowledge
Consider creating a new note that synthesizes what you've learned:
```python
await write_note(
title="Synthesis of {query.capitalize()} Information",
content='''
# Synthesis of {query.capitalize()} Information
## Overview
[Synthesis of the search results and your conversation]
## Key Insights
[Summary of main points learned from these results]
## Observations
- [insight] [Important observation from search results]
- [connection] [How this connects to other topics]
## Relations
- relates_to [[{results.results[0].title if results.results else "Related Topic"}]]
- extends [[Another Relevant Topic]]
'''
)
```
Remember that capturing synthesized knowledge is one of the most valuable features of Basic Memory.
""")
return summary
# Extract the rendered prompt from the response
result = response.json()
return result["prompt"]
src/basic_memory/mcp/prompts/utils.py
+3 -3
View File
@@ -35,7 +35,7 @@ def format_prompt_context(context: PromptContext) -> str:
Returns:
Formatted continuation summary
"""
if not context.results:
if not context.results: # pragma: no cover
return dedent(f"""
# Continuing conversation on: {context.topic}
@@ -138,11 +138,11 @@ def format_prompt_context(context: PromptContext) -> str:
- type: **{related.type}**
- title: {related.title}
""")
if related.permalink:
if related.permalink: # pragma: no cover
section_content += (
f'You can view this document with: `read_note("{related.permalink}")`'
)
else:
else: # pragma: no cover
section_content += (
f'You can view this file with: `read_file("{related.file_path}")`'
)
src/basic_memory/mcp/server.py
+77 -8
View File
@@ -1,19 +1,31 @@
"""Enhanced FastMCP server instance for Basic Memory."""
"""
Basic Memory FastMCP server.
"""
import asyncio
from contextlib import asynccontextmanager
from typing import AsyncIterator, Optional
from mcp.server.fastmcp import FastMCP
from mcp.server.fastmcp.utilities.logging import configure_logging as mcp_configure_logging
from dataclasses import dataclass
from typing import AsyncIterator, Optional, Any
from basic_memory.config import config as project_config
from dotenv import load_dotenv
from fastmcp import FastMCP
from fastmcp.utilities.logging import configure_logging as mcp_configure_logging
from mcp.server.auth.settings import AuthSettings
from basic_memory.config import app_config
from basic_memory.services.initialization import initialize_app
from basic_memory.mcp.auth_provider import BasicMemoryOAuthProvider
from basic_memory.mcp.external_auth_provider import (
create_github_provider,
create_google_provider,
)
from basic_memory.mcp.supabase_auth_provider import SupabaseOAuthProvider
# mcp console logging
mcp_configure_logging(level="ERROR")
load_dotenv()
@dataclass
class AppContext:
@@ -24,7 +36,7 @@ class AppContext:
async def app_lifespan(server: FastMCP) -> AsyncIterator[AppContext]: # pragma: no cover
"""Manage application lifecycle with type-safe context"""
# Initialize on startup
watch_task = await initialize_app(project_config)
watch_task = await initialize_app(app_config)
try:
yield AppContext(watch_task=watch_task)
finally:
@@ -33,5 +45,62 @@ async def app_lifespan(server: FastMCP) -> AsyncIterator[AppContext]: # pragma:
watch_task.cancel()
# OAuth configuration function
def create_auth_config() -> tuple[AuthSettings | None, Any | None]:
"""Create OAuth configuration if enabled."""
# Check if OAuth is enabled via environment variable
import os
if os.getenv("FASTMCP_AUTH_ENABLED", "false").lower() == "true":
from pydantic import AnyHttpUrl
# Configure OAuth settings
issuer_url = os.getenv("FASTMCP_AUTH_ISSUER_URL", "http://localhost:8000")
required_scopes = os.getenv("FASTMCP_AUTH_REQUIRED_SCOPES", "read,write")
docs_url = os.getenv("FASTMCP_AUTH_DOCS_URL") or "http://localhost:8000/docs/oauth"
auth_settings = AuthSettings(
issuer_url=AnyHttpUrl(issuer_url),
service_documentation_url=AnyHttpUrl(docs_url),
required_scopes=required_scopes.split(",") if required_scopes else ["read", "write"],
)
# Create OAuth provider based on type
provider_type = os.getenv("FASTMCP_AUTH_PROVIDER", "basic").lower()
if provider_type == "github":
auth_provider = create_github_provider()
elif provider_type == "google":
auth_provider = create_google_provider()
elif provider_type == "supabase":
supabase_url = os.getenv("SUPABASE_URL")
supabase_anon_key = os.getenv("SUPABASE_ANON_KEY")
supabase_service_key = os.getenv("SUPABASE_SERVICE_KEY")
if not supabase_url or not supabase_anon_key:
raise ValueError("SUPABASE_URL and SUPABASE_ANON_KEY must be set for Supabase auth")
auth_provider = SupabaseOAuthProvider(
supabase_url=supabase_url,
supabase_anon_key=supabase_anon_key,
supabase_service_key=supabase_service_key,
issuer_url=issuer_url,
)
else: # default to "basic"
auth_provider = BasicMemoryOAuthProvider(issuer_url=issuer_url)
return auth_settings, auth_provider
return None, None
# Create auth configuration
auth_settings, auth_provider = create_auth_config()
# Create the shared server instance
mcp = FastMCP("Basic Memory", log_level="ERROR", lifespan=app_lifespan)
mcp = FastMCP(
name="Basic Memory",
log_level="DEBUG",
auth_server_provider=auth_provider,
auth=auth_settings,
)
src/basic_memory/mcp/supabase_auth_provider.py
+463
View File
@@ -0,0 +1,463 @@
"""Supabase OAuth provider for Basic Memory MCP server."""
import os
import secrets
from dataclasses import dataclass
from datetime import datetime, timedelta
from typing import Optional, Dict, Any
import httpx
import jwt
from loguru import logger
from mcp.server.auth.provider import (
OAuthAuthorizationServerProvider,
AuthorizationParams,
AuthorizationCode,
RefreshToken,
AccessToken,
TokenError,
AuthorizeError,
)
from mcp.shared.auth import OAuthClientInformationFull, OAuthToken
@dataclass
class SupabaseAuthorizationCode(AuthorizationCode):
"""Authorization code with Supabase metadata."""
user_id: Optional[str] = None
email: Optional[str] = None
@dataclass
class SupabaseRefreshToken(RefreshToken):
"""Refresh token with Supabase metadata."""
supabase_refresh_token: Optional[str] = None
user_id: Optional[str] = None
@dataclass
class SupabaseAccessToken(AccessToken):
"""Access token with Supabase metadata."""
supabase_access_token: Optional[str] = None
user_id: Optional[str] = None
email: Optional[str] = None
class SupabaseOAuthProvider(
OAuthAuthorizationServerProvider[
SupabaseAuthorizationCode, SupabaseRefreshToken, SupabaseAccessToken
]
):
"""OAuth provider that integrates with Supabase Auth.
This provider uses Supabase as the authentication backend while
maintaining compatibility with MCP's OAuth requirements.
"""
def __init__(
self,
supabase_url: str,
supabase_anon_key: str,
supabase_service_key: Optional[str] = None,
issuer_url: str = "http://localhost:8000",
):
self.supabase_url = supabase_url.rstrip("/")
self.supabase_anon_key = supabase_anon_key
self.supabase_service_key = supabase_service_key or supabase_anon_key
self.issuer_url = issuer_url
# HTTP client for Supabase API calls
self.http_client = httpx.AsyncClient()
# Temporary storage for auth flows (in production, use Supabase DB)
self.pending_auth_codes: Dict[str, SupabaseAuthorizationCode] = {}
self.mcp_to_supabase_tokens: Dict[str, Dict[str, Any]] = {}
async def get_client(self, client_id: str) -> Optional[OAuthClientInformationFull]:
"""Get a client from Supabase.
In production, this would query a clients table in Supabase.
"""
# For now, we'll validate against a configured list of allowed clients
# In production, query Supabase DB for client info
allowed_clients = os.getenv("SUPABASE_ALLOWED_CLIENTS", "").split(",")
if client_id in allowed_clients:
return OAuthClientInformationFull(
client_id=client_id,
client_secret="", # Supabase handles secrets
redirect_uris=[], # Supabase handles redirect URIs
)
return None
async def register_client(self, client_info: OAuthClientInformationFull) -> None:
"""Register a new OAuth client in Supabase.
In production, this would insert into a clients table.
"""
# For development, we just log the registration
logger.info(f"Would register client {client_info.client_id} in Supabase")
# In production:
# await self.supabase.table('oauth_clients').insert({
# 'client_id': client_info.client_id,
# 'client_secret': client_info.client_secret,
# 'metadata': client_info.client_metadata,
# }).execute()
async def authorize(
self, client: OAuthClientInformationFull, params: AuthorizationParams
) -> str:
"""Create authorization URL redirecting to Supabase Auth.
This initiates the OAuth flow with Supabase as the identity provider.
"""
# Generate state for this auth request
state = secrets.token_urlsafe(32)
# Store the authorization request
self.pending_auth_codes[state] = SupabaseAuthorizationCode(
code=state,
scopes=params.scopes or [],
expires_at=(datetime.utcnow() + timedelta(minutes=10)).timestamp(),
client_id=client.client_id,
code_challenge=params.code_challenge,
redirect_uri=params.redirect_uri,
redirect_uri_provided_explicitly=params.redirect_uri_provided_explicitly,
)
# Build Supabase auth URL
auth_params = {
"redirect_to": f"{self.issuer_url}/auth/callback",
"scopes": " ".join(params.scopes or ["openid", "email"]),
"state": state,
}
# Use Supabase's OAuth endpoint
auth_url = f"{self.supabase_url}/auth/v1/authorize"
query_string = "&".join(f"{k}={v}" for k, v in auth_params.items())
return f"{auth_url}?{query_string}"
async def handle_supabase_callback(self, code: str, state: str) -> str:
"""Handle callback from Supabase after user authentication."""
# Get the original auth request
auth_request = self.pending_auth_codes.get(state)
if not auth_request:
raise AuthorizeError(
error="invalid_request",
error_description="Invalid state parameter",
)
# Exchange code with Supabase for tokens
token_response = await self.http_client.post(
f"{self.supabase_url}/auth/v1/token",
json={
"grant_type": "authorization_code",
"code": code,
"redirect_uri": f"{self.issuer_url}/auth/callback",
},
headers={
"apikey": self.supabase_anon_key,
"Authorization": f"Bearer {self.supabase_anon_key}",
},
)
if not token_response.is_success:
raise AuthorizeError(
error="server_error",
error_description="Failed to exchange code with Supabase",
)
supabase_tokens = token_response.json()
# Get user info from Supabase
user_response = await self.http_client.get(
f"{self.supabase_url}/auth/v1/user",
headers={
"apikey": self.supabase_anon_key,
"Authorization": f"Bearer {supabase_tokens['access_token']}",
},
)
user_data = user_response.json() if user_response.is_success else {}
# Generate MCP authorization code
mcp_code = secrets.token_urlsafe(32)
# Update auth request with user info
auth_request.code = mcp_code
auth_request.user_id = user_data.get("id")
auth_request.email = user_data.get("email")
# Store mapping
self.pending_auth_codes[mcp_code] = auth_request
self.mcp_to_supabase_tokens[mcp_code] = {
"supabase_tokens": supabase_tokens,
"user": user_data,
}
# Clean up old state
del self.pending_auth_codes[state]
# Redirect back to client
redirect_uri = str(auth_request.redirect_uri)
separator = "&" if "?" in redirect_uri else "?"
return f"{redirect_uri}{separator}code={mcp_code}&state={state}"
async def load_authorization_code(
self, client: OAuthClientInformationFull, authorization_code: str
) -> Optional[SupabaseAuthorizationCode]:
"""Load an authorization code."""
code = self.pending_auth_codes.get(authorization_code)
if code and code.client_id == client.client_id:
# Check expiration
if datetime.utcnow().timestamp() > code.expires_at:
del self.pending_auth_codes[authorization_code]
return None
return code
return None
async def exchange_authorization_code(
self, client: OAuthClientInformationFull, authorization_code: SupabaseAuthorizationCode
) -> OAuthToken:
"""Exchange authorization code for tokens."""
# Get stored Supabase tokens
token_data = self.mcp_to_supabase_tokens.get(authorization_code.code)
if not token_data:
raise TokenError(error="invalid_grant", error_description="Invalid authorization code")
supabase_tokens = token_data["supabase_tokens"]
user = token_data["user"]
# Generate MCP tokens that wrap Supabase tokens
access_token = self._generate_mcp_token(
client_id=client.client_id,
user_id=user.get("id", ""),
email=user.get("email", ""),
scopes=authorization_code.scopes,
supabase_access_token=supabase_tokens["access_token"],
)
refresh_token = secrets.token_urlsafe(32)
# Store the token mapping
self.mcp_to_supabase_tokens[access_token] = {
"client_id": client.client_id,
"user_id": user.get("id"),
"email": user.get("email"),
"supabase_access_token": supabase_tokens["access_token"],
"supabase_refresh_token": supabase_tokens["refresh_token"],
"scopes": authorization_code.scopes,
}
# Store refresh token mapping
self.mcp_to_supabase_tokens[refresh_token] = {
"client_id": client.client_id,
"user_id": user.get("id"),
"supabase_refresh_token": supabase_tokens["refresh_token"],
"scopes": authorization_code.scopes,
}
# Clean up authorization code
del self.pending_auth_codes[authorization_code.code]
del self.mcp_to_supabase_tokens[authorization_code.code]
return OAuthToken(
access_token=access_token,
token_type="bearer",
expires_in=supabase_tokens.get("expires_in", 3600),
refresh_token=refresh_token,
scope=" ".join(authorization_code.scopes) if authorization_code.scopes else None,
)
async def load_refresh_token(
self, client: OAuthClientInformationFull, refresh_token: str
) -> Optional[SupabaseRefreshToken]:
"""Load a refresh token."""
token_data = self.mcp_to_supabase_tokens.get(refresh_token)
if token_data and token_data["client_id"] == client.client_id:
return SupabaseRefreshToken(
token=refresh_token,
client_id=client.client_id,
scopes=token_data["scopes"],
supabase_refresh_token=token_data["supabase_refresh_token"],
user_id=token_data.get("user_id"),
)
return None
async def exchange_refresh_token(
self,
client: OAuthClientInformationFull,
refresh_token: SupabaseRefreshToken,
scopes: list[str],
) -> OAuthToken:
"""Exchange refresh token for new tokens using Supabase."""
# Refresh with Supabase
token_response = await self.http_client.post(
f"{self.supabase_url}/auth/v1/token",
json={
"grant_type": "refresh_token",
"refresh_token": refresh_token.supabase_refresh_token,
},
headers={
"apikey": self.supabase_anon_key,
"Authorization": f"Bearer {self.supabase_anon_key}",
},
)
if not token_response.is_success:
raise TokenError(
error="invalid_grant",
error_description="Failed to refresh with Supabase",
)
supabase_tokens = token_response.json()
# Get updated user info
user_response = await self.http_client.get(
f"{self.supabase_url}/auth/v1/user",
headers={
"apikey": self.supabase_anon_key,
"Authorization": f"Bearer {supabase_tokens['access_token']}",
},
)
user_data = user_response.json() if user_response.is_success else {}
# Generate new MCP tokens
new_access_token = self._generate_mcp_token(
client_id=client.client_id,
user_id=user_data.get("id", ""),
email=user_data.get("email", ""),
scopes=scopes or refresh_token.scopes,
supabase_access_token=supabase_tokens["access_token"],
)
new_refresh_token = secrets.token_urlsafe(32)
# Update token mappings
self.mcp_to_supabase_tokens[new_access_token] = {
"client_id": client.client_id,
"user_id": user_data.get("id"),
"email": user_data.get("email"),
"supabase_access_token": supabase_tokens["access_token"],
"supabase_refresh_token": supabase_tokens["refresh_token"],
"scopes": scopes or refresh_token.scopes,
}
self.mcp_to_supabase_tokens[new_refresh_token] = {
"client_id": client.client_id,
"user_id": user_data.get("id"),
"supabase_refresh_token": supabase_tokens["refresh_token"],
"scopes": scopes or refresh_token.scopes,
}
# Clean up old tokens
del self.mcp_to_supabase_tokens[refresh_token.token]
return OAuthToken(
access_token=new_access_token,
token_type="bearer",
expires_in=supabase_tokens.get("expires_in", 3600),
refresh_token=new_refresh_token,
scope=" ".join(scopes or refresh_token.scopes),
)
async def load_access_token(self, token: str) -> Optional[SupabaseAccessToken]:
"""Load and validate an access token."""
# First check our mapping
token_data = self.mcp_to_supabase_tokens.get(token)
if token_data:
return SupabaseAccessToken(
token=token,
client_id=token_data["client_id"],
scopes=token_data["scopes"],
supabase_access_token=token_data.get("supabase_access_token"),
user_id=token_data.get("user_id"),
email=token_data.get("email"),
)
# Try to decode as JWT
try:
# Verify with Supabase's JWT secret
payload = jwt.decode(
token,
os.getenv("SUPABASE_JWT_SECRET", ""),
algorithms=["HS256"],
audience="authenticated",
)
return SupabaseAccessToken(
token=token,
client_id=payload.get("client_id", ""),
scopes=payload.get("scopes", []),
user_id=payload.get("sub"),
email=payload.get("email"),
)
except jwt.InvalidTokenError:
pass
# Validate with Supabase
user_response = await self.http_client.get(
f"{self.supabase_url}/auth/v1/user",
headers={
"apikey": self.supabase_anon_key,
"Authorization": f"Bearer {token}",
},
)
if user_response.is_success:
user_data = user_response.json()
return SupabaseAccessToken(
token=token,
client_id="", # Unknown client for direct Supabase tokens
scopes=[],
supabase_access_token=token,
user_id=user_data.get("id"),
email=user_data.get("email"),
)
return None
async def revoke_token(self, token: SupabaseAccessToken | SupabaseRefreshToken) -> None:
"""Revoke a token."""
# Remove from our mapping
self.mcp_to_supabase_tokens.pop(token.token, None)
# In production, also revoke in Supabase:
# await self.supabase.auth.admin.sign_out(token.user_id)
def _generate_mcp_token(
self,
client_id: str,
user_id: str,
email: str,
scopes: list[str],
supabase_access_token: str,
) -> str:
"""Generate an MCP token that wraps Supabase authentication."""
payload = {
"iss": self.issuer_url,
"sub": user_id,
"client_id": client_id,
"email": email,
"scopes": scopes,
"supabase_token": supabase_access_token[:10] + "...", # Reference only
"exp": datetime.utcnow() + timedelta(hours=1),
"iat": datetime.utcnow(),
}
# Use Supabase JWT secret if available
secret = os.getenv("SUPABASE_JWT_SECRET", secrets.token_urlsafe(32))
return jwt.encode(payload, secret, algorithm="HS256")
src/basic_memory/mcp/tools/build_context.py
+5 -1
View File
@@ -4,6 +4,7 @@ from typing import Optional
from loguru import logger
from basic_memory.config import get_project_config
from basic_memory.mcp.async_client import client
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_get
@@ -71,9 +72,12 @@ async def build_context(
"""
logger.info(f"Building context from {url}")
url = normalize_memory_url(url)
project_url = get_project_config().project_url
response = await call_get(
client,
f"/memory/{memory_url_path(url)}",
f"{project_url}/memory/{memory_url_path(url)}",
params={
"depth": depth,
"timeframe": timeframe,
src/basic_memory/mcp/tools/canvas.py
+4 -1
View File
@@ -8,6 +8,7 @@ from typing import Dict, List, Any
from loguru import logger
from basic_memory.config import get_project_config
from basic_memory.mcp.async_client import client
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_put
@@ -72,6 +73,8 @@ async def canvas(
}
```
"""
project_url = get_project_config().project_url
# Ensure path has .canvas extension
file_title = title if title.endswith(".canvas") else f"{title}.canvas"
file_path = f"{folder}/{file_title}"
@@ -84,7 +87,7 @@ async def canvas(
# Write the file using the resource API
logger.info(f"Creating canvas file: {file_path}")
response = await call_put(client, f"/resource/{file_path}", json=canvas_json)
response = await call_put(client, f"{project_url}/resource/{file_path}", json=canvas_json)
# Parse response
result = response.json()
src/basic_memory/mcp/tools/delete_note.py
+4 -1
View File
@@ -1,3 +1,4 @@
from basic_memory.config import get_project_config
from basic_memory.mcp.tools.utils import call_delete
@@ -23,6 +24,8 @@ async def delete_note(identifier: str) -> bool:
# Delete by permalink
delete_note("notes/project-planning")
"""
response = await call_delete(client, f"/knowledge/entities/{identifier}")
project_url = get_project_config().project_url
response = await call_delete(client, f"{project_url}/knowledge/entities/{identifier}")
result = DeleteEntitiesResponse.model_validate(response.json())
return result.deleted
src/basic_memory/mcp/tools/project_info.py
+3 -1
View File
@@ -2,6 +2,7 @@
from loguru import logger
from basic_memory.config import get_project_config
from basic_memory.mcp.async_client import client
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_get
@@ -43,9 +44,10 @@ async def project_info() -> ProjectInfoResponse:
print(f"Basic Memory version: {info.system.version}")
"""
logger.info("Getting project info")
project_url = get_project_config().project_url
# Call the API endpoint
response = await call_get(client, "/stats/project-info")
response = await call_get(client, f"{project_url}/project/info")
# Convert response to ProjectInfoResponse
return ProjectInfoResponse.model_validate(response.json())
src/basic_memory/mcp/tools/read_content.py
+4 -1
View File
@@ -7,6 +7,7 @@ Files are read directly without any knowledge graph processing.
from loguru import logger
from basic_memory.config import get_project_config
from basic_memory.mcp.server import mcp
from basic_memory.mcp.async_client import client
from basic_memory.mcp.tools.utils import call_get
@@ -178,8 +179,10 @@ async def read_content(path: str) -> dict:
"""
logger.info("Reading file", path=path)
project_url = get_project_config().project_url
url = memory_url_path(path)
response = await call_get(client, f"/resource/{url}")
response = await call_get(client, f"{project_url}/resource/{url}")
content_type = response.headers.get("content-type", "application/octet-stream")
content_length = int(response.headers.get("content-length", 0))
src/basic_memory/mcp/tools/read_note.py
+6 -2
View File
@@ -4,6 +4,7 @@ from textwrap import dedent
from loguru import logger
from basic_memory.config import get_project_config
from basic_memory.mcp.async_client import client
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.search import search_notes
@@ -43,9 +44,12 @@ async def read_note(identifier: str, page: int = 1, page_size: int = 10) -> str:
# Read with pagination
read_note("Project Updates", page=2, page_size=5)
"""
project_url = get_project_config().project_url
# Get the file via REST API - first try direct permalink lookup
entity_path = memory_url_path(identifier)
path = f"/resource/{entity_path}"
path = f"{project_url}/resource/{entity_path}"
logger.info(f"Attempting to read note from URL: {path}")
try:
@@ -69,7 +73,7 @@ async def read_note(identifier: str, page: int = 1, page_size: int = 10) -> str:
if result.permalink:
try:
# Try to fetch the content using the found permalink
path = f"/resource/{result.permalink}"
path = f"{project_url}/resource/{result.permalink}"
response = await call_get(
client, path, params={"page": page, "page_size": page_size}
)
src/basic_memory/mcp/tools/recent_activity.py
+4 -1
View File
@@ -4,6 +4,7 @@ from typing import List, Union
from loguru import logger
from basic_memory.config import get_project_config
from basic_memory.mcp.async_client import client
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_get
@@ -114,9 +115,11 @@ async def recent_activity(
# Add validated types to params
params["type"] = [t.value for t in validated_types] # pyright: ignore
project_url = get_project_config().project_url
response = await call_get(
client,
"/memory/recent",
f"{project_url}/memory/recent",
params=params,
)
return GraphContext.model_validate(response.json())
src/basic_memory/mcp/tools/search.py
+4 -1
View File
@@ -4,6 +4,7 @@ from typing import List, Optional
from loguru import logger
from basic_memory.config import get_project_config
from basic_memory.mcp.async_client import client
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_post
@@ -103,10 +104,12 @@ async def search_notes(
if after_date:
search_query.after_date = after_date
project_url = get_project_config().project_url
logger.info(f"Searching for {search_query}")
response = await call_post(
client,
"/search/",
f"{project_url}/search/",
json=search_query.model_dump(),
params={"page": page, "page_size": page_size},
)
src/basic_memory/mcp/tools/utils.py
+1
View File
@@ -282,6 +282,7 @@ async def call_post(
timeout=timeout,
extensions=extensions,
)
logger.debug(f"response: {response.json()}")
if response.is_success:
return response
src/basic_memory/mcp/tools/write_note.py
+4 -1
View File
@@ -10,6 +10,7 @@ from basic_memory.mcp.tools.utils import call_put
from basic_memory.schemas import EntityResponse
from basic_memory.schemas.base import Entity
from basic_memory.utils import parse_tags
from basic_memory.config import get_project_config
# Define TagType as a Union that can accept either a string or a list of strings or None
TagType = Union[List[str], str, None]
@@ -78,10 +79,12 @@ async def write_note(
content=content,
entity_metadata=metadata,
)
project_url = get_project_config().project_url
print(f"project_url: {project_url}")
# Create or update via knowledge API
logger.debug("Creating entity via API", permalink=entity.permalink)
url = f"/knowledge/entities/{entity.permalink}"
url = f"{project_url}/knowledge/entities/{entity.permalink}"
response = await call_put(client, url, json=entity.model_dump())
result = EntityResponse.model_validate(response.json())
src/basic_memory/models/__init__.py
+3 -2
View File
@@ -3,12 +3,13 @@
import basic_memory
from basic_memory.models.base import Base
from basic_memory.models.knowledge import Entity, Observation, Relation
SCHEMA_VERSION = basic_memory.__version__ + "-" + "003"
from basic_memory.models.project import Project
__all__ = [
"Base",
"Entity",
"Observation",
"Relation",
"Project",
"basic_memory",
]
src/basic_memory/models/knowledge.py
+16 -4
View File
@@ -17,7 +17,6 @@ from sqlalchemy import (
from sqlalchemy.orm import Mapped, mapped_column, relationship
from basic_memory.models.base import Base
from basic_memory.utils import generate_permalink
@@ -29,6 +28,7 @@ class Entity(Base):
- Maps to a file on disk
- Maintains a checksum for change detection
- Tracks both source file and semantic properties
- Belongs to a specific project
"""
__tablename__ = "entity"
@@ -38,13 +38,21 @@ class Entity(Base):
Index("ix_entity_title", "title"),
Index("ix_entity_created_at", "created_at"), # For timeline queries
Index("ix_entity_updated_at", "updated_at"), # For timeline queries
# Unique index only for markdown files with non-null permalinks
Index("ix_entity_project_id", "project_id"), # For project filtering
# Project-specific uniqueness constraints
Index(
"uix_entity_permalink",
"uix_entity_permalink_project",
"permalink",
"project_id",
unique=True,
sqlite_where=text("content_type = 'text/markdown' AND permalink IS NOT NULL"),
),
Index(
"uix_entity_file_path_project",
"file_path",
"project_id",
unique=True,
),
)
# Core identity
@@ -54,10 +62,13 @@ class Entity(Base):
entity_metadata: Mapped[Optional[dict]] = mapped_column(JSON, nullable=True)
content_type: Mapped[str] = mapped_column(String)
# Project reference
project_id: Mapped[int] = mapped_column(Integer, ForeignKey("project.id"), nullable=False)
# Normalized path for URIs - required for markdown files only
permalink: Mapped[Optional[str]] = mapped_column(String, nullable=True, index=True)
# Actual filesystem relative path
file_path: Mapped[str] = mapped_column(String, unique=True, index=True)
file_path: Mapped[str] = mapped_column(String, index=True)
# checksum of file
checksum: Mapped[Optional[str]] = mapped_column(String, nullable=True)
@@ -66,6 +77,7 @@ class Entity(Base):
updated_at: Mapped[datetime] = mapped_column(DateTime)
# Relationships
project = relationship("Project", back_populates="entities")
observations = relationship(
"Observation", back_populates="entity", cascade="all, delete-orphan"
)
src/basic_memory/models/project.py
+80
View File
@@ -0,0 +1,80 @@
"""Project model for Basic Memory."""
from datetime import datetime
from typing import Optional
from sqlalchemy import (
Integer,
String,
Text,
Boolean,
DateTime,
Index,
event,
)
from sqlalchemy.orm import Mapped, mapped_column, relationship
from basic_memory.models.base import Base
from basic_memory.utils import generate_permalink
class Project(Base):
"""Project model for Basic Memory.
A project represents a collection of knowledge entities that are grouped together.
Projects are stored in the app-level database and provide context for all knowledge
operations.
"""
__tablename__ = "project"
__table_args__ = (
# Regular indexes
Index("ix_project_name", "name", unique=True),
Index("ix_project_permalink", "permalink", unique=True),
Index("ix_project_path", "path"),
Index("ix_project_created_at", "created_at"),
Index("ix_project_updated_at", "updated_at"),
)
# Core identity
id: Mapped[int] = mapped_column(Integer, primary_key=True)
name: Mapped[str] = mapped_column(String, unique=True)
description: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
# URL-friendly identifier generated from name
permalink: Mapped[str] = mapped_column(String, unique=True)
# Filesystem path to project directory
path: Mapped[str] = mapped_column(String)
# Status flags
is_active: Mapped[bool] = mapped_column(Boolean, default=True)
is_default: Mapped[Optional[bool]] = mapped_column(
Boolean, default=None, unique=True, nullable=True
)
# Timestamps
created_at: Mapped[datetime] = mapped_column(DateTime, default=datetime.utcnow)
updated_at: Mapped[datetime] = mapped_column(
DateTime, default=datetime.utcnow, onupdate=datetime.utcnow
)
# Define relationships to entities, observations, and relations
# These relationships will be established once we add project_id to those models
entities = relationship("Entity", back_populates="project", cascade="all, delete-orphan")
def __repr__(self) -> str: # pragma: no cover
return f"Project(id={self.id}, name='{self.name}', permalink='{self.permalink}', path='{self.path}')"
@event.listens_for(Project, "before_insert")
@event.listens_for(Project, "before_update")
def set_project_permalink(mapper, connection, project):
"""Generate URL-friendly permalink for the project if needed.
This event listener ensures the permalink is always derived from the name,
even if the name changes.
"""
# If the name changed or permalink is empty, regenerate permalink
if not project.permalink or project.permalink != generate_permalink(project.name):
project.permalink = generate_permalink(project.name)
src/basic_memory/models/search.py
+8 -5
View File
@@ -13,21 +13,24 @@ CREATE VIRTUAL TABLE IF NOT EXISTS search_index USING fts5(
permalink, -- Stable identifier (now indexed for path search)
file_path UNINDEXED, -- Physical location
type UNINDEXED, -- entity/relation/observation
-- Relation fields
-- Project context
project_id UNINDEXED, -- Project identifier
-- Relation fields
from_id UNINDEXED, -- Source entity
to_id UNINDEXED, -- Target entity
relation_type UNINDEXED, -- Type of relation
-- Observation fields
entity_id UNINDEXED, -- Parent entity
category UNINDEXED, -- Observation category
-- Common fields
metadata UNINDEXED, -- JSON metadata
created_at UNINDEXED, -- Creation timestamp
updated_at UNINDEXED, -- Last update
-- Configuration
tokenize='unicode61 tokenchars 0x2F', -- Hex code for /
prefix='1,2,3,4' -- Support longer prefixes for paths
src/basic_memory/repository/__init__.py
+2
View File
@@ -1,9 +1,11 @@
from .entity_repository import EntityRepository
from .observation_repository import ObservationRepository
from .project_repository import ProjectRepository
from .relation_repository import RelationRepository
__all__ = [
"EntityRepository",
"ObservationRepository",
"ProjectRepository",
"RelationRepository",
]
src/basic_memory/repository/entity_repository.py
+8 -3
View File
@@ -18,9 +18,14 @@ class EntityRepository(Repository[Entity]):
to strings before passing to repository methods.
"""
def __init__(self, session_maker: async_sessionmaker[AsyncSession]):
"""Initialize with session maker."""
super().__init__(session_maker, Entity)
def __init__(self, session_maker: async_sessionmaker[AsyncSession], project_id: int):
"""Initialize with session maker and project_id filter.
Args:
session_maker: SQLAlchemy session maker
project_id: Project ID to filter all operations by
"""
super().__init__(session_maker, Entity, project_id=project_id)
async def get_by_permalink(self, permalink: str) -> Optional[Entity]:
"""Get entity by permalink.
src/basic_memory/repository/observation_repository.py
+35 -3
View File
@@ -1,6 +1,6 @@
"""Repository for managing Observation objects."""
from typing import Sequence
from typing import Dict, List, Sequence
from sqlalchemy import select
from sqlalchemy.ext.asyncio import async_sessionmaker
@@ -12,8 +12,14 @@ from basic_memory.repository.repository import Repository
class ObservationRepository(Repository[Observation]):
"""Repository for Observation model with memory-specific operations."""
def __init__(self, session_maker: async_sessionmaker):
super().__init__(session_maker, Observation)
def __init__(self, session_maker: async_sessionmaker, project_id: int):
"""Initialize with session maker and project_id filter.
Args:
session_maker: SQLAlchemy session maker
project_id: Project ID to filter all operations by
"""
super().__init__(session_maker, Observation, project_id=project_id)
async def find_by_entity(self, entity_id: int) -> Sequence[Observation]:
"""Find all observations for a specific entity."""
@@ -38,3 +44,29 @@ class ObservationRepository(Repository[Observation]):
query = select(Observation.category).distinct()
result = await self.execute_query(query, use_query_options=False)
return result.scalars().all()
async def find_by_entities(self, entity_ids: List[int]) -> Dict[int, List[Observation]]:
"""Find all observations for multiple entities in a single query.
Args:
entity_ids: List of entity IDs to fetch observations for
Returns:
Dictionary mapping entity_id to list of observations
"""
if not entity_ids: # pragma: no cover
return {}
# Query observations for all entities in the list
query = select(Observation).filter(Observation.entity_id.in_(entity_ids))
result = await self.execute_query(query)
observations = result.scalars().all()
# Group observations by entity_id
observations_by_entity = {}
for obs in observations:
if obs.entity_id not in observations_by_entity:
observations_by_entity[obs.entity_id] = []
observations_by_entity[obs.entity_id].append(obs)
return observations_by_entity
src/basic_memory/repository/project_info_repository.py
+3 -2
View File
@@ -1,9 +1,10 @@
from basic_memory.repository.repository import Repository
from basic_memory.models.project import Project
class ProjectInfoRepository(Repository):
"""Repository for statistics queries."""
def __init__(self, session_maker):
# Initialize with a dummy model since we're just using the execute_query method
super().__init__(session_maker, None) # type: ignore
# Initialize with Project model as a reference
super().__init__(session_maker, Project)
src/basic_memory/repository/project_repository.py
+85
View File
@@ -0,0 +1,85 @@
"""Repository for managing projects in Basic Memory."""
from pathlib import Path
from typing import Optional, Sequence, Union
from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from basic_memory import db
from basic_memory.models.project import Project
from basic_memory.repository.repository import Repository
class ProjectRepository(Repository[Project]):
"""Repository for Project model.
Projects represent collections of knowledge entities grouped together.
Each entity, observation, and relation belongs to a specific project.
"""
def __init__(self, session_maker: async_sessionmaker[AsyncSession]):
"""Initialize with session maker."""
super().__init__(session_maker, Project)
async def get_by_name(self, name: str) -> Optional[Project]:
"""Get project by name.
Args:
name: Unique name of the project
"""
query = self.select().where(Project.name == name)
return await self.find_one(query)
async def get_by_permalink(self, permalink: str) -> Optional[Project]:
"""Get project by permalink.
Args:
permalink: URL-friendly identifier for the project
"""
query = self.select().where(Project.permalink == permalink)
return await self.find_one(query)
async def get_by_path(self, path: Union[Path, str]) -> Optional[Project]:
"""Get project by filesystem path.
Args:
path: Path to the project directory (will be converted to string internally)
"""
query = self.select().where(Project.path == str(path))
return await self.find_one(query)
async def get_default_project(self) -> Optional[Project]:
"""Get the default project (the one marked as is_default=True)."""
query = self.select().where(Project.is_default.is_not(None))
return await self.find_one(query)
async def get_active_projects(self) -> Sequence[Project]:
"""Get all active projects."""
query = self.select().where(Project.is_active == True) # noqa: E712
result = await self.execute_query(query)
return list(result.scalars().all())
async def set_as_default(self, project_id: int) -> Optional[Project]:
"""Set a project as the default and unset previous default.
Args:
project_id: ID of the project to set as default
Returns:
The updated project if found, None otherwise
"""
async with db.scoped_session(self.session_maker) as session:
# First, clear the default flag for all projects using direct SQL
await session.execute(
text("UPDATE project SET is_default = NULL WHERE is_default IS NOT NULL")
)
await session.flush()
# Set the new default project
target_project = await self.select_by_id(session, project_id)
if target_project:
target_project.is_default = True
await session.flush()
return target_project
return None # pragma: no cover
src/basic_memory/repository/relation_repository.py
+8 -2
View File
@@ -16,8 +16,14 @@ from basic_memory.repository.repository import Repository
class RelationRepository(Repository[Relation]):
"""Repository for Relation model with memory-specific operations."""
def __init__(self, session_maker: async_sessionmaker):
super().__init__(session_maker, Relation)
def __init__(self, session_maker: async_sessionmaker, project_id: int):
"""Initialize with session maker and project_id filter.
Args:
session_maker: SQLAlchemy session maker
project_id: Project ID to filter all operations by
"""
super().__init__(session_maker, Relation, project_id=project_id)
async def find_relation(
self, from_permalink: str, to_permalink: str, relation_type: str
src/basic_memory/repository/repository.py
+107 -15
View File
@@ -1,6 +1,6 @@
"""Base repository implementation."""
from typing import Type, Optional, Any, Sequence, TypeVar, List
from typing import Type, Optional, Any, Sequence, TypeVar, List, Dict
from loguru import logger
from sqlalchemy import (
@@ -27,13 +27,30 @@ T = TypeVar("T", bound=Base)
class Repository[T: Base]:
"""Base repository implementation with generic CRUD operations."""
def __init__(self, session_maker: async_sessionmaker[AsyncSession], Model: Type[T]):
def __init__(
self,
session_maker: async_sessionmaker[AsyncSession],
Model: Type[T],
project_id: Optional[int] = None,
):
self.session_maker = session_maker
self.project_id = project_id
if Model:
self.Model = Model
self.mapper = inspect(self.Model).mapper
self.primary_key: Column[Any] = self.mapper.primary_key[0]
self.valid_columns = [column.key for column in self.mapper.columns]
# Check if this model has a project_id column
self.has_project_id = "project_id" in self.valid_columns
def _set_project_id_if_needed(self, model: T) -> None:
"""Set project_id on model if needed and available."""
if (
self.has_project_id
and self.project_id is not None
and getattr(model, "project_id", None) is None
):
setattr(model, "project_id", self.project_id)
def get_model_data(self, entity_data):
model_data = {
@@ -41,6 +58,19 @@ class Repository[T: Base]:
}
return model_data
def _add_project_filter(self, query: Select) -> Select:
"""Add project_id filter to query if applicable.
Args:
query: The SQLAlchemy query to modify
Returns:
Updated query with project filter if applicable
"""
if self.has_project_id and self.project_id is not None:
query = query.filter(getattr(self.Model, "project_id") == self.project_id)
return query
async def select_by_id(self, session: AsyncSession, entity_id: int) -> Optional[T]:
"""Select an entity by ID using an existing session."""
query = (
@@ -48,6 +78,9 @@ class Repository[T: Base]:
.filter(self.primary_key == entity_id)
.options(*self.get_load_options())
)
# Add project filter if applicable
query = self._add_project_filter(query)
result = await session.execute(query)
return result.scalars().one_or_none()
@@ -56,6 +89,9 @@ class Repository[T: Base]:
query = (
select(self.Model).where(self.primary_key.in_(ids)).options(*self.get_load_options())
)
# Add project filter if applicable
query = self._add_project_filter(query)
result = await session.execute(query)
return result.scalars().all()
@@ -66,6 +102,9 @@ class Repository[T: Base]:
:return: the added model instance
"""
async with db.scoped_session(self.session_maker) as session:
# Set project_id if applicable and not already set
self._set_project_id_if_needed(model)
session.add(model)
await session.flush()
@@ -89,6 +128,10 @@ class Repository[T: Base]:
:return: the added models instances
"""
async with db.scoped_session(self.session_maker) as session:
# set the project id if not present in models
for model in models:
self._set_project_id_if_needed(model)
session.add_all(models)
await session.flush()
@@ -104,7 +147,10 @@ class Repository[T: Base]:
"""
if not entities:
entities = (self.Model,)
return select(*entities)
query = select(*entities)
# Add project filter if applicable
return self._add_project_filter(query)
async def find_all(self, skip: int = 0, limit: Optional[int] = None) -> Sequence[T]:
"""Fetch records from the database with pagination."""
@@ -112,6 +158,9 @@ class Repository[T: Base]:
async with db.scoped_session(self.session_maker) as session:
query = select(self.Model).offset(skip).options(*self.get_load_options())
# Add project filter if applicable
query = self._add_project_filter(query)
if limit:
query = query.limit(limit)
@@ -143,9 +192,9 @@ class Repository[T: Base]:
entity = result.scalars().one_or_none()
if entity:
logger.debug(f"Found {self.Model.__name__}: {getattr(entity, 'id', None)}")
logger.trace(f"Found {self.Model.__name__}: {getattr(entity, 'id', None)}")
else:
logger.debug(f"No {self.Model.__name__} found")
logger.trace(f"No {self.Model.__name__} found")
return entity
async def create(self, data: dict) -> T:
@@ -154,6 +203,15 @@ class Repository[T: Base]:
async with db.scoped_session(self.session_maker) as session:
# Only include valid columns that are provided in entity_data
model_data = self.get_model_data(data)
# Add project_id if applicable and not already provided
if (
self.has_project_id
and self.project_id is not None
and "project_id" not in model_data
):
model_data["project_id"] = self.project_id
model = self.Model(**model_data)
session.add(model)
await session.flush()
@@ -176,12 +234,20 @@ class Repository[T: Base]:
async with db.scoped_session(self.session_maker) as session:
# Only include valid columns that are provided in entity_data
model_list = [
self.Model(
**self.get_model_data(d),
)
for d in data_list
]
model_list = []
for d in data_list:
model_data = self.get_model_data(d)
# Add project_id if applicable and not already provided
if (
self.has_project_id
and self.project_id is not None
and "project_id" not in model_data
):
model_data["project_id"] = self.project_id # pragma: no cover
model_list.append(self.Model(**model_data))
session.add_all(model_list)
await session.flush()
@@ -237,7 +303,13 @@ class Repository[T: Base]:
"""Delete records matching given IDs."""
logger.debug(f"Deleting {self.Model.__name__} by ids: {ids}")
async with db.scoped_session(self.session_maker) as session:
query = delete(self.Model).where(self.primary_key.in_(ids))
conditions = [self.primary_key.in_(ids)]
# Add project_id filter if applicable
if self.has_project_id and self.project_id is not None: # pragma: no cover
conditions.append(getattr(self.Model, "project_id") == self.project_id)
query = delete(self.Model).where(and_(*conditions))
result = await session.execute(query)
logger.debug(f"Deleted {result.rowcount} records")
return result.rowcount
@@ -247,6 +319,11 @@ class Repository[T: Base]:
logger.debug(f"Deleting {self.Model.__name__} by fields: {filters}")
async with db.scoped_session(self.session_maker) as session:
conditions = [getattr(self.Model, field) == value for field, value in filters.items()]
# Add project_id filter if applicable
if self.has_project_id and self.project_id is not None:
conditions.append(getattr(self.Model, "project_id") == self.project_id)
query = delete(self.Model).where(and_(*conditions))
result = await session.execute(query)
deleted = result.rowcount > 0
@@ -258,19 +335,34 @@ class Repository[T: Base]:
async with db.scoped_session(self.session_maker) as session:
if query is None:
query = select(func.count()).select_from(self.Model)
# Add project filter if applicable
if (
isinstance(query, Select)
and self.has_project_id
and self.project_id is not None
):
query = query.where(
getattr(self.Model, "project_id") == self.project_id
) # pragma: no cover
result = await session.execute(query)
scalar = result.scalar()
count = scalar if scalar is not None else 0
logger.debug(f"Counted {count} {self.Model.__name__} records")
return count
async def execute_query(self, query: Executable, use_query_options: bool = True) -> Result[Any]:
async def execute_query(
self,
query: Executable,
params: Optional[Dict[str, Any]] = None,
use_query_options: bool = True,
) -> Result[Any]:
"""Execute a query asynchronously."""
query = query.options(*self.get_load_options()) if use_query_options else query
logger.debug(f"Executing query: {query}")
logger.trace(f"Executing query: {query}, params: {params}")
async with db.scoped_session(self.session_maker) as session:
result = await session.execute(query)
result = await session.execute(query, params)
return result
def get_load_options(self) -> List[LoaderOption]:
src/basic_memory/repository/search_repository.py
+66 -14
View File
@@ -19,6 +19,7 @@ from basic_memory.schemas.search import SearchItemType
class SearchIndexRow:
"""Search result with score and metadata."""
project_id: int
id: int
type: str
file_path: str
@@ -47,6 +48,27 @@ class SearchIndexRow:
def content(self):
return self.content_snippet
@property
def directory(self) -> str:
"""Extract directory part from file_path.
For a file at "projects/notes/ideas.md", returns "/projects/notes"
For a file at root level "README.md", returns "/"
"""
if not self.type == SearchItemType.ENTITY.value and not self.file_path:
return ""
# Split the path by slashes
parts = self.file_path.split("/")
# If there's only one part (e.g., "README.md"), it's at the root
if len(parts) <= 1:
return "/"
# Join all parts except the last one (filename)
directory_path = "/".join(parts[:-1])
return f"/{directory_path}"
def to_insert(self):
return {
"id": self.id,
@@ -64,14 +86,28 @@ class SearchIndexRow:
"category": self.category,
"created_at": self.created_at if self.created_at else None,
"updated_at": self.updated_at if self.updated_at else None,
"project_id": self.project_id,
}
class SearchRepository:
"""Repository for search index operations."""
def __init__(self, session_maker: async_sessionmaker[AsyncSession]):
def __init__(self, session_maker: async_sessionmaker[AsyncSession], project_id: int):
"""Initialize with session maker and project_id filter.
Args:
session_maker: SQLAlchemy session maker
project_id: Project ID to filter all operations by
Raises:
ValueError: If project_id is None or invalid
"""
if project_id is None or project_id <= 0: # pragma: no cover
raise ValueError("A valid project_id is required for SearchRepository")
self.session_maker = session_maker
self.project_id = project_id
async def init_search_index(self):
"""Create or recreate the search index."""
@@ -125,7 +161,7 @@ class SearchRepository:
title: Optional[str] = None,
types: Optional[List[str]] = None,
after_date: Optional[datetime] = None,
entity_types: Optional[List[SearchItemType]] = None,
search_item_types: Optional[List[SearchItemType]] = None,
limit: int = 10,
offset: int = 0,
) -> List[SearchIndexRow]:
@@ -175,8 +211,8 @@ class SearchRepository:
conditions.append("permalink MATCH :permalink")
# Handle entity type filter
if entity_types:
type_list = ", ".join(f"'{t.value}'" for t in entity_types)
if search_item_types:
type_list = ", ".join(f"'{t.value}'" for t in search_item_types)
conditions.append(f"type IN ({type_list})")
# Handle type filter
@@ -192,6 +228,10 @@ class SearchRepository:
# order by most recent first
order_by_clause = ", updated_at DESC"
# Always filter by project_id
params["project_id"] = self.project_id
conditions.append("project_id = :project_id")
# set limit on search query
params["limit"] = limit
params["offset"] = offset
@@ -201,6 +241,7 @@ class SearchRepository:
sql = f"""
SELECT
project_id,
id,
title,
permalink,
@@ -230,6 +271,7 @@ class SearchRepository:
results = [
SearchIndexRow(
project_id=self.project_id,
id=row.id,
title=row.title,
permalink=row.permalink,
@@ -249,10 +291,10 @@ class SearchRepository:
for row in rows
]
logger.debug(f"Found {len(results)} search results")
logger.trace(f"Found {len(results)} search results")
for r in results:
logger.debug(
f"Search result: type:{r.type} title: {r.title} permalink: {r.permalink} score: {r.score}"
logger.trace(
f"Search result: project_id: {r.project_id} type:{r.type} title: {r.title} permalink: {r.permalink} score: {r.score}"
)
return results
@@ -269,6 +311,10 @@ class SearchRepository:
{"permalink": search_index_row.permalink},
)
# Prepare data for insert with project_id
insert_data = search_index_row.to_insert()
insert_data["project_id"] = self.project_id
# Insert new record
await session.execute(
text("""
@@ -276,15 +322,17 @@ class SearchRepository:
id, title, content_stems, content_snippet, permalink, file_path, type, metadata,
from_id, to_id, relation_type,
entity_id, category,
created_at, updated_at
created_at, updated_at,
project_id
) VALUES (
:id, :title, :content_stems, :content_snippet, :permalink, :file_path, :type, :metadata,
:from_id, :to_id, :relation_type,
:entity_id, :category,
:created_at, :updated_at
:created_at, :updated_at,
:project_id
)
"""),
search_index_row.to_insert(),
insert_data,
)
logger.debug(f"indexed row {search_index_row}")
await session.commit()
@@ -293,8 +341,10 @@ class SearchRepository:
"""Delete an item from the search index by entity_id."""
async with db.scoped_session(self.session_maker) as session:
await session.execute(
text("DELETE FROM search_index WHERE entity_id = :entity_id"),
{"entity_id": entity_id},
text(
"DELETE FROM search_index WHERE entity_id = :entity_id AND project_id = :project_id"
),
{"entity_id": entity_id, "project_id": self.project_id},
)
await session.commit()
@@ -302,8 +352,10 @@ class SearchRepository:
"""Delete an item from the search index."""
async with db.scoped_session(self.session_maker) as session:
await session.execute(
text("DELETE FROM search_index WHERE permalink = :permalink"),
{"permalink": permalink},
text(
"DELETE FROM search_index WHERE permalink = :permalink AND project_id = :project_id"
),
{"permalink": permalink, "project_id": self.project_id},
)
await session.commit()
src/basic_memory/schemas/__init__.py
+6
View File
@@ -44,6 +44,10 @@ from basic_memory.schemas.project_info import (
ProjectInfoResponse,
)
from basic_memory.schemas.directory import (
DirectoryNode,
)
# For convenient imports, export all models
__all__ = [
# Base
@@ -71,4 +75,6 @@ __all__ = [
"ActivityMetrics",
"SystemStatus",
"ProjectInfoResponse",
# Directory
"DirectoryNode",
]
src/basic_memory/schemas/directory.py
+30
View File
@@ -0,0 +1,30 @@
"""Schemas for directory tree operations."""
from datetime import datetime
from typing import List, Optional, Literal
from pydantic import BaseModel
class DirectoryNode(BaseModel):
"""Directory node in file system."""
name: str
file_path: Optional[str] = None # Original path without leading slash (matches DB)
directory_path: str # Path with leading slash for directory navigation
type: Literal["directory", "file"]
children: List["DirectoryNode"] = [] # Default to empty list
title: Optional[str] = None
permalink: Optional[str] = None
entity_id: Optional[int] = None
entity_type: Optional[str] = None
content_type: Optional[str] = None
updated_at: Optional[datetime] = None
@property
def has_children(self) -> bool:
return bool(self.children)
# Support for recursive model
DirectoryNode.model_rebuild()
src/basic_memory/schemas/importer.py
+34
View File
@@ -0,0 +1,34 @@
"""Schemas for import services."""
from typing import Dict, Optional
from pydantic import BaseModel
class ImportResult(BaseModel):
"""Common import result schema."""
import_count: Dict[str, int]
success: bool
error_message: Optional[str] = None
class ChatImportResult(ImportResult):
"""Result schema for chat imports."""
conversations: int = 0
messages: int = 0
class ProjectImportResult(ImportResult):
"""Result schema for project imports."""
documents: int = 0
prompts: int = 0
class EntityImportResult(ImportResult):
"""Result schema for entity imports."""
entities: int = 0
relations: int = 0
src/basic_memory/schemas/memory.py
+27 -13
View File
@@ -100,27 +100,41 @@ class MemoryMetadata(BaseModel):
uri: Optional[str] = None
types: Optional[List[SearchItemType]] = None
depth: int
timeframe: str
timeframe: Optional[str] = None
generated_at: datetime
total_results: int
total_relations: int
primary_count: Optional[int] = None # Changed field name
related_count: Optional[int] = None # Changed field name
total_results: Optional[int] = None # For backward compatibility
total_relations: Optional[int] = None
total_observations: Optional[int] = None
class ContextResult(BaseModel):
"""Context result containing a primary item with its observations and related items."""
primary_result: EntitySummary | RelationSummary | ObservationSummary = Field(
description="Primary item"
)
observations: Sequence[ObservationSummary] = Field(
description="Observations belonging to this entity", default_factory=list
)
related_results: Sequence[EntitySummary | RelationSummary | ObservationSummary] = Field(
description="Related items", default_factory=list
)
class GraphContext(BaseModel):
"""Complete context response."""
# Direct matches
primary_results: Sequence[EntitySummary | RelationSummary | ObservationSummary] = Field(
description="results directly matching URI"
)
# Related entities
related_results: Sequence[EntitySummary | RelationSummary | ObservationSummary] = Field(
description="related results"
# hierarchical results
results: Sequence[ContextResult] = Field(
description="Hierarchical results with related items nested", default_factory=list
)
# Context metadata
metadata: MemoryMetadata
page: int = 1
page_size: int = 1
page: Optional[int] = None
page_size: Optional[int] = None
src/basic_memory/schemas/project_info.py
+114 -2
View File
@@ -1,5 +1,6 @@
"""Schema for project info response."""
import os
from datetime import datetime
from typing import Dict, List, Optional, Any
@@ -75,14 +76,24 @@ class SystemStatus(BaseModel):
timestamp: datetime = Field(description="Timestamp when the information was collected")
class ProjectDetail(BaseModel):
"""Detailed information about a project."""
path: str = Field(description="Path to the project directory")
active: bool = Field(description="Whether the project is active")
id: Optional[int] = Field(description="Database ID of the project if available")
is_default: bool = Field(description="Whether this is the default project")
permalink: str = Field(description="URL-friendly identifier for the project")
class ProjectInfoResponse(BaseModel):
"""Response for the project_info tool."""
# Project configuration
project_name: str = Field(description="Name of the current project")
project_path: str = Field(description="Path to the current project files")
available_projects: Dict[str, str] = Field(
description="Map of configured project names to paths"
available_projects: Dict[str, Dict[str, Any]] = Field(
description="Map of configured project names to detailed project information"
)
default_project: str = Field(description="Name of the default project")
@@ -94,3 +105,104 @@ class ProjectInfoResponse(BaseModel):
# System status
system: SystemStatus = Field(description="System and service status information")
class ProjectSwitchRequest(BaseModel):
"""Request model for switching projects."""
name: str = Field(..., description="Name of the project to switch to")
path: str = Field(..., description="Path to the project directory")
set_default: bool = Field(..., description="Set the project as the default")
class WatchEvent(BaseModel):
timestamp: datetime
path: str
action: str # new, delete, etc
status: str # success, error
checksum: Optional[str]
error: Optional[str] = None
class WatchServiceState(BaseModel):
# Service status
running: bool = False
start_time: datetime = datetime.now() # Use directly with Pydantic model
pid: int = os.getpid() # Use directly with Pydantic model
# Stats
error_count: int = 0
last_error: Optional[datetime] = None
last_scan: Optional[datetime] = None
# File counts
synced_files: int = 0
# Recent activity
recent_events: List[WatchEvent] = [] # Use directly with Pydantic model
def add_event(
self,
path: str,
action: str,
status: str,
checksum: Optional[str] = None,
error: Optional[str] = None,
) -> WatchEvent: # pragma: no cover
event = WatchEvent(
timestamp=datetime.now(),
path=path,
action=action,
status=status,
checksum=checksum,
error=error,
)
self.recent_events.insert(0, event)
self.recent_events = self.recent_events[:100] # Keep last 100
return event
def record_error(self, error: str): # pragma: no cover
self.error_count += 1
self.add_event(path="", action="sync", status="error", error=error)
self.last_error = datetime.now()
class ProjectWatchStatus(BaseModel):
"""Project with its watch status."""
name: str = Field(..., description="Name of the project")
path: str = Field(..., description="Path to the project")
watch_status: Optional[WatchServiceState] = Field(
None, description="Watch status information for the project"
)
class ProjectStatusResponse(BaseModel):
"""Response model for switching projects."""
message: str = Field(..., description="Status message about the project switch")
status: str = Field(..., description="Status of the switch (success or error)")
default: bool = Field(..., description="True if the project was set as the default")
old_project: Optional[ProjectWatchStatus] = Field(
None, description="Information about the project being switched from"
)
new_project: Optional[ProjectWatchStatus] = Field(
None, description="Information about the project being switched to"
)
class ProjectItem(BaseModel):
"""Simple representation of a project."""
name: str
path: str
is_default: bool
is_current: bool
class ProjectList(BaseModel):
"""Response model for listing projects."""
projects: List[ProjectItem]
default_project: str
current_project: str
src/basic_memory/schemas/prompt.py
+90
View File
@@ -0,0 +1,90 @@
"""Request and response schemas for prompt-related operations."""
from typing import Optional, List, Any, Dict
from pydantic import BaseModel, Field
from basic_memory.schemas.base import TimeFrame
from basic_memory.schemas.memory import EntitySummary, ObservationSummary, RelationSummary
class PromptContextItem(BaseModel):
"""Container for primary and related results to render in a prompt."""
primary_results: List[EntitySummary]
related_results: List[EntitySummary | ObservationSummary | RelationSummary]
class ContinueConversationRequest(BaseModel):
"""Request for generating a continue conversation prompt.
Used to provide context for continuing a conversation on a specific topic
or with recent activity from a given timeframe.
"""
topic: Optional[str] = Field(None, description="Topic or keyword to search for")
timeframe: Optional[TimeFrame] = Field(
None, description="How far back to look for activity (e.g. '1d', '1 week')"
)
# Limit depth to max 2 for performance reasons - higher values cause significant slowdown
search_items_limit: int = Field(
5,
description="Maximum number of search results to include in context (max 10)",
ge=1,
le=10,
)
depth: int = Field(
1,
description="How many relationship 'hops' to follow when building context (max 5)",
ge=1,
le=5,
)
# Limit related items to prevent overloading the context
related_items_limit: int = Field(
5, description="Maximum number of related items to include in context (max 10)", ge=1, le=10
)
class SearchPromptRequest(BaseModel):
"""Request for generating a search results prompt.
Used to format search results into a prompt with context and suggestions.
"""
query: str = Field(..., description="The search query text")
timeframe: Optional[TimeFrame] = Field(
None, description="Optional timeframe to limit results (e.g. '1d', '1 week')"
)
class PromptMetadata(BaseModel):
"""Metadata about a prompt response.
Contains statistical information about the prompt generation process
and results, useful for debugging and UI display.
"""
query: Optional[str] = Field(None, description="The original query or topic")
timeframe: Optional[str] = Field(None, description="The timeframe used for filtering")
search_count: int = Field(0, description="Number of search results found")
context_count: int = Field(0, description="Number of context items retrieved")
observation_count: int = Field(0, description="Total number of observations included")
relation_count: int = Field(0, description="Total number of relations included")
total_items: int = Field(0, description="Total number of all items included in the prompt")
search_limit: int = Field(0, description="Maximum search results requested")
context_depth: int = Field(0, description="Context depth used")
related_limit: int = Field(0, description="Maximum related items requested")
generated_at: str = Field(..., description="ISO timestamp when this prompt was generated")
class PromptResponse(BaseModel):
"""Response containing the rendered prompt.
Includes both the rendered prompt text and the context that was used
to render it, for potential client-side use.
"""
prompt: str = Field(..., description="The rendered prompt text")
context: Dict[str, Any] = Field(..., description="The context used to render the prompt")
metadata: PromptMetadata = Field(
..., description="Metadata about the prompt generation process"
)
src/basic_memory/schemas/search.py
+1 -1
View File
@@ -90,7 +90,7 @@ class SearchResult(BaseModel):
title: str
type: SearchItemType
score: float
entity: Optional[Permalink]
entity: Optional[Permalink] = None
permalink: Optional[str]
content: Optional[str] = None
file_path: str
src/basic_memory/services/__init__.py
+2 -1
View File
@@ -3,5 +3,6 @@
from .service import BaseService
from .file_service import FileService
from .entity_service import EntityService
from .project_service import ProjectService
__all__ = ["BaseService", "FileService", "EntityService"]
__all__ = ["BaseService", "FileService", "EntityService", "ProjectService"]
src/basic_memory/services/context_service.py
+207 -94
View File
@@ -1,6 +1,6 @@
"""Service for building rich context from the knowledge graph."""
from dataclasses import dataclass
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import List, Optional, Tuple
@@ -8,9 +8,11 @@ from loguru import logger
from sqlalchemy import text
from basic_memory.repository.entity_repository import EntityRepository
from basic_memory.repository.search_repository import SearchRepository
from basic_memory.repository.observation_repository import ObservationRepository
from basic_memory.repository.search_repository import SearchRepository, SearchIndexRow
from basic_memory.schemas.memory import MemoryUrl, memory_url_path
from basic_memory.schemas.search import SearchItemType
from basic_memory.utils import generate_permalink
@dataclass
@@ -31,6 +33,38 @@ class ContextResultRow:
entity_id: Optional[int] = None
@dataclass
class ContextResultItem:
"""A hierarchical result containing a primary item with its observations and related items."""
primary_result: ContextResultRow | SearchIndexRow
observations: List[ContextResultRow] = field(default_factory=list)
related_results: List[ContextResultRow] = field(default_factory=list)
@dataclass
class ContextMetadata:
"""Metadata about a context result."""
uri: Optional[str] = None
types: Optional[List[SearchItemType]] = None
depth: int = 1
timeframe: Optional[str] = None
generated_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
primary_count: int = 0
related_count: int = 0
total_observations: int = 0
total_relations: int = 0
@dataclass
class ContextResult:
"""Complete context result with metadata."""
results: List[ContextResultItem] = field(default_factory=list)
metadata: ContextMetadata = field(default_factory=ContextMetadata)
class ContextService:
"""Service for building rich context from memory:// URIs.
@@ -44,9 +78,11 @@ class ContextService:
self,
search_repository: SearchRepository,
entity_repository: EntityRepository,
observation_repository: ObservationRepository,
):
self.search_repository = search_repository
self.entity_repository = entity_repository
self.observation_repository = observation_repository
async def build_context(
self,
@@ -57,7 +93,8 @@ class ContextService:
limit=10,
offset=0,
max_related: int = 10,
):
include_observations: bool = True,
) -> ContextResult:
"""Build rich context from a memory:// URI."""
logger.debug(
f"Building context for URI: '{memory_url}' depth: '{depth}' since: '{since}' limit: '{limit}' offset: '{offset}' max_related: '{max_related}'"
@@ -81,7 +118,7 @@ class ContextService:
else:
logger.debug(f"Build context for '{types}'")
primary = await self.search_repository.search(
entity_types=types, after_date=since, limit=limit, offset=offset
search_item_types=types, after_date=since, limit=limit, offset=offset
)
# Get type_id pairs for traversal
@@ -94,24 +131,78 @@ class ContextService:
type_id_pairs, max_depth=depth, since=since, max_results=max_related
)
logger.debug(f"Found {len(related)} related results")
for r in related:
logger.debug(f"Found related {r.type}: {r.permalink}")
# Build response
return {
"primary_results": primary,
"related_results": related,
"metadata": {
"uri": memory_url_path(memory_url) if memory_url else None,
"types": types if types else None,
"depth": depth,
"timeframe": since.isoformat() if since else None,
"generated_at": datetime.now(timezone.utc).isoformat(),
"matched_results": len(primary),
"total_results": len(primary) + len(related),
"total_relations": sum(1 for r in related if r.type == SearchItemType.RELATION),
},
}
# Collect entity IDs from primary and related results
entity_ids = []
for result in primary:
if result.type == SearchItemType.ENTITY.value:
entity_ids.append(result.id)
for result in related:
if result.type == SearchItemType.ENTITY.value:
entity_ids.append(result.id)
# Fetch observations for all entities if requested
observations_by_entity = {}
if include_observations and entity_ids:
# Use our observation repository to get observations for all entities at once
observations_by_entity = await self.observation_repository.find_by_entities(entity_ids)
logger.debug(f"Found observations for {len(observations_by_entity)} entities")
# Create metadata dataclass
metadata = ContextMetadata(
uri=memory_url_path(memory_url) if memory_url else None,
types=types,
depth=depth,
timeframe=since.isoformat() if since else None,
primary_count=len(primary),
related_count=len(related),
total_observations=sum(len(obs) for obs in observations_by_entity.values()),
total_relations=sum(1 for r in related if r.type == SearchItemType.RELATION),
)
# Build context results list directly with ContextResultItem objects
context_results = []
# For each primary result
for primary_item in primary:
# Find all related items with this primary item as root
related_to_primary = [r for r in related if r.root_id == primary_item.id]
# Get observations for this item if it's an entity
item_observations = []
if primary_item.type == SearchItemType.ENTITY.value and include_observations:
# Convert Observation models to ContextResultRows
for obs in observations_by_entity.get(primary_item.id, []):
item_observations.append(
ContextResultRow(
type="observation",
id=obs.id,
title=f"{obs.category}: {obs.content[:50]}...",
permalink=generate_permalink(
f"{primary_item.permalink}/observations/{obs.category}/{obs.content}"
),
file_path=primary_item.file_path,
content=obs.content,
category=obs.category,
entity_id=primary_item.id,
depth=0,
root_id=primary_item.id,
created_at=primary_item.created_at, # created_at time from entity
)
)
# Create ContextResultItem directly
context_item = ContextResultItem(
primary_result=primary_item,
observations=item_observations,
related_results=related_to_primary,
)
context_results.append(context_item)
# Return the structured ContextResult
return ContextResult(results=context_results, metadata=metadata)
async def find_related(
self,
@@ -124,7 +215,6 @@ class ContextService:
Uses recursive CTE to find:
- Connected entities
- Their observations
- Relations that connect them
Note on depth:
@@ -138,105 +228,130 @@ class ContextService:
if not type_id_pairs:
return []
logger.debug(f"Finding connected items for {type_id_pairs} with depth {max_depth}")
# Extract entity IDs from type_id_pairs for the optimized query
entity_ids = [i for t, i in type_id_pairs if t == "entity"]
# Build the VALUES clause directly since SQLite doesn't handle parameterized IN well
if not entity_ids:
logger.debug("No entity IDs found in type_id_pairs")
return []
logger.debug(
f"Finding connected items for {len(entity_ids)} entities with depth {max_depth}"
)
# Build the VALUES clause for entity IDs
entity_id_values = ", ".join([str(i) for i in entity_ids])
# For compatibility with the old query, we still need this for filtering
values = ", ".join([f"('{t}', {i})" for t, i in type_id_pairs])
# Parameters for bindings
params = {"max_depth": max_depth, "max_results": max_results}
# Build date and timeframe filters conditionally based on since parameter
if since:
params["since_date"] = since.isoformat() # pyright: ignore
date_filter = "AND e.created_at >= :since_date"
relation_date_filter = "AND e_from.created_at >= :since_date"
timeframe_condition = "AND eg.relation_date >= :since_date"
else:
date_filter = ""
relation_date_filter = ""
timeframe_condition = ""
# Build date filter
date_filter = "AND base.created_at >= :since_date" if since else ""
r1_date_filter = "AND r.created_at >= :since_date" if since else ""
related_date_filter = "AND e.created_at >= :since_date" if since else ""
# Use a CTE that operates directly on entity and relation tables
# This avoids the overhead of the search_index virtual table
query = text(f"""
WITH RECURSIVE context_graph AS (
-- Base case: seed items
WITH RECURSIVE entity_graph AS (
-- Base case: seed entities
SELECT
id,
type,
title,
permalink,
file_path,
from_id,
to_id,
relation_type,
content_snippet as content,
category,
entity_id,
e.id,
'entity' as type,
e.title,
e.permalink,
e.file_path,
NULL as from_id,
NULL as to_id,
NULL as relation_type,
NULL as content,
NULL as category,
NULL as entity_id,
0 as depth,
id as root_id,
created_at,
created_at as relation_date,
e.id as root_id,
e.created_at,
e.created_at as relation_date,
0 as is_incoming
FROM search_index base
WHERE (base.type, base.id) IN ({values})
FROM entity e
WHERE e.id IN ({entity_id_values})
{date_filter}
UNION ALL -- Allow same paths at different depths
UNION ALL
-- Get relations from current entities
SELECT DISTINCT
-- Get relations from current entities
SELECT
r.id,
r.type,
r.title,
r.permalink,
r.file_path,
'relation' as type,
r.relation_type || ': ' || r.to_name as title,
-- Relation model doesn't have permalink column - we'll generate it at runtime
'' as permalink,
e_from.file_path,
r.from_id,
r.to_id,
r.relation_type,
r.content_snippet as content,
r.category,
r.entity_id,
cg.depth + 1,
cg.root_id,
r.created_at,
r.created_at as relation_date,
CASE WHEN r.from_id = cg.id THEN 0 ELSE 1 END as is_incoming
FROM context_graph cg
JOIN search_index r ON (
cg.type = 'entity' AND
r.type = 'relation' AND
(r.from_id = cg.id OR r.to_id = cg.id)
{r1_date_filter}
NULL as content,
NULL as category,
NULL as entity_id,
eg.depth + 1,
eg.root_id,
e_from.created_at, -- Use the from_entity's created_at since relation has no timestamp
e_from.created_at as relation_date,
CASE WHEN r.from_id = eg.id THEN 0 ELSE 1 END as is_incoming
FROM entity_graph eg
JOIN relation r ON (
eg.type = 'entity' AND
(r.from_id = eg.id OR r.to_id = eg.id)
)
WHERE cg.depth < :max_depth
JOIN entity e_from ON (
r.from_id = e_from.id
{relation_date_filter}
)
WHERE eg.depth < :max_depth
UNION ALL
-- Get entities connected by relations
SELECT DISTINCT
SELECT
e.id,
e.type,
'entity' as type,
e.title,
e.permalink,
CASE
WHEN e.permalink IS NULL THEN ''
ELSE e.permalink
END as permalink,
e.file_path,
e.from_id,
e.to_id,
e.relation_type,
e.content_snippet as content,
e.category,
e.entity_id,
cg.depth + 1, -- Increment depth for entities
cg.root_id,
NULL as from_id,
NULL as to_id,
NULL as relation_type,
NULL as content,
NULL as category,
NULL as entity_id,
eg.depth + 1,
eg.root_id,
e.created_at,
cg.relation_date,
cg.is_incoming
FROM context_graph cg
JOIN search_index e ON (
cg.type = 'relation' AND
e.type = 'entity' AND
eg.relation_date,
eg.is_incoming
FROM entity_graph eg
JOIN entity e ON (
eg.type = 'relation' AND
e.id = CASE
WHEN cg.is_incoming = 0 THEN cg.to_id -- Fixed entity lookup
ELSE cg.from_id
WHEN eg.is_incoming = 0 THEN eg.to_id
ELSE eg.from_id
END
{related_date_filter}
{date_filter}
)
WHERE cg.depth < :max_depth
WHERE eg.depth < :max_depth
-- Only include entities connected by relations within timeframe if specified
{timeframe_condition}
)
SELECT DISTINCT
type,
@@ -253,12 +368,10 @@ class ContextService:
MIN(depth) as depth,
root_id,
created_at
FROM context_graph
FROM entity_graph
WHERE (type, id) NOT IN ({values})
GROUP BY
type, id, title, permalink, from_id, to_id,
relation_type, category, entity_id,
root_id, created_at
type, id
ORDER BY depth, type, id
LIMIT :max_results
""")
src/basic_memory/services/directory_service.py
+89
View File
@@ -0,0 +1,89 @@
"""Directory service for managing file directories and tree structure."""
import logging
import os
from typing import Dict
from basic_memory.repository import EntityRepository
from basic_memory.schemas.directory import DirectoryNode
logger = logging.getLogger(__name__)
class DirectoryService:
"""Service for working with directory trees."""
def __init__(self, entity_repository: EntityRepository):
"""Initialize the directory service.
Args:
entity_repository: Directory repository for data access.
"""
self.entity_repository = entity_repository
async def get_directory_tree(self) -> DirectoryNode:
"""Build a hierarchical directory tree from indexed files."""
# Get all files from DB (flat list)
entity_rows = await self.entity_repository.find_all()
# Create a root directory node
root_node = DirectoryNode(name="Root", directory_path="/", type="directory")
# Map to store directory nodes by path for easy lookup
dir_map: Dict[str, DirectoryNode] = {root_node.directory_path: root_node}
# First pass: create all directory nodes
for file in entity_rows:
# Process directory path components
parts = [p for p in file.file_path.split("/") if p]
# Create directory structure
current_path = "/"
for i, part in enumerate(parts[:-1]): # Skip the filename
parent_path = current_path
# Build the directory path
current_path = (
f"{current_path}{part}" if current_path == "/" else f"{current_path}/{part}"
)
# Create directory node if it doesn't exist
if current_path not in dir_map:
dir_node = DirectoryNode(
name=part, directory_path=current_path, type="directory"
)
dir_map[current_path] = dir_node
# Add to parent's children
if parent_path in dir_map:
dir_map[parent_path].children.append(dir_node)
# Second pass: add file nodes to their parent directories
for file in entity_rows:
file_name = os.path.basename(file.file_path)
parent_dir = os.path.dirname(file.file_path)
directory_path = "/" if parent_dir == "" else f"/{parent_dir}"
# Create file node
file_node = DirectoryNode(
name=file_name,
file_path=file.file_path, # Original path from DB (no leading slash)
directory_path=f"/{file.file_path}", # Path with leading slash
type="file",
title=file.title,
permalink=file.permalink,
entity_id=file.id,
entity_type=file.entity_type,
content_type=file.content_type,
updated_at=file.updated_at,
)
# Add to parent directory's children
if directory_path in dir_map:
dir_map[directory_path].children.append(file_node)
else:
# If parent directory doesn't exist (should be rare), add to root
dir_map["/"].children.append(file_node) # pragma: no cover
# Return the root node with its children
return root_node

Some files were not shown because too many files have changed in this diff Show More