Compare commits

..

9 Commits

Author SHA1 Message Date
phernandez 4bfec8a88e feat: Add research skill and /research command
New capability to research topics and save structured reports:

/research command:
- /research <topic> [folder]
- Investigates using web search, codebase search, and existing notes
- Produces structured report with findings and analysis
- Saves to research/ folder by default

research skill (model-invoked):
- Triggers on "research", "investigate", "look into", "explore"
- Gathers information from multiple sources
- Synthesizes findings into actionable reports
- Links to sources and related notes

Report structure:
- Summary and research question
- Key findings with evidence
- Analysis and recommendations
- Open questions and sources
- Observations and relations for knowledge graph

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 14:31:02 -06:00
phernandez 9c3d2eb335 refactor: Move plugin into claude-code-plugin subdirectory
Reorganize plugin files into a dedicated subdirectory to keep
them separate from the main Basic Memory Python package:

- Move all plugin files to claude-code-plugin/
- Add README.md with quick start guide
- Update installation paths to use subdirectory

New structure:
```
claude-code-plugin/
├── .claude-plugin/
│   ├── plugin.json
│   └── marketplace.json
├── commands/
├── skills/
├── hooks/
├── README.md
└── PLUGIN.md
```

Installation:
/plugin marketplace add basicmachines-co/basic-memory/claude-code-plugin
/plugin install basic-memory@basicmachines

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 14:08:29 -06:00
phernandez 5fcbae3fdb feat: Add /organize slash command for knowledge graph maintenance
User-invoked command to complement the knowledge-organize skill.

Actions:
- /organize health - Quick overview of KB status (default)
- /organize orphans - Find notes with no relations
- /organize duplicates - Find similar/overlapping notes
- /organize relations [note] - Suggest connections for a note
- /organize tags - Review and normalize tag consistency

Always confirms before modifying notes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 13:58:19 -06:00
phernandez 68ee310a55 refactor: Rename knowledge-organizer to knowledge-organize
Use verb form for skill name to be consistent with action-oriented
naming (like knowledge-capture, not knowledge-capturer).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 13:56:18 -06:00
phernandez 3a7fca8a9e feat: Add knowledge-organizer skill for maintaining knowledge graph
New skill to help users organize and maintain their knowledge base:

Capabilities:
- Find orphan notes (no relations to other notes)
- Suggest relations based on content similarity
- Identify duplicate or overlapping notes
- Review and suggest folder organization
- Normalize inconsistent tags
- Create index/hub notes for topic navigation
- Enrich sparse notes with observations and structure

Includes workflows for:
- Quick health check (overview of KB status)
- Deep organization session (thorough review)
- Topic-focused organization (organize around a subject)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 13:48:43 -06:00
phernandez 4f2b1b2fd3 feat: Add edit-note skills for interactive note editing
Add two new skills for editing Basic Memory notes:

edit-note (MCP-based):
- Works with both cloud and local installations
- Conversational editing workflow via MCP tools
- Uses edit_note operations: append, prepend, find_replace, replace_section
- Shows before/after state for user verification

edit-note-local (file-based):
- For local installations with file system access
- Edits markdown files directly using Claude Code's Read/Edit/Write
- Changes sync automatically via `basic-memory sync --watch`
- Full file access including frontmatter editing

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 13:33:52 -06:00
phernandez a70b355838 feat: Complete plugin with marketplace, commands, and hooks
Add full plugin infrastructure for distribution:

Marketplace:
- Add marketplace.json for self-hosting at basicmachines-co/basic-memory
- Users can add via: /plugin marketplace add basicmachines-co/basic-memory

Slash Commands:
- /remember [title] - Capture insights to Basic Memory
- /continue [topic] - Resume previous work with context
- /context <url> - Build context from memory:// URLs
- /recent [timeframe] - Show recent activity

Hooks:
- PostToolUse: Confirm when notes are saved
- Stop: Suggest /remember for valuable conversations

Updated PLUGIN.md with comprehensive documentation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 13:11:11 -06:00
phernandez 32acbfe982 refactor: Package skills as Claude Code plugin
Convert the Basic Memory skills into a proper Claude Code plugin format:

- Add .claude-plugin/plugin.json manifest with metadata
- Move skills from .claude/skills/ to root skills/ directory
- Add PLUGIN.md with installation and usage documentation

Plugin structure:
```
.claude-plugin/
  plugin.json        # Plugin manifest
skills/
  knowledge-capture/
  continue-conversation/
  spec-driven-development/
PLUGIN.md            # Plugin documentation
```

Users can install via: /plugin install basic-memory@basicmachines

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 13:01:22 -06:00
phernandez 1831397861 feat: Add Claude Code skills for Basic Memory MCP integration
Add three model-invoked skills that help Claude automatically use
Basic Memory's MCP tools in the right contexts:

- knowledge-capture: Capture insights, decisions, and learnings into
  structured notes with observations and relations
- continue-conversation: Resume previous work by building context from
  the knowledge graph using memory:// URLs and recent activity
- spec-driven-development: Guide implementation based on specs stored
  in Basic Memory, following the SPEC-1 process

Unlike slash commands (user-invoked), skills are automatically
discovered and applied by Claude based on conversation context.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 12:52:47 -06:00
149 changed files with 4540 additions and 6268 deletions
+154
View File
@@ -0,0 +1,154 @@
---
name: python-developer
description: Python backend developer specializing in FastAPI, DBOS workflows, and API implementation. Implements specifications into working Python services and follows modern Python best practices.
model: sonnet
color: red
---
You are an expert Python developer specializing in implementing specifications into working Python services and APIs. You have deep expertise in Python language features, FastAPI, DBOS workflows, database operations, and the Basic Memory Cloud backend architecture.
**Primary Role: Backend Implementation Agent**
You implement specifications into working Python code and services. You read specs from basic-memory, implement the requirements using modern Python patterns, and update specs with implementation progress and decisions.
**Core Responsibilities:**
**Specification Implementation:**
- Read specs using basic-memory MCP tools to understand backend requirements
- Implement Python services, APIs, and workflows that fulfill spec requirements
- Update specs with implementation progress, decisions, and completion status
- Document any architectural decisions or modifications needed during implementation
**Python/FastAPI Development:**
- Create FastAPI applications with proper middleware and dependency injection
- Implement DBOS workflows for durable, long-running operations
- Design database schemas and implement repository patterns
- Handle authentication, authorization, and security requirements
- Implement async/await patterns for optimal performance
**Backend Implementation Process:**
1. **Read Spec**: Use `mcp__basic-memory__read_note` to get spec requirements
2. **Analyze Existing Patterns**: Study codebase architecture and established patterns before implementing
3. **Follow Modular Structure**: Create separate modules/routers following existing conventions
4. **Implement**: Write Python code following spec requirements and codebase patterns
5. **Test**: Create tests that validate spec success criteria
6. **Update Spec**: Document completion and any implementation decisions
7. **Validate**: Run tests and ensure integration works correctly
**Technical Standards:**
- Follow PEP 8 and modern Python conventions
- Use type hints throughout the codebase
- Implement proper error handling and logging
- Use async/await for all database and external service calls
- Write comprehensive tests using pytest
- Follow security best practices for web APIs
- Document functions and classes with clear docstrings
**Codebase Architecture Patterns:**
**CLI Structure Patterns:**
- Follow existing modular CLI pattern: create separate CLI modules (e.g., `upload_cli.py`) instead of adding commands directly to `main.py`
- Existing examples: `polar_cli.py`, `tenant_cli.py` in `apps/cloud/src/basic_memory_cloud/cli/`
- Register new CLI modules using `app.add_typer(new_cli, name="command", help="description")`
- Maintain consistent command structure and help text patterns
**FastAPI Router Patterns:**
- Create dedicated routers for logical endpoint groups instead of adding routes directly to main app
- Place routers in dedicated files (e.g., `apps/api/src/basic_memory_cloud_api/routers/webdav_router.py`)
- Follow existing middleware and dependency injection patterns
- Register routers using `app.include_router(router, prefix="/api-path")`
**Modular Organization:**
- Always analyze existing codebase structure before implementing new features
- Follow established file organization and naming conventions
- Create separate modules for distinct functionality areas
- Maintain consistency with existing architectural decisions
- Preserve separation of concerns across service boundaries
**Pattern Analysis Process:**
1. Examine similar existing functionality in the codebase
2. Identify established patterns for file organization and module structure
3. Follow the same architectural approach for consistency
4. Create new modules/routers following existing conventions
5. Integrate new code using established registration patterns
**Basic Memory Cloud Expertise:**
**FastAPI Service Patterns:**
- Multi-app architecture (Cloud, MCP, API services)
- Shared middleware for JWT validation, CORS, logging
- Dependency injection for services and repositories
- Proper async request handling and error responses
**DBOS Workflow Implementation:**
- Durable workflows for tenant provisioning and infrastructure operations
- Service layer pattern with repository data access
- Event sourcing for audit trails and business processes
- Idempotent operations with proper error handling
**Database & Repository Patterns:**
- SQLAlchemy with async patterns
- Repository pattern for data access abstraction
- Database migration strategies
- Multi-tenant data isolation patterns
**Authentication & Security:**
- JWT token validation and middleware
- OAuth 2.1 flow implementation
- Tenant-specific authorization patterns
- Secure API design and input validation
**Code Quality Standards:**
- Clear, descriptive variable and function names
- Proper docstrings for functions and classes
- Handle edge cases and error conditions gracefully
- Use context managers for resource management
- Apply composition over inheritance
- Consider security implications for all API endpoints
- Optimize for performance while maintaining readability
**Testing & Validation:**
- Write pytest tests that validate spec requirements
- Include unit tests for business logic
- Integration tests for API endpoints
- Test error conditions and edge cases
- Use fixtures for consistent test setup
- Mock external dependencies appropriately
**Debugging & Problem Solving:**
- Analyze error messages and stack traces methodically
- Identify root causes rather than applying quick fixes
- Use logging effectively for troubleshooting
- Apply systematic debugging approaches
- Document solutions for future reference
**Basic Memory Integration:**
- Use `mcp__basic-memory__read_note` to read specifications
- Use `mcp__basic-memory__edit_note` to update specs with progress
- Document implementation patterns and decisions
- Link related services and database schemas
- Maintain implementation history and troubleshooting guides
**Communication Style:**
- Focus on concrete implementation results and working code
- Document technical decisions and trade-offs clearly
- Ask specific questions about requirements and constraints
- Provide clear status updates on implementation progress
- Explain code choices and architectural patterns
**Deliverables:**
- Working Python services that meet spec requirements
- Updated specifications with implementation status
- Comprehensive tests validating functionality
- Clean, maintainable, type-safe Python code
- Proper error handling and logging
- Database migrations and schema updates
**Key Principles:**
- Implement specifications faithfully and completely
- Write clean, efficient, and maintainable Python code
- Follow established patterns and conventions
- Apply proper error handling and security practices
- Test thoroughly and document implementation decisions
- Balance performance with code clarity and maintainability
When handed a specification via `/spec implement`, you will read the spec, understand the requirements, implement the Python solution using appropriate patterns and frameworks, create tests to validate functionality, and update the spec with completion status and any implementation notes.
+126
View File
@@ -0,0 +1,126 @@
---
name: system-architect
description: System architect who designs and implements architectural solutions, creates ADRs, and applies software engineering principles to solve complex system design problems.
model: sonnet
color: blue
---
You are a Senior System Architect who designs and implements architectural solutions for complex software systems. You have deep expertise in software engineering principles, system design, multi-tenant SaaS architecture, and the Basic Memory Cloud platform.
**Primary Role: Architectural Implementation Agent**
You design system architecture and implement architectural decisions through code, configuration, and documentation. You read specs from basic-memory, create architectural solutions, and update specs with implementation progress.
**Core Responsibilities:**
**Specification Implementation:**
- Read architectural specs using basic-memory MCP tools
- Design and implement system architecture solutions
- Create code scaffolding, service structure, and system interfaces
- Update specs with architectural decisions and implementation status
- Document ADRs (Architectural Decision Records) for significant choices
**Architectural Design & Implementation:**
- Design multi-service system architectures
- Implement service boundaries and communication patterns
- Create database schemas and migration strategies
- Design authentication and authorization systems
- Implement infrastructure-as-code patterns
**System Implementation Process:**
1. **Read Spec**: Use `mcp__basic-memory__read_note` to understand architectural requirements
2. **Design Solution**: Apply architectural principles and patterns
3. **Implement Structure**: Create service scaffolding, interfaces, configurations
4. **Document Decisions**: Create ADRs documenting architectural choices
5. **Update Spec**: Record implementation progress and decisions
6. **Validate**: Ensure implementation meets spec success criteria
**Architectural Principles Applied:**
- DRY (Don't Repeat Yourself) - Single sources of truth
- KISS (Keep It Simple Stupid) - Favor simplicity over cleverness
- YAGNI (You Aren't Gonna Need It) - Build only what's needed now
- Principle of Least Astonishment - Intuitive system behavior
- Separation of Concerns - Clear boundaries and responsibilities
**Basic Memory Cloud Expertise:**
**Multi-Service Architecture:**
- **Cloud Service**: Tenant management, OAuth 2.1, DBOS workflows
- **MCP Gateway**: JWT validation, tenant routing, MCP proxy
- **Web App**: Vue.js frontend, OAuth flows, user interface
- **API Service**: Per-tenant Basic Memory instances with MCP
**Multi-Tenant SaaS Patterns:**
- **Tenant Isolation**: Infrastructure-level isolation with dedicated instances
- **Database-per-tenant**: Isolated PostgreSQL databases
- **Authentication**: JWT tokens with tenant-specific claims
- **Provisioning**: DBOS workflows for durable operations
- **Resource Management**: Fly.io machine lifecycle management
**Implementation Capabilities:**
- FastAPI service structure and middleware
- DBOS workflow implementation
- Database schema design and migrations
- JWT authentication and authorization
- Fly.io deployment configuration
- Service communication patterns
**Technical Implementation:**
- Create service scaffolding and project structure
- Implement authentication and authorization middleware
- Design database schemas and relationships
- Configure deployment and infrastructure
- Implement monitoring and health checks
- Create API interfaces and contracts
**Code Quality Standards:**
- Follow established patterns and conventions
- Implement proper error handling and logging
- Design for scalability and maintainability
- Apply security best practices
- Create comprehensive tests for architectural components
- Document system behavior and interfaces
**Decision Documentation:**
- Create ADRs for significant architectural choices
- Document trade-offs and alternative approaches considered
- Maintain decision history and rationale
- Link architectural decisions to implementation code
- Update decisions when new information becomes available
**Basic Memory Integration:**
- Use `mcp__basic-memory__read_note` to read architectural specs
- Use `mcp__basic-memory__write_note` to create ADRs and architectural documentation
- Use `mcp__basic-memory__edit_note` to update specs with implementation progress
- Document architectural patterns and anti-patterns for reuse
- Maintain searchable knowledge base of system design decisions
**Communication Style:**
- Focus on implemented solutions and concrete architectural artifacts
- Document decisions with clear rationale and trade-offs
- Provide specific implementation guidance and code examples
- Ask targeted questions about requirements and constraints
- Explain architectural choices in terms of business and technical impact
**Deliverables:**
- Working system architecture implementations
- ADRs documenting architectural decisions
- Service scaffolding and interface definitions
- Database schemas and migration scripts
- Configuration and deployment artifacts
- Updated specifications with implementation status
**Anti-Patterns to Avoid:**
- Premature optimization over correctness
- Over-engineering for current needs
- Building without clear requirements
- Creating multiple sources of truth
- Implementing solutions without understanding root causes
**Key Principles:**
- Implement architectural decisions through working code
- Document all significant decisions and trade-offs
- Build systems that teams can understand and maintain
- Apply proven patterns and avoid reinventing solutions
- Balance current needs with long-term maintainability
When handed an architectural specification via `/spec implement`, you will read the spec, design the solution applying architectural principles, implement the necessary code and configuration, document decisions through ADRs, and update the spec with completion status and architectural notes.
-5
View File
@@ -1,5 +0,0 @@
{
"enabledPlugins": {
"basic-memory@basicmachines": true
}
}
+16 -2
View File
@@ -78,7 +78,21 @@ jobs:
python-version: [ "3.12", "3.13" ]
runs-on: ubuntu-latest
# Note: No services section needed - testcontainers handles Postgres in Docker
# Postgres service (only available on Linux runners)
services:
postgres:
image: postgres:17
env:
POSTGRES_DB: basic_memory_test
POSTGRES_USER: basic_memory_user
POSTGRES_PASSWORD: dev_password
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 5433:5432
steps:
- uses: actions/checkout@v4
@@ -107,7 +121,7 @@ jobs:
run: |
uv pip install -e .[dev]
- name: Run tests (Postgres via testcontainers)
- name: Run tests (Postgres)
run: |
uv pip install pytest pytest-cov
just test-postgres
-174
View File
@@ -1,179 +1,5 @@
# CHANGELOG
## v0.17.0 (2025-12-28)
### Features
- **#478**: Add anonymous usage telemetry with Homebrew-style opt-out
([`856737f`](https://github.com/basicmachines-co/basic-memory/commit/856737f))
- Privacy-respecting anonymous usage analytics
- Easy opt-out via `BASIC_MEMORY_NO_ANALYTICS=1` environment variable
- Helps improve Basic Memory based on real usage patterns
- **#474**: Add auto-format files on save with built-in Python formatter
([`1fd680c`](https://github.com/basicmachines-co/basic-memory/commit/1fd680c))
- Automatic markdown formatting on file save
- Built-in Python formatter for consistent code style
- Configurable formatting options
- **#447**: Complete Phase 2 of API v2 migration - MCP tools use v2 endpoints
([`1a74d85`](https://github.com/basicmachines-co/basic-memory/commit/1a74d85))
- All MCP tools now use optimized v2 API endpoints
- Improved performance for knowledge graph operations
- Foundation for future API enhancements
### Bug Fixes
- Fix UTF-8 BOM handling in frontmatter parsing
([`85684f8`](https://github.com/basicmachines-co/basic-memory/commit/85684f8))
- Handles files with UTF-8 byte order marks correctly
- Prevents frontmatter parsing failures
- **#475**: Handle null titles in ChatGPT import
([`14ce5a3`](https://github.com/basicmachines-co/basic-memory/commit/14ce5a3))
- Gracefully handles conversations without titles
- Improved import robustness
- Remove MaxLen constraint from observation content
([`45d6caf`](https://github.com/basicmachines-co/basic-memory/commit/45d6caf))
- Allows longer observation content without truncation
- Removes arbitrary 2000 character limit
- Handle FileNotFoundError gracefully during sync
([`1652f86`](https://github.com/basicmachines-co/basic-memory/commit/1652f86))
- Prevents sync failures when files are deleted during sync
- More resilient file watching
- Use canonical project names in API response messages
([`c23927d`](https://github.com/basicmachines-co/basic-memory/commit/c23927d))
- Consistent project name formatting in all responses
- Suppress CLI warnings for cleaner output
([`d71c6e8`](https://github.com/basicmachines-co/basic-memory/commit/d71c6e8))
- Cleaner terminal output without spurious warnings
- Prevent DEBUG logs from appearing on CLI stdout
([`63b9849`](https://github.com/basicmachines-co/basic-memory/commit/63b9849))
- Debug logging no longer pollutes CLI output
- **#473**: Detect rclone version for --create-empty-src-dirs support
([`622d37e`](https://github.com/basicmachines-co/basic-memory/commit/622d37e))
- Automatic rclone version detection for compatibility
- Prevents errors on older rclone versions
- **#471**: Prevent CLI commands from hanging on exit
([`916baf8`](https://github.com/basicmachines-co/basic-memory/commit/916baf8))
- Fixes CLI hang on shutdown
- Proper async cleanup
- Add cloud_mode check to initialize_app()
([`ef7adb7`](https://github.com/basicmachines-co/basic-memory/commit/ef7adb7))
- Correct initialization for cloud deployments
### Internal
- Centralize test environment detection in config.is_test_env
([`3cd9178`](https://github.com/basicmachines-co/basic-memory/commit/3cd9178))
- Unified test environment detection
- Disables analytics in test environments
- Make test-int-postgres compatible with macOS
([`95937c6`](https://github.com/basicmachines-co/basic-memory/commit/95937c6))
- Cross-platform PostgreSQL testing support
## v0.16.3 (2025-12-20)
### Features
- **#439**: Add PostgreSQL database backend support
([`fb5e9e1`](https://github.com/basicmachines-co/basic-memory/commit/fb5e9e1))
- Full PostgreSQL/Neon database support as alternative to SQLite
- Async connection pooling with asyncpg
- Alembic migrations support for both backends
- Configurable via `BASIC_MEMORY_DATABASE_BACKEND` environment variable
- **#441**: Implement API v2 with ID-based endpoints (Phase 1)
([`28cc522`](https://github.com/basicmachines-co/basic-memory/commit/28cc522))
- New ID-based API endpoints for improved performance
- Foundation for future API enhancements
- Backward compatible with existing endpoints
- Add project_id to Relation and Observation for efficient project-scoped queries
([`a920a9f`](https://github.com/basicmachines-co/basic-memory/commit/a920a9f))
- Enables faster queries in multi-project environments
- Improved database schema for cloud deployments
- Add bulk insert with ON CONFLICT handling for relations
([`0818bda`](https://github.com/basicmachines-co/basic-memory/commit/0818bda))
- Faster relation creation during sync operations
- Handles duplicate relations gracefully
### Performance
- Lightweight permalink resolution to avoid eager loading
([`6f99d2e`](https://github.com/basicmachines-co/basic-memory/commit/6f99d2e))
- Reduces database queries during entity lookups
- Improved response times for read operations
### Bug Fixes
- **#464**: Pin FastMCP to 2.12.3 to fix MCP tools visibility
([`f227ef6`](https://github.com/basicmachines-co/basic-memory/commit/f227ef6))
- Fixes issue where MCP tools were not visible to Claude
- Reverts to last known working FastMCP version
- **#458**: Reduce watch service CPU usage by increasing reload interval
([`897b1ed`](https://github.com/basicmachines-co/basic-memory/commit/897b1ed))
- Lowers CPU usage during file watching
- More efficient resource utilization
- **#456**: Await background sync task cancellation in lifespan shutdown
([`efbc758`](https://github.com/basicmachines-co/basic-memory/commit/efbc758))
- Prevents hanging on shutdown
- Clean async task cleanup
- **#434**: Respect --project flag in background sync
([`70bb10b`](https://github.com/basicmachines-co/basic-memory/commit/70bb10b))
- Background sync now correctly uses specified project
- Fixes multi-project sync issues
- **#446**: Fix observation parsing and permalink limits
([`73d940e`](https://github.com/basicmachines-co/basic-memory/commit/73d940e))
- Handles edge cases in observation content
- Prevents permalink truncation issues
- **#424**: Handle periods in kebab_filenames mode
([`b004565`](https://github.com/basicmachines-co/basic-memory/commit/b004565))
- Fixes filename handling for files with multiple periods
- Improved kebab-case conversion
- Fix Postgres/Neon connection settings and search index dedupe
([`b5d4fb5`](https://github.com/basicmachines-co/basic-memory/commit/b5d4fb5))
- Optimized connection pooling for Postgres
- Prevents duplicate search index entries
### Testing & CI
- Replace py-pglite with testcontainers for Postgres testing
([`c462faf`](https://github.com/basicmachines-co/basic-memory/commit/c462faf))
- More reliable Postgres testing infrastructure
- Uses Docker-based test containers
- Add PostgreSQL testing to GitHub Actions workflow
([`66b91b2`](https://github.com/basicmachines-co/basic-memory/commit/66b91b2))
- CI now tests both SQLite and PostgreSQL backends
- Ensures cross-database compatibility
- **#416**: Add integration test for read_note with underscored folders
([`0c12a39`](https://github.com/basicmachines-co/basic-memory/commit/0c12a39))
- Verifies folder name handling edge cases
### Internal
- Cloud compatibility fixes and performance improvements (#454)
- Remove logfire instrumentation for cleaner production deployments
- Truncate content_stems to fix Postgres 8KB index row limit
## v0.16.2 (2025-11-16)
### Bug Fixes
+18 -94
View File
@@ -15,14 +15,10 @@ See the [README.md](README.md) file for a project overview.
### Build and Test Commands
- Install: `just install` or `pip install -e ".[dev]"`
- Run all tests (SQLite + Postgres): `just test`
- Run all tests against SQLite: `just test-sqlite`
- Run all tests against Postgres: `just test-postgres` (uses testcontainers)
- Run unit tests (SQLite): `just test-unit-sqlite`
- Run unit tests (Postgres): `just test-unit-postgres`
- Run integration tests (SQLite): `just test-int-sqlite`
- Run integration tests (Postgres): `just test-int-postgres`
- Generate HTML coverage: `just coverage`
- Run all tests (with coverage): `just test` - Runs both unit and integration tests with unified coverage
- Run unit tests only: `just test-unit` - Fast, no coverage
- Run integration tests only: `just test-int` - Fast, no coverage
- Generate HTML coverage: `just coverage` - Opens in browser
- Single test: `pytest tests/path/to/test_file.py::test_function_name`
- Run benchmarks: `pytest test-int/test_sync_performance_benchmark.py -v -m "benchmark and not slow"`
- Lint: `just lint` or `ruff check . --fix`
@@ -34,8 +30,6 @@ See the [README.md](README.md) file for a project overview.
**Note:** Project requires Python 3.12+ (uses type parameter syntax and `type` aliases introduced in 3.12)
**Postgres Testing:** Uses [testcontainers](https://testcontainers-python.readthedocs.io/) which automatically spins up a Postgres instance in Docker. No manual database setup required - just have Docker running.
### Test Structure
- `tests/` - Unit tests for individual components (mocked, fast)
@@ -58,69 +52,11 @@ See the [README.md](README.md) file for a project overview.
- Follow the repository pattern for data access
- Tools communicate to api routers via the httpx ASGI client (in process)
### Code Change Guidelines
- **Full file read before edits**: Before editing any file, read it in full first to ensure complete context; partial reads lead to corrupted edits
- **Minimize diffs**: Prefer the smallest change that satisfies the request. Avoid unrelated refactors or style rewrites unless necessary for correctness
- **No speculative getattr**: Never use `getattr(obj, "attr", default)` when unsure about attribute names. Check the class definition or source code first
- **Fail fast**: Write code with fail-fast logic by default. Do not swallow exceptions with errors or warnings
- **No fallback logic**: Do not add fallback logic unless explicitly told to and agreed with the user
- **No guessing**: Do not say "The issue is..." before you actually know what the issue is. Investigate first.
### Literate Programming Style
Code should tell a story. Comments must explain the "why" and narrative flow, not just the "what".
**Section Headers:**
For files with multiple phases of logic, add section headers so the control flow reads like chapters:
```python
# --- Authentication ---
# ... auth logic ...
# --- Data Validation ---
# ... validation logic ...
# --- Business Logic ---
# ... core logic ...
```
**Decision Point Comments:**
For conditionals that materially change behavior (gates, fallbacks, retries, feature flags), add comments with:
- **Trigger**: what condition causes this branch
- **Why**: the rationale (cost, correctness, UX, determinism)
- **Outcome**: what changes downstream
```python
# Trigger: project has no active sync watcher
# Why: avoid duplicate file system watchers consuming resources
# Outcome: starts new watcher, registers in active_watchers dict
if project_id not in active_watchers:
start_watcher(project_id)
```
**Constraint Comments:**
If code exists because of a constraint (async requirements, rate limits, schema compatibility), explain the constraint near the code:
```python
# SQLite requires WAL mode for concurrent read/write access
connection.execute("PRAGMA journal_mode=WAL")
```
**What NOT to Comment:**
Avoid comments that restate obvious code:
```python
# Bad - restates code
counter += 1 # increment counter
# Good - explains why
counter += 1 # track retries for backoff calculation
```
### Codebase Architecture
- `/alembic` - Alembic db migrations
- `/api` - FastAPI implementation of REST endpoints
- `/cli` - Typer command-line interface
- `/importers` - Import functionality for Claude, ChatGPT, and other sources
- `/markdown` - Markdown parsing and processing
- `/mcp` - Model Context Protocol server implementation
- `/models` - SQLAlchemy ORM models
@@ -140,10 +76,8 @@ counter += 1 # track retries for backoff calculation
- SQLite is used for indexing and full text search, files are source of truth
- Testing uses pytest with asyncio support (strict mode)
- Unit tests (`tests/`) use mocks when necessary; integration tests (`test-int/`) use real implementations
- By default, tests run against SQLite (fast, no Docker needed)
- Set `BASIC_MEMORY_TEST_POSTGRES=1` to run against Postgres (uses testcontainers - Docker required)
- Each test runs in a standalone environment with isolated database and tmp_path directory
- CI runs SQLite and Postgres tests in parallel for faster feedback
- Test database uses in-memory SQLite
- Each test runs in a standalone environment with in-memory SQLite and tmp_file directory
- Performance benchmarks are in `test-int/test_sync_performance_benchmark.py`
- Use pytest markers: `@pytest.mark.benchmark` for benchmarks, `@pytest.mark.slow` for slow tests
@@ -198,26 +132,22 @@ See SPEC-16 for full context manager refactor details.
### Basic Memory Commands
**Local Commands:**
- Check sync status: `basic-memory status`
- Sync knowledge: `basic-memory sync` or `basic-memory sync --watch`
- Import from Claude: `basic-memory import claude conversations`
- Import from ChatGPT: `basic-memory import chatgpt`
- Import from Memory JSON: `basic-memory import memory-json`
- Tool access: `basic-memory tool` (provides CLI access to MCP tools)
- Continue: `basic-memory tool continue-conversation --topic="search"`
**Project Management:**
- List projects: `basic-memory project list`
- Add project: `basic-memory project add "name" ~/path`
- Project info: `basic-memory project info`
- One-way sync (local -> cloud): `basic-memory project sync`
- Bidirectional sync: `basic-memory project bisync`
- Integrity check: `basic-memory project check`
- Check sync status: `basic-memory status`
- Tool access: `basic-memory tools` (provides CLI access to MCP tools)
- Guide: `basic-memory tools basic-memory-guide`
- Continue: `basic-memory tools continue-conversation --topic="search"`
**Cloud Commands (requires subscription):**
- Authenticate: `basic-memory cloud login`
- Logout: `basic-memory cloud logout`
- Check cloud status: `basic-memory cloud status`
- Setup cloud sync: `basic-memory cloud setup`
- Bidirectional sync: `basic-memory cloud sync`
- Integrity check: `basic-memory cloud check`
- Mount cloud storage: `basic-memory cloud mount`
- Unmount cloud storage: `basic-memory cloud unmount`
### MCP Capabilities
@@ -244,19 +174,18 @@ See SPEC-16 for full context manager refactor details.
- `list_memory_projects()` - List all available projects with their status
- `create_memory_project(project_name, project_path, set_default)` - Create new Basic Memory projects
- `delete_project(project_name)` - Delete a project from configuration
- `get_current_project()` - Get current project information and stats
- `sync_status()` - Check file synchronization and background operation status
**Visualization:**
- `canvas(nodes, edges, title, folder)` - Generate Obsidian canvas files for knowledge graph visualization
**ChatGPT-Compatible Tools:**
- `search(query)` - Search across knowledge base (OpenAI actions compatible)
- `fetch(id)` - Fetch full content of a search result document
- MCP Prompts for better AI interaction:
- `ai_assistant_guide()` - Guidance on effectively using Basic Memory tools for AI assistants
- `continue_conversation(topic, timeframe)` - Continue previous conversations with relevant historical context
- `search(query, after_date)` - Search with detailed, formatted results for better context understanding
- `recent_activity(timeframe)` - View recently changed items with formatted output
- `json_canvas_spec()` - Full JSON Canvas specification for Obsidian visualization
### Cloud Features (v0.15.0+)
@@ -300,11 +229,6 @@ of using AI just for code generation, we've developed a true collaborative workf
This approach has allowed us to tackle more complex challenges and build a more robust system than either humans or AI
could achieve independently.
**Problem-Solving Guidance:**
- If a solution isn't working after reasonable effort, suggest alternative approaches
- Don't persist with a problematic library or pattern when better alternatives exist
- Example: When py-pglite caused cascading test failures, switching to testcontainers-postgres was the right call
## GitHub Integration
Basic Memory has taken AI-Human collaboration to the next level by integrating Claude directly into the development workflow through GitHub:
+14 -81
View File
@@ -433,109 +433,42 @@ See the [Documentation](https://memory.basicmachines.co/) for more info, includi
- [Managing multiple Projects](https://docs.basicmemory.com/guides/cli-reference/#project)
- [Importing data from OpenAI/Claude Projects](https://docs.basicmemory.com/guides/cli-reference/#import)
## Logging
Basic Memory uses [Loguru](https://github.com/Delgan/loguru) for logging. The logging behavior varies by entry point:
| Entry Point | Default Behavior | Use Case |
|-------------|------------------|----------|
| CLI commands | File only | Prevents log output from interfering with command output |
| MCP server | File only | Stdout would corrupt the JSON-RPC protocol |
| API server | File (local) or stdout (cloud) | Docker/cloud deployments use stdout |
**Log file location:** `~/.basic-memory/basic-memory.log` (10MB rotation, 10 days retention)
### Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `BASIC_MEMORY_LOG_LEVEL` | `INFO` | Log level: DEBUG, INFO, WARNING, ERROR |
| `BASIC_MEMORY_CLOUD_MODE` | `false` | When `true`, API logs to stdout with structured context |
| `BASIC_MEMORY_ENV` | `dev` | Set to `test` for test mode (stderr only) |
### Examples
```bash
# Enable debug logging
BASIC_MEMORY_LOG_LEVEL=DEBUG basic-memory sync
# View logs
tail -f ~/.basic-memory/basic-memory.log
# Cloud/Docker mode (stdout logging with structured context)
BASIC_MEMORY_CLOUD_MODE=true uvicorn basic_memory.api.app:app
```
## Telemetry
Basic Memory collects anonymous usage statistics to help improve the software. This follows the [Homebrew model](https://docs.brew.sh/Analytics) - telemetry is on by default with easy opt-out.
**What we collect:**
- App version, Python version, OS, architecture
- Feature usage (which MCP tools and CLI commands are used)
- Error types (sanitized - no file paths or personal data)
**What we NEVER collect:**
- Note content, file names, or paths
- Personal information
- IP addresses
**Opting out:**
```bash
# Disable telemetry
basic-memory telemetry disable
# Check status
basic-memory telemetry status
# Re-enable
basic-memory telemetry enable
```
Or set the environment variable:
```bash
export BASIC_MEMORY_TELEMETRY_ENABLED=false
```
For more details, see the [Telemetry documentation](https://basicmemory.com/telemetry).
## Development
### Running Tests
Basic Memory supports dual database backends (SQLite and Postgres). By default, tests run against SQLite. Set `BASIC_MEMORY_TEST_POSTGRES=1` to run against Postgres (uses testcontainers - Docker required).
Basic Memory supports dual database backends (SQLite and Postgres). Tests are parametrized to run against both backends automatically.
**Quick Start:**
```bash
# Run all tests against SQLite (default, fast)
# Run SQLite tests (default, no Docker needed)
just test-sqlite
# Run all tests against Postgres (uses testcontainers)
# Run Postgres tests (requires Docker)
just test-postgres
# Run both SQLite and Postgres tests
just test
```
**Available Test Commands:**
- `just test` - Run all tests against both SQLite and Postgres
- `just test-sqlite` - Run all tests against SQLite (fast, no Docker needed)
- `just test-postgres` - Run all tests against Postgres (uses testcontainers)
- `just test-unit-sqlite` - Run unit tests against SQLite
- `just test-unit-postgres` - Run unit tests against Postgres
- `just test-int-sqlite` - Run integration tests against SQLite
- `just test-int-postgres` - Run integration tests against Postgres
- `just test-sqlite` - Run tests against SQLite only (fastest, no Docker needed)
- `just test-postgres` - Run tests against Postgres only (requires Docker)
- `just test-windows` - Run Windows-specific tests (auto-skips on other platforms)
- `just test-benchmark` - Run performance benchmark tests
- `just test-all` - Run all tests including Windows, Postgres, and benchmarks
**Postgres Testing:**
**Postgres Testing Requirements:**
Postgres tests use [testcontainers](https://testcontainers-python.readthedocs.io/) which automatically spins up a Postgres instance in Docker. No manual database setup required - just have Docker running.
To run Postgres tests, you need to start the test database:
```bash
docker-compose -f docker-compose-postgres.yml up -d
```
Tests will connect to `localhost:5433/basic_memory_test`.
**Test Markers:**
Tests use pytest markers for selective execution:
- `postgres` - Tests that run against Postgres backend
- `windows` - Windows-specific database optimizations
- `benchmark` - Performance tests (excluded from default runs)
@@ -0,0 +1,23 @@
{
"name": "basicmachines",
"owner": {
"name": "Basic Machines",
"email": "hello@basicmachines.co"
},
"metadata": {
"description": "Official plugins from Basic Machines for knowledge management and AI-assisted development",
"version": "0.1.0"
},
"plugins": [
{
"name": "basic-memory",
"source": ".",
"description": "Skills, commands, and hooks for Basic Memory MCP - capture knowledge, continue conversations, and follow spec-driven development",
"version": "0.1.0",
"author": {
"name": "Basic Machines"
},
"keywords": ["memory", "knowledge", "mcp", "specs", "context"]
}
]
}
@@ -0,0 +1,9 @@
{
"name": "basic-memory",
"description": "Claude Code skills for Basic Memory - capture knowledge, continue conversations, and follow spec-driven development using the Basic Memory MCP server",
"version": "0.1.0",
"author": {
"name": "Basic Machines"
},
"repository": "https://github.com/basicmachines-co/basic-memory"
}
+313
View File
@@ -0,0 +1,313 @@
# Basic Memory Plugin for Claude Code
This plugin provides skills, commands, and hooks for working with [Basic Memory](https://basicmemory.io) - a local-first knowledge management system built on the Model Context Protocol (MCP).
## Prerequisites
You need the Basic Memory MCP server running. Install it via:
```bash
# Install basic-memory
pip install basic-memory
# Or with pipx
pipx install basic-memory
```
Then add it to your Claude Code MCP configuration.
## Installation
### Add the Marketplace
```
/plugin marketplace add basicmachines-co/basic-memory/claude-code-plugin
```
### Install the Plugin
```
/plugin install basic-memory@basicmachines
```
### Or via Repository Settings
Add to your `.claude/settings.json`:
```json
{
"plugins": {
"extraKnownMarketplaces": {
"basicmachines": {
"source": {
"source": "github",
"repo": "basicmachines-co/basic-memory",
"path": "claude-code-plugin"
}
}
},
"installed": ["basic-memory@basicmachines"]
}
}
```
---
## Slash Commands
User-invoked commands for explicit interaction with Basic Memory.
### `/remember [title] [folder]`
Capture insights, decisions, or learnings from the current conversation.
```
/remember "FastAPI Async Pattern"
/remember "Auth Decision" decisions
```
Creates a structured note with:
- Context from the conversation
- Observations with `[decision]`, `[insight]`, `[pattern]` categories
- Relations linking to related concepts
### `/continue [topic]`
Resume previous work by building context from Basic Memory.
```
/continue postgres migration
/continue SPEC-24
/continue
```
If no topic is provided, shows recent activity and asks what to dive into.
### `/context <memory://url> [depth] [timeframe]`
Build context from a specific memory:// URL.
```
/context memory://SPEC-24
/context memory://architecture/* 3 2weeks
```
### `/recent [timeframe] [project]`
Show recent activity in Basic Memory.
```
/recent
/recent 1week
/recent today specs
```
### `/organize [action] [project]`
Organize and maintain your knowledge graph.
```
/organize # Quick health check
/organize orphans # Find unlinked notes
/organize duplicates # Find similar notes
/organize relations "Note" # Suggest links for a note
/organize tags # Review tag consistency
```
Actions:
- `health` - Overview of knowledge base status (default)
- `orphans` - Find notes with no relations
- `duplicates` - Find overlapping notes
- `relations` - Suggest connections
- `tags` - Review tag consistency
### `/research <topic> [folder]`
Research a topic and save a structured report to Basic Memory.
```
/research MCP protocol
/research "database migrations"
/research "auth options" decisions
```
Produces a report with:
- Summary and key findings
- Analysis and recommendations
- Sources and related notes
- Saved to `research/` folder by default
---
## Skills
Model-invoked capabilities that Claude uses automatically based on context.
### knowledge-capture
Automatically captures insights, decisions, and learnings into structured notes.
**Triggers when:**
- Important decisions are made
- Technical insights are discovered
- Problems are solved
- Design trade-offs are discussed
### continue-conversation
Resumes previous work by building context from the knowledge graph.
**Triggers when:**
- Starting a new session
- User mentions previous work ("continue with...", "back to...")
- Need context about ongoing projects
### spec-driven-development
Guides implementation based on specifications stored in Basic Memory.
**Triggers when:**
- Implementing a feature defined by a spec
- Creating new specifications
- Reviewing implementation against criteria
### edit-note
Interactively edit notes using MCP tools in a conversational workflow.
**Triggers when:**
- User wants to edit, update, or modify a note
- User asks to change specific content in a note
- User wants to add observations or relations
**How it works:**
1. Fetches the note via MCP
2. Shows current content
3. Applies edits using `edit_note` operations (append, prepend, find_replace, replace_section)
4. Shows the updated result
**Best for:** Cloud users or when you want conversational editing.
### edit-note-local
Edit notes directly as local markdown files with automatic sync.
**Triggers when:**
- User has local Basic Memory installation
- User wants to make substantial file edits
- User prefers working with full file content
**How it works:**
1. Finds the note's file path via MCP
2. Uses Claude Code's Read/Edit/Write tools on the actual file
3. Basic Memory's `sync --watch` picks up changes automatically
**Best for:** Local users who want full file access and git integration.
### knowledge-organize
Help organize, link, and maintain the knowledge graph.
**Triggers when:**
- User wants to organize their notes
- User asks about orphan or unlinked notes
- User wants to find connections between notes
- User mentions duplicates or similar notes
- User asks for help with folder organization
**Capabilities:**
- **Find orphan notes** - Identify notes with no relations
- **Suggest relations** - Propose meaningful links between notes
- **Identify duplicates** - Find notes covering similar topics
- **Folder organization** - Review and suggest folder structure
- **Tag consistency** - Normalize and improve tagging
- **Create index notes** - Generate hub notes linking related topics
- **Enrich sparse notes** - Suggest observations and structure
**Best for:** Periodic knowledge base maintenance and improving discoverability.
### research
Research topics thoroughly and produce structured reports saved to Basic Memory.
**Triggers when:**
- User asks to research or investigate something
- User wants to understand a concept or technology
- User needs context before making a decision
- Phrases like "research", "look into", "explore", "investigate"
**What it produces:**
- Structured report with summary, findings, and analysis
- Recommendations when applicable
- Links to sources and related notes
- Saved to `research/` folder
**Best for:** Building knowledge base through investigation and documentation.
---
## Hooks
Automated behaviors that enhance the Basic Memory workflow.
### PostToolUse: write_note
Confirms when notes are saved to Basic Memory.
### Stop
After significant conversations, suggests using `/remember` to capture valuable insights (only when genuinely useful).
---
## MCP Tools Used
This plugin leverages Basic Memory's MCP tools:
| Tool | Purpose |
|------|---------|
| `write_note` | Create/update markdown notes |
| `read_note` | Read notes by title or permalink |
| `search_notes` | Full-text search across content |
| `build_context` | Navigate knowledge graph via memory:// URLs |
| `recent_activity` | Get recently updated information |
| `edit_note` | Incrementally update notes |
---
## Plugin Structure
```
claude-code-plugin/
├── .claude-plugin/
│ ├── plugin.json # Plugin manifest
│ └── marketplace.json # Self-hosted marketplace
├── commands/
│ ├── remember.md # /remember command
│ ├── continue.md # /continue command
│ ├── context.md # /context command
│ ├── recent.md # /recent command
│ ├── organize.md # /organize command
│ └── research.md # /research command
├── skills/
│ ├── knowledge-capture/
│ ├── continue-conversation/
│ ├── spec-driven-development/
│ ├── edit-note/
│ ├── edit-note-local/
│ ├── knowledge-organize/
│ └── research/
├── hooks/
│ └── hooks.json # Hook definitions
├── README.md # Quick start guide
└── PLUGIN.md # Full documentation
```
---
## Related
- [Basic Memory Documentation](https://docs.basicmemory.io)
- [Basic Memory GitHub](https://github.com/basicmachines-co/basic-memory)
- [Model Context Protocol](https://modelcontextprotocol.io)
- [Claude Code Plugins](https://code.claude.com/docs/en/plugins)
+100
View File
@@ -0,0 +1,100 @@
# Basic Memory Plugin for Claude Code
A Claude Code plugin that integrates [Basic Memory](https://basicmemory.io) - a local-first knowledge management system built on the Model Context Protocol (MCP).
## What This Plugin Does
This plugin helps Claude Code work seamlessly with your Basic Memory knowledge base:
- **Capture knowledge** from conversations automatically
- **Resume previous work** by building context from your knowledge graph
- **Edit notes** interactively through conversation
- **Organize your knowledge** by finding orphans, suggesting links, and maintaining structure
## Installation
### 1. Install Basic Memory
```bash
pip install basic-memory
# or
pipx install basic-memory
```
### 2. Add the Marketplace
```
/plugin marketplace add basicmachines-co/basic-memory/claude-code-plugin
```
### 3. Install the Plugin
```
/plugin install basic-memory@basicmachines
```
## Commands
| Command | Description |
|---------|-------------|
| `/remember [title]` | Capture insights from the current conversation |
| `/continue [topic]` | Resume previous work with context |
| `/context <memory://url>` | Build context from a specific note |
| `/recent [timeframe]` | Show recent activity |
| `/organize [action]` | Maintain your knowledge graph |
| `/research <topic>` | Research a topic and save a report |
### Examples
```bash
# Capture what we just discussed
/remember "Database Design Decision"
# Pick up where we left off
/continue postgres migration
# Check recent changes
/recent 1week
# Find orphan notes and suggest links
/organize orphans
# Research a topic and save findings
/research "MCP protocol"
```
## Skills
Skills are model-invoked - Claude uses them automatically when the context fits.
| Skill | What It Does |
|-------|--------------|
| `knowledge-capture` | Auto-captures decisions and insights into structured notes |
| `continue-conversation` | Builds context when resuming previous work |
| `spec-driven-development` | Guides implementation based on specs in Basic Memory |
| `edit-note` | Edits notes via MCP tools (cloud-compatible) |
| `edit-note-local` | Edits notes as files (local installations) |
| `knowledge-organize` | Helps organize and link notes |
| `research` | Researches topics and produces saved reports |
## Hooks
| Event | Behavior |
|-------|----------|
| `PostToolUse: write_note` | Confirms when notes are saved |
| `Stop` | Suggests capturing valuable insights after conversations |
## Requirements
- [Claude Code](https://claude.com/claude-code)
- [Basic Memory](https://basicmemory.io) with MCP server configured
## Documentation
- [Full Plugin Documentation](./PLUGIN.md)
- [Basic Memory Docs](https://docs.basicmemory.io)
- [Claude Code Plugins](https://code.claude.com/docs/en/plugins)
## License
MIT - See the [Basic Memory repository](https://github.com/basicmachines-co/basic-memory) for details.
+39
View File
@@ -0,0 +1,39 @@
---
description: Build context from a Basic Memory URL
argument-hint: <memory://url> [depth] [timeframe]
allowed-tools: mcp__basic-memory__build_context, mcp__basic-memory__read_note
---
# Context
Build context from a Basic Memory memory:// URL.
## Arguments
- `$1` - Memory URL (e.g., `memory://topic`, `memory://folder/*`, `memory://SPEC-24`)
- `$2` - Depth of relation traversal (optional, default: 2)
- `$3` - Timeframe for recent changes (optional, default: "7d")
## Your Task
Navigate the knowledge graph and build comprehensive context.
1. **Build context** using `mcp__basic-memory__build_context`:
- url: "$1"
- depth: $2 or 2
- timeframe: "$3" or "7d"
2. **Present the context**:
- Main note content
- Related notes found via relations
- Recent changes within timeframe
- Key observations and decisions
3. **Read additional notes** if needed for more detail.
## Memory URL Formats
- `memory://note-title` - Single note by title
- `memory://folder/*` - All notes in a folder
- `memory://SPEC-*` - Pattern matching
- `memory://specs/SPEC-24` - Note in specific project folder
+46
View File
@@ -0,0 +1,46 @@
---
description: Resume previous work from Basic Memory context
argument-hint: [topic]
allowed-tools: mcp__basic-memory__build_context, mcp__basic-memory__recent_activity, mcp__basic-memory__search_notes, mcp__basic-memory__read_note
---
# Continue
Resume previous work by building context from Basic Memory.
## Arguments
- `$ARGUMENTS` - Topic, note title, or search terms to find previous context
## Your Task
Build context to continue previous work seamlessly.
1. **Find relevant context**:
If a specific topic is provided ("$ARGUMENTS"):
- Search for matching notes: `mcp__basic-memory__search_notes`
- Build context from matches: `mcp__basic-memory__build_context`
- Read key notes for details: `mcp__basic-memory__read_note`
If no topic provided:
- Get recent activity: `mcp__basic-memory__recent_activity` with timeframe "3d"
- Present what's been happening
- Ask which topic to dive into
2. **Present context**:
- Summarize current state of the work
- Highlight recent changes or progress
- List open items or next steps
- Show related context from the knowledge graph
3. **Be ready to continue**:
- Understand what was done before
- Know what needs to happen next
- Have relevant context loaded
## Tips
- Use `memory://topic` URL format with `build_context`
- Check multiple projects if needed (main, specs)
- Follow relations to find connected knowledge
+87
View File
@@ -0,0 +1,87 @@
---
description: Organize and maintain your Basic Memory knowledge graph
argument-hint: [health|orphans|duplicates|relations|tags] [project]
allowed-tools: mcp__basic-memory__search_notes, mcp__basic-memory__read_note, mcp__basic-memory__list_directory, mcp__basic-memory__edit_note, mcp__basic-memory__write_note
---
# Organize
Help organize, link, and maintain your Basic Memory knowledge graph.
## Arguments
- `$1` - Action (optional): `health`, `orphans`, `duplicates`, `relations`, `tags` (default: `health`)
- `$2` - Project (optional): defaults to "main"
## Actions
### `/organize` or `/organize health`
Run a quick health check:
1. Count total notes
2. Identify orphan notes (no relations)
3. Check for potential duplicates
4. Show folder distribution
5. Report any issues found
### `/organize orphans`
Find and address orphan notes:
1. Search for notes with empty Relations sections
2. List orphans found
3. For each orphan, suggest potential relations based on content
4. Offer to add relations or create index notes
### `/organize duplicates`
Find potentially duplicate notes:
1. Search for notes with similar titles
2. Compare content for overlap
3. Suggest: merge, differentiate, or link with `supersedes`
### `/organize relations [note-title]`
Suggest relations for a specific note (or recent notes if not specified):
1. Read the target note
2. Search for related content
3. Suggest relation types:
- `relates-to` - General connection
- `extends` - Builds upon
- `implements` - Realizes concept
- `depends-on` - Requires understanding of
4. Offer to add selected relations
### `/organize tags`
Review tag consistency:
1. Gather all tags across notes
2. Find similar/duplicate tags (e.g., `arch` vs `architecture`)
3. Identify over-used or under-used tags
4. Suggest normalization
## Your Task
Execute: `/organize $ARGUMENTS`
Based on the action requested:
1. **Gather data** using search and list tools
2. **Analyze** for the specific issue (orphans, duplicates, etc.)
3. **Present findings** clearly with counts and examples
4. **Offer solutions** - ask before making changes
5. **Apply fixes** using edit_note or write_note when user approves
Always confirm before modifying notes. Show what will change and get approval.
## Examples
```
/organize # Quick health check
/organize health # Same as above
/organize orphans # Find unlinked notes
/organize duplicates # Find similar notes
/organize relations # Suggest links for recent notes
/organize relations "My Note" # Suggest links for specific note
/organize tags # Review tag consistency
/organize health specs # Health check on specs project
```
+40
View File
@@ -0,0 +1,40 @@
---
description: Show recent activity in Basic Memory
argument-hint: [timeframe] [project]
allowed-tools: mcp__basic-memory__recent_activity, mcp__basic-memory__read_note
---
# Recent
Show recent activity in Basic Memory.
## Arguments
- `$1` - Timeframe (optional): "today", "1d", "3d", "1 week", "2 weeks" (default: "3d")
- `$2` - Project (optional): "main", "specs", etc.
## Your Task
Show what's been happening in Basic Memory recently.
1. **Get recent activity** using `mcp__basic-memory__recent_activity`:
- timeframe: "$1" or "3d"
- project: "$2" or check all projects
2. **Present activity**:
- List recently modified notes
- Group by type or folder if helpful
- Highlight key changes
- Show dates of modifications
3. **Offer to dive deeper**:
- Ask if user wants to read any specific notes
- Suggest continuing work on active items
## Timeframe Examples
- `today` - Just today
- `1d` or `yesterday` - Last 24 hours
- `3d` - Last 3 days
- `1 week` - Last week
- `2 weeks` - Last 2 weeks
+43
View File
@@ -0,0 +1,43 @@
---
description: Capture insights, decisions, or learnings to Basic Memory
argument-hint: [title] [optional: folder]
allowed-tools: mcp__basic-memory__write_note, mcp__basic-memory__search_notes
---
# Remember
Capture what we just discussed into a Basic Memory note.
## Arguments
- `$1` - Title for the note (required)
- `$2` - Folder to save in (optional, defaults to "notes")
## Your Task
Create a structured note capturing the key insights from our conversation.
1. **Analyze the conversation** for:
- Decisions made
- Insights discovered
- Problems solved
- Patterns identified
- Trade-offs discussed
2. **Structure the note** with:
- Clear title: "$1" (or generate one if not provided)
- Context section explaining the situation
- Main content with key points
- Observations using `[category]` format:
- `[decision]` - Choices made
- `[insight]` - Understanding gained
- `[pattern]` - Reusable approaches
- `[learning]` - Lessons learned
- Relations to link related concepts with `[[WikiLinks]]`
3. **Save using** `mcp__basic-memory__write_note`:
- folder: "$2" or "notes"
- Include relevant tags
- Project: use "main" unless user specifies otherwise
4. **Confirm** what was captured and where it was saved.
+140
View File
@@ -0,0 +1,140 @@
---
description: Research a topic and save a structured report to Basic Memory
argument-hint: <topic> [folder]
allowed-tools: mcp__basic-memory__write_note, mcp__basic-memory__search_notes, mcp__basic-memory__read_note, mcp__basic-memory__build_context, WebSearch, WebFetch, Grep, Glob, Read
---
# Research
Research a topic thoroughly and produce a structured report saved to Basic Memory.
## Arguments
- `$1` - Topic to research (required)
- `$2` - Folder to save report (optional, default: "research")
## Your Task
Conduct thorough research on: **$ARGUMENTS**
### 1. Check Existing Knowledge
First, see what we already know:
```python
mcp__basic-memory__search_notes(query="$1", project="main")
```
Read any relevant existing notes to avoid duplicating research.
### 2. Gather Information
Depending on the topic, use appropriate tools:
**For codebase topics:**
- Search code with Grep/Glob
- Read relevant files
- Check tests for examples
**For external topics:**
- Use WebSearch for current information
- Fetch documentation with WebFetch
- Look for official sources
**For Basic Memory context:**
- Build context from related notes
- Check for prior decisions or research
### 3. Analyze Findings
Synthesize what you learned:
- Identify key concepts
- Note patterns and trade-offs
- Form recommendations if applicable
- Flag uncertainties
### 4. Produce Report
Create a structured report with this format:
```markdown
---
title: "Research: [Topic]"
type: research
tags:
- research
- [relevant-tags]
---
# Research: [Topic]
## Summary
[2-3 sentence executive summary]
## Research Question
[What we investigated and why]
## Key Findings
### [Finding 1]
[Details and evidence]
### [Finding 2]
[Details and evidence]
### [Finding 3]
[Details and evidence]
## Analysis
[Synthesis, patterns, trade-offs, recommendations]
## Open Questions
- [Areas needing more investigation]
## Sources
- [Links to sources]
- [[Related Notes]] from Basic Memory
## Observations
- [finding] Key insight #research
- [recommendation] Suggested approach based on research
## Relations
- researches [[Topic]]
- relates-to [[Related Concepts]]
```
### 5. Save Report
```python
mcp__basic-memory__write_note(
title="Research: $1",
content="[report content]",
folder="$2" or "research",
tags=["research", ...],
project="main"
)
```
### 6. Present Summary
After saving, present:
- Key findings summary
- Main recommendation (if applicable)
- Where the report was saved
- Offer to dive deeper into any aspect
## Examples
```
/research MCP protocol
/research "database migration patterns"
/research "authentication options" decisions
/research "React vs Vue" architecture
```
+26
View File
@@ -0,0 +1,26 @@
{
"hooks": {
"PostToolUse": [
{
"matcher": "mcp__basic-memory__write_note",
"hooks": [
{
"type": "command",
"command": "echo '✓ Note saved to Basic Memory'"
}
]
}
],
"Stop": [
{
"matcher": "*",
"hooks": [
{
"type": "prompt",
"prompt": "If this conversation contained valuable insights, decisions, or learnings that should be preserved, suggest using `/remember [title]` to capture them in Basic Memory. Only suggest this if there's genuinely valuable content worth preserving - don't suggest for trivial interactions."
}
]
}
]
}
}
@@ -0,0 +1,211 @@
---
name: continue-conversation
description: Resume previous work by building context from Basic Memory knowledge graph using memory URLs and recent activity
---
# Continue Conversation
This skill helps you resume previous work by building context from the Basic Memory knowledge graph, enabling seamless continuation across sessions.
## When to Use
Use this skill when:
- Starting a new session and need to pick up where you left off
- User mentions previous work ("continue with...", "back to...", "where were we with...")
- Need context about ongoing projects or specs
- User asks about something discussed in a previous conversation
- Working on a multi-session task
## Building Context
### 1. Identify What to Continue
Ask if unclear:
- What topic or project to resume?
- What timeframe to look at?
- Any specific aspect to focus on?
### 2. Gather Context with MCP Tools
**Option A: Known Topic - Use build_context**
```python
# Navigate knowledge graph from a known starting point
mcp__basic-memory__build_context(
url="memory://topic-or-note-name",
depth=2, # How many relation hops to follow
timeframe="7d", # Recent changes
project="main" # or "specs" for specifications
)
```
Memory URL formats:
- `memory://note-title` - Single note
- `memory://folder/*` - All notes in folder
- `memory://specs/SPEC-24*` - Pattern matching
**Option B: Recent Activity - What's been happening?**
```python
# See what's changed recently
mcp__basic-memory__recent_activity(
timeframe="3d", # "1d", "1 week", "2 weeks"
depth=1,
project="main"
)
```
**Option C: Search for Context**
```python
# Find relevant notes
mcp__basic-memory__search_notes(
query="search terms",
page_size=10,
project="main"
)
```
### 3. Read Key Notes
Once you identify relevant notes:
```python
mcp__basic-memory__read_note(
identifier="note-title-or-permalink",
project="main"
)
```
### 4. Present Context to User
Summarize what you found:
- Current state of the work
- Recent changes or progress
- Open items or next steps
- Related context that might be helpful
## Context Strategies by Scenario
### Resuming a Spec Implementation
```python
# 1. Read the spec
mcp__basic-memory__read_note(
identifier="SPEC-24: Postgres Database Migration",
project="specs"
)
# 2. Check recent activity on related topics
mcp__basic-memory__build_context(
url="memory://SPEC-24*",
timeframe="7d",
project="specs"
)
# 3. Look at what's been done in the codebase
# (Use regular file tools for this)
```
### Continuing General Work
```python
# 1. Check recent activity across projects
mcp__basic-memory__recent_activity(
timeframe="3d",
project="main"
)
# 2. Read any notes from recent sessions
mcp__basic-memory__read_note(
identifier="relevant-note",
project="main"
)
```
### Following Up on a Topic
```python
# 1. Search for the topic
mcp__basic-memory__search_notes(
query="topic keywords",
project="main"
)
# 2. Build context from best match
mcp__basic-memory__build_context(
url="memory://found-note-permalink",
depth=2,
project="main"
)
```
## Timeframe Reference
Natural language timeframes:
- `"today"` - Current day
- `"yesterday"` - Previous day
- `"3d"` or `"3 days"` - Last 3 days
- `"1 week"` or `"7d"` - Last week
- `"2 weeks"` - Last 2 weeks
- `"1 month"` - Last month
## Project Reference
Common projects:
- `main` - Primary knowledge base
- `specs` - Specifications and design docs
- `basic-memory-llc` - Business/company notes
- `getting-started` - Tutorial content
List available projects:
```python
mcp__basic-memory__list_memory_projects()
```
## Example Conversations
### User: "Let's continue with the Postgres migration"
```
1. Read SPEC-24 from specs project
2. Check for related notes about implementation progress
3. Summarize:
- Spec overview and goals
- What's been completed (checkmarks)
- What's pending (checkboxes)
- Any blockers or decisions needed
```
### User: "What was I working on yesterday?"
```
1. Get recent activity for last 2 days
2. List modified notes with brief descriptions
3. Ask which topic to dive into
```
### User: "Back to the async client pattern"
```
1. Search for "async client pattern"
2. Build context from matching note
3. Include related notes via relations
4. Present the full picture
```
## Best Practices
1. **Start broad, then narrow** - Get overview first, then specific details
2. **Follow relations** - Knowledge graph connections are valuable
3. **Check multiple projects** - Specs might be separate from implementation notes
4. **Present incrementally** - Share what you find as you go
5. **Confirm understanding** - Verify the context is what user needs
6. **Update as you go** - Capture new progress in notes during the session
## Combining with Other Skills
After building context, you might:
- Use **knowledge-capture** to document new progress
- Use **spec-driven-development** if continuing a spec implementation
- Create new notes linking to the context you gathered
@@ -0,0 +1,261 @@
---
name: edit-note-local
description: Edit Basic Memory notes directly as local files - enables full file editing with automatic sync (local installations only)
---
# Edit Note Local
This skill enables direct file-based editing of Basic Memory notes. It works by editing the actual markdown files in the knowledge base, which Basic Memory's sync service automatically picks up. This provides a more seamless editing experience for local installations.
## When to Use
Use this skill when:
- User has a local Basic Memory installation (not cloud-only)
- User wants to make substantial edits to a note
- User prefers working with the full file content
- User wants changes to sync automatically via `basic-memory sync --watch`
**Note:** This skill requires local file access. For cloud-only users, use the `edit-note` skill instead.
## Editing Workflow
### 1. Find the Note's File Path
First, get the note metadata to find its file location:
```python
# Search for the note
mcp__basic-memory__search_notes(
query="note title or keywords",
project="main"
)
# Read the note to get file_path from metadata
mcp__basic-memory__read_note(
identifier="note-title",
project="main"
)
```
The response includes `file_path` which gives the relative path within the knowledge base.
### 2. Determine Full File Path
Basic Memory projects have a root directory. Common locations:
- Default: `~/basic-memory/`
- Custom: Check project configuration
Construct the full path:
```
{project_root}/{file_path}
```
For example:
- Project root: `/Users/username/basic-memory`
- File path from note: `notes/My Note.md`
- Full path: `/Users/username/basic-memory/notes/My Note.md`
### 3. Read the File
Use Claude Code's Read tool to get the full file content:
```python
Read(file_path="/Users/username/basic-memory/notes/My Note.md")
```
Display the content to the user, explaining the structure:
- Frontmatter (YAML between `---` markers)
- Main content
- Observations section
- Relations section
### 4. Edit the File
Use Claude Code's Edit tool for precise changes:
```python
Edit(
file_path="/Users/username/basic-memory/notes/My Note.md",
old_string="text to replace",
new_string="new text"
)
```
Or use Write for complete rewrites:
```python
Write(
file_path="/Users/username/basic-memory/notes/My Note.md",
content="Complete new file content..."
)
```
### 5. Sync Happens Automatically
If the user has `basic-memory sync --watch` running, changes are picked up automatically. Otherwise, they can run:
```bash
basic-memory sync
```
## File Structure Reference
Basic Memory notes follow this markdown structure:
```markdown
---
title: Note Title
type: note
permalink: note-title
tags:
- tag1
- tag2
---
# Note Title
## Context
Background and situation explanation.
## Main Content
The primary content of the note...
## Observations
- [category] Observation text #optional-tag
- [decision] A decision that was made #tag
- [insight] An insight discovered
## Relations
- relates-to [[Other Note]]
- implements [[Parent Concept]]
- learned-from [[Source Note]]
```
## Editing Patterns
### Edit Frontmatter Tags
```python
Edit(
file_path="/path/to/note.md",
old_string="tags:\n- old-tag",
new_string="tags:\n- old-tag\n- new-tag"
)
```
### Add New Section
```python
Edit(
file_path="/path/to/note.md",
old_string="## Observations",
new_string="## New Section\n\nNew content here.\n\n## Observations"
)
```
### Modify Observation
```python
Edit(
file_path="/path/to/note.md",
old_string="- [decision] Old decision",
new_string="- [decision] Updated decision with new info #updated"
)
```
### Add Relation
```python
Edit(
file_path="/path/to/note.md",
old_string="## Relations\n",
new_string="## Relations\n\n- relates-to [[New Related Note]]\n"
)
```
### Complete Rewrite
For major changes, read the file, construct new content preserving the frontmatter structure, and write:
```python
Write(
file_path="/path/to/note.md",
content="""---
title: Note Title
type: note
permalink: note-title
tags:
- updated
---
# Note Title
Completely rewritten content...
## Observations
- [rewrite] Complete rewrite of this note #major-update
## Relations
- updates [[Previous Version]]
"""
)
```
## Finding the Project Root
To find where Basic Memory stores files, you can:
1. **Check common locations:**
- `~/basic-memory/`
- `~/Documents/basic-memory/`
- Current working directory
2. **Use the list_directory tool:**
```python
mcp__basic-memory__list_directory(
dir_name="/",
project="main"
)
```
3. **Ask the user:** "Where is your Basic Memory knowledge base located?"
## Advantages of Local Editing
1. **Full file access** - Edit any part of the file including frontmatter
2. **Multi-line edits** - Make complex structural changes easily
3. **Batch operations** - Edit multiple files in sequence
4. **Version control** - Changes tracked by git if the folder is a repo
5. **Instant preview** - Use any markdown editor alongside
6. **Auto-sync** - `sync --watch` picks up changes automatically
## Best Practices
1. **Preserve frontmatter** - Don't break the YAML structure
2. **Keep valid markdown** - Maintain proper formatting
3. **Preserve permalinks** - Changing them can break links
4. **Show diffs** - Tell the user what changed
5. **Suggest sync** - Remind about `basic-memory sync` if not watching
6. **Handle missing files** - Check if file exists before editing
## Example Conversation
**User:** "I want to completely rewrite my architecture decision note"
**Claude:**
1. Searches for the note via MCP
2. Gets the file path
3. Reads the current file content
4. Asks: "Here's the current note. What would you like the new version to say?"
**User:** Provides new content
**Claude:**
1. Preserves the frontmatter (title, permalink, type)
2. Writes the new content using Write tool
3. Confirms: "Updated the file at `/path/to/note.md`. If you have sync --watch running, it's already indexed. Otherwise run `basic-memory sync`."
@@ -0,0 +1,209 @@
---
name: edit-note
description: Interactively edit Basic Memory notes using MCP tools - view, modify, and update notes in a conversational workflow (works with cloud and local)
---
# Edit Note
This skill enables interactive editing of Basic Memory notes using MCP tools. It works with both Basic Memory Cloud and local installations since it operates through the MCP interface rather than direct file access.
## When to Use
Use this skill when:
- User wants to edit an existing note
- User asks to update, change, or modify note content
- User wants to refine observations or relations in a note
- User says things like "edit my note about...", "update the...", "change X to Y in..."
## Editing Workflow
### 1. Fetch the Current Note
First, retrieve the note to show the user what exists:
```python
mcp__basic-memory__read_note(
identifier="Note Title or permalink",
project="main" # or specified project
)
```
Present the note content clearly, highlighting:
- Current title and metadata
- Main content sections
- Observations (with categories)
- Relations (with link targets)
### 2. Understand the Edit Request
Ask clarifying questions if needed:
- Which section to modify?
- What specifically to change?
- Add new content or replace existing?
### 3. Apply the Edit
Use the appropriate `edit_note` operation:
**Append** - Add content to the end:
```python
mcp__basic-memory__edit_note(
identifier="note-title",
operation="append",
content="\n\n## New Section\n\nNew content here...",
project="main"
)
```
**Prepend** - Add content to the beginning:
```python
mcp__basic-memory__edit_note(
identifier="note-title",
operation="prepend",
content="# Updated Header\n\n",
project="main"
)
```
**Find and Replace** - Replace specific text:
```python
mcp__basic-memory__edit_note(
identifier="note-title",
operation="find_replace",
find_text="old text to find",
content="new replacement text",
project="main"
)
```
**Replace Section** - Replace an entire section by heading:
```python
mcp__basic-memory__edit_note(
identifier="note-title",
operation="replace_section",
section="## Section Heading",
content="## Section Heading\n\nCompletely new section content...",
project="main"
)
```
### 4. Show the Result
After editing, fetch and display the updated note:
```python
mcp__basic-memory__read_note(
identifier="note-title",
project="main"
)
```
Highlight what changed so the user can verify.
## Edit Operations Reference
| Operation | Use Case | Required Parameters |
|-----------|----------|---------------------|
| `append` | Add to end | `content` |
| `prepend` | Add to beginning | `content` |
| `find_replace` | Change specific text | `find_text`, `content` |
| `replace_section` | Rewrite a section | `section`, `content` |
## Common Edit Patterns
### Adding a New Observation
```python
mcp__basic-memory__edit_note(
identifier="note-title",
operation="find_replace",
find_text="## Observations",
content="## Observations\n\n- [new-category] New observation here #tag",
project="main"
)
```
Or append to observations section:
```python
mcp__basic-memory__edit_note(
identifier="note-title",
operation="append",
content="\n- [insight] Additional insight discovered #tag",
project="main"
)
```
### Adding a New Relation
```python
mcp__basic-memory__edit_note(
identifier="note-title",
operation="find_replace",
find_text="## Relations",
content="## Relations\n\n- relates-to [[New Related Note]]",
project="main"
)
```
### Updating a Specific Observation
```python
mcp__basic-memory__edit_note(
identifier="note-title",
operation="find_replace",
find_text="- [decision] Old decision text",
content="- [decision] Updated decision with new context #updated",
project="main"
)
```
### Rewriting the Context Section
```python
mcp__basic-memory__edit_note(
identifier="note-title",
operation="replace_section",
section="## Context",
content="## Context\n\nCompletely rewritten context explaining the new situation...",
project="main"
)
```
## Multi-Step Editing Session
For complex edits, work iteratively:
1. **Show current state** → Read and display the note
2. **First edit** → Apply one change
3. **Show result** → Display updated note
4. **Next edit** → Apply another change if needed
5. **Confirm complete** → Final display and confirmation
This keeps the user informed and allows course correction.
## Best Practices
1. **Always show before and after** - User should see what changed
2. **One edit at a time** - For complex changes, do multiple operations
3. **Preserve structure** - Maintain the note's markdown format
4. **Be careful with find_replace** - Ensure the find_text is unique
5. **Confirm destructive changes** - Ask before replacing large sections
6. **Keep observations formatted** - Maintain `[category]` prefix format
7. **Keep relations formatted** - Maintain `- relation-type [[Target]]` format
## Example Conversation
**User:** "Edit my note about the async client pattern - add an observation about testing"
**Claude:**
1. Fetches "Async Client Pattern" note
2. Displays current content
3. Asks: "What observation about testing would you like to add?"
**User:** "That the context manager pattern makes mocking easier in tests"
**Claude:**
1. Uses `edit_note` with `append` to add:
`- [testing] Context manager pattern simplifies mocking in unit tests #testability`
2. Fetches and displays updated note
3. Confirms: "Added the testing observation. Here's the updated note..."
@@ -0,0 +1,211 @@
---
name: knowledge-capture
description: Capture insights, decisions, and learnings from conversations into structured Basic Memory notes with observations and relations
---
# Knowledge Capture
This skill helps you capture valuable information from conversations into Basic Memory's knowledge graph using structured notes with observations and relations.
## When to Use
Use this skill when:
- Important decisions are made during a conversation
- Technical insights or patterns are discovered
- Problems are solved and the solution should be preserved
- Design trade-offs are discussed
- Architecture or implementation approaches are chosen
- Learnings from debugging or investigation emerge
## Capture Process
### 1. Identify Valuable Information
Look for:
- **Decisions**: Choices made and their rationale
- **Insights**: New understanding or "aha" moments
- **Patterns**: Reusable approaches or solutions
- **Trade-offs**: Options considered and why one was chosen
- **Learnings**: What worked, what didn't, and why
- **Context**: Background that would help future understanding
### 2. Structure the Note
Use Basic Memory's knowledge format:
```markdown
---
title: Descriptive Title
type: note
tags:
- relevant
- tags
---
# Title
## Context
Brief background explaining the situation.
## Content
Main content organized logically.
## Observations
- [decision] What was decided and why #tag
- [insight] Key understanding gained #tag
- [pattern] Reusable approach identified #tag
- [learning] What we learned from this #tag
- [tradeoff] Option A chosen over B because... #tag
## Relations
- relates-to [[Related Concept]]
- implements [[Parent Spec or Design]]
- learned-from [[Source of Learning]]
```
### 3. Choose Appropriate Categories
Common observation categories:
- `[decision]` - Choices made
- `[insight]` - Understanding gained
- `[pattern]` - Reusable approaches
- `[learning]` - Lessons learned
- `[tradeoff]` - Options weighed
- `[problem]` - Issues identified
- `[solution]` - Fixes applied
- `[architecture]` - Structural decisions
- `[implementation]` - Code-level choices
- `[constraint]` - Limitations discovered
- `[requirement]` - Needs identified
### 4. Create Meaningful Relations
Link to related knowledge:
- `relates-to` - General association
- `implements` - Realizes a spec or design
- `extends` - Builds upon existing concept
- `learned-from` - Source of insight
- `enables` - Makes something possible
- `depends-on` - Requires another concept
- `solves` - Addresses a problem
## MCP Tools to Use
```python
# Write a new note
mcp__basic-memory__write_note(
title="Your Note Title",
content="Full markdown content...",
folder="appropriate/folder",
tags=["tag1", "tag2"],
project="main" # or appropriate project
)
# Search for related notes to link
mcp__basic-memory__search_notes(
query="relevant terms",
project="main"
)
# Read existing notes for context
mcp__basic-memory__read_note(
identifier="note-title-or-permalink",
project="main"
)
```
## Folder Organization
Choose appropriate folders:
- `decisions/` - Architecture and design decisions
- `learnings/` - Insights and lessons learned
- `patterns/` - Reusable approaches
- `debug-logs/` - Problem investigations
- `conversations/` - Imported conversation summaries
## Examples
### Capturing a Technical Decision
```markdown
---
title: FastAPI Async Client Pattern
type: note
tags:
- architecture
- fastapi
- async
---
# FastAPI Async Client Pattern
## Context
During implementation of MCP tools, we needed to decide how to handle HTTP client lifecycle.
## Decision
Use context manager pattern for HTTP clients instead of module-level singletons.
## Rationale
- Proper resource management
- Supports three deployment modes (local ASGI, CLI cloud, cloud app)
- Auth happens at client creation, not per-request
- Enables dependency injection for testing
## Observations
- [decision] Context manager pattern for HTTP clients enables proper resource cleanup #architecture
- [pattern] Factory pattern allows different client configurations per deployment mode #flexibility
- [tradeoff] Slightly more verbose than singleton but much more flexible #engineering
## Relations
- implements [[SPEC-16 MCP Cloud Service Consolidation]]
- enables [[Cloud App Integration]]
```
### Capturing a Debugging Insight
```markdown
---
title: SQLite WAL Mode Performance Fix
type: note
tags:
- debugging
- sqlite
- performance
---
# SQLite WAL Mode Performance Fix
## Problem
Sync operations were slow with multiple concurrent writes.
## Investigation
Found that default SQLite journaling was causing lock contention.
## Solution
Enabled WAL (Write-Ahead Logging) mode for the database connection.
## Observations
- [problem] Default SQLite journaling causes lock contention under concurrent writes #performance
- [solution] WAL mode significantly improves concurrent write performance #sqlite
- [learning] Always consider WAL mode for SQLite in applications with concurrent access #database
## Relations
- solves [[Sync Performance Issues]]
- relates-to [[SPEC-19 Sync Performance]]
```
## Best Practices
1. **Capture immediately** - Write notes while context is fresh
2. **Be specific** - Include concrete details, not vague summaries
3. **Link liberally** - More relations = better knowledge graph
4. **Use tags** - Enable discovery via search
5. **Include context** - Future you won't remember the situation
6. **Prefer facts over opinions** - Observations should be verifiable
7. **Keep notes atomic** - One concept per note when possible
@@ -0,0 +1,283 @@
---
name: knowledge-organize
description: Help organize, link, and maintain the Basic Memory knowledge graph - find orphan notes, suggest relations, identify duplicates, and improve overall knowledge structure
---
# Knowledge Organize
This skill helps users maintain a healthy, well-connected knowledge graph. As notes accumulate, it becomes valuable to periodically organize, link, and curate the knowledge base.
## When to Use
Use this skill when:
- User asks to organize their notes
- User wants to find connections between notes
- User mentions orphan or unlinked notes
- User wants to clean up or improve their knowledge base
- User asks about duplicate or similar notes
- User wants help with folder organization
- User asks to review or audit their notes
- Phrases like "help me organize", "find related notes", "what's not linked", "clean up my notes"
## Organization Capabilities
### 1. Find Orphan Notes
Identify notes that have no relations to other notes - they're isolated in the knowledge graph.
```python
# Get all notes
mcp__basic-memory__search_notes(
query="*",
page_size=50,
project="main"
)
# For each note, check if it has relations
# Orphans have empty Relations sections
```
**What to do with orphans:**
- Suggest potential relations based on content similarity
- Ask if they should be linked to existing topics
- Propose creating hub notes to connect related orphans
### 2. Suggest Relations
Analyze note content and suggest meaningful connections.
```python
# Read a note
mcp__basic-memory__read_note(
identifier="note-to-analyze",
project="main"
)
# Search for potentially related notes
mcp__basic-memory__search_notes(
query="key terms from the note",
project="main"
)
# Suggest relations based on:
# - Shared topics or concepts
# - Complementary content (problem/solution, question/answer)
# - Sequential relationship (part 1, part 2)
# - Hierarchical (parent concept, child detail)
```
**Relation types to suggest:**
- `relates-to` - General topical connection
- `extends` - Builds upon or expands
- `implements` - Realizes a concept
- `depends-on` - Requires understanding of
- `contradicts` - Presents alternative view
- `learned-from` - Source of insight
- `enables` - Makes something possible
### 3. Identify Similar/Duplicate Notes
Find notes that may cover the same topic.
```python
# Search for notes with similar titles or content
mcp__basic-memory__search_notes(
query="topic keywords",
project="main"
)
# Compare results for overlap
# Look for:
# - Similar titles
# - Overlapping observations
# - Same tags
# - Related timestamps (created around same time)
```
**Actions for duplicates:**
- Merge into a single comprehensive note
- Link them with `supersedes` or `updates` relations
- Differentiate by adding context about their distinct focus
### 4. Folder Organization Review
Analyze folder structure and suggest improvements.
```python
# List directory structure
mcp__basic-memory__list_directory(
dir_name="/",
depth=3,
project="main"
)
# Identify:
# - Overcrowded folders
# - Single-note folders
# - Inconsistent naming
# - Notes that might belong elsewhere
```
**Organization suggestions:**
- Group related notes into topic folders
- Create subfolders for large categories
- Suggest consistent naming conventions
- Move misplaced notes
### 5. Tag Consistency
Review and normalize tags across notes.
```python
# Search notes to analyze tag patterns
mcp__basic-memory__search_notes(
query="*",
page_size=100,
project="main"
)
# Look for:
# - Similar tags (architecture vs arch)
# - Unused tags
# - Over-used generic tags
# - Missing tags on relevant notes
```
**Tag improvements:**
- Suggest tag standardization (pick one variant)
- Propose new tags for common themes
- Identify notes missing obvious tags
### 6. Create Index/Hub Notes
Generate notes that serve as navigation hubs for related topics.
```python
# After identifying a cluster of related notes
mcp__basic-memory__write_note(
title="Architecture Decisions Index",
content="""---
title: Architecture Decisions Index
type: index
tags:
- architecture
- index
---
# Architecture Decisions Index
A hub linking all architecture-related decisions and patterns.
## Decisions
- [[Database Selection Decision]]
- [[API Design Patterns]]
- [[Authentication Architecture]]
## Patterns
- [[Repository Pattern]]
- [[Async Client Pattern]]
## Observations
- [index] Central hub for architecture knowledge #navigation
## Relations
- indexes [[Architecture]]
""",
folder="indexes",
project="main"
)
```
### 7. Enrich Sparse Notes
Find notes lacking observations or structure and suggest improvements.
```python
# Read a sparse note
mcp__basic-memory__read_note(
identifier="sparse-note",
project="main"
)
# If missing:
# - Observations section → suggest categories
# - Relations section → suggest links
# - Tags → suggest relevant tags
# - Context → suggest adding background
```
## Organization Workflows
### Quick Health Check
A fast overview of knowledge base status:
1. Count total notes
2. Identify orphan count
3. List recently modified
4. Check for obvious duplicates
5. Report folder distribution
### Deep Organization Session
Thorough review and improvement:
1. **Audit phase** - Catalog all notes, identify issues
2. **Orphan phase** - Address unlinked notes
3. **Relation phase** - Suggest new connections
4. **Duplicate phase** - Merge or differentiate similar notes
5. **Structure phase** - Reorganize folders if needed
6. **Index phase** - Create hub notes for major topics
### Topic-Focused Organization
Organize around a specific subject:
1. Find all notes related to topic
2. Map existing relations
3. Identify gaps in the topic graph
4. Suggest new notes to fill gaps
5. Create topic index note
## Best Practices
1. **Work incrementally** - Don't reorganize everything at once
2. **Confirm before changing** - Always ask before moving/editing notes
3. **Preserve permalinks** - Moving is okay, changing permalinks breaks links
4. **Show the graph** - Help user visualize connections
5. **Explain suggestions** - Say why a relation makes sense
6. **Respect user's system** - Enhance their organization, don't impose a new one
## Example Conversations
**User:** "Help me organize my notes"
**Claude:**
1. Runs health check on the knowledge base
2. Reports: "You have 47 notes. I found 12 orphan notes and 3 potential duplicates."
3. Asks: "Would you like to start by connecting the orphan notes, or review the duplicates first?"
**User:** "Find notes that should be linked to my API design note"
**Claude:**
1. Reads the API design note
2. Searches for related content
3. Suggests: "I found 5 notes that could relate:
- 'REST Best Practices' → relates-to
- 'Authentication Flow' → implements
- 'Rate Limiting Decision' → extends
Would you like me to add any of these relations?"
**User:** "Are there any notes about similar topics?"
**Claude:**
1. Analyzes note titles and content
2. Identifies clusters of similar notes
3. Reports: "I found these potential overlaps:
- 'Auth Flow' and 'Authentication Design' cover similar ground
- 'DB Schema v1' and 'DB Schema v2' might need a 'supersedes' relation
Would you like to review any of these?"
+213
View File
@@ -0,0 +1,213 @@
---
name: research
description: Research a topic thoroughly and produce a structured report saved to Basic Memory - investigate concepts, gather context, and document findings
---
# Research
This skill helps conduct thorough research on a topic and produces a structured report that gets saved to Basic Memory for future reference.
## When to Use
Use this skill when:
- User asks to research or investigate something
- User wants to understand a concept, technology, or approach
- User needs context gathered before making a decision
- User asks "what is...", "how does... work", "explore...", "investigate..."
- User wants findings documented for later
- Phrases like "research this", "look into", "find out about", "explore options for"
## Research Process
### 1. Understand the Research Question
Clarify what specifically to investigate:
- What is the core question or topic?
- What scope - broad overview or deep dive?
- Any specific aspects to focus on?
- What will the research inform (a decision, implementation, understanding)?
### 2. Gather Information
Use available tools to collect information:
**For codebase research:**
- Search the codebase for relevant code
- Read documentation and comments
- Trace how things connect
- Look at tests for usage examples
**For concept research:**
- Use web search for current information
- Fetch documentation from official sources
- Look for examples and best practices
- Compare alternatives if relevant
**For Basic Memory context:**
```python
# Check what we already know
mcp__basic-memory__search_notes(
query="topic keywords",
project="main"
)
# Build context from related notes
mcp__basic-memory__build_context(
url="memory://related-topic",
depth=2,
project="main"
)
```
### 3. Analyze and Synthesize
Organize findings into coherent insights:
- Identify key concepts and how they relate
- Note patterns, trade-offs, and considerations
- Highlight what's most relevant to the user's needs
- Flag uncertainties or areas needing more investigation
### 4. Produce the Report
Create a structured research report:
```markdown
---
title: "Research: [Topic]"
type: research
tags:
- research
- [topic-tags]
---
# Research: [Topic]
## Summary
[2-3 sentence executive summary of findings]
## Research Question
[What we set out to understand]
## Key Findings
### [Finding 1]
[Details, evidence, implications]
### [Finding 2]
[Details, evidence, implications]
### [Finding 3]
[Details, evidence, implications]
## Analysis
[Synthesis of findings - patterns, trade-offs, recommendations]
## Open Questions
- [Things that need more investigation]
- [Uncertainties or assumptions]
## Sources
- [Where information came from]
- [[Related Note]] - relevant prior knowledge
## Observations
- [finding] Key insight discovered #research
- [pattern] Pattern identified during research
- [recommendation] Suggested approach based on findings
## Relations
- researches [[Topic]]
- informs [[Decision or Implementation]]
- relates-to [[Related Concepts]]
```
### 5. Save to Basic Memory
```python
mcp__basic-memory__write_note(
title="Research: [Topic]",
content="[Full report content]",
folder="research",
tags=["research", "topic-tags"],
project="main"
)
```
## Report Styles
Adjust based on the research type:
### Quick Investigation
- Focused summary
- 2-3 key findings
- Direct recommendation
- Saved to `research/` folder
### Deep Dive
- Comprehensive analysis
- Multiple sections
- Detailed evidence
- Comparison of options
- Saved to `research/` folder
### Decision Support
- Options evaluated
- Pros/cons for each
- Clear recommendation with rationale
- Saved to `decisions/` or `research/` folder
### Technical Exploration
- How it works
- Architecture/design
- Code examples
- Integration considerations
- Saved to `research/` folder
## Best Practices
1. **Start with what we know** - Check Basic Memory for existing context
2. **Be thorough but focused** - Cover the topic well without tangents
3. **Cite sources** - Link to where information came from
4. **Be honest about uncertainty** - Flag what's unclear or needs verification
5. **Make it actionable** - Include recommendations when appropriate
6. **Link to related knowledge** - Connect to existing notes
7. **Save for future reference** - Always save the report to Basic Memory
## Example Conversations
**User:** "Research how other projects handle database migrations"
**Claude:**
1. Searches codebase for migration patterns
2. Checks Basic Memory for related decisions
3. Looks up best practices online
4. Produces report comparing approaches
5. Saves to `research/Database Migration Approaches.md`
6. Presents summary with recommendation
**User:** "Investigate the MCP protocol"
**Claude:**
1. Fetches MCP documentation
2. Searches for examples in codebase
3. Checks Basic Memory for prior context
4. Produces comprehensive report on MCP
5. Saves to `research/MCP Protocol Overview.md`
6. Presents key concepts and how to use them
**User:** "Look into authentication options for the API"
**Claude:**
1. Researches common auth patterns (JWT, OAuth, API keys)
2. Checks existing codebase auth implementation
3. Evaluates trade-offs for the use case
4. Produces decision-support report
5. Saves to `research/API Authentication Options.md`
6. Recommends approach with rationale
@@ -0,0 +1,292 @@
---
name: spec-driven-development
description: Guide implementation based on specs stored in Basic Memory, following the SPEC-1 specification-driven development process
---
# Spec-Driven Development
This skill guides implementation work based on specifications stored in the Basic Memory "specs" project, following the process defined in SPEC-1.
## When to Use
Use this skill when:
- Implementing a feature defined by a spec
- Creating a new specification before implementation
- Reviewing implementation against spec criteria
- Need to understand what a spec requires
- Updating spec progress as work completes
## The Spec-Driven Process
From SPEC-1, the workflow is:
1. **Create** - Write spec as complete thought in Basic Memory "specs" project
2. **Discuss** - Iterate and refine the specification
3. **Implement** - Execute implementation directly
4. **Validate** - Review implementation against spec criteria
5. **Document** - Update spec with learnings and decisions
## Spec Structure
Every spec contains:
- **Why** - The reasoning and problem being solved
- **What** - What is affected or changed
- **How** - High-level approach to implementation
- **How to Evaluate** - Testing/validation procedure
### Progress Tracking Format
Specs use living documentation with checklists:
```markdown
### Feature Area
- ✅ Basic functionality implemented
- ✅ Props and events defined
- [ ] Add sorting controls
- [ ] Improve accessibility
- [x] Currently implementing responsive design
```
- `✅` - Completed items
- `[ ]` - Pending items
- `[x]` - In-progress items
## Working with Specs
### Reading a Spec
```python
# Get the full spec
mcp__basic-memory__read_note(
identifier="SPEC-24: Postgres Database Migration",
project="specs"
)
# Or search for it
mcp__basic-memory__search_notes(
query="postgres migration",
project="specs"
)
```
### Creating a New Spec
```python
# 1. First, find the next spec number
mcp__basic-memory__search_notes(
query="SPEC-",
project="specs"
)
# 2. Create the spec with proper structure
mcp__basic-memory__write_note(
title="SPEC-30: Your Feature Name",
content="""---
title: 'SPEC-30: Your Feature Name'
type: spec
tags:
- feature-area
- component
---
# SPEC-30: Your Feature Name
## Why
[Problem statement and motivation]
## What
[What is affected or changed]
- Affected areas
- Components involved
- Scope boundaries
## How (High Level)
[Implementation approach]
### Phase 1: Foundation
- [ ] Task 1
- [ ] Task 2
### Phase 2: Core Features
- [ ] Task 3
- [ ] Task 4
## How to Evaluate
### Success Criteria
- [ ] Criterion 1
- [ ] Criterion 2
### Testing Procedure
1. Step 1
2. Step 2
## Observations
- [goal] Primary objective #tag
- [constraint] Known limitation #tag
## Relations
- relates-to [[Related Spec]]
- depends-on [[Dependency]]
""",
folder="", # Root of specs project
project="specs"
)
```
### Updating Spec Progress
```python
# Mark items complete as you implement
mcp__basic-memory__edit_note(
identifier="SPEC-24: Postgres Database Migration",
operation="find_replace",
find_text="- [ ] Create migration scripts",
content="- ✅ Create migration scripts",
project="specs"
)
# Or add new observations
mcp__basic-memory__edit_note(
identifier="SPEC-24: Postgres Database Migration",
operation="append",
content="\n- [learning] Alembic autogenerate works well for model changes #migration",
project="specs"
)
```
### Reviewing Implementation
When reviewing against a spec:
1. **Read the spec's "How to Evaluate" section**
2. **Check each success criterion:**
- Functional completeness
- Test coverage (count test files, check categories)
- Code quality (TypeScript, linting, performance)
- Architecture compliance
- Documentation completeness
3. **Be honest** - Don't overstate completeness
4. **Document findings** - Update spec with review results
5. **Identify gaps** - Clearly note what still needs work
## Implementation Workflow
### Starting Implementation
1. **Read the spec thoroughly**
```python
mcp__basic-memory__read_note(
identifier="SPEC-XX: Feature Name",
project="specs"
)
```
2. **Understand dependencies**
- Check Relations section for dependencies
- Read related specs if needed
3. **Plan your approach**
- Break "How" section into concrete tasks
- Identify what to implement first
4. **Mark first item in-progress**
```python
mcp__basic-memory__edit_note(
identifier="SPEC-XX",
operation="find_replace",
find_text="- [ ] First task",
content="- [x] First task",
project="specs"
)
```
### During Implementation
1. **Update progress as you complete items**
2. **Add observations for decisions made**
3. **Note any deviations from the spec**
4. **Capture learnings that might help future specs**
### After Implementation
1. **Run full evaluation against criteria**
2. **Mark all completed items with ✅**
3. **Add final observations**
4. **Document any follow-up work needed**
## Spec Naming Convention
Format: `SPEC-X: Descriptive Title`
Examples:
- `SPEC-24: Postgres Database Migration`
- `SPEC-25: Cloud Index Service`
- `SPEC-26: Multi-User Security and Permissions`
## Common Spec Patterns
### Feature Spec
```markdown
## Why
User need or problem
## What
- New UI components
- API endpoints
- Database changes
## How
Implementation phases with checkboxes
```
### Architecture Spec
```markdown
## Why
Technical debt or scalability need
## What
- System components affected
- Data flow changes
- Integration points
## How
Migration strategy with rollback plan
```
### Process Spec
```markdown
## Why
Workflow improvement need
## What
- Process steps changed
- Tools involved
- Team impact
## How
Rollout plan and adoption strategy
```
## Best Practices
1. **Spec first, code second** - Write spec before implementation
2. **Keep specs living** - Update as understanding evolves
3. **Be specific in criteria** - Vague criteria = vague completion
4. **Link related specs** - Build the knowledge graph
5. **Capture decisions** - Future you will thank you
6. **Review honestly** - Incomplete is okay, dishonest isn't
7. **Close the loop** - Mark items done as you complete them
## Using with Slash Commands
The `/spec` command provides quick access:
- `/spec create [name]` - Create new specification
- `/spec status` - Show all spec statuses
- `/spec show [name]` - Read a specific spec
- `/spec review [name]` - Validate implementation
+29 -45
View File
@@ -7,60 +7,44 @@ install:
@echo ""
@echo "💡 Remember to activate the virtual environment by running: source .venv/bin/activate"
# Run all tests with unified coverage report
test: test-unit test-int
# Run unit tests only (fast, no coverage)
test-unit:
uv run pytest -p pytest_mock -v --no-cov tests
# Run integration tests only (fast, no coverage)
test-int:
uv run pytest -p pytest_mock -v --no-cov test-int
# ==============================================================================
# DATABASE BACKEND TESTING
# ==============================================================================
# Basic Memory supports dual database backends (SQLite and Postgres).
# By default, tests run against SQLite (fast, no dependencies).
# Set BASIC_MEMORY_TEST_POSTGRES=1 to run against Postgres (uses testcontainers).
# Tests are parametrized to run against both backends automatically.
#
# Quick Start:
# just test # Run all tests against SQLite (default)
# just test-sqlite # Run all tests against SQLite
# just test-postgres # Run all tests against Postgres (testcontainers)
# just test-unit-sqlite # Run unit tests against SQLite
# just test-unit-postgres # Run unit tests against Postgres
# just test-int-sqlite # Run integration tests against SQLite
# just test-int-postgres # Run integration tests against Postgres
# just test-sqlite # Run SQLite tests (default, no Docker needed)
# just test-postgres # Run Postgres tests (requires Docker)
#
# CI runs both in parallel for faster feedback.
# For Postgres tests, first start the database:
# docker-compose -f docker-compose-postgres.yml up -d
# ==============================================================================
# Run all tests against SQLite and Postgres
test: test-sqlite test-postgres
# Run tests against SQLite only (default backend, skip Postgres/Benchmark tests)
# This is the fastest option and doesn't require any Docker setup.
# Use this for local development and quick feedback.
# Includes Windows-specific tests which will auto-skip on non-Windows platforms.
test-sqlite:
uv run pytest -p pytest_mock -v --no-cov -m "not postgres and not benchmark" tests test-int
# Run all tests against SQLite
test-sqlite: test-unit-sqlite test-int-sqlite
# Run all tests against Postgres (uses testcontainers)
test-postgres: test-unit-postgres test-int-postgres
# Run unit tests against SQLite
test-unit-sqlite:
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov tests
# Run unit tests against Postgres
test-unit-postgres:
BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov tests
# Run integration tests against SQLite
test-int-sqlite:
uv run pytest -p pytest_mock -v --no-cov test-int
# Run integration tests against Postgres
# Note: Uses timeout due to FastMCP Client + asyncpg cleanup hang (tests pass, process hangs on exit)
# See: https://github.com/jlowin/fastmcp/issues/1311
test-int-postgres:
#!/usr/bin/env bash
set -euo pipefail
# Use gtimeout (macOS/Homebrew) or timeout (Linux)
TIMEOUT_CMD=$(command -v gtimeout || command -v timeout || echo "")
if [[ -n "$TIMEOUT_CMD" ]]; then
$TIMEOUT_CMD --signal=KILL 600 bash -c 'BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov test-int' || test $? -eq 137
else
echo "⚠️ No timeout command found, running without timeout..."
BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov test-int
fi
# Run tests against Postgres only (requires docker-compose-postgres.yml up)
# First start Postgres: docker-compose -f docker-compose-postgres.yml up -d
# Tests will connect to localhost:5433/basic_memory_test
# To reset the database: just postgres-reset
test-postgres:
uv run pytest -p pytest_mock -v --no-cov -m "postgres and not benchmark" tests test-int
# Reset Postgres test database (drops and recreates schema)
# Useful when Alembic migration state gets out of sync during development
@@ -75,7 +59,7 @@ postgres-reset:
postgres-migrate:
@cd src/basic_memory/alembic && \
BASIC_MEMORY_DATABASE_BACKEND=postgres \
BASIC_MEMORY_DATABASE_URL=${POSTGRES_TEST_URL:-postgresql+asyncpg://basic_memory_user:dev_password@localhost:5433/basic_memory_test} \
BASIC_MEMORY_DATABASE_URL=${POSTGRES_TEST_URL:-postgresql://basic_memory_user:dev_password@localhost:5433/basic_memory_test} \
uv run alembic upgrade head
@echo "✅ Migrations applied to Postgres test database"
+4 -10
View File
@@ -29,19 +29,14 @@ dependencies = [
"alembic>=1.14.1",
"pillow>=11.1.0",
"pybars3>=0.9.7",
"fastmcp==2.12.3", # Pinned - 2.14.x breaks MCP tools visibility (issue #463)
"fastmcp>=2.10.2",
"pyjwt>=2.10.1",
"python-dotenv>=1.1.0",
"pytest-aio>=1.9.0",
"aiofiles>=24.1.0", # Optional observability (disabled by default via config)
"aiofiles>=24.1.0", # Async file I/O
"logfire>=0.73.0", # Optional observability (disabled by default via config)
"asyncpg>=0.30.0",
"nest-asyncio>=1.6.0", # For Alembic migrations with Postgres
"pytest-asyncio>=1.2.0",
"psycopg==3.3.1",
"mdformat>=0.7.22",
"mdformat-gfm>=0.3.7",
"mdformat-frontmatter>=2.0.8",
"openpanel>=0.0.1", # Anonymous usage telemetry (Homebrew-style opt-out)
]
@@ -86,8 +81,7 @@ dev = [
"pytest-xdist>=3.0.0",
"ruff>=0.1.6",
"freezegun>=1.5.5",
"testcontainers[postgres]>=4.0.0",
"psycopg>=3.2.0",
]
[tool.hatch.version]
+1 -1
View File
@@ -1,7 +1,7 @@
"""basic-memory - Local-first knowledge management combining Zettelkasten with knowledge graphs"""
# Package version - updated by release automation
__version__ = "0.17.0"
__version__ = "0.16.2"
# API version for FastAPI - independent of package version
__api_version__ = "v0"
@@ -1,56 +0,0 @@
"""Add cascade delete FK from search_index to entity
Revision ID: a2b3c4d5e6f7
Revises: f8a9b2c3d4e5
Create Date: 2025-12-02 07:00:00.000000
"""
from typing import Sequence, Union
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "a2b3c4d5e6f7"
down_revision: Union[str, None] = "f8a9b2c3d4e5"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Add FK with CASCADE delete from search_index.entity_id to entity.id.
This migration is Postgres-only because:
- SQLite uses FTS5 virtual tables which don't support foreign keys
- The FK enables automatic cleanup of search_index entries when entities are deleted
"""
connection = op.get_bind()
dialect = connection.dialect.name
if dialect == "postgresql":
# First, clean up any orphaned search_index entries where entity no longer exists
op.execute("""
DELETE FROM search_index
WHERE entity_id IS NOT NULL
AND entity_id NOT IN (SELECT id FROM entity)
""")
# Add FK with CASCADE - nullable FK allows search_index entries without entity_id
op.create_foreign_key(
"fk_search_index_entity_id",
"search_index",
"entity",
["entity_id"],
["id"],
ondelete="CASCADE",
)
def downgrade() -> None:
"""Remove the FK constraint."""
connection = op.get_bind()
dialect = connection.dialect.name
if dialect == "postgresql":
op.drop_constraint("fk_search_index_entity_id", "search_index", type_="foreignkey")
@@ -1,239 +0,0 @@
"""Add project_id to relation/observation and pg_trgm for fuzzy link resolution
Revision ID: f8a9b2c3d4e5
Revises: 314f1ea54dc4
Create Date: 2025-12-01 12:00:00.000000
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy import text
def column_exists(connection, table: str, column: str) -> bool:
"""Check if a column exists in a table (idempotent migration support)."""
if connection.dialect.name == "postgresql":
result = connection.execute(
text(
"SELECT 1 FROM information_schema.columns "
"WHERE table_name = :table AND column_name = :column"
),
{"table": table, "column": column},
)
return result.fetchone() is not None
else:
# SQLite
result = connection.execute(text(f"PRAGMA table_info({table})"))
columns = [row[1] for row in result]
return column in columns
def index_exists(connection, index_name: str) -> bool:
"""Check if an index exists (idempotent migration support)."""
if connection.dialect.name == "postgresql":
result = connection.execute(
text("SELECT 1 FROM pg_indexes WHERE indexname = :index_name"),
{"index_name": index_name},
)
return result.fetchone() is not None
else:
# SQLite
result = connection.execute(
text("SELECT 1 FROM sqlite_master WHERE type='index' AND name = :index_name"),
{"index_name": index_name},
)
return result.fetchone() is not None
# revision identifiers, used by Alembic.
revision: str = "f8a9b2c3d4e5"
down_revision: Union[str, None] = "314f1ea54dc4"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Add project_id to relation and observation tables, plus pg_trgm indexes.
This migration:
1. Adds project_id column to relation and observation tables (denormalization)
2. Backfills project_id from the associated entity
3. Enables pg_trgm extension for trigram-based fuzzy matching (Postgres only)
4. Creates GIN indexes on entity title and permalink for fast similarity searches
5. Creates partial index on unresolved relations for efficient bulk resolution
"""
connection = op.get_bind()
dialect = connection.dialect.name
# -------------------------------------------------------------------------
# Add project_id to relation table
# -------------------------------------------------------------------------
# Step 1: Add project_id column as nullable first (idempotent)
if not column_exists(connection, "relation", "project_id"):
op.add_column("relation", sa.Column("project_id", sa.Integer(), nullable=True))
# Step 2: Backfill project_id from entity.project_id via from_id
if dialect == "postgresql":
op.execute("""
UPDATE relation
SET project_id = entity.project_id
FROM entity
WHERE relation.from_id = entity.id
""")
else:
# SQLite syntax
op.execute("""
UPDATE relation
SET project_id = (
SELECT entity.project_id
FROM entity
WHERE entity.id = relation.from_id
)
""")
# Step 3: Make project_id NOT NULL and add foreign key
if dialect == "postgresql":
op.alter_column("relation", "project_id", nullable=False)
op.create_foreign_key(
"fk_relation_project_id",
"relation",
"project",
["project_id"],
["id"],
)
else:
# SQLite requires batch operations for ALTER COLUMN
with op.batch_alter_table("relation") as batch_op:
batch_op.alter_column("project_id", nullable=False)
batch_op.create_foreign_key(
"fk_relation_project_id",
"project",
["project_id"],
["id"],
)
# Step 4: Create index on relation.project_id (idempotent)
if not index_exists(connection, "ix_relation_project_id"):
op.create_index("ix_relation_project_id", "relation", ["project_id"])
# -------------------------------------------------------------------------
# Add project_id to observation table
# -------------------------------------------------------------------------
# Step 1: Add project_id column as nullable first (idempotent)
if not column_exists(connection, "observation", "project_id"):
op.add_column("observation", sa.Column("project_id", sa.Integer(), nullable=True))
# Step 2: Backfill project_id from entity.project_id via entity_id
if dialect == "postgresql":
op.execute("""
UPDATE observation
SET project_id = entity.project_id
FROM entity
WHERE observation.entity_id = entity.id
""")
else:
# SQLite syntax
op.execute("""
UPDATE observation
SET project_id = (
SELECT entity.project_id
FROM entity
WHERE entity.id = observation.entity_id
)
""")
# Step 3: Make project_id NOT NULL and add foreign key
if dialect == "postgresql":
op.alter_column("observation", "project_id", nullable=False)
op.create_foreign_key(
"fk_observation_project_id",
"observation",
"project",
["project_id"],
["id"],
)
else:
# SQLite requires batch operations for ALTER COLUMN
with op.batch_alter_table("observation") as batch_op:
batch_op.alter_column("project_id", nullable=False)
batch_op.create_foreign_key(
"fk_observation_project_id",
"project",
["project_id"],
["id"],
)
# Step 4: Create index on observation.project_id (idempotent)
if not index_exists(connection, "ix_observation_project_id"):
op.create_index("ix_observation_project_id", "observation", ["project_id"])
# Postgres-specific: pg_trgm and GIN indexes
if dialect == "postgresql":
# Enable pg_trgm extension for fuzzy string matching
op.execute("CREATE EXTENSION IF NOT EXISTS pg_trgm")
# Create trigram indexes on entity table for fuzzy matching
# GIN indexes with gin_trgm_ops support similarity searches
op.execute("""
CREATE INDEX IF NOT EXISTS idx_entity_title_trgm
ON entity USING gin (title gin_trgm_ops)
""")
op.execute("""
CREATE INDEX IF NOT EXISTS idx_entity_permalink_trgm
ON entity USING gin (permalink gin_trgm_ops)
""")
# Create partial index on unresolved relations for efficient bulk resolution
# This makes "WHERE to_id IS NULL AND project_id = X" queries very fast
op.execute("""
CREATE INDEX IF NOT EXISTS idx_relation_unresolved
ON relation (project_id, to_name)
WHERE to_id IS NULL
""")
# Create index on relation.to_name for join performance in bulk resolution
op.execute("""
CREATE INDEX IF NOT EXISTS idx_relation_to_name
ON relation (to_name)
""")
def downgrade() -> None:
"""Remove project_id from relation/observation and pg_trgm indexes."""
connection = op.get_bind()
dialect = connection.dialect.name
if dialect == "postgresql":
# Drop Postgres-specific indexes
op.execute("DROP INDEX IF EXISTS idx_relation_to_name")
op.execute("DROP INDEX IF EXISTS idx_relation_unresolved")
op.execute("DROP INDEX IF EXISTS idx_entity_permalink_trgm")
op.execute("DROP INDEX IF EXISTS idx_entity_title_trgm")
# Note: We don't drop the pg_trgm extension as other code may depend on it
# Drop project_id from observation
op.drop_index("ix_observation_project_id", table_name="observation")
op.drop_constraint("fk_observation_project_id", "observation", type_="foreignkey")
op.drop_column("observation", "project_id")
# Drop project_id from relation
op.drop_index("ix_relation_project_id", table_name="relation")
op.drop_constraint("fk_relation_project_id", "relation", type_="foreignkey")
op.drop_column("relation", "project_id")
else:
# SQLite requires batch operations
op.drop_index("ix_observation_project_id", table_name="observation")
with op.batch_alter_table("observation") as batch_op:
batch_op.drop_constraint("fk_observation_project_id", type_="foreignkey")
batch_op.drop_column("project_id")
op.drop_index("ix_relation_project_id", table_name="relation")
with op.batch_alter_table("relation") as batch_op:
batch_op.drop_constraint("fk_relation_project_id", type_="foreignkey")
batch_op.drop_column("project_id")
+7 -21
View File
@@ -30,7 +30,7 @@ from basic_memory.api.v2.routers import (
prompt_router as v2_prompt,
importer_router as v2_importer,
)
from basic_memory.config import ConfigManager, init_api_logging
from basic_memory.config import ConfigManager
from basic_memory.services.initialization import initialize_file_sync, initialize_app
@@ -38,9 +38,6 @@ from basic_memory.services.initialization import initialize_file_sync, initializ
async def lifespan(app: FastAPI): # pragma: no cover
"""Lifecycle manager for the FastAPI app. Not called in stdio mcp mode"""
# Initialize logging for API (stdout in cloud mode, file otherwise)
init_api_logging()
app_config = ConfigManager().config
logger.info("Starting Basic Memory API")
@@ -53,21 +50,12 @@ async def lifespan(app: FastAPI): # pragma: no cover
app.state.session_maker = session_maker
logger.info("Database connections cached in app state")
# Start file sync if enabled
if app_config.sync_changes and not app_config.is_test_env:
logger.info(f"Sync changes enabled: {app_config.sync_changes}")
logger.info(f"Sync changes enabled: {app_config.sync_changes}")
if app_config.sync_changes:
# start file sync task in background
async def _file_sync_runner() -> None:
await initialize_file_sync(app_config)
app.state.sync_task = asyncio.create_task(_file_sync_runner())
app.state.sync_task = asyncio.create_task(initialize_file_sync(app_config))
else:
if app_config.is_test_env:
logger.info("Test environment detected. Skipping file sync service.")
else:
logger.info("Sync changes disabled. Skipping file sync service.")
app.state.sync_task = None
logger.info("Sync changes disabled. Skipping file sync service.")
# proceed with startup
yield
@@ -76,10 +64,6 @@ async def lifespan(app: FastAPI): # pragma: no cover
if app.state.sync_task:
logger.info("Stopping sync...")
app.state.sync_task.cancel() # pyright: ignore
try:
await app.state.sync_task
except asyncio.CancelledError:
logger.info("Sync task cancelled successfully")
await db.shutdown_db()
@@ -116,6 +100,8 @@ app.include_router(v2_project, prefix="/v2")
app.include_router(project.project_resource_router)
app.include_router(management.router)
# Auth routes are handled by FastMCP automatically when auth is enabled
@app.exception_handler(Exception)
async def exception_handler(request, exc): # pragma: no cover
@@ -274,7 +274,7 @@ async def add_project(
raise HTTPException(status_code=500, detail="Failed to retrieve newly created project")
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message=f"Project '{new_project.name}' added successfully",
message=f"Project '{project_data.name}' added successfully",
status="success",
default=project_data.set_default,
new_project=ProjectItem(
@@ -329,7 +329,7 @@ async def remove_project(
await project_service.remove_project(name, delete_notes=delete_notes)
return ProjectStatusResponse(
message=f"Project '{old_project.name}' removed successfully",
message=f"Project '{name}' removed successfully",
status="success",
default=False,
old_project=ProjectItem(
+25 -35
View File
@@ -2,9 +2,9 @@
import tempfile
from pathlib import Path
from typing import Annotated, Union
from typing import Annotated
from fastapi import APIRouter, HTTPException, BackgroundTasks, Body, Response
from fastapi import APIRouter, HTTPException, BackgroundTasks, Body
from fastapi.responses import FileResponse, JSONResponse
from loguru import logger
@@ -25,17 +25,6 @@ from datetime import datetime
router = APIRouter(prefix="/resource", tags=["resources"])
def _mtime_to_datetime(entity: EntityModel) -> datetime:
"""Convert entity mtime (file modification time) to datetime.
Returns the file's actual modification time, falling back to updated_at
if mtime is not available.
"""
if entity.mtime:
return datetime.fromtimestamp(entity.mtime).astimezone()
return entity.updated_at
def get_entity_ids(item: SearchIndexRow) -> set[int]:
match item.type:
case SearchItemType.ENTITY:
@@ -50,7 +39,7 @@ def get_entity_ids(item: SearchIndexRow) -> set[int]:
raise ValueError(f"Unexpected type: {item.type}")
@router.get("/{identifier:path}", response_model=None)
@router.get("/{identifier:path}")
async def get_resource_content(
config: ProjectConfigDep,
link_resolver: LinkResolverDep,
@@ -61,7 +50,7 @@ async def get_resource_content(
identifier: str,
page: int = 1,
page_size: int = 10,
) -> Union[Response, FileResponse]:
) -> FileResponse:
"""Get resource content by identifier: name or permalink."""
logger.debug(f"Getting content for: {identifier}")
@@ -92,16 +81,13 @@ async def get_resource_content(
# return single response
if len(results) == 1:
entity = results[0]
# Check file exists via file_service (for cloud compatibility)
if not await file_service.exists(entity.file_path):
file_path = Path(f"{config.home}/{entity.file_path}")
if not file_path.exists():
raise HTTPException(
status_code=404,
detail=f"File not found: {entity.file_path}",
detail=f"File not found: {file_path}",
)
# Read content via file_service as bytes (works with both local and S3)
content = await file_service.read_file_bytes(entity.file_path)
content_type = file_service.content_type(entity.file_path)
return Response(content=content, media_type=content_type)
return FileResponse(path=file_path)
# for multiple files, initialize a temporary file for writing the results
with tempfile.NamedTemporaryFile(delete=False, mode="w", suffix=".md") as tmp_file:
@@ -111,7 +97,7 @@ async def get_resource_content(
# Read content for each entity
content = await file_service.read_entity_content(result)
memory_url = normalize_memory_url(result.permalink)
modified_date = _mtime_to_datetime(result).isoformat()
modified_date = result.updated_at.isoformat()
checksum = result.checksum[:8] if result.checksum else ""
# Prepare the delimited content
@@ -185,17 +171,21 @@ async def write_resource(
else:
content_str = str(content)
# Cloud compatibility: do not assume a local filesystem path structure.
# Delegate directory creation + writes to the configured FileService (local or S3).
await file_service.ensure_directory(Path(file_path).parent)
checksum = await file_service.write_file(file_path, content_str)
# Get full file path
full_path = Path(f"{config.home}/{file_path}")
# Ensure parent directory exists
full_path.parent.mkdir(parents=True, exist_ok=True)
# Write content to file
checksum = await file_service.write_file(full_path, content_str)
# Get file info
file_metadata = await file_service.get_file_metadata(file_path)
file_stats = file_service.file_stats(full_path)
# Determine file details
file_name = Path(file_path).name
content_type = file_service.content_type(file_path)
content_type = file_service.content_type(full_path)
entity_type = "canvas" if file_path.endswith(".canvas") else "file"
@@ -212,7 +202,7 @@ async def write_resource(
"content_type": content_type,
"file_path": file_path,
"checksum": checksum,
"updated_at": file_metadata.modified_at,
"updated_at": datetime.fromtimestamp(file_stats.st_mtime).astimezone(),
},
)
status_code = 200
@@ -224,8 +214,8 @@ async def write_resource(
content_type=content_type,
file_path=file_path,
checksum=checksum,
created_at=file_metadata.created_at,
updated_at=file_metadata.modified_at,
created_at=datetime.fromtimestamp(file_stats.st_ctime).astimezone(),
updated_at=datetime.fromtimestamp(file_stats.st_mtime).astimezone(),
)
entity = await entity_repository.add(entity)
status_code = 201
@@ -239,9 +229,9 @@ async def write_resource(
content={
"file_path": file_path,
"checksum": checksum,
"size": file_metadata.size,
"created_at": file_metadata.created_at.timestamp(),
"modified_at": file_metadata.modified_at.timestamp(),
"size": file_stats.st_size,
"created_at": file_stats.st_ctime,
"modified_at": file_stats.st_mtime,
},
)
except Exception as e: # pragma: no cover
+14 -28
View File
@@ -24,26 +24,8 @@ async def to_graph_context(
page: Optional[int] = None,
page_size: Optional[int] = None,
):
# First pass: collect all entity IDs needed for relations
entity_ids_needed: set[int] = set()
for context_item in context_result.results:
for item in (
[context_item.primary_result] + context_item.observations + context_item.related_results
):
if item.type == SearchItemType.RELATION:
if item.from_id: # pyright: ignore
entity_ids_needed.add(item.from_id) # pyright: ignore
if item.to_id:
entity_ids_needed.add(item.to_id)
# Batch fetch all entities at once
entity_lookup: dict[int, str] = {}
if entity_ids_needed:
entities = await entity_repository.find_by_ids(list(entity_ids_needed))
entity_lookup = {e.id: e.title for e in entities}
# Helper function to convert items to summaries
def to_summary(item: SearchIndexRow | ContextResultRow):
async def to_summary(item: SearchIndexRow | ContextResultRow):
match item.type:
case SearchItemType.ENTITY:
return EntitySummary(
@@ -66,8 +48,8 @@ async def to_graph_context(
created_at=item.created_at,
)
case SearchItemType.RELATION:
from_title = entity_lookup.get(item.from_id) if item.from_id else None # pyright: ignore
to_title = entity_lookup.get(item.to_id) if item.to_id else None
from_entity = await entity_repository.find_by_id(item.from_id) # pyright: ignore
to_entity = await entity_repository.find_by_id(item.to_id) if item.to_id else None
return RelationSummary(
relation_id=item.id,
entity_id=item.entity_id, # pyright: ignore
@@ -75,9 +57,9 @@ async def to_graph_context(
file_path=item.file_path,
permalink=item.permalink, # pyright: ignore
relation_type=item.relation_type, # pyright: ignore
from_entity=from_title,
from_entity=from_entity.title if from_entity else None,
from_entity_id=item.from_id, # pyright: ignore
to_entity=to_title,
to_entity=to_entity.title if to_entity else None,
to_entity_id=item.to_id,
created_at=item.created_at,
)
@@ -88,19 +70,23 @@ async def to_graph_context(
hierarchical_results = []
for context_item in context_result.results:
# Process primary result
primary_result = to_summary(context_item.primary_result)
primary_result = await to_summary(context_item.primary_result)
# Process observations (always ObservationSummary, validated by context_service)
observations = [to_summary(obs) for obs in context_item.observations]
# Process observations
observations = []
for obs in context_item.observations:
observations.append(await to_summary(obs))
# Process related results
related = [to_summary(rel) for rel in context_item.related_results]
related = []
for rel in context_item.related_results:
related.append(await to_summary(rel))
# Add to hierarchical results
hierarchical_results.append(
ContextResult(
primary_result=primary_result,
observations=observations, # pyright: ignore[reportArgumentType]
observations=observations,
related_results=related,
)
)
@@ -96,7 +96,9 @@ async def resolve_identifier(
# Try to resolve the identifier
entity = await link_resolver.resolve_link(data.identifier)
if not entity:
raise HTTPException(status_code=404, detail=f"Entity not found: '{data.identifier}'")
raise HTTPException(
status_code=404, detail=f"Could not resolve identifier: '{data.identifier}'"
)
# Determine resolution method
resolution_method = "search" # default
@@ -25,89 +25,11 @@ from basic_memory.schemas.project_info import (
ProjectItem,
ProjectStatusResponse,
)
from basic_memory.schemas.v2 import ProjectResolveRequest, ProjectResolveResponse
from basic_memory.utils import normalize_project_path, generate_permalink
from basic_memory.utils import normalize_project_path
router = APIRouter(prefix="/projects", tags=["project_management-v2"])
@router.post("/resolve", response_model=ProjectResolveResponse)
async def resolve_project_identifier(
data: ProjectResolveRequest,
project_repository: ProjectRepositoryDep,
) -> ProjectResolveResponse:
"""Resolve a project identifier (name or permalink) to a project ID.
This endpoint provides efficient lookup of projects by name without
needing to fetch the entire project list. Supports case-insensitive
matching on both name and permalink.
Args:
data: Request containing the identifier to resolve
Returns:
Project information including the numeric ID
Raises:
HTTPException: 404 if project not found
Example:
POST /v2/projects/resolve
{"identifier": "my-project"}
Returns:
{
"project_id": 1,
"name": "my-project",
"permalink": "my-project",
"path": "/path/to/project",
"is_active": true,
"is_default": false,
"resolution_method": "name"
}
"""
logger.info(f"API v2 request: resolve_project_identifier for '{data.identifier}'")
# Generate permalink for comparison
identifier_permalink = generate_permalink(data.identifier)
# Try to find project by ID first (if identifier is numeric)
resolution_method = "name"
project = None
if data.identifier.isdigit():
project_id = int(data.identifier)
project = await project_repository.get_by_id(project_id)
if project:
resolution_method = "id"
# If not found by ID, try by permalink first (exact match)
if not project:
project = await project_repository.get_by_permalink(identifier_permalink)
if project:
resolution_method = "permalink"
# If not found by permalink, try case-insensitive name search
# Uses efficient database query instead of fetching all projects
if not project:
project = await project_repository.get_by_name_case_insensitive(data.identifier)
if project:
resolution_method = "name"
if not project:
raise HTTPException(status_code=404, detail=f"Project not found: '{data.identifier}'")
return ProjectResolveResponse(
project_id=project.id,
name=project.name,
permalink=generate_permalink(project.name),
path=normalize_project_path(project.path),
is_active=project.is_active if hasattr(project, "is_active") else True,
is_default=project.is_default or False,
resolution_method=resolution_method,
)
@router.get("/{project_id}", response_model=ProjectItem)
async def get_project_by_id(
project_id: ProjectIdPathDep,
@@ -11,7 +11,8 @@ Key differences from v1:
from pathlib import Path
from fastapi import APIRouter, HTTPException, Response
from fastapi import APIRouter, HTTPException
from fastapi.responses import FileResponse
from loguru import logger
from basic_memory.deps import (
@@ -29,6 +30,7 @@ from basic_memory.schemas.v2.resource import (
ResourceResponse,
)
from basic_memory.utils import validate_project_path
from datetime import datetime
router = APIRouter(prefix="/resource", tags=["resources-v2"])
@@ -40,7 +42,7 @@ async def get_resource_content(
config: ProjectConfigV2Dep,
entity_service: EntityServiceV2Dep,
file_service: FileServiceV2Dep,
) -> Response:
) -> FileResponse:
"""Get raw resource content by entity ID.
Args:
@@ -51,7 +53,7 @@ async def get_resource_content(
file_service: File service for reading file content
Returns:
Response with entity content
FileResponse with entity content
Raises:
HTTPException: 404 if entity or file not found
@@ -74,18 +76,14 @@ async def get_resource_content(
detail="Entity contains invalid file path",
)
# Check file exists via file_service (for cloud compatibility)
if not await file_service.exists(entity.file_path):
file_path = Path(f"{config.home}/{entity.file_path}")
if not file_path.exists():
raise HTTPException(
status_code=404,
detail=f"File not found: {entity.file_path}",
detail=f"File not found: {file_path}",
)
# Read content via file_service as bytes (works with both local and S3)
content = await file_service.read_file_bytes(entity.file_path)
content_type = file_service.content_type(entity.file_path)
return Response(content=content, media_type=content_type)
return FileResponse(path=file_path)
@router.post("", response_model=ResourceResponse)
@@ -135,17 +133,21 @@ async def create_resource(
f"Use PUT /resource/{existing_entity.id} to update it.",
)
# Cloud compatibility: avoid assuming a local filesystem path.
# Delegate directory creation + writes to FileService (local or S3).
await file_service.ensure_directory(Path(data.file_path).parent)
checksum = await file_service.write_file(data.file_path, data.content)
# Get full file path
full_path = Path(f"{config.home}/{data.file_path}")
# Ensure parent directory exists
full_path.parent.mkdir(parents=True, exist_ok=True)
# Write content to file
checksum = await file_service.write_file(full_path, data.content)
# Get file info
file_metadata = await file_service.get_file_metadata(data.file_path)
file_stats = file_service.file_stats(full_path)
# Determine file details
file_name = Path(data.file_path).name
content_type = file_service.content_type(data.file_path)
content_type = file_service.content_type(full_path)
entity_type = "canvas" if data.file_path.endswith(".canvas") else "file"
# Create a new entity model
@@ -155,8 +157,8 @@ async def create_resource(
content_type=content_type,
file_path=data.file_path,
checksum=checksum,
created_at=file_metadata.created_at,
updated_at=file_metadata.modified_at,
created_at=datetime.fromtimestamp(file_stats.st_ctime).astimezone(),
updated_at=datetime.fromtimestamp(file_stats.st_mtime).astimezone(),
)
entity = await entity_repository.add(entity)
@@ -168,9 +170,9 @@ async def create_resource(
entity_id=entity.id,
file_path=data.file_path,
checksum=checksum,
size=file_metadata.size,
created_at=file_metadata.created_at.timestamp(),
modified_at=file_metadata.modified_at.timestamp(),
size=file_stats.st_size,
created_at=file_stats.st_ctime,
modified_at=file_stats.st_mtime,
)
except HTTPException:
# Re-raise HTTP exceptions without wrapping
@@ -230,27 +232,31 @@ async def update_resource(
"Path must be relative and stay within project boundaries.",
)
# Get full paths
old_full_path = Path(f"{config.home}/{entity.file_path}")
new_full_path = Path(f"{config.home}/{target_file_path}")
# If moving file, handle the move
if data.file_path and data.file_path != entity.file_path:
# Ensure new parent directory exists (no-op for S3)
await file_service.ensure_directory(Path(target_file_path).parent)
# Ensure new parent directory exists
new_full_path.parent.mkdir(parents=True, exist_ok=True)
# If old file exists, remove it via file_service (for cloud compatibility)
if await file_service.exists(entity.file_path):
await file_service.delete_file(entity.file_path)
# If old file exists, remove it
if old_full_path.exists():
old_full_path.unlink()
else:
# Ensure directory exists for in-place update
await file_service.ensure_directory(Path(target_file_path).parent)
new_full_path.parent.mkdir(parents=True, exist_ok=True)
# Write content to target file
checksum = await file_service.write_file(target_file_path, data.content)
checksum = await file_service.write_file(new_full_path, data.content)
# Get file info
file_metadata = await file_service.get_file_metadata(target_file_path)
file_stats = file_service.file_stats(new_full_path)
# Determine file details
file_name = Path(target_file_path).name
content_type = file_service.content_type(target_file_path)
content_type = file_service.content_type(new_full_path)
entity_type = "canvas" if target_file_path.endswith(".canvas") else "file"
# Update entity
@@ -262,7 +268,7 @@ async def update_resource(
"content_type": content_type,
"file_path": target_file_path,
"checksum": checksum,
"updated_at": file_metadata.modified_at,
"updated_at": datetime.fromtimestamp(file_stats.st_mtime).astimezone(),
},
)
@@ -274,9 +280,9 @@ async def update_resource(
entity_id=entity_id,
file_path=target_file_path,
checksum=checksum,
size=file_metadata.size,
created_at=file_metadata.created_at.timestamp(),
modified_at=file_metadata.modified_at.timestamp(),
size=file_stats.st_size,
created_at=file_stats.st_ctime,
modified_at=file_stats.st_mtime,
)
except HTTPException:
# Re-raise HTTP exceptions without wrapping
+5 -35
View File
@@ -1,21 +1,8 @@
# Suppress Logfire "not configured" warning - we only use Logfire in cloud/server contexts
import os
from typing import Optional
os.environ.setdefault("LOGFIRE_IGNORE_NO_CONFIG", "1")
import typer
# Remove loguru's default handler IMMEDIATELY, before any other imports.
# This prevents DEBUG logs from appearing on stdout during module-level
# initialization (e.g., template_loader.TemplateLoader() logs at DEBUG level).
from loguru import logger
logger.remove()
from typing import Optional # noqa: E402
import typer # noqa: E402
from basic_memory.config import ConfigManager, init_cli_logging # noqa: E402
from basic_memory.telemetry import show_notice_if_needed, track_app_started # noqa: E402
from basic_memory.config import ConfigManager
def version_callback(value: bool) -> None:
@@ -44,25 +31,8 @@ def app_callback(
) -> None:
"""Basic Memory - Local-first personal knowledge management."""
# Initialize logging for CLI (file only, no stdout)
init_cli_logging()
# Show telemetry notice and track CLI startup
# Skip for 'mcp' command - it handles its own telemetry in lifespan
# Skip for 'telemetry' command - avoid issues when user is managing telemetry
if ctx.invoked_subcommand not in {"mcp", "telemetry"}:
show_notice_if_needed()
track_app_started("cli")
# Run initialization for commands that don't use the API
# Skip for 'mcp' command - it has its own lifespan that handles initialization
# Skip for API-using commands (status, sync, etc.) - they handle initialization via deps.py
api_commands = {"mcp", "status", "sync", "project", "tool"}
if (
not version
and ctx.invoked_subcommand is not None
and ctx.invoked_subcommand not in api_commands
):
# Run initialization for every command unless --version was specified
if not version and ctx.invoked_subcommand is not None:
from basic_memory.services.initialization import ensure_initialization
app_config = ConfigManager().config
+1 -3
View File
@@ -1,7 +1,7 @@
"""CLI commands for basic-memory."""
from . import status, db, import_memory_json, mcp, import_claude_conversations
from . import import_claude_projects, import_chatgpt, tool, project, format, telemetry
from . import import_claude_projects, import_chatgpt, tool, project
__all__ = [
"status",
@@ -13,6 +13,4 @@ __all__ = [
"import_chatgpt",
"tool",
"project",
"format",
"telemetry",
]
@@ -9,14 +9,11 @@ This module provides simplified, project-scoped rclone operations:
Replaces tenant-wide sync with project-scoped workflows.
"""
import re
import subprocess
from dataclasses import dataclass
from functools import lru_cache
from pathlib import Path
from typing import Optional
from loguru import logger
from rich.console import Console
from basic_memory.cli.commands.cloud.rclone_installer import is_rclone_installed
@@ -24,9 +21,6 @@ from basic_memory.utils import normalize_project_path
console = Console()
# Minimum rclone version for --create-empty-src-dirs support
MIN_RCLONE_VERSION_EMPTY_DIRS = (1, 64, 0)
class RcloneError(Exception):
"""Exception raised for rclone command errors."""
@@ -49,42 +43,6 @@ def check_rclone_installed() -> None:
)
@lru_cache(maxsize=1)
def get_rclone_version() -> tuple[int, int, int] | None:
"""Get rclone version as (major, minor, patch) tuple.
Returns:
Version tuple like (1, 64, 2), or None if version cannot be determined.
Note:
Result is cached since rclone version won't change during runtime.
"""
try:
result = subprocess.run(["rclone", "version"], capture_output=True, text=True, timeout=10)
# Parse "rclone v1.64.2" or "rclone v1.60.1-DEV"
match = re.search(r"v(\d+)\.(\d+)\.(\d+)", result.stdout)
if match:
version = (int(match.group(1)), int(match.group(2)), int(match.group(3)))
logger.debug(f"Detected rclone version: {version}")
return version
except Exception as e:
logger.warning(f"Could not determine rclone version: {e}")
return None
def supports_create_empty_src_dirs() -> bool:
"""Check if installed rclone supports --create-empty-src-dirs flag.
Returns:
True if rclone version >= 1.64.0, False otherwise.
"""
version = get_rclone_version()
if version is None:
# If we can't determine version, assume older and skip the flag
return False
return version >= MIN_RCLONE_VERSION_EMPTY_DIRS
@dataclass
class SyncProject:
"""Project configured for cloud sync.
@@ -260,6 +218,7 @@ def project_bisync(
"bisync",
str(local_path),
remote_path,
"--create-empty-src-dirs",
"--resilient",
"--conflict-resolve=newer",
"--max-delete=25",
@@ -270,10 +229,6 @@ def project_bisync(
str(state_path),
]
# Add --create-empty-src-dirs if rclone version supports it (v1.64+)
if supports_create_empty_src_dirs():
cmd.append("--create-empty-src-dirs")
if verbose:
cmd.append("--verbose")
else:
+1 -27
View File
@@ -1,14 +1,12 @@
"""utility functions for commands"""
import asyncio
from typing import Optional, TypeVar, Coroutine, Any
from typing import Optional
from mcp.server.fastmcp.exceptions import ToolError
import typer
from rich.console import Console
from basic_memory import db
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.tools.utils import call_post, call_get
@@ -17,30 +15,6 @@ from basic_memory.schemas import ProjectInfoResponse
console = Console()
T = TypeVar("T")
def run_with_cleanup(coro: Coroutine[Any, Any, T]) -> T:
"""Run an async coroutine with proper database cleanup.
This helper ensures database connections are cleaned up before the event
loop closes, preventing process hangs in CLI commands.
Args:
coro: The coroutine to run
Returns:
The result of the coroutine
"""
async def _with_cleanup() -> T:
try:
return await coro
finally:
await db.shutdown_db()
return asyncio.run(_with_cleanup())
async def run_sync(project: Optional[str] = None, force_full: bool = False):
"""Run sync operation via API endpoint.
-198
View File
@@ -1,198 +0,0 @@
"""Format command for basic-memory CLI."""
import asyncio
from pathlib import Path
from typing import Annotated, Optional
import typer
from loguru import logger
from rich.console import Console
from rich.progress import Progress, SpinnerColumn, TextColumn
from basic_memory.cli.app import app
from basic_memory.config import ConfigManager, get_project_config
from basic_memory.file_utils import format_file
console = Console()
def is_markdown_extension(path: Path) -> bool:
"""Check if file has a markdown extension."""
return path.suffix.lower() in (".md", ".markdown")
async def format_single_file(file_path: Path, app_config) -> tuple[Path, bool, Optional[str]]:
"""Format a single file.
Returns:
Tuple of (path, success, error_message)
"""
try:
result = await format_file(
file_path, app_config, is_markdown=is_markdown_extension(file_path)
)
if result is not None:
return (file_path, True, None)
else:
return (file_path, False, "No formatter configured or formatting skipped")
except Exception as e:
return (file_path, False, str(e))
async def format_files(
paths: list[Path], app_config, show_progress: bool = True
) -> tuple[int, int, list[tuple[Path, str]]]:
"""Format multiple files.
Returns:
Tuple of (formatted_count, skipped_count, errors)
"""
formatted = 0
skipped = 0
errors: list[tuple[Path, str]] = []
if show_progress:
with Progress(
SpinnerColumn(),
TextColumn("[progress.description]{task.description}"),
console=console,
) as progress:
task = progress.add_task("Formatting files...", total=len(paths))
for file_path in paths:
path, success, error = await format_single_file(file_path, app_config)
if success:
formatted += 1
elif error and "No formatter configured" not in error:
errors.append((path, error))
else:
skipped += 1
progress.update(task, advance=1)
else:
for file_path in paths:
path, success, error = await format_single_file(file_path, app_config)
if success:
formatted += 1
elif error and "No formatter configured" not in error:
errors.append((path, error))
else:
skipped += 1
return formatted, skipped, errors
async def run_format(
path: Optional[Path] = None,
project: Optional[str] = None,
) -> None:
"""Run the format command."""
app_config = ConfigManager().config
# Check if formatting is enabled
if (
not app_config.format_on_save
and not app_config.formatter_command
and not app_config.formatters
):
console.print(
"[yellow]No formatters configured. Set format_on_save=true and "
"formatter_command or formatters in your config.[/yellow]"
)
console.print(
"\nExample config (~/.basic-memory/config.json):\n"
' "format_on_save": true,\n'
' "formatter_command": "prettier --write {file}"\n'
)
raise typer.Exit(1)
# Temporarily enable format_on_save for this command
# (so format_file actually runs the formatter)
original_format_on_save = app_config.format_on_save
app_config.format_on_save = True
try:
# Determine which files to format
if path:
# Format specific file or directory
if path.is_file():
files = [path]
elif path.is_dir():
# Find all markdown and json files
files = (
list(path.rglob("*.md"))
+ list(path.rglob("*.json"))
+ list(path.rglob("*.canvas"))
)
else:
console.print(f"[red]Path not found: {path}[/red]")
raise typer.Exit(1)
else:
# Format all files in project
project_config = get_project_config(project)
project_path = Path(project_config.home)
if not project_path.exists():
console.print(f"[red]Project path not found: {project_path}[/red]")
raise typer.Exit(1)
# Find all markdown and json files
files = (
list(project_path.rglob("*.md"))
+ list(project_path.rglob("*.json"))
+ list(project_path.rglob("*.canvas"))
)
if not files:
console.print("[yellow]No files found to format.[/yellow]")
return
console.print(f"Found {len(files)} file(s) to format...")
formatted, skipped, errors = await format_files(files, app_config)
# Print summary
console.print()
if formatted > 0:
console.print(f"[green]Formatted: {formatted} file(s)[/green]")
if skipped > 0:
console.print(f"[dim]Skipped: {skipped} file(s) (no formatter for extension)[/dim]")
if errors:
console.print(f"[red]Errors: {len(errors)} file(s)[/red]")
for path, error in errors:
console.print(f" [red]{path}[/red]: {error}")
finally:
# Restore original setting
app_config.format_on_save = original_format_on_save
@app.command()
def format(
path: Annotated[
Optional[Path],
typer.Argument(help="File or directory to format. Defaults to current project."),
] = None,
project: Annotated[
Optional[str],
typer.Option("--project", "-p", help="Project name to format."),
] = None,
) -> None:
"""Format files using configured formatters.
Uses the formatter_command or formatters settings from your config.
By default, formats all .md, .json, and .canvas files in the current project.
Examples:
basic-memory format # Format all files in current project
basic-memory format --project research # Format files in specific project
basic-memory format notes/meeting.md # Format a specific file
basic-memory format notes/ # Format all files in directory
"""
try:
asyncio.run(run_format(path, project))
except Exception as e:
if not isinstance(e, typer.Exit):
logger.error(f"Error formatting files: {e}")
console.print(f"[red]Error formatting files: {e}[/red]")
raise typer.Exit(code=1)
raise
@@ -7,7 +7,7 @@ from typing import Annotated
import typer
from basic_memory.cli.app import import_app
from basic_memory.config import ConfigManager, get_project_config
from basic_memory.config import get_project_config
from basic_memory.importers import ChatGPTImporter
from basic_memory.markdown import EntityParser, MarkdownProcessor
from loguru import logger
@@ -20,9 +20,8 @@ console = Console()
async def get_markdown_processor() -> MarkdownProcessor:
"""Get MarkdownProcessor instance."""
config = get_project_config()
app_config = ConfigManager().config
entity_parser = EntityParser(config.home)
return MarkdownProcessor(entity_parser, app_config=app_config)
return MarkdownProcessor(entity_parser)
@import_app.command(name="chatgpt", help="Import conversations from ChatGPT JSON export.")
@@ -7,7 +7,7 @@ from typing import Annotated
import typer
from basic_memory.cli.app import claude_app
from basic_memory.config import ConfigManager, get_project_config
from basic_memory.config import get_project_config
from basic_memory.importers.claude_conversations_importer import ClaudeConversationsImporter
from basic_memory.markdown import EntityParser, MarkdownProcessor
from loguru import logger
@@ -20,9 +20,8 @@ console = Console()
async def get_markdown_processor() -> MarkdownProcessor:
"""Get MarkdownProcessor instance."""
config = get_project_config()
app_config = ConfigManager().config
entity_parser = EntityParser(config.home)
return MarkdownProcessor(entity_parser, app_config=app_config)
return MarkdownProcessor(entity_parser)
@claude_app.command(name="conversations", help="Import chat conversations from Claude.ai.")
@@ -7,7 +7,7 @@ from typing import Annotated
import typer
from basic_memory.cli.app import claude_app
from basic_memory.config import ConfigManager, get_project_config
from basic_memory.config import get_project_config
from basic_memory.importers.claude_projects_importer import ClaudeProjectsImporter
from basic_memory.markdown import EntityParser, MarkdownProcessor
from loguru import logger
@@ -20,9 +20,8 @@ console = Console()
async def get_markdown_processor() -> MarkdownProcessor:
"""Get MarkdownProcessor instance."""
config = get_project_config()
app_config = ConfigManager().config
entity_parser = EntityParser(config.home)
return MarkdownProcessor(entity_parser, app_config=app_config)
return MarkdownProcessor(entity_parser)
@claude_app.command(name="projects", help="Import projects from Claude.ai.")
@@ -7,7 +7,7 @@ from typing import Annotated
import typer
from basic_memory.cli.app import import_app
from basic_memory.config import ConfigManager, get_project_config
from basic_memory.config import get_project_config
from basic_memory.importers.memory_json_importer import MemoryJsonImporter
from basic_memory.markdown import EntityParser, MarkdownProcessor
from loguru import logger
@@ -20,9 +20,8 @@ console = Console()
async def get_markdown_processor() -> MarkdownProcessor:
"""Get MarkdownProcessor instance."""
config = get_project_config()
app_config = ConfigManager().config
entity_parser = EntityParser(config.home)
return MarkdownProcessor(entity_parser, app_config=app_config)
return MarkdownProcessor(entity_parser)
@import_app.command()
+26 -8
View File
@@ -1,13 +1,14 @@
"""MCP server command with streamable HTTP transport."""
import asyncio
import os
import typer
from typing import Optional
from basic_memory.cli.app import app
from basic_memory.config import ConfigManager, init_mcp_logging
from basic_memory.config import ConfigManager
# Import mcp instance (has lifespan that handles initialization and file sync)
# Import mcp instance
from basic_memory.mcp.server import mcp as mcp_server # pragma: no cover
# Import mcp tools to register them
@@ -16,6 +17,8 @@ import basic_memory.mcp.tools # noqa: F401 # pragma: no cover
# Import prompts to register them
import basic_memory.mcp.prompts # noqa: F401 # pragma: no cover
from loguru import logger
import threading
from basic_memory.services.initialization import initialize_file_sync
config = ConfigManager().config
@@ -40,11 +43,7 @@ if not config.cloud_mode_enabled:
- stdio: Standard I/O (good for local usage)
- streamable-http: Recommended for web deployments (default)
- sse: Server-Sent Events (for compatibility with existing clients)
Initialization, file sync, and cleanup are handled by the MCP server's lifespan.
"""
# Initialize logging for MCP (file only, stdout breaks protocol)
init_mcp_logging()
# Validate and set project constraint if specified
if project:
@@ -58,8 +57,27 @@ if not config.cloud_mode_enabled:
os.environ["BASIC_MEMORY_MCP_PROJECT"] = project_name
logger.info(f"MCP server constrained to project: {project_name}")
# Run the MCP server (blocks)
# Lifespan handles: initialization, migrations, file sync, cleanup
app_config = ConfigManager().config
def run_file_sync():
"""Run file sync in a separate thread with its own event loop."""
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
loop.run_until_complete(initialize_file_sync(app_config))
except Exception as e:
logger.error(f"File sync error: {e}", err=True)
finally:
loop.close()
logger.info(f"Sync changes enabled: {app_config.sync_changes}")
if app_config.sync_changes:
# Start the sync thread
sync_thread = threading.Thread(target=run_file_sync, daemon=True)
sync_thread.start()
logger.info("Started file sync in background")
# Now run the MCP server (blocks)
logger.info(f"Starting MCP server with {transport.upper()} transport")
if transport == "stdio":
+9 -22
View File
@@ -16,9 +16,14 @@ from datetime import datetime
from rich.panel import Panel
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.tools.utils import call_get, call_post, call_delete, call_put, call_patch
from basic_memory.schemas.project_info import ProjectList, ProjectStatusResponse
from basic_memory.mcp.tools.utils import call_get
from basic_memory.schemas.project_info import ProjectList
from basic_memory.mcp.tools.utils import call_post
from basic_memory.schemas.project_info import ProjectStatusResponse
from basic_memory.mcp.tools.utils import call_delete
from basic_memory.mcp.tools.utils import call_put
from basic_memory.utils import generate_permalink, normalize_project_path
from basic_memory.mcp.tools.utils import call_patch
# Import rclone commands for project sync
from basic_memory.cli.commands.cloud.rclone_commands import (
@@ -249,17 +254,9 @@ def remove_project(
async def _remove_project():
async with get_client() as client:
# Convert name to permalink for efficient resolution
project_permalink = generate_permalink(name)
# Use v2 project resolver to find project ID by permalink
resolve_data = {"identifier": project_permalink}
response = await call_post(client, "/v2/projects/resolve", json=resolve_data)
target_project = response.json()
# Use v2 API with project ID
response = await call_delete(
client, f"/v2/projects/{target_project['project_id']}?delete_notes={delete_notes}"
client, f"/projects/{project_permalink}?delete_notes={delete_notes}"
)
return ProjectStatusResponse.model_validate(response.json())
@@ -332,18 +329,8 @@ def set_default_project(
async def _set_default():
async with get_client() as client:
# Convert name to permalink for efficient resolution
project_permalink = generate_permalink(name)
# Use v2 project resolver to find project ID by permalink
resolve_data = {"identifier": project_permalink}
response = await call_post(client, "/v2/projects/resolve", json=resolve_data)
target_project = response.json()
# Use v2 API with project ID
response = await call_put(
client, f"/v2/projects/{target_project['project_id']}/default"
)
response = await call_put(client, f"/projects/{project_permalink}/default")
return ProjectStatusResponse.model_validate(response.json())
try:
+2 -3
View File
@@ -1,5 +1,6 @@
"""Status command for basic-memory CLI."""
import asyncio
from typing import Set, Dict
from typing import Annotated, Optional
@@ -164,10 +165,8 @@ def status(
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed file information"),
):
"""Show sync status between files and database."""
from basic_memory.cli.commands.command_utils import run_with_cleanup
try:
run_with_cleanup(run_status(project, verbose)) # pragma: no cover
asyncio.run(run_status(project, verbose)) # pragma: no cover
except Exception as e:
logger.error(f"Error checking status: {e}")
typer.echo(f"Error checking status: {e}", err=True)
@@ -1,81 +0,0 @@
"""Telemetry commands for basic-memory CLI."""
import typer
from rich.console import Console
from rich.panel import Panel
from basic_memory.cli.app import app
from basic_memory.config import ConfigManager
console = Console()
# Create telemetry subcommand group
telemetry_app = typer.Typer(help="Manage anonymous telemetry settings")
app.add_typer(telemetry_app, name="telemetry")
@telemetry_app.command("enable")
def enable() -> None:
"""Enable anonymous telemetry.
Telemetry helps improve Basic Memory by collecting anonymous usage data.
No personal data, note content, or file paths are ever collected.
"""
config_manager = ConfigManager()
config = config_manager.config
config.telemetry_enabled = True
config_manager.save_config(config)
console.print("[green]Telemetry enabled[/green]")
console.print("[dim]Thank you for helping improve Basic Memory![/dim]")
@telemetry_app.command("disable")
def disable() -> None:
"""Disable anonymous telemetry.
You can re-enable telemetry anytime with: bm telemetry enable
"""
config_manager = ConfigManager()
config = config_manager.config
config.telemetry_enabled = False
config_manager.save_config(config)
console.print("[yellow]Telemetry disabled[/yellow]")
@telemetry_app.command("status")
def status() -> None:
"""Show current telemetry status and what's collected."""
from basic_memory.telemetry import get_install_id, TELEMETRY_DOCS_URL
config = ConfigManager().config
status_text = (
"[green]enabled[/green]" if config.telemetry_enabled else "[yellow]disabled[/yellow]"
)
console.print(f"\nTelemetry: {status_text}")
console.print(f"Install ID: [dim]{get_install_id()}[/dim]")
console.print()
what_we_collect = """
[bold]What we collect:[/bold]
- App version, Python version, OS, architecture
- Feature usage (which MCP tools and CLI commands)
- Sync statistics (entity count, duration)
- Error types (sanitized, no file paths)
[bold]What we NEVER collect:[/bold]
- Note content, file names, or paths
- Personal information
- IP addresses
"""
console.print(
Panel(
what_we_collect.strip(),
title="Telemetry Details",
border_style="blue",
expand=False,
)
)
console.print(f"[dim]Details: {TELEMETRY_DOCS_URL}[/dim]")
-7
View File
@@ -13,16 +13,9 @@ from basic_memory.cli.commands import ( # noqa: F401 # pragma: no cover
mcp,
project,
status,
telemetry,
tool,
)
# Re-apply warning filter AFTER all imports
# (authlib adds a DeprecationWarning filter that overrides ours)
import warnings # pragma: no cover
warnings.filterwarnings("ignore") # pragma: no cover
if __name__ == "__main__": # pragma: no cover
# start the app
app()
+70 -148
View File
@@ -9,9 +9,10 @@ from typing import Any, Dict, Literal, Optional, List, Tuple
from enum import Enum
from loguru import logger
from pydantic import BaseModel, Field, model_validator
from pydantic import BaseModel, Field, field_validator
from pydantic_settings import BaseSettings, SettingsConfigDict
import basic_memory
from basic_memory.utils import setup_logging, generate_permalink
@@ -99,32 +100,13 @@ class BasicMemoryConfig(BaseSettings):
description="Database connection URL. For Postgres, use postgresql+asyncpg://user:pass@host:port/db. If not set, SQLite will use default path.",
)
# Database connection pool configuration (Postgres only)
db_pool_size: int = Field(
default=20,
description="Number of connections to keep in the pool (Postgres only)",
gt=0,
)
db_pool_overflow: int = Field(
default=40,
description="Max additional connections beyond pool_size under load (Postgres only)",
gt=0,
)
db_pool_recycle: int = Field(
default=180,
description="Recycle connections after N seconds to prevent stale connections. Default 180s works well with Neon's ~5 minute scale-to-zero (Postgres only)",
gt=0,
)
# Watch service configuration
sync_delay: int = Field(
default=1000, description="Milliseconds to wait after changes before syncing", gt=0
)
watch_project_reload_interval: int = Field(
default=300,
description="Seconds between reloading project list in watch service. Higher values reduce CPU usage by minimizing watcher restarts. Default 300s (5 min) balances efficiency with responsiveness to new projects.",
gt=0,
default=30, description="Seconds between reloading project list in watch service", gt=0
)
# update permalinks on move
@@ -165,28 +147,6 @@ class BasicMemoryConfig(BaseSettings):
description="Skip expensive initialization synchronization. Useful for cloud/stateless deployments where project reconciliation is not needed.",
)
# File formatting configuration
format_on_save: bool = Field(
default=False,
description="Automatically format files after saving using configured formatter. Disabled by default.",
)
formatter_command: Optional[str] = Field(
default=None,
description="External formatter command. Use {file} as placeholder for file path. If not set, uses built-in mdformat (Python, no Node.js required). Set to 'npx prettier --write {file}' for Prettier.",
)
formatters: Dict[str, str] = Field(
default_factory=dict,
description="Per-extension formatters. Keys are extensions (without dot), values are commands. Example: {'md': 'prettier --write {file}', 'json': 'prettier --write {file}'}",
)
formatter_timeout: float = Field(
default=5.0,
description="Maximum seconds to wait for formatter to complete",
gt=0,
)
# Project path constraints
project_root: Optional[str] = Field(
default=None,
@@ -221,34 +181,6 @@ class BasicMemoryConfig(BaseSettings):
description="Cloud project sync configuration mapping project names to their local paths and sync state",
)
# Telemetry configuration (Homebrew-style opt-out)
telemetry_enabled: bool = Field(
default=True,
description="Send anonymous usage statistics to help improve Basic Memory. Disable with: bm telemetry disable",
)
telemetry_notice_shown: bool = Field(
default=False,
description="Whether the one-time telemetry notice has been shown to the user",
)
@property
def is_test_env(self) -> bool:
"""Check if running in a test environment.
Returns True if any of:
- env field is set to "test"
- BASIC_MEMORY_ENV environment variable is "test"
- PYTEST_CURRENT_TEST environment variable is set (pytest is running)
Used to disable features like telemetry and file watchers during tests.
"""
return (
self.env == "test"
or os.getenv("BASIC_MEMORY_ENV", "").lower() == "test"
or os.getenv("PYTEST_CURRENT_TEST") is not None
)
@property
def cloud_mode_enabled(self) -> bool:
"""Check if cloud mode is enabled.
@@ -265,36 +197,6 @@ class BasicMemoryConfig(BaseSettings):
# Fall back to config file value
return self.cloud_mode
@classmethod
def for_cloud_tenant(
cls,
database_url: str,
projects: Optional[Dict[str, str]] = None,
) -> "BasicMemoryConfig":
"""Create config for cloud tenant - no config.json, database is source of truth.
This factory method creates a BasicMemoryConfig suitable for cloud deployments
where:
- Database is Postgres (Neon), not SQLite
- Projects are discovered from the database, not config file
- Path validation is skipped (no local filesystem in cloud)
- Initialization sync is skipped (stateless deployment)
Args:
database_url: Postgres connection URL for tenant database
projects: Optional project mapping (usually empty, discovered from DB)
Returns:
BasicMemoryConfig configured for cloud mode
"""
return cls(
database_backend=DatabaseBackend.POSTGRES,
database_url=database_url,
projects=projects or {},
cloud_mode=True,
skip_initialization_sync=True,
)
model_config = SettingsConfigDict(
env_prefix="BASIC_MEMORY_",
extra="ignore",
@@ -311,10 +213,6 @@ class BasicMemoryConfig(BaseSettings):
def model_post_init(self, __context: Any) -> None:
"""Ensure configuration is valid after initialization."""
# Skip project initialization in cloud mode - projects are discovered from DB
if self.database_backend == DatabaseBackend.POSTGRES:
return
# Ensure at least one project exists; if none exist then create main
if not self.projects: # pragma: no cover
self.projects["main"] = str(
@@ -357,26 +255,19 @@ class BasicMemoryConfig(BaseSettings):
"""Get all configured projects as ProjectConfig objects."""
return [ProjectConfig(name=name, home=Path(path)) for name, path in self.projects.items()]
@model_validator(mode="after")
def ensure_project_paths_exists(self) -> "BasicMemoryConfig": # pragma: no cover
"""Ensure project paths exist.
Skips path creation when using Postgres backend (cloud mode) since
cloud tenants don't use local filesystem paths.
"""
# Skip path creation for cloud mode - no local filesystem
if self.database_backend == DatabaseBackend.POSTGRES:
return self
for name, path_value in self.projects.items():
@field_validator("projects")
@classmethod
def ensure_project_paths_exists(cls, v: Dict[str, str]) -> Dict[str, str]: # pragma: no cover
"""Ensure project path exists."""
for name, path_value in v.items():
path = Path(path_value)
if not path.exists():
if not Path(path).exists():
try:
path.mkdir(parents=True)
except Exception as e:
logger.error(f"Failed to create project path: {e}")
raise e
return self
return v
@property
def data_dir_path(self):
@@ -579,38 +470,69 @@ def save_basic_memory_config(file_path: Path, config: BasicMemoryConfig) -> None
logger.error(f"Failed to save config: {e}")
# Logging initialization functions for different entry points
# setup logging to a single log file in user home directory
user_home = Path.home()
log_dir = user_home / DATA_DIR_NAME
log_dir.mkdir(parents=True, exist_ok=True)
def init_cli_logging() -> None: # pragma: no cover
"""Initialize logging for CLI commands - file only.
CLI commands should not log to stdout to avoid interfering with
command output and shell integration.
# Process info for logging
def get_process_name(): # pragma: no cover
"""
log_level = os.getenv("BASIC_MEMORY_LOG_LEVEL", "INFO")
setup_logging(log_level=log_level, log_to_file=True)
def init_mcp_logging() -> None: # pragma: no cover
"""Initialize logging for MCP server - file only.
MCP server must not log to stdout as it would corrupt the
JSON-RPC protocol communication.
get the type of process for logging
"""
log_level = os.getenv("BASIC_MEMORY_LOG_LEVEL", "INFO")
setup_logging(log_level=log_level, log_to_file=True)
import sys
def init_api_logging() -> None: # pragma: no cover
"""Initialize logging for API server.
Cloud mode (BASIC_MEMORY_CLOUD_MODE=1): stdout with structured context
Local mode: file only
"""
log_level = os.getenv("BASIC_MEMORY_LOG_LEVEL", "INFO")
cloud_mode = os.getenv("BASIC_MEMORY_CLOUD_MODE", "").lower() in ("1", "true")
if cloud_mode:
setup_logging(log_level=log_level, log_to_stdout=True, structured_context=True)
if "sync" in sys.argv:
return "sync"
elif "mcp" in sys.argv:
return "mcp"
elif "cli" in sys.argv:
return "cli"
else:
setup_logging(log_level=log_level, log_to_file=True)
return "api"
process_name = get_process_name()
# Global flag to track if logging has been set up
_LOGGING_SETUP = False
# Logging
def setup_basic_memory_logging(): # pragma: no cover
"""Set up logging for basic-memory, ensuring it only happens once."""
global _LOGGING_SETUP
if _LOGGING_SETUP:
# We can't log before logging is set up
# print("Skipping duplicate logging setup")
return
# Check for console logging environment variable - accept more truthy values
console_logging_env = os.getenv("BASIC_MEMORY_CONSOLE_LOGGING", "false").lower()
console_logging = console_logging_env in ("true", "1", "yes", "on")
# Check for log level environment variable first, fall back to config
log_level = os.getenv("BASIC_MEMORY_LOG_LEVEL")
if not log_level:
config_manager = ConfigManager()
log_level = config_manager.config.log_level
config_manager = ConfigManager()
config = get_project_config()
setup_logging(
env=config_manager.config.env,
home_dir=user_home, # Use user home for logs
log_level=log_level,
log_file=f"{DATA_DIR_NAME}/basic-memory-{process_name}.log",
console=console_logging,
)
logger.info(f"Basic Memory {basic_memory.__version__} (Project: {config.project})")
_LOGGING_SETUP = True
# Set up logging
setup_basic_memory_logging()
+4 -36
View File
@@ -1,6 +1,5 @@
import asyncio
import os
import sys
from contextlib import asynccontextmanager
from enum import Enum, auto
from pathlib import Path
@@ -24,21 +23,6 @@ from sqlalchemy.pool import NullPool
from basic_memory.repository.postgres_search_repository import PostgresSearchRepository
from basic_memory.repository.sqlite_search_repository import SQLiteSearchRepository
# -----------------------------------------------------------------------------
# Windows event loop policy
# -----------------------------------------------------------------------------
# On Windows, the default ProactorEventLoop has known rough edges with aiosqlite
# during shutdown/teardown (threads posting results to a loop that's closing),
# which can manifest as:
# - "RuntimeError: Event loop is closed"
# - "IndexError: pop from an empty deque"
#
# The SelectorEventLoop doesn't support subprocess operations, so code that uses
# asyncio.create_subprocess_shell() (like sync_service._quick_count_files) must
# detect Windows and use fallback implementations.
if sys.platform == "win32": # pragma: no cover
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
# Module level state
_engine: Optional[AsyncEngine] = None
_session_maker: Optional[async_sessionmaker[AsyncSession]] = None
@@ -206,37 +190,21 @@ def _create_sqlite_engine(db_url: str, db_type: DatabaseType) -> AsyncEngine:
return engine
def _create_postgres_engine(db_url: str, config: BasicMemoryConfig) -> AsyncEngine:
def _create_postgres_engine(db_url: str) -> AsyncEngine:
"""Create Postgres async engine with appropriate configuration.
Args:
db_url: Postgres connection URL (postgresql+asyncpg://...)
config: BasicMemoryConfig with pool settings
Returns:
Configured async engine for Postgres
"""
# Use NullPool connection issues.
# Assume connection pooler like PgBouncer handles connection pooling.
# Postgres with asyncpg - use standard async connection
engine = create_async_engine(
db_url,
echo=False,
poolclass=NullPool, # No pooling - fresh connection per request
connect_args={
# Disable statement cache to avoid issues with prepared statements on reconnect
"statement_cache_size": 0,
# Allow 30s for commands (Neon cold start can take 2-5s, sometimes longer)
"command_timeout": 30,
# Allow 30s for initial connection (Neon wake-up time)
"timeout": 30,
"server_settings": {
"application_name": "basic-memory",
# Statement timeout for queries (30s to allow for cold start)
"statement_timeout": "30s",
},
},
pool_pre_ping=True, # Verify connections before using them
)
logger.debug("Created Postgres engine with NullPool (no connection pooling)")
return engine
@@ -260,7 +228,7 @@ def _create_engine_and_session(
# Delegate to backend-specific engine creation
# Check explicit POSTGRES type first, then config setting
if db_type == DatabaseType.POSTGRES or config.database_backend == DatabaseBackend.POSTGRES:
engine = _create_postgres_engine(db_url, config)
engine = _create_postgres_engine(db_url)
else:
engine = _create_sqlite_engine(db_url, db_type)
+12 -22
View File
@@ -351,33 +351,28 @@ async def get_entity_parser_v2(project_config: ProjectConfigV2Dep) -> EntityPars
EntityParserV2Dep = Annotated["EntityParser", Depends(get_entity_parser_v2)]
async def get_markdown_processor(
entity_parser: EntityParserDep, app_config: AppConfigDep
) -> MarkdownProcessor:
return MarkdownProcessor(entity_parser, app_config=app_config)
async def get_markdown_processor(entity_parser: EntityParserDep) -> MarkdownProcessor:
return MarkdownProcessor(entity_parser)
MarkdownProcessorDep = Annotated[MarkdownProcessor, Depends(get_markdown_processor)]
async def get_markdown_processor_v2(
entity_parser: EntityParserV2Dep, app_config: AppConfigDep
) -> MarkdownProcessor:
return MarkdownProcessor(entity_parser, app_config=app_config)
async def get_markdown_processor_v2(entity_parser: EntityParserV2Dep) -> MarkdownProcessor:
return MarkdownProcessor(entity_parser)
MarkdownProcessorV2Dep = Annotated[MarkdownProcessor, Depends(get_markdown_processor_v2)]
async def get_file_service(
project_config: ProjectConfigDep,
markdown_processor: MarkdownProcessorDep,
app_config: AppConfigDep,
project_config: ProjectConfigDep, markdown_processor: MarkdownProcessorDep
) -> FileService:
file_service = FileService(project_config.home, markdown_processor, app_config=app_config)
logger.debug(
f"Created FileService for project: {project_config.name}, base_path: {project_config.home} "
f"Creating FileService for project: {project_config.name}, base_path: {project_config.home}"
)
file_service = FileService(project_config.home, markdown_processor)
logger.debug(f"Created FileService for project: {file_service} ")
return file_service
@@ -385,14 +380,13 @@ FileServiceDep = Annotated[FileService, Depends(get_file_service)]
async def get_file_service_v2(
project_config: ProjectConfigV2Dep,
markdown_processor: MarkdownProcessorV2Dep,
app_config: AppConfigDep,
project_config: ProjectConfigV2Dep, markdown_processor: MarkdownProcessorV2Dep
) -> FileService:
file_service = FileService(project_config.home, markdown_processor, app_config=app_config)
logger.debug(
f"Created FileService for project: {project_config.name}, base_path: {project_config.home}"
f"Creating FileService for project: {project_config.name}, base_path: {project_config.home}"
)
file_service = FileService(project_config.home, markdown_processor)
logger.debug(f"Created FileService for project: {file_service} ")
return file_service
@@ -406,7 +400,6 @@ async def get_entity_service(
entity_parser: EntityParserDep,
file_service: FileServiceDep,
link_resolver: "LinkResolverDep",
search_service: "SearchServiceDep",
app_config: AppConfigDep,
) -> EntityService:
"""Create EntityService with repository."""
@@ -417,7 +410,6 @@ async def get_entity_service(
entity_parser=entity_parser,
file_service=file_service,
link_resolver=link_resolver,
search_service=search_service,
app_config=app_config,
)
@@ -432,7 +424,6 @@ async def get_entity_service_v2(
entity_parser: EntityParserV2Dep,
file_service: FileServiceV2Dep,
link_resolver: "LinkResolverV2Dep",
search_service: "SearchServiceV2Dep",
app_config: AppConfigDep,
) -> EntityService:
"""Create EntityService for v2 API."""
@@ -443,7 +434,6 @@ async def get_entity_service_v2(
entity_parser=entity_parser,
file_service=file_service,
link_resolver=link_resolver,
search_service=search_service,
app_config=app_config,
)
+3 -212
View File
@@ -1,13 +1,9 @@
"""Utilities for file operations."""
import asyncio
import hashlib
import shlex
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
import re
from typing import TYPE_CHECKING, Any, Dict, Optional, Union
from typing import Any, Dict, Union
import aiofiles
import yaml
@@ -16,23 +12,6 @@ from loguru import logger
from basic_memory.utils import FilePath
if TYPE_CHECKING:
from basic_memory.config import BasicMemoryConfig
@dataclass
class FileMetadata:
"""File metadata for cloud-compatible file operations.
This dataclass provides a cloud-agnostic way to represent file metadata,
enabling S3FileService to return metadata from head_object responses
instead of mock stat_result with zeros.
"""
size: int
created_at: datetime
modified_at: datetime
class FileError(Exception):
"""Base exception for file operations."""
@@ -74,28 +53,6 @@ async def compute_checksum(content: Union[str, bytes]) -> str:
raise FileError(f"Failed to compute checksum: {e}")
# UTF-8 BOM character that can appear at the start of files
UTF8_BOM = "\ufeff"
def strip_bom(content: str) -> str:
"""Strip UTF-8 BOM from the start of content if present.
BOM (Byte Order Mark) characters can be present in files created on Windows
or copied from certain sources. They should be stripped before processing
frontmatter. See issue #452.
Args:
content: Content that may start with BOM
Returns:
Content with BOM removed if present
"""
if content and content.startswith(UTF8_BOM):
return content[1:]
return content
async def write_file_atomic(path: FilePath, content: str) -> None:
"""
Write file with atomic operation using temporary file.
@@ -127,168 +84,6 @@ async def write_file_atomic(path: FilePath, content: str) -> None:
raise FileWriteError(f"Failed to write file {path}: {e}")
async def format_markdown_builtin(path: Path) -> Optional[str]:
"""
Format a markdown file using the built-in mdformat formatter.
Uses mdformat with GFM (GitHub Flavored Markdown) support for consistent
formatting without requiring Node.js or external tools.
Args:
path: Path to the markdown file to format
Returns:
Formatted content if successful, None if formatting failed.
"""
try:
import mdformat
except ImportError:
logger.warning(
"mdformat not installed, skipping built-in formatting",
path=str(path),
)
return None
try:
# Read original content
async with aiofiles.open(path, mode="r", encoding="utf-8") as f:
content = await f.read()
# Format using mdformat with GFM and frontmatter extensions
# mdformat is synchronous, so we run it in a thread executor
loop = asyncio.get_event_loop()
formatted_content = await loop.run_in_executor(
None,
lambda: mdformat.text(
content,
extensions={"gfm", "frontmatter"}, # GFM + YAML frontmatter support
options={"wrap": "no"}, # Don't wrap lines
),
)
# Only write if content changed
if formatted_content != content:
async with aiofiles.open(path, mode="w", encoding="utf-8") as f:
await f.write(formatted_content)
logger.debug(
"Formatted file with mdformat",
path=str(path),
changed=formatted_content != content,
)
return formatted_content
except Exception as e:
logger.warning(
"mdformat formatting failed",
path=str(path),
error=str(e),
)
return None
async def format_file(
path: Path,
config: "BasicMemoryConfig",
is_markdown: bool = False,
) -> Optional[str]:
"""
Format a file using configured formatter.
By default, uses the built-in mdformat formatter for markdown files (pure Python,
no Node.js required). External formatters like Prettier can be configured via
formatter_command or per-extension formatters.
Args:
path: File to format
config: Configuration with formatter settings
is_markdown: Whether this is a markdown file (caller should use FileService.is_markdown)
Returns:
Formatted content if successful, None if formatting was skipped or failed.
Failures are logged as warnings but don't raise exceptions.
"""
if not config.format_on_save:
return None
extension = path.suffix.lstrip(".")
formatter = config.formatters.get(extension) or config.formatter_command
# Use built-in mdformat for markdown files when no external formatter configured
if not formatter:
if is_markdown:
return await format_markdown_builtin(path)
else:
logger.debug("No formatter configured for extension", extension=extension)
return None
# Use external formatter
# Replace {file} placeholder with the actual path
cmd = formatter.replace("{file}", str(path))
try:
# Parse command into args list for safer execution (no shell=True)
args = shlex.split(cmd)
proc = await asyncio.create_subprocess_exec(
*args,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
try:
stdout, stderr = await asyncio.wait_for(
proc.communicate(),
timeout=config.formatter_timeout,
)
except asyncio.TimeoutError:
proc.kill()
await proc.wait()
logger.warning(
"Formatter timed out",
path=str(path),
timeout=config.formatter_timeout,
)
return None
if proc.returncode != 0:
logger.warning(
"Formatter exited with non-zero status",
path=str(path),
returncode=proc.returncode,
stderr=stderr.decode("utf-8", errors="replace") if stderr else "",
)
# Still try to read the file - formatter may have partially worked
# or the file may be unchanged
# Read formatted content
async with aiofiles.open(path, mode="r", encoding="utf-8") as f:
formatted_content = await f.read()
logger.debug(
"Formatted file successfully",
path=str(path),
formatter=args[0] if args else formatter,
)
return formatted_content
except FileNotFoundError:
# Formatter executable not found
logger.warning(
"Formatter executable not found",
command=cmd.split()[0] if cmd else "",
path=str(path),
)
return None
except Exception as e:
logger.warning(
"Formatter failed",
path=str(path),
error=str(e),
)
return None
def has_frontmatter(content: str) -> bool:
"""
Check if content contains valid YAML frontmatter.
@@ -302,8 +97,7 @@ def has_frontmatter(content: str) -> bool:
if not content:
return False
# Strip BOM before checking for frontmatter markers
content = strip_bom(content).strip()
content = content.strip()
if not content.startswith("---"):
return False
@@ -324,8 +118,6 @@ def parse_frontmatter(content: str) -> Dict[str, Any]:
ParseError: If frontmatter is invalid or parsing fails
"""
try:
# Strip BOM before parsing frontmatter
content = strip_bom(content)
if not content.strip().startswith("---"):
raise ParseError("Content has no frontmatter")
@@ -367,8 +159,7 @@ def remove_frontmatter(content: str) -> str:
Raises:
ParseError: If content starts with frontmatter marker but is malformed
"""
# Strip BOM before processing
content = strip_bom(content).strip()
content = content.strip()
# Return as-is if no frontmatter marker
if not content.startswith("---"):
+2 -5
View File
@@ -5,18 +5,15 @@ from datetime import datetime
from typing import Any
def clean_filename(name: str | None) -> str: # pragma: no cover
def clean_filename(name: str) -> str: # pragma: no cover
"""Clean a string to be used as a filename.
Args:
name: The string to clean (can be None).
name: The string to clean.
Returns:
A cleaned string suitable for use as a filename.
"""
# Handle None or empty input
if not name:
return "untitled"
# Replace common punctuation and whitespace with underscores
name = re.sub(r"[\s\-,.:/\\\[\]\(\)]+", "_", name)
# Remove any non-alphanumeric or underscore characters
@@ -23,7 +23,6 @@ from basic_memory.markdown.schemas import (
)
from basic_memory.utils import parse_tags
md = MarkdownIt().use(observation_plugin).use(relation_plugin)
@@ -227,12 +226,6 @@ class EntityParser:
Returns:
EntityMarkdown with parsed content
"""
# Strip BOM before parsing (can be present in files from Windows or certain sources)
# See issue #452
from basic_memory.file_utils import strip_bom
content = strip_bom(content)
# Parse frontmatter with proper error handling for malformed YAML
try:
post = frontmatter.loads(content)
@@ -1,19 +1,15 @@
from pathlib import Path
from typing import TYPE_CHECKING, Optional
from typing import Optional
from collections import OrderedDict
from frontmatter import Post
from loguru import logger
from basic_memory import file_utils
from basic_memory.file_utils import dump_frontmatter
from basic_memory.markdown.entity_parser import EntityParser
from basic_memory.markdown.schemas import EntityMarkdown, Observation, Relation
if TYPE_CHECKING:
from basic_memory.config import BasicMemoryConfig
class DirtyFileError(Exception):
"""Raised when attempting to write to a file that has been modified."""
@@ -39,14 +35,9 @@ class MarkdownProcessor:
3. Track schema changes (that's done by the database)
"""
def __init__(
self,
entity_parser: EntityParser,
app_config: Optional["BasicMemoryConfig"] = None,
):
"""Initialize processor with parser and optional config."""
def __init__(self, entity_parser: EntityParser):
"""Initialize processor with base path and parser."""
self.entity_parser = entity_parser
self.app_config = app_config
async def read_file(self, path: Path) -> EntityMarkdown:
"""Read and parse file into EntityMarkdown schema.
@@ -131,17 +122,7 @@ class MarkdownProcessor:
# Write atomically and return checksum of updated file
path.parent.mkdir(parents=True, exist_ok=True)
await file_utils.write_file_atomic(path, final_content)
# Format file if configured (MarkdownProcessor always handles markdown files)
content_for_checksum = final_content
if self.app_config:
formatted_content = await file_utils.format_file(
path, self.app_config, is_markdown=True
)
if formatted_content is not None:
content_for_checksum = formatted_content
return await file_utils.compute_checksum(content_for_checksum)
return await file_utils.compute_checksum(final_content)
def format_observations(self, observations: list[Observation]) -> str:
"""Format observations section in standard way.
+2 -4
View File
@@ -30,9 +30,7 @@ def is_observation(token: Token) -> bool:
# Check for proper observation format: [category] content
match = re.match(r"^\[([^\[\]()]+)\]\s+(.+)", content)
# Check for standalone hashtags (words starting with #)
# This excludes # in HTML attributes like color="#4285F4"
has_tags = any(part.startswith("#") for part in content.split())
has_tags = "#" in content
return bool(match) or has_tags
@@ -162,7 +160,7 @@ def parse_inline_relations(content: str) -> List[Dict[str, Any]]:
target = content[start + 2 : end].strip()
if target:
relations.append({"type": "links_to", "target": target, "context": None})
relations.append({"type": "links to", "target": target, "context": None})
start = end + 2
+1 -10
View File
@@ -3,7 +3,6 @@
from pathlib import Path
from typing import Any, Optional
from frontmatter import Post
from basic_memory.file_utils import has_frontmatter, remove_frontmatter, parse_frontmatter
@@ -13,10 +12,7 @@ from basic_memory.models import Observation as ObservationModel
def entity_model_from_markdown(
file_path: Path,
markdown: EntityMarkdown,
entity: Optional[Entity] = None,
project_id: Optional[int] = None,
file_path: Path, markdown: EntityMarkdown, entity: Optional[Entity] = None
) -> Entity:
"""
Convert markdown entity to model. Does not include relations.
@@ -25,7 +21,6 @@ def entity_model_from_markdown(
file_path: Path to the markdown file
markdown: Parsed markdown entity
entity: Optional existing entity to update
project_id: Project ID for new observations (uses entity.project_id if not provided)
Returns:
Entity model populated from markdown
@@ -55,13 +50,9 @@ def entity_model_from_markdown(
metadata = markdown.frontmatter.metadata or {}
model.entity_metadata = {k: str(v) for k, v in metadata.items() if v is not None}
# Get project_id from entity if not provided
obs_project_id = project_id or (model.project_id if hasattr(model, "project_id") else None)
# Convert observations
model.observations = [
ObservationModel(
project_id=obs_project_id,
content=obs.content,
category=obs.category,
context=obs.context,
-1
View File
@@ -95,7 +95,6 @@ async def get_client() -> AsyncIterator[AsyncClient]:
yield client
else:
# Local mode: ASGI transport for in-process calls
# Note: ASGI transport does NOT trigger FastAPI lifespan, so no special handling needed
logger.info("Creating ASGI client for local Basic Memory API")
async with AsyncClient(
transport=ASGITransport(app=fastapi_app), base_url="http://test", timeout=timeout
-72
View File
@@ -2,80 +2,8 @@
Basic Memory FastMCP server.
"""
import asyncio
from contextlib import asynccontextmanager
from fastmcp import FastMCP
from loguru import logger
from basic_memory import db
from basic_memory.config import ConfigManager
from basic_memory.services.initialization import initialize_app, initialize_file_sync
from basic_memory.telemetry import show_notice_if_needed, track_app_started
@asynccontextmanager
async def lifespan(app: FastMCP):
"""Lifecycle manager for the MCP server.
Handles:
- Database initialization and migrations
- Telemetry notice and tracking
- File sync in background (if enabled and not in cloud mode)
- Proper cleanup on shutdown
"""
app_config = ConfigManager().config
logger.info("Starting Basic Memory MCP server")
# Show telemetry notice (first run only) and track startup
show_notice_if_needed()
track_app_started("mcp")
# Track if we created the engine (vs test fixtures providing it)
# This prevents disposing an engine provided by test fixtures when
# multiple Client connections are made in the same test
engine_was_none = db._engine is None
# Initialize app (runs migrations, reconciles projects)
await initialize_app(app_config)
# Start file sync as background task (if enabled and not in cloud mode)
sync_task = None
if app_config.is_test_env:
logger.info("Test environment detected - skipping local file sync")
elif app_config.sync_changes and not app_config.cloud_mode_enabled:
logger.info("Starting file sync in background")
async def _file_sync_runner() -> None:
await initialize_file_sync(app_config)
sync_task = asyncio.create_task(_file_sync_runner())
elif app_config.cloud_mode_enabled:
logger.info("Cloud mode enabled - skipping local file sync")
else:
logger.info("Sync changes disabled - skipping file sync")
try:
yield
finally:
# Shutdown
logger.info("Shutting down Basic Memory MCP server")
if sync_task:
sync_task.cancel()
try:
await sync_task
except asyncio.CancelledError:
logger.info("File sync task cancelled")
# Only shutdown DB if we created it (not if test fixture provided it)
if engine_was_none:
await db.shutdown_db()
logger.info("Database connections closed")
else:
logger.debug("Skipping DB shutdown - engine provided externally")
mcp = FastMCP(
name="Basic Memory",
lifespan=lifespan,
)
+3 -3
View File
@@ -9,7 +9,6 @@ from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.project_context import get_active_project
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_get
from basic_memory.telemetry import track_mcp_tool
from basic_memory.schemas.base import TimeFrame
from basic_memory.schemas.memory import (
GraphContext,
@@ -88,7 +87,6 @@ async def build_context(
Raises:
ToolError: If project doesn't exist or depth parameter is invalid
"""
track_mcp_tool("build_context")
logger.info(f"Building context from {url} in project {project}")
# Convert string depth to integer if needed
@@ -106,9 +104,11 @@ async def build_context(
# Get the active project using the new stateless approach
active_project = await get_active_project(client, project, context)
project_url = active_project.project_url
response = await call_get(
client,
f"/v2/projects/{active_project.id}/memory/{memory_url_path(url)}",
f"{project_url}/memory/{memory_url_path(url)}",
params={
"depth": depth,
"timeframe": timeframe,
+12 -34
View File
@@ -12,8 +12,7 @@ from fastmcp import Context
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.project_context import get_active_project
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_put, call_post, resolve_entity_id
from basic_memory.telemetry import track_mcp_tool
from basic_memory.mcp.tools.utils import call_put
@mcp.tool(
@@ -95,9 +94,9 @@ async def canvas(
Raises:
ToolError: If project doesn't exist or folder path is invalid
"""
track_mcp_tool("canvas")
async with get_client() as client:
active_project = await get_active_project(client, project, context)
project_url = active_project.project_url
# Ensure path has .canvas extension
file_title = title if title.endswith(".canvas") else f"{title}.canvas"
@@ -109,44 +108,23 @@ async def canvas(
# Convert to JSON
canvas_json = json.dumps(canvas_data, indent=2)
# Try to create the canvas file first (optimistic create)
# Write the file using the resource API
logger.info(f"Creating canvas file: {file_path} in project {project}")
try:
response = await call_post(
client,
f"/v2/projects/{active_project.id}/resource",
json={"file_path": file_path, "content": canvas_json},
)
action = "Created"
except Exception as e:
# If creation failed due to conflict (already exists), try to update
if (
"409" in str(e)
or "conflict" in str(e).lower()
or "already exists" in str(e).lower()
):
logger.info(f"Canvas file exists, updating instead: {file_path}")
try:
entity_id = await resolve_entity_id(client, active_project.id, file_path)
# For update, send content in JSON body
response = await call_put(
client,
f"/v2/projects/{active_project.id}/resource/{entity_id}",
json={"content": canvas_json},
)
action = "Updated"
except Exception as update_error:
# Re-raise the original error if update also fails
raise e from update_error
else:
# Re-raise if it's not a conflict error
raise
# Send canvas_json as content string, not as json parameter
# The resource endpoint expects Body() string content, not JSON-encoded data
response = await call_put(
client,
f"{project_url}/resource/{file_path}",
content=canvas_json,
headers={"Content-Type": "text/plain"},
)
# Parse response
result = response.json()
logger.debug(result)
# Build summary
action = "Created" if response.status_code == 201 else "Updated"
summary = [f"# {action}: {file_path}", "\nThe canvas is ready to open in Obsidian."]
return "\n".join(summary)
@@ -15,7 +15,6 @@ from basic_memory.mcp.tools.search import search_notes
from basic_memory.mcp.tools.read_note import read_note
from basic_memory.schemas.search import SearchResponse
from basic_memory.config import ConfigManager
from basic_memory.telemetry import track_mcp_tool
def _format_search_results_for_chatgpt(results: SearchResponse) -> List[Dict[str, Any]]:
@@ -89,7 +88,6 @@ async def search(
List with one dict: `{ "type": "text", "text": "{...JSON...}" }`
where the JSON body contains `results`, `total_count`, and echo of `query`.
"""
track_mcp_tool("search")
logger.info(f"ChatGPT search request: query='{query}'")
try:
@@ -153,7 +151,6 @@ async def fetch(
List with one dict: `{ "type": "text", "text": "{...JSON...}" }`
where the JSON body includes `id`, `title`, `text`, `url`, and metadata.
"""
track_mcp_tool("fetch")
logger.info(f"ChatGPT fetch request: id='{id}'")
try:
+3 -20
View File
@@ -3,13 +3,11 @@ from typing import Optional
from loguru import logger
from fastmcp import Context
from mcp.server.fastmcp.exceptions import ToolError
from basic_memory.mcp.project_context import get_active_project
from basic_memory.mcp.tools.utils import call_delete, resolve_entity_id
from basic_memory.mcp.tools.utils import call_delete
from basic_memory.mcp.server import mcp
from basic_memory.mcp.async_client import get_client
from basic_memory.telemetry import track_mcp_tool
from basic_memory.schemas import DeleteEntitiesResponse
@@ -204,27 +202,12 @@ async def delete_note(
with suggestions for finding the correct identifier, including search
commands and alternative formats to try.
"""
track_mcp_tool("delete_note")
async with get_client() as client:
active_project = await get_active_project(client, project, context)
project_url = active_project.project_url
try:
# Resolve identifier to entity ID
entity_id = await resolve_entity_id(client, active_project.id, identifier)
except ToolError as e:
# If entity not found, return False (note doesn't exist)
if "Entity not found" in str(e) or "not found" in str(e).lower():
logger.warning(f"Note not found for deletion: {identifier}")
return False
# For other resolution errors, return formatted error message
logger.error(f"Delete failed for '{identifier}': {e}, project: {active_project.name}")
return _format_delete_error_response(active_project.name, str(e), identifier)
try:
# Call the DELETE endpoint
response = await call_delete(
client, f"/v2/projects/{active_project.id}/knowledge/entities/{entity_id}"
)
response = await call_delete(client, f"{project_url}/knowledge/entities/{identifier}")
result = DeleteEntitiesResponse.model_validate(response.json())
if result.deleted:
+3 -7
View File
@@ -8,8 +8,7 @@ from fastmcp import Context
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.project_context import get_active_project, add_project_metadata
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_patch, resolve_entity_id
from basic_memory.telemetry import track_mcp_tool
from basic_memory.mcp.tools.utils import call_patch
from basic_memory.schemas import EntityResponse
@@ -215,9 +214,9 @@ async def edit_note(
search_notes() first to find the correct identifier. The tool provides detailed
error messages with suggestions if operations fail.
"""
track_mcp_tool("edit_note")
async with get_client() as client:
active_project = await get_active_project(client, project, context)
project_url = active_project.project_url
logger.info("MCP tool call", tool="edit_note", identifier=identifier, operation=operation)
@@ -236,9 +235,6 @@ async def edit_note(
# Use the PATCH endpoint to edit the entity
try:
# Resolve identifier to entity ID
entity_id = await resolve_entity_id(client, active_project.id, identifier)
# Prepare the edit request data
edit_data = {
"operation": operation,
@@ -254,7 +250,7 @@ async def edit_note(
edit_data["expected_replacements"] = str(expected_replacements)
# Call the PATCH endpoint
url = f"/v2/projects/{active_project.id}/knowledge/entities/{entity_id}"
url = f"{project_url}/knowledge/entities/{identifier}"
response = await call_patch(client, url, json=edit_data)
result = EntityResponse.model_validate(response.json())
+2 -3
View File
@@ -9,7 +9,6 @@ from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.project_context import get_active_project
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_get
from basic_memory.telemetry import track_mcp_tool
@mcp.tool(
@@ -64,9 +63,9 @@ async def list_directory(
Raises:
ToolError: If project doesn't exist or directory path is invalid
"""
track_mcp_tool("list_directory")
async with get_client() as client:
active_project = await get_active_project(client, project, context)
project_url = active_project.project_url
# Prepare query parameters
params = {
@@ -83,7 +82,7 @@ async def list_directory(
# Call the API endpoint
response = await call_get(
client,
f"/v2/projects/{active_project.id}/directory/list",
f"{project_url}/directory/list",
params=params,
)
+10 -16
View File
@@ -8,11 +8,10 @@ from fastmcp import Context
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_get, call_put, resolve_entity_id
from basic_memory.mcp.tools.utils import call_post, call_get
from basic_memory.mcp.project_context import get_active_project
from basic_memory.schemas import EntityResponse
from basic_memory.schemas.project_info import ProjectList
from basic_memory.telemetry import track_mcp_tool
from basic_memory.utils import validate_project_path
@@ -396,11 +395,11 @@ async def move_note(
- Re-indexes the entity for search
- Maintains all observations and relations
"""
track_mcp_tool("move_note")
async with get_client() as client:
logger.debug(f"Moving note: {identifier} to {destination_path} in project: {project}")
active_project = await get_active_project(client, project, context)
project_url = active_project.project_url
# Validate destination path to prevent path traversal attacks
project_path = active_project.home
@@ -435,10 +434,8 @@ move_note("{identifier}", "notes/{destination_path.split("/")[-1] if "/" in dest
# Get the source entity information for extension validation
source_ext = "md" # Default to .md if we can't determine source extension
try:
# Resolve identifier to entity ID
entity_id = await resolve_entity_id(client, active_project.id, identifier)
# Fetch source entity information to get the current file extension
url = f"/v2/projects/{active_project.id}/knowledge/entities/{entity_id}"
url = f"{project_url}/knowledge/entities/{identifier}"
response = await call_get(client, url)
source_entity = EntityResponse.model_validate(response.json())
if "." in source_entity.file_path:
@@ -470,10 +467,8 @@ move_note("{identifier}", "notes/{destination_path.split("/")[-1] if "/" in dest
# Get the source entity to check its file extension
try:
# Resolve identifier to entity ID (might already be cached from above)
entity_id = await resolve_entity_id(client, active_project.id, identifier)
# Fetch source entity information
url = f"/v2/projects/{active_project.id}/knowledge/entities/{entity_id}"
url = f"{project_url}/knowledge/entities/{identifier}"
response = await call_get(client, url)
source_entity = EntityResponse.model_validate(response.json())
@@ -510,17 +505,16 @@ move_note("{identifier}", "notes/{destination_path.split("/")[-1] if "/" in dest
logger.debug(f"Could not fetch source entity for extension check: {e}")
try:
# Resolve identifier to entity ID for the move operation
entity_id = await resolve_entity_id(client, active_project.id, identifier)
# Prepare move request (v2 API only needs destination_path)
# Prepare move request
move_data = {
"identifier": identifier,
"destination_path": destination_path,
"project": active_project.name,
}
# Call the v2 move API endpoint (PUT method, entity_id in URL)
url = f"/v2/projects/{active_project.id}/knowledge/entities/{entity_id}/move"
response = await call_put(client, url, json=move_data)
# Call the move API endpoint
url = f"{project_url}/knowledge/move"
response = await call_post(client, url, json=move_data)
result = EntityResponse.model_validate(response.json())
# Build success message
@@ -15,7 +15,6 @@ from basic_memory.schemas.project_info import (
ProjectStatusResponse,
ProjectInfoRequest,
)
from basic_memory.telemetry import track_mcp_tool
from basic_memory.utils import generate_permalink
@@ -41,7 +40,6 @@ async def list_memory_projects(context: Context | None = None) -> str:
Example:
list_memory_projects()
"""
track_mcp_tool("list_memory_projects")
async with get_client() as client:
if context: # pragma: no cover
await context.info("Listing all available projects")
@@ -94,7 +92,6 @@ async def create_memory_project(
create_memory_project("my-research", "~/Documents/research")
create_memory_project("work-notes", "/home/user/work", set_default=True)
"""
track_mcp_tool("create_memory_project")
async with get_client() as client:
# Check if server is constrained to a specific project
constrained_project = os.environ.get("BASIC_MEMORY_MCP_PROJECT")
@@ -150,7 +147,6 @@ async def delete_project(project_name: str, context: Context | None = None) -> s
This action cannot be undone. The project will need to be re-added
to access its content through Basic Memory again.
"""
track_mcp_tool("delete_project")
async with get_client() as client:
# Check if server is constrained to a specific project
constrained_project = os.environ.get("BASIC_MEMORY_MCP_PROJECT")
@@ -183,8 +179,11 @@ async def delete_project(project_name: str, context: Context | None = None) -> s
f"Project '{project_name}' not found. Available projects: {', '.join(available_projects)}"
)
# Call v2 API to delete project using project ID
response = await call_delete(client, f"/v2/projects/{target_project.id}")
# Call API to delete project using URL encoding for special characters
from urllib.parse import quote
encoded_name = quote(target_project.name, safe="")
response = await call_delete(client, f"/projects/{encoded_name}")
status_response = ProjectStatusResponse.model_validate(response.json())
result = f"{status_response.message}\n\n"
+3 -13
View File
@@ -13,14 +13,12 @@ from typing import Optional
from loguru import logger
from PIL import Image as PILImage
from fastmcp import Context
from mcp.server.fastmcp.exceptions import ToolError
from basic_memory.mcp.project_context import get_active_project
from basic_memory.mcp.server import mcp
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.tools.utils import call_get, resolve_entity_id
from basic_memory.mcp.tools.utils import call_get
from basic_memory.schemas.memory import memory_url_path
from basic_memory.telemetry import track_mcp_tool
from basic_memory.utils import validate_project_path
@@ -201,11 +199,11 @@ async def read_content(
HTTPError: If project doesn't exist or is inaccessible
SecurityError: If path attempts path traversal
"""
track_mcp_tool("read_content")
logger.info("Reading file", path=path, project=project)
async with get_client() as client:
active_project = await get_active_project(client, project, context)
project_url = active_project.project_url
url = memory_url_path(path)
@@ -223,15 +221,7 @@ async def read_content(
"error": f"Path '{path}' is not allowed - paths must stay within project boundaries",
}
# Resolve path to entity ID
try:
entity_id = await resolve_entity_id(client, active_project.id, url)
except ToolError:
# Convert resolution errors to "Resource not found" for consistency
raise ToolError(f"Resource not found: {url}")
# Call the v2 resource endpoint
response = await call_get(client, f"/v2/projects/{active_project.id}/resource/{entity_id}")
response = await call_get(client, f"{project_url}/resource/{url}")
content_type = response.headers.get("content-type", "application/octet-stream")
content_length = int(response.headers.get("content-length", 0))
+12 -24
View File
@@ -10,8 +10,7 @@ from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.project_context import get_active_project
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.search import search_notes
from basic_memory.mcp.tools.utils import call_get, resolve_entity_id
from basic_memory.telemetry import track_mcp_tool
from basic_memory.mcp.tools.utils import call_get
from basic_memory.schemas.memory import memory_url_path
from basic_memory.utils import validate_project_path
@@ -78,7 +77,6 @@ async def read_note(
If the exact note isn't found, this tool provides helpful suggestions
including related notes, search commands, and note creation templates.
"""
track_mcp_tool("read_note")
async with get_client() as client:
# Get and validate the project
active_project = await get_active_project(client, project, context)
@@ -99,29 +97,23 @@ async def read_note(
)
return f"# Error\n\nIdentifier '{identifier}' is not allowed - paths must stay within project boundaries"
# Get the file via REST API - first try direct identifier resolution
project_url = active_project.project_url
# Get the file via REST API - first try direct permalink lookup
entity_path = memory_url_path(identifier)
logger.info(
f"Attempting to read note from Project: {active_project.name} identifier: {entity_path}"
)
path = f"{project_url}/resource/{entity_path}"
logger.info(f"Attempting to read note from Project: {active_project.name} URL: {path}")
try:
# Try to resolve identifier to entity ID
entity_id = await resolve_entity_id(client, active_project.id, entity_path)
# Fetch content using entity ID
response = await call_get(
client,
f"/v2/projects/{active_project.id}/resource/{entity_id}",
params={"page": page, "page_size": page_size},
)
# Try direct lookup first
response = await call_get(client, path, params={"page": page, "page_size": page_size})
# If successful, return the content
if response.status_code == 200:
logger.info("Returning read_note result from resource: {path}", path=entity_path)
return response.text
except Exception as e: # pragma: no cover
logger.info(f"Direct lookup failed for '{entity_path}': {e}")
logger.info(f"Direct lookup failed for '{path}': {e}")
# Continue to fallback methods
# Fallback 1: Try title search via API
@@ -135,14 +127,10 @@ async def read_note(
result = title_results.results[0] # Get the first/best match
if result.permalink:
try:
# Resolve the permalink to entity ID
entity_id = await resolve_entity_id(client, active_project.id, result.permalink)
# Fetch content using the entity ID
# Try to fetch the content using the found permalink
path = f"{project_url}/resource/{result.permalink}"
response = await call_get(
client,
f"/v2/projects/{active_project.id}/resource/{entity_id}",
params={"page": page, "page_size": page_size},
client, path, params={"page": page, "page_size": page_size}
)
if response.status_code == 200:
@@ -9,7 +9,6 @@ from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.project_context import get_active_project, resolve_project_parameter
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_get
from basic_memory.telemetry import track_mcp_tool
from basic_memory.schemas.base import TimeFrame
from basic_memory.schemas.memory import (
GraphContext,
@@ -99,7 +98,6 @@ async def recent_activity(
- For focused queries, consider using build_context with a specific URI
- Max timeframe is 1 year in the past
"""
track_mcp_tool("recent_activity")
async with get_client() as client:
# Build common parameters for API calls
params = {
@@ -249,10 +247,11 @@ async def recent_activity(
)
active_project = await get_active_project(client, resolved_project, context)
project_url = active_project.project_url
response = await call_get(
client,
f"/v2/projects/{active_project.id}/memory/recent",
f"{project_url}/memory/recent",
params=params,
)
activity_data = GraphContext.model_validate(response.json())
@@ -275,9 +274,10 @@ async def _get_project_activity(
Returns:
ProjectActivity with activity data or empty activity on error
"""
project_url = f"/{project_info.permalink}"
activity_response = await call_get(
client,
f"/v2/projects/{project_info.id}/memory/recent",
f"{project_url}/memory/recent",
params=params,
)
activity = GraphContext.model_validate(activity_response.json())
+2 -3
View File
@@ -10,7 +10,6 @@ from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.project_context import get_active_project
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_post
from basic_memory.telemetry import track_mcp_tool
from basic_memory.schemas.search import SearchItemType, SearchQuery, SearchResponse
@@ -331,7 +330,6 @@ async def search_notes(
# Explicit project specification
results = await search_notes("project planning", project="my-project")
"""
track_mcp_tool("search_notes")
# Create a SearchQuery object based on the parameters
search_query = SearchQuery()
@@ -357,13 +355,14 @@ async def search_notes(
async with get_client() as client:
active_project = await get_active_project(client, project, context)
project_url = active_project.project_url
logger.info(f"Searching for {search_query} in project {active_project.name}")
try:
response = await call_post(
client,
f"/v2/projects/{active_project.id}/search/",
f"{project_url}/search/",
json=search_query.model_dump(),
params={"page": page, "page_size": page_size},
)
-28
View File
@@ -435,34 +435,6 @@ async def call_post(
raise ToolError(error_message) from e
async def resolve_entity_id(client: AsyncClient, project_id: int, identifier: str) -> int:
"""Resolve a string identifier to an entity ID using the v2 API.
Args:
client: HTTP client for API calls
project_id: Project ID
identifier: The identifier to resolve (permalink, title, or path)
Returns:
The resolved entity ID
Raises:
ToolError: If the identifier cannot be resolved
"""
try:
response = await call_post(
client, f"/v2/projects/{project_id}/knowledge/resolve", json={"identifier": identifier}
)
data = response.json()
return data["entity_id"]
except HTTPStatusError as e:
if e.response.status_code == 404:
raise ToolError(f"Entity not found: '{identifier}'")
raise ToolError(f"Error resolving identifier '{identifier}': {e}")
except Exception as e:
raise ToolError(f"Unexpected error resolving identifier '{identifier}': {e}")
async def call_delete(
client: AsyncClient,
url: URL | str,
+1 -2
View File
@@ -8,7 +8,6 @@ from fastmcp import Context
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.read_note import read_note
from basic_memory.telemetry import track_mcp_tool
@mcp.tool(
@@ -55,7 +54,7 @@ async def view_note(
HTTPError: If project doesn't exist or is inaccessible
SecurityError: If identifier attempts path traversal
"""
track_mcp_tool("view_note")
logger.info(f"Viewing note: {identifier} in project: {project}")
# Call the existing read_note logic
+10 -33
View File
@@ -7,8 +7,7 @@ from loguru import logger
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.project_context import get_active_project, add_project_metadata
from basic_memory.mcp.server import mcp
from basic_memory.mcp.tools.utils import call_put, call_post, resolve_entity_id
from basic_memory.telemetry import track_mcp_tool
from basic_memory.mcp.tools.utils import call_put
from basic_memory.schemas import EntityResponse
from fastmcp import Context
from basic_memory.schemas.base import Entity
@@ -117,7 +116,6 @@ async def write_note(
HTTPError: If project doesn't exist or is inaccessible
SecurityError: If folder path attempts path traversal
"""
track_mcp_tool("write_note")
async with get_client() as client:
logger.info(
f"MCP tool call tool=write_note project={project} folder={folder}, title={title}, tags={tags}"
@@ -152,37 +150,16 @@ async def write_note(
content=content,
entity_metadata=metadata,
)
project_url = active_project.permalink
# Try to create the entity first (optimistic create)
logger.debug(f"Attempting to create entity permalink={entity.permalink}")
action = "Created" # Default to created
try:
url = f"/v2/projects/{active_project.id}/knowledge/entities"
response = await call_post(client, url, json=entity.model_dump())
result = EntityResponse.model_validate(response.json())
action = "Created"
except Exception as e:
# If creation failed due to conflict (already exists), try to update
if (
"409" in str(e)
or "conflict" in str(e).lower()
or "already exists" in str(e).lower()
):
logger.debug(f"Entity exists, updating instead permalink={entity.permalink}")
try:
if not entity.permalink:
raise ValueError("Entity permalink is required for updates")
entity_id = await resolve_entity_id(client, active_project.id, entity.permalink)
url = f"/v2/projects/{active_project.id}/knowledge/entities/{entity_id}"
response = await call_put(client, url, json=entity.model_dump())
result = EntityResponse.model_validate(response.json())
action = "Updated"
except Exception as update_error:
# Re-raise the original error if update also fails
raise e from update_error
else:
# Re-raise if it's not a conflict error
raise
# Create or update via knowledge API
logger.debug(f"Creating entity via API permalink={entity.permalink}")
url = f"{project_url}/knowledge/entities/{entity.permalink}"
response = await call_put(client, url, json=entity.model_dump())
result = EntityResponse.model_validate(response.json())
# Format semantic summary based on status code
action = "Created" if response.status_code == 201 else "Updated"
summary = [
f"# {action} note",
f"project: {active_project.name}",
+2
View File
@@ -4,6 +4,7 @@ import basic_memory
from basic_memory.models.base import Base
from basic_memory.models.knowledge import Entity, Observation, Relation
from basic_memory.models.project import Project
from basic_memory.models.search import SearchIndex
__all__ = [
"Base",
@@ -11,5 +12,6 @@ __all__ = [
"Observation",
"Relation",
"Project",
"SearchIndex",
"basic_memory",
]
+1 -8
View File
@@ -145,7 +145,6 @@ class Observation(Base):
)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
project_id: Mapped[int] = mapped_column(Integer, ForeignKey("project.id"), index=True)
entity_id: Mapped[int] = mapped_column(Integer, ForeignKey("entity.id", ondelete="CASCADE"))
content: Mapped[str] = mapped_column(Text)
category: Mapped[str] = mapped_column(String, nullable=False, default="note")
@@ -163,14 +162,9 @@ class Observation(Base):
We can construct these because observations are always defined in
and owned by a single entity.
Content is truncated to 200 chars to stay under PostgreSQL's
btree index limit of 2704 bytes.
"""
# Truncate content to avoid exceeding PostgreSQL's btree index limit
content_for_permalink = self.content[:200] if len(self.content) > 200 else self.content
return generate_permalink(
f"{self.entity.permalink}/observations/{self.category}/{content_for_permalink}"
f"{self.entity.permalink}/observations/{self.category}/{self.content}"
)
def __repr__(self) -> str: # pragma: no cover
@@ -192,7 +186,6 @@ class Relation(Base):
)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
project_id: Mapped[int] = mapped_column(Integer, ForeignKey("project.id"), index=True)
from_id: Mapped[int] = mapped_column(Integer, ForeignKey("entity.id", ondelete="CASCADE"))
to_id: Mapped[Optional[int]] = mapped_column(
Integer, ForeignKey("entity.id", ondelete="CASCADE"), nullable=True
+44 -43
View File
@@ -1,52 +1,53 @@
"""Search DDL statements for SQLite and Postgres.
"""Search models and tables."""
The search_index table is created via raw DDL, not ORM models, because:
- SQLite uses FTS5 virtual tables (cannot be represented as ORM)
- Postgres uses composite primary keys and generated tsvector columns
- Both backends use raw SQL for all search operations via SearchIndexRow dataclass
"""
from sqlalchemy import DDL, Column, Integer, String, DateTime, Text
from sqlalchemy.dialects.postgresql import JSONB
from sqlalchemy.types import JSON
from sqlalchemy import DDL
from basic_memory.models.base import Base
# Define Postgres search_index table with composite primary key and tsvector
# This DDL matches the Alembic migration schema (314f1ea54dc4)
# Used by tests to create the table without running full migrations
# NOTE: Split into separate DDL statements because asyncpg doesn't support
# multiple statements in a single execute call.
CREATE_POSTGRES_SEARCH_INDEX_TABLE = DDL("""
CREATE TABLE IF NOT EXISTS search_index (
id INTEGER NOT NULL,
project_id INTEGER NOT NULL,
title TEXT,
content_stems TEXT,
content_snippet TEXT,
permalink VARCHAR,
file_path VARCHAR,
type VARCHAR,
from_id INTEGER,
to_id INTEGER,
relation_type VARCHAR,
entity_id INTEGER,
category VARCHAR,
metadata JSONB,
created_at TIMESTAMP WITH TIME ZONE,
updated_at TIMESTAMP WITH TIME ZONE,
textsearchable_index_col tsvector GENERATED ALWAYS AS (
to_tsvector('english', coalesce(title, '') || ' ' || coalesce(content_stems, ''))
) STORED,
PRIMARY KEY (id, type, project_id),
FOREIGN KEY (project_id) REFERENCES project(id) ON DELETE CASCADE
)
""")
class SearchIndex(Base):
"""Search index table for Postgres only.
CREATE_POSTGRES_SEARCH_INDEX_FTS = DDL("""
CREATE INDEX IF NOT EXISTS idx_search_index_fts ON search_index USING gin(textsearchable_index_col)
""")
For SQLite: This model is skipped; FTS5 virtual table is created via DDL instead.
For Postgres: This is the actual table structure with tsvector support.
"""
__tablename__ = "search_index"
# Primary key (rowid in SQLite FTS5, explicit id in Postgres)
id = Column(Integer, primary_key=True, autoincrement=True)
# Core searchable fields
title = Column(Text, nullable=True)
content_stems = Column(Text, nullable=True)
content_snippet = Column(Text, nullable=True)
permalink = Column(String(255), nullable=True, index=True)
file_path = Column(Text, nullable=True)
type = Column(String(50), nullable=True)
# Project context
project_id = Column(Integer, nullable=True, index=True)
# Relation fields
from_id = Column(Integer, nullable=True)
to_id = Column(Integer, nullable=True)
relation_type = Column(String(100), nullable=True)
# Observation fields
entity_id = Column(Integer, nullable=True)
category = Column(String(100), nullable=True)
# Common fields
# Use JSONB for Postgres, JSON for SQLite
# Note: 'metadata' is a reserved name in SQLAlchemy, so we use 'metadata_' and map to 'metadata'
metadata_ = Column("metadata", JSON().with_variant(JSONB(), "postgresql"), nullable=True)
created_at = Column(DateTime(timezone=True), nullable=True)
updated_at = Column(DateTime(timezone=True), nullable=True)
# Note: textsearchable_index_col (tsvector) will be added by migration for Postgres only
CREATE_POSTGRES_SEARCH_INDEX_METADATA = DDL("""
CREATE INDEX IF NOT EXISTS idx_search_index_metadata_gin ON search_index USING gin(metadata jsonb_path_ops)
""")
# Define FTS5 virtual table creation for SQLite only
# This DDL is executed separately for SQLite databases
@@ -3,7 +3,6 @@
from pathlib import Path
from typing import List, Optional, Sequence, Union, Any
from loguru import logger
from sqlalchemy import select
from sqlalchemy.exc import IntegrityError
@@ -77,99 +76,6 @@ class EntityRepository(Repository[Entity]):
)
return await self.find_one(query)
# -------------------------------------------------------------------------
# Lightweight methods for permalink resolution (no eager loading)
# -------------------------------------------------------------------------
async def permalink_exists(self, permalink: str) -> bool:
"""Check if a permalink exists without loading the full entity.
This is much faster than get_by_permalink() as it skips eager loading
of observations and relations. Use for existence checks in bulk operations.
Args:
permalink: Permalink to check
Returns:
True if permalink exists, False otherwise
"""
query = select(Entity.id).where(Entity.permalink == permalink).limit(1)
query = self._add_project_filter(query)
result = await self.execute_query(query, use_query_options=False)
return result.scalar_one_or_none() is not None
async def get_file_path_for_permalink(self, permalink: str) -> Optional[str]:
"""Get the file_path for a permalink without loading the full entity.
Use when you only need the file_path, not the full entity with relations.
Args:
permalink: Permalink to look up
Returns:
file_path string if found, None otherwise
"""
query = select(Entity.file_path).where(Entity.permalink == permalink)
query = self._add_project_filter(query)
result = await self.execute_query(query, use_query_options=False)
return result.scalar_one_or_none()
async def get_permalink_for_file_path(self, file_path: Union[Path, str]) -> Optional[str]:
"""Get the permalink for a file_path without loading the full entity.
Use when you only need the permalink, not the full entity with relations.
Args:
file_path: File path to look up
Returns:
permalink string if found, None otherwise
"""
query = select(Entity.permalink).where(Entity.file_path == Path(file_path).as_posix())
query = self._add_project_filter(query)
result = await self.execute_query(query, use_query_options=False)
return result.scalar_one_or_none()
async def get_all_permalinks(self) -> List[str]:
"""Get all permalinks for this project.
Optimized for bulk operations - returns only permalink strings
without loading entities or relationships.
Returns:
List of all permalinks in the project
"""
query = select(Entity.permalink)
query = self._add_project_filter(query)
result = await self.execute_query(query, use_query_options=False)
return list(result.scalars().all())
async def get_permalink_to_file_path_map(self) -> dict[str, str]:
"""Get a mapping of permalink -> file_path for all entities.
Optimized for bulk permalink resolution - loads minimal data in one query.
Returns:
Dict mapping permalink to file_path
"""
query = select(Entity.permalink, Entity.file_path)
query = self._add_project_filter(query)
result = await self.execute_query(query, use_query_options=False)
return {row.permalink: row.file_path for row in result.all()}
async def get_file_path_to_permalink_map(self) -> dict[str, str]:
"""Get a mapping of file_path -> permalink for all entities.
Optimized for bulk permalink resolution - loads minimal data in one query.
Returns:
Dict mapping file_path to permalink
"""
query = select(Entity.file_path, Entity.permalink)
query = self._add_project_filter(query)
result = await self.execute_query(query, use_query_options=False)
return {row.file_path: row.permalink for row in result.all()}
async def get_by_file_paths(
self, session: AsyncSession, file_paths: Sequence[Union[Path, str]]
) -> List[Row[Any]]:
@@ -2,7 +2,6 @@
from typing import Dict, List, Sequence
from sqlalchemy import select
from sqlalchemy.ext.asyncio import async_sessionmaker
@@ -5,7 +5,6 @@ import re
from datetime import datetime
from typing import List, Optional
from loguru import logger
from sqlalchemy import text
@@ -258,7 +257,7 @@ class PostgresSearchRepository(SearchRepositoryBase):
{score_expr} as score
FROM search_index
WHERE {where_clause}
ORDER BY score DESC, id ASC {order_by_clause}
ORDER BY score DESC {order_by_clause}
LIMIT :limit
OFFSET :offset
"""
@@ -3,7 +3,6 @@
from pathlib import Path
from typing import Optional, Sequence, Union
from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
@@ -24,7 +23,7 @@ class ProjectRepository(Repository[Project]):
super().__init__(session_maker, Project)
async def get_by_name(self, name: str) -> Optional[Project]:
"""Get project by name (exact match).
"""Get project by name.
Args:
name: Unique name of the project
@@ -32,18 +31,6 @@ class ProjectRepository(Repository[Project]):
query = self.select().where(Project.name == name)
return await self.find_one(query)
async def get_by_name_case_insensitive(self, name: str) -> Optional[Project]:
"""Get project by name (case-insensitive match).
Args:
name: Project name (case-insensitive)
Returns:
Project if found, None otherwise
"""
query = self.select().where(Project.name.ilike(name))
return await self.find_one(query)
async def get_by_permalink(self, permalink: str) -> Optional[Project]:
"""Get project by permalink.
@@ -1,11 +1,9 @@
"""Repository for managing Relation objects."""
from sqlalchemy import and_, delete
from typing import Sequence, List, Optional
from sqlalchemy import and_, delete, select
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
from sqlalchemy import select
from sqlalchemy.ext.asyncio import async_sessionmaker
from sqlalchemy.orm import selectinload, aliased
from sqlalchemy.orm.interfaces import LoaderOption
@@ -88,59 +86,5 @@ class RelationRepository(Repository[Relation]):
result = await self.execute_query(query)
return result.scalars().all()
async def add_all_ignore_duplicates(self, relations: List[Relation]) -> int:
"""Bulk insert relations, ignoring duplicates.
Uses ON CONFLICT DO NOTHING to skip relations that would violate the
unique constraint on (from_id, to_name, relation_type). This is useful
for bulk operations where the same link may appear multiple times in
a document.
Works with both SQLite and PostgreSQL dialects.
Args:
relations: List of Relation objects to insert
Returns:
Number of relations actually inserted (excludes duplicates)
"""
if not relations:
return 0
# Convert Relation objects to dicts for insert
values = [
{
"project_id": r.project_id if r.project_id else self.project_id,
"from_id": r.from_id,
"to_id": r.to_id,
"to_name": r.to_name,
"relation_type": r.relation_type,
"context": r.context,
}
for r in relations
]
async with db.scoped_session(self.session_maker) as session:
# Check dialect to use appropriate insert
dialect_name = session.bind.dialect.name if session.bind else "sqlite"
if dialect_name == "postgresql":
# PostgreSQL: use RETURNING to count inserted rows
# (rowcount is 0 for ON CONFLICT DO NOTHING)
stmt = (
pg_insert(Relation)
.values(values)
.on_conflict_do_nothing()
.returning(Relation.id)
)
result = await session.execute(stmt)
return len(result.fetchall())
else:
# SQLite: rowcount works correctly
stmt = sqlite_insert(Relation).values(values)
stmt = stmt.on_conflict_do_nothing()
result = await session.execute(stmt)
return result.rowcount if result.rowcount > 0 else 0
def get_load_options(self) -> List[LoaderOption]:
return [selectinload(Relation.from_entity), selectinload(Relation.to_entity)]
@@ -2,7 +2,6 @@
from typing import Type, Optional, Any, Sequence, TypeVar, List, Dict
from loguru import logger
from sqlalchemy import (
select,
@@ -4,7 +4,6 @@ from abc import ABC, abstractmethod
from datetime import datetime
from typing import Any, Dict, List, Optional
from loguru import logger
from sqlalchemy import Executable, Result, text
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
@@ -5,7 +5,6 @@ import re
from datetime import datetime
from typing import List, Optional
from loguru import logger
from sqlalchemy import text
+1 -1
View File
@@ -183,7 +183,7 @@ ObservationStr = Annotated[
str,
BeforeValidator(str.strip), # Clean whitespace
MinLen(1), # Ensure non-empty after stripping
# No MaxLen - matches DB Text column which has no length restriction
MaxLen(1000), # Keep reasonable length
]
+1 -5
View File
@@ -1,12 +1,10 @@
"""V2 API schemas - ID-based entity and project references."""
"""V2 API schemas - ID-based entity references."""
from basic_memory.schemas.v2.entity import (
EntityResolveRequest,
EntityResolveResponse,
EntityResponseV2,
MoveEntityRequestV2,
ProjectResolveRequest,
ProjectResolveResponse,
)
from basic_memory.schemas.v2.resource import (
CreateResourceRequest,
@@ -19,8 +17,6 @@ __all__ = [
"EntityResolveResponse",
"EntityResponseV2",
"MoveEntityRequestV2",
"ProjectResolveRequest",
"ProjectResolveResponse",
"CreateResourceRequest",
"UpdateResourceRequest",
"ResourceResponse",
+1 -34
View File
@@ -1,4 +1,4 @@
"""V2 entity and project schemas with ID-first design."""
"""V2 entity schemas with ID-first design."""
from datetime import datetime
from typing import Dict, List, Literal, Optional
@@ -94,36 +94,3 @@ class EntityResponseV2(BaseModel):
)
model_config = ConfigDict(from_attributes=True)
class ProjectResolveRequest(BaseModel):
"""Request to resolve a project identifier to a project ID.
Supports resolution of:
- Project names (e.g., "my-project")
- Permalinks (e.g., "my-project")
"""
identifier: str = Field(
...,
description="Project identifier to resolve (name or permalink)",
min_length=1,
max_length=255,
)
class ProjectResolveResponse(BaseModel):
"""Response from project identifier resolution.
Returns the project ID and associated metadata for the resolved project.
"""
project_id: int = Field(..., description="Numeric project ID (primary identifier)")
name: str = Field(..., description="Project name")
permalink: str = Field(..., description="Project permalink")
path: str = Field(..., description="Project file path")
is_active: bool = Field(..., description="Whether the project is active")
is_default: bool = Field(..., description="Whether the project is the default")
resolution_method: Literal["id", "name", "permalink"] = Field(
..., description="How the identifier was resolved"
)
@@ -4,7 +4,6 @@ from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import List, Optional, Tuple
from loguru import logger
from sqlalchemy import text
+2 -15
View File
@@ -3,10 +3,8 @@
import fnmatch
import logging
import os
from datetime import datetime
from typing import Dict, List, Optional, Sequence
from basic_memory.models import Entity
from basic_memory.repository import EntityRepository
from basic_memory.schemas.directory import DirectoryNode
@@ -14,17 +12,6 @@ from basic_memory.schemas.directory import DirectoryNode
logger = logging.getLogger(__name__)
def _mtime_to_datetime(entity: Entity) -> datetime:
"""Convert entity mtime (file modification time) to datetime.
Returns the file's actual modification time, falling back to updated_at
if mtime is not available.
"""
if entity.mtime:
return datetime.fromtimestamp(entity.mtime).astimezone()
return entity.updated_at
class DirectoryService:
"""Service for working with directory trees."""
@@ -90,7 +77,7 @@ class DirectoryService:
entity_id=file.id,
entity_type=file.entity_type,
content_type=file.content_type,
updated_at=_mtime_to_datetime(file),
updated_at=file.updated_at,
)
# Add to parent directory's children
@@ -254,7 +241,7 @@ class DirectoryService:
entity_id=file.id,
entity_type=file.entity_type,
content_type=file.content_type,
updated_at=_mtime_to_datetime(file),
updated_at=file.updated_at,
)
# Add to parent directory's children
+32 -64
View File
@@ -8,7 +8,6 @@ import yaml
from loguru import logger
from sqlalchemy.exc import IntegrityError
from basic_memory.config import ProjectConfig, BasicMemoryConfig
from basic_memory.file_utils import (
has_frontmatter,
@@ -29,7 +28,6 @@ from basic_memory.schemas.base import Permalink
from basic_memory.services import BaseService, FileService
from basic_memory.services.exceptions import EntityCreationError, EntityNotFoundError
from basic_memory.services.link_resolver import LinkResolver
from basic_memory.services.search_service import SearchService
from basic_memory.utils import generate_permalink
@@ -44,7 +42,6 @@ class EntityService(BaseService[EntityModel]):
relation_repository: RelationRepository,
file_service: FileService,
link_resolver: LinkResolver,
search_service: Optional[SearchService] = None,
app_config: Optional[BasicMemoryConfig] = None,
):
super().__init__(entity_repository)
@@ -53,7 +50,6 @@ class EntityService(BaseService[EntityModel]):
self.entity_parser = entity_parser
self.file_service = file_service
self.link_resolver = link_resolver
self.search_service = search_service
self.app_config = app_config
async def detect_file_path_conflicts(
@@ -110,9 +106,6 @@ class EntityService(BaseService[EntityModel]):
4. Generate new unique permalink from file path
Enhanced to detect and handle character-related conflicts.
Note: Uses lightweight repository methods that skip eager loading of
observations and relations for better performance during bulk operations.
"""
file_path_str = Path(file_path).as_posix()
@@ -129,20 +122,16 @@ class EntityService(BaseService[EntityModel]):
# If markdown has explicit permalink, try to validate it
if markdown and markdown.frontmatter.permalink:
desired_permalink = markdown.frontmatter.permalink
# Use lightweight method - we only need to check file_path
existing_file_path = await self.repository.get_file_path_for_permalink(
desired_permalink
)
existing = await self.repository.get_by_permalink(desired_permalink)
# If no conflict or it's our own file, use as is
if not existing_file_path or existing_file_path == file_path_str:
if not existing or existing.file_path == file_path_str:
return desired_permalink
# For existing files, try to find current permalink
# Use lightweight method - we only need the permalink
existing_permalink = await self.repository.get_permalink_for_file_path(file_path_str)
if existing_permalink:
return existing_permalink
existing = await self.repository.get_by_file_path(file_path_str)
if existing:
return existing.permalink
# New file - generate permalink
if markdown and markdown.frontmatter.permalink:
@@ -151,10 +140,9 @@ class EntityService(BaseService[EntityModel]):
desired_permalink = generate_permalink(file_path_str)
# Make unique if needed - enhanced to handle character conflicts
# Use lightweight existence check instead of loading full entity
permalink = desired_permalink
suffix = 1
while await self.repository.permalink_exists(permalink):
while await self.repository.get_by_permalink(permalink):
permalink = f"{desired_permalink}-{suffix}"
suffix += 1
logger.debug(f"creating unique permalink: {permalink}")
@@ -236,11 +224,8 @@ class EntityService(BaseService[EntityModel]):
final_content = dump_frontmatter(post)
checksum = await self.file_service.write_file(file_path, final_content)
# parse entity from content we just wrote (avoids re-reading file for cloud compatibility)
entity_markdown = await self.entity_parser.parse_markdown_content(
file_path=file_path,
content=final_content,
)
# parse entity from file
entity_markdown = await self.entity_parser.parse_file(file_path)
# create entity
created = await self.create_entity_from_markdown(file_path, entity_markdown)
@@ -260,12 +245,8 @@ class EntityService(BaseService[EntityModel]):
# Convert file path string to Path
file_path = Path(entity.file_path)
# Read existing content via file_service (for cloud compatibility)
existing_content = await self.file_service.read_file_content(file_path)
existing_markdown = await self.entity_parser.parse_markdown_content(
file_path=file_path,
content=existing_content,
)
# Read existing frontmatter from the file if it exists
existing_markdown = await self.entity_parser.parse_file(file_path)
# Parse content frontmatter to check for user-specified permalink and entity_type
content_markdown = None
@@ -321,11 +302,8 @@ class EntityService(BaseService[EntityModel]):
final_content = dump_frontmatter(merged_post)
checksum = await self.file_service.write_file(file_path, final_content)
# parse entity from content we just wrote (avoids re-reading file for cloud compatibility)
entity_markdown = await self.entity_parser.parse_markdown_content(
file_path=file_path,
content=final_content,
)
# parse entity from file
entity_markdown = await self.entity_parser.parse_file(file_path)
# update entity in db
entity = await self.update_entity_and_observations(file_path, entity_markdown)
@@ -357,11 +335,7 @@ class EntityService(BaseService[EntityModel]):
)
entity = entities[0]
# Delete from search index first (if search_service is available)
if self.search_service:
await self.search_service.handle_delete(entity)
# Delete file
# Delete file first
await self.file_service.delete_entity_file(entity)
# Delete from DB (this will cascade to observations/relations)
@@ -404,9 +378,7 @@ class EntityService(BaseService[EntityModel]):
Uses UPSERT approach to handle permalink/file_path conflicts cleanly.
"""
logger.debug(f"Creating entity: {markdown.frontmatter.title} file_path: {file_path}")
model = entity_model_from_markdown(
file_path, markdown, project_id=self.repository.project_id
)
model = entity_model_from_markdown(file_path, markdown)
# Mark as incomplete because we still need to add relations
model.checksum = None
@@ -436,7 +408,6 @@ class EntityService(BaseService[EntityModel]):
# add new observations
observations = [
Observation(
project_id=self.observation_repository.project_id,
entity_id=db_entity.id,
content=obs.content,
category=obs.category,
@@ -503,7 +474,6 @@ class EntityService(BaseService[EntityModel]):
# Create the relation
relation = Relation(
project_id=self.relation_repository.project_id,
from_id=db_entity.id,
to_id=target_id,
to_name=target_name,
@@ -576,11 +546,8 @@ class EntityService(BaseService[EntityModel]):
# Write the updated content back to the file
checksum = await self.file_service.write_file(file_path, new_content)
# Parse the content we just wrote (avoids re-reading file for cloud compatibility)
entity_markdown = await self.entity_parser.parse_markdown_content(
file_path=file_path,
content=new_content,
)
# Parse the updated file to get new observations/relations
entity_markdown = await self.entity_parser.parse_file(file_path)
# Update entity and its relationships
entity = await self.update_entity_and_observations(file_path, entity_markdown)
@@ -799,20 +766,23 @@ class EntityService(BaseService[EntityModel]):
raise ValueError(f"Invalid destination path: {destination_path}")
# 3. Validate paths
# NOTE: In tenantless/cloud mode, we cannot rely on local filesystem paths.
# Use FileService for existence checks and moving.
if not await self.file_service.exists(current_path):
source_file = project_config.home / current_path
destination_file = project_config.home / destination_path
# Validate source exists
if not source_file.exists():
raise ValueError(f"Source file not found: {current_path}")
if await self.file_service.exists(destination_path):
# Check if destination already exists
if destination_file.exists():
raise ValueError(f"Destination already exists: {destination_path}")
try:
# 4. Ensure destination directory if needed (no-op for S3)
await self.file_service.ensure_directory(Path(destination_path).parent)
# 4. Create destination directory if needed
destination_file.parent.mkdir(parents=True, exist_ok=True)
# 5. Move physical file via FileService (filesystem rename or cloud move)
await self.file_service.move_file(current_path, destination_path)
# 5. Move physical file
source_file.rename(destination_file)
logger.info(f"Moved file: {current_path} -> {destination_path}")
# 6. Prepare database updates
@@ -851,14 +821,12 @@ class EntityService(BaseService[EntityModel]):
except Exception as e:
# Rollback: try to restore original file location if move succeeded
try:
if await self.file_service.exists(
destination_path
) and not await self.file_service.exists(current_path):
await self.file_service.move_file(destination_path, current_path)
if destination_file.exists() and not source_file.exists():
try:
destination_file.rename(source_file)
logger.info(f"Rolled back file move: {destination_path} -> {current_path}")
except Exception as rollback_error: # pragma: no cover
logger.error(f"Failed to rollback file move: {rollback_error}")
except Exception as rollback_error: # pragma: no cover
logger.error(f"Failed to rollback file move: {rollback_error}")
# Re-raise the original error with context
raise ValueError(f"Move failed: {str(e)}") from e
+12 -123
View File
@@ -3,19 +3,15 @@
import asyncio
import hashlib
import mimetypes
from datetime import datetime
from os import stat_result
from pathlib import Path
from typing import TYPE_CHECKING, Any, Dict, Optional, Tuple, Union
from typing import Any, Dict, Tuple, Union
import aiofiles
import yaml
from basic_memory import file_utils
if TYPE_CHECKING:
from basic_memory.config import BasicMemoryConfig
from basic_memory.file_utils import FileError, FileMetadata, ParseError
from basic_memory.file_utils import FileError, ParseError
from basic_memory.markdown.markdown_processor import MarkdownProcessor
from basic_memory.models import Entity as EntityModel
from basic_memory.schemas import Entity as EntitySchema
@@ -45,11 +41,9 @@ class FileService:
base_path: Path,
markdown_processor: MarkdownProcessor,
max_concurrent_files: int = 10,
app_config: Optional["BasicMemoryConfig"] = None,
):
self.base_path = base_path.resolve() # Get absolute path
self.markdown_processor = markdown_processor
self.app_config = app_config
# Semaphore to limit concurrent file operations
# Prevents OOM on large projects by processing files in batches
self._file_semaphore = asyncio.Semaphore(max_concurrent_files)
@@ -154,15 +148,12 @@ class FileService:
Handles both absolute and relative paths. Relative paths are resolved
against base_path.
If format_on_save is enabled in config, runs the configured formatter
after writing and returns the checksum of the formatted content.
Args:
path: Where to write (Path or string)
content: Content to write
Returns:
Checksum of written content (or formatted content if formatting enabled)
Checksum of written content
Raises:
FileOperationError: If write fails
@@ -185,17 +176,8 @@ class FileService:
await file_utils.write_file_atomic(full_path, content)
# Format file if configured
final_content = content
if self.app_config:
formatted_content = await file_utils.format_file(
full_path, self.app_config, is_markdown=self.is_markdown(path)
)
if formatted_content is not None:
final_content = formatted_content
# Compute and return checksum of final content
checksum = await file_utils.compute_checksum(final_content)
# Compute and return checksum
checksum = await file_utils.compute_checksum(content)
logger.debug(f"File write completed path={full_path}, {checksum=}")
return checksum
@@ -238,41 +220,6 @@ class FileService:
logger.exception("File read error", path=str(full_path), error=str(e))
raise FileOperationError(f"Failed to read file: {e}")
async def read_file_bytes(self, path: FilePath) -> bytes:
"""Read file content as bytes using true async I/O with aiofiles.
This method reads files in binary mode, suitable for non-text files
like images, PDFs, etc. For cloud compatibility with S3FileService.
Args:
path: Path to read (Path or string)
Returns:
File content as bytes
Raises:
FileOperationError: If read fails
"""
# Convert string to Path if needed
path_obj = self.base_path / path if isinstance(path, str) else path
full_path = path_obj if path_obj.is_absolute() else self.base_path / path_obj
try:
logger.debug("Reading file bytes", operation="read_file_bytes", path=str(full_path))
async with aiofiles.open(full_path, mode="rb") as f:
content = await f.read()
logger.debug(
"File read completed",
path=str(full_path),
content_length=len(content),
)
return content
except Exception as e:
logger.exception("File read error", path=str(full_path), error=str(e))
raise FileOperationError(f"Failed to read file: {e}")
async def read_file(self, path: FilePath) -> Tuple[str, str]:
"""Read file and compute checksum using true async I/O.
@@ -329,43 +276,6 @@ class FileService:
full_path = path_obj if path_obj.is_absolute() else self.base_path / path_obj
full_path.unlink(missing_ok=True)
async def move_file(self, source: FilePath, destination: FilePath) -> None:
"""Move/rename a file from source to destination.
This method abstracts the underlying storage (filesystem vs cloud).
Default implementation uses atomic filesystem rename, but cloud-backed
implementations (e.g., S3) can override to copy+delete.
Args:
source: Source path (relative to base_path or absolute)
destination: Destination path (relative to base_path or absolute)
Raises:
FileOperationError: If the move fails
"""
# Convert strings to Paths and resolve relative paths against base_path
src_obj = self.base_path / source if isinstance(source, str) else source
dst_obj = self.base_path / destination if isinstance(destination, str) else destination
src_full = src_obj if src_obj.is_absolute() else self.base_path / src_obj
dst_full = dst_obj if dst_obj.is_absolute() else self.base_path / dst_obj
try:
# Ensure destination directory exists
await self.ensure_directory(dst_full.parent)
# Use semaphore for concurrency control and run blocking rename in executor
async with self._file_semaphore:
loop = asyncio.get_event_loop()
await loop.run_in_executor(None, lambda: src_full.rename(dst_full))
except Exception as e:
logger.exception(
"File move error",
source=str(src_full),
destination=str(dst_full),
error=str(e),
)
raise FileOperationError(f"Failed to move file {source} -> {destination}: {e}")
async def update_frontmatter(self, path: FilePath, updates: Dict[str, Any]) -> str:
"""Update frontmatter fields in a file while preserving all content.
@@ -422,17 +332,7 @@ class FileService:
)
await file_utils.write_file_atomic(full_path, final_content)
# Format file if configured
content_for_checksum = final_content
if self.app_config:
formatted_content = await file_utils.format_file(
full_path, self.app_config, is_markdown=self.is_markdown(path)
)
if formatted_content is not None:
content_for_checksum = formatted_content
return await file_utils.compute_checksum(content_for_checksum)
return await file_utils.compute_checksum(final_content)
except Exception as e:
# Only log real errors (not YAML parsing, which is handled above)
@@ -481,31 +381,20 @@ class FileService:
logger.error("Failed to compute checksum", path=str(full_path), error=str(e))
raise FileError(f"Failed to compute checksum for {path}: {e}")
async def get_file_metadata(self, path: FilePath) -> FileMetadata:
"""Return file metadata for a given path.
This method is async to support cloud implementations (S3FileService)
where file metadata requires async operations (head_object).
def file_stats(self, path: FilePath) -> stat_result:
"""Return file stats for a given path.
Args:
path: Path to the file (Path or string)
Returns:
FileMetadata with size, created_at, and modified_at
File statistics
"""
# Convert string to Path if needed
path_obj = self.base_path / path if isinstance(path, str) else path
full_path = path_obj if path_obj.is_absolute() else self.base_path / path_obj
# Run blocking stat() in thread pool to maintain async compatibility
loop = asyncio.get_event_loop()
stat_result = await loop.run_in_executor(None, full_path.stat)
return FileMetadata(
size=stat_result.st_size,
created_at=datetime.fromtimestamp(stat_result.st_ctime).astimezone(),
modified_at=datetime.fromtimestamp(stat_result.st_mtime).astimezone(),
)
# get file timestamps
return full_path.stat()
def content_type(self, path: FilePath) -> str:
"""Return content_type for a given path.
+26 -51
View File
@@ -5,11 +5,8 @@ to ensure consistent application startup across all entry points.
"""
import asyncio
import os
import sys
from pathlib import Path
from loguru import logger
from basic_memory import db
@@ -30,12 +27,15 @@ async def initialize_database(app_config: BasicMemoryConfig) -> None:
Database migrations are now handled automatically when the database
connection is first established via get_or_create_db().
"""
# Trigger database initialization and migrations by getting the database connection
try:
await db.get_or_create_db(app_config.database_path)
logger.info("Database initialization completed")
except Exception as e:
logger.error(f"Error during database initialization: {e}")
raise
logger.error(f"Error initializing database: {e}")
# Allow application to continue - it might still work
# depending on what the error was, and will fail with a
# more specific error if the database is actually unusable
async def reconcile_projects_with_config(app_config: BasicMemoryConfig):
@@ -49,29 +49,31 @@ async def reconcile_projects_with_config(app_config: BasicMemoryConfig):
"""
logger.info("Reconciling projects from config with database...")
# Get database session (engine already created by initialize_database)
# Get database session - migrations handled centrally
_, session_maker = await db.get_or_create_db(
db_path=app_config.database_path,
db_type=db.DatabaseType.FILESYSTEM,
ensure_migrations=False,
)
project_repository = ProjectRepository(session_maker)
# Import ProjectService here to avoid circular imports
from basic_memory.services.project_service import ProjectService
# Create project service and synchronize projects
project_service = ProjectService(repository=project_repository)
try:
# Create project service and synchronize projects
project_service = ProjectService(repository=project_repository)
await project_service.synchronize_projects()
logger.info("Projects successfully reconciled between config and database")
except Exception as e:
# Log the error but continue with initialization
logger.error(f"Error during project synchronization: {e}")
logger.info("Continuing with initialization despite synchronization error")
async def initialize_file_sync(
app_config: BasicMemoryConfig,
) -> None:
):
"""Initialize file synchronization services. This function starts the watch service and does not return
Args:
@@ -80,20 +82,15 @@ async def initialize_file_sync(
Returns:
The watch service task that's monitoring file changes
"""
# Never start file watching during tests. Even "background" watchers add tasks/threads
# and can interact badly with strict asyncio teardown (especially on Windows/aiosqlite).
# Skip file sync in test environments to avoid interference with tests
if app_config.is_test_env:
logger.info("Test environment detected - skipping file sync initialization")
return None
# delay import
from basic_memory.sync import WatchService
# Get database session (migrations already run if needed)
# Load app configuration - migrations handled centrally
_, session_maker = await db.get_or_create_db(
db_path=app_config.database_path,
db_type=db.DatabaseType.FILESYSTEM,
ensure_migrations=False,
)
project_repository = ProjectRepository(session_maker)
@@ -107,12 +104,6 @@ async def initialize_file_sync(
# Get active projects
active_projects = await project_repository.get_active_projects()
# Filter to constrained project if MCP server was started with --project
constrained_project = os.environ.get("BASIC_MEMORY_MCP_PROJECT")
if constrained_project:
active_projects = [p for p in active_projects if p.name == constrained_project]
logger.info(f"Background sync constrained to project: {constrained_project}")
# Start sync for all projects as background tasks (non-blocking)
async def sync_project_background(project: Project):
"""Sync a single project in the background."""
@@ -140,10 +131,12 @@ async def initialize_file_sync(
# Then start the watch service in the background
logger.info("Starting watch service for all projects")
# run the watch service
await watch_service.run()
logger.info("Watch service started")
try:
await watch_service.run()
logger.info("Watch service started")
except Exception as e: # pragma: no cover
logger.error(f"Error starting watch service: {e}")
return None
@@ -162,11 +155,6 @@ async def initialize_app(
Args:
app_config: The Basic Memory project configuration
"""
# Skip initialization in cloud mode - cloud manages its own projects
if app_config.cloud_mode_enabled:
logger.debug("Skipping initialization in cloud mode - projects managed by cloud")
return
logger.info("Initializing app...")
# Initialize database first
await initialize_database(app_config)
@@ -193,24 +181,11 @@ def ensure_initialization(app_config: BasicMemoryConfig) -> None:
logger.debug("Skipping initialization in cloud mode - projects managed by cloud")
return
async def _init_and_cleanup():
"""Initialize app and clean up database connections.
Database connections created during initialization must be cleaned up
before the event loop closes, otherwise the process will hang indefinitely.
"""
try:
await initialize_app(app_config)
finally:
# Always cleanup database connections to prevent process hang
await db.shutdown_db()
# On Windows, use SelectorEventLoop to avoid ProactorEventLoop cleanup issues
# The ProactorEventLoop can raise "IndexError: pop from an empty deque" during
# event loop cleanup when there are pending handles. SelectorEventLoop is more
# stable for our use case (no subprocess pipes or named pipes needed).
if sys.platform == "win32":
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
asyncio.run(_init_and_cleanup())
logger.info("Initialization completed successfully")
try:
result = asyncio.run(initialize_app(app_config))
logger.info(f"Initialization completed successfully: result={result}")
except Exception as e: # pragma: no cover
logger.exception(f"Error during initialization: {e}")
# Continue execution even if initialization fails
# The command might still work, or will fail with a
# more specific error message
@@ -2,7 +2,6 @@
from typing import Optional, Tuple
from loguru import logger
from basic_memory.models import Entity
+6 -6
View File
@@ -8,7 +8,6 @@ from datetime import datetime
from pathlib import Path
from typing import Dict, Optional, Sequence
from loguru import logger
from sqlalchemy import text
@@ -24,6 +23,9 @@ from basic_memory.config import WATCH_STATUS_JSON, ConfigManager, get_project_co
from basic_memory.utils import generate_permalink
config = ConfigManager().config
class ProjectService:
"""Service for managing Basic Memory projects."""
@@ -141,7 +143,6 @@ class ProjectService:
"""
# If project_root is set, constrain all projects to that directory
project_root = self.config_manager.config.project_root
sanitized_name = None
if project_root:
base_path = Path(project_root)
@@ -198,15 +199,14 @@ class ProjectService:
f"Projects cannot share directory trees."
)
if not self.config_manager.config.cloud_mode:
# First add to config file (this will validate the project doesn't exist)
self.config_manager.add_project(name, resolved_path)
# First add to config file (this will validate the project doesn't exist)
project_config = self.config_manager.add_project(name, resolved_path)
# Then add to database
project_data = {
"name": name,
"path": resolved_path,
"permalink": sanitized_name,
"permalink": generate_permalink(project_config.name),
"is_active": True,
# Don't set is_default=False to avoid UNIQUE constraint issues
# Let it default to NULL, only set to True when explicitly making default
+6 -37
View File
@@ -4,7 +4,6 @@ import ast
from datetime import datetime
from typing import List, Optional, Set
from dateparser import parse
from fastapi import BackgroundTasks
from loguru import logger
@@ -16,21 +15,6 @@ from basic_memory.repository.search_repository import SearchRepository, SearchIn
from basic_memory.schemas.search import SearchQuery, SearchItemType
from basic_memory.services import FileService
# Maximum size for content_stems field to stay under Postgres's 8KB index row limit.
# We use 6000 characters to leave headroom for other indexed columns and overhead.
MAX_CONTENT_STEMS_SIZE = 6000
def _mtime_to_datetime(entity: Entity) -> datetime:
"""Convert entity mtime (file modification time) to datetime.
Returns the file's actual modification time, falling back to updated_at
if mtime is not available.
"""
if entity.mtime:
return datetime.fromtimestamp(entity.mtime).astimezone()
return entity.updated_at
class SearchService:
"""Service for search operations.
@@ -209,7 +193,7 @@ class SearchService:
"entity_type": entity.entity_type,
},
created_at=entity.created_at,
updated_at=_mtime_to_datetime(entity),
updated_at=entity.updated_at,
project_id=entity.project_id,
)
)
@@ -272,10 +256,6 @@ class SearchService:
entity_content_stems = "\n".join(p for p in content_stems if p and p.strip())
# Truncate to stay under Postgres's 8KB index row limit
if len(entity_content_stems) > MAX_CONTENT_STEMS_SIZE:
entity_content_stems = entity_content_stems[:MAX_CONTENT_STEMS_SIZE]
# Add entity row
rows_to_index.append(
SearchIndexRow(
@@ -291,28 +271,17 @@ class SearchService:
"entity_type": entity.entity_type,
},
created_at=entity.created_at,
updated_at=_mtime_to_datetime(entity),
updated_at=entity.updated_at,
project_id=entity.project_id,
)
)
# Add observation rows - dedupe by permalink to avoid unique constraint violations
# Two observations with same entity/category/content generate identical permalinks
seen_permalinks: set[str] = {entity.permalink} if entity.permalink else set()
# Add observation rows
for obs in entity.observations:
obs_permalink = obs.permalink
if obs_permalink in seen_permalinks:
logger.debug(f"Skipping duplicate observation permalink: {obs_permalink}")
continue
seen_permalinks.add(obs_permalink)
# Index with parent entity's file path since that's where it's defined
obs_content_stems = "\n".join(
p for p in self._generate_variants(obs.content) if p and p.strip()
)
# Truncate to stay under Postgres's 8KB index row limit
if len(obs_content_stems) > MAX_CONTENT_STEMS_SIZE:
obs_content_stems = obs_content_stems[:MAX_CONTENT_STEMS_SIZE]
rows_to_index.append(
SearchIndexRow(
id=obs.id,
@@ -320,7 +289,7 @@ class SearchService:
title=f"{obs.category}: {obs.content[:100]}...",
content_stems=obs_content_stems,
content_snippet=obs.content,
permalink=obs_permalink,
permalink=obs.permalink,
file_path=entity.file_path,
category=obs.category,
entity_id=entity.id,
@@ -328,7 +297,7 @@ class SearchService:
"tags": obs.tags,
},
created_at=entity.created_at,
updated_at=_mtime_to_datetime(entity),
updated_at=entity.updated_at,
project_id=entity.project_id,
)
)
@@ -358,7 +327,7 @@ class SearchService:
to_id=rel.to_id,
relation_type=rel.relation_type,
created_at=entity.created_at,
updated_at=_mtime_to_datetime(entity),
updated_at=entity.updated_at,
project_id=entity.project_id,
)
)
+107 -121
View File
@@ -2,7 +2,6 @@
import asyncio
import os
import sys
import time
from collections import OrderedDict
from dataclasses import dataclass, field
@@ -11,7 +10,7 @@ from pathlib import Path
from typing import AsyncIterator, Dict, List, Optional, Set, Tuple
import aiofiles.os
import logfire
from loguru import logger
from sqlalchemy.exc import IntegrityError
@@ -216,12 +215,17 @@ class SyncService:
f"path={path}, error={error}"
)
# Record metric for file failure
logfire.metric_counter("sync.circuit_breaker.failures").add(1)
# Log when threshold is reached
if failure_info.count >= MAX_CONSECUTIVE_FAILURES:
logger.error(
f"File {path} has failed {MAX_CONSECUTIVE_FAILURES} times and will be skipped. "
f"First failure: {failure_info.first_failure}, Last error: {error}"
)
# Record metric for file being blocked by circuit breaker
logfire.metric_counter("sync.circuit_breaker.blocked_files").add(1)
else:
# Create new failure record
self._file_failures[path] = FileFailureInfo(
@@ -251,6 +255,7 @@ class SyncService:
logger.info(f"Clearing failure history for {path} after successful sync")
del self._file_failures[path]
@logfire.instrument()
async def sync(
self, directory: Path, project_name: Optional[str] = None, force_full: bool = False
) -> SyncReport:
@@ -277,58 +282,63 @@ class SyncService:
)
# sync moves first
for old_path, new_path in report.moves.items():
# in the case where a file has been deleted and replaced by another file
# it will show up in the move and modified lists, so handle it in modified
if new_path in report.modified:
report.modified.remove(new_path)
logger.debug(
f"File marked as moved and modified: old_path={old_path}, new_path={new_path}"
)
else:
await self.handle_move(old_path, new_path)
with logfire.span("process_moves", move_count=len(report.moves)):
for old_path, new_path in report.moves.items():
# in the case where a file has been deleted and replaced by another file
# it will show up in the move and modified lists, so handle it in modified
if new_path in report.modified:
report.modified.remove(new_path)
logger.debug(
f"File marked as moved and modified: old_path={old_path}, new_path={new_path}"
)
else:
await self.handle_move(old_path, new_path)
# deleted next
for path in report.deleted:
await self.handle_delete(path)
with logfire.span("process_deletes", delete_count=len(report.deleted)):
for path in report.deleted:
await self.handle_delete(path)
# then new and modified
for path in report.new:
entity, _ = await self.sync_file(path, new=True)
with logfire.span("process_new_files", new_count=len(report.new)):
for path in report.new:
entity, _ = await self.sync_file(path, new=True)
# Track if file was skipped
if entity is None and await self._should_skip_file(path):
failure_info = self._file_failures[path]
report.skipped_files.append(
SkippedFile(
path=path,
reason=failure_info.last_error,
failure_count=failure_info.count,
first_failed=failure_info.first_failure,
# Track if file was skipped
if entity is None and await self._should_skip_file(path):
failure_info = self._file_failures[path]
report.skipped_files.append(
SkippedFile(
path=path,
reason=failure_info.last_error,
failure_count=failure_info.count,
first_failed=failure_info.first_failure,
)
)
)
for path in report.modified:
entity, _ = await self.sync_file(path, new=False)
with logfire.span("process_modified_files", modified_count=len(report.modified)):
for path in report.modified:
entity, _ = await self.sync_file(path, new=False)
# Track if file was skipped
if entity is None and await self._should_skip_file(path):
failure_info = self._file_failures[path]
report.skipped_files.append(
SkippedFile(
path=path,
reason=failure_info.last_error,
failure_count=failure_info.count,
first_failed=failure_info.first_failure,
# Track if file was skipped
if entity is None and await self._should_skip_file(path):
failure_info = self._file_failures[path]
report.skipped_files.append(
SkippedFile(
path=path,
reason=failure_info.last_error,
failure_count=failure_info.count,
first_failed=failure_info.first_failure,
)
)
)
# Only resolve relations if there were actual changes
# If no files changed, no new unresolved relations could have been created
if report.total > 0:
await self.resolve_relations()
else:
logger.info("Skipping relation resolution - no file changes detected")
with logfire.span("resolve_relations"):
if report.total > 0:
await self.resolve_relations()
else:
logger.info("Skipping relation resolution - no file changes detected")
# Update scan watermark after successful sync
# Use the timestamp from sync start (not end) to ensure we catch files
@@ -351,6 +361,15 @@ class SyncService:
duration_ms = int((time.time() - start_time) * 1000)
# Record metrics for sync operation
logfire.metric_histogram("sync.duration", unit="ms").record(duration_ms)
logfire.metric_counter("sync.files.new").add(len(report.new))
logfire.metric_counter("sync.files.modified").add(len(report.modified))
logfire.metric_counter("sync.files.deleted").add(len(report.deleted))
logfire.metric_counter("sync.files.moved").add(len(report.moves))
if report.skipped_files:
logfire.metric_counter("sync.files.skipped").add(len(report.skipped_files))
# Log summary with skipped files if any
if report.skipped_files:
logger.warning(
@@ -371,6 +390,7 @@ class SyncService:
return report
@logfire.instrument()
async def scan(self, directory, force_full: bool = False):
"""Smart scan using watermark and file count for large project optimization.
@@ -452,6 +472,12 @@ class SyncService:
logger.warning("No scan watermark available, falling back to full scan")
file_paths_to_scan = await self._scan_directory_full(directory)
# Record scan type metric
logfire.metric_counter(f"sync.scan.{scan_type}").add(1)
logfire.metric_histogram("sync.scan.files_scanned", unit="files").record(
len(file_paths_to_scan)
)
# Step 3: Process each file with mtime-based comparison
scanned_paths: Set[str] = set()
changed_checksums: Dict[str, str] = {}
@@ -563,6 +589,7 @@ class SyncService:
report.checksums = changed_checksums
scan_duration_ms = int((time.time() - scan_start_time) * 1000)
logfire.metric_histogram("sync.scan.duration", unit="ms").record(scan_duration_ms)
logger.info(
f"Completed {scan_type} scan for directory {directory} in {scan_duration_ms}ms, "
@@ -572,6 +599,7 @@ class SyncService:
)
return report
@logfire.instrument()
async def sync_file(
self, path: str, new: bool = True
) -> Tuple[Optional[Entity], Optional[str]]:
@@ -610,16 +638,6 @@ class SyncService:
)
return entity, checksum
except FileNotFoundError:
# File exists in database but not on filesystem
# This indicates a database/filesystem inconsistency - treat as deletion
logger.warning(
f"File not found during sync, treating as deletion: path={path}. "
"This may indicate a race condition or manual file deletion."
)
await self.handle_delete(path)
return None, None
except Exception as e:
# Check if this is a fatal error (or caused by one)
# Fatal errors like project deletion should terminate sync immediately
@@ -636,6 +654,7 @@ class SyncService:
return None, None
@logfire.instrument()
async def sync_markdown_file(self, path: str, new: bool = True) -> Tuple[Optional[Entity], str]:
"""Sync a markdown file with full processing.
@@ -653,19 +672,12 @@ class SyncService:
file_contains_frontmatter = has_frontmatter(file_content)
# Get file timestamps for tracking modification times
file_metadata = await self.file_service.get_file_metadata(path)
created = file_metadata.created_at
modified = file_metadata.modified_at
file_stats = self.file_service.file_stats(path)
created = datetime.fromtimestamp(file_stats.st_ctime).astimezone()
modified = datetime.fromtimestamp(file_stats.st_mtime).astimezone()
# Parse markdown content with file metadata (avoids redundant file read/stat)
# This enables cloud implementations (S3FileService) to provide metadata from head_object
abs_path = self.file_service.base_path / path
entity_markdown = await self.entity_parser.parse_markdown_content(
file_path=abs_path,
content=file_content,
mtime=file_metadata.modified_at.timestamp(),
ctime=file_metadata.created_at.timestamp(),
)
# entity markdown will always contain front matter, so it can be used up create/update the entity
entity_markdown = await self.entity_parser.parse_file(path)
# if the file contains frontmatter, resolve a permalink (unless disabled)
if file_contains_frontmatter and not self.app_config.disable_permalinks:
@@ -711,8 +723,8 @@ class SyncService:
"checksum": final_checksum,
"created_at": created,
"updated_at": modified,
"mtime": file_metadata.modified_at.timestamp(),
"size": file_metadata.size,
"mtime": file_stats.st_mtime,
"size": file_stats.st_size,
},
)
@@ -725,6 +737,7 @@ class SyncService:
# Return the final checksum to ensure everything is consistent
return entity, final_checksum
@logfire.instrument()
async def sync_regular_file(self, path: str, new: bool = True) -> Tuple[Optional[Entity], str]:
"""Sync a non-markdown file with basic tracking.
@@ -741,9 +754,9 @@ class SyncService:
await self.entity_service.resolve_permalink(path, skip_conflict_check=True)
# get file timestamps
file_metadata = await self.file_service.get_file_metadata(path)
created = file_metadata.created_at
modified = file_metadata.modified_at
file_stats = self.file_service.file_stats(path)
created = datetime.fromtimestamp(file_stats.st_ctime).astimezone()
modified = datetime.fromtimestamp(file_stats.st_mtime).astimezone()
# get mime type
content_type = self.file_service.content_type(path)
@@ -759,8 +772,8 @@ class SyncService:
created_at=created,
updated_at=modified,
content_type=content_type,
mtime=file_metadata.modified_at.timestamp(),
size=file_metadata.size,
mtime=file_stats.st_mtime,
size=file_stats.st_size,
)
)
return entity, checksum
@@ -776,15 +789,15 @@ class SyncService:
logger.error(f"Entity not found after constraint violation, path={path}")
raise ValueError(f"Entity not found after constraint violation: {path}")
# Re-get file metadata since we're in update path
file_metadata_for_update = await self.file_service.get_file_metadata(path)
# Re-get file stats since we're in update path
file_stats_for_update = self.file_service.file_stats(path)
updated = await self.entity_repository.update(
entity.id,
{
"file_path": path,
"checksum": checksum,
"mtime": file_metadata_for_update.modified_at.timestamp(),
"size": file_metadata_for_update.size,
"mtime": file_stats_for_update.st_mtime,
"size": file_stats_for_update.st_size,
},
)
@@ -798,8 +811,8 @@ class SyncService:
raise
else:
# Get file timestamps for updating modification time
file_metadata = await self.file_service.get_file_metadata(path)
modified = file_metadata.modified_at
file_stats = self.file_service.file_stats(path)
modified = datetime.fromtimestamp(file_stats.st_mtime).astimezone()
entity = await self.entity_repository.get_by_file_path(path)
if entity is None: # pragma: no cover
@@ -814,8 +827,8 @@ class SyncService:
"file_path": path,
"checksum": checksum,
"updated_at": modified,
"mtime": file_metadata.modified_at.timestamp(),
"size": file_metadata.size,
"mtime": file_stats.st_mtime,
"size": file_stats.st_size,
},
)
@@ -825,6 +838,7 @@ class SyncService:
return updated, checksum
@logfire.instrument()
async def handle_delete(self, file_path: str):
"""Handle complete entity deletion including search index cleanup."""
@@ -856,6 +870,7 @@ class SyncService:
else:
await self.search_service.delete_by_entity_id(entity.id)
@logfire.instrument()
async def handle_move(self, old_path, new_path):
logger.debug("Moving entity", old_path=old_path, new_path=new_path)
@@ -960,6 +975,7 @@ class SyncService:
# update search index
await self.search_service.index_entity(updated)
@logfire.instrument()
async def resolve_relations(self, entity_id: int | None = None):
"""Try to resolve unresolved relations.
@@ -1010,27 +1026,16 @@ class SyncService:
"to_name": resolved_entity.title,
},
)
# update search index only on successful resolution
await self.search_service.index_entity(resolved_entity)
except IntegrityError:
# IntegrityError means a relation with this (from_id, to_id, relation_type)
# already exists. The UPDATE was rolled back, so our unresolved relation
# (to_id=NULL) still exists in the database. We delete it because:
# 1. It's redundant - a resolved relation already captures this relationship
# 2. If we don't delete it, future syncs will try to resolve it again
# and get the same IntegrityError
except IntegrityError: # pragma: no cover
logger.debug(
"Deleting duplicate unresolved relation "
"Ignoring duplicate relation "
f"relation_id={relation.id} "
f"from_id={relation.from_id} "
f"to_name={relation.to_name} "
f"resolved_to_id={resolved_entity.id}"
f"to_name={relation.to_name}"
)
try:
await self.relation_repository.delete(relation.id)
except Exception as e:
# Log but don't fail - the relation may have been deleted already
logger.debug(f"Could not delete duplicate relation {relation.id}: {e}")
# update search index
await self.search_service.index_entity(resolved_entity)
async def _quick_count_files(self, directory: Path) -> int:
"""Fast file count using find command.
@@ -1038,22 +1043,12 @@ class SyncService:
Uses subprocess to leverage OS-level file counting which is much faster
than Python iteration, especially on network filesystems like TigrisFS.
On Windows, subprocess is not supported with SelectorEventLoop (which we use
to avoid aiosqlite cleanup issues), so we fall back to Python-based counting.
Args:
directory: Directory to count files in
Returns:
Number of files in directory (recursive)
"""
# Windows with SelectorEventLoop doesn't support subprocess
if sys.platform == "win32":
count = 0
async for _ in self.scan_directory(directory):
count += 1
return count
process = await asyncio.create_subprocess_shell(
f'find "{directory}" -type f | wc -l',
stdout=asyncio.subprocess.PIPE,
@@ -1068,6 +1063,8 @@ class SyncService:
f"error: {error_msg}. Falling back to manual count. "
f"This will slow down watermark detection!"
)
# Track optimization failures for visibility
logfire.metric_counter("sync.scan.file_count_failure").add(1)
# Fallback: count using scan_directory
count = 0
async for _ in self.scan_directory(directory):
@@ -1084,9 +1081,6 @@ class SyncService:
This is dramatically faster than scanning all files and comparing mtimes,
especially on network filesystems like TigrisFS where stat operations are expensive.
On Windows, subprocess is not supported with SelectorEventLoop (which we use
to avoid aiosqlite cleanup issues), so we implement mtime filtering in Python.
Args:
directory: Directory to scan
since_timestamp: Unix timestamp to find files newer than
@@ -1094,16 +1088,6 @@ class SyncService:
Returns:
List of relative file paths modified since the timestamp (respects .bmignore)
"""
# Windows with SelectorEventLoop doesn't support subprocess
# Implement mtime filtering in Python to preserve watermark optimization
if sys.platform == "win32":
file_paths = []
async for file_path_str, stat_info in self.scan_directory(directory):
if stat_info.st_mtime > since_timestamp:
rel_path = Path(file_path_str).relative_to(directory).as_posix()
file_paths.append(rel_path)
return file_paths
# Convert timestamp to find-compatible format
since_date = datetime.fromtimestamp(since_timestamp).strftime("%Y-%m-%d %H:%M:%S")
@@ -1121,6 +1105,8 @@ class SyncService:
f"error: {error_msg}. Falling back to full scan. "
f"This will cause slow syncs on large projects!"
)
# Track optimization failures for visibility
logfire.metric_counter("sync.scan.optimization_failure").add(1)
# Fallback to full scan
return await self._scan_directory_full(directory)
@@ -1220,8 +1206,8 @@ async def get_sync_service(project: Project) -> SyncService: # pragma: no cover
project_path = Path(project.path)
entity_parser = EntityParser(project_path)
markdown_processor = MarkdownProcessor(entity_parser, app_config=app_config)
file_service = FileService(project_path, markdown_processor, app_config=app_config)
markdown_processor = MarkdownProcessor(entity_parser)
file_service = FileService(project_path, markdown_processor)
# Initialize repositories
entity_repository = EntityRepository(session_maker, project_id=project.id)

Some files were not shown because too many files have changed in this diff Show More