mirror of
https://github.com/basicmachines-co/basic-memory
synced 2026-06-21 13:47:35 +00:00
Compare commits
9 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 4bfec8a88e | |||
| 9c3d2eb335 | |||
| 5fcbae3fdb | |||
| 68ee310a55 | |||
| 3a7fca8a9e | |||
| 4f2b1b2fd3 | |||
| a70b355838 | |||
| 32acbfe982 | |||
| 1831397861 |
@@ -1,5 +0,0 @@
|
||||
{
|
||||
"enabledPlugins": {
|
||||
"basic-memory@basicmachines": true
|
||||
}
|
||||
}
|
||||
@@ -78,7 +78,21 @@ jobs:
|
||||
python-version: [ "3.12", "3.13" ]
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
# Note: No services section needed - testcontainers handles Postgres in Docker
|
||||
# Postgres service (only available on Linux runners)
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:17
|
||||
env:
|
||||
POSTGRES_DB: basic_memory_test
|
||||
POSTGRES_USER: basic_memory_user
|
||||
POSTGRES_PASSWORD: dev_password
|
||||
options: >-
|
||||
--health-cmd pg_isready
|
||||
--health-interval 10s
|
||||
--health-timeout 5s
|
||||
--health-retries 5
|
||||
ports:
|
||||
- 5433:5432
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
@@ -107,7 +121,7 @@ jobs:
|
||||
run: |
|
||||
uv pip install -e .[dev]
|
||||
|
||||
- name: Run tests (Postgres via testcontainers)
|
||||
- name: Run tests (Postgres)
|
||||
run: |
|
||||
uv pip install pytest pytest-cov
|
||||
just test-postgres
|
||||
@@ -1,98 +1,5 @@
|
||||
# CHANGELOG
|
||||
|
||||
## v0.16.3 (2025-12-20)
|
||||
|
||||
### Features
|
||||
|
||||
- **#439**: Add PostgreSQL database backend support
|
||||
([`fb5e9e1`](https://github.com/basicmachines-co/basic-memory/commit/fb5e9e1))
|
||||
- Full PostgreSQL/Neon database support as alternative to SQLite
|
||||
- Async connection pooling with asyncpg
|
||||
- Alembic migrations support for both backends
|
||||
- Configurable via `BASIC_MEMORY_DATABASE_BACKEND` environment variable
|
||||
|
||||
- **#441**: Implement API v2 with ID-based endpoints (Phase 1)
|
||||
([`28cc522`](https://github.com/basicmachines-co/basic-memory/commit/28cc522))
|
||||
- New ID-based API endpoints for improved performance
|
||||
- Foundation for future API enhancements
|
||||
- Backward compatible with existing endpoints
|
||||
|
||||
- Add project_id to Relation and Observation for efficient project-scoped queries
|
||||
([`a920a9f`](https://github.com/basicmachines-co/basic-memory/commit/a920a9f))
|
||||
- Enables faster queries in multi-project environments
|
||||
- Improved database schema for cloud deployments
|
||||
|
||||
- Add bulk insert with ON CONFLICT handling for relations
|
||||
([`0818bda`](https://github.com/basicmachines-co/basic-memory/commit/0818bda))
|
||||
- Faster relation creation during sync operations
|
||||
- Handles duplicate relations gracefully
|
||||
|
||||
### Performance
|
||||
|
||||
- Lightweight permalink resolution to avoid eager loading
|
||||
([`6f99d2e`](https://github.com/basicmachines-co/basic-memory/commit/6f99d2e))
|
||||
- Reduces database queries during entity lookups
|
||||
- Improved response times for read operations
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- **#464**: Pin FastMCP to 2.12.3 to fix MCP tools visibility
|
||||
([`f227ef6`](https://github.com/basicmachines-co/basic-memory/commit/f227ef6))
|
||||
- Fixes issue where MCP tools were not visible to Claude
|
||||
- Reverts to last known working FastMCP version
|
||||
|
||||
- **#458**: Reduce watch service CPU usage by increasing reload interval
|
||||
([`897b1ed`](https://github.com/basicmachines-co/basic-memory/commit/897b1ed))
|
||||
- Lowers CPU usage during file watching
|
||||
- More efficient resource utilization
|
||||
|
||||
- **#456**: Await background sync task cancellation in lifespan shutdown
|
||||
([`efbc758`](https://github.com/basicmachines-co/basic-memory/commit/efbc758))
|
||||
- Prevents hanging on shutdown
|
||||
- Clean async task cleanup
|
||||
|
||||
- **#434**: Respect --project flag in background sync
|
||||
([`70bb10b`](https://github.com/basicmachines-co/basic-memory/commit/70bb10b))
|
||||
- Background sync now correctly uses specified project
|
||||
- Fixes multi-project sync issues
|
||||
|
||||
- **#446**: Fix observation parsing and permalink limits
|
||||
([`73d940e`](https://github.com/basicmachines-co/basic-memory/commit/73d940e))
|
||||
- Handles edge cases in observation content
|
||||
- Prevents permalink truncation issues
|
||||
|
||||
- **#424**: Handle periods in kebab_filenames mode
|
||||
([`b004565`](https://github.com/basicmachines-co/basic-memory/commit/b004565))
|
||||
- Fixes filename handling for files with multiple periods
|
||||
- Improved kebab-case conversion
|
||||
|
||||
- Fix Postgres/Neon connection settings and search index dedupe
|
||||
([`b5d4fb5`](https://github.com/basicmachines-co/basic-memory/commit/b5d4fb5))
|
||||
- Optimized connection pooling for Postgres
|
||||
- Prevents duplicate search index entries
|
||||
|
||||
### Testing & CI
|
||||
|
||||
- Replace py-pglite with testcontainers for Postgres testing
|
||||
([`c462faf`](https://github.com/basicmachines-co/basic-memory/commit/c462faf))
|
||||
- More reliable Postgres testing infrastructure
|
||||
- Uses Docker-based test containers
|
||||
|
||||
- Add PostgreSQL testing to GitHub Actions workflow
|
||||
([`66b91b2`](https://github.com/basicmachines-co/basic-memory/commit/66b91b2))
|
||||
- CI now tests both SQLite and PostgreSQL backends
|
||||
- Ensures cross-database compatibility
|
||||
|
||||
- **#416**: Add integration test for read_note with underscored folders
|
||||
([`0c12a39`](https://github.com/basicmachines-co/basic-memory/commit/0c12a39))
|
||||
- Verifies folder name handling edge cases
|
||||
|
||||
### Internal
|
||||
|
||||
- Cloud compatibility fixes and performance improvements (#454)
|
||||
- Remove logfire instrumentation for cleaner production deployments
|
||||
- Truncate content_stems to fix Postgres 8KB index row limit
|
||||
|
||||
## v0.16.2 (2025-11-16)
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
@@ -15,14 +15,10 @@ See the [README.md](README.md) file for a project overview.
|
||||
### Build and Test Commands
|
||||
|
||||
- Install: `just install` or `pip install -e ".[dev]"`
|
||||
- Run all tests (SQLite + Postgres): `just test`
|
||||
- Run all tests against SQLite: `just test-sqlite`
|
||||
- Run all tests against Postgres: `just test-postgres` (uses testcontainers)
|
||||
- Run unit tests (SQLite): `just test-unit-sqlite`
|
||||
- Run unit tests (Postgres): `just test-unit-postgres`
|
||||
- Run integration tests (SQLite): `just test-int-sqlite`
|
||||
- Run integration tests (Postgres): `just test-int-postgres`
|
||||
- Generate HTML coverage: `just coverage`
|
||||
- Run all tests (with coverage): `just test` - Runs both unit and integration tests with unified coverage
|
||||
- Run unit tests only: `just test-unit` - Fast, no coverage
|
||||
- Run integration tests only: `just test-int` - Fast, no coverage
|
||||
- Generate HTML coverage: `just coverage` - Opens in browser
|
||||
- Single test: `pytest tests/path/to/test_file.py::test_function_name`
|
||||
- Run benchmarks: `pytest test-int/test_sync_performance_benchmark.py -v -m "benchmark and not slow"`
|
||||
- Lint: `just lint` or `ruff check . --fix`
|
||||
@@ -34,8 +30,6 @@ See the [README.md](README.md) file for a project overview.
|
||||
|
||||
**Note:** Project requires Python 3.12+ (uses type parameter syntax and `type` aliases introduced in 3.12)
|
||||
|
||||
**Postgres Testing:** Uses [testcontainers](https://testcontainers-python.readthedocs.io/) which automatically spins up a Postgres instance in Docker. No manual database setup required - just have Docker running.
|
||||
|
||||
### Test Structure
|
||||
|
||||
- `tests/` - Unit tests for individual components (mocked, fast)
|
||||
@@ -82,10 +76,8 @@ See the [README.md](README.md) file for a project overview.
|
||||
- SQLite is used for indexing and full text search, files are source of truth
|
||||
- Testing uses pytest with asyncio support (strict mode)
|
||||
- Unit tests (`tests/`) use mocks when necessary; integration tests (`test-int/`) use real implementations
|
||||
- By default, tests run against SQLite (fast, no Docker needed)
|
||||
- Set `BASIC_MEMORY_TEST_POSTGRES=1` to run against Postgres (uses testcontainers - Docker required)
|
||||
- Each test runs in a standalone environment with isolated database and tmp_path directory
|
||||
- CI runs SQLite and Postgres tests in parallel for faster feedback
|
||||
- Test database uses in-memory SQLite
|
||||
- Each test runs in a standalone environment with in-memory SQLite and tmp_file directory
|
||||
- Performance benchmarks are in `test-int/test_sync_performance_benchmark.py`
|
||||
- Use pytest markers: `@pytest.mark.benchmark` for benchmarks, `@pytest.mark.slow` for slow tests
|
||||
|
||||
@@ -237,11 +229,6 @@ of using AI just for code generation, we've developed a true collaborative workf
|
||||
This approach has allowed us to tackle more complex challenges and build a more robust system than either humans or AI
|
||||
could achieve independently.
|
||||
|
||||
**Problem-Solving Guidance:**
|
||||
- If a solution isn't working after reasonable effort, suggest alternative approaches
|
||||
- Don't persist with a problematic library or pattern when better alternatives exist
|
||||
- Example: When py-pglite caused cascading test failures, switching to testcontainers-postgres was the right call
|
||||
|
||||
## GitHub Integration
|
||||
|
||||
Basic Memory has taken AI-Human collaboration to the next level by integrating Claude directly into the development workflow through GitHub:
|
||||
|
||||
@@ -433,76 +433,42 @@ See the [Documentation](https://memory.basicmachines.co/) for more info, includi
|
||||
- [Managing multiple Projects](https://docs.basicmemory.com/guides/cli-reference/#project)
|
||||
- [Importing data from OpenAI/Claude Projects](https://docs.basicmemory.com/guides/cli-reference/#import)
|
||||
|
||||
## Logging
|
||||
|
||||
Basic Memory uses [Loguru](https://github.com/Delgan/loguru) for logging. The logging behavior varies by entry point:
|
||||
|
||||
| Entry Point | Default Behavior | Use Case |
|
||||
|-------------|------------------|----------|
|
||||
| CLI commands | File only | Prevents log output from interfering with command output |
|
||||
| MCP server | File only | Stdout would corrupt the JSON-RPC protocol |
|
||||
| API server | File (local) or stdout (cloud) | Docker/cloud deployments use stdout |
|
||||
|
||||
**Log file location:** `~/.basic-memory/basic-memory.log` (10MB rotation, 10 days retention)
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `BASIC_MEMORY_LOG_LEVEL` | `INFO` | Log level: DEBUG, INFO, WARNING, ERROR |
|
||||
| `BASIC_MEMORY_CLOUD_MODE` | `false` | When `true`, API logs to stdout with structured context |
|
||||
| `BASIC_MEMORY_ENV` | `dev` | Set to `test` for test mode (stderr only) |
|
||||
|
||||
### Examples
|
||||
|
||||
```bash
|
||||
# Enable debug logging
|
||||
BASIC_MEMORY_LOG_LEVEL=DEBUG basic-memory sync
|
||||
|
||||
# View logs
|
||||
tail -f ~/.basic-memory/basic-memory.log
|
||||
|
||||
# Cloud/Docker mode (stdout logging with structured context)
|
||||
BASIC_MEMORY_CLOUD_MODE=true uvicorn basic_memory.api.app:app
|
||||
```
|
||||
|
||||
## Development
|
||||
|
||||
### Running Tests
|
||||
|
||||
Basic Memory supports dual database backends (SQLite and Postgres). By default, tests run against SQLite. Set `BASIC_MEMORY_TEST_POSTGRES=1` to run against Postgres (uses testcontainers - Docker required).
|
||||
Basic Memory supports dual database backends (SQLite and Postgres). Tests are parametrized to run against both backends automatically.
|
||||
|
||||
**Quick Start:**
|
||||
```bash
|
||||
# Run all tests against SQLite (default, fast)
|
||||
# Run SQLite tests (default, no Docker needed)
|
||||
just test-sqlite
|
||||
|
||||
# Run all tests against Postgres (uses testcontainers)
|
||||
# Run Postgres tests (requires Docker)
|
||||
just test-postgres
|
||||
|
||||
# Run both SQLite and Postgres tests
|
||||
just test
|
||||
```
|
||||
|
||||
**Available Test Commands:**
|
||||
|
||||
- `just test` - Run all tests against both SQLite and Postgres
|
||||
- `just test-sqlite` - Run all tests against SQLite (fast, no Docker needed)
|
||||
- `just test-postgres` - Run all tests against Postgres (uses testcontainers)
|
||||
- `just test-unit-sqlite` - Run unit tests against SQLite
|
||||
- `just test-unit-postgres` - Run unit tests against Postgres
|
||||
- `just test-int-sqlite` - Run integration tests against SQLite
|
||||
- `just test-int-postgres` - Run integration tests against Postgres
|
||||
- `just test-sqlite` - Run tests against SQLite only (fastest, no Docker needed)
|
||||
- `just test-postgres` - Run tests against Postgres only (requires Docker)
|
||||
- `just test-windows` - Run Windows-specific tests (auto-skips on other platforms)
|
||||
- `just test-benchmark` - Run performance benchmark tests
|
||||
- `just test-all` - Run all tests including Windows, Postgres, and benchmarks
|
||||
|
||||
**Postgres Testing:**
|
||||
**Postgres Testing Requirements:**
|
||||
|
||||
Postgres tests use [testcontainers](https://testcontainers-python.readthedocs.io/) which automatically spins up a Postgres instance in Docker. No manual database setup required - just have Docker running.
|
||||
To run Postgres tests, you need to start the test database:
|
||||
```bash
|
||||
docker-compose -f docker-compose-postgres.yml up -d
|
||||
```
|
||||
|
||||
Tests will connect to `localhost:5433/basic_memory_test`.
|
||||
|
||||
**Test Markers:**
|
||||
|
||||
Tests use pytest markers for selective execution:
|
||||
- `postgres` - Tests that run against Postgres backend
|
||||
- `windows` - Windows-specific database optimizations
|
||||
- `benchmark` - Performance tests (excluded from default runs)
|
||||
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"name": "basicmachines",
|
||||
"owner": {
|
||||
"name": "Basic Machines",
|
||||
"email": "hello@basicmachines.co"
|
||||
},
|
||||
"metadata": {
|
||||
"description": "Official plugins from Basic Machines for knowledge management and AI-assisted development",
|
||||
"version": "0.1.0"
|
||||
},
|
||||
"plugins": [
|
||||
{
|
||||
"name": "basic-memory",
|
||||
"source": ".",
|
||||
"description": "Skills, commands, and hooks for Basic Memory MCP - capture knowledge, continue conversations, and follow spec-driven development",
|
||||
"version": "0.1.0",
|
||||
"author": {
|
||||
"name": "Basic Machines"
|
||||
},
|
||||
"keywords": ["memory", "knowledge", "mcp", "specs", "context"]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"name": "basic-memory",
|
||||
"description": "Claude Code skills for Basic Memory - capture knowledge, continue conversations, and follow spec-driven development using the Basic Memory MCP server",
|
||||
"version": "0.1.0",
|
||||
"author": {
|
||||
"name": "Basic Machines"
|
||||
},
|
||||
"repository": "https://github.com/basicmachines-co/basic-memory"
|
||||
}
|
||||
@@ -0,0 +1,313 @@
|
||||
# Basic Memory Plugin for Claude Code
|
||||
|
||||
This plugin provides skills, commands, and hooks for working with [Basic Memory](https://basicmemory.io) - a local-first knowledge management system built on the Model Context Protocol (MCP).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
You need the Basic Memory MCP server running. Install it via:
|
||||
|
||||
```bash
|
||||
# Install basic-memory
|
||||
pip install basic-memory
|
||||
|
||||
# Or with pipx
|
||||
pipx install basic-memory
|
||||
```
|
||||
|
||||
Then add it to your Claude Code MCP configuration.
|
||||
|
||||
## Installation
|
||||
|
||||
### Add the Marketplace
|
||||
|
||||
```
|
||||
/plugin marketplace add basicmachines-co/basic-memory/claude-code-plugin
|
||||
```
|
||||
|
||||
### Install the Plugin
|
||||
|
||||
```
|
||||
/plugin install basic-memory@basicmachines
|
||||
```
|
||||
|
||||
### Or via Repository Settings
|
||||
|
||||
Add to your `.claude/settings.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"plugins": {
|
||||
"extraKnownMarketplaces": {
|
||||
"basicmachines": {
|
||||
"source": {
|
||||
"source": "github",
|
||||
"repo": "basicmachines-co/basic-memory",
|
||||
"path": "claude-code-plugin"
|
||||
}
|
||||
}
|
||||
},
|
||||
"installed": ["basic-memory@basicmachines"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Slash Commands
|
||||
|
||||
User-invoked commands for explicit interaction with Basic Memory.
|
||||
|
||||
### `/remember [title] [folder]`
|
||||
|
||||
Capture insights, decisions, or learnings from the current conversation.
|
||||
|
||||
```
|
||||
/remember "FastAPI Async Pattern"
|
||||
/remember "Auth Decision" decisions
|
||||
```
|
||||
|
||||
Creates a structured note with:
|
||||
- Context from the conversation
|
||||
- Observations with `[decision]`, `[insight]`, `[pattern]` categories
|
||||
- Relations linking to related concepts
|
||||
|
||||
### `/continue [topic]`
|
||||
|
||||
Resume previous work by building context from Basic Memory.
|
||||
|
||||
```
|
||||
/continue postgres migration
|
||||
/continue SPEC-24
|
||||
/continue
|
||||
```
|
||||
|
||||
If no topic is provided, shows recent activity and asks what to dive into.
|
||||
|
||||
### `/context <memory://url> [depth] [timeframe]`
|
||||
|
||||
Build context from a specific memory:// URL.
|
||||
|
||||
```
|
||||
/context memory://SPEC-24
|
||||
/context memory://architecture/* 3 2weeks
|
||||
```
|
||||
|
||||
### `/recent [timeframe] [project]`
|
||||
|
||||
Show recent activity in Basic Memory.
|
||||
|
||||
```
|
||||
/recent
|
||||
/recent 1week
|
||||
/recent today specs
|
||||
```
|
||||
|
||||
### `/organize [action] [project]`
|
||||
|
||||
Organize and maintain your knowledge graph.
|
||||
|
||||
```
|
||||
/organize # Quick health check
|
||||
/organize orphans # Find unlinked notes
|
||||
/organize duplicates # Find similar notes
|
||||
/organize relations "Note" # Suggest links for a note
|
||||
/organize tags # Review tag consistency
|
||||
```
|
||||
|
||||
Actions:
|
||||
- `health` - Overview of knowledge base status (default)
|
||||
- `orphans` - Find notes with no relations
|
||||
- `duplicates` - Find overlapping notes
|
||||
- `relations` - Suggest connections
|
||||
- `tags` - Review tag consistency
|
||||
|
||||
### `/research <topic> [folder]`
|
||||
|
||||
Research a topic and save a structured report to Basic Memory.
|
||||
|
||||
```
|
||||
/research MCP protocol
|
||||
/research "database migrations"
|
||||
/research "auth options" decisions
|
||||
```
|
||||
|
||||
Produces a report with:
|
||||
- Summary and key findings
|
||||
- Analysis and recommendations
|
||||
- Sources and related notes
|
||||
- Saved to `research/` folder by default
|
||||
|
||||
---
|
||||
|
||||
## Skills
|
||||
|
||||
Model-invoked capabilities that Claude uses automatically based on context.
|
||||
|
||||
### knowledge-capture
|
||||
|
||||
Automatically captures insights, decisions, and learnings into structured notes.
|
||||
|
||||
**Triggers when:**
|
||||
- Important decisions are made
|
||||
- Technical insights are discovered
|
||||
- Problems are solved
|
||||
- Design trade-offs are discussed
|
||||
|
||||
### continue-conversation
|
||||
|
||||
Resumes previous work by building context from the knowledge graph.
|
||||
|
||||
**Triggers when:**
|
||||
- Starting a new session
|
||||
- User mentions previous work ("continue with...", "back to...")
|
||||
- Need context about ongoing projects
|
||||
|
||||
### spec-driven-development
|
||||
|
||||
Guides implementation based on specifications stored in Basic Memory.
|
||||
|
||||
**Triggers when:**
|
||||
- Implementing a feature defined by a spec
|
||||
- Creating new specifications
|
||||
- Reviewing implementation against criteria
|
||||
|
||||
### edit-note
|
||||
|
||||
Interactively edit notes using MCP tools in a conversational workflow.
|
||||
|
||||
**Triggers when:**
|
||||
- User wants to edit, update, or modify a note
|
||||
- User asks to change specific content in a note
|
||||
- User wants to add observations or relations
|
||||
|
||||
**How it works:**
|
||||
1. Fetches the note via MCP
|
||||
2. Shows current content
|
||||
3. Applies edits using `edit_note` operations (append, prepend, find_replace, replace_section)
|
||||
4. Shows the updated result
|
||||
|
||||
**Best for:** Cloud users or when you want conversational editing.
|
||||
|
||||
### edit-note-local
|
||||
|
||||
Edit notes directly as local markdown files with automatic sync.
|
||||
|
||||
**Triggers when:**
|
||||
- User has local Basic Memory installation
|
||||
- User wants to make substantial file edits
|
||||
- User prefers working with full file content
|
||||
|
||||
**How it works:**
|
||||
1. Finds the note's file path via MCP
|
||||
2. Uses Claude Code's Read/Edit/Write tools on the actual file
|
||||
3. Basic Memory's `sync --watch` picks up changes automatically
|
||||
|
||||
**Best for:** Local users who want full file access and git integration.
|
||||
|
||||
### knowledge-organize
|
||||
|
||||
Help organize, link, and maintain the knowledge graph.
|
||||
|
||||
**Triggers when:**
|
||||
- User wants to organize their notes
|
||||
- User asks about orphan or unlinked notes
|
||||
- User wants to find connections between notes
|
||||
- User mentions duplicates or similar notes
|
||||
- User asks for help with folder organization
|
||||
|
||||
**Capabilities:**
|
||||
- **Find orphan notes** - Identify notes with no relations
|
||||
- **Suggest relations** - Propose meaningful links between notes
|
||||
- **Identify duplicates** - Find notes covering similar topics
|
||||
- **Folder organization** - Review and suggest folder structure
|
||||
- **Tag consistency** - Normalize and improve tagging
|
||||
- **Create index notes** - Generate hub notes linking related topics
|
||||
- **Enrich sparse notes** - Suggest observations and structure
|
||||
|
||||
**Best for:** Periodic knowledge base maintenance and improving discoverability.
|
||||
|
||||
### research
|
||||
|
||||
Research topics thoroughly and produce structured reports saved to Basic Memory.
|
||||
|
||||
**Triggers when:**
|
||||
- User asks to research or investigate something
|
||||
- User wants to understand a concept or technology
|
||||
- User needs context before making a decision
|
||||
- Phrases like "research", "look into", "explore", "investigate"
|
||||
|
||||
**What it produces:**
|
||||
- Structured report with summary, findings, and analysis
|
||||
- Recommendations when applicable
|
||||
- Links to sources and related notes
|
||||
- Saved to `research/` folder
|
||||
|
||||
**Best for:** Building knowledge base through investigation and documentation.
|
||||
|
||||
---
|
||||
|
||||
## Hooks
|
||||
|
||||
Automated behaviors that enhance the Basic Memory workflow.
|
||||
|
||||
### PostToolUse: write_note
|
||||
|
||||
Confirms when notes are saved to Basic Memory.
|
||||
|
||||
### Stop
|
||||
|
||||
After significant conversations, suggests using `/remember` to capture valuable insights (only when genuinely useful).
|
||||
|
||||
---
|
||||
|
||||
## MCP Tools Used
|
||||
|
||||
This plugin leverages Basic Memory's MCP tools:
|
||||
|
||||
| Tool | Purpose |
|
||||
|------|---------|
|
||||
| `write_note` | Create/update markdown notes |
|
||||
| `read_note` | Read notes by title or permalink |
|
||||
| `search_notes` | Full-text search across content |
|
||||
| `build_context` | Navigate knowledge graph via memory:// URLs |
|
||||
| `recent_activity` | Get recently updated information |
|
||||
| `edit_note` | Incrementally update notes |
|
||||
|
||||
---
|
||||
|
||||
## Plugin Structure
|
||||
|
||||
```
|
||||
claude-code-plugin/
|
||||
├── .claude-plugin/
|
||||
│ ├── plugin.json # Plugin manifest
|
||||
│ └── marketplace.json # Self-hosted marketplace
|
||||
├── commands/
|
||||
│ ├── remember.md # /remember command
|
||||
│ ├── continue.md # /continue command
|
||||
│ ├── context.md # /context command
|
||||
│ ├── recent.md # /recent command
|
||||
│ ├── organize.md # /organize command
|
||||
│ └── research.md # /research command
|
||||
├── skills/
|
||||
│ ├── knowledge-capture/
|
||||
│ ├── continue-conversation/
|
||||
│ ├── spec-driven-development/
|
||||
│ ├── edit-note/
|
||||
│ ├── edit-note-local/
|
||||
│ ├── knowledge-organize/
|
||||
│ └── research/
|
||||
├── hooks/
|
||||
│ └── hooks.json # Hook definitions
|
||||
├── README.md # Quick start guide
|
||||
└── PLUGIN.md # Full documentation
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- [Basic Memory Documentation](https://docs.basicmemory.io)
|
||||
- [Basic Memory GitHub](https://github.com/basicmachines-co/basic-memory)
|
||||
- [Model Context Protocol](https://modelcontextprotocol.io)
|
||||
- [Claude Code Plugins](https://code.claude.com/docs/en/plugins)
|
||||
@@ -0,0 +1,100 @@
|
||||
# Basic Memory Plugin for Claude Code
|
||||
|
||||
A Claude Code plugin that integrates [Basic Memory](https://basicmemory.io) - a local-first knowledge management system built on the Model Context Protocol (MCP).
|
||||
|
||||
## What This Plugin Does
|
||||
|
||||
This plugin helps Claude Code work seamlessly with your Basic Memory knowledge base:
|
||||
|
||||
- **Capture knowledge** from conversations automatically
|
||||
- **Resume previous work** by building context from your knowledge graph
|
||||
- **Edit notes** interactively through conversation
|
||||
- **Organize your knowledge** by finding orphans, suggesting links, and maintaining structure
|
||||
|
||||
## Installation
|
||||
|
||||
### 1. Install Basic Memory
|
||||
|
||||
```bash
|
||||
pip install basic-memory
|
||||
# or
|
||||
pipx install basic-memory
|
||||
```
|
||||
|
||||
### 2. Add the Marketplace
|
||||
|
||||
```
|
||||
/plugin marketplace add basicmachines-co/basic-memory/claude-code-plugin
|
||||
```
|
||||
|
||||
### 3. Install the Plugin
|
||||
|
||||
```
|
||||
/plugin install basic-memory@basicmachines
|
||||
```
|
||||
|
||||
## Commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `/remember [title]` | Capture insights from the current conversation |
|
||||
| `/continue [topic]` | Resume previous work with context |
|
||||
| `/context <memory://url>` | Build context from a specific note |
|
||||
| `/recent [timeframe]` | Show recent activity |
|
||||
| `/organize [action]` | Maintain your knowledge graph |
|
||||
| `/research <topic>` | Research a topic and save a report |
|
||||
|
||||
### Examples
|
||||
|
||||
```bash
|
||||
# Capture what we just discussed
|
||||
/remember "Database Design Decision"
|
||||
|
||||
# Pick up where we left off
|
||||
/continue postgres migration
|
||||
|
||||
# Check recent changes
|
||||
/recent 1week
|
||||
|
||||
# Find orphan notes and suggest links
|
||||
/organize orphans
|
||||
|
||||
# Research a topic and save findings
|
||||
/research "MCP protocol"
|
||||
```
|
||||
|
||||
## Skills
|
||||
|
||||
Skills are model-invoked - Claude uses them automatically when the context fits.
|
||||
|
||||
| Skill | What It Does |
|
||||
|-------|--------------|
|
||||
| `knowledge-capture` | Auto-captures decisions and insights into structured notes |
|
||||
| `continue-conversation` | Builds context when resuming previous work |
|
||||
| `spec-driven-development` | Guides implementation based on specs in Basic Memory |
|
||||
| `edit-note` | Edits notes via MCP tools (cloud-compatible) |
|
||||
| `edit-note-local` | Edits notes as files (local installations) |
|
||||
| `knowledge-organize` | Helps organize and link notes |
|
||||
| `research` | Researches topics and produces saved reports |
|
||||
|
||||
## Hooks
|
||||
|
||||
| Event | Behavior |
|
||||
|-------|----------|
|
||||
| `PostToolUse: write_note` | Confirms when notes are saved |
|
||||
| `Stop` | Suggests capturing valuable insights after conversations |
|
||||
|
||||
## Requirements
|
||||
|
||||
- [Claude Code](https://claude.com/claude-code)
|
||||
- [Basic Memory](https://basicmemory.io) with MCP server configured
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Full Plugin Documentation](./PLUGIN.md)
|
||||
- [Basic Memory Docs](https://docs.basicmemory.io)
|
||||
- [Claude Code Plugins](https://code.claude.com/docs/en/plugins)
|
||||
|
||||
## License
|
||||
|
||||
MIT - See the [Basic Memory repository](https://github.com/basicmachines-co/basic-memory) for details.
|
||||
@@ -0,0 +1,39 @@
|
||||
---
|
||||
description: Build context from a Basic Memory URL
|
||||
argument-hint: <memory://url> [depth] [timeframe]
|
||||
allowed-tools: mcp__basic-memory__build_context, mcp__basic-memory__read_note
|
||||
---
|
||||
|
||||
# Context
|
||||
|
||||
Build context from a Basic Memory memory:// URL.
|
||||
|
||||
## Arguments
|
||||
|
||||
- `$1` - Memory URL (e.g., `memory://topic`, `memory://folder/*`, `memory://SPEC-24`)
|
||||
- `$2` - Depth of relation traversal (optional, default: 2)
|
||||
- `$3` - Timeframe for recent changes (optional, default: "7d")
|
||||
|
||||
## Your Task
|
||||
|
||||
Navigate the knowledge graph and build comprehensive context.
|
||||
|
||||
1. **Build context** using `mcp__basic-memory__build_context`:
|
||||
- url: "$1"
|
||||
- depth: $2 or 2
|
||||
- timeframe: "$3" or "7d"
|
||||
|
||||
2. **Present the context**:
|
||||
- Main note content
|
||||
- Related notes found via relations
|
||||
- Recent changes within timeframe
|
||||
- Key observations and decisions
|
||||
|
||||
3. **Read additional notes** if needed for more detail.
|
||||
|
||||
## Memory URL Formats
|
||||
|
||||
- `memory://note-title` - Single note by title
|
||||
- `memory://folder/*` - All notes in a folder
|
||||
- `memory://SPEC-*` - Pattern matching
|
||||
- `memory://specs/SPEC-24` - Note in specific project folder
|
||||
@@ -0,0 +1,46 @@
|
||||
---
|
||||
description: Resume previous work from Basic Memory context
|
||||
argument-hint: [topic]
|
||||
allowed-tools: mcp__basic-memory__build_context, mcp__basic-memory__recent_activity, mcp__basic-memory__search_notes, mcp__basic-memory__read_note
|
||||
---
|
||||
|
||||
# Continue
|
||||
|
||||
Resume previous work by building context from Basic Memory.
|
||||
|
||||
## Arguments
|
||||
|
||||
- `$ARGUMENTS` - Topic, note title, or search terms to find previous context
|
||||
|
||||
## Your Task
|
||||
|
||||
Build context to continue previous work seamlessly.
|
||||
|
||||
1. **Find relevant context**:
|
||||
|
||||
If a specific topic is provided ("$ARGUMENTS"):
|
||||
- Search for matching notes: `mcp__basic-memory__search_notes`
|
||||
- Build context from matches: `mcp__basic-memory__build_context`
|
||||
- Read key notes for details: `mcp__basic-memory__read_note`
|
||||
|
||||
If no topic provided:
|
||||
- Get recent activity: `mcp__basic-memory__recent_activity` with timeframe "3d"
|
||||
- Present what's been happening
|
||||
- Ask which topic to dive into
|
||||
|
||||
2. **Present context**:
|
||||
- Summarize current state of the work
|
||||
- Highlight recent changes or progress
|
||||
- List open items or next steps
|
||||
- Show related context from the knowledge graph
|
||||
|
||||
3. **Be ready to continue**:
|
||||
- Understand what was done before
|
||||
- Know what needs to happen next
|
||||
- Have relevant context loaded
|
||||
|
||||
## Tips
|
||||
|
||||
- Use `memory://topic` URL format with `build_context`
|
||||
- Check multiple projects if needed (main, specs)
|
||||
- Follow relations to find connected knowledge
|
||||
@@ -0,0 +1,87 @@
|
||||
---
|
||||
description: Organize and maintain your Basic Memory knowledge graph
|
||||
argument-hint: [health|orphans|duplicates|relations|tags] [project]
|
||||
allowed-tools: mcp__basic-memory__search_notes, mcp__basic-memory__read_note, mcp__basic-memory__list_directory, mcp__basic-memory__edit_note, mcp__basic-memory__write_note
|
||||
---
|
||||
|
||||
# Organize
|
||||
|
||||
Help organize, link, and maintain your Basic Memory knowledge graph.
|
||||
|
||||
## Arguments
|
||||
|
||||
- `$1` - Action (optional): `health`, `orphans`, `duplicates`, `relations`, `tags` (default: `health`)
|
||||
- `$2` - Project (optional): defaults to "main"
|
||||
|
||||
## Actions
|
||||
|
||||
### `/organize` or `/organize health`
|
||||
|
||||
Run a quick health check:
|
||||
1. Count total notes
|
||||
2. Identify orphan notes (no relations)
|
||||
3. Check for potential duplicates
|
||||
4. Show folder distribution
|
||||
5. Report any issues found
|
||||
|
||||
### `/organize orphans`
|
||||
|
||||
Find and address orphan notes:
|
||||
1. Search for notes with empty Relations sections
|
||||
2. List orphans found
|
||||
3. For each orphan, suggest potential relations based on content
|
||||
4. Offer to add relations or create index notes
|
||||
|
||||
### `/organize duplicates`
|
||||
|
||||
Find potentially duplicate notes:
|
||||
1. Search for notes with similar titles
|
||||
2. Compare content for overlap
|
||||
3. Suggest: merge, differentiate, or link with `supersedes`
|
||||
|
||||
### `/organize relations [note-title]`
|
||||
|
||||
Suggest relations for a specific note (or recent notes if not specified):
|
||||
1. Read the target note
|
||||
2. Search for related content
|
||||
3. Suggest relation types:
|
||||
- `relates-to` - General connection
|
||||
- `extends` - Builds upon
|
||||
- `implements` - Realizes concept
|
||||
- `depends-on` - Requires understanding of
|
||||
4. Offer to add selected relations
|
||||
|
||||
### `/organize tags`
|
||||
|
||||
Review tag consistency:
|
||||
1. Gather all tags across notes
|
||||
2. Find similar/duplicate tags (e.g., `arch` vs `architecture`)
|
||||
3. Identify over-used or under-used tags
|
||||
4. Suggest normalization
|
||||
|
||||
## Your Task
|
||||
|
||||
Execute: `/organize $ARGUMENTS`
|
||||
|
||||
Based on the action requested:
|
||||
|
||||
1. **Gather data** using search and list tools
|
||||
2. **Analyze** for the specific issue (orphans, duplicates, etc.)
|
||||
3. **Present findings** clearly with counts and examples
|
||||
4. **Offer solutions** - ask before making changes
|
||||
5. **Apply fixes** using edit_note or write_note when user approves
|
||||
|
||||
Always confirm before modifying notes. Show what will change and get approval.
|
||||
|
||||
## Examples
|
||||
|
||||
```
|
||||
/organize # Quick health check
|
||||
/organize health # Same as above
|
||||
/organize orphans # Find unlinked notes
|
||||
/organize duplicates # Find similar notes
|
||||
/organize relations # Suggest links for recent notes
|
||||
/organize relations "My Note" # Suggest links for specific note
|
||||
/organize tags # Review tag consistency
|
||||
/organize health specs # Health check on specs project
|
||||
```
|
||||
@@ -0,0 +1,40 @@
|
||||
---
|
||||
description: Show recent activity in Basic Memory
|
||||
argument-hint: [timeframe] [project]
|
||||
allowed-tools: mcp__basic-memory__recent_activity, mcp__basic-memory__read_note
|
||||
---
|
||||
|
||||
# Recent
|
||||
|
||||
Show recent activity in Basic Memory.
|
||||
|
||||
## Arguments
|
||||
|
||||
- `$1` - Timeframe (optional): "today", "1d", "3d", "1 week", "2 weeks" (default: "3d")
|
||||
- `$2` - Project (optional): "main", "specs", etc.
|
||||
|
||||
## Your Task
|
||||
|
||||
Show what's been happening in Basic Memory recently.
|
||||
|
||||
1. **Get recent activity** using `mcp__basic-memory__recent_activity`:
|
||||
- timeframe: "$1" or "3d"
|
||||
- project: "$2" or check all projects
|
||||
|
||||
2. **Present activity**:
|
||||
- List recently modified notes
|
||||
- Group by type or folder if helpful
|
||||
- Highlight key changes
|
||||
- Show dates of modifications
|
||||
|
||||
3. **Offer to dive deeper**:
|
||||
- Ask if user wants to read any specific notes
|
||||
- Suggest continuing work on active items
|
||||
|
||||
## Timeframe Examples
|
||||
|
||||
- `today` - Just today
|
||||
- `1d` or `yesterday` - Last 24 hours
|
||||
- `3d` - Last 3 days
|
||||
- `1 week` - Last week
|
||||
- `2 weeks` - Last 2 weeks
|
||||
@@ -0,0 +1,43 @@
|
||||
---
|
||||
description: Capture insights, decisions, or learnings to Basic Memory
|
||||
argument-hint: [title] [optional: folder]
|
||||
allowed-tools: mcp__basic-memory__write_note, mcp__basic-memory__search_notes
|
||||
---
|
||||
|
||||
# Remember
|
||||
|
||||
Capture what we just discussed into a Basic Memory note.
|
||||
|
||||
## Arguments
|
||||
|
||||
- `$1` - Title for the note (required)
|
||||
- `$2` - Folder to save in (optional, defaults to "notes")
|
||||
|
||||
## Your Task
|
||||
|
||||
Create a structured note capturing the key insights from our conversation.
|
||||
|
||||
1. **Analyze the conversation** for:
|
||||
- Decisions made
|
||||
- Insights discovered
|
||||
- Problems solved
|
||||
- Patterns identified
|
||||
- Trade-offs discussed
|
||||
|
||||
2. **Structure the note** with:
|
||||
- Clear title: "$1" (or generate one if not provided)
|
||||
- Context section explaining the situation
|
||||
- Main content with key points
|
||||
- Observations using `[category]` format:
|
||||
- `[decision]` - Choices made
|
||||
- `[insight]` - Understanding gained
|
||||
- `[pattern]` - Reusable approaches
|
||||
- `[learning]` - Lessons learned
|
||||
- Relations to link related concepts with `[[WikiLinks]]`
|
||||
|
||||
3. **Save using** `mcp__basic-memory__write_note`:
|
||||
- folder: "$2" or "notes"
|
||||
- Include relevant tags
|
||||
- Project: use "main" unless user specifies otherwise
|
||||
|
||||
4. **Confirm** what was captured and where it was saved.
|
||||
@@ -0,0 +1,140 @@
|
||||
---
|
||||
description: Research a topic and save a structured report to Basic Memory
|
||||
argument-hint: <topic> [folder]
|
||||
allowed-tools: mcp__basic-memory__write_note, mcp__basic-memory__search_notes, mcp__basic-memory__read_note, mcp__basic-memory__build_context, WebSearch, WebFetch, Grep, Glob, Read
|
||||
---
|
||||
|
||||
# Research
|
||||
|
||||
Research a topic thoroughly and produce a structured report saved to Basic Memory.
|
||||
|
||||
## Arguments
|
||||
|
||||
- `$1` - Topic to research (required)
|
||||
- `$2` - Folder to save report (optional, default: "research")
|
||||
|
||||
## Your Task
|
||||
|
||||
Conduct thorough research on: **$ARGUMENTS**
|
||||
|
||||
### 1. Check Existing Knowledge
|
||||
|
||||
First, see what we already know:
|
||||
```python
|
||||
mcp__basic-memory__search_notes(query="$1", project="main")
|
||||
```
|
||||
|
||||
Read any relevant existing notes to avoid duplicating research.
|
||||
|
||||
### 2. Gather Information
|
||||
|
||||
Depending on the topic, use appropriate tools:
|
||||
|
||||
**For codebase topics:**
|
||||
- Search code with Grep/Glob
|
||||
- Read relevant files
|
||||
- Check tests for examples
|
||||
|
||||
**For external topics:**
|
||||
- Use WebSearch for current information
|
||||
- Fetch documentation with WebFetch
|
||||
- Look for official sources
|
||||
|
||||
**For Basic Memory context:**
|
||||
- Build context from related notes
|
||||
- Check for prior decisions or research
|
||||
|
||||
### 3. Analyze Findings
|
||||
|
||||
Synthesize what you learned:
|
||||
- Identify key concepts
|
||||
- Note patterns and trade-offs
|
||||
- Form recommendations if applicable
|
||||
- Flag uncertainties
|
||||
|
||||
### 4. Produce Report
|
||||
|
||||
Create a structured report with this format:
|
||||
|
||||
```markdown
|
||||
---
|
||||
title: "Research: [Topic]"
|
||||
type: research
|
||||
tags:
|
||||
- research
|
||||
- [relevant-tags]
|
||||
---
|
||||
|
||||
# Research: [Topic]
|
||||
|
||||
## Summary
|
||||
|
||||
[2-3 sentence executive summary]
|
||||
|
||||
## Research Question
|
||||
|
||||
[What we investigated and why]
|
||||
|
||||
## Key Findings
|
||||
|
||||
### [Finding 1]
|
||||
[Details and evidence]
|
||||
|
||||
### [Finding 2]
|
||||
[Details and evidence]
|
||||
|
||||
### [Finding 3]
|
||||
[Details and evidence]
|
||||
|
||||
## Analysis
|
||||
|
||||
[Synthesis, patterns, trade-offs, recommendations]
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [Areas needing more investigation]
|
||||
|
||||
## Sources
|
||||
|
||||
- [Links to sources]
|
||||
- [[Related Notes]] from Basic Memory
|
||||
|
||||
## Observations
|
||||
|
||||
- [finding] Key insight #research
|
||||
- [recommendation] Suggested approach based on research
|
||||
|
||||
## Relations
|
||||
|
||||
- researches [[Topic]]
|
||||
- relates-to [[Related Concepts]]
|
||||
```
|
||||
|
||||
### 5. Save Report
|
||||
|
||||
```python
|
||||
mcp__basic-memory__write_note(
|
||||
title="Research: $1",
|
||||
content="[report content]",
|
||||
folder="$2" or "research",
|
||||
tags=["research", ...],
|
||||
project="main"
|
||||
)
|
||||
```
|
||||
|
||||
### 6. Present Summary
|
||||
|
||||
After saving, present:
|
||||
- Key findings summary
|
||||
- Main recommendation (if applicable)
|
||||
- Where the report was saved
|
||||
- Offer to dive deeper into any aspect
|
||||
|
||||
## Examples
|
||||
|
||||
```
|
||||
/research MCP protocol
|
||||
/research "database migration patterns"
|
||||
/research "authentication options" decisions
|
||||
/research "React vs Vue" architecture
|
||||
```
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"hooks": {
|
||||
"PostToolUse": [
|
||||
{
|
||||
"matcher": "mcp__basic-memory__write_note",
|
||||
"hooks": [
|
||||
{
|
||||
"type": "command",
|
||||
"command": "echo '✓ Note saved to Basic Memory'"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"Stop": [
|
||||
{
|
||||
"matcher": "*",
|
||||
"hooks": [
|
||||
{
|
||||
"type": "prompt",
|
||||
"prompt": "If this conversation contained valuable insights, decisions, or learnings that should be preserved, suggest using `/remember [title]` to capture them in Basic Memory. Only suggest this if there's genuinely valuable content worth preserving - don't suggest for trivial interactions."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,211 @@
|
||||
---
|
||||
name: continue-conversation
|
||||
description: Resume previous work by building context from Basic Memory knowledge graph using memory URLs and recent activity
|
||||
---
|
||||
|
||||
# Continue Conversation
|
||||
|
||||
This skill helps you resume previous work by building context from the Basic Memory knowledge graph, enabling seamless continuation across sessions.
|
||||
|
||||
## When to Use
|
||||
|
||||
Use this skill when:
|
||||
- Starting a new session and need to pick up where you left off
|
||||
- User mentions previous work ("continue with...", "back to...", "where were we with...")
|
||||
- Need context about ongoing projects or specs
|
||||
- User asks about something discussed in a previous conversation
|
||||
- Working on a multi-session task
|
||||
|
||||
## Building Context
|
||||
|
||||
### 1. Identify What to Continue
|
||||
|
||||
Ask if unclear:
|
||||
- What topic or project to resume?
|
||||
- What timeframe to look at?
|
||||
- Any specific aspect to focus on?
|
||||
|
||||
### 2. Gather Context with MCP Tools
|
||||
|
||||
**Option A: Known Topic - Use build_context**
|
||||
|
||||
```python
|
||||
# Navigate knowledge graph from a known starting point
|
||||
mcp__basic-memory__build_context(
|
||||
url="memory://topic-or-note-name",
|
||||
depth=2, # How many relation hops to follow
|
||||
timeframe="7d", # Recent changes
|
||||
project="main" # or "specs" for specifications
|
||||
)
|
||||
```
|
||||
|
||||
Memory URL formats:
|
||||
- `memory://note-title` - Single note
|
||||
- `memory://folder/*` - All notes in folder
|
||||
- `memory://specs/SPEC-24*` - Pattern matching
|
||||
|
||||
**Option B: Recent Activity - What's been happening?**
|
||||
|
||||
```python
|
||||
# See what's changed recently
|
||||
mcp__basic-memory__recent_activity(
|
||||
timeframe="3d", # "1d", "1 week", "2 weeks"
|
||||
depth=1,
|
||||
project="main"
|
||||
)
|
||||
```
|
||||
|
||||
**Option C: Search for Context**
|
||||
|
||||
```python
|
||||
# Find relevant notes
|
||||
mcp__basic-memory__search_notes(
|
||||
query="search terms",
|
||||
page_size=10,
|
||||
project="main"
|
||||
)
|
||||
```
|
||||
|
||||
### 3. Read Key Notes
|
||||
|
||||
Once you identify relevant notes:
|
||||
|
||||
```python
|
||||
mcp__basic-memory__read_note(
|
||||
identifier="note-title-or-permalink",
|
||||
project="main"
|
||||
)
|
||||
```
|
||||
|
||||
### 4. Present Context to User
|
||||
|
||||
Summarize what you found:
|
||||
- Current state of the work
|
||||
- Recent changes or progress
|
||||
- Open items or next steps
|
||||
- Related context that might be helpful
|
||||
|
||||
## Context Strategies by Scenario
|
||||
|
||||
### Resuming a Spec Implementation
|
||||
|
||||
```python
|
||||
# 1. Read the spec
|
||||
mcp__basic-memory__read_note(
|
||||
identifier="SPEC-24: Postgres Database Migration",
|
||||
project="specs"
|
||||
)
|
||||
|
||||
# 2. Check recent activity on related topics
|
||||
mcp__basic-memory__build_context(
|
||||
url="memory://SPEC-24*",
|
||||
timeframe="7d",
|
||||
project="specs"
|
||||
)
|
||||
|
||||
# 3. Look at what's been done in the codebase
|
||||
# (Use regular file tools for this)
|
||||
```
|
||||
|
||||
### Continuing General Work
|
||||
|
||||
```python
|
||||
# 1. Check recent activity across projects
|
||||
mcp__basic-memory__recent_activity(
|
||||
timeframe="3d",
|
||||
project="main"
|
||||
)
|
||||
|
||||
# 2. Read any notes from recent sessions
|
||||
mcp__basic-memory__read_note(
|
||||
identifier="relevant-note",
|
||||
project="main"
|
||||
)
|
||||
```
|
||||
|
||||
### Following Up on a Topic
|
||||
|
||||
```python
|
||||
# 1. Search for the topic
|
||||
mcp__basic-memory__search_notes(
|
||||
query="topic keywords",
|
||||
project="main"
|
||||
)
|
||||
|
||||
# 2. Build context from best match
|
||||
mcp__basic-memory__build_context(
|
||||
url="memory://found-note-permalink",
|
||||
depth=2,
|
||||
project="main"
|
||||
)
|
||||
```
|
||||
|
||||
## Timeframe Reference
|
||||
|
||||
Natural language timeframes:
|
||||
- `"today"` - Current day
|
||||
- `"yesterday"` - Previous day
|
||||
- `"3d"` or `"3 days"` - Last 3 days
|
||||
- `"1 week"` or `"7d"` - Last week
|
||||
- `"2 weeks"` - Last 2 weeks
|
||||
- `"1 month"` - Last month
|
||||
|
||||
## Project Reference
|
||||
|
||||
Common projects:
|
||||
- `main` - Primary knowledge base
|
||||
- `specs` - Specifications and design docs
|
||||
- `basic-memory-llc` - Business/company notes
|
||||
- `getting-started` - Tutorial content
|
||||
|
||||
List available projects:
|
||||
```python
|
||||
mcp__basic-memory__list_memory_projects()
|
||||
```
|
||||
|
||||
## Example Conversations
|
||||
|
||||
### User: "Let's continue with the Postgres migration"
|
||||
|
||||
```
|
||||
1. Read SPEC-24 from specs project
|
||||
2. Check for related notes about implementation progress
|
||||
3. Summarize:
|
||||
- Spec overview and goals
|
||||
- What's been completed (checkmarks)
|
||||
- What's pending (checkboxes)
|
||||
- Any blockers or decisions needed
|
||||
```
|
||||
|
||||
### User: "What was I working on yesterday?"
|
||||
|
||||
```
|
||||
1. Get recent activity for last 2 days
|
||||
2. List modified notes with brief descriptions
|
||||
3. Ask which topic to dive into
|
||||
```
|
||||
|
||||
### User: "Back to the async client pattern"
|
||||
|
||||
```
|
||||
1. Search for "async client pattern"
|
||||
2. Build context from matching note
|
||||
3. Include related notes via relations
|
||||
4. Present the full picture
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Start broad, then narrow** - Get overview first, then specific details
|
||||
2. **Follow relations** - Knowledge graph connections are valuable
|
||||
3. **Check multiple projects** - Specs might be separate from implementation notes
|
||||
4. **Present incrementally** - Share what you find as you go
|
||||
5. **Confirm understanding** - Verify the context is what user needs
|
||||
6. **Update as you go** - Capture new progress in notes during the session
|
||||
|
||||
## Combining with Other Skills
|
||||
|
||||
After building context, you might:
|
||||
- Use **knowledge-capture** to document new progress
|
||||
- Use **spec-driven-development** if continuing a spec implementation
|
||||
- Create new notes linking to the context you gathered
|
||||
@@ -0,0 +1,261 @@
|
||||
---
|
||||
name: edit-note-local
|
||||
description: Edit Basic Memory notes directly as local files - enables full file editing with automatic sync (local installations only)
|
||||
---
|
||||
|
||||
# Edit Note Local
|
||||
|
||||
This skill enables direct file-based editing of Basic Memory notes. It works by editing the actual markdown files in the knowledge base, which Basic Memory's sync service automatically picks up. This provides a more seamless editing experience for local installations.
|
||||
|
||||
## When to Use
|
||||
|
||||
Use this skill when:
|
||||
- User has a local Basic Memory installation (not cloud-only)
|
||||
- User wants to make substantial edits to a note
|
||||
- User prefers working with the full file content
|
||||
- User wants changes to sync automatically via `basic-memory sync --watch`
|
||||
|
||||
**Note:** This skill requires local file access. For cloud-only users, use the `edit-note` skill instead.
|
||||
|
||||
## Editing Workflow
|
||||
|
||||
### 1. Find the Note's File Path
|
||||
|
||||
First, get the note metadata to find its file location:
|
||||
|
||||
```python
|
||||
# Search for the note
|
||||
mcp__basic-memory__search_notes(
|
||||
query="note title or keywords",
|
||||
project="main"
|
||||
)
|
||||
|
||||
# Read the note to get file_path from metadata
|
||||
mcp__basic-memory__read_note(
|
||||
identifier="note-title",
|
||||
project="main"
|
||||
)
|
||||
```
|
||||
|
||||
The response includes `file_path` which gives the relative path within the knowledge base.
|
||||
|
||||
### 2. Determine Full File Path
|
||||
|
||||
Basic Memory projects have a root directory. Common locations:
|
||||
- Default: `~/basic-memory/`
|
||||
- Custom: Check project configuration
|
||||
|
||||
Construct the full path:
|
||||
```
|
||||
{project_root}/{file_path}
|
||||
```
|
||||
|
||||
For example:
|
||||
- Project root: `/Users/username/basic-memory`
|
||||
- File path from note: `notes/My Note.md`
|
||||
- Full path: `/Users/username/basic-memory/notes/My Note.md`
|
||||
|
||||
### 3. Read the File
|
||||
|
||||
Use Claude Code's Read tool to get the full file content:
|
||||
|
||||
```python
|
||||
Read(file_path="/Users/username/basic-memory/notes/My Note.md")
|
||||
```
|
||||
|
||||
Display the content to the user, explaining the structure:
|
||||
- Frontmatter (YAML between `---` markers)
|
||||
- Main content
|
||||
- Observations section
|
||||
- Relations section
|
||||
|
||||
### 4. Edit the File
|
||||
|
||||
Use Claude Code's Edit tool for precise changes:
|
||||
|
||||
```python
|
||||
Edit(
|
||||
file_path="/Users/username/basic-memory/notes/My Note.md",
|
||||
old_string="text to replace",
|
||||
new_string="new text"
|
||||
)
|
||||
```
|
||||
|
||||
Or use Write for complete rewrites:
|
||||
|
||||
```python
|
||||
Write(
|
||||
file_path="/Users/username/basic-memory/notes/My Note.md",
|
||||
content="Complete new file content..."
|
||||
)
|
||||
```
|
||||
|
||||
### 5. Sync Happens Automatically
|
||||
|
||||
If the user has `basic-memory sync --watch` running, changes are picked up automatically. Otherwise, they can run:
|
||||
```bash
|
||||
basic-memory sync
|
||||
```
|
||||
|
||||
## File Structure Reference
|
||||
|
||||
Basic Memory notes follow this markdown structure:
|
||||
|
||||
```markdown
|
||||
---
|
||||
title: Note Title
|
||||
type: note
|
||||
permalink: note-title
|
||||
tags:
|
||||
- tag1
|
||||
- tag2
|
||||
---
|
||||
|
||||
# Note Title
|
||||
|
||||
## Context
|
||||
|
||||
Background and situation explanation.
|
||||
|
||||
## Main Content
|
||||
|
||||
The primary content of the note...
|
||||
|
||||
## Observations
|
||||
|
||||
- [category] Observation text #optional-tag
|
||||
- [decision] A decision that was made #tag
|
||||
- [insight] An insight discovered
|
||||
|
||||
## Relations
|
||||
|
||||
- relates-to [[Other Note]]
|
||||
- implements [[Parent Concept]]
|
||||
- learned-from [[Source Note]]
|
||||
```
|
||||
|
||||
## Editing Patterns
|
||||
|
||||
### Edit Frontmatter Tags
|
||||
|
||||
```python
|
||||
Edit(
|
||||
file_path="/path/to/note.md",
|
||||
old_string="tags:\n- old-tag",
|
||||
new_string="tags:\n- old-tag\n- new-tag"
|
||||
)
|
||||
```
|
||||
|
||||
### Add New Section
|
||||
|
||||
```python
|
||||
Edit(
|
||||
file_path="/path/to/note.md",
|
||||
old_string="## Observations",
|
||||
new_string="## New Section\n\nNew content here.\n\n## Observations"
|
||||
)
|
||||
```
|
||||
|
||||
### Modify Observation
|
||||
|
||||
```python
|
||||
Edit(
|
||||
file_path="/path/to/note.md",
|
||||
old_string="- [decision] Old decision",
|
||||
new_string="- [decision] Updated decision with new info #updated"
|
||||
)
|
||||
```
|
||||
|
||||
### Add Relation
|
||||
|
||||
```python
|
||||
Edit(
|
||||
file_path="/path/to/note.md",
|
||||
old_string="## Relations\n",
|
||||
new_string="## Relations\n\n- relates-to [[New Related Note]]\n"
|
||||
)
|
||||
```
|
||||
|
||||
### Complete Rewrite
|
||||
|
||||
For major changes, read the file, construct new content preserving the frontmatter structure, and write:
|
||||
|
||||
```python
|
||||
Write(
|
||||
file_path="/path/to/note.md",
|
||||
content="""---
|
||||
title: Note Title
|
||||
type: note
|
||||
permalink: note-title
|
||||
tags:
|
||||
- updated
|
||||
---
|
||||
|
||||
# Note Title
|
||||
|
||||
Completely rewritten content...
|
||||
|
||||
## Observations
|
||||
|
||||
- [rewrite] Complete rewrite of this note #major-update
|
||||
|
||||
## Relations
|
||||
|
||||
- updates [[Previous Version]]
|
||||
"""
|
||||
)
|
||||
```
|
||||
|
||||
## Finding the Project Root
|
||||
|
||||
To find where Basic Memory stores files, you can:
|
||||
|
||||
1. **Check common locations:**
|
||||
- `~/basic-memory/`
|
||||
- `~/Documents/basic-memory/`
|
||||
- Current working directory
|
||||
|
||||
2. **Use the list_directory tool:**
|
||||
```python
|
||||
mcp__basic-memory__list_directory(
|
||||
dir_name="/",
|
||||
project="main"
|
||||
)
|
||||
```
|
||||
|
||||
3. **Ask the user:** "Where is your Basic Memory knowledge base located?"
|
||||
|
||||
## Advantages of Local Editing
|
||||
|
||||
1. **Full file access** - Edit any part of the file including frontmatter
|
||||
2. **Multi-line edits** - Make complex structural changes easily
|
||||
3. **Batch operations** - Edit multiple files in sequence
|
||||
4. **Version control** - Changes tracked by git if the folder is a repo
|
||||
5. **Instant preview** - Use any markdown editor alongside
|
||||
6. **Auto-sync** - `sync --watch` picks up changes automatically
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Preserve frontmatter** - Don't break the YAML structure
|
||||
2. **Keep valid markdown** - Maintain proper formatting
|
||||
3. **Preserve permalinks** - Changing them can break links
|
||||
4. **Show diffs** - Tell the user what changed
|
||||
5. **Suggest sync** - Remind about `basic-memory sync` if not watching
|
||||
6. **Handle missing files** - Check if file exists before editing
|
||||
|
||||
## Example Conversation
|
||||
|
||||
**User:** "I want to completely rewrite my architecture decision note"
|
||||
|
||||
**Claude:**
|
||||
1. Searches for the note via MCP
|
||||
2. Gets the file path
|
||||
3. Reads the current file content
|
||||
4. Asks: "Here's the current note. What would you like the new version to say?"
|
||||
|
||||
**User:** Provides new content
|
||||
|
||||
**Claude:**
|
||||
1. Preserves the frontmatter (title, permalink, type)
|
||||
2. Writes the new content using Write tool
|
||||
3. Confirms: "Updated the file at `/path/to/note.md`. If you have sync --watch running, it's already indexed. Otherwise run `basic-memory sync`."
|
||||
@@ -0,0 +1,209 @@
|
||||
---
|
||||
name: edit-note
|
||||
description: Interactively edit Basic Memory notes using MCP tools - view, modify, and update notes in a conversational workflow (works with cloud and local)
|
||||
---
|
||||
|
||||
# Edit Note
|
||||
|
||||
This skill enables interactive editing of Basic Memory notes using MCP tools. It works with both Basic Memory Cloud and local installations since it operates through the MCP interface rather than direct file access.
|
||||
|
||||
## When to Use
|
||||
|
||||
Use this skill when:
|
||||
- User wants to edit an existing note
|
||||
- User asks to update, change, or modify note content
|
||||
- User wants to refine observations or relations in a note
|
||||
- User says things like "edit my note about...", "update the...", "change X to Y in..."
|
||||
|
||||
## Editing Workflow
|
||||
|
||||
### 1. Fetch the Current Note
|
||||
|
||||
First, retrieve the note to show the user what exists:
|
||||
|
||||
```python
|
||||
mcp__basic-memory__read_note(
|
||||
identifier="Note Title or permalink",
|
||||
project="main" # or specified project
|
||||
)
|
||||
```
|
||||
|
||||
Present the note content clearly, highlighting:
|
||||
- Current title and metadata
|
||||
- Main content sections
|
||||
- Observations (with categories)
|
||||
- Relations (with link targets)
|
||||
|
||||
### 2. Understand the Edit Request
|
||||
|
||||
Ask clarifying questions if needed:
|
||||
- Which section to modify?
|
||||
- What specifically to change?
|
||||
- Add new content or replace existing?
|
||||
|
||||
### 3. Apply the Edit
|
||||
|
||||
Use the appropriate `edit_note` operation:
|
||||
|
||||
**Append** - Add content to the end:
|
||||
```python
|
||||
mcp__basic-memory__edit_note(
|
||||
identifier="note-title",
|
||||
operation="append",
|
||||
content="\n\n## New Section\n\nNew content here...",
|
||||
project="main"
|
||||
)
|
||||
```
|
||||
|
||||
**Prepend** - Add content to the beginning:
|
||||
```python
|
||||
mcp__basic-memory__edit_note(
|
||||
identifier="note-title",
|
||||
operation="prepend",
|
||||
content="# Updated Header\n\n",
|
||||
project="main"
|
||||
)
|
||||
```
|
||||
|
||||
**Find and Replace** - Replace specific text:
|
||||
```python
|
||||
mcp__basic-memory__edit_note(
|
||||
identifier="note-title",
|
||||
operation="find_replace",
|
||||
find_text="old text to find",
|
||||
content="new replacement text",
|
||||
project="main"
|
||||
)
|
||||
```
|
||||
|
||||
**Replace Section** - Replace an entire section by heading:
|
||||
```python
|
||||
mcp__basic-memory__edit_note(
|
||||
identifier="note-title",
|
||||
operation="replace_section",
|
||||
section="## Section Heading",
|
||||
content="## Section Heading\n\nCompletely new section content...",
|
||||
project="main"
|
||||
)
|
||||
```
|
||||
|
||||
### 4. Show the Result
|
||||
|
||||
After editing, fetch and display the updated note:
|
||||
|
||||
```python
|
||||
mcp__basic-memory__read_note(
|
||||
identifier="note-title",
|
||||
project="main"
|
||||
)
|
||||
```
|
||||
|
||||
Highlight what changed so the user can verify.
|
||||
|
||||
## Edit Operations Reference
|
||||
|
||||
| Operation | Use Case | Required Parameters |
|
||||
|-----------|----------|---------------------|
|
||||
| `append` | Add to end | `content` |
|
||||
| `prepend` | Add to beginning | `content` |
|
||||
| `find_replace` | Change specific text | `find_text`, `content` |
|
||||
| `replace_section` | Rewrite a section | `section`, `content` |
|
||||
|
||||
## Common Edit Patterns
|
||||
|
||||
### Adding a New Observation
|
||||
|
||||
```python
|
||||
mcp__basic-memory__edit_note(
|
||||
identifier="note-title",
|
||||
operation="find_replace",
|
||||
find_text="## Observations",
|
||||
content="## Observations\n\n- [new-category] New observation here #tag",
|
||||
project="main"
|
||||
)
|
||||
```
|
||||
|
||||
Or append to observations section:
|
||||
```python
|
||||
mcp__basic-memory__edit_note(
|
||||
identifier="note-title",
|
||||
operation="append",
|
||||
content="\n- [insight] Additional insight discovered #tag",
|
||||
project="main"
|
||||
)
|
||||
```
|
||||
|
||||
### Adding a New Relation
|
||||
|
||||
```python
|
||||
mcp__basic-memory__edit_note(
|
||||
identifier="note-title",
|
||||
operation="find_replace",
|
||||
find_text="## Relations",
|
||||
content="## Relations\n\n- relates-to [[New Related Note]]",
|
||||
project="main"
|
||||
)
|
||||
```
|
||||
|
||||
### Updating a Specific Observation
|
||||
|
||||
```python
|
||||
mcp__basic-memory__edit_note(
|
||||
identifier="note-title",
|
||||
operation="find_replace",
|
||||
find_text="- [decision] Old decision text",
|
||||
content="- [decision] Updated decision with new context #updated",
|
||||
project="main"
|
||||
)
|
||||
```
|
||||
|
||||
### Rewriting the Context Section
|
||||
|
||||
```python
|
||||
mcp__basic-memory__edit_note(
|
||||
identifier="note-title",
|
||||
operation="replace_section",
|
||||
section="## Context",
|
||||
content="## Context\n\nCompletely rewritten context explaining the new situation...",
|
||||
project="main"
|
||||
)
|
||||
```
|
||||
|
||||
## Multi-Step Editing Session
|
||||
|
||||
For complex edits, work iteratively:
|
||||
|
||||
1. **Show current state** → Read and display the note
|
||||
2. **First edit** → Apply one change
|
||||
3. **Show result** → Display updated note
|
||||
4. **Next edit** → Apply another change if needed
|
||||
5. **Confirm complete** → Final display and confirmation
|
||||
|
||||
This keeps the user informed and allows course correction.
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Always show before and after** - User should see what changed
|
||||
2. **One edit at a time** - For complex changes, do multiple operations
|
||||
3. **Preserve structure** - Maintain the note's markdown format
|
||||
4. **Be careful with find_replace** - Ensure the find_text is unique
|
||||
5. **Confirm destructive changes** - Ask before replacing large sections
|
||||
6. **Keep observations formatted** - Maintain `[category]` prefix format
|
||||
7. **Keep relations formatted** - Maintain `- relation-type [[Target]]` format
|
||||
|
||||
## Example Conversation
|
||||
|
||||
**User:** "Edit my note about the async client pattern - add an observation about testing"
|
||||
|
||||
**Claude:**
|
||||
1. Fetches "Async Client Pattern" note
|
||||
2. Displays current content
|
||||
3. Asks: "What observation about testing would you like to add?"
|
||||
|
||||
**User:** "That the context manager pattern makes mocking easier in tests"
|
||||
|
||||
**Claude:**
|
||||
1. Uses `edit_note` with `append` to add:
|
||||
`- [testing] Context manager pattern simplifies mocking in unit tests #testability`
|
||||
2. Fetches and displays updated note
|
||||
3. Confirms: "Added the testing observation. Here's the updated note..."
|
||||
@@ -0,0 +1,211 @@
|
||||
---
|
||||
name: knowledge-capture
|
||||
description: Capture insights, decisions, and learnings from conversations into structured Basic Memory notes with observations and relations
|
||||
---
|
||||
|
||||
# Knowledge Capture
|
||||
|
||||
This skill helps you capture valuable information from conversations into Basic Memory's knowledge graph using structured notes with observations and relations.
|
||||
|
||||
## When to Use
|
||||
|
||||
Use this skill when:
|
||||
- Important decisions are made during a conversation
|
||||
- Technical insights or patterns are discovered
|
||||
- Problems are solved and the solution should be preserved
|
||||
- Design trade-offs are discussed
|
||||
- Architecture or implementation approaches are chosen
|
||||
- Learnings from debugging or investigation emerge
|
||||
|
||||
## Capture Process
|
||||
|
||||
### 1. Identify Valuable Information
|
||||
|
||||
Look for:
|
||||
- **Decisions**: Choices made and their rationale
|
||||
- **Insights**: New understanding or "aha" moments
|
||||
- **Patterns**: Reusable approaches or solutions
|
||||
- **Trade-offs**: Options considered and why one was chosen
|
||||
- **Learnings**: What worked, what didn't, and why
|
||||
- **Context**: Background that would help future understanding
|
||||
|
||||
### 2. Structure the Note
|
||||
|
||||
Use Basic Memory's knowledge format:
|
||||
|
||||
```markdown
|
||||
---
|
||||
title: Descriptive Title
|
||||
type: note
|
||||
tags:
|
||||
- relevant
|
||||
- tags
|
||||
---
|
||||
|
||||
# Title
|
||||
|
||||
## Context
|
||||
Brief background explaining the situation.
|
||||
|
||||
## Content
|
||||
Main content organized logically.
|
||||
|
||||
## Observations
|
||||
|
||||
- [decision] What was decided and why #tag
|
||||
- [insight] Key understanding gained #tag
|
||||
- [pattern] Reusable approach identified #tag
|
||||
- [learning] What we learned from this #tag
|
||||
- [tradeoff] Option A chosen over B because... #tag
|
||||
|
||||
## Relations
|
||||
|
||||
- relates-to [[Related Concept]]
|
||||
- implements [[Parent Spec or Design]]
|
||||
- learned-from [[Source of Learning]]
|
||||
```
|
||||
|
||||
### 3. Choose Appropriate Categories
|
||||
|
||||
Common observation categories:
|
||||
- `[decision]` - Choices made
|
||||
- `[insight]` - Understanding gained
|
||||
- `[pattern]` - Reusable approaches
|
||||
- `[learning]` - Lessons learned
|
||||
- `[tradeoff]` - Options weighed
|
||||
- `[problem]` - Issues identified
|
||||
- `[solution]` - Fixes applied
|
||||
- `[architecture]` - Structural decisions
|
||||
- `[implementation]` - Code-level choices
|
||||
- `[constraint]` - Limitations discovered
|
||||
- `[requirement]` - Needs identified
|
||||
|
||||
### 4. Create Meaningful Relations
|
||||
|
||||
Link to related knowledge:
|
||||
- `relates-to` - General association
|
||||
- `implements` - Realizes a spec or design
|
||||
- `extends` - Builds upon existing concept
|
||||
- `learned-from` - Source of insight
|
||||
- `enables` - Makes something possible
|
||||
- `depends-on` - Requires another concept
|
||||
- `solves` - Addresses a problem
|
||||
|
||||
## MCP Tools to Use
|
||||
|
||||
```python
|
||||
# Write a new note
|
||||
mcp__basic-memory__write_note(
|
||||
title="Your Note Title",
|
||||
content="Full markdown content...",
|
||||
folder="appropriate/folder",
|
||||
tags=["tag1", "tag2"],
|
||||
project="main" # or appropriate project
|
||||
)
|
||||
|
||||
# Search for related notes to link
|
||||
mcp__basic-memory__search_notes(
|
||||
query="relevant terms",
|
||||
project="main"
|
||||
)
|
||||
|
||||
# Read existing notes for context
|
||||
mcp__basic-memory__read_note(
|
||||
identifier="note-title-or-permalink",
|
||||
project="main"
|
||||
)
|
||||
```
|
||||
|
||||
## Folder Organization
|
||||
|
||||
Choose appropriate folders:
|
||||
- `decisions/` - Architecture and design decisions
|
||||
- `learnings/` - Insights and lessons learned
|
||||
- `patterns/` - Reusable approaches
|
||||
- `debug-logs/` - Problem investigations
|
||||
- `conversations/` - Imported conversation summaries
|
||||
|
||||
## Examples
|
||||
|
||||
### Capturing a Technical Decision
|
||||
|
||||
```markdown
|
||||
---
|
||||
title: FastAPI Async Client Pattern
|
||||
type: note
|
||||
tags:
|
||||
- architecture
|
||||
- fastapi
|
||||
- async
|
||||
---
|
||||
|
||||
# FastAPI Async Client Pattern
|
||||
|
||||
## Context
|
||||
During implementation of MCP tools, we needed to decide how to handle HTTP client lifecycle.
|
||||
|
||||
## Decision
|
||||
Use context manager pattern for HTTP clients instead of module-level singletons.
|
||||
|
||||
## Rationale
|
||||
- Proper resource management
|
||||
- Supports three deployment modes (local ASGI, CLI cloud, cloud app)
|
||||
- Auth happens at client creation, not per-request
|
||||
- Enables dependency injection for testing
|
||||
|
||||
## Observations
|
||||
|
||||
- [decision] Context manager pattern for HTTP clients enables proper resource cleanup #architecture
|
||||
- [pattern] Factory pattern allows different client configurations per deployment mode #flexibility
|
||||
- [tradeoff] Slightly more verbose than singleton but much more flexible #engineering
|
||||
|
||||
## Relations
|
||||
|
||||
- implements [[SPEC-16 MCP Cloud Service Consolidation]]
|
||||
- enables [[Cloud App Integration]]
|
||||
```
|
||||
|
||||
### Capturing a Debugging Insight
|
||||
|
||||
```markdown
|
||||
---
|
||||
title: SQLite WAL Mode Performance Fix
|
||||
type: note
|
||||
tags:
|
||||
- debugging
|
||||
- sqlite
|
||||
- performance
|
||||
---
|
||||
|
||||
# SQLite WAL Mode Performance Fix
|
||||
|
||||
## Problem
|
||||
Sync operations were slow with multiple concurrent writes.
|
||||
|
||||
## Investigation
|
||||
Found that default SQLite journaling was causing lock contention.
|
||||
|
||||
## Solution
|
||||
Enabled WAL (Write-Ahead Logging) mode for the database connection.
|
||||
|
||||
## Observations
|
||||
|
||||
- [problem] Default SQLite journaling causes lock contention under concurrent writes #performance
|
||||
- [solution] WAL mode significantly improves concurrent write performance #sqlite
|
||||
- [learning] Always consider WAL mode for SQLite in applications with concurrent access #database
|
||||
|
||||
## Relations
|
||||
|
||||
- solves [[Sync Performance Issues]]
|
||||
- relates-to [[SPEC-19 Sync Performance]]
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Capture immediately** - Write notes while context is fresh
|
||||
2. **Be specific** - Include concrete details, not vague summaries
|
||||
3. **Link liberally** - More relations = better knowledge graph
|
||||
4. **Use tags** - Enable discovery via search
|
||||
5. **Include context** - Future you won't remember the situation
|
||||
6. **Prefer facts over opinions** - Observations should be verifiable
|
||||
7. **Keep notes atomic** - One concept per note when possible
|
||||
@@ -0,0 +1,283 @@
|
||||
---
|
||||
name: knowledge-organize
|
||||
description: Help organize, link, and maintain the Basic Memory knowledge graph - find orphan notes, suggest relations, identify duplicates, and improve overall knowledge structure
|
||||
---
|
||||
|
||||
# Knowledge Organize
|
||||
|
||||
This skill helps users maintain a healthy, well-connected knowledge graph. As notes accumulate, it becomes valuable to periodically organize, link, and curate the knowledge base.
|
||||
|
||||
## When to Use
|
||||
|
||||
Use this skill when:
|
||||
- User asks to organize their notes
|
||||
- User wants to find connections between notes
|
||||
- User mentions orphan or unlinked notes
|
||||
- User wants to clean up or improve their knowledge base
|
||||
- User asks about duplicate or similar notes
|
||||
- User wants help with folder organization
|
||||
- User asks to review or audit their notes
|
||||
- Phrases like "help me organize", "find related notes", "what's not linked", "clean up my notes"
|
||||
|
||||
## Organization Capabilities
|
||||
|
||||
### 1. Find Orphan Notes
|
||||
|
||||
Identify notes that have no relations to other notes - they're isolated in the knowledge graph.
|
||||
|
||||
```python
|
||||
# Get all notes
|
||||
mcp__basic-memory__search_notes(
|
||||
query="*",
|
||||
page_size=50,
|
||||
project="main"
|
||||
)
|
||||
|
||||
# For each note, check if it has relations
|
||||
# Orphans have empty Relations sections
|
||||
```
|
||||
|
||||
**What to do with orphans:**
|
||||
- Suggest potential relations based on content similarity
|
||||
- Ask if they should be linked to existing topics
|
||||
- Propose creating hub notes to connect related orphans
|
||||
|
||||
### 2. Suggest Relations
|
||||
|
||||
Analyze note content and suggest meaningful connections.
|
||||
|
||||
```python
|
||||
# Read a note
|
||||
mcp__basic-memory__read_note(
|
||||
identifier="note-to-analyze",
|
||||
project="main"
|
||||
)
|
||||
|
||||
# Search for potentially related notes
|
||||
mcp__basic-memory__search_notes(
|
||||
query="key terms from the note",
|
||||
project="main"
|
||||
)
|
||||
|
||||
# Suggest relations based on:
|
||||
# - Shared topics or concepts
|
||||
# - Complementary content (problem/solution, question/answer)
|
||||
# - Sequential relationship (part 1, part 2)
|
||||
# - Hierarchical (parent concept, child detail)
|
||||
```
|
||||
|
||||
**Relation types to suggest:**
|
||||
- `relates-to` - General topical connection
|
||||
- `extends` - Builds upon or expands
|
||||
- `implements` - Realizes a concept
|
||||
- `depends-on` - Requires understanding of
|
||||
- `contradicts` - Presents alternative view
|
||||
- `learned-from` - Source of insight
|
||||
- `enables` - Makes something possible
|
||||
|
||||
### 3. Identify Similar/Duplicate Notes
|
||||
|
||||
Find notes that may cover the same topic.
|
||||
|
||||
```python
|
||||
# Search for notes with similar titles or content
|
||||
mcp__basic-memory__search_notes(
|
||||
query="topic keywords",
|
||||
project="main"
|
||||
)
|
||||
|
||||
# Compare results for overlap
|
||||
# Look for:
|
||||
# - Similar titles
|
||||
# - Overlapping observations
|
||||
# - Same tags
|
||||
# - Related timestamps (created around same time)
|
||||
```
|
||||
|
||||
**Actions for duplicates:**
|
||||
- Merge into a single comprehensive note
|
||||
- Link them with `supersedes` or `updates` relations
|
||||
- Differentiate by adding context about their distinct focus
|
||||
|
||||
### 4. Folder Organization Review
|
||||
|
||||
Analyze folder structure and suggest improvements.
|
||||
|
||||
```python
|
||||
# List directory structure
|
||||
mcp__basic-memory__list_directory(
|
||||
dir_name="/",
|
||||
depth=3,
|
||||
project="main"
|
||||
)
|
||||
|
||||
# Identify:
|
||||
# - Overcrowded folders
|
||||
# - Single-note folders
|
||||
# - Inconsistent naming
|
||||
# - Notes that might belong elsewhere
|
||||
```
|
||||
|
||||
**Organization suggestions:**
|
||||
- Group related notes into topic folders
|
||||
- Create subfolders for large categories
|
||||
- Suggest consistent naming conventions
|
||||
- Move misplaced notes
|
||||
|
||||
### 5. Tag Consistency
|
||||
|
||||
Review and normalize tags across notes.
|
||||
|
||||
```python
|
||||
# Search notes to analyze tag patterns
|
||||
mcp__basic-memory__search_notes(
|
||||
query="*",
|
||||
page_size=100,
|
||||
project="main"
|
||||
)
|
||||
|
||||
# Look for:
|
||||
# - Similar tags (architecture vs arch)
|
||||
# - Unused tags
|
||||
# - Over-used generic tags
|
||||
# - Missing tags on relevant notes
|
||||
```
|
||||
|
||||
**Tag improvements:**
|
||||
- Suggest tag standardization (pick one variant)
|
||||
- Propose new tags for common themes
|
||||
- Identify notes missing obvious tags
|
||||
|
||||
### 6. Create Index/Hub Notes
|
||||
|
||||
Generate notes that serve as navigation hubs for related topics.
|
||||
|
||||
```python
|
||||
# After identifying a cluster of related notes
|
||||
mcp__basic-memory__write_note(
|
||||
title="Architecture Decisions Index",
|
||||
content="""---
|
||||
title: Architecture Decisions Index
|
||||
type: index
|
||||
tags:
|
||||
- architecture
|
||||
- index
|
||||
---
|
||||
|
||||
# Architecture Decisions Index
|
||||
|
||||
A hub linking all architecture-related decisions and patterns.
|
||||
|
||||
## Decisions
|
||||
|
||||
- [[Database Selection Decision]]
|
||||
- [[API Design Patterns]]
|
||||
- [[Authentication Architecture]]
|
||||
|
||||
## Patterns
|
||||
|
||||
- [[Repository Pattern]]
|
||||
- [[Async Client Pattern]]
|
||||
|
||||
## Observations
|
||||
|
||||
- [index] Central hub for architecture knowledge #navigation
|
||||
|
||||
## Relations
|
||||
|
||||
- indexes [[Architecture]]
|
||||
""",
|
||||
folder="indexes",
|
||||
project="main"
|
||||
)
|
||||
```
|
||||
|
||||
### 7. Enrich Sparse Notes
|
||||
|
||||
Find notes lacking observations or structure and suggest improvements.
|
||||
|
||||
```python
|
||||
# Read a sparse note
|
||||
mcp__basic-memory__read_note(
|
||||
identifier="sparse-note",
|
||||
project="main"
|
||||
)
|
||||
|
||||
# If missing:
|
||||
# - Observations section → suggest categories
|
||||
# - Relations section → suggest links
|
||||
# - Tags → suggest relevant tags
|
||||
# - Context → suggest adding background
|
||||
```
|
||||
|
||||
## Organization Workflows
|
||||
|
||||
### Quick Health Check
|
||||
|
||||
A fast overview of knowledge base status:
|
||||
|
||||
1. Count total notes
|
||||
2. Identify orphan count
|
||||
3. List recently modified
|
||||
4. Check for obvious duplicates
|
||||
5. Report folder distribution
|
||||
|
||||
### Deep Organization Session
|
||||
|
||||
Thorough review and improvement:
|
||||
|
||||
1. **Audit phase** - Catalog all notes, identify issues
|
||||
2. **Orphan phase** - Address unlinked notes
|
||||
3. **Relation phase** - Suggest new connections
|
||||
4. **Duplicate phase** - Merge or differentiate similar notes
|
||||
5. **Structure phase** - Reorganize folders if needed
|
||||
6. **Index phase** - Create hub notes for major topics
|
||||
|
||||
### Topic-Focused Organization
|
||||
|
||||
Organize around a specific subject:
|
||||
|
||||
1. Find all notes related to topic
|
||||
2. Map existing relations
|
||||
3. Identify gaps in the topic graph
|
||||
4. Suggest new notes to fill gaps
|
||||
5. Create topic index note
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Work incrementally** - Don't reorganize everything at once
|
||||
2. **Confirm before changing** - Always ask before moving/editing notes
|
||||
3. **Preserve permalinks** - Moving is okay, changing permalinks breaks links
|
||||
4. **Show the graph** - Help user visualize connections
|
||||
5. **Explain suggestions** - Say why a relation makes sense
|
||||
6. **Respect user's system** - Enhance their organization, don't impose a new one
|
||||
|
||||
## Example Conversations
|
||||
|
||||
**User:** "Help me organize my notes"
|
||||
|
||||
**Claude:**
|
||||
1. Runs health check on the knowledge base
|
||||
2. Reports: "You have 47 notes. I found 12 orphan notes and 3 potential duplicates."
|
||||
3. Asks: "Would you like to start by connecting the orphan notes, or review the duplicates first?"
|
||||
|
||||
**User:** "Find notes that should be linked to my API design note"
|
||||
|
||||
**Claude:**
|
||||
1. Reads the API design note
|
||||
2. Searches for related content
|
||||
3. Suggests: "I found 5 notes that could relate:
|
||||
- 'REST Best Practices' → relates-to
|
||||
- 'Authentication Flow' → implements
|
||||
- 'Rate Limiting Decision' → extends
|
||||
Would you like me to add any of these relations?"
|
||||
|
||||
**User:** "Are there any notes about similar topics?"
|
||||
|
||||
**Claude:**
|
||||
1. Analyzes note titles and content
|
||||
2. Identifies clusters of similar notes
|
||||
3. Reports: "I found these potential overlaps:
|
||||
- 'Auth Flow' and 'Authentication Design' cover similar ground
|
||||
- 'DB Schema v1' and 'DB Schema v2' might need a 'supersedes' relation
|
||||
Would you like to review any of these?"
|
||||
@@ -0,0 +1,213 @@
|
||||
---
|
||||
name: research
|
||||
description: Research a topic thoroughly and produce a structured report saved to Basic Memory - investigate concepts, gather context, and document findings
|
||||
---
|
||||
|
||||
# Research
|
||||
|
||||
This skill helps conduct thorough research on a topic and produces a structured report that gets saved to Basic Memory for future reference.
|
||||
|
||||
## When to Use
|
||||
|
||||
Use this skill when:
|
||||
- User asks to research or investigate something
|
||||
- User wants to understand a concept, technology, or approach
|
||||
- User needs context gathered before making a decision
|
||||
- User asks "what is...", "how does... work", "explore...", "investigate..."
|
||||
- User wants findings documented for later
|
||||
- Phrases like "research this", "look into", "find out about", "explore options for"
|
||||
|
||||
## Research Process
|
||||
|
||||
### 1. Understand the Research Question
|
||||
|
||||
Clarify what specifically to investigate:
|
||||
- What is the core question or topic?
|
||||
- What scope - broad overview or deep dive?
|
||||
- Any specific aspects to focus on?
|
||||
- What will the research inform (a decision, implementation, understanding)?
|
||||
|
||||
### 2. Gather Information
|
||||
|
||||
Use available tools to collect information:
|
||||
|
||||
**For codebase research:**
|
||||
- Search the codebase for relevant code
|
||||
- Read documentation and comments
|
||||
- Trace how things connect
|
||||
- Look at tests for usage examples
|
||||
|
||||
**For concept research:**
|
||||
- Use web search for current information
|
||||
- Fetch documentation from official sources
|
||||
- Look for examples and best practices
|
||||
- Compare alternatives if relevant
|
||||
|
||||
**For Basic Memory context:**
|
||||
```python
|
||||
# Check what we already know
|
||||
mcp__basic-memory__search_notes(
|
||||
query="topic keywords",
|
||||
project="main"
|
||||
)
|
||||
|
||||
# Build context from related notes
|
||||
mcp__basic-memory__build_context(
|
||||
url="memory://related-topic",
|
||||
depth=2,
|
||||
project="main"
|
||||
)
|
||||
```
|
||||
|
||||
### 3. Analyze and Synthesize
|
||||
|
||||
Organize findings into coherent insights:
|
||||
- Identify key concepts and how they relate
|
||||
- Note patterns, trade-offs, and considerations
|
||||
- Highlight what's most relevant to the user's needs
|
||||
- Flag uncertainties or areas needing more investigation
|
||||
|
||||
### 4. Produce the Report
|
||||
|
||||
Create a structured research report:
|
||||
|
||||
```markdown
|
||||
---
|
||||
title: "Research: [Topic]"
|
||||
type: research
|
||||
tags:
|
||||
- research
|
||||
- [topic-tags]
|
||||
---
|
||||
|
||||
# Research: [Topic]
|
||||
|
||||
## Summary
|
||||
|
||||
[2-3 sentence executive summary of findings]
|
||||
|
||||
## Research Question
|
||||
|
||||
[What we set out to understand]
|
||||
|
||||
## Key Findings
|
||||
|
||||
### [Finding 1]
|
||||
[Details, evidence, implications]
|
||||
|
||||
### [Finding 2]
|
||||
[Details, evidence, implications]
|
||||
|
||||
### [Finding 3]
|
||||
[Details, evidence, implications]
|
||||
|
||||
## Analysis
|
||||
|
||||
[Synthesis of findings - patterns, trade-offs, recommendations]
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [Things that need more investigation]
|
||||
- [Uncertainties or assumptions]
|
||||
|
||||
## Sources
|
||||
|
||||
- [Where information came from]
|
||||
- [[Related Note]] - relevant prior knowledge
|
||||
|
||||
## Observations
|
||||
|
||||
- [finding] Key insight discovered #research
|
||||
- [pattern] Pattern identified during research
|
||||
- [recommendation] Suggested approach based on findings
|
||||
|
||||
## Relations
|
||||
|
||||
- researches [[Topic]]
|
||||
- informs [[Decision or Implementation]]
|
||||
- relates-to [[Related Concepts]]
|
||||
```
|
||||
|
||||
### 5. Save to Basic Memory
|
||||
|
||||
```python
|
||||
mcp__basic-memory__write_note(
|
||||
title="Research: [Topic]",
|
||||
content="[Full report content]",
|
||||
folder="research",
|
||||
tags=["research", "topic-tags"],
|
||||
project="main"
|
||||
)
|
||||
```
|
||||
|
||||
## Report Styles
|
||||
|
||||
Adjust based on the research type:
|
||||
|
||||
### Quick Investigation
|
||||
- Focused summary
|
||||
- 2-3 key findings
|
||||
- Direct recommendation
|
||||
- Saved to `research/` folder
|
||||
|
||||
### Deep Dive
|
||||
- Comprehensive analysis
|
||||
- Multiple sections
|
||||
- Detailed evidence
|
||||
- Comparison of options
|
||||
- Saved to `research/` folder
|
||||
|
||||
### Decision Support
|
||||
- Options evaluated
|
||||
- Pros/cons for each
|
||||
- Clear recommendation with rationale
|
||||
- Saved to `decisions/` or `research/` folder
|
||||
|
||||
### Technical Exploration
|
||||
- How it works
|
||||
- Architecture/design
|
||||
- Code examples
|
||||
- Integration considerations
|
||||
- Saved to `research/` folder
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Start with what we know** - Check Basic Memory for existing context
|
||||
2. **Be thorough but focused** - Cover the topic well without tangents
|
||||
3. **Cite sources** - Link to where information came from
|
||||
4. **Be honest about uncertainty** - Flag what's unclear or needs verification
|
||||
5. **Make it actionable** - Include recommendations when appropriate
|
||||
6. **Link to related knowledge** - Connect to existing notes
|
||||
7. **Save for future reference** - Always save the report to Basic Memory
|
||||
|
||||
## Example Conversations
|
||||
|
||||
**User:** "Research how other projects handle database migrations"
|
||||
|
||||
**Claude:**
|
||||
1. Searches codebase for migration patterns
|
||||
2. Checks Basic Memory for related decisions
|
||||
3. Looks up best practices online
|
||||
4. Produces report comparing approaches
|
||||
5. Saves to `research/Database Migration Approaches.md`
|
||||
6. Presents summary with recommendation
|
||||
|
||||
**User:** "Investigate the MCP protocol"
|
||||
|
||||
**Claude:**
|
||||
1. Fetches MCP documentation
|
||||
2. Searches for examples in codebase
|
||||
3. Checks Basic Memory for prior context
|
||||
4. Produces comprehensive report on MCP
|
||||
5. Saves to `research/MCP Protocol Overview.md`
|
||||
6. Presents key concepts and how to use them
|
||||
|
||||
**User:** "Look into authentication options for the API"
|
||||
|
||||
**Claude:**
|
||||
1. Researches common auth patterns (JWT, OAuth, API keys)
|
||||
2. Checks existing codebase auth implementation
|
||||
3. Evaluates trade-offs for the use case
|
||||
4. Produces decision-support report
|
||||
5. Saves to `research/API Authentication Options.md`
|
||||
6. Recommends approach with rationale
|
||||
@@ -0,0 +1,292 @@
|
||||
---
|
||||
name: spec-driven-development
|
||||
description: Guide implementation based on specs stored in Basic Memory, following the SPEC-1 specification-driven development process
|
||||
---
|
||||
|
||||
# Spec-Driven Development
|
||||
|
||||
This skill guides implementation work based on specifications stored in the Basic Memory "specs" project, following the process defined in SPEC-1.
|
||||
|
||||
## When to Use
|
||||
|
||||
Use this skill when:
|
||||
- Implementing a feature defined by a spec
|
||||
- Creating a new specification before implementation
|
||||
- Reviewing implementation against spec criteria
|
||||
- Need to understand what a spec requires
|
||||
- Updating spec progress as work completes
|
||||
|
||||
## The Spec-Driven Process
|
||||
|
||||
From SPEC-1, the workflow is:
|
||||
|
||||
1. **Create** - Write spec as complete thought in Basic Memory "specs" project
|
||||
2. **Discuss** - Iterate and refine the specification
|
||||
3. **Implement** - Execute implementation directly
|
||||
4. **Validate** - Review implementation against spec criteria
|
||||
5. **Document** - Update spec with learnings and decisions
|
||||
|
||||
## Spec Structure
|
||||
|
||||
Every spec contains:
|
||||
- **Why** - The reasoning and problem being solved
|
||||
- **What** - What is affected or changed
|
||||
- **How** - High-level approach to implementation
|
||||
- **How to Evaluate** - Testing/validation procedure
|
||||
|
||||
### Progress Tracking Format
|
||||
|
||||
Specs use living documentation with checklists:
|
||||
|
||||
```markdown
|
||||
### Feature Area
|
||||
- ✅ Basic functionality implemented
|
||||
- ✅ Props and events defined
|
||||
- [ ] Add sorting controls
|
||||
- [ ] Improve accessibility
|
||||
- [x] Currently implementing responsive design
|
||||
```
|
||||
|
||||
- `✅` - Completed items
|
||||
- `[ ]` - Pending items
|
||||
- `[x]` - In-progress items
|
||||
|
||||
## Working with Specs
|
||||
|
||||
### Reading a Spec
|
||||
|
||||
```python
|
||||
# Get the full spec
|
||||
mcp__basic-memory__read_note(
|
||||
identifier="SPEC-24: Postgres Database Migration",
|
||||
project="specs"
|
||||
)
|
||||
|
||||
# Or search for it
|
||||
mcp__basic-memory__search_notes(
|
||||
query="postgres migration",
|
||||
project="specs"
|
||||
)
|
||||
```
|
||||
|
||||
### Creating a New Spec
|
||||
|
||||
```python
|
||||
# 1. First, find the next spec number
|
||||
mcp__basic-memory__search_notes(
|
||||
query="SPEC-",
|
||||
project="specs"
|
||||
)
|
||||
|
||||
# 2. Create the spec with proper structure
|
||||
mcp__basic-memory__write_note(
|
||||
title="SPEC-30: Your Feature Name",
|
||||
content="""---
|
||||
title: 'SPEC-30: Your Feature Name'
|
||||
type: spec
|
||||
tags:
|
||||
- feature-area
|
||||
- component
|
||||
---
|
||||
|
||||
# SPEC-30: Your Feature Name
|
||||
|
||||
## Why
|
||||
|
||||
[Problem statement and motivation]
|
||||
|
||||
## What
|
||||
|
||||
[What is affected or changed]
|
||||
- Affected areas
|
||||
- Components involved
|
||||
- Scope boundaries
|
||||
|
||||
## How (High Level)
|
||||
|
||||
[Implementation approach]
|
||||
|
||||
### Phase 1: Foundation
|
||||
- [ ] Task 1
|
||||
- [ ] Task 2
|
||||
|
||||
### Phase 2: Core Features
|
||||
- [ ] Task 3
|
||||
- [ ] Task 4
|
||||
|
||||
## How to Evaluate
|
||||
|
||||
### Success Criteria
|
||||
- [ ] Criterion 1
|
||||
- [ ] Criterion 2
|
||||
|
||||
### Testing Procedure
|
||||
1. Step 1
|
||||
2. Step 2
|
||||
|
||||
## Observations
|
||||
|
||||
- [goal] Primary objective #tag
|
||||
- [constraint] Known limitation #tag
|
||||
|
||||
## Relations
|
||||
|
||||
- relates-to [[Related Spec]]
|
||||
- depends-on [[Dependency]]
|
||||
""",
|
||||
folder="", # Root of specs project
|
||||
project="specs"
|
||||
)
|
||||
```
|
||||
|
||||
### Updating Spec Progress
|
||||
|
||||
```python
|
||||
# Mark items complete as you implement
|
||||
mcp__basic-memory__edit_note(
|
||||
identifier="SPEC-24: Postgres Database Migration",
|
||||
operation="find_replace",
|
||||
find_text="- [ ] Create migration scripts",
|
||||
content="- ✅ Create migration scripts",
|
||||
project="specs"
|
||||
)
|
||||
|
||||
# Or add new observations
|
||||
mcp__basic-memory__edit_note(
|
||||
identifier="SPEC-24: Postgres Database Migration",
|
||||
operation="append",
|
||||
content="\n- [learning] Alembic autogenerate works well for model changes #migration",
|
||||
project="specs"
|
||||
)
|
||||
```
|
||||
|
||||
### Reviewing Implementation
|
||||
|
||||
When reviewing against a spec:
|
||||
|
||||
1. **Read the spec's "How to Evaluate" section**
|
||||
2. **Check each success criterion:**
|
||||
- Functional completeness
|
||||
- Test coverage (count test files, check categories)
|
||||
- Code quality (TypeScript, linting, performance)
|
||||
- Architecture compliance
|
||||
- Documentation completeness
|
||||
3. **Be honest** - Don't overstate completeness
|
||||
4. **Document findings** - Update spec with review results
|
||||
5. **Identify gaps** - Clearly note what still needs work
|
||||
|
||||
## Implementation Workflow
|
||||
|
||||
### Starting Implementation
|
||||
|
||||
1. **Read the spec thoroughly**
|
||||
```python
|
||||
mcp__basic-memory__read_note(
|
||||
identifier="SPEC-XX: Feature Name",
|
||||
project="specs"
|
||||
)
|
||||
```
|
||||
|
||||
2. **Understand dependencies**
|
||||
- Check Relations section for dependencies
|
||||
- Read related specs if needed
|
||||
|
||||
3. **Plan your approach**
|
||||
- Break "How" section into concrete tasks
|
||||
- Identify what to implement first
|
||||
|
||||
4. **Mark first item in-progress**
|
||||
```python
|
||||
mcp__basic-memory__edit_note(
|
||||
identifier="SPEC-XX",
|
||||
operation="find_replace",
|
||||
find_text="- [ ] First task",
|
||||
content="- [x] First task",
|
||||
project="specs"
|
||||
)
|
||||
```
|
||||
|
||||
### During Implementation
|
||||
|
||||
1. **Update progress as you complete items**
|
||||
2. **Add observations for decisions made**
|
||||
3. **Note any deviations from the spec**
|
||||
4. **Capture learnings that might help future specs**
|
||||
|
||||
### After Implementation
|
||||
|
||||
1. **Run full evaluation against criteria**
|
||||
2. **Mark all completed items with ✅**
|
||||
3. **Add final observations**
|
||||
4. **Document any follow-up work needed**
|
||||
|
||||
## Spec Naming Convention
|
||||
|
||||
Format: `SPEC-X: Descriptive Title`
|
||||
|
||||
Examples:
|
||||
- `SPEC-24: Postgres Database Migration`
|
||||
- `SPEC-25: Cloud Index Service`
|
||||
- `SPEC-26: Multi-User Security and Permissions`
|
||||
|
||||
## Common Spec Patterns
|
||||
|
||||
### Feature Spec
|
||||
```markdown
|
||||
## Why
|
||||
User need or problem
|
||||
|
||||
## What
|
||||
- New UI components
|
||||
- API endpoints
|
||||
- Database changes
|
||||
|
||||
## How
|
||||
Implementation phases with checkboxes
|
||||
```
|
||||
|
||||
### Architecture Spec
|
||||
```markdown
|
||||
## Why
|
||||
Technical debt or scalability need
|
||||
|
||||
## What
|
||||
- System components affected
|
||||
- Data flow changes
|
||||
- Integration points
|
||||
|
||||
## How
|
||||
Migration strategy with rollback plan
|
||||
```
|
||||
|
||||
### Process Spec
|
||||
```markdown
|
||||
## Why
|
||||
Workflow improvement need
|
||||
|
||||
## What
|
||||
- Process steps changed
|
||||
- Tools involved
|
||||
- Team impact
|
||||
|
||||
## How
|
||||
Rollout plan and adoption strategy
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Spec first, code second** - Write spec before implementation
|
||||
2. **Keep specs living** - Update as understanding evolves
|
||||
3. **Be specific in criteria** - Vague criteria = vague completion
|
||||
4. **Link related specs** - Build the knowledge graph
|
||||
5. **Capture decisions** - Future you will thank you
|
||||
6. **Review honestly** - Incomplete is okay, dishonest isn't
|
||||
7. **Close the loop** - Mark items done as you complete them
|
||||
|
||||
## Using with Slash Commands
|
||||
|
||||
The `/spec` command provides quick access:
|
||||
- `/spec create [name]` - Create new specification
|
||||
- `/spec status` - Show all spec statuses
|
||||
- `/spec show [name]` - Read a specific spec
|
||||
- `/spec review [name]` - Validate implementation
|
||||
@@ -7,51 +7,44 @@ install:
|
||||
@echo ""
|
||||
@echo "💡 Remember to activate the virtual environment by running: source .venv/bin/activate"
|
||||
|
||||
# Run all tests with unified coverage report
|
||||
test: test-unit test-int
|
||||
|
||||
# Run unit tests only (fast, no coverage)
|
||||
test-unit:
|
||||
uv run pytest -p pytest_mock -v --no-cov tests
|
||||
|
||||
# Run integration tests only (fast, no coverage)
|
||||
test-int:
|
||||
uv run pytest -p pytest_mock -v --no-cov test-int
|
||||
|
||||
# ==============================================================================
|
||||
# DATABASE BACKEND TESTING
|
||||
# ==============================================================================
|
||||
# Basic Memory supports dual database backends (SQLite and Postgres).
|
||||
# By default, tests run against SQLite (fast, no dependencies).
|
||||
# Set BASIC_MEMORY_TEST_POSTGRES=1 to run against Postgres (uses testcontainers).
|
||||
# Tests are parametrized to run against both backends automatically.
|
||||
#
|
||||
# Quick Start:
|
||||
# just test # Run all tests against SQLite (default)
|
||||
# just test-sqlite # Run all tests against SQLite
|
||||
# just test-postgres # Run all tests against Postgres (testcontainers)
|
||||
# just test-unit-sqlite # Run unit tests against SQLite
|
||||
# just test-unit-postgres # Run unit tests against Postgres
|
||||
# just test-int-sqlite # Run integration tests against SQLite
|
||||
# just test-int-postgres # Run integration tests against Postgres
|
||||
# just test-sqlite # Run SQLite tests (default, no Docker needed)
|
||||
# just test-postgres # Run Postgres tests (requires Docker)
|
||||
#
|
||||
# CI runs both in parallel for faster feedback.
|
||||
# For Postgres tests, first start the database:
|
||||
# docker-compose -f docker-compose-postgres.yml up -d
|
||||
# ==============================================================================
|
||||
|
||||
# Run all tests against SQLite and Postgres
|
||||
test: test-sqlite test-postgres
|
||||
# Run tests against SQLite only (default backend, skip Postgres/Benchmark tests)
|
||||
# This is the fastest option and doesn't require any Docker setup.
|
||||
# Use this for local development and quick feedback.
|
||||
# Includes Windows-specific tests which will auto-skip on non-Windows platforms.
|
||||
test-sqlite:
|
||||
uv run pytest -p pytest_mock -v --no-cov -m "not postgres and not benchmark" tests test-int
|
||||
|
||||
# Run all tests against SQLite
|
||||
test-sqlite: test-unit-sqlite test-int-sqlite
|
||||
|
||||
# Run all tests against Postgres (uses testcontainers)
|
||||
test-postgres: test-unit-postgres test-int-postgres
|
||||
|
||||
# Run unit tests against SQLite
|
||||
test-unit-sqlite:
|
||||
uv run pytest -p pytest_mock -v --no-cov tests
|
||||
|
||||
# Run unit tests against Postgres
|
||||
test-unit-postgres:
|
||||
BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov tests
|
||||
|
||||
# Run integration tests against SQLite
|
||||
test-int-sqlite:
|
||||
uv run pytest -p pytest_mock -v --no-cov test-int
|
||||
|
||||
# Run integration tests against Postgres
|
||||
# Note: Uses timeout due to FastMCP Client + asyncpg cleanup hang (tests pass, process hangs on exit)
|
||||
# See: https://github.com/jlowin/fastmcp/issues/1311
|
||||
test-int-postgres:
|
||||
timeout --signal=KILL 600 bash -c 'BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov test-int' || test $? -eq 137
|
||||
# Run tests against Postgres only (requires docker-compose-postgres.yml up)
|
||||
# First start Postgres: docker-compose -f docker-compose-postgres.yml up -d
|
||||
# Tests will connect to localhost:5433/basic_memory_test
|
||||
# To reset the database: just postgres-reset
|
||||
test-postgres:
|
||||
uv run pytest -p pytest_mock -v --no-cov -m "postgres and not benchmark" tests test-int
|
||||
|
||||
# Reset Postgres test database (drops and recreates schema)
|
||||
# Useful when Alembic migration state gets out of sync during development
|
||||
@@ -66,7 +59,7 @@ postgres-reset:
|
||||
postgres-migrate:
|
||||
@cd src/basic_memory/alembic && \
|
||||
BASIC_MEMORY_DATABASE_BACKEND=postgres \
|
||||
BASIC_MEMORY_DATABASE_URL=${POSTGRES_TEST_URL:-postgresql+asyncpg://basic_memory_user:dev_password@localhost:5433/basic_memory_test} \
|
||||
BASIC_MEMORY_DATABASE_URL=${POSTGRES_TEST_URL:-postgresql://basic_memory_user:dev_password@localhost:5433/basic_memory_test} \
|
||||
uv run alembic upgrade head
|
||||
@echo "✅ Migrations applied to Postgres test database"
|
||||
|
||||
|
||||
+4
-6
@@ -29,15 +29,14 @@ dependencies = [
|
||||
"alembic>=1.14.1",
|
||||
"pillow>=11.1.0",
|
||||
"pybars3>=0.9.7",
|
||||
"fastmcp==2.12.3", # Pinned - 2.14.x breaks MCP tools visibility (issue #463)
|
||||
"fastmcp>=2.10.2",
|
||||
"pyjwt>=2.10.1",
|
||||
"python-dotenv>=1.1.0",
|
||||
"pytest-aio>=1.9.0",
|
||||
"aiofiles>=24.1.0", # Optional observability (disabled by default via config)
|
||||
"aiofiles>=24.1.0", # Async file I/O
|
||||
"logfire>=0.73.0", # Optional observability (disabled by default via config)
|
||||
"asyncpg>=0.30.0",
|
||||
"nest-asyncio>=1.6.0", # For Alembic migrations with Postgres
|
||||
"pytest-asyncio>=1.2.0",
|
||||
"psycopg==3.3.1",
|
||||
]
|
||||
|
||||
|
||||
@@ -82,8 +81,7 @@ dev = [
|
||||
"pytest-xdist>=3.0.0",
|
||||
"ruff>=0.1.6",
|
||||
"freezegun>=1.5.5",
|
||||
"testcontainers[postgres]>=4.0.0",
|
||||
"psycopg>=3.2.0",
|
||||
|
||||
]
|
||||
|
||||
[tool.hatch.version]
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
"""basic-memory - Local-first knowledge management combining Zettelkasten with knowledge graphs"""
|
||||
|
||||
# Package version - updated by release automation
|
||||
__version__ = "0.16.3"
|
||||
__version__ = "0.16.2"
|
||||
|
||||
# API version for FastAPI - independent of package version
|
||||
__api_version__ = "v0"
|
||||
|
||||
@@ -1,56 +0,0 @@
|
||||
"""Add cascade delete FK from search_index to entity
|
||||
|
||||
Revision ID: a2b3c4d5e6f7
|
||||
Revises: f8a9b2c3d4e5
|
||||
Create Date: 2025-12-02 07:00:00.000000
|
||||
|
||||
"""
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "a2b3c4d5e6f7"
|
||||
down_revision: Union[str, None] = "f8a9b2c3d4e5"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
"""Add FK with CASCADE delete from search_index.entity_id to entity.id.
|
||||
|
||||
This migration is Postgres-only because:
|
||||
- SQLite uses FTS5 virtual tables which don't support foreign keys
|
||||
- The FK enables automatic cleanup of search_index entries when entities are deleted
|
||||
"""
|
||||
connection = op.get_bind()
|
||||
dialect = connection.dialect.name
|
||||
|
||||
if dialect == "postgresql":
|
||||
# First, clean up any orphaned search_index entries where entity no longer exists
|
||||
op.execute("""
|
||||
DELETE FROM search_index
|
||||
WHERE entity_id IS NOT NULL
|
||||
AND entity_id NOT IN (SELECT id FROM entity)
|
||||
""")
|
||||
|
||||
# Add FK with CASCADE - nullable FK allows search_index entries without entity_id
|
||||
op.create_foreign_key(
|
||||
"fk_search_index_entity_id",
|
||||
"search_index",
|
||||
"entity",
|
||||
["entity_id"],
|
||||
["id"],
|
||||
ondelete="CASCADE",
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
"""Remove the FK constraint."""
|
||||
connection = op.get_bind()
|
||||
dialect = connection.dialect.name
|
||||
|
||||
if dialect == "postgresql":
|
||||
op.drop_constraint("fk_search_index_entity_id", "search_index", type_="foreignkey")
|
||||
-199
@@ -1,199 +0,0 @@
|
||||
"""Add project_id to relation/observation and pg_trgm for fuzzy link resolution
|
||||
|
||||
Revision ID: f8a9b2c3d4e5
|
||||
Revises: 314f1ea54dc4
|
||||
Create Date: 2025-12-01 12:00:00.000000
|
||||
|
||||
"""
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "f8a9b2c3d4e5"
|
||||
down_revision: Union[str, None] = "314f1ea54dc4"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
"""Add project_id to relation and observation tables, plus pg_trgm indexes.
|
||||
|
||||
This migration:
|
||||
1. Adds project_id column to relation and observation tables (denormalization)
|
||||
2. Backfills project_id from the associated entity
|
||||
3. Enables pg_trgm extension for trigram-based fuzzy matching (Postgres only)
|
||||
4. Creates GIN indexes on entity title and permalink for fast similarity searches
|
||||
5. Creates partial index on unresolved relations for efficient bulk resolution
|
||||
"""
|
||||
connection = op.get_bind()
|
||||
dialect = connection.dialect.name
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Add project_id to relation table
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
# Step 1: Add project_id column as nullable first
|
||||
op.add_column("relation", sa.Column("project_id", sa.Integer(), nullable=True))
|
||||
|
||||
# Step 2: Backfill project_id from entity.project_id via from_id
|
||||
if dialect == "postgresql":
|
||||
op.execute("""
|
||||
UPDATE relation
|
||||
SET project_id = entity.project_id
|
||||
FROM entity
|
||||
WHERE relation.from_id = entity.id
|
||||
""")
|
||||
else:
|
||||
# SQLite syntax
|
||||
op.execute("""
|
||||
UPDATE relation
|
||||
SET project_id = (
|
||||
SELECT entity.project_id
|
||||
FROM entity
|
||||
WHERE entity.id = relation.from_id
|
||||
)
|
||||
""")
|
||||
|
||||
# Step 3: Make project_id NOT NULL and add foreign key
|
||||
if dialect == "postgresql":
|
||||
op.alter_column("relation", "project_id", nullable=False)
|
||||
op.create_foreign_key(
|
||||
"fk_relation_project_id",
|
||||
"relation",
|
||||
"project",
|
||||
["project_id"],
|
||||
["id"],
|
||||
)
|
||||
else:
|
||||
# SQLite requires batch operations for ALTER COLUMN
|
||||
with op.batch_alter_table("relation") as batch_op:
|
||||
batch_op.alter_column("project_id", nullable=False)
|
||||
batch_op.create_foreign_key(
|
||||
"fk_relation_project_id",
|
||||
"project",
|
||||
["project_id"],
|
||||
["id"],
|
||||
)
|
||||
|
||||
# Step 4: Create index on relation.project_id
|
||||
op.create_index("ix_relation_project_id", "relation", ["project_id"])
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Add project_id to observation table
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
# Step 1: Add project_id column as nullable first
|
||||
op.add_column("observation", sa.Column("project_id", sa.Integer(), nullable=True))
|
||||
|
||||
# Step 2: Backfill project_id from entity.project_id via entity_id
|
||||
if dialect == "postgresql":
|
||||
op.execute("""
|
||||
UPDATE observation
|
||||
SET project_id = entity.project_id
|
||||
FROM entity
|
||||
WHERE observation.entity_id = entity.id
|
||||
""")
|
||||
else:
|
||||
# SQLite syntax
|
||||
op.execute("""
|
||||
UPDATE observation
|
||||
SET project_id = (
|
||||
SELECT entity.project_id
|
||||
FROM entity
|
||||
WHERE entity.id = observation.entity_id
|
||||
)
|
||||
""")
|
||||
|
||||
# Step 3: Make project_id NOT NULL and add foreign key
|
||||
if dialect == "postgresql":
|
||||
op.alter_column("observation", "project_id", nullable=False)
|
||||
op.create_foreign_key(
|
||||
"fk_observation_project_id",
|
||||
"observation",
|
||||
"project",
|
||||
["project_id"],
|
||||
["id"],
|
||||
)
|
||||
else:
|
||||
# SQLite requires batch operations for ALTER COLUMN
|
||||
with op.batch_alter_table("observation") as batch_op:
|
||||
batch_op.alter_column("project_id", nullable=False)
|
||||
batch_op.create_foreign_key(
|
||||
"fk_observation_project_id",
|
||||
"project",
|
||||
["project_id"],
|
||||
["id"],
|
||||
)
|
||||
|
||||
# Step 4: Create index on observation.project_id
|
||||
op.create_index("ix_observation_project_id", "observation", ["project_id"])
|
||||
|
||||
# Postgres-specific: pg_trgm and GIN indexes
|
||||
if dialect == "postgresql":
|
||||
# Enable pg_trgm extension for fuzzy string matching
|
||||
op.execute("CREATE EXTENSION IF NOT EXISTS pg_trgm")
|
||||
|
||||
# Create trigram indexes on entity table for fuzzy matching
|
||||
# GIN indexes with gin_trgm_ops support similarity searches
|
||||
op.execute("""
|
||||
CREATE INDEX IF NOT EXISTS idx_entity_title_trgm
|
||||
ON entity USING gin (title gin_trgm_ops)
|
||||
""")
|
||||
|
||||
op.execute("""
|
||||
CREATE INDEX IF NOT EXISTS idx_entity_permalink_trgm
|
||||
ON entity USING gin (permalink gin_trgm_ops)
|
||||
""")
|
||||
|
||||
# Create partial index on unresolved relations for efficient bulk resolution
|
||||
# This makes "WHERE to_id IS NULL AND project_id = X" queries very fast
|
||||
op.execute("""
|
||||
CREATE INDEX IF NOT EXISTS idx_relation_unresolved
|
||||
ON relation (project_id, to_name)
|
||||
WHERE to_id IS NULL
|
||||
""")
|
||||
|
||||
# Create index on relation.to_name for join performance in bulk resolution
|
||||
op.execute("""
|
||||
CREATE INDEX IF NOT EXISTS idx_relation_to_name
|
||||
ON relation (to_name)
|
||||
""")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
"""Remove project_id from relation/observation and pg_trgm indexes."""
|
||||
connection = op.get_bind()
|
||||
dialect = connection.dialect.name
|
||||
|
||||
if dialect == "postgresql":
|
||||
# Drop Postgres-specific indexes
|
||||
op.execute("DROP INDEX IF EXISTS idx_relation_to_name")
|
||||
op.execute("DROP INDEX IF EXISTS idx_relation_unresolved")
|
||||
op.execute("DROP INDEX IF EXISTS idx_entity_permalink_trgm")
|
||||
op.execute("DROP INDEX IF EXISTS idx_entity_title_trgm")
|
||||
# Note: We don't drop the pg_trgm extension as other code may depend on it
|
||||
|
||||
# Drop project_id from observation
|
||||
op.drop_index("ix_observation_project_id", table_name="observation")
|
||||
op.drop_constraint("fk_observation_project_id", "observation", type_="foreignkey")
|
||||
op.drop_column("observation", "project_id")
|
||||
|
||||
# Drop project_id from relation
|
||||
op.drop_index("ix_relation_project_id", table_name="relation")
|
||||
op.drop_constraint("fk_relation_project_id", "relation", type_="foreignkey")
|
||||
op.drop_column("relation", "project_id")
|
||||
else:
|
||||
# SQLite requires batch operations
|
||||
op.drop_index("ix_observation_project_id", table_name="observation")
|
||||
with op.batch_alter_table("observation") as batch_op:
|
||||
batch_op.drop_constraint("fk_observation_project_id", type_="foreignkey")
|
||||
batch_op.drop_column("project_id")
|
||||
|
||||
op.drop_index("ix_relation_project_id", table_name="relation")
|
||||
with op.batch_alter_table("relation") as batch_op:
|
||||
batch_op.drop_constraint("fk_relation_project_id", type_="foreignkey")
|
||||
batch_op.drop_column("project_id")
|
||||
@@ -30,7 +30,7 @@ from basic_memory.api.v2.routers import (
|
||||
prompt_router as v2_prompt,
|
||||
importer_router as v2_importer,
|
||||
)
|
||||
from basic_memory.config import ConfigManager, init_api_logging
|
||||
from basic_memory.config import ConfigManager
|
||||
from basic_memory.services.initialization import initialize_file_sync, initialize_app
|
||||
|
||||
|
||||
@@ -38,9 +38,6 @@ from basic_memory.services.initialization import initialize_file_sync, initializ
|
||||
async def lifespan(app: FastAPI): # pragma: no cover
|
||||
"""Lifecycle manager for the FastAPI app. Not called in stdio mcp mode"""
|
||||
|
||||
# Initialize logging for API (stdout in cloud mode, file otherwise)
|
||||
init_api_logging()
|
||||
|
||||
app_config = ConfigManager().config
|
||||
logger.info("Starting Basic Memory API")
|
||||
|
||||
@@ -59,7 +56,6 @@ async def lifespan(app: FastAPI): # pragma: no cover
|
||||
app.state.sync_task = asyncio.create_task(initialize_file_sync(app_config))
|
||||
else:
|
||||
logger.info("Sync changes disabled. Skipping file sync service.")
|
||||
app.state.sync_task = None
|
||||
|
||||
# proceed with startup
|
||||
yield
|
||||
@@ -68,10 +64,6 @@ async def lifespan(app: FastAPI): # pragma: no cover
|
||||
if app.state.sync_task:
|
||||
logger.info("Stopping sync...")
|
||||
app.state.sync_task.cancel() # pyright: ignore
|
||||
try:
|
||||
await app.state.sync_task
|
||||
except asyncio.CancelledError:
|
||||
logger.info("Sync task cancelled successfully")
|
||||
|
||||
await db.shutdown_db()
|
||||
|
||||
@@ -108,6 +100,8 @@ app.include_router(v2_project, prefix="/v2")
|
||||
app.include_router(project.project_resource_router)
|
||||
app.include_router(management.router)
|
||||
|
||||
# Auth routes are handled by FastMCP automatically when auth is enabled
|
||||
|
||||
|
||||
@app.exception_handler(Exception)
|
||||
async def exception_handler(request, exc): # pragma: no cover
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Annotated, Union
|
||||
from typing import Annotated
|
||||
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Body, Response
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Body
|
||||
from fastapi.responses import FileResponse, JSONResponse
|
||||
from loguru import logger
|
||||
|
||||
@@ -25,17 +25,6 @@ from datetime import datetime
|
||||
router = APIRouter(prefix="/resource", tags=["resources"])
|
||||
|
||||
|
||||
def _mtime_to_datetime(entity: EntityModel) -> datetime:
|
||||
"""Convert entity mtime (file modification time) to datetime.
|
||||
|
||||
Returns the file's actual modification time, falling back to updated_at
|
||||
if mtime is not available.
|
||||
"""
|
||||
if entity.mtime:
|
||||
return datetime.fromtimestamp(entity.mtime).astimezone()
|
||||
return entity.updated_at
|
||||
|
||||
|
||||
def get_entity_ids(item: SearchIndexRow) -> set[int]:
|
||||
match item.type:
|
||||
case SearchItemType.ENTITY:
|
||||
@@ -50,7 +39,7 @@ def get_entity_ids(item: SearchIndexRow) -> set[int]:
|
||||
raise ValueError(f"Unexpected type: {item.type}")
|
||||
|
||||
|
||||
@router.get("/{identifier:path}", response_model=None)
|
||||
@router.get("/{identifier:path}")
|
||||
async def get_resource_content(
|
||||
config: ProjectConfigDep,
|
||||
link_resolver: LinkResolverDep,
|
||||
@@ -61,7 +50,7 @@ async def get_resource_content(
|
||||
identifier: str,
|
||||
page: int = 1,
|
||||
page_size: int = 10,
|
||||
) -> Union[Response, FileResponse]:
|
||||
) -> FileResponse:
|
||||
"""Get resource content by identifier: name or permalink."""
|
||||
logger.debug(f"Getting content for: {identifier}")
|
||||
|
||||
@@ -92,16 +81,13 @@ async def get_resource_content(
|
||||
# return single response
|
||||
if len(results) == 1:
|
||||
entity = results[0]
|
||||
# Check file exists via file_service (for cloud compatibility)
|
||||
if not await file_service.exists(entity.file_path):
|
||||
file_path = Path(f"{config.home}/{entity.file_path}")
|
||||
if not file_path.exists():
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"File not found: {entity.file_path}",
|
||||
detail=f"File not found: {file_path}",
|
||||
)
|
||||
# Read content via file_service as bytes (works with both local and S3)
|
||||
content = await file_service.read_file_bytes(entity.file_path)
|
||||
content_type = file_service.content_type(entity.file_path)
|
||||
return Response(content=content, media_type=content_type)
|
||||
return FileResponse(path=file_path)
|
||||
|
||||
# for multiple files, initialize a temporary file for writing the results
|
||||
with tempfile.NamedTemporaryFile(delete=False, mode="w", suffix=".md") as tmp_file:
|
||||
@@ -111,7 +97,7 @@ async def get_resource_content(
|
||||
# Read content for each entity
|
||||
content = await file_service.read_entity_content(result)
|
||||
memory_url = normalize_memory_url(result.permalink)
|
||||
modified_date = _mtime_to_datetime(result).isoformat()
|
||||
modified_date = result.updated_at.isoformat()
|
||||
checksum = result.checksum[:8] if result.checksum else ""
|
||||
|
||||
# Prepare the delimited content
|
||||
@@ -185,17 +171,21 @@ async def write_resource(
|
||||
else:
|
||||
content_str = str(content)
|
||||
|
||||
# Cloud compatibility: do not assume a local filesystem path structure.
|
||||
# Delegate directory creation + writes to the configured FileService (local or S3).
|
||||
await file_service.ensure_directory(Path(file_path).parent)
|
||||
checksum = await file_service.write_file(file_path, content_str)
|
||||
# Get full file path
|
||||
full_path = Path(f"{config.home}/{file_path}")
|
||||
|
||||
# Ensure parent directory exists
|
||||
full_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Write content to file
|
||||
checksum = await file_service.write_file(full_path, content_str)
|
||||
|
||||
# Get file info
|
||||
file_metadata = await file_service.get_file_metadata(file_path)
|
||||
file_stats = file_service.file_stats(full_path)
|
||||
|
||||
# Determine file details
|
||||
file_name = Path(file_path).name
|
||||
content_type = file_service.content_type(file_path)
|
||||
content_type = file_service.content_type(full_path)
|
||||
|
||||
entity_type = "canvas" if file_path.endswith(".canvas") else "file"
|
||||
|
||||
@@ -212,7 +202,7 @@ async def write_resource(
|
||||
"content_type": content_type,
|
||||
"file_path": file_path,
|
||||
"checksum": checksum,
|
||||
"updated_at": file_metadata.modified_at,
|
||||
"updated_at": datetime.fromtimestamp(file_stats.st_mtime).astimezone(),
|
||||
},
|
||||
)
|
||||
status_code = 200
|
||||
@@ -224,8 +214,8 @@ async def write_resource(
|
||||
content_type=content_type,
|
||||
file_path=file_path,
|
||||
checksum=checksum,
|
||||
created_at=file_metadata.created_at,
|
||||
updated_at=file_metadata.modified_at,
|
||||
created_at=datetime.fromtimestamp(file_stats.st_ctime).astimezone(),
|
||||
updated_at=datetime.fromtimestamp(file_stats.st_mtime).astimezone(),
|
||||
)
|
||||
entity = await entity_repository.add(entity)
|
||||
status_code = 201
|
||||
@@ -239,9 +229,9 @@ async def write_resource(
|
||||
content={
|
||||
"file_path": file_path,
|
||||
"checksum": checksum,
|
||||
"size": file_metadata.size,
|
||||
"created_at": file_metadata.created_at.timestamp(),
|
||||
"modified_at": file_metadata.modified_at.timestamp(),
|
||||
"size": file_stats.st_size,
|
||||
"created_at": file_stats.st_ctime,
|
||||
"modified_at": file_stats.st_mtime,
|
||||
},
|
||||
)
|
||||
except Exception as e: # pragma: no cover
|
||||
|
||||
@@ -24,26 +24,8 @@ async def to_graph_context(
|
||||
page: Optional[int] = None,
|
||||
page_size: Optional[int] = None,
|
||||
):
|
||||
# First pass: collect all entity IDs needed for relations
|
||||
entity_ids_needed: set[int] = set()
|
||||
for context_item in context_result.results:
|
||||
for item in (
|
||||
[context_item.primary_result] + context_item.observations + context_item.related_results
|
||||
):
|
||||
if item.type == SearchItemType.RELATION:
|
||||
if item.from_id: # pyright: ignore
|
||||
entity_ids_needed.add(item.from_id) # pyright: ignore
|
||||
if item.to_id:
|
||||
entity_ids_needed.add(item.to_id)
|
||||
|
||||
# Batch fetch all entities at once
|
||||
entity_lookup: dict[int, str] = {}
|
||||
if entity_ids_needed:
|
||||
entities = await entity_repository.find_by_ids(list(entity_ids_needed))
|
||||
entity_lookup = {e.id: e.title for e in entities}
|
||||
|
||||
# Helper function to convert items to summaries
|
||||
def to_summary(item: SearchIndexRow | ContextResultRow):
|
||||
async def to_summary(item: SearchIndexRow | ContextResultRow):
|
||||
match item.type:
|
||||
case SearchItemType.ENTITY:
|
||||
return EntitySummary(
|
||||
@@ -66,8 +48,8 @@ async def to_graph_context(
|
||||
created_at=item.created_at,
|
||||
)
|
||||
case SearchItemType.RELATION:
|
||||
from_title = entity_lookup.get(item.from_id) if item.from_id else None # pyright: ignore
|
||||
to_title = entity_lookup.get(item.to_id) if item.to_id else None
|
||||
from_entity = await entity_repository.find_by_id(item.from_id) # pyright: ignore
|
||||
to_entity = await entity_repository.find_by_id(item.to_id) if item.to_id else None
|
||||
return RelationSummary(
|
||||
relation_id=item.id,
|
||||
entity_id=item.entity_id, # pyright: ignore
|
||||
@@ -75,9 +57,9 @@ async def to_graph_context(
|
||||
file_path=item.file_path,
|
||||
permalink=item.permalink, # pyright: ignore
|
||||
relation_type=item.relation_type, # pyright: ignore
|
||||
from_entity=from_title,
|
||||
from_entity=from_entity.title if from_entity else None,
|
||||
from_entity_id=item.from_id, # pyright: ignore
|
||||
to_entity=to_title,
|
||||
to_entity=to_entity.title if to_entity else None,
|
||||
to_entity_id=item.to_id,
|
||||
created_at=item.created_at,
|
||||
)
|
||||
@@ -88,19 +70,23 @@ async def to_graph_context(
|
||||
hierarchical_results = []
|
||||
for context_item in context_result.results:
|
||||
# Process primary result
|
||||
primary_result = to_summary(context_item.primary_result)
|
||||
primary_result = await to_summary(context_item.primary_result)
|
||||
|
||||
# Process observations (always ObservationSummary, validated by context_service)
|
||||
observations = [to_summary(obs) for obs in context_item.observations]
|
||||
# Process observations
|
||||
observations = []
|
||||
for obs in context_item.observations:
|
||||
observations.append(await to_summary(obs))
|
||||
|
||||
# Process related results
|
||||
related = [to_summary(rel) for rel in context_item.related_results]
|
||||
related = []
|
||||
for rel in context_item.related_results:
|
||||
related.append(await to_summary(rel))
|
||||
|
||||
# Add to hierarchical results
|
||||
hierarchical_results.append(
|
||||
ContextResult(
|
||||
primary_result=primary_result,
|
||||
observations=observations, # pyright: ignore[reportArgumentType]
|
||||
observations=observations,
|
||||
related_results=related,
|
||||
)
|
||||
)
|
||||
|
||||
@@ -11,7 +11,8 @@ Key differences from v1:
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Response
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from fastapi.responses import FileResponse
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory.deps import (
|
||||
@@ -29,6 +30,7 @@ from basic_memory.schemas.v2.resource import (
|
||||
ResourceResponse,
|
||||
)
|
||||
from basic_memory.utils import validate_project_path
|
||||
from datetime import datetime
|
||||
|
||||
router = APIRouter(prefix="/resource", tags=["resources-v2"])
|
||||
|
||||
@@ -40,7 +42,7 @@ async def get_resource_content(
|
||||
config: ProjectConfigV2Dep,
|
||||
entity_service: EntityServiceV2Dep,
|
||||
file_service: FileServiceV2Dep,
|
||||
) -> Response:
|
||||
) -> FileResponse:
|
||||
"""Get raw resource content by entity ID.
|
||||
|
||||
Args:
|
||||
@@ -51,7 +53,7 @@ async def get_resource_content(
|
||||
file_service: File service for reading file content
|
||||
|
||||
Returns:
|
||||
Response with entity content
|
||||
FileResponse with entity content
|
||||
|
||||
Raises:
|
||||
HTTPException: 404 if entity or file not found
|
||||
@@ -74,18 +76,14 @@ async def get_resource_content(
|
||||
detail="Entity contains invalid file path",
|
||||
)
|
||||
|
||||
# Check file exists via file_service (for cloud compatibility)
|
||||
if not await file_service.exists(entity.file_path):
|
||||
file_path = Path(f"{config.home}/{entity.file_path}")
|
||||
if not file_path.exists():
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"File not found: {entity.file_path}",
|
||||
detail=f"File not found: {file_path}",
|
||||
)
|
||||
|
||||
# Read content via file_service as bytes (works with both local and S3)
|
||||
content = await file_service.read_file_bytes(entity.file_path)
|
||||
content_type = file_service.content_type(entity.file_path)
|
||||
|
||||
return Response(content=content, media_type=content_type)
|
||||
return FileResponse(path=file_path)
|
||||
|
||||
|
||||
@router.post("", response_model=ResourceResponse)
|
||||
@@ -135,17 +133,21 @@ async def create_resource(
|
||||
f"Use PUT /resource/{existing_entity.id} to update it.",
|
||||
)
|
||||
|
||||
# Cloud compatibility: avoid assuming a local filesystem path.
|
||||
# Delegate directory creation + writes to FileService (local or S3).
|
||||
await file_service.ensure_directory(Path(data.file_path).parent)
|
||||
checksum = await file_service.write_file(data.file_path, data.content)
|
||||
# Get full file path
|
||||
full_path = Path(f"{config.home}/{data.file_path}")
|
||||
|
||||
# Ensure parent directory exists
|
||||
full_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Write content to file
|
||||
checksum = await file_service.write_file(full_path, data.content)
|
||||
|
||||
# Get file info
|
||||
file_metadata = await file_service.get_file_metadata(data.file_path)
|
||||
file_stats = file_service.file_stats(full_path)
|
||||
|
||||
# Determine file details
|
||||
file_name = Path(data.file_path).name
|
||||
content_type = file_service.content_type(data.file_path)
|
||||
content_type = file_service.content_type(full_path)
|
||||
entity_type = "canvas" if data.file_path.endswith(".canvas") else "file"
|
||||
|
||||
# Create a new entity model
|
||||
@@ -155,8 +157,8 @@ async def create_resource(
|
||||
content_type=content_type,
|
||||
file_path=data.file_path,
|
||||
checksum=checksum,
|
||||
created_at=file_metadata.created_at,
|
||||
updated_at=file_metadata.modified_at,
|
||||
created_at=datetime.fromtimestamp(file_stats.st_ctime).astimezone(),
|
||||
updated_at=datetime.fromtimestamp(file_stats.st_mtime).astimezone(),
|
||||
)
|
||||
entity = await entity_repository.add(entity)
|
||||
|
||||
@@ -168,9 +170,9 @@ async def create_resource(
|
||||
entity_id=entity.id,
|
||||
file_path=data.file_path,
|
||||
checksum=checksum,
|
||||
size=file_metadata.size,
|
||||
created_at=file_metadata.created_at.timestamp(),
|
||||
modified_at=file_metadata.modified_at.timestamp(),
|
||||
size=file_stats.st_size,
|
||||
created_at=file_stats.st_ctime,
|
||||
modified_at=file_stats.st_mtime,
|
||||
)
|
||||
except HTTPException:
|
||||
# Re-raise HTTP exceptions without wrapping
|
||||
@@ -230,27 +232,31 @@ async def update_resource(
|
||||
"Path must be relative and stay within project boundaries.",
|
||||
)
|
||||
|
||||
# Get full paths
|
||||
old_full_path = Path(f"{config.home}/{entity.file_path}")
|
||||
new_full_path = Path(f"{config.home}/{target_file_path}")
|
||||
|
||||
# If moving file, handle the move
|
||||
if data.file_path and data.file_path != entity.file_path:
|
||||
# Ensure new parent directory exists (no-op for S3)
|
||||
await file_service.ensure_directory(Path(target_file_path).parent)
|
||||
# Ensure new parent directory exists
|
||||
new_full_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# If old file exists, remove it via file_service (for cloud compatibility)
|
||||
if await file_service.exists(entity.file_path):
|
||||
await file_service.delete_file(entity.file_path)
|
||||
# If old file exists, remove it
|
||||
if old_full_path.exists():
|
||||
old_full_path.unlink()
|
||||
else:
|
||||
# Ensure directory exists for in-place update
|
||||
await file_service.ensure_directory(Path(target_file_path).parent)
|
||||
new_full_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Write content to target file
|
||||
checksum = await file_service.write_file(target_file_path, data.content)
|
||||
checksum = await file_service.write_file(new_full_path, data.content)
|
||||
|
||||
# Get file info
|
||||
file_metadata = await file_service.get_file_metadata(target_file_path)
|
||||
file_stats = file_service.file_stats(new_full_path)
|
||||
|
||||
# Determine file details
|
||||
file_name = Path(target_file_path).name
|
||||
content_type = file_service.content_type(target_file_path)
|
||||
content_type = file_service.content_type(new_full_path)
|
||||
entity_type = "canvas" if target_file_path.endswith(".canvas") else "file"
|
||||
|
||||
# Update entity
|
||||
@@ -262,7 +268,7 @@ async def update_resource(
|
||||
"content_type": content_type,
|
||||
"file_path": target_file_path,
|
||||
"checksum": checksum,
|
||||
"updated_at": file_metadata.modified_at,
|
||||
"updated_at": datetime.fromtimestamp(file_stats.st_mtime).astimezone(),
|
||||
},
|
||||
)
|
||||
|
||||
@@ -274,9 +280,9 @@ async def update_resource(
|
||||
entity_id=entity_id,
|
||||
file_path=target_file_path,
|
||||
checksum=checksum,
|
||||
size=file_metadata.size,
|
||||
created_at=file_metadata.created_at.timestamp(),
|
||||
modified_at=file_metadata.modified_at.timestamp(),
|
||||
size=file_stats.st_size,
|
||||
created_at=file_stats.st_ctime,
|
||||
modified_at=file_stats.st_mtime,
|
||||
)
|
||||
except HTTPException:
|
||||
# Re-raise HTTP exceptions without wrapping
|
||||
|
||||
@@ -2,7 +2,7 @@ from typing import Optional
|
||||
|
||||
import typer
|
||||
|
||||
from basic_memory.config import ConfigManager, init_cli_logging
|
||||
from basic_memory.config import ConfigManager
|
||||
|
||||
|
||||
def version_callback(value: bool) -> None:
|
||||
@@ -31,9 +31,6 @@ def app_callback(
|
||||
) -> None:
|
||||
"""Basic Memory - Local-first personal knowledge management."""
|
||||
|
||||
# Initialize logging for CLI (file only, no stdout)
|
||||
init_cli_logging()
|
||||
|
||||
# Run initialization for every command unless --version was specified
|
||||
if not version and ctx.invoked_subcommand is not None:
|
||||
from basic_memory.services.initialization import ensure_initialization
|
||||
|
||||
@@ -6,7 +6,7 @@ import typer
|
||||
from typing import Optional
|
||||
|
||||
from basic_memory.cli.app import app
|
||||
from basic_memory.config import ConfigManager, init_mcp_logging
|
||||
from basic_memory.config import ConfigManager
|
||||
|
||||
# Import mcp instance
|
||||
from basic_memory.mcp.server import mcp as mcp_server # pragma: no cover
|
||||
@@ -44,8 +44,6 @@ if not config.cloud_mode_enabled:
|
||||
- streamable-http: Recommended for web deployments (default)
|
||||
- sse: Server-Sent Events (for compatibility with existing clients)
|
||||
"""
|
||||
# Initialize logging for MCP (file only, stdout breaks protocol)
|
||||
init_mcp_logging()
|
||||
|
||||
# Validate and set project constraint if specified
|
||||
if project:
|
||||
|
||||
+70
-98
@@ -9,9 +9,10 @@ from typing import Any, Dict, Literal, Optional, List, Tuple
|
||||
from enum import Enum
|
||||
|
||||
from loguru import logger
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
import basic_memory
|
||||
from basic_memory.utils import setup_logging, generate_permalink
|
||||
|
||||
|
||||
@@ -99,32 +100,13 @@ class BasicMemoryConfig(BaseSettings):
|
||||
description="Database connection URL. For Postgres, use postgresql+asyncpg://user:pass@host:port/db. If not set, SQLite will use default path.",
|
||||
)
|
||||
|
||||
# Database connection pool configuration (Postgres only)
|
||||
db_pool_size: int = Field(
|
||||
default=20,
|
||||
description="Number of connections to keep in the pool (Postgres only)",
|
||||
gt=0,
|
||||
)
|
||||
db_pool_overflow: int = Field(
|
||||
default=40,
|
||||
description="Max additional connections beyond pool_size under load (Postgres only)",
|
||||
gt=0,
|
||||
)
|
||||
db_pool_recycle: int = Field(
|
||||
default=180,
|
||||
description="Recycle connections after N seconds to prevent stale connections. Default 180s works well with Neon's ~5 minute scale-to-zero (Postgres only)",
|
||||
gt=0,
|
||||
)
|
||||
|
||||
# Watch service configuration
|
||||
sync_delay: int = Field(
|
||||
default=1000, description="Milliseconds to wait after changes before syncing", gt=0
|
||||
)
|
||||
|
||||
watch_project_reload_interval: int = Field(
|
||||
default=300,
|
||||
description="Seconds between reloading project list in watch service. Higher values reduce CPU usage by minimizing watcher restarts. Default 300s (5 min) balances efficiency with responsiveness to new projects.",
|
||||
gt=0,
|
||||
default=30, description="Seconds between reloading project list in watch service", gt=0
|
||||
)
|
||||
|
||||
# update permalinks on move
|
||||
@@ -215,36 +197,6 @@ class BasicMemoryConfig(BaseSettings):
|
||||
# Fall back to config file value
|
||||
return self.cloud_mode
|
||||
|
||||
@classmethod
|
||||
def for_cloud_tenant(
|
||||
cls,
|
||||
database_url: str,
|
||||
projects: Optional[Dict[str, str]] = None,
|
||||
) -> "BasicMemoryConfig":
|
||||
"""Create config for cloud tenant - no config.json, database is source of truth.
|
||||
|
||||
This factory method creates a BasicMemoryConfig suitable for cloud deployments
|
||||
where:
|
||||
- Database is Postgres (Neon), not SQLite
|
||||
- Projects are discovered from the database, not config file
|
||||
- Path validation is skipped (no local filesystem in cloud)
|
||||
- Initialization sync is skipped (stateless deployment)
|
||||
|
||||
Args:
|
||||
database_url: Postgres connection URL for tenant database
|
||||
projects: Optional project mapping (usually empty, discovered from DB)
|
||||
|
||||
Returns:
|
||||
BasicMemoryConfig configured for cloud mode
|
||||
"""
|
||||
return cls(
|
||||
database_backend=DatabaseBackend.POSTGRES,
|
||||
database_url=database_url,
|
||||
projects=projects or {},
|
||||
cloud_mode=True,
|
||||
skip_initialization_sync=True,
|
||||
)
|
||||
|
||||
model_config = SettingsConfigDict(
|
||||
env_prefix="BASIC_MEMORY_",
|
||||
extra="ignore",
|
||||
@@ -261,10 +213,6 @@ class BasicMemoryConfig(BaseSettings):
|
||||
|
||||
def model_post_init(self, __context: Any) -> None:
|
||||
"""Ensure configuration is valid after initialization."""
|
||||
# Skip project initialization in cloud mode - projects are discovered from DB
|
||||
if self.database_backend == DatabaseBackend.POSTGRES:
|
||||
return
|
||||
|
||||
# Ensure at least one project exists; if none exist then create main
|
||||
if not self.projects: # pragma: no cover
|
||||
self.projects["main"] = str(
|
||||
@@ -307,26 +255,19 @@ class BasicMemoryConfig(BaseSettings):
|
||||
"""Get all configured projects as ProjectConfig objects."""
|
||||
return [ProjectConfig(name=name, home=Path(path)) for name, path in self.projects.items()]
|
||||
|
||||
@model_validator(mode="after")
|
||||
def ensure_project_paths_exists(self) -> "BasicMemoryConfig": # pragma: no cover
|
||||
"""Ensure project paths exist.
|
||||
|
||||
Skips path creation when using Postgres backend (cloud mode) since
|
||||
cloud tenants don't use local filesystem paths.
|
||||
"""
|
||||
# Skip path creation for cloud mode - no local filesystem
|
||||
if self.database_backend == DatabaseBackend.POSTGRES:
|
||||
return self
|
||||
|
||||
for name, path_value in self.projects.items():
|
||||
@field_validator("projects")
|
||||
@classmethod
|
||||
def ensure_project_paths_exists(cls, v: Dict[str, str]) -> Dict[str, str]: # pragma: no cover
|
||||
"""Ensure project path exists."""
|
||||
for name, path_value in v.items():
|
||||
path = Path(path_value)
|
||||
if not path.exists():
|
||||
if not Path(path).exists():
|
||||
try:
|
||||
path.mkdir(parents=True)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to create project path: {e}")
|
||||
raise e
|
||||
return self
|
||||
return v
|
||||
|
||||
@property
|
||||
def data_dir_path(self):
|
||||
@@ -529,38 +470,69 @@ def save_basic_memory_config(file_path: Path, config: BasicMemoryConfig) -> None
|
||||
logger.error(f"Failed to save config: {e}")
|
||||
|
||||
|
||||
# Logging initialization functions for different entry points
|
||||
# setup logging to a single log file in user home directory
|
||||
user_home = Path.home()
|
||||
log_dir = user_home / DATA_DIR_NAME
|
||||
log_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
def init_cli_logging() -> None: # pragma: no cover
|
||||
"""Initialize logging for CLI commands - file only.
|
||||
|
||||
CLI commands should not log to stdout to avoid interfering with
|
||||
command output and shell integration.
|
||||
# Process info for logging
|
||||
def get_process_name(): # pragma: no cover
|
||||
"""
|
||||
log_level = os.getenv("BASIC_MEMORY_LOG_LEVEL", "INFO")
|
||||
setup_logging(log_level=log_level, log_to_file=True)
|
||||
|
||||
|
||||
def init_mcp_logging() -> None: # pragma: no cover
|
||||
"""Initialize logging for MCP server - file only.
|
||||
|
||||
MCP server must not log to stdout as it would corrupt the
|
||||
JSON-RPC protocol communication.
|
||||
get the type of process for logging
|
||||
"""
|
||||
log_level = os.getenv("BASIC_MEMORY_LOG_LEVEL", "INFO")
|
||||
setup_logging(log_level=log_level, log_to_file=True)
|
||||
import sys
|
||||
|
||||
|
||||
def init_api_logging() -> None: # pragma: no cover
|
||||
"""Initialize logging for API server.
|
||||
|
||||
Cloud mode (BASIC_MEMORY_CLOUD_MODE=1): stdout with structured context
|
||||
Local mode: file only
|
||||
"""
|
||||
log_level = os.getenv("BASIC_MEMORY_LOG_LEVEL", "INFO")
|
||||
cloud_mode = os.getenv("BASIC_MEMORY_CLOUD_MODE", "").lower() in ("1", "true")
|
||||
if cloud_mode:
|
||||
setup_logging(log_level=log_level, log_to_stdout=True, structured_context=True)
|
||||
if "sync" in sys.argv:
|
||||
return "sync"
|
||||
elif "mcp" in sys.argv:
|
||||
return "mcp"
|
||||
elif "cli" in sys.argv:
|
||||
return "cli"
|
||||
else:
|
||||
setup_logging(log_level=log_level, log_to_file=True)
|
||||
return "api"
|
||||
|
||||
|
||||
process_name = get_process_name()
|
||||
|
||||
# Global flag to track if logging has been set up
|
||||
_LOGGING_SETUP = False
|
||||
|
||||
|
||||
# Logging
|
||||
|
||||
|
||||
def setup_basic_memory_logging(): # pragma: no cover
|
||||
"""Set up logging for basic-memory, ensuring it only happens once."""
|
||||
global _LOGGING_SETUP
|
||||
if _LOGGING_SETUP:
|
||||
# We can't log before logging is set up
|
||||
# print("Skipping duplicate logging setup")
|
||||
return
|
||||
|
||||
# Check for console logging environment variable - accept more truthy values
|
||||
console_logging_env = os.getenv("BASIC_MEMORY_CONSOLE_LOGGING", "false").lower()
|
||||
console_logging = console_logging_env in ("true", "1", "yes", "on")
|
||||
|
||||
# Check for log level environment variable first, fall back to config
|
||||
log_level = os.getenv("BASIC_MEMORY_LOG_LEVEL")
|
||||
if not log_level:
|
||||
config_manager = ConfigManager()
|
||||
log_level = config_manager.config.log_level
|
||||
|
||||
config_manager = ConfigManager()
|
||||
config = get_project_config()
|
||||
setup_logging(
|
||||
env=config_manager.config.env,
|
||||
home_dir=user_home, # Use user home for logs
|
||||
log_level=log_level,
|
||||
log_file=f"{DATA_DIR_NAME}/basic-memory-{process_name}.log",
|
||||
console=console_logging,
|
||||
)
|
||||
|
||||
logger.info(f"Basic Memory {basic_memory.__version__} (Project: {config.project})")
|
||||
_LOGGING_SETUP = True
|
||||
|
||||
|
||||
# Set up logging
|
||||
setup_basic_memory_logging()
|
||||
|
||||
+4
-20
@@ -190,37 +190,21 @@ def _create_sqlite_engine(db_url: str, db_type: DatabaseType) -> AsyncEngine:
|
||||
return engine
|
||||
|
||||
|
||||
def _create_postgres_engine(db_url: str, config: BasicMemoryConfig) -> AsyncEngine:
|
||||
def _create_postgres_engine(db_url: str) -> AsyncEngine:
|
||||
"""Create Postgres async engine with appropriate configuration.
|
||||
|
||||
Args:
|
||||
db_url: Postgres connection URL (postgresql+asyncpg://...)
|
||||
config: BasicMemoryConfig with pool settings
|
||||
|
||||
Returns:
|
||||
Configured async engine for Postgres
|
||||
"""
|
||||
# Use NullPool connection issues.
|
||||
# Assume connection pooler like PgBouncer handles connection pooling.
|
||||
# Postgres with asyncpg - use standard async connection
|
||||
engine = create_async_engine(
|
||||
db_url,
|
||||
echo=False,
|
||||
poolclass=NullPool, # No pooling - fresh connection per request
|
||||
connect_args={
|
||||
# Disable statement cache to avoid issues with prepared statements on reconnect
|
||||
"statement_cache_size": 0,
|
||||
# Allow 30s for commands (Neon cold start can take 2-5s, sometimes longer)
|
||||
"command_timeout": 30,
|
||||
# Allow 30s for initial connection (Neon wake-up time)
|
||||
"timeout": 30,
|
||||
"server_settings": {
|
||||
"application_name": "basic-memory",
|
||||
# Statement timeout for queries (30s to allow for cold start)
|
||||
"statement_timeout": "30s",
|
||||
},
|
||||
},
|
||||
pool_pre_ping=True, # Verify connections before using them
|
||||
)
|
||||
logger.debug("Created Postgres engine with NullPool (no connection pooling)")
|
||||
|
||||
return engine
|
||||
|
||||
@@ -244,7 +228,7 @@ def _create_engine_and_session(
|
||||
# Delegate to backend-specific engine creation
|
||||
# Check explicit POSTGRES type first, then config setting
|
||||
if db_type == DatabaseType.POSTGRES or config.database_backend == DatabaseBackend.POSTGRES:
|
||||
engine = _create_postgres_engine(db_url, config)
|
||||
engine = _create_postgres_engine(db_url)
|
||||
else:
|
||||
engine = _create_sqlite_engine(db_url, db_type)
|
||||
|
||||
|
||||
@@ -368,10 +368,11 @@ MarkdownProcessorV2Dep = Annotated[MarkdownProcessor, Depends(get_markdown_proce
|
||||
async def get_file_service(
|
||||
project_config: ProjectConfigDep, markdown_processor: MarkdownProcessorDep
|
||||
) -> FileService:
|
||||
file_service = FileService(project_config.home, markdown_processor)
|
||||
logger.debug(
|
||||
f"Created FileService for project: {project_config.name}, base_path: {project_config.home} "
|
||||
f"Creating FileService for project: {project_config.name}, base_path: {project_config.home}"
|
||||
)
|
||||
file_service = FileService(project_config.home, markdown_processor)
|
||||
logger.debug(f"Created FileService for project: {file_service} ")
|
||||
return file_service
|
||||
|
||||
|
||||
@@ -381,10 +382,11 @@ FileServiceDep = Annotated[FileService, Depends(get_file_service)]
|
||||
async def get_file_service_v2(
|
||||
project_config: ProjectConfigV2Dep, markdown_processor: MarkdownProcessorV2Dep
|
||||
) -> FileService:
|
||||
file_service = FileService(project_config.home, markdown_processor)
|
||||
logger.debug(
|
||||
f"Created FileService for project: {project_config.name}, base_path: {project_config.home}"
|
||||
f"Creating FileService for project: {project_config.name}, base_path: {project_config.home}"
|
||||
)
|
||||
file_service = FileService(project_config.home, markdown_processor)
|
||||
logger.debug(f"Created FileService for project: {file_service} ")
|
||||
return file_service
|
||||
|
||||
|
||||
|
||||
@@ -1,8 +1,6 @@
|
||||
"""Utilities for file operations."""
|
||||
|
||||
import hashlib
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
import re
|
||||
from typing import Any, Dict, Union
|
||||
@@ -15,20 +13,6 @@ from loguru import logger
|
||||
from basic_memory.utils import FilePath
|
||||
|
||||
|
||||
@dataclass
|
||||
class FileMetadata:
|
||||
"""File metadata for cloud-compatible file operations.
|
||||
|
||||
This dataclass provides a cloud-agnostic way to represent file metadata,
|
||||
enabling S3FileService to return metadata from head_object responses
|
||||
instead of mock stat_result with zeros.
|
||||
"""
|
||||
|
||||
size: int
|
||||
created_at: datetime
|
||||
modified_at: datetime
|
||||
|
||||
|
||||
class FileError(Exception):
|
||||
"""Base exception for file operations."""
|
||||
|
||||
|
||||
@@ -23,7 +23,6 @@ from basic_memory.markdown.schemas import (
|
||||
)
|
||||
from basic_memory.utils import parse_tags
|
||||
|
||||
|
||||
md = MarkdownIt().use(observation_plugin).use(relation_plugin)
|
||||
|
||||
|
||||
|
||||
@@ -5,7 +5,6 @@ from collections import OrderedDict
|
||||
from frontmatter import Post
|
||||
from loguru import logger
|
||||
|
||||
|
||||
from basic_memory import file_utils
|
||||
from basic_memory.file_utils import dump_frontmatter
|
||||
from basic_memory.markdown.entity_parser import EntityParser
|
||||
|
||||
@@ -30,9 +30,7 @@ def is_observation(token: Token) -> bool:
|
||||
|
||||
# Check for proper observation format: [category] content
|
||||
match = re.match(r"^\[([^\[\]()]+)\]\s+(.+)", content)
|
||||
# Check for standalone hashtags (words starting with #)
|
||||
# This excludes # in HTML attributes like color="#4285F4"
|
||||
has_tags = any(part.startswith("#") for part in content.split())
|
||||
has_tags = "#" in content
|
||||
return bool(match) or has_tags
|
||||
|
||||
|
||||
@@ -162,7 +160,7 @@ def parse_inline_relations(content: str) -> List[Dict[str, Any]]:
|
||||
|
||||
target = content[start + 2 : end].strip()
|
||||
if target:
|
||||
relations.append({"type": "links_to", "target": target, "context": None})
|
||||
relations.append({"type": "links to", "target": target, "context": None})
|
||||
|
||||
start = end + 2
|
||||
|
||||
|
||||
@@ -3,7 +3,6 @@
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional
|
||||
|
||||
|
||||
from frontmatter import Post
|
||||
|
||||
from basic_memory.file_utils import has_frontmatter, remove_frontmatter, parse_frontmatter
|
||||
@@ -13,10 +12,7 @@ from basic_memory.models import Observation as ObservationModel
|
||||
|
||||
|
||||
def entity_model_from_markdown(
|
||||
file_path: Path,
|
||||
markdown: EntityMarkdown,
|
||||
entity: Optional[Entity] = None,
|
||||
project_id: Optional[int] = None,
|
||||
file_path: Path, markdown: EntityMarkdown, entity: Optional[Entity] = None
|
||||
) -> Entity:
|
||||
"""
|
||||
Convert markdown entity to model. Does not include relations.
|
||||
@@ -25,7 +21,6 @@ def entity_model_from_markdown(
|
||||
file_path: Path to the markdown file
|
||||
markdown: Parsed markdown entity
|
||||
entity: Optional existing entity to update
|
||||
project_id: Project ID for new observations (uses entity.project_id if not provided)
|
||||
|
||||
Returns:
|
||||
Entity model populated from markdown
|
||||
@@ -55,13 +50,9 @@ def entity_model_from_markdown(
|
||||
metadata = markdown.frontmatter.metadata or {}
|
||||
model.entity_metadata = {k: str(v) for k, v in metadata.items() if v is not None}
|
||||
|
||||
# Get project_id from entity if not provided
|
||||
obs_project_id = project_id or (model.project_id if hasattr(model, "project_id") else None)
|
||||
|
||||
# Convert observations
|
||||
model.observations = [
|
||||
ObservationModel(
|
||||
project_id=obs_project_id,
|
||||
content=obs.content,
|
||||
category=obs.category,
|
||||
context=obs.context,
|
||||
|
||||
@@ -4,6 +4,7 @@ import basic_memory
|
||||
from basic_memory.models.base import Base
|
||||
from basic_memory.models.knowledge import Entity, Observation, Relation
|
||||
from basic_memory.models.project import Project
|
||||
from basic_memory.models.search import SearchIndex
|
||||
|
||||
__all__ = [
|
||||
"Base",
|
||||
@@ -11,5 +12,6 @@ __all__ = [
|
||||
"Observation",
|
||||
"Relation",
|
||||
"Project",
|
||||
"SearchIndex",
|
||||
"basic_memory",
|
||||
]
|
||||
|
||||
@@ -145,7 +145,6 @@ class Observation(Base):
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
project_id: Mapped[int] = mapped_column(Integer, ForeignKey("project.id"), index=True)
|
||||
entity_id: Mapped[int] = mapped_column(Integer, ForeignKey("entity.id", ondelete="CASCADE"))
|
||||
content: Mapped[str] = mapped_column(Text)
|
||||
category: Mapped[str] = mapped_column(String, nullable=False, default="note")
|
||||
@@ -163,14 +162,9 @@ class Observation(Base):
|
||||
|
||||
We can construct these because observations are always defined in
|
||||
and owned by a single entity.
|
||||
|
||||
Content is truncated to 200 chars to stay under PostgreSQL's
|
||||
btree index limit of 2704 bytes.
|
||||
"""
|
||||
# Truncate content to avoid exceeding PostgreSQL's btree index limit
|
||||
content_for_permalink = self.content[:200] if len(self.content) > 200 else self.content
|
||||
return generate_permalink(
|
||||
f"{self.entity.permalink}/observations/{self.category}/{content_for_permalink}"
|
||||
f"{self.entity.permalink}/observations/{self.category}/{self.content}"
|
||||
)
|
||||
|
||||
def __repr__(self) -> str: # pragma: no cover
|
||||
@@ -192,7 +186,6 @@ class Relation(Base):
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
project_id: Mapped[int] = mapped_column(Integer, ForeignKey("project.id"), index=True)
|
||||
from_id: Mapped[int] = mapped_column(Integer, ForeignKey("entity.id", ondelete="CASCADE"))
|
||||
to_id: Mapped[Optional[int]] = mapped_column(
|
||||
Integer, ForeignKey("entity.id", ondelete="CASCADE"), nullable=True
|
||||
|
||||
@@ -1,52 +1,53 @@
|
||||
"""Search DDL statements for SQLite and Postgres.
|
||||
"""Search models and tables."""
|
||||
|
||||
The search_index table is created via raw DDL, not ORM models, because:
|
||||
- SQLite uses FTS5 virtual tables (cannot be represented as ORM)
|
||||
- Postgres uses composite primary keys and generated tsvector columns
|
||||
- Both backends use raw SQL for all search operations via SearchIndexRow dataclass
|
||||
"""
|
||||
from sqlalchemy import DDL, Column, Integer, String, DateTime, Text
|
||||
from sqlalchemy.dialects.postgresql import JSONB
|
||||
from sqlalchemy.types import JSON
|
||||
|
||||
from sqlalchemy import DDL
|
||||
from basic_memory.models.base import Base
|
||||
|
||||
|
||||
# Define Postgres search_index table with composite primary key and tsvector
|
||||
# This DDL matches the Alembic migration schema (314f1ea54dc4)
|
||||
# Used by tests to create the table without running full migrations
|
||||
# NOTE: Split into separate DDL statements because asyncpg doesn't support
|
||||
# multiple statements in a single execute call.
|
||||
CREATE_POSTGRES_SEARCH_INDEX_TABLE = DDL("""
|
||||
CREATE TABLE IF NOT EXISTS search_index (
|
||||
id INTEGER NOT NULL,
|
||||
project_id INTEGER NOT NULL,
|
||||
title TEXT,
|
||||
content_stems TEXT,
|
||||
content_snippet TEXT,
|
||||
permalink VARCHAR,
|
||||
file_path VARCHAR,
|
||||
type VARCHAR,
|
||||
from_id INTEGER,
|
||||
to_id INTEGER,
|
||||
relation_type VARCHAR,
|
||||
entity_id INTEGER,
|
||||
category VARCHAR,
|
||||
metadata JSONB,
|
||||
created_at TIMESTAMP WITH TIME ZONE,
|
||||
updated_at TIMESTAMP WITH TIME ZONE,
|
||||
textsearchable_index_col tsvector GENERATED ALWAYS AS (
|
||||
to_tsvector('english', coalesce(title, '') || ' ' || coalesce(content_stems, ''))
|
||||
) STORED,
|
||||
PRIMARY KEY (id, type, project_id),
|
||||
FOREIGN KEY (project_id) REFERENCES project(id) ON DELETE CASCADE
|
||||
)
|
||||
""")
|
||||
class SearchIndex(Base):
|
||||
"""Search index table for Postgres only.
|
||||
|
||||
CREATE_POSTGRES_SEARCH_INDEX_FTS = DDL("""
|
||||
CREATE INDEX IF NOT EXISTS idx_search_index_fts ON search_index USING gin(textsearchable_index_col)
|
||||
""")
|
||||
For SQLite: This model is skipped; FTS5 virtual table is created via DDL instead.
|
||||
For Postgres: This is the actual table structure with tsvector support.
|
||||
"""
|
||||
|
||||
__tablename__ = "search_index"
|
||||
|
||||
# Primary key (rowid in SQLite FTS5, explicit id in Postgres)
|
||||
id = Column(Integer, primary_key=True, autoincrement=True)
|
||||
|
||||
# Core searchable fields
|
||||
title = Column(Text, nullable=True)
|
||||
content_stems = Column(Text, nullable=True)
|
||||
content_snippet = Column(Text, nullable=True)
|
||||
permalink = Column(String(255), nullable=True, index=True)
|
||||
file_path = Column(Text, nullable=True)
|
||||
type = Column(String(50), nullable=True)
|
||||
|
||||
# Project context
|
||||
project_id = Column(Integer, nullable=True, index=True)
|
||||
|
||||
# Relation fields
|
||||
from_id = Column(Integer, nullable=True)
|
||||
to_id = Column(Integer, nullable=True)
|
||||
relation_type = Column(String(100), nullable=True)
|
||||
|
||||
# Observation fields
|
||||
entity_id = Column(Integer, nullable=True)
|
||||
category = Column(String(100), nullable=True)
|
||||
|
||||
# Common fields
|
||||
# Use JSONB for Postgres, JSON for SQLite
|
||||
# Note: 'metadata' is a reserved name in SQLAlchemy, so we use 'metadata_' and map to 'metadata'
|
||||
metadata_ = Column("metadata", JSON().with_variant(JSONB(), "postgresql"), nullable=True)
|
||||
created_at = Column(DateTime(timezone=True), nullable=True)
|
||||
updated_at = Column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
# Note: textsearchable_index_col (tsvector) will be added by migration for Postgres only
|
||||
|
||||
CREATE_POSTGRES_SEARCH_INDEX_METADATA = DDL("""
|
||||
CREATE INDEX IF NOT EXISTS idx_search_index_metadata_gin ON search_index USING gin(metadata jsonb_path_ops)
|
||||
""")
|
||||
|
||||
# Define FTS5 virtual table creation for SQLite only
|
||||
# This DDL is executed separately for SQLite databases
|
||||
|
||||
@@ -3,7 +3,6 @@
|
||||
from pathlib import Path
|
||||
from typing import List, Optional, Sequence, Union, Any
|
||||
|
||||
|
||||
from loguru import logger
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
@@ -77,99 +76,6 @@ class EntityRepository(Repository[Entity]):
|
||||
)
|
||||
return await self.find_one(query)
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Lightweight methods for permalink resolution (no eager loading)
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
async def permalink_exists(self, permalink: str) -> bool:
|
||||
"""Check if a permalink exists without loading the full entity.
|
||||
|
||||
This is much faster than get_by_permalink() as it skips eager loading
|
||||
of observations and relations. Use for existence checks in bulk operations.
|
||||
|
||||
Args:
|
||||
permalink: Permalink to check
|
||||
|
||||
Returns:
|
||||
True if permalink exists, False otherwise
|
||||
"""
|
||||
query = select(Entity.id).where(Entity.permalink == permalink).limit(1)
|
||||
query = self._add_project_filter(query)
|
||||
result = await self.execute_query(query, use_query_options=False)
|
||||
return result.scalar_one_or_none() is not None
|
||||
|
||||
async def get_file_path_for_permalink(self, permalink: str) -> Optional[str]:
|
||||
"""Get the file_path for a permalink without loading the full entity.
|
||||
|
||||
Use when you only need the file_path, not the full entity with relations.
|
||||
|
||||
Args:
|
||||
permalink: Permalink to look up
|
||||
|
||||
Returns:
|
||||
file_path string if found, None otherwise
|
||||
"""
|
||||
query = select(Entity.file_path).where(Entity.permalink == permalink)
|
||||
query = self._add_project_filter(query)
|
||||
result = await self.execute_query(query, use_query_options=False)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
async def get_permalink_for_file_path(self, file_path: Union[Path, str]) -> Optional[str]:
|
||||
"""Get the permalink for a file_path without loading the full entity.
|
||||
|
||||
Use when you only need the permalink, not the full entity with relations.
|
||||
|
||||
Args:
|
||||
file_path: File path to look up
|
||||
|
||||
Returns:
|
||||
permalink string if found, None otherwise
|
||||
"""
|
||||
query = select(Entity.permalink).where(Entity.file_path == Path(file_path).as_posix())
|
||||
query = self._add_project_filter(query)
|
||||
result = await self.execute_query(query, use_query_options=False)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
async def get_all_permalinks(self) -> List[str]:
|
||||
"""Get all permalinks for this project.
|
||||
|
||||
Optimized for bulk operations - returns only permalink strings
|
||||
without loading entities or relationships.
|
||||
|
||||
Returns:
|
||||
List of all permalinks in the project
|
||||
"""
|
||||
query = select(Entity.permalink)
|
||||
query = self._add_project_filter(query)
|
||||
result = await self.execute_query(query, use_query_options=False)
|
||||
return list(result.scalars().all())
|
||||
|
||||
async def get_permalink_to_file_path_map(self) -> dict[str, str]:
|
||||
"""Get a mapping of permalink -> file_path for all entities.
|
||||
|
||||
Optimized for bulk permalink resolution - loads minimal data in one query.
|
||||
|
||||
Returns:
|
||||
Dict mapping permalink to file_path
|
||||
"""
|
||||
query = select(Entity.permalink, Entity.file_path)
|
||||
query = self._add_project_filter(query)
|
||||
result = await self.execute_query(query, use_query_options=False)
|
||||
return {row.permalink: row.file_path for row in result.all()}
|
||||
|
||||
async def get_file_path_to_permalink_map(self) -> dict[str, str]:
|
||||
"""Get a mapping of file_path -> permalink for all entities.
|
||||
|
||||
Optimized for bulk permalink resolution - loads minimal data in one query.
|
||||
|
||||
Returns:
|
||||
Dict mapping file_path to permalink
|
||||
"""
|
||||
query = select(Entity.file_path, Entity.permalink)
|
||||
query = self._add_project_filter(query)
|
||||
result = await self.execute_query(query, use_query_options=False)
|
||||
return {row.file_path: row.permalink for row in result.all()}
|
||||
|
||||
async def get_by_file_paths(
|
||||
self, session: AsyncSession, file_paths: Sequence[Union[Path, str]]
|
||||
) -> List[Row[Any]]:
|
||||
|
||||
@@ -2,7 +2,6 @@
|
||||
|
||||
from typing import Dict, List, Sequence
|
||||
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker
|
||||
|
||||
|
||||
@@ -5,7 +5,6 @@ import re
|
||||
from datetime import datetime
|
||||
from typing import List, Optional
|
||||
|
||||
|
||||
from loguru import logger
|
||||
from sqlalchemy import text
|
||||
|
||||
@@ -258,7 +257,7 @@ class PostgresSearchRepository(SearchRepositoryBase):
|
||||
{score_expr} as score
|
||||
FROM search_index
|
||||
WHERE {where_clause}
|
||||
ORDER BY score DESC, id ASC {order_by_clause}
|
||||
ORDER BY score DESC {order_by_clause}
|
||||
LIMIT :limit
|
||||
OFFSET :offset
|
||||
"""
|
||||
|
||||
@@ -3,7 +3,6 @@
|
||||
from pathlib import Path
|
||||
from typing import Optional, Sequence, Union
|
||||
|
||||
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
|
||||
@@ -1,11 +1,9 @@
|
||||
"""Repository for managing Relation objects."""
|
||||
|
||||
from sqlalchemy import and_, delete
|
||||
from typing import Sequence, List, Optional
|
||||
|
||||
|
||||
from sqlalchemy import and_, delete, select
|
||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker
|
||||
from sqlalchemy.orm import selectinload, aliased
|
||||
from sqlalchemy.orm.interfaces import LoaderOption
|
||||
@@ -88,59 +86,5 @@ class RelationRepository(Repository[Relation]):
|
||||
result = await self.execute_query(query)
|
||||
return result.scalars().all()
|
||||
|
||||
async def add_all_ignore_duplicates(self, relations: List[Relation]) -> int:
|
||||
"""Bulk insert relations, ignoring duplicates.
|
||||
|
||||
Uses ON CONFLICT DO NOTHING to skip relations that would violate the
|
||||
unique constraint on (from_id, to_name, relation_type). This is useful
|
||||
for bulk operations where the same link may appear multiple times in
|
||||
a document.
|
||||
|
||||
Works with both SQLite and PostgreSQL dialects.
|
||||
|
||||
Args:
|
||||
relations: List of Relation objects to insert
|
||||
|
||||
Returns:
|
||||
Number of relations actually inserted (excludes duplicates)
|
||||
"""
|
||||
if not relations:
|
||||
return 0
|
||||
|
||||
# Convert Relation objects to dicts for insert
|
||||
values = [
|
||||
{
|
||||
"project_id": r.project_id if r.project_id else self.project_id,
|
||||
"from_id": r.from_id,
|
||||
"to_id": r.to_id,
|
||||
"to_name": r.to_name,
|
||||
"relation_type": r.relation_type,
|
||||
"context": r.context,
|
||||
}
|
||||
for r in relations
|
||||
]
|
||||
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
# Check dialect to use appropriate insert
|
||||
dialect_name = session.bind.dialect.name if session.bind else "sqlite"
|
||||
|
||||
if dialect_name == "postgresql":
|
||||
# PostgreSQL: use RETURNING to count inserted rows
|
||||
# (rowcount is 0 for ON CONFLICT DO NOTHING)
|
||||
stmt = (
|
||||
pg_insert(Relation)
|
||||
.values(values)
|
||||
.on_conflict_do_nothing()
|
||||
.returning(Relation.id)
|
||||
)
|
||||
result = await session.execute(stmt)
|
||||
return len(result.fetchall())
|
||||
else:
|
||||
# SQLite: rowcount works correctly
|
||||
stmt = sqlite_insert(Relation).values(values)
|
||||
stmt = stmt.on_conflict_do_nothing()
|
||||
result = await session.execute(stmt)
|
||||
return result.rowcount if result.rowcount > 0 else 0
|
||||
|
||||
def get_load_options(self) -> List[LoaderOption]:
|
||||
return [selectinload(Relation.from_entity), selectinload(Relation.to_entity)]
|
||||
|
||||
@@ -2,7 +2,6 @@
|
||||
|
||||
from typing import Type, Optional, Any, Sequence, TypeVar, List, Dict
|
||||
|
||||
|
||||
from loguru import logger
|
||||
from sqlalchemy import (
|
||||
select,
|
||||
|
||||
@@ -4,7 +4,6 @@ from abc import ABC, abstractmethod
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
|
||||
from loguru import logger
|
||||
from sqlalchemy import Executable, Result, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
@@ -5,7 +5,6 @@ import re
|
||||
from datetime import datetime
|
||||
from typing import List, Optional
|
||||
|
||||
|
||||
from loguru import logger
|
||||
from sqlalchemy import text
|
||||
|
||||
|
||||
@@ -4,7 +4,6 @@ from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from typing import List, Optional, Tuple
|
||||
|
||||
|
||||
from loguru import logger
|
||||
from sqlalchemy import text
|
||||
|
||||
|
||||
@@ -3,10 +3,8 @@
|
||||
import fnmatch
|
||||
import logging
|
||||
import os
|
||||
from datetime import datetime
|
||||
from typing import Dict, List, Optional, Sequence
|
||||
|
||||
|
||||
from basic_memory.models import Entity
|
||||
from basic_memory.repository import EntityRepository
|
||||
from basic_memory.schemas.directory import DirectoryNode
|
||||
@@ -14,17 +12,6 @@ from basic_memory.schemas.directory import DirectoryNode
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _mtime_to_datetime(entity: Entity) -> datetime:
|
||||
"""Convert entity mtime (file modification time) to datetime.
|
||||
|
||||
Returns the file's actual modification time, falling back to updated_at
|
||||
if mtime is not available.
|
||||
"""
|
||||
if entity.mtime:
|
||||
return datetime.fromtimestamp(entity.mtime).astimezone()
|
||||
return entity.updated_at
|
||||
|
||||
|
||||
class DirectoryService:
|
||||
"""Service for working with directory trees."""
|
||||
|
||||
@@ -90,7 +77,7 @@ class DirectoryService:
|
||||
entity_id=file.id,
|
||||
entity_type=file.entity_type,
|
||||
content_type=file.content_type,
|
||||
updated_at=_mtime_to_datetime(file),
|
||||
updated_at=file.updated_at,
|
||||
)
|
||||
|
||||
# Add to parent directory's children
|
||||
@@ -254,7 +241,7 @@ class DirectoryService:
|
||||
entity_id=file.id,
|
||||
entity_type=file.entity_type,
|
||||
content_type=file.content_type,
|
||||
updated_at=_mtime_to_datetime(file),
|
||||
updated_at=file.updated_at,
|
||||
)
|
||||
|
||||
# Add to parent directory's children
|
||||
|
||||
@@ -8,7 +8,6 @@ import yaml
|
||||
from loguru import logger
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
|
||||
|
||||
from basic_memory.config import ProjectConfig, BasicMemoryConfig
|
||||
from basic_memory.file_utils import (
|
||||
has_frontmatter,
|
||||
@@ -107,9 +106,6 @@ class EntityService(BaseService[EntityModel]):
|
||||
4. Generate new unique permalink from file path
|
||||
|
||||
Enhanced to detect and handle character-related conflicts.
|
||||
|
||||
Note: Uses lightweight repository methods that skip eager loading of
|
||||
observations and relations for better performance during bulk operations.
|
||||
"""
|
||||
file_path_str = Path(file_path).as_posix()
|
||||
|
||||
@@ -126,20 +122,16 @@ class EntityService(BaseService[EntityModel]):
|
||||
# If markdown has explicit permalink, try to validate it
|
||||
if markdown and markdown.frontmatter.permalink:
|
||||
desired_permalink = markdown.frontmatter.permalink
|
||||
# Use lightweight method - we only need to check file_path
|
||||
existing_file_path = await self.repository.get_file_path_for_permalink(
|
||||
desired_permalink
|
||||
)
|
||||
existing = await self.repository.get_by_permalink(desired_permalink)
|
||||
|
||||
# If no conflict or it's our own file, use as is
|
||||
if not existing_file_path or existing_file_path == file_path_str:
|
||||
if not existing or existing.file_path == file_path_str:
|
||||
return desired_permalink
|
||||
|
||||
# For existing files, try to find current permalink
|
||||
# Use lightweight method - we only need the permalink
|
||||
existing_permalink = await self.repository.get_permalink_for_file_path(file_path_str)
|
||||
if existing_permalink:
|
||||
return existing_permalink
|
||||
existing = await self.repository.get_by_file_path(file_path_str)
|
||||
if existing:
|
||||
return existing.permalink
|
||||
|
||||
# New file - generate permalink
|
||||
if markdown and markdown.frontmatter.permalink:
|
||||
@@ -148,10 +140,9 @@ class EntityService(BaseService[EntityModel]):
|
||||
desired_permalink = generate_permalink(file_path_str)
|
||||
|
||||
# Make unique if needed - enhanced to handle character conflicts
|
||||
# Use lightweight existence check instead of loading full entity
|
||||
permalink = desired_permalink
|
||||
suffix = 1
|
||||
while await self.repository.permalink_exists(permalink):
|
||||
while await self.repository.get_by_permalink(permalink):
|
||||
permalink = f"{desired_permalink}-{suffix}"
|
||||
suffix += 1
|
||||
logger.debug(f"creating unique permalink: {permalink}")
|
||||
@@ -233,11 +224,8 @@ class EntityService(BaseService[EntityModel]):
|
||||
final_content = dump_frontmatter(post)
|
||||
checksum = await self.file_service.write_file(file_path, final_content)
|
||||
|
||||
# parse entity from content we just wrote (avoids re-reading file for cloud compatibility)
|
||||
entity_markdown = await self.entity_parser.parse_markdown_content(
|
||||
file_path=file_path,
|
||||
content=final_content,
|
||||
)
|
||||
# parse entity from file
|
||||
entity_markdown = await self.entity_parser.parse_file(file_path)
|
||||
|
||||
# create entity
|
||||
created = await self.create_entity_from_markdown(file_path, entity_markdown)
|
||||
@@ -257,12 +245,8 @@ class EntityService(BaseService[EntityModel]):
|
||||
# Convert file path string to Path
|
||||
file_path = Path(entity.file_path)
|
||||
|
||||
# Read existing content via file_service (for cloud compatibility)
|
||||
existing_content = await self.file_service.read_file_content(file_path)
|
||||
existing_markdown = await self.entity_parser.parse_markdown_content(
|
||||
file_path=file_path,
|
||||
content=existing_content,
|
||||
)
|
||||
# Read existing frontmatter from the file if it exists
|
||||
existing_markdown = await self.entity_parser.parse_file(file_path)
|
||||
|
||||
# Parse content frontmatter to check for user-specified permalink and entity_type
|
||||
content_markdown = None
|
||||
@@ -318,11 +302,8 @@ class EntityService(BaseService[EntityModel]):
|
||||
final_content = dump_frontmatter(merged_post)
|
||||
checksum = await self.file_service.write_file(file_path, final_content)
|
||||
|
||||
# parse entity from content we just wrote (avoids re-reading file for cloud compatibility)
|
||||
entity_markdown = await self.entity_parser.parse_markdown_content(
|
||||
file_path=file_path,
|
||||
content=final_content,
|
||||
)
|
||||
# parse entity from file
|
||||
entity_markdown = await self.entity_parser.parse_file(file_path)
|
||||
|
||||
# update entity in db
|
||||
entity = await self.update_entity_and_observations(file_path, entity_markdown)
|
||||
@@ -397,9 +378,7 @@ class EntityService(BaseService[EntityModel]):
|
||||
Uses UPSERT approach to handle permalink/file_path conflicts cleanly.
|
||||
"""
|
||||
logger.debug(f"Creating entity: {markdown.frontmatter.title} file_path: {file_path}")
|
||||
model = entity_model_from_markdown(
|
||||
file_path, markdown, project_id=self.repository.project_id
|
||||
)
|
||||
model = entity_model_from_markdown(file_path, markdown)
|
||||
|
||||
# Mark as incomplete because we still need to add relations
|
||||
model.checksum = None
|
||||
@@ -429,7 +408,6 @@ class EntityService(BaseService[EntityModel]):
|
||||
# add new observations
|
||||
observations = [
|
||||
Observation(
|
||||
project_id=self.observation_repository.project_id,
|
||||
entity_id=db_entity.id,
|
||||
content=obs.content,
|
||||
category=obs.category,
|
||||
@@ -496,7 +474,6 @@ class EntityService(BaseService[EntityModel]):
|
||||
|
||||
# Create the relation
|
||||
relation = Relation(
|
||||
project_id=self.relation_repository.project_id,
|
||||
from_id=db_entity.id,
|
||||
to_id=target_id,
|
||||
to_name=target_name,
|
||||
@@ -569,11 +546,8 @@ class EntityService(BaseService[EntityModel]):
|
||||
# Write the updated content back to the file
|
||||
checksum = await self.file_service.write_file(file_path, new_content)
|
||||
|
||||
# Parse the content we just wrote (avoids re-reading file for cloud compatibility)
|
||||
entity_markdown = await self.entity_parser.parse_markdown_content(
|
||||
file_path=file_path,
|
||||
content=new_content,
|
||||
)
|
||||
# Parse the updated file to get new observations/relations
|
||||
entity_markdown = await self.entity_parser.parse_file(file_path)
|
||||
|
||||
# Update entity and its relationships
|
||||
entity = await self.update_entity_and_observations(file_path, entity_markdown)
|
||||
@@ -792,20 +766,23 @@ class EntityService(BaseService[EntityModel]):
|
||||
raise ValueError(f"Invalid destination path: {destination_path}")
|
||||
|
||||
# 3. Validate paths
|
||||
# NOTE: In tenantless/cloud mode, we cannot rely on local filesystem paths.
|
||||
# Use FileService for existence checks and moving.
|
||||
if not await self.file_service.exists(current_path):
|
||||
source_file = project_config.home / current_path
|
||||
destination_file = project_config.home / destination_path
|
||||
|
||||
# Validate source exists
|
||||
if not source_file.exists():
|
||||
raise ValueError(f"Source file not found: {current_path}")
|
||||
|
||||
if await self.file_service.exists(destination_path):
|
||||
# Check if destination already exists
|
||||
if destination_file.exists():
|
||||
raise ValueError(f"Destination already exists: {destination_path}")
|
||||
|
||||
try:
|
||||
# 4. Ensure destination directory if needed (no-op for S3)
|
||||
await self.file_service.ensure_directory(Path(destination_path).parent)
|
||||
# 4. Create destination directory if needed
|
||||
destination_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 5. Move physical file via FileService (filesystem rename or cloud move)
|
||||
await self.file_service.move_file(current_path, destination_path)
|
||||
# 5. Move physical file
|
||||
source_file.rename(destination_file)
|
||||
logger.info(f"Moved file: {current_path} -> {destination_path}")
|
||||
|
||||
# 6. Prepare database updates
|
||||
@@ -844,14 +821,12 @@ class EntityService(BaseService[EntityModel]):
|
||||
|
||||
except Exception as e:
|
||||
# Rollback: try to restore original file location if move succeeded
|
||||
try:
|
||||
if await self.file_service.exists(
|
||||
destination_path
|
||||
) and not await self.file_service.exists(current_path):
|
||||
await self.file_service.move_file(destination_path, current_path)
|
||||
if destination_file.exists() and not source_file.exists():
|
||||
try:
|
||||
destination_file.rename(source_file)
|
||||
logger.info(f"Rolled back file move: {destination_path} -> {current_path}")
|
||||
except Exception as rollback_error: # pragma: no cover
|
||||
logger.error(f"Failed to rollback file move: {rollback_error}")
|
||||
except Exception as rollback_error: # pragma: no cover
|
||||
logger.error(f"Failed to rollback file move: {rollback_error}")
|
||||
|
||||
# Re-raise the original error with context
|
||||
raise ValueError(f"Move failed: {str(e)}") from e
|
||||
|
||||
@@ -3,16 +3,15 @@
|
||||
import asyncio
|
||||
import hashlib
|
||||
import mimetypes
|
||||
from datetime import datetime
|
||||
from os import stat_result
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Tuple, Union
|
||||
|
||||
import aiofiles
|
||||
|
||||
import yaml
|
||||
|
||||
from basic_memory import file_utils
|
||||
from basic_memory.file_utils import FileError, FileMetadata, ParseError
|
||||
from basic_memory.file_utils import FileError, ParseError
|
||||
from basic_memory.markdown.markdown_processor import MarkdownProcessor
|
||||
from basic_memory.models import Entity as EntityModel
|
||||
from basic_memory.schemas import Entity as EntitySchema
|
||||
@@ -221,41 +220,6 @@ class FileService:
|
||||
logger.exception("File read error", path=str(full_path), error=str(e))
|
||||
raise FileOperationError(f"Failed to read file: {e}")
|
||||
|
||||
async def read_file_bytes(self, path: FilePath) -> bytes:
|
||||
"""Read file content as bytes using true async I/O with aiofiles.
|
||||
|
||||
This method reads files in binary mode, suitable for non-text files
|
||||
like images, PDFs, etc. For cloud compatibility with S3FileService.
|
||||
|
||||
Args:
|
||||
path: Path to read (Path or string)
|
||||
|
||||
Returns:
|
||||
File content as bytes
|
||||
|
||||
Raises:
|
||||
FileOperationError: If read fails
|
||||
"""
|
||||
# Convert string to Path if needed
|
||||
path_obj = self.base_path / path if isinstance(path, str) else path
|
||||
full_path = path_obj if path_obj.is_absolute() else self.base_path / path_obj
|
||||
|
||||
try:
|
||||
logger.debug("Reading file bytes", operation="read_file_bytes", path=str(full_path))
|
||||
async with aiofiles.open(full_path, mode="rb") as f:
|
||||
content = await f.read()
|
||||
|
||||
logger.debug(
|
||||
"File read completed",
|
||||
path=str(full_path),
|
||||
content_length=len(content),
|
||||
)
|
||||
return content
|
||||
|
||||
except Exception as e:
|
||||
logger.exception("File read error", path=str(full_path), error=str(e))
|
||||
raise FileOperationError(f"Failed to read file: {e}")
|
||||
|
||||
async def read_file(self, path: FilePath) -> Tuple[str, str]:
|
||||
"""Read file and compute checksum using true async I/O.
|
||||
|
||||
@@ -312,43 +276,6 @@ class FileService:
|
||||
full_path = path_obj if path_obj.is_absolute() else self.base_path / path_obj
|
||||
full_path.unlink(missing_ok=True)
|
||||
|
||||
async def move_file(self, source: FilePath, destination: FilePath) -> None:
|
||||
"""Move/rename a file from source to destination.
|
||||
|
||||
This method abstracts the underlying storage (filesystem vs cloud).
|
||||
Default implementation uses atomic filesystem rename, but cloud-backed
|
||||
implementations (e.g., S3) can override to copy+delete.
|
||||
|
||||
Args:
|
||||
source: Source path (relative to base_path or absolute)
|
||||
destination: Destination path (relative to base_path or absolute)
|
||||
|
||||
Raises:
|
||||
FileOperationError: If the move fails
|
||||
"""
|
||||
# Convert strings to Paths and resolve relative paths against base_path
|
||||
src_obj = self.base_path / source if isinstance(source, str) else source
|
||||
dst_obj = self.base_path / destination if isinstance(destination, str) else destination
|
||||
src_full = src_obj if src_obj.is_absolute() else self.base_path / src_obj
|
||||
dst_full = dst_obj if dst_obj.is_absolute() else self.base_path / dst_obj
|
||||
|
||||
try:
|
||||
# Ensure destination directory exists
|
||||
await self.ensure_directory(dst_full.parent)
|
||||
|
||||
# Use semaphore for concurrency control and run blocking rename in executor
|
||||
async with self._file_semaphore:
|
||||
loop = asyncio.get_event_loop()
|
||||
await loop.run_in_executor(None, lambda: src_full.rename(dst_full))
|
||||
except Exception as e:
|
||||
logger.exception(
|
||||
"File move error",
|
||||
source=str(src_full),
|
||||
destination=str(dst_full),
|
||||
error=str(e),
|
||||
)
|
||||
raise FileOperationError(f"Failed to move file {source} -> {destination}: {e}")
|
||||
|
||||
async def update_frontmatter(self, path: FilePath, updates: Dict[str, Any]) -> str:
|
||||
"""Update frontmatter fields in a file while preserving all content.
|
||||
|
||||
@@ -454,31 +381,20 @@ class FileService:
|
||||
logger.error("Failed to compute checksum", path=str(full_path), error=str(e))
|
||||
raise FileError(f"Failed to compute checksum for {path}: {e}")
|
||||
|
||||
async def get_file_metadata(self, path: FilePath) -> FileMetadata:
|
||||
"""Return file metadata for a given path.
|
||||
|
||||
This method is async to support cloud implementations (S3FileService)
|
||||
where file metadata requires async operations (head_object).
|
||||
def file_stats(self, path: FilePath) -> stat_result:
|
||||
"""Return file stats for a given path.
|
||||
|
||||
Args:
|
||||
path: Path to the file (Path or string)
|
||||
|
||||
Returns:
|
||||
FileMetadata with size, created_at, and modified_at
|
||||
File statistics
|
||||
"""
|
||||
# Convert string to Path if needed
|
||||
path_obj = self.base_path / path if isinstance(path, str) else path
|
||||
full_path = path_obj if path_obj.is_absolute() else self.base_path / path_obj
|
||||
|
||||
# Run blocking stat() in thread pool to maintain async compatibility
|
||||
loop = asyncio.get_event_loop()
|
||||
stat_result = await loop.run_in_executor(None, full_path.stat)
|
||||
|
||||
return FileMetadata(
|
||||
size=stat_result.st_size,
|
||||
created_at=datetime.fromtimestamp(stat_result.st_ctime).astimezone(),
|
||||
modified_at=datetime.fromtimestamp(stat_result.st_mtime).astimezone(),
|
||||
)
|
||||
# get file timestamps
|
||||
return full_path.stat()
|
||||
|
||||
def content_type(self, path: FilePath) -> str:
|
||||
"""Return content_type for a given path.
|
||||
|
||||
@@ -5,10 +5,8 @@ to ensure consistent application startup across all entry points.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory import db
|
||||
@@ -106,12 +104,6 @@ async def initialize_file_sync(
|
||||
# Get active projects
|
||||
active_projects = await project_repository.get_active_projects()
|
||||
|
||||
# Filter to constrained project if MCP server was started with --project
|
||||
constrained_project = os.environ.get("BASIC_MEMORY_MCP_PROJECT")
|
||||
if constrained_project:
|
||||
active_projects = [p for p in active_projects if p.name == constrained_project]
|
||||
logger.info(f"Background sync constrained to project: {constrained_project}")
|
||||
|
||||
# Start sync for all projects as background tasks (non-blocking)
|
||||
async def sync_project_background(project: Project):
|
||||
"""Sync a single project in the background."""
|
||||
|
||||
@@ -2,7 +2,6 @@
|
||||
|
||||
from typing import Optional, Tuple
|
||||
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory.models import Entity
|
||||
|
||||
@@ -8,7 +8,6 @@ from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Dict, Optional, Sequence
|
||||
|
||||
|
||||
from loguru import logger
|
||||
from sqlalchemy import text
|
||||
|
||||
@@ -24,6 +23,9 @@ from basic_memory.config import WATCH_STATUS_JSON, ConfigManager, get_project_co
|
||||
from basic_memory.utils import generate_permalink
|
||||
|
||||
|
||||
config = ConfigManager().config
|
||||
|
||||
|
||||
class ProjectService:
|
||||
"""Service for managing Basic Memory projects."""
|
||||
|
||||
@@ -141,7 +143,6 @@ class ProjectService:
|
||||
"""
|
||||
# If project_root is set, constrain all projects to that directory
|
||||
project_root = self.config_manager.config.project_root
|
||||
sanitized_name = None
|
||||
if project_root:
|
||||
base_path = Path(project_root)
|
||||
|
||||
@@ -198,15 +199,14 @@ class ProjectService:
|
||||
f"Projects cannot share directory trees."
|
||||
)
|
||||
|
||||
if not self.config_manager.config.cloud_mode:
|
||||
# First add to config file (this will validate the project doesn't exist)
|
||||
self.config_manager.add_project(name, resolved_path)
|
||||
# First add to config file (this will validate the project doesn't exist)
|
||||
project_config = self.config_manager.add_project(name, resolved_path)
|
||||
|
||||
# Then add to database
|
||||
project_data = {
|
||||
"name": name,
|
||||
"path": resolved_path,
|
||||
"permalink": sanitized_name,
|
||||
"permalink": generate_permalink(project_config.name),
|
||||
"is_active": True,
|
||||
# Don't set is_default=False to avoid UNIQUE constraint issues
|
||||
# Let it default to NULL, only set to True when explicitly making default
|
||||
|
||||
@@ -4,7 +4,6 @@ import ast
|
||||
from datetime import datetime
|
||||
from typing import List, Optional, Set
|
||||
|
||||
|
||||
from dateparser import parse
|
||||
from fastapi import BackgroundTasks
|
||||
from loguru import logger
|
||||
@@ -16,21 +15,6 @@ from basic_memory.repository.search_repository import SearchRepository, SearchIn
|
||||
from basic_memory.schemas.search import SearchQuery, SearchItemType
|
||||
from basic_memory.services import FileService
|
||||
|
||||
# Maximum size for content_stems field to stay under Postgres's 8KB index row limit.
|
||||
# We use 6000 characters to leave headroom for other indexed columns and overhead.
|
||||
MAX_CONTENT_STEMS_SIZE = 6000
|
||||
|
||||
|
||||
def _mtime_to_datetime(entity: Entity) -> datetime:
|
||||
"""Convert entity mtime (file modification time) to datetime.
|
||||
|
||||
Returns the file's actual modification time, falling back to updated_at
|
||||
if mtime is not available.
|
||||
"""
|
||||
if entity.mtime:
|
||||
return datetime.fromtimestamp(entity.mtime).astimezone()
|
||||
return entity.updated_at
|
||||
|
||||
|
||||
class SearchService:
|
||||
"""Service for search operations.
|
||||
@@ -209,7 +193,7 @@ class SearchService:
|
||||
"entity_type": entity.entity_type,
|
||||
},
|
||||
created_at=entity.created_at,
|
||||
updated_at=_mtime_to_datetime(entity),
|
||||
updated_at=entity.updated_at,
|
||||
project_id=entity.project_id,
|
||||
)
|
||||
)
|
||||
@@ -272,10 +256,6 @@ class SearchService:
|
||||
|
||||
entity_content_stems = "\n".join(p for p in content_stems if p and p.strip())
|
||||
|
||||
# Truncate to stay under Postgres's 8KB index row limit
|
||||
if len(entity_content_stems) > MAX_CONTENT_STEMS_SIZE:
|
||||
entity_content_stems = entity_content_stems[:MAX_CONTENT_STEMS_SIZE]
|
||||
|
||||
# Add entity row
|
||||
rows_to_index.append(
|
||||
SearchIndexRow(
|
||||
@@ -291,28 +271,17 @@ class SearchService:
|
||||
"entity_type": entity.entity_type,
|
||||
},
|
||||
created_at=entity.created_at,
|
||||
updated_at=_mtime_to_datetime(entity),
|
||||
updated_at=entity.updated_at,
|
||||
project_id=entity.project_id,
|
||||
)
|
||||
)
|
||||
|
||||
# Add observation rows - dedupe by permalink to avoid unique constraint violations
|
||||
# Two observations with same entity/category/content generate identical permalinks
|
||||
seen_permalinks: set[str] = {entity.permalink} if entity.permalink else set()
|
||||
# Add observation rows
|
||||
for obs in entity.observations:
|
||||
obs_permalink = obs.permalink
|
||||
if obs_permalink in seen_permalinks:
|
||||
logger.debug(f"Skipping duplicate observation permalink: {obs_permalink}")
|
||||
continue
|
||||
seen_permalinks.add(obs_permalink)
|
||||
|
||||
# Index with parent entity's file path since that's where it's defined
|
||||
obs_content_stems = "\n".join(
|
||||
p for p in self._generate_variants(obs.content) if p and p.strip()
|
||||
)
|
||||
# Truncate to stay under Postgres's 8KB index row limit
|
||||
if len(obs_content_stems) > MAX_CONTENT_STEMS_SIZE:
|
||||
obs_content_stems = obs_content_stems[:MAX_CONTENT_STEMS_SIZE]
|
||||
rows_to_index.append(
|
||||
SearchIndexRow(
|
||||
id=obs.id,
|
||||
@@ -320,7 +289,7 @@ class SearchService:
|
||||
title=f"{obs.category}: {obs.content[:100]}...",
|
||||
content_stems=obs_content_stems,
|
||||
content_snippet=obs.content,
|
||||
permalink=obs_permalink,
|
||||
permalink=obs.permalink,
|
||||
file_path=entity.file_path,
|
||||
category=obs.category,
|
||||
entity_id=entity.id,
|
||||
@@ -328,7 +297,7 @@ class SearchService:
|
||||
"tags": obs.tags,
|
||||
},
|
||||
created_at=entity.created_at,
|
||||
updated_at=_mtime_to_datetime(entity),
|
||||
updated_at=entity.updated_at,
|
||||
project_id=entity.project_id,
|
||||
)
|
||||
)
|
||||
@@ -358,7 +327,7 @@ class SearchService:
|
||||
to_id=rel.to_id,
|
||||
relation_type=rel.relation_type,
|
||||
created_at=entity.created_at,
|
||||
updated_at=_mtime_to_datetime(entity),
|
||||
updated_at=entity.updated_at,
|
||||
project_id=entity.project_id,
|
||||
)
|
||||
)
|
||||
|
||||
@@ -10,7 +10,7 @@ from pathlib import Path
|
||||
from typing import AsyncIterator, Dict, List, Optional, Set, Tuple
|
||||
|
||||
import aiofiles.os
|
||||
|
||||
import logfire
|
||||
from loguru import logger
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
|
||||
@@ -215,12 +215,17 @@ class SyncService:
|
||||
f"path={path}, error={error}"
|
||||
)
|
||||
|
||||
# Record metric for file failure
|
||||
logfire.metric_counter("sync.circuit_breaker.failures").add(1)
|
||||
|
||||
# Log when threshold is reached
|
||||
if failure_info.count >= MAX_CONSECUTIVE_FAILURES:
|
||||
logger.error(
|
||||
f"File {path} has failed {MAX_CONSECUTIVE_FAILURES} times and will be skipped. "
|
||||
f"First failure: {failure_info.first_failure}, Last error: {error}"
|
||||
)
|
||||
# Record metric for file being blocked by circuit breaker
|
||||
logfire.metric_counter("sync.circuit_breaker.blocked_files").add(1)
|
||||
else:
|
||||
# Create new failure record
|
||||
self._file_failures[path] = FileFailureInfo(
|
||||
@@ -250,6 +255,7 @@ class SyncService:
|
||||
logger.info(f"Clearing failure history for {path} after successful sync")
|
||||
del self._file_failures[path]
|
||||
|
||||
@logfire.instrument()
|
||||
async def sync(
|
||||
self, directory: Path, project_name: Optional[str] = None, force_full: bool = False
|
||||
) -> SyncReport:
|
||||
@@ -276,58 +282,63 @@ class SyncService:
|
||||
)
|
||||
|
||||
# sync moves first
|
||||
for old_path, new_path in report.moves.items():
|
||||
# in the case where a file has been deleted and replaced by another file
|
||||
# it will show up in the move and modified lists, so handle it in modified
|
||||
if new_path in report.modified:
|
||||
report.modified.remove(new_path)
|
||||
logger.debug(
|
||||
f"File marked as moved and modified: old_path={old_path}, new_path={new_path}"
|
||||
)
|
||||
else:
|
||||
await self.handle_move(old_path, new_path)
|
||||
with logfire.span("process_moves", move_count=len(report.moves)):
|
||||
for old_path, new_path in report.moves.items():
|
||||
# in the case where a file has been deleted and replaced by another file
|
||||
# it will show up in the move and modified lists, so handle it in modified
|
||||
if new_path in report.modified:
|
||||
report.modified.remove(new_path)
|
||||
logger.debug(
|
||||
f"File marked as moved and modified: old_path={old_path}, new_path={new_path}"
|
||||
)
|
||||
else:
|
||||
await self.handle_move(old_path, new_path)
|
||||
|
||||
# deleted next
|
||||
for path in report.deleted:
|
||||
await self.handle_delete(path)
|
||||
with logfire.span("process_deletes", delete_count=len(report.deleted)):
|
||||
for path in report.deleted:
|
||||
await self.handle_delete(path)
|
||||
|
||||
# then new and modified
|
||||
for path in report.new:
|
||||
entity, _ = await self.sync_file(path, new=True)
|
||||
with logfire.span("process_new_files", new_count=len(report.new)):
|
||||
for path in report.new:
|
||||
entity, _ = await self.sync_file(path, new=True)
|
||||
|
||||
# Track if file was skipped
|
||||
if entity is None and await self._should_skip_file(path):
|
||||
failure_info = self._file_failures[path]
|
||||
report.skipped_files.append(
|
||||
SkippedFile(
|
||||
path=path,
|
||||
reason=failure_info.last_error,
|
||||
failure_count=failure_info.count,
|
||||
first_failed=failure_info.first_failure,
|
||||
# Track if file was skipped
|
||||
if entity is None and await self._should_skip_file(path):
|
||||
failure_info = self._file_failures[path]
|
||||
report.skipped_files.append(
|
||||
SkippedFile(
|
||||
path=path,
|
||||
reason=failure_info.last_error,
|
||||
failure_count=failure_info.count,
|
||||
first_failed=failure_info.first_failure,
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
for path in report.modified:
|
||||
entity, _ = await self.sync_file(path, new=False)
|
||||
with logfire.span("process_modified_files", modified_count=len(report.modified)):
|
||||
for path in report.modified:
|
||||
entity, _ = await self.sync_file(path, new=False)
|
||||
|
||||
# Track if file was skipped
|
||||
if entity is None and await self._should_skip_file(path):
|
||||
failure_info = self._file_failures[path]
|
||||
report.skipped_files.append(
|
||||
SkippedFile(
|
||||
path=path,
|
||||
reason=failure_info.last_error,
|
||||
failure_count=failure_info.count,
|
||||
first_failed=failure_info.first_failure,
|
||||
# Track if file was skipped
|
||||
if entity is None and await self._should_skip_file(path):
|
||||
failure_info = self._file_failures[path]
|
||||
report.skipped_files.append(
|
||||
SkippedFile(
|
||||
path=path,
|
||||
reason=failure_info.last_error,
|
||||
failure_count=failure_info.count,
|
||||
first_failed=failure_info.first_failure,
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
# Only resolve relations if there were actual changes
|
||||
# If no files changed, no new unresolved relations could have been created
|
||||
if report.total > 0:
|
||||
await self.resolve_relations()
|
||||
else:
|
||||
logger.info("Skipping relation resolution - no file changes detected")
|
||||
with logfire.span("resolve_relations"):
|
||||
if report.total > 0:
|
||||
await self.resolve_relations()
|
||||
else:
|
||||
logger.info("Skipping relation resolution - no file changes detected")
|
||||
|
||||
# Update scan watermark after successful sync
|
||||
# Use the timestamp from sync start (not end) to ensure we catch files
|
||||
@@ -350,6 +361,15 @@ class SyncService:
|
||||
|
||||
duration_ms = int((time.time() - start_time) * 1000)
|
||||
|
||||
# Record metrics for sync operation
|
||||
logfire.metric_histogram("sync.duration", unit="ms").record(duration_ms)
|
||||
logfire.metric_counter("sync.files.new").add(len(report.new))
|
||||
logfire.metric_counter("sync.files.modified").add(len(report.modified))
|
||||
logfire.metric_counter("sync.files.deleted").add(len(report.deleted))
|
||||
logfire.metric_counter("sync.files.moved").add(len(report.moves))
|
||||
if report.skipped_files:
|
||||
logfire.metric_counter("sync.files.skipped").add(len(report.skipped_files))
|
||||
|
||||
# Log summary with skipped files if any
|
||||
if report.skipped_files:
|
||||
logger.warning(
|
||||
@@ -370,6 +390,7 @@ class SyncService:
|
||||
|
||||
return report
|
||||
|
||||
@logfire.instrument()
|
||||
async def scan(self, directory, force_full: bool = False):
|
||||
"""Smart scan using watermark and file count for large project optimization.
|
||||
|
||||
@@ -451,6 +472,12 @@ class SyncService:
|
||||
logger.warning("No scan watermark available, falling back to full scan")
|
||||
file_paths_to_scan = await self._scan_directory_full(directory)
|
||||
|
||||
# Record scan type metric
|
||||
logfire.metric_counter(f"sync.scan.{scan_type}").add(1)
|
||||
logfire.metric_histogram("sync.scan.files_scanned", unit="files").record(
|
||||
len(file_paths_to_scan)
|
||||
)
|
||||
|
||||
# Step 3: Process each file with mtime-based comparison
|
||||
scanned_paths: Set[str] = set()
|
||||
changed_checksums: Dict[str, str] = {}
|
||||
@@ -562,6 +589,7 @@ class SyncService:
|
||||
report.checksums = changed_checksums
|
||||
|
||||
scan_duration_ms = int((time.time() - scan_start_time) * 1000)
|
||||
logfire.metric_histogram("sync.scan.duration", unit="ms").record(scan_duration_ms)
|
||||
|
||||
logger.info(
|
||||
f"Completed {scan_type} scan for directory {directory} in {scan_duration_ms}ms, "
|
||||
@@ -571,6 +599,7 @@ class SyncService:
|
||||
)
|
||||
return report
|
||||
|
||||
@logfire.instrument()
|
||||
async def sync_file(
|
||||
self, path: str, new: bool = True
|
||||
) -> Tuple[Optional[Entity], Optional[str]]:
|
||||
@@ -625,6 +654,7 @@ class SyncService:
|
||||
|
||||
return None, None
|
||||
|
||||
@logfire.instrument()
|
||||
async def sync_markdown_file(self, path: str, new: bool = True) -> Tuple[Optional[Entity], str]:
|
||||
"""Sync a markdown file with full processing.
|
||||
|
||||
@@ -642,19 +672,12 @@ class SyncService:
|
||||
file_contains_frontmatter = has_frontmatter(file_content)
|
||||
|
||||
# Get file timestamps for tracking modification times
|
||||
file_metadata = await self.file_service.get_file_metadata(path)
|
||||
created = file_metadata.created_at
|
||||
modified = file_metadata.modified_at
|
||||
file_stats = self.file_service.file_stats(path)
|
||||
created = datetime.fromtimestamp(file_stats.st_ctime).astimezone()
|
||||
modified = datetime.fromtimestamp(file_stats.st_mtime).astimezone()
|
||||
|
||||
# Parse markdown content with file metadata (avoids redundant file read/stat)
|
||||
# This enables cloud implementations (S3FileService) to provide metadata from head_object
|
||||
abs_path = self.file_service.base_path / path
|
||||
entity_markdown = await self.entity_parser.parse_markdown_content(
|
||||
file_path=abs_path,
|
||||
content=file_content,
|
||||
mtime=file_metadata.modified_at.timestamp(),
|
||||
ctime=file_metadata.created_at.timestamp(),
|
||||
)
|
||||
# entity markdown will always contain front matter, so it can be used up create/update the entity
|
||||
entity_markdown = await self.entity_parser.parse_file(path)
|
||||
|
||||
# if the file contains frontmatter, resolve a permalink (unless disabled)
|
||||
if file_contains_frontmatter and not self.app_config.disable_permalinks:
|
||||
@@ -700,8 +723,8 @@ class SyncService:
|
||||
"checksum": final_checksum,
|
||||
"created_at": created,
|
||||
"updated_at": modified,
|
||||
"mtime": file_metadata.modified_at.timestamp(),
|
||||
"size": file_metadata.size,
|
||||
"mtime": file_stats.st_mtime,
|
||||
"size": file_stats.st_size,
|
||||
},
|
||||
)
|
||||
|
||||
@@ -714,6 +737,7 @@ class SyncService:
|
||||
# Return the final checksum to ensure everything is consistent
|
||||
return entity, final_checksum
|
||||
|
||||
@logfire.instrument()
|
||||
async def sync_regular_file(self, path: str, new: bool = True) -> Tuple[Optional[Entity], str]:
|
||||
"""Sync a non-markdown file with basic tracking.
|
||||
|
||||
@@ -730,9 +754,9 @@ class SyncService:
|
||||
await self.entity_service.resolve_permalink(path, skip_conflict_check=True)
|
||||
|
||||
# get file timestamps
|
||||
file_metadata = await self.file_service.get_file_metadata(path)
|
||||
created = file_metadata.created_at
|
||||
modified = file_metadata.modified_at
|
||||
file_stats = self.file_service.file_stats(path)
|
||||
created = datetime.fromtimestamp(file_stats.st_ctime).astimezone()
|
||||
modified = datetime.fromtimestamp(file_stats.st_mtime).astimezone()
|
||||
|
||||
# get mime type
|
||||
content_type = self.file_service.content_type(path)
|
||||
@@ -748,8 +772,8 @@ class SyncService:
|
||||
created_at=created,
|
||||
updated_at=modified,
|
||||
content_type=content_type,
|
||||
mtime=file_metadata.modified_at.timestamp(),
|
||||
size=file_metadata.size,
|
||||
mtime=file_stats.st_mtime,
|
||||
size=file_stats.st_size,
|
||||
)
|
||||
)
|
||||
return entity, checksum
|
||||
@@ -765,15 +789,15 @@ class SyncService:
|
||||
logger.error(f"Entity not found after constraint violation, path={path}")
|
||||
raise ValueError(f"Entity not found after constraint violation: {path}")
|
||||
|
||||
# Re-get file metadata since we're in update path
|
||||
file_metadata_for_update = await self.file_service.get_file_metadata(path)
|
||||
# Re-get file stats since we're in update path
|
||||
file_stats_for_update = self.file_service.file_stats(path)
|
||||
updated = await self.entity_repository.update(
|
||||
entity.id,
|
||||
{
|
||||
"file_path": path,
|
||||
"checksum": checksum,
|
||||
"mtime": file_metadata_for_update.modified_at.timestamp(),
|
||||
"size": file_metadata_for_update.size,
|
||||
"mtime": file_stats_for_update.st_mtime,
|
||||
"size": file_stats_for_update.st_size,
|
||||
},
|
||||
)
|
||||
|
||||
@@ -787,8 +811,8 @@ class SyncService:
|
||||
raise
|
||||
else:
|
||||
# Get file timestamps for updating modification time
|
||||
file_metadata = await self.file_service.get_file_metadata(path)
|
||||
modified = file_metadata.modified_at
|
||||
file_stats = self.file_service.file_stats(path)
|
||||
modified = datetime.fromtimestamp(file_stats.st_mtime).astimezone()
|
||||
|
||||
entity = await self.entity_repository.get_by_file_path(path)
|
||||
if entity is None: # pragma: no cover
|
||||
@@ -803,8 +827,8 @@ class SyncService:
|
||||
"file_path": path,
|
||||
"checksum": checksum,
|
||||
"updated_at": modified,
|
||||
"mtime": file_metadata.modified_at.timestamp(),
|
||||
"size": file_metadata.size,
|
||||
"mtime": file_stats.st_mtime,
|
||||
"size": file_stats.st_size,
|
||||
},
|
||||
)
|
||||
|
||||
@@ -814,6 +838,7 @@ class SyncService:
|
||||
|
||||
return updated, checksum
|
||||
|
||||
@logfire.instrument()
|
||||
async def handle_delete(self, file_path: str):
|
||||
"""Handle complete entity deletion including search index cleanup."""
|
||||
|
||||
@@ -845,6 +870,7 @@ class SyncService:
|
||||
else:
|
||||
await self.search_service.delete_by_entity_id(entity.id)
|
||||
|
||||
@logfire.instrument()
|
||||
async def handle_move(self, old_path, new_path):
|
||||
logger.debug("Moving entity", old_path=old_path, new_path=new_path)
|
||||
|
||||
@@ -949,6 +975,7 @@ class SyncService:
|
||||
# update search index
|
||||
await self.search_service.index_entity(updated)
|
||||
|
||||
@logfire.instrument()
|
||||
async def resolve_relations(self, entity_id: int | None = None):
|
||||
"""Try to resolve unresolved relations.
|
||||
|
||||
@@ -999,27 +1026,16 @@ class SyncService:
|
||||
"to_name": resolved_entity.title,
|
||||
},
|
||||
)
|
||||
# update search index only on successful resolution
|
||||
await self.search_service.index_entity(resolved_entity)
|
||||
except IntegrityError:
|
||||
# IntegrityError means a relation with this (from_id, to_id, relation_type)
|
||||
# already exists. The UPDATE was rolled back, so our unresolved relation
|
||||
# (to_id=NULL) still exists in the database. We delete it because:
|
||||
# 1. It's redundant - a resolved relation already captures this relationship
|
||||
# 2. If we don't delete it, future syncs will try to resolve it again
|
||||
# and get the same IntegrityError
|
||||
except IntegrityError: # pragma: no cover
|
||||
logger.debug(
|
||||
"Deleting duplicate unresolved relation "
|
||||
"Ignoring duplicate relation "
|
||||
f"relation_id={relation.id} "
|
||||
f"from_id={relation.from_id} "
|
||||
f"to_name={relation.to_name} "
|
||||
f"resolved_to_id={resolved_entity.id}"
|
||||
f"to_name={relation.to_name}"
|
||||
)
|
||||
try:
|
||||
await self.relation_repository.delete(relation.id)
|
||||
except Exception as e:
|
||||
# Log but don't fail - the relation may have been deleted already
|
||||
logger.debug(f"Could not delete duplicate relation {relation.id}: {e}")
|
||||
|
||||
# update search index
|
||||
await self.search_service.index_entity(resolved_entity)
|
||||
|
||||
async def _quick_count_files(self, directory: Path) -> int:
|
||||
"""Fast file count using find command.
|
||||
@@ -1047,6 +1063,8 @@ class SyncService:
|
||||
f"error: {error_msg}. Falling back to manual count. "
|
||||
f"This will slow down watermark detection!"
|
||||
)
|
||||
# Track optimization failures for visibility
|
||||
logfire.metric_counter("sync.scan.file_count_failure").add(1)
|
||||
# Fallback: count using scan_directory
|
||||
count = 0
|
||||
async for _ in self.scan_directory(directory):
|
||||
@@ -1087,6 +1105,8 @@ class SyncService:
|
||||
f"error: {error_msg}. Falling back to full scan. "
|
||||
f"This will cause slow syncs on large projects!"
|
||||
)
|
||||
# Track optimization failures for visibility
|
||||
logfire.metric_counter("sync.scan.optimization_failure").add(1)
|
||||
# Fallback to full scan
|
||||
return await self._scan_directory_full(directory)
|
||||
|
||||
|
||||
@@ -5,10 +5,7 @@ import os
|
||||
from collections import defaultdict
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import List, Optional, Set, Sequence, Callable, Awaitable, TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from basic_memory.sync.sync_service import SyncService
|
||||
from typing import List, Optional, Set, Sequence
|
||||
|
||||
from basic_memory.config import BasicMemoryConfig, WATCH_STATUS_JSON
|
||||
from basic_memory.ignore_utils import load_gitignore_patterns, should_ignore_path
|
||||
@@ -74,17 +71,12 @@ class WatchServiceState(BaseModel):
|
||||
self.last_error = datetime.now()
|
||||
|
||||
|
||||
# Type alias for sync service factory function
|
||||
SyncServiceFactory = Callable[[Project], Awaitable["SyncService"]]
|
||||
|
||||
|
||||
class WatchService:
|
||||
def __init__(
|
||||
self,
|
||||
app_config: BasicMemoryConfig,
|
||||
project_repository: ProjectRepository,
|
||||
quiet: bool = False,
|
||||
sync_service_factory: Optional[SyncServiceFactory] = None,
|
||||
):
|
||||
self.app_config = app_config
|
||||
self.project_repository = project_repository
|
||||
@@ -92,20 +84,10 @@ class WatchService:
|
||||
self.status_path = Path.home() / ".basic-memory" / WATCH_STATUS_JSON
|
||||
self.status_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
self._ignore_patterns_cache: dict[Path, Set[str]] = {}
|
||||
self._sync_service_factory = sync_service_factory
|
||||
|
||||
# quiet mode for mcp so it doesn't mess up stdout
|
||||
self.console = Console(quiet=quiet)
|
||||
|
||||
async def _get_sync_service(self, project: Project) -> "SyncService":
|
||||
"""Get sync service for a project, using factory if provided."""
|
||||
if self._sync_service_factory:
|
||||
return await self._sync_service_factory(project)
|
||||
# Fall back to default factory
|
||||
from basic_memory.sync.sync_service import get_sync_service
|
||||
|
||||
return await get_sync_service(project)
|
||||
|
||||
async def _schedule_restart(self, stop_event: asyncio.Event):
|
||||
"""Schedule a restart of the watch service after the configured interval."""
|
||||
await asyncio.sleep(self.app_config.watch_project_reload_interval)
|
||||
@@ -251,6 +233,9 @@ class WatchService:
|
||||
|
||||
async def handle_changes(self, project: Project, changes: Set[FileChange]) -> None:
|
||||
"""Process a batch of file changes"""
|
||||
# avoid circular imports
|
||||
from basic_memory.sync.sync_service import get_sync_service
|
||||
|
||||
# Check if project still exists in configuration before processing
|
||||
# This prevents deleted projects from being recreated by background sync
|
||||
from basic_memory.config import ConfigManager
|
||||
@@ -265,7 +250,7 @@ class WatchService:
|
||||
)
|
||||
return
|
||||
|
||||
sync_service = await self._get_sync_service(project)
|
||||
sync_service = await get_sync_service(project)
|
||||
file_service = sync_service.file_service
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
+64
-65
@@ -5,9 +5,9 @@ import os
|
||||
import logging
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Protocol, Union, runtime_checkable, List
|
||||
from typing import Optional, Protocol, Union, runtime_checkable, List
|
||||
|
||||
from loguru import logger
|
||||
from unidecode import unidecode
|
||||
@@ -67,6 +67,9 @@ class PathLike(Protocol):
|
||||
# This preserves compatibility with existing code while we migrate
|
||||
FilePath = Union[Path, str]
|
||||
|
||||
# Disable the "Queue is full" warning
|
||||
logging.getLogger("opentelemetry.sdk.metrics._internal.instrument").setLevel(logging.ERROR)
|
||||
|
||||
|
||||
def generate_permalink(file_path: Union[Path, str, PathLike], split_extension: bool = True) -> str:
|
||||
"""Generate a stable permalink from a file path.
|
||||
@@ -203,35 +206,29 @@ def generate_permalink(file_path: Union[Path, str, PathLike], split_extension: b
|
||||
|
||||
|
||||
def setup_logging(
|
||||
env: str,
|
||||
home_dir: Path,
|
||||
log_file: Optional[str] = None,
|
||||
log_level: str = "INFO",
|
||||
log_to_file: bool = False,
|
||||
log_to_stdout: bool = False,
|
||||
structured_context: bool = False,
|
||||
console: bool = True,
|
||||
) -> None: # pragma: no cover
|
||||
"""Configure logging with explicit settings.
|
||||
|
||||
This function provides a simple, explicit interface for configuring logging.
|
||||
Each entry point (CLI, MCP, API) should call this with appropriate settings.
|
||||
"""
|
||||
Configure logging for the application.
|
||||
|
||||
Args:
|
||||
log_level: DEBUG, INFO, WARNING, ERROR
|
||||
log_to_file: Write to ~/.basic-memory/basic-memory.log with rotation
|
||||
log_to_stdout: Write to stderr (for Docker/cloud deployments)
|
||||
structured_context: Bind tenant_id, fly_region, etc. for cloud observability
|
||||
env: The environment name (dev, test, prod)
|
||||
home_dir: The root directory for the application
|
||||
log_file: The name of the log file to write to
|
||||
log_level: The logging level to use
|
||||
console: Whether to log to the console
|
||||
"""
|
||||
# Remove default handler and any existing handlers
|
||||
logger.remove()
|
||||
|
||||
# In test mode, only log to stdout regardless of settings
|
||||
env = os.getenv("BASIC_MEMORY_ENV", "dev")
|
||||
if env == "test":
|
||||
logger.add(sys.stderr, level=log_level, backtrace=True, diagnose=True, colorize=True)
|
||||
return
|
||||
|
||||
# Add file handler with rotation
|
||||
if log_to_file:
|
||||
log_path = Path.home() / ".basic-memory" / "basic-memory.log"
|
||||
log_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
# Add file handler if we are not running tests and a log file is specified
|
||||
if log_file and env != "test":
|
||||
# Setup file logger
|
||||
log_path = home_dir / log_file
|
||||
logger.add(
|
||||
str(log_path),
|
||||
level=log_level,
|
||||
@@ -239,28 +236,42 @@ def setup_logging(
|
||||
retention="10 days",
|
||||
backtrace=True,
|
||||
diagnose=True,
|
||||
enqueue=True, # Thread-safe async logging
|
||||
enqueue=True,
|
||||
colorize=False,
|
||||
)
|
||||
|
||||
# Add stdout handler (for Docker/cloud)
|
||||
if log_to_stdout:
|
||||
# Add console logger if requested or in test mode
|
||||
if env == "test" or console:
|
||||
logger.add(sys.stderr, level=log_level, backtrace=True, diagnose=True, colorize=True)
|
||||
|
||||
# Bind structured context for cloud observability
|
||||
if structured_context:
|
||||
logger.configure(
|
||||
extra={
|
||||
"tenant_id": os.getenv("BASIC_MEMORY_TENANT_ID", "local"),
|
||||
"fly_app_name": os.getenv("FLY_APP_NAME", "local"),
|
||||
"fly_machine_id": os.getenv("FLY_MACHINE_ID", "local"),
|
||||
"fly_region": os.getenv("FLY_REGION", "local"),
|
||||
}
|
||||
)
|
||||
logger.info(f"ENV: '{env}' Log level: '{log_level}' Logging to {log_file}")
|
||||
|
||||
# Bind environment context for structured logging (works in both local and cloud)
|
||||
tenant_id = os.getenv("BASIC_MEMORY_TENANT_ID", "local")
|
||||
fly_app_name = os.getenv("FLY_APP_NAME", "local")
|
||||
fly_machine_id = os.getenv("FLY_MACHINE_ID", "local")
|
||||
fly_region = os.getenv("FLY_REGION", "local")
|
||||
|
||||
logger.configure(
|
||||
extra={
|
||||
"tenant_id": tenant_id,
|
||||
"fly_app_name": fly_app_name,
|
||||
"fly_machine_id": fly_machine_id,
|
||||
"fly_region": fly_region,
|
||||
}
|
||||
)
|
||||
|
||||
# Reduce noise from third-party libraries
|
||||
logging.getLogger("httpx").setLevel(logging.WARNING)
|
||||
logging.getLogger("watchfiles.main").setLevel(logging.WARNING)
|
||||
noisy_loggers = {
|
||||
# HTTP client logs
|
||||
"httpx": logging.WARNING,
|
||||
# File watching logs
|
||||
"watchfiles.main": logging.WARNING,
|
||||
}
|
||||
|
||||
# Set log levels for noisy loggers
|
||||
for logger_name, level in noisy_loggers.items():
|
||||
logging.getLogger(logger_name).setLevel(level)
|
||||
|
||||
|
||||
def parse_tags(tags: Union[List[str], str, None]) -> List[str]:
|
||||
@@ -329,7 +340,7 @@ def normalize_file_path_for_comparison(file_path: str) -> str:
|
||||
This function normalizes file paths to help detect potential conflicts:
|
||||
- Converts to lowercase for case-insensitive comparison
|
||||
- Normalizes Unicode characters
|
||||
- Converts backslashes to forward slashes for cross-platform consistency
|
||||
- Handles path separators consistently
|
||||
|
||||
Args:
|
||||
file_path: The file path to normalize
|
||||
@@ -338,15 +349,19 @@ def normalize_file_path_for_comparison(file_path: str) -> str:
|
||||
Normalized file path for comparison purposes
|
||||
"""
|
||||
import unicodedata
|
||||
from pathlib import PureWindowsPath
|
||||
|
||||
# Use PureWindowsPath to ensure backslashes are treated as separators
|
||||
# regardless of current platform, then convert to POSIX-style
|
||||
normalized = PureWindowsPath(file_path).as_posix().lower()
|
||||
# Convert to lowercase for case-insensitive comparison
|
||||
normalized = file_path.lower()
|
||||
|
||||
# Normalize Unicode characters (NFD normalization)
|
||||
normalized = unicodedata.normalize("NFD", normalized)
|
||||
|
||||
# Replace path separators with forward slashes
|
||||
normalized = normalized.replace("\\", "/")
|
||||
|
||||
# Remove multiple slashes
|
||||
normalized = re.sub(r"/+", "/", normalized)
|
||||
|
||||
return normalized
|
||||
|
||||
|
||||
@@ -430,37 +445,21 @@ def validate_project_path(path: str, project_path: Path) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def ensure_timezone_aware(dt: datetime, cloud_mode: bool | None = None) -> datetime:
|
||||
"""Ensure a datetime is timezone-aware.
|
||||
def ensure_timezone_aware(dt: datetime) -> datetime:
|
||||
"""Ensure a datetime is timezone-aware using system timezone.
|
||||
|
||||
If the datetime is naive, convert it to timezone-aware. The interpretation
|
||||
depends on cloud_mode:
|
||||
- In cloud mode (PostgreSQL/asyncpg): naive datetimes are interpreted as UTC
|
||||
- In local mode (SQLite): naive datetimes are interpreted as local time
|
||||
|
||||
asyncpg uses binary protocol which returns timestamps in UTC but as naive
|
||||
datetimes. In cloud deployments, cloud_mode=True handles this correctly.
|
||||
If the datetime is naive, convert it to timezone-aware using the system's local timezone.
|
||||
If it's already timezone-aware, return it unchanged.
|
||||
|
||||
Args:
|
||||
dt: The datetime to ensure is timezone-aware
|
||||
cloud_mode: Optional explicit cloud_mode setting. If None, loads from config.
|
||||
|
||||
Returns:
|
||||
A timezone-aware datetime
|
||||
"""
|
||||
if dt.tzinfo is None:
|
||||
# Determine cloud_mode: use explicit parameter if provided, otherwise load from config
|
||||
if cloud_mode is None:
|
||||
from basic_memory.config import ConfigManager
|
||||
|
||||
cloud_mode = ConfigManager().config.cloud_mode_enabled
|
||||
|
||||
if cloud_mode:
|
||||
# Cloud/PostgreSQL mode: naive datetimes from asyncpg are already UTC
|
||||
return dt.replace(tzinfo=timezone.utc)
|
||||
else:
|
||||
# Local/SQLite mode: naive datetimes are in local time
|
||||
return dt.astimezone()
|
||||
# Naive datetime - assume it's in local time and add timezone
|
||||
return dt.astimezone()
|
||||
else:
|
||||
# Already timezone-aware
|
||||
return dt
|
||||
|
||||
+49
-78
@@ -50,23 +50,18 @@ The `app` fixture ensures FastAPI dependency overrides are active, and
|
||||
`mcp_server` provides the MCP server with proper project session initialization.
|
||||
"""
|
||||
|
||||
import os
|
||||
from typing import AsyncGenerator, Literal
|
||||
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
from pathlib import Path
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
|
||||
from sqlalchemy.pool import NullPool
|
||||
from testcontainers.postgres import PostgresContainer
|
||||
|
||||
from httpx import AsyncClient, ASGITransport
|
||||
|
||||
from basic_memory.config import BasicMemoryConfig, ProjectConfig, ConfigManager, DatabaseBackend
|
||||
from basic_memory.db import engine_session_factory, DatabaseType
|
||||
from basic_memory.models import Project
|
||||
from basic_memory.models.base import Base
|
||||
from basic_memory.repository.project_repository import ProjectRepository
|
||||
from fastapi import FastAPI
|
||||
|
||||
@@ -77,38 +72,25 @@ from basic_memory.deps import get_project_config, get_engine_factory, get_app_co
|
||||
from basic_memory.mcp import tools # noqa: F401
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Database Backend Selection (env var approach)
|
||||
# =============================================================================
|
||||
# By default, integration tests run against SQLite.
|
||||
# Set BASIC_MEMORY_TEST_POSTGRES=1 to run against Postgres (uses testcontainers).
|
||||
@pytest.fixture(
|
||||
params=[
|
||||
pytest.param("sqlite", id="sqlite"),
|
||||
pytest.param("postgres", id="postgres", marks=pytest.mark.postgres),
|
||||
]
|
||||
)
|
||||
def db_backend(request) -> Literal["sqlite", "postgres"]:
|
||||
"""Parametrize tests to run against both SQLite and Postgres.
|
||||
|
||||
Usage:
|
||||
pytest # Runs tests against SQLite only (default)
|
||||
pytest -m postgres # Runs tests against Postgres only
|
||||
pytest -m "not postgres" # Runs tests against SQLite only
|
||||
pytest --run-all-backends # Runs tests against both backends
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def db_backend() -> Literal["sqlite", "postgres"]:
|
||||
"""Determine database backend from environment variable.
|
||||
|
||||
Default: sqlite
|
||||
Set BASIC_MEMORY_TEST_POSTGRES=1 to use postgres
|
||||
Note: Only tests that use database fixtures (engine_factory, session_maker, etc.)
|
||||
will be parametrized. Tests that don't use the database won't be affected.
|
||||
"""
|
||||
if os.environ.get("BASIC_MEMORY_TEST_POSTGRES", "").lower() in ("1", "true", "yes"):
|
||||
return "postgres"
|
||||
return "sqlite"
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def postgres_container(db_backend):
|
||||
"""Session-scoped Postgres container for integration tests.
|
||||
|
||||
Uses testcontainers to spin up a real Postgres instance.
|
||||
Only starts if db_backend is "postgres".
|
||||
"""
|
||||
if db_backend != "postgres":
|
||||
yield None
|
||||
return
|
||||
|
||||
with PostgresContainer("postgres:16-alpine") as postgres:
|
||||
yield postgres
|
||||
return request.param
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
@@ -116,57 +98,50 @@ async def engine_factory(
|
||||
app_config,
|
||||
config_manager,
|
||||
db_backend: Literal["sqlite", "postgres"],
|
||||
postgres_container,
|
||||
tmp_path,
|
||||
) -> AsyncGenerator[tuple, None]:
|
||||
"""Create engine and session factory for the configured database backend."""
|
||||
from basic_memory.models.search import (
|
||||
CREATE_SEARCH_INDEX,
|
||||
CREATE_POSTGRES_SEARCH_INDEX_TABLE,
|
||||
CREATE_POSTGRES_SEARCH_INDEX_FTS,
|
||||
CREATE_POSTGRES_SEARCH_INDEX_METADATA,
|
||||
)
|
||||
from basic_memory.models.search import CREATE_SEARCH_INDEX
|
||||
from basic_memory import db
|
||||
|
||||
# Determine database type based on backend
|
||||
if db_backend == "postgres":
|
||||
# Postgres mode using testcontainers
|
||||
sync_url = postgres_container.get_connection_url()
|
||||
async_url = sync_url.replace("postgresql+psycopg2", "postgresql+asyncpg")
|
||||
db_type = DatabaseType.FILESYSTEM
|
||||
else:
|
||||
db_type = DatabaseType.FILESYSTEM # Integration tests use file-based SQLite
|
||||
|
||||
engine = create_async_engine(
|
||||
async_url,
|
||||
echo=False,
|
||||
poolclass=NullPool,
|
||||
)
|
||||
# Use tmp_path for SQLite, use config database_path for Postgres
|
||||
if db_backend == "sqlite":
|
||||
db_path = tmp_path / "test.db"
|
||||
else:
|
||||
db_path = app_config.database_path
|
||||
|
||||
session_maker = async_sessionmaker(
|
||||
bind=engine,
|
||||
class_=AsyncSession,
|
||||
expire_on_commit=False,
|
||||
autoflush=False,
|
||||
)
|
||||
if db_backend == "postgres":
|
||||
# Postgres: Create fresh engine for each test with full schema reset
|
||||
config_manager._config = app_config
|
||||
|
||||
# Drop and recreate all tables for test isolation
|
||||
async with engine.begin() as conn:
|
||||
await conn.execute(text("DROP TABLE IF EXISTS search_index CASCADE"))
|
||||
await conn.run_sync(Base.metadata.drop_all)
|
||||
await conn.run_sync(Base.metadata.create_all)
|
||||
# asyncpg requires separate execute calls for each statement
|
||||
await conn.execute(CREATE_POSTGRES_SEARCH_INDEX_TABLE)
|
||||
await conn.execute(CREATE_POSTGRES_SEARCH_INDEX_FTS)
|
||||
await conn.execute(CREATE_POSTGRES_SEARCH_INDEX_METADATA)
|
||||
# Use context manager to handle engine disposal properly
|
||||
async with engine_session_factory(db_path, db_type) as (engine, session_maker):
|
||||
# Drop and recreate schema for complete isolation
|
||||
async with engine.begin() as conn:
|
||||
await conn.execute(text("DROP SCHEMA IF EXISTS public CASCADE"))
|
||||
await conn.execute(text("CREATE SCHEMA public"))
|
||||
await conn.execute(text("GRANT ALL ON SCHEMA public TO basic_memory_user"))
|
||||
await conn.execute(text("GRANT ALL ON SCHEMA public TO public"))
|
||||
|
||||
yield engine, session_maker
|
||||
# Run migrations to create production tables
|
||||
from basic_memory.db import run_migrations
|
||||
|
||||
await engine.dispose()
|
||||
await run_migrations(app_config, db_type)
|
||||
|
||||
yield engine, session_maker
|
||||
|
||||
else:
|
||||
# SQLite: Create fresh database (fast with tmp files)
|
||||
db_path = tmp_path / "test.db"
|
||||
db_type = DatabaseType.FILESYSTEM
|
||||
|
||||
async with engine_session_factory(db_path, db_type) as (engine, session_maker):
|
||||
# Create all tables via ORM
|
||||
from basic_memory.models.base import Base
|
||||
|
||||
async with engine.begin() as conn:
|
||||
await conn.run_sync(Base.metadata.create_all)
|
||||
|
||||
@@ -206,11 +181,7 @@ def config_home(tmp_path, monkeypatch) -> Path:
|
||||
|
||||
@pytest.fixture
|
||||
def app_config(
|
||||
config_home,
|
||||
db_backend: Literal["sqlite", "postgres"],
|
||||
postgres_container,
|
||||
tmp_path,
|
||||
monkeypatch,
|
||||
config_home, db_backend: Literal["sqlite", "postgres"], tmp_path, monkeypatch
|
||||
) -> BasicMemoryConfig:
|
||||
"""Create test app configuration."""
|
||||
# Disable cloud mode for CLI tests
|
||||
@@ -219,12 +190,12 @@ def app_config(
|
||||
# Create a basic config with test-project like unit tests do
|
||||
projects = {"test-project": str(config_home)}
|
||||
|
||||
# Configure database backend based on env var
|
||||
# Configure database backend based on test parameter
|
||||
if db_backend == "postgres":
|
||||
database_backend = DatabaseBackend.POSTGRES
|
||||
# Get URL from testcontainer and convert to asyncpg driver
|
||||
sync_url = postgres_container.get_connection_url()
|
||||
database_url = sync_url.replace("postgresql+psycopg2", "postgresql+asyncpg")
|
||||
database_url = (
|
||||
"postgresql+asyncpg://basic_memory_user:dev_password@localhost:5433/basic_memory_test"
|
||||
)
|
||||
else:
|
||||
database_backend = DatabaseBackend.SQLITE
|
||||
database_url = None
|
||||
|
||||
@@ -1,70 +0,0 @@
|
||||
"""
|
||||
Integration test for FastAPI lifespan shutdown behavior.
|
||||
|
||||
This test verifies the asyncio cancellation pattern used by the API lifespan:
|
||||
when the background sync task is cancelled during shutdown, it must be *awaited*
|
||||
before database shutdown begins. This prevents "hang on exit" scenarios in
|
||||
`asyncio.run(...)` callers (e.g. CLI/MCP clients using httpx ASGITransport).
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
|
||||
|
||||
def test_lifespan_shutdown_awaits_sync_task_cancellation(app, monkeypatch):
|
||||
"""
|
||||
Ensure lifespan shutdown awaits the cancelled background sync task.
|
||||
|
||||
Why this is deterministic:
|
||||
- Cancelling a task does not make it "done" immediately; it becomes done only
|
||||
once the event loop schedules it and it processes the CancelledError.
|
||||
- In the buggy version, shutdown proceeded directly to db.shutdown_db()
|
||||
immediately after calling cancel(), so at *entry* to shutdown_db the task
|
||||
is still not done.
|
||||
- In the fixed version, lifespan does `await sync_task` before shutdown_db,
|
||||
so by the time shutdown_db is called, the task is done (cancelled).
|
||||
"""
|
||||
|
||||
# Import the *module* (not the package-level FastAPI `basic_memory.api.app` export)
|
||||
# so monkeypatching affects the exact symbols referenced inside lifespan().
|
||||
#
|
||||
# Note: `basic_memory/api/__init__.py` re-exports `app`, so `import basic_memory.api.app`
|
||||
# can resolve to the FastAPI instance rather than the `basic_memory.api.app` module.
|
||||
import importlib
|
||||
|
||||
api_app_module = importlib.import_module("basic_memory.api.app")
|
||||
|
||||
# Keep startup cheap: we don't need real DB init for this ordering test.
|
||||
async def _noop_initialize_app(_app_config):
|
||||
return None
|
||||
|
||||
async def _fake_get_or_create_db(*_args, **_kwargs):
|
||||
return object(), object()
|
||||
|
||||
monkeypatch.setattr(api_app_module, "initialize_app", _noop_initialize_app)
|
||||
monkeypatch.setattr(api_app_module.db, "get_or_create_db", _fake_get_or_create_db)
|
||||
|
||||
# Make the sync task long-lived so it must be cancelled on shutdown.
|
||||
async def _fake_initialize_file_sync(_app_config):
|
||||
await asyncio.Event().wait()
|
||||
|
||||
monkeypatch.setattr(api_app_module, "initialize_file_sync", _fake_initialize_file_sync)
|
||||
|
||||
# Assert ordering: shutdown_db must be called only after the sync_task is done.
|
||||
async def _assert_sync_task_done_before_db_shutdown():
|
||||
assert api_app_module.app.state.sync_task is not None
|
||||
assert api_app_module.app.state.sync_task.done()
|
||||
|
||||
monkeypatch.setattr(api_app_module.db, "shutdown_db", _assert_sync_task_done_before_db_shutdown)
|
||||
|
||||
async def _run_client_once():
|
||||
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
|
||||
# Any request is sufficient to trigger lifespan startup/shutdown.
|
||||
await client.get("/__nonexistent__")
|
||||
|
||||
# Use asyncio.run to match the CLI/MCP execution model where loop teardown
|
||||
# would hang if a background task is left running.
|
||||
asyncio.run(_run_client_once())
|
||||
|
||||
|
||||
@@ -46,57 +46,3 @@ async def test_read_note_after_write(mcp_server, app, test_project):
|
||||
assert "# Test Note" in result_text
|
||||
assert "This is test content." in result_text
|
||||
assert "test/test-note" in result_text # permalink
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_read_note_underscored_folder_by_permalink(mcp_server, app, test_project):
|
||||
"""Test read_note with permalink from underscored folder.
|
||||
|
||||
Reproduces bug #416: read_note fails to find notes when given permalinks
|
||||
from underscored folder names (e.g., _archive/, _drafts/), even though
|
||||
the permalink is copied directly from the note's YAML frontmatter.
|
||||
"""
|
||||
|
||||
async with Client(mcp_server) as client:
|
||||
# Create a note in an underscored folder
|
||||
write_result = await client.call_tool(
|
||||
"write_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"title": "Example Note",
|
||||
"folder": "_archive/articles",
|
||||
"content": "# Example Note\n\nThis is a test note in an underscored folder.",
|
||||
"tags": "test,archive",
|
||||
},
|
||||
)
|
||||
|
||||
assert len(write_result.content) == 1
|
||||
assert write_result.content[0].type == "text"
|
||||
write_text = write_result.content[0].text
|
||||
|
||||
# Verify the file path includes the underscore
|
||||
assert "_archive/articles/Example Note.md" in write_text
|
||||
|
||||
# Verify the permalink has underscores stripped (this is the expected behavior)
|
||||
assert "archive/articles/example-note" in write_text
|
||||
|
||||
# Now try to read the note using the permalink (without underscores)
|
||||
# This is the exact scenario from the bug report - using the permalink
|
||||
# that was generated in the YAML frontmatter
|
||||
read_result = await client.call_tool(
|
||||
"read_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"identifier": "archive/articles/example-note", # permalink without underscores
|
||||
},
|
||||
)
|
||||
|
||||
# This should succeed - the note should be found by its permalink
|
||||
assert len(read_result.content) == 1
|
||||
assert read_result.content[0].type == "text"
|
||||
result_text = read_result.content[0].text
|
||||
|
||||
# Should contain the note content
|
||||
assert "# Example Note" in result_text
|
||||
assert "This is a test note in an underscored folder." in result_text
|
||||
assert "archive/articles/example-note" in result_text # permalink
|
||||
|
||||
@@ -5,6 +5,7 @@ and other SQLite configuration settings work correctly in production scenarios.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from unittest.mock import patch
|
||||
from sqlalchemy import text
|
||||
|
||||
|
||||
@@ -141,6 +142,23 @@ async def test_null_pool_on_windows(tmp_path, monkeypatch):
|
||||
assert isinstance(engine.pool, NullPool)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.skipif(
|
||||
__import__("os").name == "nt", reason="Non-Windows test - cannot mock POSIX paths on Windows"
|
||||
)
|
||||
async def test_regular_pool_on_non_windows(tmp_path):
|
||||
"""Test that regular pooling is used on non-Windows platforms."""
|
||||
from basic_memory.db import engine_session_factory, DatabaseType
|
||||
from sqlalchemy.pool import NullPool
|
||||
|
||||
db_path = tmp_path / "test_posix_pool.db"
|
||||
|
||||
with patch("basic_memory.db.os.name", "posix"):
|
||||
async with engine_session_factory(db_path, DatabaseType.FILESYSTEM) as (engine, _):
|
||||
# Engine should NOT be using NullPool on non-Windows
|
||||
assert not isinstance(engine.pool, NullPool)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.windows
|
||||
@pytest.mark.skipif(
|
||||
|
||||
@@ -0,0 +1,372 @@
|
||||
"""
|
||||
Performance benchmark tests for sync operations.
|
||||
|
||||
These tests measure baseline performance for indexing operations to track
|
||||
improvements from optimizations. Tests are marked with @pytest.mark.benchmark
|
||||
and can be run separately.
|
||||
|
||||
Usage:
|
||||
# Run all benchmarks
|
||||
pytest test-int/test_sync_performance_benchmark.py -v
|
||||
|
||||
# Run specific benchmark
|
||||
pytest test-int/test_sync_performance_benchmark.py::test_benchmark_sync_100_files -v
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
from pathlib import Path
|
||||
from textwrap import dedent
|
||||
|
||||
import pytest
|
||||
|
||||
from basic_memory.config import BasicMemoryConfig, ProjectConfig
|
||||
from basic_memory.sync.sync_service import get_sync_service
|
||||
|
||||
|
||||
async def create_benchmark_file(path: Path, file_num: int, total_files: int) -> None:
|
||||
"""Create a realistic test markdown file with observations and relations.
|
||||
|
||||
Args:
|
||||
path: Path to create the file at
|
||||
file_num: Current file number (for unique content)
|
||||
total_files: Total number of files being created (for relation targets)
|
||||
"""
|
||||
# Create realistic content with varying complexity
|
||||
has_relations = file_num < (total_files - 1) # Most files have relations
|
||||
num_observations = min(3 + (file_num % 5), 10) # 3-10 observations per file
|
||||
|
||||
# Generate relation targets (some will be forward references)
|
||||
relations = []
|
||||
if has_relations:
|
||||
# Reference 1-3 other files
|
||||
num_relations = min(1 + (file_num % 3), 3)
|
||||
for i in range(num_relations):
|
||||
target_num = (file_num + i + 1) % total_files
|
||||
relations.append(f"- relates_to [[test-file-{target_num:04d}]]")
|
||||
|
||||
content = dedent(f"""
|
||||
---
|
||||
type: note
|
||||
tags: [benchmark, test, category-{file_num % 10}]
|
||||
---
|
||||
# Test File {file_num:04d}
|
||||
|
||||
This is benchmark test file {file_num} of {total_files}.
|
||||
It contains realistic markdown content to simulate actual usage.
|
||||
|
||||
## Observations
|
||||
{chr(10).join([f"- [category-{i % 5}] Observation {i} for file {file_num} with some content #tag{i}" for i in range(num_observations)])}
|
||||
|
||||
## Relations
|
||||
{chr(10).join(relations) if relations else "- No relations for this file"}
|
||||
|
||||
## Additional Content
|
||||
|
||||
This section contains additional prose to simulate real documents.
|
||||
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed do eiusmod
|
||||
tempor incididunt ut labore et dolore magna aliqua.
|
||||
|
||||
### Subsection
|
||||
|
||||
More content here to make the file realistic. This helps test the
|
||||
full indexing pipeline including content extraction and search indexing.
|
||||
""").strip()
|
||||
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(content, encoding="utf-8")
|
||||
|
||||
|
||||
async def generate_benchmark_files(project_dir: Path, num_files: int) -> None:
|
||||
"""Generate benchmark test files.
|
||||
|
||||
Args:
|
||||
project_dir: Directory to create files in
|
||||
num_files: Number of files to generate
|
||||
"""
|
||||
print(f"\nGenerating {num_files} test files...")
|
||||
start = time.time()
|
||||
|
||||
# Create files in batches for faster generation
|
||||
batch_size = 100
|
||||
for batch_start in range(0, num_files, batch_size):
|
||||
batch_end = min(batch_start + batch_size, num_files)
|
||||
tasks = [
|
||||
create_benchmark_file(
|
||||
project_dir / f"category-{i % 10}" / f"test-file-{i:04d}.md", i, num_files
|
||||
)
|
||||
for i in range(batch_start, batch_end)
|
||||
]
|
||||
await asyncio.gather(*tasks)
|
||||
print(f" Created files {batch_start}-{batch_end} ({batch_end}/{num_files})")
|
||||
|
||||
duration = time.time() - start
|
||||
print(f" File generation completed in {duration:.2f}s ({num_files / duration:.1f} files/sec)")
|
||||
|
||||
|
||||
def get_db_size(db_path: Path) -> tuple[int, str]:
|
||||
"""Get database file size.
|
||||
|
||||
Returns:
|
||||
Tuple of (size_bytes, formatted_size)
|
||||
"""
|
||||
if not db_path.exists():
|
||||
return 0, "0 B"
|
||||
|
||||
size_bytes = db_path.stat().st_size
|
||||
|
||||
# Format size
|
||||
for unit in ["B", "KB", "MB", "GB"]:
|
||||
if size_bytes < 1024.0:
|
||||
return size_bytes, f"{size_bytes:.2f} {unit}"
|
||||
size_bytes /= 1024.0
|
||||
|
||||
return int(size_bytes * 1024**4), f"{size_bytes:.2f} TB"
|
||||
|
||||
|
||||
async def run_sync_benchmark(
|
||||
project_config: ProjectConfig, app_config: BasicMemoryConfig, num_files: int, test_name: str
|
||||
) -> dict:
|
||||
"""Run a sync benchmark and collect metrics.
|
||||
|
||||
Args:
|
||||
project_config: Project configuration
|
||||
app_config: App configuration
|
||||
num_files: Number of files to benchmark
|
||||
test_name: Name of the test for reporting
|
||||
|
||||
Returns:
|
||||
Dictionary with benchmark results
|
||||
"""
|
||||
project_dir = project_config.home
|
||||
db_path = app_config.database_path
|
||||
|
||||
print(f"\n{'=' * 70}")
|
||||
print(f"BENCHMARK: {test_name}")
|
||||
print(f"{'=' * 70}")
|
||||
|
||||
# Generate test files
|
||||
await generate_benchmark_files(project_dir, num_files)
|
||||
|
||||
# Get initial DB size
|
||||
initial_db_size, initial_db_formatted = get_db_size(db_path)
|
||||
print(f"\nInitial database size: {initial_db_formatted}")
|
||||
|
||||
# Create sync service
|
||||
from basic_memory.repository import ProjectRepository
|
||||
from basic_memory import db
|
||||
|
||||
_, session_maker = await db.get_or_create_db(
|
||||
db_path=app_config.database_path,
|
||||
db_type=db.DatabaseType.FILESYSTEM,
|
||||
)
|
||||
project_repository = ProjectRepository(session_maker)
|
||||
|
||||
# Get or create project
|
||||
projects = await project_repository.find_all()
|
||||
if projects:
|
||||
project = projects[0]
|
||||
else:
|
||||
project = await project_repository.create(
|
||||
{
|
||||
"name": project_config.name,
|
||||
"path": str(project_config.home),
|
||||
"is_active": True,
|
||||
"is_default": True,
|
||||
}
|
||||
)
|
||||
|
||||
sync_service = await get_sync_service(project)
|
||||
|
||||
# Initialize search index (required for FTS5 table)
|
||||
await sync_service.search_service.init_search_index()
|
||||
|
||||
# Run sync and measure time
|
||||
print(f"\nStarting sync of {num_files} files...")
|
||||
sync_start = time.time()
|
||||
|
||||
report = await sync_service.sync(project_dir, project_name=project.name)
|
||||
|
||||
sync_duration = time.time() - sync_start
|
||||
|
||||
# Get final DB size
|
||||
final_db_size, final_db_formatted = get_db_size(db_path)
|
||||
db_growth = final_db_size - initial_db_size
|
||||
db_growth_formatted = f"{db_growth / 1024 / 1024:.2f} MB"
|
||||
|
||||
# Calculate metrics
|
||||
files_per_sec = num_files / sync_duration if sync_duration > 0 else 0
|
||||
ms_per_file = (sync_duration * 1000) / num_files if num_files > 0 else 0
|
||||
|
||||
# Print results
|
||||
print(f"\n{'-' * 70}")
|
||||
print("RESULTS:")
|
||||
print(f"{'-' * 70}")
|
||||
print(f"Files processed: {num_files}")
|
||||
print(f" New: {len(report.new)}")
|
||||
print(f" Modified: {len(report.modified)}")
|
||||
print(f" Deleted: {len(report.deleted)}")
|
||||
print(f" Moved: {len(report.moves)}")
|
||||
print("\nPerformance:")
|
||||
print(f" Total time: {sync_duration:.2f}s")
|
||||
print(f" Files/sec: {files_per_sec:.1f}")
|
||||
print(f" ms/file: {ms_per_file:.1f}")
|
||||
print("\nDatabase:")
|
||||
print(f" Initial size: {initial_db_formatted}")
|
||||
print(f" Final size: {final_db_formatted}")
|
||||
print(f" Growth: {db_growth_formatted}")
|
||||
print(f" Growth per file: {(db_growth / num_files / 1024):.2f} KB")
|
||||
print(f"{'=' * 70}\n")
|
||||
|
||||
return {
|
||||
"test_name": test_name,
|
||||
"num_files": num_files,
|
||||
"sync_duration_sec": sync_duration,
|
||||
"files_per_sec": files_per_sec,
|
||||
"ms_per_file": ms_per_file,
|
||||
"new_files": len(report.new),
|
||||
"modified_files": len(report.modified),
|
||||
"deleted_files": len(report.deleted),
|
||||
"moved_files": len(report.moves),
|
||||
"initial_db_size": initial_db_size,
|
||||
"final_db_size": final_db_size,
|
||||
"db_growth_bytes": db_growth,
|
||||
"db_growth_per_file_bytes": db_growth / num_files if num_files > 0 else 0,
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.benchmark
|
||||
@pytest.mark.asyncio
|
||||
async def test_benchmark_sync_100_files(app_config, project_config, config_manager):
|
||||
"""Benchmark: Sync 100 files (small repository)."""
|
||||
results = await run_sync_benchmark(
|
||||
project_config, app_config, num_files=100, test_name="Sync 100 files (small repository)"
|
||||
)
|
||||
|
||||
# Basic assertions to ensure sync worked
|
||||
# Note: May be slightly more than 100 due to OS-generated files (.DS_Store, etc.)
|
||||
assert results["new_files"] >= 100
|
||||
assert results["sync_duration_sec"] > 0
|
||||
assert results["files_per_sec"] > 0
|
||||
|
||||
|
||||
@pytest.mark.benchmark
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.skip
|
||||
async def test_benchmark_sync_500_files(app_config, project_config, config_manager):
|
||||
"""Benchmark: Sync 500 files (medium repository)."""
|
||||
results = await run_sync_benchmark(
|
||||
project_config, app_config, num_files=500, test_name="Sync 500 files (medium repository)"
|
||||
)
|
||||
|
||||
# Basic assertions
|
||||
# Note: May be slightly more than 500 due to OS-generated files
|
||||
assert results["new_files"] >= 500
|
||||
assert results["sync_duration_sec"] > 0
|
||||
assert results["files_per_sec"] > 0
|
||||
|
||||
|
||||
@pytest.mark.benchmark
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.slow
|
||||
@pytest.mark.skip
|
||||
async def test_benchmark_sync_1000_files(app_config, project_config, config_manager):
|
||||
"""Benchmark: Sync 1000 files (large repository).
|
||||
|
||||
This test is marked as 'slow' and can be skipped in regular test runs:
|
||||
pytest -m "not slow"
|
||||
"""
|
||||
results = await run_sync_benchmark(
|
||||
project_config, app_config, num_files=1000, test_name="Sync 1000 files (large repository)"
|
||||
)
|
||||
|
||||
# Basic assertions
|
||||
# Note: May be slightly more than 1000 due to OS-generated files
|
||||
assert results["new_files"] >= 1000
|
||||
assert results["sync_duration_sec"] > 0
|
||||
assert results["files_per_sec"] > 0
|
||||
|
||||
|
||||
@pytest.mark.benchmark
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.skip
|
||||
async def test_benchmark_resync_no_changes(app_config, project_config, config_manager):
|
||||
"""Benchmark: Re-sync with no changes (should be fast).
|
||||
|
||||
This tests the performance of scanning files when nothing has changed,
|
||||
which is important for cloud restarts.
|
||||
"""
|
||||
project_dir = project_config.home
|
||||
num_files = 100
|
||||
|
||||
# First sync
|
||||
print(f"\nFirst sync of {num_files} files...")
|
||||
await generate_benchmark_files(project_dir, num_files)
|
||||
|
||||
from basic_memory.repository import ProjectRepository
|
||||
from basic_memory import db
|
||||
|
||||
_, session_maker = await db.get_or_create_db(
|
||||
db_path=app_config.database_path,
|
||||
db_type=db.DatabaseType.FILESYSTEM,
|
||||
)
|
||||
project_repository = ProjectRepository(session_maker)
|
||||
projects = await project_repository.find_all()
|
||||
if projects:
|
||||
project = projects[0]
|
||||
else:
|
||||
project = await project_repository.create(
|
||||
{
|
||||
"name": project_config.name,
|
||||
"path": str(project_config.home),
|
||||
"is_active": True,
|
||||
"is_default": True,
|
||||
}
|
||||
)
|
||||
|
||||
sync_service = await get_sync_service(project)
|
||||
|
||||
# Initialize search index
|
||||
await sync_service.search_service.init_search_index()
|
||||
|
||||
await sync_service.sync(project_dir, project_name=project.name)
|
||||
|
||||
# Second sync (no changes)
|
||||
print("\nRe-sync with no changes...")
|
||||
resync_start = time.time()
|
||||
report = await sync_service.sync(project_dir, project_name=project.name)
|
||||
resync_duration = time.time() - resync_start
|
||||
|
||||
print(f"\n{'-' * 70}")
|
||||
print("RE-SYNC RESULTS (no changes):")
|
||||
print(f"{'-' * 70}")
|
||||
print(f"Files scanned: {num_files}")
|
||||
print(f"Changes detected: {report.total}")
|
||||
print(f" New: {len(report.new)}")
|
||||
print(f" Modified: {len(report.modified)}")
|
||||
print(f" Deleted: {len(report.deleted)}")
|
||||
print(f" Moved: {len(report.moves)}")
|
||||
print(f"Duration: {resync_duration:.2f}s")
|
||||
print(f"Files/sec: {num_files / resync_duration:.1f}")
|
||||
|
||||
# Debug: Show what changed
|
||||
if report.total > 0:
|
||||
print("\n⚠️ UNEXPECTED CHANGES DETECTED:")
|
||||
if report.new:
|
||||
print(f" New files ({len(report.new)}): {list(report.new)[:5]}")
|
||||
if report.modified:
|
||||
print(f" Modified files ({len(report.modified)}): {list(report.modified)[:5]}")
|
||||
if report.deleted:
|
||||
print(f" Deleted files ({len(report.deleted)}): {list(report.deleted)[:5]}")
|
||||
if report.moves:
|
||||
print(f" Moved files ({len(report.moves)}): {dict(list(report.moves.items())[:5])}")
|
||||
|
||||
print(f"{'=' * 70}\n")
|
||||
|
||||
# Should be no changes
|
||||
assert report.total == 0, (
|
||||
f"Expected no changes but got {report.total}: new={len(report.new)}, modified={len(report.modified)}, deleted={len(report.deleted)}, moves={len(report.moves)}"
|
||||
)
|
||||
assert len(report.new) == 0
|
||||
assert len(report.modified) == 0
|
||||
assert len(report.deleted) == 0
|
||||
@@ -9,19 +9,17 @@ from basic_memory.mcp.async_client import create_client
|
||||
|
||||
|
||||
def test_create_client_uses_asgi_when_no_remote_env():
|
||||
"""Test that create_client uses ASGI transport when cloud mode is disabled."""
|
||||
# Ensure env vars are not set and config cloud_mode is False
|
||||
"""Test that create_client uses ASGI transport when BASIC_MEMORY_USE_REMOTE_API is not set."""
|
||||
# Ensure env vars are not set (pop if they exist)
|
||||
with patch.dict("os.environ", clear=False):
|
||||
os.environ.pop("BASIC_MEMORY_USE_REMOTE_API", None)
|
||||
os.environ.pop("BASIC_MEMORY_CLOUD_MODE", None)
|
||||
|
||||
# Also patch the config's cloud_mode to ensure it's False
|
||||
with patch.object(ConfigManager().config, "cloud_mode", False):
|
||||
client = create_client()
|
||||
client = create_client()
|
||||
|
||||
assert isinstance(client, AsyncClient)
|
||||
assert isinstance(client._transport, ASGITransport)
|
||||
assert str(client.base_url) == "http://test"
|
||||
assert isinstance(client, AsyncClient)
|
||||
assert isinstance(client._transport, ASGITransport)
|
||||
assert str(client.base_url) == "http://test"
|
||||
|
||||
|
||||
def test_create_client_uses_http_when_cloud_mode_env_set():
|
||||
@@ -39,18 +37,16 @@ def test_create_client_uses_http_when_cloud_mode_env_set():
|
||||
|
||||
def test_create_client_configures_extended_timeouts():
|
||||
"""Test that create_client configures 30-second timeouts for long operations."""
|
||||
# Ensure env vars are not set and config cloud_mode is False
|
||||
# Ensure env vars are not set (pop if they exist)
|
||||
with patch.dict("os.environ", clear=False):
|
||||
os.environ.pop("BASIC_MEMORY_USE_REMOTE_API", None)
|
||||
os.environ.pop("BASIC_MEMORY_CLOUD_MODE", None)
|
||||
|
||||
# Also patch the config's cloud_mode to ensure it's False
|
||||
with patch.object(ConfigManager().config, "cloud_mode", False):
|
||||
client = create_client()
|
||||
client = create_client()
|
||||
|
||||
# Verify timeout configuration
|
||||
assert isinstance(client.timeout, Timeout)
|
||||
assert client.timeout.connect == 10.0 # 10 seconds for connection
|
||||
assert client.timeout.read == 30.0 # 30 seconds for reading
|
||||
assert client.timeout.write == 30.0 # 30 seconds for writing
|
||||
assert client.timeout.pool == 30.0 # 30 seconds for pool
|
||||
# Verify timeout configuration
|
||||
assert isinstance(client.timeout, Timeout)
|
||||
assert client.timeout.connect == 10.0 # 10 seconds for connection
|
||||
assert client.timeout.read == 30.0 # 30 seconds for reading
|
||||
assert client.timeout.write == 30.0 # 30 seconds for writing
|
||||
assert client.timeout.pool == 30.0 # 30 seconds for pool
|
||||
|
||||
@@ -218,13 +218,9 @@ async def test_get_resource_entities(client, project_config, entity_repository,
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_resource_entities_pagination(
|
||||
client, project_config, entity_repository, project_url, db_backend
|
||||
client, project_config, entity_repository, project_url
|
||||
):
|
||||
"""Test getting content by permalink match."""
|
||||
if db_backend == "postgres":
|
||||
pytest.skip(
|
||||
"Pagination differs: relations expand to multiple entities, ordering is undefined"
|
||||
)
|
||||
# Create entity
|
||||
content1 = "# Test Content\n"
|
||||
data = {
|
||||
|
||||
+1
-28
@@ -1,8 +1,5 @@
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import AsyncGenerator
|
||||
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
from fastapi import FastAPI
|
||||
from httpx import AsyncClient, ASGITransport
|
||||
@@ -11,31 +8,7 @@ from basic_memory.api.app import app as fastapi_app
|
||||
from basic_memory.deps import get_project_config, get_engine_factory, get_app_config
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def isolated_home(tmp_path, monkeypatch) -> Path:
|
||||
"""Isolate tests from user's HOME directory.
|
||||
|
||||
This prevents tests from reading/writing to ~/.basic-memory/.bmignore
|
||||
or other user-specific configuration.
|
||||
|
||||
Sets BASIC_MEMORY_HOME to tmp_path directly so the default project
|
||||
writes files to tmp_path, which is where tests expect to find them.
|
||||
"""
|
||||
# Clear config cache to ensure fresh config for each test
|
||||
from basic_memory import config as config_module
|
||||
|
||||
config_module._CONFIG_CACHE = None
|
||||
|
||||
monkeypatch.setenv("HOME", str(tmp_path))
|
||||
if os.name == "nt":
|
||||
monkeypatch.setenv("USERPROFILE", str(tmp_path))
|
||||
# Set to tmp_path directly (not tmp_path/basic-memory) so default project
|
||||
# home is tmp_path - tests expect to find imported files there
|
||||
monkeypatch.setenv("BASIC_MEMORY_HOME", str(tmp_path))
|
||||
return tmp_path
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
@pytest_asyncio.fixture(autouse=True)
|
||||
async def app(app_config, project_config, engine_factory, test_config, aiolib) -> FastAPI:
|
||||
"""Create test FastAPI application."""
|
||||
app = fastapi_app
|
||||
|
||||
+93
-113
@@ -1,23 +1,16 @@
|
||||
"""Common test fixtures."""
|
||||
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from textwrap import dedent
|
||||
from typing import AsyncGenerator
|
||||
from typing import AsyncGenerator, Literal
|
||||
|
||||
import os
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.ext.asyncio import (
|
||||
AsyncEngine,
|
||||
AsyncSession,
|
||||
async_sessionmaker,
|
||||
create_async_engine,
|
||||
)
|
||||
from sqlalchemy.pool import NullPool
|
||||
from testcontainers.postgres import PostgresContainer
|
||||
from sqlalchemy.ext.asyncio import AsyncEngine, AsyncSession, async_sessionmaker
|
||||
|
||||
from basic_memory import db
|
||||
from basic_memory.config import ProjectConfig, BasicMemoryConfig, ConfigManager, DatabaseBackend
|
||||
@@ -44,47 +37,32 @@ from basic_memory.sync.sync_service import SyncService
|
||||
from basic_memory.sync.watch_service import WatchService
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Database Backend Selection (env var approach)
|
||||
# =============================================================================
|
||||
# By default, tests run against SQLite.
|
||||
# Set BASIC_MEMORY_TEST_POSTGRES=1 to run against Postgres (uses testcontainers).
|
||||
# This allows running sqlite/postgres tests in parallel in CI.
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def db_backend():
|
||||
"""Determine database backend from environment variable.
|
||||
|
||||
Default: sqlite
|
||||
Set BASIC_MEMORY_TEST_POSTGRES=1 to use postgres
|
||||
"""
|
||||
if os.environ.get("BASIC_MEMORY_TEST_POSTGRES", "").lower() in ("1", "true", "yes"):
|
||||
return "postgres"
|
||||
return "sqlite"
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def postgres_container(db_backend):
|
||||
"""Session-scoped Postgres container for tests.
|
||||
|
||||
Uses testcontainers to spin up a real Postgres instance in Docker.
|
||||
The container is started once per test session and shared across all tests.
|
||||
Only starts if db_backend is "postgres".
|
||||
"""
|
||||
if db_backend != "postgres":
|
||||
yield None
|
||||
return
|
||||
|
||||
with PostgresContainer("postgres:16-alpine") as postgres:
|
||||
yield postgres
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def anyio_backend():
|
||||
return "asyncio"
|
||||
|
||||
|
||||
@pytest.fixture(
|
||||
params=[
|
||||
pytest.param("sqlite", id="sqlite"),
|
||||
pytest.param("postgres", id="postgres", marks=pytest.mark.postgres),
|
||||
]
|
||||
)
|
||||
def db_backend(request) -> Literal["sqlite", "postgres"]:
|
||||
"""Parametrize tests to run against both SQLite and Postgres.
|
||||
|
||||
Usage:
|
||||
pytest # Runs tests against SQLite only (default)
|
||||
pytest -m postgres # Runs tests against Postgres only
|
||||
pytest -m "not postgres" # Runs tests against SQLite only
|
||||
pytest --run-all-backends # Runs tests against both backends
|
||||
|
||||
Note: Only tests that use database fixtures (engine_factory, session_maker, etc.)
|
||||
will be parametrized. Tests that don't use the database won't be affected.
|
||||
"""
|
||||
return request.param
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def project_root() -> Path:
|
||||
return Path(__file__).parent.parent
|
||||
@@ -103,18 +81,24 @@ def config_home(tmp_path, monkeypatch) -> Path:
|
||||
|
||||
|
||||
@pytest.fixture(scope="function")
|
||||
def app_config(config_home, db_backend, postgres_container, monkeypatch) -> BasicMemoryConfig:
|
||||
"""Create test app configuration for the appropriate backend."""
|
||||
def app_config(
|
||||
config_home, db_backend: Literal["sqlite", "postgres"], monkeypatch
|
||||
) -> BasicMemoryConfig:
|
||||
"""Create test app configuration."""
|
||||
# Create a basic config without depending on test_project to avoid circular dependency
|
||||
projects = {"test-project": str(config_home)}
|
||||
|
||||
# Set backend based on parameterized db_backend fixture
|
||||
# Configure database backend based on test parameter
|
||||
if db_backend == "postgres":
|
||||
backend = DatabaseBackend.POSTGRES
|
||||
# Get URL from testcontainer and convert to asyncpg driver
|
||||
sync_url = postgres_container.get_connection_url()
|
||||
database_url = sync_url.replace("postgresql+psycopg2", "postgresql+asyncpg")
|
||||
database_backend = DatabaseBackend.POSTGRES
|
||||
# Use env var if set, otherwise use default matching docker-compose-postgres.yml
|
||||
# These are local test credentials only - NOT for production
|
||||
database_url = os.getenv(
|
||||
"POSTGRES_TEST_URL",
|
||||
"postgresql+asyncpg://basic_memory_user:dev_password@localhost:5433/basic_memory_test",
|
||||
)
|
||||
else:
|
||||
backend = DatabaseBackend.SQLITE
|
||||
database_backend = DatabaseBackend.SQLITE
|
||||
database_url = None
|
||||
|
||||
app_config = BasicMemoryConfig(
|
||||
@@ -122,7 +106,7 @@ def app_config(config_home, db_backend, postgres_container, monkeypatch) -> Basi
|
||||
projects=projects,
|
||||
default_project="test-project",
|
||||
update_permalinks_on_move=True,
|
||||
database_backend=backend,
|
||||
database_backend=database_backend,
|
||||
database_url=database_url,
|
||||
)
|
||||
|
||||
@@ -178,66 +162,76 @@ def test_config(config_home, project_config, app_config, config_manager) -> Test
|
||||
async def engine_factory(
|
||||
app_config,
|
||||
config_manager,
|
||||
db_backend,
|
||||
postgres_container,
|
||||
db_backend: Literal["sqlite", "postgres"],
|
||||
) -> AsyncGenerator[tuple[AsyncEngine, async_sessionmaker[AsyncSession]], None]:
|
||||
"""Engine factory for SQLite or Postgres tests.
|
||||
|
||||
Uses parameterized db_backend fixture to run tests against both backends.
|
||||
"""
|
||||
"""Create engine and session factory for the configured database backend."""
|
||||
from basic_memory.models.search import CREATE_SEARCH_INDEX
|
||||
|
||||
if db_backend == "postgres":
|
||||
# Postgres mode using testcontainers
|
||||
# Get async connection URL (asyncpg driver - same as production)
|
||||
sync_url = postgres_container.get_connection_url()
|
||||
async_url = sync_url.replace("postgresql+psycopg2", "postgresql+asyncpg")
|
||||
# Postgres: Create fresh engine for each test with full schema reset
|
||||
config_manager._config = app_config
|
||||
db_type = DatabaseType.FILESYSTEM
|
||||
|
||||
engine = create_async_engine(
|
||||
async_url,
|
||||
echo=False,
|
||||
poolclass=NullPool, # NullPool for better test isolation
|
||||
)
|
||||
# Use context manager to handle engine disposal properly
|
||||
async with db.engine_session_factory(db_path=app_config.database_path, db_type=db_type) as (
|
||||
engine,
|
||||
session_maker,
|
||||
):
|
||||
# Drop and recreate schema for complete isolation
|
||||
async with engine.begin() as conn:
|
||||
await conn.execute(text("DROP SCHEMA IF EXISTS public CASCADE"))
|
||||
await conn.execute(text("CREATE SCHEMA public"))
|
||||
await conn.execute(text("GRANT ALL ON SCHEMA public TO basic_memory_user"))
|
||||
await conn.execute(text("GRANT ALL ON SCHEMA public TO public"))
|
||||
|
||||
session_maker = async_sessionmaker(
|
||||
bind=engine,
|
||||
class_=AsyncSession,
|
||||
expire_on_commit=False,
|
||||
autoflush=False,
|
||||
)
|
||||
# Run migrations to create production tables (including search_index with correct schema)
|
||||
# Alembic handles duplicate migration checks, so it's safe to call this for each test
|
||||
from basic_memory.db import run_migrations
|
||||
|
||||
from basic_memory.models.search import (
|
||||
CREATE_POSTGRES_SEARCH_INDEX_TABLE,
|
||||
CREATE_POSTGRES_SEARCH_INDEX_FTS,
|
||||
CREATE_POSTGRES_SEARCH_INDEX_METADATA,
|
||||
)
|
||||
await run_migrations(app_config, db_type)
|
||||
|
||||
# Drop and recreate all tables for test isolation
|
||||
async with engine.begin() as conn:
|
||||
# Must drop search_index first (has FK to project, blocks drop_all)
|
||||
await conn.execute(text("DROP TABLE IF EXISTS search_index CASCADE"))
|
||||
await conn.run_sync(Base.metadata.drop_all)
|
||||
await conn.run_sync(Base.metadata.create_all)
|
||||
# Create search_index via DDL (not ORM - uses composite PK + tsvector)
|
||||
# asyncpg requires separate execute calls for each statement
|
||||
await conn.execute(CREATE_POSTGRES_SEARCH_INDEX_TABLE)
|
||||
await conn.execute(CREATE_POSTGRES_SEARCH_INDEX_FTS)
|
||||
await conn.execute(CREATE_POSTGRES_SEARCH_INDEX_METADATA)
|
||||
# For Postgres, migrations create all production tables with correct schemas
|
||||
# We only need to create test-specific tables (like ModelTest) that aren't in migrations
|
||||
# Don't create search_index via ORM - it's already created by migration with composite PK
|
||||
async with engine.begin() as conn:
|
||||
# List of tables created by migrations - don't recreate them via ORM
|
||||
production_tables = {
|
||||
"entity",
|
||||
"observation",
|
||||
"relation",
|
||||
"project",
|
||||
"search_index",
|
||||
"alembic_version",
|
||||
}
|
||||
|
||||
yield engine, session_maker
|
||||
# Get test-specific tables that aren't created by migrations
|
||||
test_tables = [
|
||||
table
|
||||
for table in Base.metadata.sorted_tables
|
||||
if table.name not in production_tables
|
||||
]
|
||||
if test_tables:
|
||||
await conn.run_sync(
|
||||
lambda sync_conn: Base.metadata.create_all(sync_conn, tables=test_tables)
|
||||
)
|
||||
|
||||
await engine.dispose()
|
||||
yield engine, session_maker
|
||||
else:
|
||||
# SQLite mode
|
||||
# SQLite: Create fresh in-memory database for each test
|
||||
db_type = DatabaseType.MEMORY
|
||||
async with db.engine_session_factory(db_path=app_config.database_path, db_type=db_type) as (
|
||||
engine,
|
||||
session_maker,
|
||||
):
|
||||
# Create all tables via ORM, then add search_index via FTS5 DDL
|
||||
# Create all tables via ORM
|
||||
async with engine.begin() as conn:
|
||||
await conn.run_sync(Base.metadata.create_all)
|
||||
await conn.execute(CREATE_SEARCH_INDEX)
|
||||
|
||||
# Drop any SearchIndex ORM table, then create FTS5 virtual table
|
||||
async with db.scoped_session(session_maker) as session:
|
||||
await session.execute(text("DROP TABLE IF EXISTS search_index"))
|
||||
await session.execute(CREATE_SEARCH_INDEX)
|
||||
await session.commit()
|
||||
|
||||
# Yield after setup is complete
|
||||
yield engine, session_maker
|
||||
@@ -549,22 +543,8 @@ async def test_graph(
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def watch_service(app_config: BasicMemoryConfig, project_repository, sync_service) -> WatchService:
|
||||
"""Create WatchService with injected sync_service factory.
|
||||
|
||||
The sync_service_factory allows tests to use the fixture-provided sync_service
|
||||
instead of the production get_sync_service() which creates its own db connection.
|
||||
"""
|
||||
|
||||
async def sync_service_factory(project):
|
||||
"""Return the test fixture's sync_service regardless of project."""
|
||||
return sync_service
|
||||
|
||||
return WatchService(
|
||||
app_config=app_config,
|
||||
project_repository=project_repository,
|
||||
sync_service_factory=sync_service_factory,
|
||||
)
|
||||
def watch_service(app_config: BasicMemoryConfig, project_repository) -> WatchService:
|
||||
return WatchService(app_config=app_config, project_repository=project_repository)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
|
||||
@@ -85,11 +85,11 @@ async def test_parse_complete_file(project_config, entity_parser, valid_entity_c
|
||||
), "missing [[Auth API Spec]]"
|
||||
|
||||
# inline links in content
|
||||
assert Relation(type="links_to", target="Random Link", context=None) in entity.relations, (
|
||||
assert Relation(type="links to", target="Random Link", context=None) in entity.relations, (
|
||||
"missing [[Random Link]]"
|
||||
)
|
||||
assert (
|
||||
Relation(type="links_to", target="Random Link with Title|Titled Link", context=None)
|
||||
Relation(type="links to", target="Random Link with Title|Titled Link", context=None)
|
||||
in entity.relations
|
||||
), "missing [[Random Link with Title|Titled Link]]"
|
||||
|
||||
@@ -179,7 +179,7 @@ async def test_parse_file_without_section_headers(project_config, entity_parser)
|
||||
assert entity.observations[0].tags == ["test"]
|
||||
|
||||
assert len(entity.relations) == 2
|
||||
assert entity.relations[0].type == "links_to"
|
||||
assert entity.relations[0].type == "links to"
|
||||
assert entity.relations[0].target == "Random Link"
|
||||
|
||||
assert entity.relations[1].type == "references"
|
||||
|
||||
@@ -121,44 +121,6 @@ def test_observation_excludes_markdown_and_wiki_links():
|
||||
assert not is_observation(token), "No space after category should not be valid observation"
|
||||
|
||||
|
||||
def test_observation_excludes_html_color_codes():
|
||||
"""Test that HTML color codes are NOT interpreted as hashtags.
|
||||
|
||||
This test validates the fix for issue #446 where:
|
||||
- HTML color codes like #4285F4 in attributes were incorrectly
|
||||
causing lines to be parsed as observations.
|
||||
"""
|
||||
# HTML color code in font tag should NOT be an observation
|
||||
token = Token("inline", '**<font color="#4285F4">Jane:</font>** Welcome to the show', 0)
|
||||
assert not is_observation(token), "HTML color codes should not trigger hashtag detection"
|
||||
|
||||
# Color code in style attribute
|
||||
token = Token("inline", '<span style="color:#FF5733">Styled text</span>', 0)
|
||||
assert not is_observation(token), "Color codes in style should not be observations"
|
||||
|
||||
# Multiple color codes
|
||||
token = Token(
|
||||
"inline", '<font color="#4285F4">Blue</font> and <font color="#EA4335">Red</font>', 0
|
||||
)
|
||||
assert not is_observation(token), "Multiple color codes should not be observations"
|
||||
|
||||
# Hex color without quotes (edge case)
|
||||
token = Token("inline", "background-color:#FFFFFF is white", 0)
|
||||
assert not is_observation(token), "Inline hex colors should not be observations"
|
||||
|
||||
# But standalone hashtags SHOULD still work
|
||||
token = Token("inline", "This has a #realtag in it", 0)
|
||||
assert is_observation(token), "Standalone hashtags should still work"
|
||||
|
||||
# Multiple real hashtags
|
||||
token = Token("inline", "Tags: #design #feature #important", 0)
|
||||
assert is_observation(token), "Multiple standalone hashtags should work"
|
||||
|
||||
# Mix of color code and real tag - should be observation because of real tag
|
||||
token = Token("inline", '<font color="#4285F4">Text</font> #actualtag', 0)
|
||||
assert is_observation(token), "Real hashtag with color code should still be observation"
|
||||
|
||||
|
||||
def test_relation_plugin():
|
||||
"""Test relation plugin."""
|
||||
md = MarkdownIt().use(relation_plugin)
|
||||
@@ -181,7 +143,7 @@ def test_relation_plugin():
|
||||
token = [t for t in md.parse(content) if t.type == "inline"][0]
|
||||
rels = token.meta["relations"]
|
||||
assert len(rels) == 2
|
||||
assert rels[0]["type"] == "links_to"
|
||||
assert rels[0]["type"] == "links to"
|
||||
assert rels[0]["target"] == "Link"
|
||||
assert rels[1]["target"] == "Another Link"
|
||||
|
||||
@@ -246,4 +208,4 @@ def test_combined_plugins():
|
||||
text_token = inline_tokens[4]
|
||||
assert "relations" in text_token.meta
|
||||
link = text_token.meta["relations"][0]
|
||||
assert link["type"] == "links_to"
|
||||
assert link["type"] == "links to"
|
||||
|
||||
@@ -88,7 +88,7 @@ async def test_missing_sections(tmp_path):
|
||||
entity = await parser.parse_file(test_file)
|
||||
assert len(entity.relations) == 1
|
||||
assert entity.relations[0].target == "links"
|
||||
assert entity.relations[0].type == "links_to"
|
||||
assert entity.relations[0].type == "links to"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
@@ -50,7 +50,6 @@ def test_entity_data():
|
||||
}
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
@pytest_asyncio.fixture(autouse=True)
|
||||
async def init_search_index(search_service: SearchService):
|
||||
"""Initialize search index. Request this fixture explicitly in tests that need it."""
|
||||
await search_service.init_search_index()
|
||||
|
||||
@@ -1,9 +1,7 @@
|
||||
"""Tests for the move_note MCP tool."""
|
||||
|
||||
import pytest
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch, MagicMock
|
||||
from contextlib import asynccontextmanager
|
||||
from unittest.mock import patch
|
||||
|
||||
from basic_memory.mcp.tools.move_note import move_note, _format_move_error_response
|
||||
from basic_memory.mcp.tools.write_note import write_note
|
||||
@@ -889,71 +887,36 @@ class TestMoveNoteSecurityValidation:
|
||||
|
||||
|
||||
class TestMoveNoteErrorHandling:
|
||||
"""Test move note exception handling.
|
||||
|
||||
These are pure unit tests that mock get_client and other dependencies.
|
||||
They don't need the database or ASGI app.
|
||||
"""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_client(self):
|
||||
"""Create a mock async client context manager."""
|
||||
mock = MagicMock()
|
||||
|
||||
@asynccontextmanager
|
||||
async def mock_get_client():
|
||||
yield mock
|
||||
|
||||
return mock_get_client, mock
|
||||
"""Test move note exception handling."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_move_note_exception_handling(self, mock_client):
|
||||
async def test_move_note_exception_handling(self):
|
||||
"""Test exception handling in move_note."""
|
||||
mock_get_client, _ = mock_client
|
||||
with patch("basic_memory.mcp.tools.move_note.get_active_project") as mock_get_project:
|
||||
mock_get_project.return_value.project_url = "http://test"
|
||||
mock_get_project.return_value.name = "test-project"
|
||||
|
||||
with patch("basic_memory.mcp.tools.move_note.get_client", mock_get_client):
|
||||
with patch("basic_memory.mcp.tools.move_note.get_active_project") as mock_get_project:
|
||||
mock_get_project.return_value.project_url = "http://test"
|
||||
mock_get_project.return_value.name = "test-project"
|
||||
mock_get_project.return_value.home = Path("/tmp/test")
|
||||
with patch(
|
||||
"basic_memory.mcp.tools.move_note.call_post",
|
||||
side_effect=Exception("entity not found"),
|
||||
):
|
||||
result = await move_note.fn("test-note", "target/file.md", project="test-project")
|
||||
|
||||
with patch(
|
||||
"basic_memory.mcp.tools.move_note.call_post",
|
||||
side_effect=Exception("entity not found"),
|
||||
):
|
||||
with patch(
|
||||
"basic_memory.mcp.tools.move_note.call_get",
|
||||
side_effect=Exception("not found"),
|
||||
):
|
||||
result = await move_note.fn(
|
||||
"test-note", "target/file.md", project="test-project"
|
||||
)
|
||||
|
||||
assert isinstance(result, str)
|
||||
assert "# Move Failed - Note Not Found" in result
|
||||
assert isinstance(result, str)
|
||||
assert "# Move Failed - Note Not Found" in result
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_move_note_permission_error_handling(self, mock_client):
|
||||
async def test_move_note_permission_error_handling(self):
|
||||
"""Test permission error handling in move_note."""
|
||||
mock_get_client, _ = mock_client
|
||||
with patch("basic_memory.mcp.tools.move_note.get_active_project") as mock_get_project:
|
||||
mock_get_project.return_value.project_url = "http://test"
|
||||
mock_get_project.return_value.name = "test-project"
|
||||
|
||||
with patch("basic_memory.mcp.tools.move_note.get_client", mock_get_client):
|
||||
with patch("basic_memory.mcp.tools.move_note.get_active_project") as mock_get_project:
|
||||
mock_get_project.return_value.project_url = "http://test"
|
||||
mock_get_project.return_value.name = "test-project"
|
||||
mock_get_project.return_value.home = Path("/tmp/test")
|
||||
with patch(
|
||||
"basic_memory.mcp.tools.move_note.call_post",
|
||||
side_effect=Exception("permission denied"),
|
||||
):
|
||||
result = await move_note.fn("test-note", "target/file.md", project="test-project")
|
||||
|
||||
with patch(
|
||||
"basic_memory.mcp.tools.move_note.call_post",
|
||||
side_effect=Exception("permission denied"),
|
||||
):
|
||||
with patch(
|
||||
"basic_memory.mcp.tools.move_note.call_get",
|
||||
side_effect=Exception("not found"),
|
||||
):
|
||||
result = await move_note.fn(
|
||||
"test-note", "target/file.md", project="test-project"
|
||||
)
|
||||
|
||||
assert isinstance(result, str)
|
||||
assert "# Move Failed - Permission Error" in result
|
||||
assert isinstance(result, str)
|
||||
assert "# Move Failed - Permission Error" in result
|
||||
|
||||
@@ -18,12 +18,10 @@ async def entity_with_observations(session_maker, sample_entity):
|
||||
async with db.scoped_session(session_maker) as session:
|
||||
observations = [
|
||||
Observation(
|
||||
project_id=sample_entity.project_id,
|
||||
entity_id=sample_entity.id,
|
||||
content="First observation",
|
||||
),
|
||||
Observation(
|
||||
project_id=sample_entity.project_id,
|
||||
entity_id=sample_entity.id,
|
||||
content="Second observation",
|
||||
),
|
||||
@@ -61,7 +59,6 @@ async def related_results(session_maker, test_project: Project):
|
||||
await session.flush()
|
||||
|
||||
relation = Relation(
|
||||
project_id=test_project.id,
|
||||
from_id=source.id,
|
||||
to_id=target.id,
|
||||
to_name=target.title,
|
||||
@@ -179,55 +176,6 @@ async def test_update_entity(entity_repository: EntityRepository, sample_entity:
|
||||
assert db_entity.title == "Updated title"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_entity_returns_with_relations_and_observations(
|
||||
entity_repository: EntityRepository, entity_with_observations, test_project: Project
|
||||
):
|
||||
"""Test that update() returns entity with observations and relations eagerly loaded."""
|
||||
entity = entity_with_observations
|
||||
|
||||
# Create a target entity and relation
|
||||
async with db.scoped_session(entity_repository.session_maker) as session:
|
||||
target = Entity(
|
||||
project_id=test_project.id,
|
||||
title="target",
|
||||
entity_type="test",
|
||||
permalink="target/target",
|
||||
file_path="target/target.md",
|
||||
content_type="text/markdown",
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
)
|
||||
session.add(target)
|
||||
await session.flush()
|
||||
|
||||
relation = Relation(
|
||||
project_id=test_project.id,
|
||||
from_id=entity.id,
|
||||
to_id=target.id,
|
||||
to_name=target.title,
|
||||
relation_type="connects_to",
|
||||
)
|
||||
session.add(relation)
|
||||
|
||||
# Now update the entity
|
||||
updated = await entity_repository.update(entity.id, {"title": "Updated with relations"})
|
||||
|
||||
# Verify returned entity has observations and relations accessible
|
||||
# (would raise DetachedInstanceError if not eagerly loaded)
|
||||
assert updated is not None
|
||||
assert updated.title == "Updated with relations"
|
||||
|
||||
# Access observations - should NOT raise DetachedInstanceError
|
||||
assert len(updated.observations) == 2
|
||||
assert updated.observations[0].content in ["First observation", "Second observation"]
|
||||
|
||||
# Access relations - should NOT raise DetachedInstanceError
|
||||
assert len(updated.relations) == 1
|
||||
assert updated.relations[0].relation_type == "connects_to"
|
||||
assert updated.relations[0].to_name == "target"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_delete_entity(entity_repository: EntityRepository, sample_entity):
|
||||
"""Test deleting an entity."""
|
||||
@@ -789,7 +737,6 @@ async def test_get_all_file_paths_performance(entity_repository: EntityRepositor
|
||||
|
||||
# Add observations to entity1
|
||||
observation = Observation(
|
||||
project_id=entity_repository.project_id,
|
||||
entity_id=entity1.id,
|
||||
content="Test observation",
|
||||
category="note",
|
||||
@@ -798,7 +745,6 @@ async def test_get_all_file_paths_performance(entity_repository: EntityRepositor
|
||||
|
||||
# Add relation between entities
|
||||
relation = Relation(
|
||||
project_id=entity_repository.project_id,
|
||||
from_id=entity1.id,
|
||||
to_id=entity2.id,
|
||||
to_name=entity2.title,
|
||||
@@ -864,191 +810,3 @@ async def test_get_all_file_paths_project_isolation(
|
||||
# Should only include files from project 1
|
||||
assert len(file_paths) == 1
|
||||
assert file_paths == ["test/file1.md"]
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Tests for lightweight permalink resolution methods
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_permalink_exists(entity_repository: EntityRepository, sample_entity: Entity):
|
||||
"""Test checking if a permalink exists without loading full entity."""
|
||||
# Existing permalink should return True
|
||||
assert await entity_repository.permalink_exists(sample_entity.permalink) is True # pyright: ignore [reportArgumentType]
|
||||
|
||||
# Non-existent permalink should return False
|
||||
assert await entity_repository.permalink_exists("nonexistent/permalink") is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_permalink_exists_project_isolation(
|
||||
entity_repository: EntityRepository, session_maker
|
||||
):
|
||||
"""Test that permalink_exists respects project isolation."""
|
||||
async with db.scoped_session(session_maker) as session:
|
||||
# Create entity in repository's project
|
||||
entity1 = Entity(
|
||||
project_id=entity_repository.project_id,
|
||||
title="Project 1 Entity",
|
||||
entity_type="test",
|
||||
permalink="test/entity1",
|
||||
file_path="test/entity1.md",
|
||||
content_type="text/markdown",
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
)
|
||||
session.add(entity1)
|
||||
|
||||
# Create a second project with same permalink
|
||||
project2 = Project(name="other-project", path="/tmp/other")
|
||||
session.add(project2)
|
||||
await session.flush()
|
||||
|
||||
entity2 = Entity(
|
||||
project_id=project2.id,
|
||||
title="Project 2 Entity",
|
||||
entity_type="test",
|
||||
permalink="test/entity2",
|
||||
file_path="test/entity2.md",
|
||||
content_type="text/markdown",
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
)
|
||||
session.add(entity2)
|
||||
|
||||
# Should find entity1's permalink in project 1
|
||||
assert await entity_repository.permalink_exists("test/entity1") is True
|
||||
|
||||
# Should NOT find entity2's permalink (it's in project 2)
|
||||
assert await entity_repository.permalink_exists("test/entity2") is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_file_path_for_permalink(
|
||||
entity_repository: EntityRepository, sample_entity: Entity
|
||||
):
|
||||
"""Test getting file_path for a permalink without loading full entity."""
|
||||
# Existing permalink should return file_path
|
||||
file_path = await entity_repository.get_file_path_for_permalink(sample_entity.permalink) # pyright: ignore [reportArgumentType]
|
||||
assert file_path == sample_entity.file_path
|
||||
|
||||
# Non-existent permalink should return None
|
||||
result = await entity_repository.get_file_path_for_permalink("nonexistent/permalink")
|
||||
assert result is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_permalink_for_file_path(
|
||||
entity_repository: EntityRepository, sample_entity: Entity
|
||||
):
|
||||
"""Test getting permalink for a file_path without loading full entity."""
|
||||
# Existing file_path should return permalink
|
||||
permalink = await entity_repository.get_permalink_for_file_path(sample_entity.file_path)
|
||||
assert permalink == sample_entity.permalink
|
||||
|
||||
# Non-existent file_path should return None
|
||||
result = await entity_repository.get_permalink_for_file_path("nonexistent/path.md")
|
||||
assert result is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_all_permalinks(entity_repository: EntityRepository, session_maker):
|
||||
"""Test getting all permalinks without loading full entities."""
|
||||
async with db.scoped_session(session_maker) as session:
|
||||
entity1 = Entity(
|
||||
project_id=entity_repository.project_id,
|
||||
title="Entity 1",
|
||||
entity_type="test",
|
||||
permalink="test/entity1",
|
||||
file_path="test/entity1.md",
|
||||
content_type="text/markdown",
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
)
|
||||
entity2 = Entity(
|
||||
project_id=entity_repository.project_id,
|
||||
title="Entity 2",
|
||||
entity_type="test",
|
||||
permalink="test/entity2",
|
||||
file_path="test/entity2.md",
|
||||
content_type="text/markdown",
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
)
|
||||
session.add_all([entity1, entity2])
|
||||
|
||||
permalinks = await entity_repository.get_all_permalinks()
|
||||
|
||||
assert len(permalinks) == 2
|
||||
assert set(permalinks) == {"test/entity1", "test/entity2"}
|
||||
|
||||
# Results should be strings, not entities
|
||||
for permalink in permalinks:
|
||||
assert isinstance(permalink, str)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_permalink_to_file_path_map(entity_repository: EntityRepository, session_maker):
|
||||
"""Test getting permalink -> file_path mapping for bulk operations."""
|
||||
async with db.scoped_session(session_maker) as session:
|
||||
entity1 = Entity(
|
||||
project_id=entity_repository.project_id,
|
||||
title="Entity 1",
|
||||
entity_type="test",
|
||||
permalink="test/entity1",
|
||||
file_path="test/entity1.md",
|
||||
content_type="text/markdown",
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
)
|
||||
entity2 = Entity(
|
||||
project_id=entity_repository.project_id,
|
||||
title="Entity 2",
|
||||
entity_type="test",
|
||||
permalink="test/entity2",
|
||||
file_path="test/entity2.md",
|
||||
content_type="text/markdown",
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
)
|
||||
session.add_all([entity1, entity2])
|
||||
|
||||
mapping = await entity_repository.get_permalink_to_file_path_map()
|
||||
|
||||
assert len(mapping) == 2
|
||||
assert mapping["test/entity1"] == "test/entity1.md"
|
||||
assert mapping["test/entity2"] == "test/entity2.md"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_file_path_to_permalink_map(entity_repository: EntityRepository, session_maker):
|
||||
"""Test getting file_path -> permalink mapping for bulk operations."""
|
||||
async with db.scoped_session(session_maker) as session:
|
||||
entity1 = Entity(
|
||||
project_id=entity_repository.project_id,
|
||||
title="Entity 1",
|
||||
entity_type="test",
|
||||
permalink="test/entity1",
|
||||
file_path="test/entity1.md",
|
||||
content_type="text/markdown",
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
)
|
||||
entity2 = Entity(
|
||||
project_id=entity_repository.project_id,
|
||||
title="Entity 2",
|
||||
entity_type="test",
|
||||
permalink="test/entity2",
|
||||
file_path="test/entity2.md",
|
||||
content_type="text/markdown",
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
)
|
||||
session.add_all([entity1, entity2])
|
||||
|
||||
mapping = await entity_repository.get_file_path_to_permalink_map()
|
||||
|
||||
assert len(mapping) == 2
|
||||
assert mapping["test/entity1.md"] == "test/entity1"
|
||||
assert mapping["test/entity2.md"] == "test/entity2"
|
||||
|
||||
@@ -28,7 +28,6 @@ async def test_upsert_entity_with_observations_conflict(entity_repository: Entit
|
||||
|
||||
# Add observations to the entity
|
||||
obs1 = Observation(
|
||||
project_id=entity_repository.project_id,
|
||||
content="This is a test observation",
|
||||
category="testing",
|
||||
tags=["test"],
|
||||
@@ -57,13 +56,11 @@ async def test_upsert_entity_with_observations_conflict(entity_repository: Entit
|
||||
|
||||
# Add different observations
|
||||
obs2 = Observation(
|
||||
project_id=entity_repository.project_id,
|
||||
content="This is an updated observation",
|
||||
category="updated",
|
||||
tags=["updated"],
|
||||
)
|
||||
obs3 = Observation(
|
||||
project_id=entity_repository.project_id,
|
||||
content="This is a second observation",
|
||||
category="second",
|
||||
tags=["second"],
|
||||
|
||||
@@ -22,7 +22,6 @@ async def repo(observation_repository):
|
||||
async def sample_observation(repo, sample_entity: Entity):
|
||||
"""Create a sample observation for testing"""
|
||||
observation_data = {
|
||||
"project_id": sample_entity.project_id,
|
||||
"entity_id": sample_entity.id,
|
||||
"content": "Test observation",
|
||||
"context": "test-context",
|
||||
@@ -36,7 +35,6 @@ async def test_create_observation(
|
||||
):
|
||||
"""Test creating a new observation"""
|
||||
observation_data = {
|
||||
"project_id": sample_entity.project_id,
|
||||
"entity_id": sample_entity.id,
|
||||
"content": "Test content",
|
||||
"context": "test-context",
|
||||
@@ -54,7 +52,6 @@ async def test_create_observation_entity_does_not_exist(
|
||||
):
|
||||
"""Test creating a new observation"""
|
||||
observation_data = {
|
||||
"project_id": sample_entity.project_id,
|
||||
"entity_id": 99999, # Non-existent entity ID (integer for Postgres compatibility)
|
||||
"content": "Test content",
|
||||
"context": "test-context",
|
||||
@@ -107,12 +104,10 @@ async def test_delete_observations(session_maker: async_sessionmaker, repo, test
|
||||
|
||||
# Create test observations
|
||||
obs1 = Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content="Test observation 1",
|
||||
)
|
||||
obs2 = Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content="Test observation 2",
|
||||
)
|
||||
@@ -149,7 +144,6 @@ async def test_delete_observation_by_id(
|
||||
|
||||
# Create test observation
|
||||
obs = Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content="Test observation",
|
||||
)
|
||||
@@ -186,12 +180,10 @@ async def test_delete_observation_by_content(
|
||||
|
||||
# Create test observations
|
||||
obs1 = Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content="Delete this observation",
|
||||
)
|
||||
obs2 = Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content="Keep this observation",
|
||||
)
|
||||
@@ -228,19 +220,16 @@ async def test_find_by_category(session_maker: async_sessionmaker, repo, test_pr
|
||||
# Create test observations with different categories
|
||||
observations = [
|
||||
Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content="Tech observation",
|
||||
category="tech",
|
||||
),
|
||||
Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content="Design observation",
|
||||
category="design",
|
||||
),
|
||||
Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content="Another tech observation",
|
||||
category="tech",
|
||||
@@ -289,25 +278,21 @@ async def test_observation_categories(
|
||||
# Create observations with various categories
|
||||
observations = [
|
||||
Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content="First tech note",
|
||||
category="tech",
|
||||
),
|
||||
Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content="Second tech note",
|
||||
category="tech", # Duplicate category
|
||||
),
|
||||
Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content="Design note",
|
||||
category="design",
|
||||
),
|
||||
Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content="Feature note",
|
||||
category="feature",
|
||||
@@ -356,7 +341,6 @@ async def test_find_by_category_case_sensitivity(
|
||||
|
||||
# Create a test observation
|
||||
obs = Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content="Tech note",
|
||||
category="tech", # lowercase in database
|
||||
@@ -372,97 +356,3 @@ async def test_find_by_category_case_sensitivity(
|
||||
|
||||
upper_case = await repo.find_by_category("TECH")
|
||||
assert len(upper_case) == 0 # Currently case-sensitive
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_observation_permalink_truncates_long_content(
|
||||
session_maker: async_sessionmaker, repo, test_project: Project
|
||||
):
|
||||
"""Test that observation permalinks truncate long content.
|
||||
|
||||
This test validates the fix for issue #446 where:
|
||||
- Long observation content (like transcript dialogue) created permalinks
|
||||
exceeding PostgreSQL's btree index limit of 2704 bytes.
|
||||
- Content is now truncated to 200 chars in the permalink property.
|
||||
"""
|
||||
async with db.scoped_session(session_maker) as session:
|
||||
entity = Entity(
|
||||
project_id=test_project.id,
|
||||
title="test_entity",
|
||||
entity_type="test",
|
||||
permalink="test/test-entity",
|
||||
file_path="test/test_entity.md",
|
||||
content_type="text/markdown",
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
)
|
||||
session.add(entity)
|
||||
await session.flush()
|
||||
|
||||
# Create observation with very long content (5000+ chars to simulate transcript)
|
||||
long_content = "A" * 5000 # Well over the 200 char limit
|
||||
obs = Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content=long_content,
|
||||
category="transcript",
|
||||
)
|
||||
session.add(obs)
|
||||
await session.flush()
|
||||
|
||||
# Access the permalink property
|
||||
permalink = obs.permalink
|
||||
|
||||
# The full content would create a permalink like:
|
||||
# test/test-entity/observations/transcript/AAAA...5000 chars
|
||||
# With truncation, it should be much shorter
|
||||
|
||||
# Content portion should be truncated to 200 chars
|
||||
# Permalink format: entity_permalink/observations/category/content
|
||||
assert len(permalink) < 300 # Should be well under 300 chars total
|
||||
assert len(long_content[:200]) == 200 # Verify truncation length
|
||||
|
||||
# Verify the permalink contains expected parts
|
||||
assert "test/test-entity" in permalink or "test-entity" in permalink
|
||||
assert "observations" in permalink
|
||||
assert "transcript" in permalink
|
||||
|
||||
# Full 5000-char content should NOT be in permalink
|
||||
assert long_content not in permalink
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_observation_permalink_short_content_unchanged(
|
||||
session_maker: async_sessionmaker, repo, test_project: Project
|
||||
):
|
||||
"""Test that short observation content is not unnecessarily truncated."""
|
||||
async with db.scoped_session(session_maker) as session:
|
||||
entity = Entity(
|
||||
project_id=test_project.id,
|
||||
title="test_entity",
|
||||
entity_type="test",
|
||||
permalink="test/test-entity",
|
||||
file_path="test/test_entity.md",
|
||||
content_type="text/markdown",
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
)
|
||||
session.add(entity)
|
||||
await session.flush()
|
||||
|
||||
# Create observation with short content
|
||||
short_content = "Short observation content"
|
||||
obs = Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content=short_content,
|
||||
category="note",
|
||||
)
|
||||
session.add(obs)
|
||||
await session.flush()
|
||||
|
||||
permalink = obs.permalink
|
||||
|
||||
# Short content should be fully included (after permalink normalization)
|
||||
# The generate_permalink function normalizes the content
|
||||
assert "short-observation-content" in permalink.lower()
|
||||
|
||||
@@ -50,18 +50,16 @@ async def target_entity(session_maker, test_project: Project):
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def test_relations(session_maker, source_entity, target_entity, test_project: Project):
|
||||
async def test_relations(session_maker, source_entity, target_entity):
|
||||
"""Create test relations."""
|
||||
relations = [
|
||||
Relation(
|
||||
project_id=test_project.id,
|
||||
from_id=source_entity.id,
|
||||
to_id=target_entity.id,
|
||||
to_name=target_entity.title,
|
||||
relation_type="connects_to",
|
||||
),
|
||||
Relation(
|
||||
project_id=test_project.id,
|
||||
from_id=source_entity.id,
|
||||
to_id=target_entity.id,
|
||||
to_name=target_entity.title,
|
||||
@@ -351,156 +349,3 @@ async def test_delete_nonexistent_relation(relation_repository):
|
||||
"""Test deleting a relation that doesn't exist."""
|
||||
result = await relation_repository.delete_by_fields(relation_type="nonexistent")
|
||||
assert result is False
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Tests for add_all_ignore_duplicates
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_add_all_ignore_duplicates_basic(
|
||||
relation_repository: RelationRepository, sample_entity: Entity, related_entity: Entity
|
||||
):
|
||||
"""Test bulk inserting relations with ON CONFLICT DO NOTHING."""
|
||||
relations = [
|
||||
Relation(
|
||||
from_id=sample_entity.id,
|
||||
to_id=related_entity.id,
|
||||
to_name=related_entity.title,
|
||||
relation_type="links_to",
|
||||
),
|
||||
Relation(
|
||||
from_id=sample_entity.id,
|
||||
to_id=related_entity.id,
|
||||
to_name=related_entity.title,
|
||||
relation_type="references",
|
||||
),
|
||||
]
|
||||
|
||||
inserted = await relation_repository.add_all_ignore_duplicates(relations)
|
||||
|
||||
# Both should be inserted
|
||||
assert inserted == 2
|
||||
|
||||
# Verify they exist
|
||||
found = await relation_repository.find_by_entities(sample_entity.id, related_entity.id)
|
||||
assert len(found) == 2
|
||||
relation_types = {r.relation_type for r in found}
|
||||
assert relation_types == {"links_to", "references"}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_add_all_ignore_duplicates_skips_duplicates(
|
||||
relation_repository: RelationRepository, sample_entity: Entity, related_entity: Entity
|
||||
):
|
||||
"""Test that duplicate relations are silently ignored."""
|
||||
# Same relation appearing multiple times (common when same [[link]] appears twice in doc)
|
||||
relations = [
|
||||
Relation(
|
||||
from_id=sample_entity.id,
|
||||
to_id=None, # Unresolved
|
||||
to_name="Some Target",
|
||||
relation_type="links_to",
|
||||
),
|
||||
Relation(
|
||||
from_id=sample_entity.id,
|
||||
to_id=None,
|
||||
to_name="Some Target", # Duplicate!
|
||||
relation_type="links_to",
|
||||
),
|
||||
Relation(
|
||||
from_id=sample_entity.id,
|
||||
to_id=None,
|
||||
to_name="Some Target", # Triple duplicate!
|
||||
relation_type="links_to",
|
||||
),
|
||||
]
|
||||
|
||||
inserted = await relation_repository.add_all_ignore_duplicates(relations)
|
||||
|
||||
# Only 1 should be inserted (duplicates ignored)
|
||||
assert inserted == 1
|
||||
|
||||
# Verify only one exists
|
||||
all_relations = await relation_repository.find_all()
|
||||
matching = [r for r in all_relations if r.to_name == "Some Target"]
|
||||
assert len(matching) == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_add_all_ignore_duplicates_empty_list(relation_repository: RelationRepository):
|
||||
"""Test with empty list returns 0."""
|
||||
inserted = await relation_repository.add_all_ignore_duplicates([])
|
||||
assert inserted == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_add_all_ignore_duplicates_mixed(
|
||||
relation_repository: RelationRepository, sample_entity: Entity, related_entity: Entity
|
||||
):
|
||||
"""Test with mix of new and duplicate relations."""
|
||||
# First, insert one relation
|
||||
first_relation = Relation(
|
||||
from_id=sample_entity.id,
|
||||
to_id=None,
|
||||
to_name="Existing Target",
|
||||
relation_type="links_to",
|
||||
)
|
||||
await relation_repository.add_all_ignore_duplicates([first_relation])
|
||||
|
||||
# Now try to insert a mix of new and duplicate
|
||||
relations = [
|
||||
Relation(
|
||||
from_id=sample_entity.id,
|
||||
to_id=None,
|
||||
to_name="Existing Target", # Duplicate of first_relation
|
||||
relation_type="links_to",
|
||||
),
|
||||
Relation(
|
||||
from_id=sample_entity.id,
|
||||
to_id=None,
|
||||
to_name="New Target 1", # New
|
||||
relation_type="links_to",
|
||||
),
|
||||
Relation(
|
||||
from_id=sample_entity.id,
|
||||
to_id=None,
|
||||
to_name="New Target 2", # New
|
||||
relation_type="references",
|
||||
),
|
||||
]
|
||||
|
||||
inserted = await relation_repository.add_all_ignore_duplicates(relations)
|
||||
|
||||
# Only 2 new ones should be inserted
|
||||
assert inserted == 2
|
||||
|
||||
# Verify total count
|
||||
all_relations = await relation_repository.find_all()
|
||||
from_sample = [r for r in all_relations if r.from_id == sample_entity.id]
|
||||
assert len(from_sample) == 3 # 1 existing + 2 new
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_add_all_ignore_duplicates_with_context(
|
||||
relation_repository: RelationRepository, sample_entity: Entity, related_entity: Entity
|
||||
):
|
||||
"""Test that context field is properly inserted."""
|
||||
relations = [
|
||||
Relation(
|
||||
from_id=sample_entity.id,
|
||||
to_id=related_entity.id,
|
||||
to_name=related_entity.title,
|
||||
relation_type="links_to",
|
||||
context="some context here",
|
||||
),
|
||||
]
|
||||
|
||||
inserted = await relation_repository.add_all_ignore_duplicates(relations)
|
||||
assert inserted == 1
|
||||
|
||||
# Verify context was saved
|
||||
found = await relation_repository.find_by_entities(sample_entity.id, related_entity.id)
|
||||
assert len(found) == 1
|
||||
assert found[0].context == "some context here"
|
||||
|
||||
@@ -286,7 +286,6 @@ async def test_project_isolation_in_find_related(session_maker, app_config):
|
||||
|
||||
# Create relation in project1 (between entities of project1)
|
||||
relation_p1 = Relation(
|
||||
project_id=project1.id,
|
||||
from_id=entity1_p1.id,
|
||||
to_id=entity2_p1.id,
|
||||
to_name="Entity2_P1",
|
||||
|
||||
@@ -432,14 +432,14 @@ async def test_create_with_content(entity_service: EntityService, file_service:
|
||||
assert entity.observations[0].context == "Reduces merge conflicts"
|
||||
|
||||
assert len(entity.relations) == 4
|
||||
assert entity.relations[0].relation_type == "links_to"
|
||||
assert entity.relations[0].relation_type == "links to"
|
||||
assert entity.relations[0].to_name == "Git"
|
||||
assert entity.relations[1].relation_type == "links_to"
|
||||
assert entity.relations[1].relation_type == "links to"
|
||||
assert entity.relations[1].to_name == "Trunk Based Development"
|
||||
assert entity.relations[2].relation_type == "implements"
|
||||
assert entity.relations[2].to_name == "Branch Strategy"
|
||||
assert entity.relations[2].context == "Our standard workflow"
|
||||
assert entity.relations[3].relation_type == "links_to"
|
||||
assert entity.relations[3].relation_type == "links to"
|
||||
assert entity.relations[3].to_name == "Git Cheat Sheet"
|
||||
|
||||
# Verify file has new content but preserved metadata
|
||||
@@ -557,14 +557,14 @@ async def test_update_with_content(entity_service: EntityService, file_service:
|
||||
assert entity.observations[0].context == "Reduces merge conflicts"
|
||||
|
||||
assert len(entity.relations) == 4
|
||||
assert entity.relations[0].relation_type == "links_to"
|
||||
assert entity.relations[0].relation_type == "links to"
|
||||
assert entity.relations[0].to_name == "Git"
|
||||
assert entity.relations[1].relation_type == "links_to"
|
||||
assert entity.relations[1].relation_type == "links to"
|
||||
assert entity.relations[1].to_name == "Trunk Based Development"
|
||||
assert entity.relations[2].relation_type == "implements"
|
||||
assert entity.relations[2].to_name == "Branch Strategy"
|
||||
assert entity.relations[2].context == "Our standard workflow"
|
||||
assert entity.relations[3].relation_type == "links_to"
|
||||
assert entity.relations[3].relation_type == "links to"
|
||||
assert entity.relations[3].to_name == "Git Cheat Sheet"
|
||||
|
||||
# Verify file has new content but preserved metadata
|
||||
@@ -1772,7 +1772,7 @@ async def test_move_entity_with_complex_observations(
|
||||
# Check relations
|
||||
relation_types = {rel.relation_type for rel in moved_entity.relations}
|
||||
assert "implements" in relation_types
|
||||
assert "links_to" in relation_types
|
||||
assert "links to" in relation_types
|
||||
|
||||
relation_targets = {rel.to_name for rel in moved_entity.relations}
|
||||
assert "Branch Strategy" in relation_targets
|
||||
|
||||
@@ -162,77 +162,3 @@ async def test_write_unicode_content(tmp_path: Path, file_service: FileService):
|
||||
content, _ = await file_service.read_file(test_path)
|
||||
|
||||
assert content == test_content
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_read_file_content(tmp_path: Path, file_service: FileService):
|
||||
"""Test read_file_content returns just the content without checksum."""
|
||||
test_path = tmp_path / "test.md"
|
||||
test_content = "test content\nwith multiple lines"
|
||||
|
||||
# Write file
|
||||
await file_service.write_file(test_path, test_content)
|
||||
|
||||
# Read content only
|
||||
content = await file_service.read_file_content(test_path)
|
||||
assert content == test_content
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_read_file_content_missing_file(tmp_path: Path, file_service: FileService):
|
||||
"""Test read_file_content raises error for missing files."""
|
||||
test_path = tmp_path / "missing.md"
|
||||
|
||||
with pytest.raises(FileOperationError):
|
||||
await file_service.read_file_content(test_path)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_read_file_bytes(tmp_path: Path, file_service: FileService):
|
||||
"""Test read_file_bytes for binary file reading."""
|
||||
test_path = tmp_path / "test.bin"
|
||||
# Create binary content with non-UTF8 bytes
|
||||
binary_content = b"\x00\x01\x02\x03\xff\xfe\xfd"
|
||||
|
||||
# Write binary file directly
|
||||
test_path.write_bytes(binary_content)
|
||||
|
||||
# Read back using read_file_bytes
|
||||
content = await file_service.read_file_bytes(test_path)
|
||||
assert content == binary_content
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_read_file_bytes_image(tmp_path: Path, file_service: FileService):
|
||||
"""Test read_file_bytes with image-like binary content."""
|
||||
test_path = tmp_path / "test.png"
|
||||
# PNG header signature
|
||||
png_header = b"\x89PNG\r\n\x1a\n"
|
||||
fake_image_content = png_header + b"\x00" * 100
|
||||
|
||||
test_path.write_bytes(fake_image_content)
|
||||
|
||||
content = await file_service.read_file_bytes(test_path)
|
||||
assert content == fake_image_content
|
||||
assert content.startswith(png_header)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_read_file_bytes_missing_file(tmp_path: Path, file_service: FileService):
|
||||
"""Test read_file_bytes raises error for missing files."""
|
||||
test_path = tmp_path / "missing.bin"
|
||||
|
||||
with pytest.raises(FileOperationError):
|
||||
await file_service.read_file_bytes(test_path)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_read_file_bytes_text_file(tmp_path: Path, file_service: FileService):
|
||||
"""Test read_file_bytes can read text files as bytes."""
|
||||
test_path = tmp_path / "test.txt"
|
||||
text_content = "Hello, World!"
|
||||
|
||||
test_path.write_text(text_content)
|
||||
|
||||
content = await file_service.read_file_bytes(test_path)
|
||||
assert content == text_content.encode("utf-8")
|
||||
|
||||
@@ -178,71 +178,3 @@ async def test_initialize_file_sync_background_tasks(
|
||||
|
||||
# Watch service should still be started
|
||||
mock_watch_service.run.assert_called_once()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@patch("basic_memory.services.initialization.db.get_or_create_db")
|
||||
@patch("basic_memory.sync.sync_service.get_sync_service")
|
||||
@patch("basic_memory.sync.WatchService")
|
||||
@patch("basic_memory.services.initialization.asyncio.create_task")
|
||||
@patch.dict("os.environ", {"BASIC_MEMORY_MCP_PROJECT": "project1"})
|
||||
async def test_initialize_file_sync_respects_project_constraint(
|
||||
mock_create_task, mock_watch_service_class, mock_get_sync_service, mock_get_db, app_config
|
||||
):
|
||||
"""Test that file sync only syncs the constrained project when BASIC_MEMORY_MCP_PROJECT is set."""
|
||||
# Setup mocks
|
||||
mock_session_maker = AsyncMock()
|
||||
mock_get_db.return_value = (None, mock_session_maker)
|
||||
|
||||
mock_watch_service = AsyncMock()
|
||||
mock_watch_service.run = AsyncMock()
|
||||
mock_watch_service_class.return_value = mock_watch_service
|
||||
|
||||
mock_repository = AsyncMock()
|
||||
mock_project1 = MagicMock()
|
||||
mock_project1.name = "project1"
|
||||
mock_project1.path = "/path/to/project1"
|
||||
mock_project1.id = 1
|
||||
|
||||
mock_project2 = MagicMock()
|
||||
mock_project2.name = "project2"
|
||||
mock_project2.path = "/path/to/project2"
|
||||
mock_project2.id = 2
|
||||
|
||||
mock_project3 = MagicMock()
|
||||
mock_project3.name = "project3"
|
||||
mock_project3.path = "/path/to/project3"
|
||||
mock_project3.id = 3
|
||||
|
||||
mock_sync_service = AsyncMock()
|
||||
mock_sync_service.sync = AsyncMock()
|
||||
mock_get_sync_service.return_value = mock_sync_service
|
||||
|
||||
# Mock background tasks
|
||||
mock_task = MagicMock()
|
||||
mock_create_task.return_value = mock_task
|
||||
|
||||
# Mock the repository
|
||||
with patch("basic_memory.services.initialization.ProjectRepository") as mock_repo_class:
|
||||
mock_repo_class.return_value = mock_repository
|
||||
# Return all 3 projects from get_active_projects
|
||||
mock_repository.get_active_projects.return_value = [
|
||||
mock_project1,
|
||||
mock_project2,
|
||||
mock_project3,
|
||||
]
|
||||
|
||||
# Run the function
|
||||
result = await initialize_file_sync(app_config)
|
||||
|
||||
# Assertions
|
||||
mock_repository.get_active_projects.assert_called_once()
|
||||
|
||||
# Should only create 1 background task for project1 (the constrained project)
|
||||
assert mock_create_task.call_count == 1
|
||||
|
||||
# Verify the function returns None
|
||||
assert result is None
|
||||
|
||||
# Watch service should still be started
|
||||
mock_watch_service.run.assert_called_once()
|
||||
|
||||
@@ -890,6 +890,11 @@ async def test_add_project_without_project_root_allows_arbitrary_paths(
|
||||
if "BASIC_MEMORY_PROJECT_ROOT" in os.environ:
|
||||
monkeypatch.delenv("BASIC_MEMORY_PROJECT_ROOT")
|
||||
|
||||
# Force reload config without project_root
|
||||
from basic_memory.services import project_service as ps_module
|
||||
|
||||
monkeypatch.setattr(ps_module, "config", config_manager.load_config())
|
||||
|
||||
# Create a test directory
|
||||
test_dir = Path(temp_dir) / "arbitrary-location"
|
||||
test_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
@@ -755,182 +755,3 @@ async def test_search_title_via_repository_direct(search_service, session_maker,
|
||||
# Should find the entity without throwing FTS5 syntax errors
|
||||
assert len(results) >= 1
|
||||
assert any(result.title == "Note (with parentheses)" for result in results)
|
||||
|
||||
|
||||
# Tests for duplicate observation permalink deduplication
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_index_entity_with_duplicate_observations(
|
||||
search_service, session_maker, test_project
|
||||
):
|
||||
"""Test that indexing an entity with duplicate observations doesn't cause unique constraint violations.
|
||||
|
||||
Two observations with the same category and content generate identical permalinks,
|
||||
which would violate the unique constraint on the search_index table.
|
||||
"""
|
||||
from basic_memory.repository import EntityRepository, ObservationRepository
|
||||
from unittest.mock import AsyncMock
|
||||
from datetime import datetime
|
||||
|
||||
entity_repo = EntityRepository(session_maker, project_id=test_project.id)
|
||||
obs_repo = ObservationRepository(session_maker, project_id=test_project.id)
|
||||
|
||||
# Create entity
|
||||
entity_data = {
|
||||
"title": "Entity With Duplicate Observations",
|
||||
"entity_type": "note",
|
||||
"entity_metadata": {},
|
||||
"content_type": "text/markdown",
|
||||
"file_path": "test/duplicate-obs.md",
|
||||
"permalink": "test/duplicate-obs",
|
||||
"project_id": test_project.id,
|
||||
"created_at": datetime.now(),
|
||||
"updated_at": datetime.now(),
|
||||
}
|
||||
|
||||
entity = await entity_repo.create(entity_data)
|
||||
|
||||
# Create duplicate observations - same category and content
|
||||
duplicate_content = "This is a duplicated observation"
|
||||
await obs_repo.create(
|
||||
{"entity_id": entity.id, "category": "note", "content": duplicate_content}
|
||||
)
|
||||
await obs_repo.create(
|
||||
{"entity_id": entity.id, "category": "note", "content": duplicate_content}
|
||||
)
|
||||
|
||||
# Reload entity with observations (get_by_permalink eagerly loads observations)
|
||||
entity = await entity_repo.get_by_permalink("test/duplicate-obs")
|
||||
|
||||
# Verify we have duplicate observations
|
||||
assert len(entity.observations) == 2
|
||||
assert entity.observations[0].permalink == entity.observations[1].permalink
|
||||
|
||||
# Mock file service to avoid file I/O
|
||||
search_service.file_service.read_entity_content = AsyncMock(return_value="")
|
||||
|
||||
# This should not raise a unique constraint violation
|
||||
await search_service.index_entity(entity)
|
||||
|
||||
# Verify entity is searchable
|
||||
results = await search_service.search(SearchQuery(text="Duplicate Observations"))
|
||||
assert len(results) >= 1
|
||||
assert any(r.title == "Entity With Duplicate Observations" for r in results)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_index_entity_dedupes_observations_by_permalink(
|
||||
search_service, session_maker, test_project
|
||||
):
|
||||
"""Test that only unique observation permalinks are indexed.
|
||||
|
||||
When an entity has observations with identical permalinks, only the first one
|
||||
should be indexed to avoid unique constraint violations.
|
||||
"""
|
||||
from basic_memory.repository import EntityRepository, ObservationRepository
|
||||
from unittest.mock import AsyncMock
|
||||
from datetime import datetime
|
||||
|
||||
entity_repo = EntityRepository(session_maker, project_id=test_project.id)
|
||||
obs_repo = ObservationRepository(session_maker, project_id=test_project.id)
|
||||
|
||||
# Create entity
|
||||
entity_data = {
|
||||
"title": "Dedupe Test Entity",
|
||||
"entity_type": "note",
|
||||
"entity_metadata": {},
|
||||
"content_type": "text/markdown",
|
||||
"file_path": "test/dedupe-test.md",
|
||||
"permalink": "test/dedupe-test",
|
||||
"project_id": test_project.id,
|
||||
"created_at": datetime.now(),
|
||||
"updated_at": datetime.now(),
|
||||
}
|
||||
|
||||
entity = await entity_repo.create(entity_data)
|
||||
|
||||
# Create three observations: two duplicates and one unique
|
||||
duplicate_content = "Duplicate observation content"
|
||||
unique_content = "Unique observation content"
|
||||
|
||||
await obs_repo.create(
|
||||
{"entity_id": entity.id, "category": "note", "content": duplicate_content}
|
||||
)
|
||||
await obs_repo.create(
|
||||
{"entity_id": entity.id, "category": "note", "content": duplicate_content}
|
||||
)
|
||||
await obs_repo.create({"entity_id": entity.id, "category": "note", "content": unique_content})
|
||||
|
||||
# Reload entity with observations (get_by_permalink eagerly loads observations)
|
||||
entity = await entity_repo.get_by_permalink("test/dedupe-test")
|
||||
assert len(entity.observations) == 3
|
||||
|
||||
# Mock file service to avoid file I/O
|
||||
search_service.file_service.read_entity_content = AsyncMock(return_value="")
|
||||
|
||||
# Index the entity
|
||||
await search_service.index_entity(entity)
|
||||
|
||||
# Search for the unique observation - should find it
|
||||
results = await search_service.search(SearchQuery(text="Unique observation"))
|
||||
assert len(results) >= 1
|
||||
|
||||
# Search for duplicate observation - should find it (only one indexed)
|
||||
results = await search_service.search(SearchQuery(text="Duplicate observation"))
|
||||
assert len(results) >= 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_index_entity_multiple_categories_same_content(
|
||||
search_service, session_maker, test_project
|
||||
):
|
||||
"""Test that observations with same content but different categories are not deduped.
|
||||
|
||||
The permalink includes the category, so observations with different categories
|
||||
but same content should have different permalinks and both be indexed.
|
||||
"""
|
||||
from basic_memory.repository import EntityRepository, ObservationRepository
|
||||
from unittest.mock import AsyncMock
|
||||
from datetime import datetime
|
||||
|
||||
entity_repo = EntityRepository(session_maker, project_id=test_project.id)
|
||||
obs_repo = ObservationRepository(session_maker, project_id=test_project.id)
|
||||
|
||||
# Create entity
|
||||
entity_data = {
|
||||
"title": "Multi Category Entity",
|
||||
"entity_type": "note",
|
||||
"entity_metadata": {},
|
||||
"content_type": "text/markdown",
|
||||
"file_path": "test/multi-category.md",
|
||||
"permalink": "test/multi-category",
|
||||
"project_id": test_project.id,
|
||||
"created_at": datetime.now(),
|
||||
"updated_at": datetime.now(),
|
||||
}
|
||||
|
||||
entity = await entity_repo.create(entity_data)
|
||||
|
||||
# Create observations with same content but different categories
|
||||
shared_content = "Shared content across categories"
|
||||
await obs_repo.create({"entity_id": entity.id, "category": "tech", "content": shared_content})
|
||||
await obs_repo.create({"entity_id": entity.id, "category": "design", "content": shared_content})
|
||||
|
||||
# Reload entity with observations (get_by_permalink eagerly loads observations)
|
||||
entity = await entity_repo.get_by_permalink("test/multi-category")
|
||||
assert len(entity.observations) == 2
|
||||
|
||||
# Verify permalinks are different due to different categories
|
||||
permalinks = {obs.permalink for obs in entity.observations}
|
||||
assert len(permalinks) == 2 # Should be 2 unique permalinks
|
||||
|
||||
# Mock file service to avoid file I/O
|
||||
search_service.file_service.read_entity_content = AsyncMock(return_value="")
|
||||
|
||||
# Index the entity - both should be indexed since permalinks differ
|
||||
await search_service.index_entity(entity)
|
||||
|
||||
# Search for the shared content - should find both observations
|
||||
results = await search_service.search(SearchQuery(text="Shared content"))
|
||||
assert len(results) >= 2
|
||||
|
||||
@@ -106,89 +106,6 @@ Target content
|
||||
assert source.relations[0].to_name == target.title
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_resolve_relations_deletes_duplicate_unresolved_relation(
|
||||
sync_service: SyncService,
|
||||
project_config: ProjectConfig,
|
||||
entity_service: EntityService,
|
||||
):
|
||||
"""Test that resolve_relations deletes duplicate unresolved relations on IntegrityError.
|
||||
|
||||
When resolving a forward reference would create a duplicate (from_id, to_id, relation_type),
|
||||
the unresolved relation should be deleted since a resolved version already exists.
|
||||
"""
|
||||
from unittest.mock import patch
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from basic_memory.models import Relation
|
||||
|
||||
project_dir = project_config.home
|
||||
|
||||
# Create source entity
|
||||
source_content = """
|
||||
---
|
||||
type: knowledge
|
||||
---
|
||||
# Source Entity
|
||||
Content
|
||||
"""
|
||||
await create_test_file(project_dir / "source.md", source_content)
|
||||
|
||||
# Create target entity
|
||||
target_content = """
|
||||
---
|
||||
type: knowledge
|
||||
---
|
||||
# Target Entity
|
||||
Content
|
||||
"""
|
||||
await create_test_file(project_dir / "target.md", target_content)
|
||||
|
||||
# Sync to create both entities
|
||||
await sync_service.sync(project_config.home)
|
||||
|
||||
source = await entity_service.get_by_permalink("source")
|
||||
await entity_service.get_by_permalink("target")
|
||||
|
||||
# Create an unresolved relation that will resolve to target
|
||||
unresolved_relation = Relation(
|
||||
from_id=source.id,
|
||||
to_id=None, # Unresolved
|
||||
to_name="target", # Will resolve to target entity
|
||||
relation_type="relates_to",
|
||||
)
|
||||
await sync_service.relation_repository.add(unresolved_relation)
|
||||
unresolved_id = unresolved_relation.id
|
||||
|
||||
# Verify we have the unresolved relation
|
||||
source = await entity_service.get_by_permalink("source")
|
||||
assert len(source.outgoing_relations) == 1
|
||||
assert source.outgoing_relations[0].to_id is None
|
||||
|
||||
# Mock the repository update to raise IntegrityError (simulating existing duplicate)
|
||||
|
||||
async def mock_update_raises_integrity_error(entity_id, data):
|
||||
# Simulate: a resolved relation with same (from_id, to_id, relation_type) already exists
|
||||
raise IntegrityError(
|
||||
"UNIQUE constraint failed: relation.from_id, relation.to_id, relation.relation_type",
|
||||
None,
|
||||
None, # pyright: ignore [reportArgumentType]
|
||||
)
|
||||
|
||||
with patch.object(
|
||||
sync_service.relation_repository, "update", side_effect=mock_update_raises_integrity_error
|
||||
):
|
||||
# Call resolve_relations - should hit IntegrityError and delete the duplicate
|
||||
await sync_service.resolve_relations()
|
||||
|
||||
# Verify the unresolved relation was deleted
|
||||
deleted = await sync_service.relation_repository.find_by_id(unresolved_id)
|
||||
assert deleted is None
|
||||
|
||||
# Verify no unresolved relations remain
|
||||
unresolved = await sync_service.relation_repository.find_unresolved_relations()
|
||||
assert len(unresolved) == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_sync(
|
||||
sync_service: SyncService, project_config: ProjectConfig, entity_service: EntityService
|
||||
@@ -712,8 +629,11 @@ async def test_sync_preserves_timestamps(
|
||||
sync_service: SyncService,
|
||||
project_config: ProjectConfig,
|
||||
entity_service: EntityService,
|
||||
db_backend,
|
||||
):
|
||||
"""Test that sync preserves file timestamps and frontmatter dates."""
|
||||
if db_backend == "postgres":
|
||||
pytest.skip("Postgres timestamp handling differs from SQLite")
|
||||
project_dir = project_config.home
|
||||
|
||||
# Create a file with explicit frontmatter dates
|
||||
@@ -765,6 +685,7 @@ async def test_sync_updates_timestamps_on_file_modification(
|
||||
sync_service: SyncService,
|
||||
project_config: ProjectConfig,
|
||||
entity_service: EntityService,
|
||||
db_backend,
|
||||
):
|
||||
"""Test that sync updates entity timestamps when files are modified.
|
||||
|
||||
@@ -773,6 +694,9 @@ async def test_sync_updates_timestamps_on_file_modification(
|
||||
not the database operation time. This is critical for accurate temporal ordering in
|
||||
search and recent_activity queries.
|
||||
"""
|
||||
if db_backend == "postgres":
|
||||
pytest.skip("Postgres timestamp handling differs from SQLite")
|
||||
|
||||
project_dir = project_config.home
|
||||
|
||||
# Create initial file
|
||||
|
||||
@@ -8,7 +8,7 @@ import pytest
|
||||
from watchfiles import Change
|
||||
|
||||
from basic_memory.models.project import Project
|
||||
from basic_memory.sync.watch_service import WatchServiceState
|
||||
from basic_memory.sync.watch_service import WatchService, WatchServiceState
|
||||
|
||||
|
||||
async def create_test_file(path: Path, content: str = "test content") -> None:
|
||||
@@ -17,7 +17,10 @@ async def create_test_file(path: Path, content: str = "test content") -> None:
|
||||
path.write_text(content)
|
||||
|
||||
|
||||
# Note: watch_service fixture is defined in conftest.py with sync_service_factory
|
||||
@pytest.fixture
|
||||
def watch_service(sync_service, file_service, project_config):
|
||||
"""Create watch service instance."""
|
||||
return WatchService(sync_service, file_service, project_config)
|
||||
|
||||
|
||||
def test_watch_service_init(watch_service, project_config):
|
||||
|
||||
@@ -85,8 +85,8 @@ async def test_run_handles_no_projects():
|
||||
with patch.object(watch_service, "write_status", return_value=None):
|
||||
await watch_service.run()
|
||||
|
||||
# Should have slept for the configured reload interval when no projects found
|
||||
mock_sleep.assert_called_with(config.watch_project_reload_interval)
|
||||
# Should have slept for 30 seconds when no projects found
|
||||
mock_sleep.assert_called_with(30)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
@@ -1,96 +0,0 @@
|
||||
"""Tests for timezone utilities."""
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
|
||||
from basic_memory.utils import ensure_timezone_aware
|
||||
|
||||
|
||||
class TestEnsureTimezoneAware:
|
||||
"""Tests for ensure_timezone_aware function."""
|
||||
|
||||
def test_already_timezone_aware_returns_unchanged(self):
|
||||
"""Timezone-aware datetime should be returned unchanged."""
|
||||
dt = datetime(2024, 1, 15, 12, 30, 0, tzinfo=timezone.utc)
|
||||
result = ensure_timezone_aware(dt)
|
||||
assert result == dt
|
||||
assert result.tzinfo == timezone.utc
|
||||
|
||||
def test_naive_datetime_cloud_mode_true_interprets_as_utc(self):
|
||||
"""In cloud mode, naive datetimes should be interpreted as UTC."""
|
||||
naive_dt = datetime(2024, 1, 15, 12, 30, 0)
|
||||
result = ensure_timezone_aware(naive_dt, cloud_mode=True)
|
||||
|
||||
# Should have UTC timezone
|
||||
assert result.tzinfo == timezone.utc
|
||||
# Time values should be unchanged (just tagged as UTC)
|
||||
assert result.year == 2024
|
||||
assert result.month == 1
|
||||
assert result.day == 15
|
||||
assert result.hour == 12
|
||||
assert result.minute == 30
|
||||
|
||||
def test_naive_datetime_cloud_mode_false_interprets_as_local(self):
|
||||
"""In local mode, naive datetimes should be interpreted as local time."""
|
||||
naive_dt = datetime(2024, 1, 15, 12, 30, 0)
|
||||
result = ensure_timezone_aware(naive_dt, cloud_mode=False)
|
||||
|
||||
# Should have some timezone info (local)
|
||||
assert result.tzinfo is not None
|
||||
# The datetime should be converted to local timezone
|
||||
# We can't assert exact timezone as it depends on system
|
||||
|
||||
def test_cloud_mode_true_does_not_shift_time(self):
|
||||
"""Cloud mode should use replace() not astimezone() - time values unchanged."""
|
||||
naive_dt = datetime(2024, 6, 15, 18, 0, 0) # Summer time
|
||||
result = ensure_timezone_aware(naive_dt, cloud_mode=True)
|
||||
|
||||
# Hour should remain 18, not be shifted by timezone offset
|
||||
assert result.hour == 18
|
||||
assert result.tzinfo == timezone.utc
|
||||
|
||||
def test_explicit_cloud_mode_skips_config_loading(self):
|
||||
"""When cloud_mode is explicitly passed, config should not be loaded."""
|
||||
# This test verifies we can call ensure_timezone_aware without
|
||||
# triggering ConfigManager import when cloud_mode is explicit
|
||||
naive_dt = datetime(2024, 1, 15, 12, 30, 0)
|
||||
|
||||
# Should work without any config setup
|
||||
result_cloud = ensure_timezone_aware(naive_dt, cloud_mode=True)
|
||||
assert result_cloud.tzinfo == timezone.utc
|
||||
|
||||
result_local = ensure_timezone_aware(naive_dt, cloud_mode=False)
|
||||
assert result_local.tzinfo is not None
|
||||
|
||||
def test_none_cloud_mode_falls_back_to_config(self):
|
||||
"""When cloud_mode is None, should load from config."""
|
||||
from unittest.mock import patch, MagicMock
|
||||
|
||||
naive_dt = datetime(2024, 1, 15, 12, 30, 0)
|
||||
|
||||
# Mock ConfigManager to return cloud_mode_enabled=True
|
||||
mock_config = MagicMock()
|
||||
mock_config.cloud_mode_enabled = True
|
||||
|
||||
with patch("basic_memory.config.ConfigManager") as mock_manager:
|
||||
mock_manager.return_value.config = mock_config
|
||||
result = ensure_timezone_aware(naive_dt, cloud_mode=None)
|
||||
|
||||
# Should have used cloud mode (UTC)
|
||||
assert result.tzinfo == timezone.utc
|
||||
|
||||
def test_asyncpg_naive_utc_scenario(self):
|
||||
"""Simulate asyncpg returning naive datetime that's actually UTC.
|
||||
|
||||
asyncpg binary protocol returns timestamps in UTC but as naive datetimes.
|
||||
In cloud mode, we interpret these as UTC rather than local time.
|
||||
"""
|
||||
# Simulate what asyncpg returns: a naive datetime that's actually UTC
|
||||
asyncpg_result = datetime(2024, 1, 15, 18, 30, 0) # 6:30 PM UTC
|
||||
|
||||
# In cloud mode, interpret as UTC
|
||||
cloud_result = ensure_timezone_aware(asyncpg_result, cloud_mode=True)
|
||||
assert cloud_result == datetime(2024, 1, 15, 18, 30, 0, tzinfo=timezone.utc)
|
||||
|
||||
# The hour should remain 18, not shifted
|
||||
assert cloud_result.hour == 18
|
||||
@@ -126,19 +126,18 @@ dependencies = [
|
||||
{ name = "fastapi", extra = ["standard"] },
|
||||
{ name = "fastmcp" },
|
||||
{ name = "greenlet" },
|
||||
{ name = "logfire" },
|
||||
{ name = "loguru" },
|
||||
{ name = "markdown-it-py" },
|
||||
{ name = "mcp" },
|
||||
{ name = "nest-asyncio" },
|
||||
{ name = "pillow" },
|
||||
{ name = "psycopg" },
|
||||
{ name = "pybars3" },
|
||||
{ name = "pydantic", extra = ["email", "timezone"] },
|
||||
{ name = "pydantic-settings" },
|
||||
{ name = "pyjwt" },
|
||||
{ name = "pyright" },
|
||||
{ name = "pytest-aio" },
|
||||
{ name = "pytest-asyncio" },
|
||||
{ name = "python-dotenv" },
|
||||
{ name = "python-frontmatter" },
|
||||
{ name = "pyyaml" },
|
||||
@@ -154,14 +153,12 @@ dev = [
|
||||
{ name = "freezegun" },
|
||||
{ name = "gevent" },
|
||||
{ name = "icecream" },
|
||||
{ name = "psycopg" },
|
||||
{ name = "pytest" },
|
||||
{ name = "pytest-asyncio" },
|
||||
{ name = "pytest-cov" },
|
||||
{ name = "pytest-mock" },
|
||||
{ name = "pytest-xdist" },
|
||||
{ name = "ruff" },
|
||||
{ name = "testcontainers" },
|
||||
]
|
||||
|
||||
[package.metadata]
|
||||
@@ -172,21 +169,20 @@ requires-dist = [
|
||||
{ name = "asyncpg", specifier = ">=0.30.0" },
|
||||
{ name = "dateparser", specifier = ">=1.2.0" },
|
||||
{ name = "fastapi", extras = ["standard"], specifier = ">=0.115.8" },
|
||||
{ name = "fastmcp", specifier = "==2.12.3" },
|
||||
{ name = "fastmcp", specifier = ">=2.10.2" },
|
||||
{ name = "greenlet", specifier = ">=3.1.1" },
|
||||
{ name = "logfire", specifier = ">=0.73.0" },
|
||||
{ name = "loguru", specifier = ">=0.7.3" },
|
||||
{ name = "markdown-it-py", specifier = ">=3.0.0" },
|
||||
{ name = "mcp", specifier = ">=1.2.0" },
|
||||
{ name = "nest-asyncio", specifier = ">=1.6.0" },
|
||||
{ name = "pillow", specifier = ">=11.1.0" },
|
||||
{ name = "psycopg", specifier = "==3.3.1" },
|
||||
{ name = "pybars3", specifier = ">=0.9.7" },
|
||||
{ name = "pydantic", extras = ["email", "timezone"], specifier = ">=2.10.3" },
|
||||
{ name = "pydantic-settings", specifier = ">=2.6.1" },
|
||||
{ name = "pyjwt", specifier = ">=2.10.1" },
|
||||
{ name = "pyright", specifier = ">=1.1.390" },
|
||||
{ name = "pytest-aio", specifier = ">=1.9.0" },
|
||||
{ name = "pytest-asyncio", specifier = ">=1.2.0" },
|
||||
{ name = "python-dotenv", specifier = ">=1.1.0" },
|
||||
{ name = "python-frontmatter", specifier = ">=1.1.0" },
|
||||
{ name = "pyyaml", specifier = ">=6.0.1" },
|
||||
@@ -202,14 +198,12 @@ dev = [
|
||||
{ name = "freezegun", specifier = ">=1.5.5" },
|
||||
{ name = "gevent", specifier = ">=24.11.1" },
|
||||
{ name = "icecream", specifier = ">=2.1.3" },
|
||||
{ name = "psycopg", specifier = ">=3.2.0" },
|
||||
{ name = "pytest", specifier = ">=8.3.4" },
|
||||
{ name = "pytest-asyncio", specifier = ">=0.24.0" },
|
||||
{ name = "pytest-cov", specifier = ">=4.1.0" },
|
||||
{ name = "pytest-mock", specifier = ">=3.12.0" },
|
||||
{ name = "pytest-xdist", specifier = ">=3.0.0" },
|
||||
{ name = "ruff", specifier = ">=0.1.6" },
|
||||
{ name = "testcontainers", extras = ["postgres"], specifier = ">=4.0.0" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -455,20 +449,6 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/68/1b/e0a87d256e40e8c888847551b20a017a6b98139178505dc7ffb96f04e954/dnspython-2.7.0-py3-none-any.whl", hash = "sha256:b4c34b7d10b51bcc3a5071e7b8dee77939f1e878477eeecc965e9835f63c6c86", size = 313632, upload-time = "2024-10-05T20:14:57.687Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "docker"
|
||||
version = "7.1.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "pywin32", marker = "sys_platform == 'win32'" },
|
||||
{ name = "requests" },
|
||||
{ name = "urllib3" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/91/9b/4a2ea29aeba62471211598dac5d96825bb49348fa07e906ea930394a83ce/docker-7.1.0.tar.gz", hash = "sha256:ad8c70e6e3f8926cb8a92619b832b4ea5299e2831c14284663184e200546fa6c", size = 117834, upload-time = "2024-05-23T11:13:57.216Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/e3/26/57c6fb270950d476074c087527a558ccb6f4436657314bfb6cdf484114c4/docker-7.1.0-py3-none-any.whl", hash = "sha256:c96b93b7f0a746f9e77d325bcfb87422a3d8bd4f03136ae8a85b37f1898d5fc0", size = 147774, upload-time = "2024-05-23T11:13:55.01Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "docstring-parser"
|
||||
version = "0.17.0"
|
||||
@@ -594,7 +574,7 @@ wheels = [
|
||||
|
||||
[[package]]
|
||||
name = "fastmcp"
|
||||
version = "2.12.3"
|
||||
version = "2.11.3"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "authlib" },
|
||||
@@ -609,9 +589,9 @@ dependencies = [
|
||||
{ name = "python-dotenv" },
|
||||
{ name = "rich" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/99/5e/035fdfa23646de8811776cd62d93440e334e8a4557b35c63c1bff125c08c/fastmcp-2.12.3.tar.gz", hash = "sha256:541dd569d5b6c083140b04d997ba3dc47f7c10695cee700d0a733ce63b20bb65", size = 5246812, upload-time = "2025-09-12T12:28:07.136Z" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/06/80/13aec687ec21727b0fe6d26c6fe2febb33ae24e24c980929a706db3a8bc2/fastmcp-2.11.3.tar.gz", hash = "sha256:e8e3834a3e0b513712b8e63a6f0d4cbe19093459a1da3f7fbf8ef2810cfd34e3", size = 2692092, upload-time = "2025-08-11T21:38:46.493Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/96/79/0fd386e61819e205563d4eb15da76564b80dc2edd3c64b46f2706235daec/fastmcp-2.12.3-py3-none-any.whl", hash = "sha256:aee50872923a9cba731861fc0120e7dbe4642a2685ba251b2b202b82fb6c25a9", size = 314031, upload-time = "2025-09-12T12:28:05.024Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/61/05/63f63ad5b6789a730d94b8cb3910679c5da1ed5b4e38c957140ac9edcf0e/fastmcp-2.11.3-py3-none-any.whl", hash = "sha256:28f22126c90fd36e5de9cc68b9c271b6d832dcf322256f23d220b68afb3352cc", size = 260231, upload-time = "2025-08-11T21:38:44.746Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -657,6 +637,18 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/60/16/b71171e97ec7b4ded8669542f4369d88d5a289e2704efbbde51e858e062a/gevent-25.5.1-cp314-cp314-macosx_11_0_universal2.whl", hash = "sha256:0bacf89a65489d26c7087669af89938d5bfd9f7afb12a07b57855b9fad6ccbd0", size = 2937113, upload-time = "2025-05-12T11:12:03.191Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "googleapis-common-protos"
|
||||
version = "1.70.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "protobuf" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/39/24/33db22342cf4a2ea27c9955e6713140fedd51e8b141b5ce5260897020f1a/googleapis_common_protos-1.70.0.tar.gz", hash = "sha256:0e1b44e0ea153e6594f9f394fef15193a68aaaea2d843f83e2742717ca753257", size = 145903, upload-time = "2025-04-14T10:17:02.924Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/86/f1/62a193f0227cf15a920390abe675f386dec35f7ae3ffe6da582d3ade42c7/googleapis_common_protos-1.70.0-py3-none-any.whl", hash = "sha256:b8bfcca8c25a2bb253e0e0b0adaf8c00773e5e6af6fd92397576680b807e0fd8", size = 294530, upload-time = "2025-04-14T10:17:01.271Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "greenlet"
|
||||
version = "3.2.4"
|
||||
@@ -788,6 +780,18 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/76/c6/c88e154df9c4e1a2a66ccf0005a88dfb2650c1dffb6f5ce603dfbd452ce3/idna-3.10-py3-none-any.whl", hash = "sha256:946d195a0d259cbba61165e88e65941f16e9b36ea6ddb97f00452bae8b1287d3", size = 70442, upload-time = "2024-09-15T18:07:37.964Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "importlib-metadata"
|
||||
version = "8.7.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "zipp" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/76/66/650a33bd90f786193e4de4b3ad86ea60b53c89b669a5c7be931fac31cdb0/importlib_metadata-8.7.0.tar.gz", hash = "sha256:d13b81ad223b890aa16c5471f2ac3056cf76c5f10f82d6f9292f0b415f389000", size = 56641, upload-time = "2025-04-27T15:29:01.736Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/20/b0/36bd937216ec521246249be3bf9855081de4c5e06a0c9b4219dbeda50373/importlib_metadata-8.7.0-py3-none-any.whl", hash = "sha256:e5dd1551894c77868a30651cef00984d50e1002d06942a7101d34870c5f02afd", size = 27656, upload-time = "2025-04-27T15:29:00.214Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "iniconfig"
|
||||
version = "2.1.0"
|
||||
@@ -875,6 +879,24 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/e7/1e/fb441c07b6662ec1fc92b249225ba6e6e5221b05623cb0131d082f782edc/lazy_object_proxy-1.11.0-py3-none-any.whl", hash = "sha256:a56a5093d433341ff7da0e89f9b486031ccd222ec8e52ec84d0ec1cdc819674b", size = 16635, upload-time = "2025-04-16T16:53:47.198Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "logfire"
|
||||
version = "4.13.2"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "executing" },
|
||||
{ name = "opentelemetry-exporter-otlp-proto-http" },
|
||||
{ name = "opentelemetry-instrumentation" },
|
||||
{ name = "opentelemetry-sdk" },
|
||||
{ name = "protobuf" },
|
||||
{ name = "rich" },
|
||||
{ name = "typing-extensions" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/11/51/004bbe0276fcecfea8d4a4c76ad33426c9b34428e0723f22c6bb801dbc22/logfire-4.13.2.tar.gz", hash = "sha256:4e756e140c3b8fd25653d20437ebcb75734975f5382de6ae28be775c75575d95", size = 547796, upload-time = "2025-10-13T16:17:53.392Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/4b/5f/0803848cc5ce524ff830e5f2a2f1400fd6ee72be705d87d0432cec42b1e4/logfire-4.13.2-py3-none-any.whl", hash = "sha256:887e99897a1818864aa5bfc595b02c93264ce23d1860866369eff6b6e2dde1c6", size = 228152, upload-time = "2025-10-13T16:17:50.641Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "loguru"
|
||||
version = "0.7.3"
|
||||
@@ -1069,6 +1091,103 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/27/dd/b3fd642260cb17532f66cc1e8250f3507d1e580483e209dc1e9d13bd980d/openapi_spec_validator-0.7.2-py3-none-any.whl", hash = "sha256:4bbdc0894ec85f1d1bea1d6d9c8b2c3c8d7ccaa13577ef40da9c006c9fd0eb60", size = 39713, upload-time = "2025-06-07T14:48:54.077Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "opentelemetry-api"
|
||||
version = "1.37.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "importlib-metadata" },
|
||||
{ name = "typing-extensions" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/63/04/05040d7ce33a907a2a02257e601992f0cdf11c73b33f13c4492bf6c3d6d5/opentelemetry_api-1.37.0.tar.gz", hash = "sha256:540735b120355bd5112738ea53621f8d5edb35ebcd6fe21ada3ab1c61d1cd9a7", size = 64923, upload-time = "2025-09-11T10:29:01.662Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/91/48/28ed9e55dcf2f453128df738210a980e09f4e468a456fa3c763dbc8be70a/opentelemetry_api-1.37.0-py3-none-any.whl", hash = "sha256:accf2024d3e89faec14302213bc39550ec0f4095d1cf5ca688e1bfb1c8612f47", size = 65732, upload-time = "2025-09-11T10:28:41.826Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "opentelemetry-exporter-otlp-proto-common"
|
||||
version = "1.37.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "opentelemetry-proto" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/dc/6c/10018cbcc1e6fff23aac67d7fd977c3d692dbe5f9ef9bb4db5c1268726cc/opentelemetry_exporter_otlp_proto_common-1.37.0.tar.gz", hash = "sha256:c87a1bdd9f41fdc408d9cc9367bb53f8d2602829659f2b90be9f9d79d0bfe62c", size = 20430, upload-time = "2025-09-11T10:29:03.605Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/08/13/b4ef09837409a777f3c0af2a5b4ba9b7af34872bc43609dda0c209e4060d/opentelemetry_exporter_otlp_proto_common-1.37.0-py3-none-any.whl", hash = "sha256:53038428449c559b0c564b8d718df3314da387109c4d36bd1b94c9a641b0292e", size = 18359, upload-time = "2025-09-11T10:28:44.939Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "opentelemetry-exporter-otlp-proto-http"
|
||||
version = "1.37.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "googleapis-common-protos" },
|
||||
{ name = "opentelemetry-api" },
|
||||
{ name = "opentelemetry-exporter-otlp-proto-common" },
|
||||
{ name = "opentelemetry-proto" },
|
||||
{ name = "opentelemetry-sdk" },
|
||||
{ name = "requests" },
|
||||
{ name = "typing-extensions" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/5d/e3/6e320aeb24f951449e73867e53c55542bebbaf24faeee7623ef677d66736/opentelemetry_exporter_otlp_proto_http-1.37.0.tar.gz", hash = "sha256:e52e8600f1720d6de298419a802108a8f5afa63c96809ff83becb03f874e44ac", size = 17281, upload-time = "2025-09-11T10:29:04.844Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/e9/e9/70d74a664d83976556cec395d6bfedd9b85ec1498b778367d5f93e373397/opentelemetry_exporter_otlp_proto_http-1.37.0-py3-none-any.whl", hash = "sha256:54c42b39945a6cc9d9a2a33decb876eabb9547e0dcb49df090122773447f1aef", size = 19576, upload-time = "2025-09-11T10:28:46.726Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "opentelemetry-instrumentation"
|
||||
version = "0.58b0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "opentelemetry-api" },
|
||||
{ name = "opentelemetry-semantic-conventions" },
|
||||
{ name = "packaging" },
|
||||
{ name = "wrapt" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/f6/36/7c307d9be8ce4ee7beb86d7f1d31027f2a6a89228240405a858d6e4d64f9/opentelemetry_instrumentation-0.58b0.tar.gz", hash = "sha256:df640f3ac715a3e05af145c18f527f4422c6ab6c467e40bd24d2ad75a00cb705", size = 31549, upload-time = "2025-09-11T11:42:14.084Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/d4/db/5ff1cd6c5ca1d12ecf1b73be16fbb2a8af2114ee46d4b0e6d4b23f4f4db7/opentelemetry_instrumentation-0.58b0-py3-none-any.whl", hash = "sha256:50f97ac03100676c9f7fc28197f8240c7290ca1baa12da8bfbb9a1de4f34cc45", size = 33019, upload-time = "2025-09-11T11:41:00.624Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "opentelemetry-proto"
|
||||
version = "1.37.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "protobuf" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/dd/ea/a75f36b463a36f3c5a10c0b5292c58b31dbdde74f6f905d3d0ab2313987b/opentelemetry_proto-1.37.0.tar.gz", hash = "sha256:30f5c494faf66f77faeaefa35ed4443c5edb3b0aa46dad073ed7210e1a789538", size = 46151, upload-time = "2025-09-11T10:29:11.04Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/c4/25/f89ea66c59bd7687e218361826c969443c4fa15dfe89733f3bf1e2a9e971/opentelemetry_proto-1.37.0-py3-none-any.whl", hash = "sha256:8ed8c066ae8828bbf0c39229979bdf583a126981142378a9cbe9d6fd5701c6e2", size = 72534, upload-time = "2025-09-11T10:28:56.831Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "opentelemetry-sdk"
|
||||
version = "1.37.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "opentelemetry-api" },
|
||||
{ name = "opentelemetry-semantic-conventions" },
|
||||
{ name = "typing-extensions" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/f4/62/2e0ca80d7fe94f0b193135375da92c640d15fe81f636658d2acf373086bc/opentelemetry_sdk-1.37.0.tar.gz", hash = "sha256:cc8e089c10953ded765b5ab5669b198bbe0af1b3f89f1007d19acd32dc46dda5", size = 170404, upload-time = "2025-09-11T10:29:11.779Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/9f/62/9f4ad6a54126fb00f7ed4bb5034964c6e4f00fcd5a905e115bd22707e20d/opentelemetry_sdk-1.37.0-py3-none-any.whl", hash = "sha256:8f3c3c22063e52475c5dbced7209495c2c16723d016d39287dfc215d1771257c", size = 131941, upload-time = "2025-09-11T10:28:57.83Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "opentelemetry-semantic-conventions"
|
||||
version = "0.58b0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "opentelemetry-api" },
|
||||
{ name = "typing-extensions" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/aa/1b/90701d91e6300d9f2fb352153fb1721ed99ed1f6ea14fa992c756016e63a/opentelemetry_semantic_conventions-0.58b0.tar.gz", hash = "sha256:6bd46f51264279c433755767bb44ad00f1c9e2367e1b42af563372c5a6fa0c25", size = 129867, upload-time = "2025-09-11T10:29:12.597Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/07/90/68152b7465f50285d3ce2481b3aec2f82822e3f52e5152eeeaf516bab841/opentelemetry_semantic_conventions-0.58b0-py3-none-any.whl", hash = "sha256:5564905ab1458b96684db1340232729fce3b5375a06e140e8904c78e4f815b28", size = 207954, upload-time = "2025-09-11T10:28:59.218Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "packaging"
|
||||
version = "25.0"
|
||||
@@ -1172,16 +1291,18 @@ wheels = [
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "psycopg"
|
||||
version = "3.3.1"
|
||||
name = "protobuf"
|
||||
version = "6.33.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "typing-extensions", marker = "python_full_version < '3.13'" },
|
||||
{ name = "tzdata", marker = "sys_platform == 'win32'" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/18/ed/3a30e8ef82d4128c76aa9bd6b2a7fe6c16c283811e6655997f5047801b47/psycopg-3.3.1.tar.gz", hash = "sha256:ccfa30b75874eef809c0fbbb176554a2640cc1735a612accc2e2396a92442fc6", size = 165596, upload-time = "2025-12-02T21:09:55.545Z" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/19/ff/64a6c8f420818bb873713988ca5492cba3a7946be57e027ac63495157d97/protobuf-6.33.0.tar.gz", hash = "sha256:140303d5c8d2037730c548f8c7b93b20bb1dc301be280c378b82b8894589c954", size = 443463, upload-time = "2025-10-15T20:39:52.159Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/b6/f3/0b4a4c25a47c2d907afa97674287dab61bc9941c9ac3972a67100e33894d/psycopg-3.3.1-py3-none-any.whl", hash = "sha256:e44d8eae209752efe46318f36dd0fdf5863e928009338d736843bb1084f6435c", size = 212760, upload-time = "2025-12-02T21:02:36.029Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/7e/ee/52b3fa8feb6db4a833dfea4943e175ce645144532e8a90f72571ad85df4e/protobuf-6.33.0-cp310-abi3-win32.whl", hash = "sha256:d6101ded078042a8f17959eccd9236fb7a9ca20d3b0098bbcb91533a5680d035", size = 425593, upload-time = "2025-10-15T20:39:40.29Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/7b/c6/7a465f1825872c55e0341ff4a80198743f73b69ce5d43ab18043699d1d81/protobuf-6.33.0-cp310-abi3-win_amd64.whl", hash = "sha256:9a031d10f703f03768f2743a1c403af050b6ae1f3480e9c140f39c45f81b13ee", size = 436882, upload-time = "2025-10-15T20:39:42.841Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/e1/a9/b6eee662a6951b9c3640e8e452ab3e09f117d99fc10baa32d1581a0d4099/protobuf-6.33.0-cp39-abi3-macosx_10_9_universal2.whl", hash = "sha256:905b07a65f1a4b72412314082c7dbfae91a9e8b68a0cc1577515f8df58ecf455", size = 427521, upload-time = "2025-10-15T20:39:43.803Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/10/35/16d31e0f92c6d2f0e77c2a3ba93185130ea13053dd16200a57434c882f2b/protobuf-6.33.0-cp39-abi3-manylinux2014_aarch64.whl", hash = "sha256:e0697ece353e6239b90ee43a9231318302ad8353c70e6e45499fa52396debf90", size = 324445, upload-time = "2025-10-15T20:39:44.932Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/e6/eb/2a981a13e35cda8b75b5585aaffae2eb904f8f351bdd3870769692acbd8a/protobuf-6.33.0-cp39-abi3-manylinux2014_s390x.whl", hash = "sha256:e0a1715e4f27355afd9570f3ea369735afc853a6c3951a6afe1f80d8569ad298", size = 339159, upload-time = "2025-10-15T20:39:46.186Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/21/51/0b1cbad62074439b867b4e04cc09b93f6699d78fd191bed2bbb44562e077/protobuf-6.33.0-cp39-abi3-manylinux2014_x86_64.whl", hash = "sha256:35be49fd3f4fefa4e6e2aacc35e8b837d6703c37a2168a55ac21e9b1bc7559ef", size = 323172, upload-time = "2025-10-15T20:39:47.465Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/07/d1/0a28c21707807c6aacd5dc9c3704b2aa1effbf37adebd8caeaf68b17a636/protobuf-6.33.0-py3-none-any.whl", hash = "sha256:25c9e1963c6734448ea2d308cfa610e692b801304ba0908d7bfa564ac5132995", size = 170477, upload-time = "2025-10-15T20:39:51.311Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -1354,15 +1475,14 @@ wheels = [
|
||||
|
||||
[[package]]
|
||||
name = "pytest-asyncio"
|
||||
version = "1.3.0"
|
||||
version = "1.1.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "pytest" },
|
||||
{ name = "typing-extensions", marker = "python_full_version < '3.13'" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/90/2c/8af215c0f776415f3590cac4f9086ccefd6fd463befeae41cd4d3f193e5a/pytest_asyncio-1.3.0.tar.gz", hash = "sha256:d7f52f36d231b80ee124cd216ffb19369aa168fc10095013c6b014a34d3ee9e5", size = 50087, upload-time = "2025-11-10T16:07:47.256Z" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/4e/51/f8794af39eeb870e87a8c8068642fc07bce0c854d6865d7dd0f2a9d338c2/pytest_asyncio-1.1.0.tar.gz", hash = "sha256:796aa822981e01b68c12e4827b8697108f7205020f24b5793b3c41555dab68ea", size = 46652, upload-time = "2025-07-16T04:29:26.393Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/e5/35/f8b19922b6a25bc0880171a2f1a003eaeb93657475193ab516fd87cac9da/pytest_asyncio-1.3.0-py3-none-any.whl", hash = "sha256:611e26147c7f77640e6d0a92a38ed17c3e9848063698d5c93d5aa7aa11cebff5", size = 15075, upload-time = "2025-11-10T16:07:45.537Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/c7/9d/bf86eddabf8c6c9cb1ea9a869d6873b46f105a5d292d3a6f7071f5b07935/pytest_asyncio-1.1.0-py3-none-any.whl", hash = "sha256:5fe2d69607b0bd75c656d1211f969cadba035030156745ee09e7d71740e58ecf", size = 15157, upload-time = "2025-07-16T04:29:24.929Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -1884,22 +2004,6 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/f7/1f/b876b1f83aef204198a42dc101613fefccb32258e5428b5f9259677864b4/starlette-0.47.2-py3-none-any.whl", hash = "sha256:c5847e96134e5c5371ee9fac6fdf1a67336d5815e09eb2a01fdb57a351ef915b", size = 72984, upload-time = "2025-07-20T17:31:56.738Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "testcontainers"
|
||||
version = "4.13.3"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "docker" },
|
||||
{ name = "python-dotenv" },
|
||||
{ name = "typing-extensions" },
|
||||
{ name = "urllib3" },
|
||||
{ name = "wrapt" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/fc/b3/c272537f3ea2f312555efeb86398cc382cd07b740d5f3c730918c36e64e1/testcontainers-4.13.3.tar.gz", hash = "sha256:9d82a7052c9a53c58b69e1dc31da8e7a715e8b3ec1c4df5027561b47e2efe646", size = 79064, upload-time = "2025-11-14T05:08:47.584Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/73/27/c2f24b19dafa197c514abe70eda69bc031c5152c6b1f1e5b20099e2ceedd/testcontainers-4.13.3-py3-none-any.whl", hash = "sha256:063278c4805ffa6dd85e56648a9da3036939e6c0ac1001e851c9276b19b05970", size = 124784, upload-time = "2025-11-14T05:08:46.053Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "typer"
|
||||
version = "0.16.0"
|
||||
@@ -2187,6 +2291,15 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/1f/f6/a933bd70f98e9cf3e08167fc5cd7aaaca49147e48411c0bd5ae701bb2194/wrapt-1.17.3-py3-none-any.whl", hash = "sha256:7171ae35d2c33d326ac19dd8facb1e82e5fd04ef8c6c0e394d7af55a55051c22", size = 23591, upload-time = "2025-08-12T05:53:20.674Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "zipp"
|
||||
version = "3.23.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/e3/02/0f2892c661036d50ede074e376733dca2ae7c6eb617489437771209d4180/zipp-3.23.0.tar.gz", hash = "sha256:a07157588a12518c9d4034df3fbbee09c814741a33ff63c05fa29d26a2404166", size = 25547, upload-time = "2025-06-08T17:06:39.4Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/2e/54/647ade08bf0db230bfea292f893923872fd20be6ac6f53b2b936ba839d75/zipp-3.23.0-py3-none-any.whl", hash = "sha256:071652d6115ed432f5ce1d34c336c0adfd6a884660d1e9712a256d3d3bd4b14e", size = 10276, upload-time = "2025-06-08T17:06:38.034Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "zope-event"
|
||||
version = "5.1.1"
|
||||
|
||||
Reference in New Issue
Block a user