Compare commits

...

33 Commits

Author SHA1 Message Date
claude[bot] 8c067caa59 fix: Pin FastMCP to 2.12.3 to restore MCP tools visibility
Version 2.14.x introduced breaking changes that prevent Claude Desktop
from seeing MCP tools. Pinning to 2.12.3 until we can properly migrate.

Fixes #463

Co-authored-by: Drew Cain <groksrc@users.noreply.github.com>
2025-12-20 01:16:44 +00:00
Paul Hernandez 897b1edaa4 fix: Reduce watch service CPU usage by increasing reload interval (#458)
Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-12-17 11:37:01 -06:00
Paul Hernandez 0c12a39a98 test: Add integration test for issue #416 (read_note with underscored folders) (#453)
Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-12-17 09:56:45 -06:00
Paul Hernandez efbc758325 fix: await background sync task cancellation in lifespan shutdown (#456)
Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
2025-12-17 09:53:34 -06:00
Paul Hernandez a0f20eb102 chore: more Tenantless fixes (#457)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-16 18:34:05 -06:00
Paul Hernandez 78673d8e51 chore: Cloud compatibility fixes and performance improvements (#454)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-15 20:07:55 -06:00
phernandez 126c0495c0 Merge branch 'main' of github.com:basicmachines-co/basic-memory 2025-12-13 15:24:43 -06:00
Paul Hernandez 4a43d7df4a remove logfire instrumentation
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-13 15:22:14 -06:00
Paul Hernandez c462faf046 Replace py-pglite with testcontainers for Postgres testing (#449)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-10 22:17:56 -06:00
Cedric Hurst 70bb10be1d fix: respect --project flag in background sync (fixes #434) (#436)
Signed-off-by: Cedric Hurst <cedric@spantree.net>
2025-12-08 12:58:18 -06:00
phernandez fbf9045d78 use asyncpg for just db-migrate
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-05 15:01:35 -06:00
phernandez 1094210c52 fix broken sqlite migration
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-02 20:23:55 -06:00
phernandez 391feb639f add delete cascade to entity to delete search_index (postgres only)
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-02 10:04:21 -06:00
phernandez a920a9ff29 feat: Add project_id to Relation and Observation for efficient project-scoped queries
Denormalizes project_id onto Relation and Observation tables to enable
efficient project-scoped queries without joins. Migration backfills
from associated entity and adds pg_trgm extension with GIN indexes
for fuzzy link resolution on PostgreSQL.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-01 21:54:56 -06:00
phernandez 05efe8701c test: Verify update() returns entity with eager-loaded relations
Add test confirming entity_repository.update() returns the entity with
observations and relations eagerly loaded, eliminating the need for a
separate find_by_id() call after update.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-12-01 16:22:21 -06:00
phernandez 0eaf30bb06 remove conflict constraint name from relation_repository.py
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-30 19:31:09 -06:00
phernandez 0818bda565 feat: Add bulk insert with ON CONFLICT handling for relations
Add add_all_ignore_duplicates() method to RelationRepository for bulk
inserting relations with ON CONFLICT DO NOTHING. This handles cases
where the same [[wiki link]] appears multiple times in a document,
silently ignoring duplicates based on the (from_id, to_name, relation_type)
unique constraint.

Works with both SQLite and PostgreSQL dialects.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-30 14:59:52 -06:00
phernandez 6f99d2e551 perf: lightweight permalink resolution to avoid eager loading
Add optimized repository methods for resolve_permalink() that skip
eager loading of observations and relations:

- permalink_exists(): Check existence without loading entity
- get_file_path_for_permalink(): Get only file_path column
- get_permalink_for_file_path(): Get only permalink column
- get_all_permalinks(): Get all permalinks as strings
- get_permalink_to_file_path_map(): Bulk lookup mapping
- get_file_path_to_permalink_map(): Reverse mapping

Updated entity_service.resolve_permalink() to use these lightweight
methods instead of loading full entities with all relationships.

Also added logfire instrumentation to markdown utils.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-30 14:34:31 -06:00
phernandez 73d940e064 fix: observation parsing and permalink limits (#446)
1. Hashtag detection now checks for standalone words starting with #
   instead of just checking if # appears anywhere in content.
   This prevents HTML color codes like #4285F4 from being
   interpreted as hashtags.

2. Observation permalinks now truncate content to 200 chars
   to stay under PostgreSQL's btree index limit of 2704 bytes.

Added tests for both fixes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-30 00:12:04 -06:00
phernandez c3678a11d2 truncate content_stems to fix Postgres 8KB index row limit
Large documents (like ~1MB conversation imports) exceed Postgres's 8KB
index row limit, causing ProgramLimitExceededError. Truncate content_stems
to 6000 characters (with headroom for other columns) before indexing.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-29 20:23:09 -06:00
phernandez 203d684c24 fix integrity error handling when setting forward relation refs
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-29 19:03:12 -06:00
phernandez a872220924 disable pooling for postgres db
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-29 13:24:47 -06:00
phernandez 7d763a66ff use entity.mtime for updated at in api
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-29 13:21:58 -06:00
phernandez b5d4fb559c fix: postgres/neon connection settings and search index dedupe
- Reduce db_pool_recycle from 3600s to 180s for Neon scale-to-zero
- Add connect_args for Neon serverless (statement cache, timeouts, app name)
- Dedupe observation permalinks in search indexing to avoid unique constraint violations
- Add tests for duplicate observation permalink handling

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-29 11:24:08 -06:00
phernandez 830775276d remove record_return=True from logfire spans
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 18:36:09 -06:00
phernandez ed894fc3ed get db pool sizes from config
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 16:50:57 -06:00
phernandez 704338edcf remove logfire.instrument_fastapi(app) from app.py
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 13:52:18 -06:00
phernandez 0ca02a7ebe add logfire instrumentation to services and repository code
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-28 12:47:29 -06:00
jope-bm 28cc5225a7 feat: Implement API v2 with ID-based endpoints (Phase 1) (#441)
Signed-off-by: Joe P <joe@basicmemory.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: phernandez <paul@basicmachines.co>
2025-11-27 10:35:55 -06:00
phernandez 9b7bbc7116 formatting and logic change to resolve_relations, remove fuzzy search 2025-11-25 22:54:37 -06:00
phernandez 138c283d6c add postgres db type 2025-11-25 20:25:56 -06:00
phernandez 7a8954c37e add extra logic for cloud-indexing improvements 2025-11-25 13:52:58 -06:00
phernandez 10c7c19c03 fix db url for sqlite migrations
Signed-off-by: phernandez <paul@basicmachines.co>
2025-11-21 13:21:20 -06:00
109 changed files with 7859 additions and 1377 deletions
+5
View File
@@ -0,0 +1,5 @@
{
"enabledPlugins": {
"basic-memory@basicmachines": true
}
}
+2 -16
View File
@@ -78,21 +78,7 @@ jobs:
python-version: [ "3.12", "3.13" ]
runs-on: ubuntu-latest
# Postgres service (only available on Linux runners)
services:
postgres:
image: postgres:17
env:
POSTGRES_DB: basic_memory_test
POSTGRES_USER: basic_memory_user
POSTGRES_PASSWORD: dev_password
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 5433:5432
# Note: No services section needed - testcontainers handles Postgres in Docker
steps:
- uses: actions/checkout@v4
@@ -121,7 +107,7 @@ jobs:
run: |
uv pip install -e .[dev]
- name: Run tests (Postgres)
- name: Run tests (Postgres via testcontainers)
run: |
uv pip install pytest pytest-cov
just test-postgres
+2 -1
View File
@@ -52,4 +52,5 @@ ENV/
# claude action
claude-output
**/.claude/settings.local.json
**/.claude/settings.local.json
.mcp.json
+20 -6
View File
@@ -15,10 +15,14 @@ See the [README.md](README.md) file for a project overview.
### Build and Test Commands
- Install: `just install` or `pip install -e ".[dev]"`
- Run all tests (with coverage): `just test` - Runs both unit and integration tests with unified coverage
- Run unit tests only: `just test-unit` - Fast, no coverage
- Run integration tests only: `just test-int` - Fast, no coverage
- Generate HTML coverage: `just coverage` - Opens in browser
- Run all tests (SQLite + Postgres): `just test`
- Run all tests against SQLite: `just test-sqlite`
- Run all tests against Postgres: `just test-postgres` (uses testcontainers)
- Run unit tests (SQLite): `just test-unit-sqlite`
- Run unit tests (Postgres): `just test-unit-postgres`
- Run integration tests (SQLite): `just test-int-sqlite`
- Run integration tests (Postgres): `just test-int-postgres`
- Generate HTML coverage: `just coverage`
- Single test: `pytest tests/path/to/test_file.py::test_function_name`
- Run benchmarks: `pytest test-int/test_sync_performance_benchmark.py -v -m "benchmark and not slow"`
- Lint: `just lint` or `ruff check . --fix`
@@ -30,6 +34,8 @@ See the [README.md](README.md) file for a project overview.
**Note:** Project requires Python 3.12+ (uses type parameter syntax and `type` aliases introduced in 3.12)
**Postgres Testing:** Uses [testcontainers](https://testcontainers-python.readthedocs.io/) which automatically spins up a Postgres instance in Docker. No manual database setup required - just have Docker running.
### Test Structure
- `tests/` - Unit tests for individual components (mocked, fast)
@@ -76,8 +82,10 @@ See the [README.md](README.md) file for a project overview.
- SQLite is used for indexing and full text search, files are source of truth
- Testing uses pytest with asyncio support (strict mode)
- Unit tests (`tests/`) use mocks when necessary; integration tests (`test-int/`) use real implementations
- Test database uses in-memory SQLite
- Each test runs in a standalone environment with in-memory SQLite and tmp_file directory
- By default, tests run against SQLite (fast, no Docker needed)
- Set `BASIC_MEMORY_TEST_POSTGRES=1` to run against Postgres (uses testcontainers - Docker required)
- Each test runs in a standalone environment with isolated database and tmp_path directory
- CI runs SQLite and Postgres tests in parallel for faster feedback
- Performance benchmarks are in `test-int/test_sync_performance_benchmark.py`
- Use pytest markers: `@pytest.mark.benchmark` for benchmarks, `@pytest.mark.slow` for slow tests
@@ -229,6 +237,11 @@ of using AI just for code generation, we've developed a true collaborative workf
This approach has allowed us to tackle more complex challenges and build a more robust system than either humans or AI
could achieve independently.
**Problem-Solving Guidance:**
- If a solution isn't working after reasonable effort, suggest alternative approaches
- Don't persist with a problematic library or pattern when better alternatives exist
- Example: When py-pglite caused cascading test failures, switching to testcontainers-postgres was the right call
## GitHub Integration
Basic Memory has taken AI-Human collaboration to the next level by integrating Claude directly into the development workflow through GitHub:
@@ -264,5 +277,6 @@ With GitHub integration, the development workflow includes:
2. **Contribution tracking** - All of Claude's contributions are properly attributed in the Git history
3. **Branch management** - Claude can create feature branches for implementations
4. **Documentation maintenance** - Claude can keep documentation updated as the code evolves
5. **Code Commits**: ALWAYS sign off commits with `git commit -s`
This level of integration represents a new paradigm in AI-human collaboration, where the AI assistant becomes a full-fledged team member rather than just a tool for generating code snippets.
+48 -14
View File
@@ -433,42 +433,76 @@ See the [Documentation](https://memory.basicmachines.co/) for more info, includi
- [Managing multiple Projects](https://docs.basicmemory.com/guides/cli-reference/#project)
- [Importing data from OpenAI/Claude Projects](https://docs.basicmemory.com/guides/cli-reference/#import)
## Logging
Basic Memory uses [Loguru](https://github.com/Delgan/loguru) for logging. The logging behavior varies by entry point:
| Entry Point | Default Behavior | Use Case |
|-------------|------------------|----------|
| CLI commands | File only | Prevents log output from interfering with command output |
| MCP server | File only | Stdout would corrupt the JSON-RPC protocol |
| API server | File (local) or stdout (cloud) | Docker/cloud deployments use stdout |
**Log file location:** `~/.basic-memory/basic-memory.log` (10MB rotation, 10 days retention)
### Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `BASIC_MEMORY_LOG_LEVEL` | `INFO` | Log level: DEBUG, INFO, WARNING, ERROR |
| `BASIC_MEMORY_CLOUD_MODE` | `false` | When `true`, API logs to stdout with structured context |
| `BASIC_MEMORY_ENV` | `dev` | Set to `test` for test mode (stderr only) |
### Examples
```bash
# Enable debug logging
BASIC_MEMORY_LOG_LEVEL=DEBUG basic-memory sync
# View logs
tail -f ~/.basic-memory/basic-memory.log
# Cloud/Docker mode (stdout logging with structured context)
BASIC_MEMORY_CLOUD_MODE=true uvicorn basic_memory.api.app:app
```
## Development
### Running Tests
Basic Memory supports dual database backends (SQLite and Postgres). Tests are parametrized to run against both backends automatically.
Basic Memory supports dual database backends (SQLite and Postgres). By default, tests run against SQLite. Set `BASIC_MEMORY_TEST_POSTGRES=1` to run against Postgres (uses testcontainers - Docker required).
**Quick Start:**
```bash
# Run SQLite tests (default, no Docker needed)
# Run all tests against SQLite (default, fast)
just test-sqlite
# Run Postgres tests (requires Docker)
# Run all tests against Postgres (uses testcontainers)
just test-postgres
# Run both SQLite and Postgres tests
just test
```
**Available Test Commands:**
- `just test-sqlite` - Run tests against SQLite only (fastest, no Docker needed)
- `just test-postgres` - Run tests against Postgres only (requires Docker)
- `just test` - Run all tests against both SQLite and Postgres
- `just test-sqlite` - Run all tests against SQLite (fast, no Docker needed)
- `just test-postgres` - Run all tests against Postgres (uses testcontainers)
- `just test-unit-sqlite` - Run unit tests against SQLite
- `just test-unit-postgres` - Run unit tests against Postgres
- `just test-int-sqlite` - Run integration tests against SQLite
- `just test-int-postgres` - Run integration tests against Postgres
- `just test-windows` - Run Windows-specific tests (auto-skips on other platforms)
- `just test-benchmark` - Run performance benchmark tests
- `just test-all` - Run all tests including Windows, Postgres, and benchmarks
**Postgres Testing Requirements:**
**Postgres Testing:**
To run Postgres tests, you need to start the test database:
```bash
docker-compose -f docker-compose-postgres.yml up -d
```
Tests will connect to `localhost:5433/basic_memory_test`.
Postgres tests use [testcontainers](https://testcontainers-python.readthedocs.io/) which automatically spins up a Postgres instance in Docker. No manual database setup required - just have Docker running.
**Test Markers:**
Tests use pytest markers for selective execution:
- `postgres` - Tests that run against Postgres backend
- `windows` - Windows-specific database optimizations
- `benchmark` - Performance tests (excluded from default runs)
+36 -29
View File
@@ -7,44 +7,51 @@ install:
@echo ""
@echo "💡 Remember to activate the virtual environment by running: source .venv/bin/activate"
# Run all tests with unified coverage report
test: test-unit test-int
# Run unit tests only (fast, no coverage)
test-unit:
uv run pytest -p pytest_mock -v --no-cov tests
# Run integration tests only (fast, no coverage)
test-int:
uv run pytest -p pytest_mock -v --no-cov test-int
# ==============================================================================
# DATABASE BACKEND TESTING
# ==============================================================================
# Basic Memory supports dual database backends (SQLite and Postgres).
# Tests are parametrized to run against both backends automatically.
# By default, tests run against SQLite (fast, no dependencies).
# Set BASIC_MEMORY_TEST_POSTGRES=1 to run against Postgres (uses testcontainers).
#
# Quick Start:
# just test-sqlite # Run SQLite tests (default, no Docker needed)
# just test-postgres # Run Postgres tests (requires Docker)
# just test # Run all tests against SQLite (default)
# just test-sqlite # Run all tests against SQLite
# just test-postgres # Run all tests against Postgres (testcontainers)
# just test-unit-sqlite # Run unit tests against SQLite
# just test-unit-postgres # Run unit tests against Postgres
# just test-int-sqlite # Run integration tests against SQLite
# just test-int-postgres # Run integration tests against Postgres
#
# For Postgres tests, first start the database:
# docker-compose -f docker-compose-postgres.yml up -d
# CI runs both in parallel for faster feedback.
# ==============================================================================
# Run tests against SQLite only (default backend, skip Postgres/Benchmark tests)
# This is the fastest option and doesn't require any Docker setup.
# Use this for local development and quick feedback.
# Includes Windows-specific tests which will auto-skip on non-Windows platforms.
test-sqlite:
uv run pytest -p pytest_mock -v --no-cov -m "not postgres and not benchmark" tests test-int
# Run all tests against SQLite and Postgres
test: test-sqlite test-postgres
# Run tests against Postgres only (requires docker-compose-postgres.yml up)
# First start Postgres: docker-compose -f docker-compose-postgres.yml up -d
# Tests will connect to localhost:5433/basic_memory_test
# To reset the database: just postgres-reset
test-postgres:
uv run pytest -p pytest_mock -v --no-cov -m "postgres and not benchmark" tests test-int
# Run all tests against SQLite
test-sqlite: test-unit-sqlite test-int-sqlite
# Run all tests against Postgres (uses testcontainers)
test-postgres: test-unit-postgres test-int-postgres
# Run unit tests against SQLite
test-unit-sqlite:
uv run pytest -p pytest_mock -v --no-cov tests
# Run unit tests against Postgres
test-unit-postgres:
BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov tests
# Run integration tests against SQLite
test-int-sqlite:
uv run pytest -p pytest_mock -v --no-cov test-int
# Run integration tests against Postgres
# Note: Uses timeout due to FastMCP Client + asyncpg cleanup hang (tests pass, process hangs on exit)
# See: https://github.com/jlowin/fastmcp/issues/1311
test-int-postgres:
timeout --signal=KILL 600 bash -c 'BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov test-int' || test $? -eq 137
# Reset Postgres test database (drops and recreates schema)
# Useful when Alembic migration state gets out of sync during development
@@ -59,7 +66,7 @@ postgres-reset:
postgres-migrate:
@cd src/basic_memory/alembic && \
BASIC_MEMORY_DATABASE_BACKEND=postgres \
BASIC_MEMORY_DATABASE_URL=${POSTGRES_TEST_URL:-postgresql://basic_memory_user:dev_password@localhost:5433/basic_memory_test} \
BASIC_MEMORY_DATABASE_URL=${POSTGRES_TEST_URL:-postgresql+asyncpg://basic_memory_user:dev_password@localhost:5433/basic_memory_test} \
uv run alembic upgrade head
@echo "✅ Migrations applied to Postgres test database"
+7 -6
View File
@@ -15,7 +15,6 @@ dependencies = [
"aiosqlite>=0.20.0",
"greenlet>=3.1.1",
"pydantic[email,timezone]>=2.10.3",
"icecream>=2.1.3",
"mcp>=1.2.0",
"pydantic-settings>=2.6.1",
"loguru>=0.7.3",
@@ -30,13 +29,15 @@ dependencies = [
"alembic>=1.14.1",
"pillow>=11.1.0",
"pybars3>=0.9.7",
"fastmcp>=2.10.2",
"fastmcp==2.12.3",
"pyjwt>=2.10.1",
"python-dotenv>=1.1.0",
"pytest-aio>=1.9.0",
"aiofiles>=24.1.0", # Async file I/O
"logfire>=0.73.0", # Optional observability (disabled by default via config)
"aiofiles>=24.1.0", # Optional observability (disabled by default via config)
"asyncpg>=0.30.0",
"nest-asyncio>=1.6.0", # For Alembic migrations with Postgres
"pytest-asyncio>=1.2.0",
"psycopg==3.3.1",
]
@@ -81,8 +82,8 @@ dev = [
"pytest-xdist>=3.0.0",
"ruff>=0.1.6",
"freezegun>=1.5.5",
"nest-asyncio>=1.6.0",
"psycopg2-binary>=2.9.0", # For Alembic migrations with Postgres
"testcontainers[postgres]>=4.0.0",
"psycopg>=3.2.0",
]
[tool.hatch.version]
+91 -25
View File
@@ -1,14 +1,25 @@
"""Alembic environment configuration."""
import asyncio
import os
from logging.config import fileConfig
from sqlalchemy import engine_from_config
from sqlalchemy import pool
# Allow nested event loops (needed for pytest-asyncio and other async contexts)
# Note: nest_asyncio doesn't work with uvloop, so we handle that case separately
try:
import nest_asyncio
nest_asyncio.apply()
except (ImportError, ValueError):
# nest_asyncio not available or can't patch this loop type (e.g., uvloop)
pass
from sqlalchemy import engine_from_config, pool
from sqlalchemy.ext.asyncio import AsyncEngine, create_async_engine
from alembic import context
from basic_memory.config import ConfigManager, DatabaseBackend
from basic_memory.config import ConfigManager
# set config.env to "test" for pytest to prevent logging to file in utils.setup_logging()
os.environ["BASIC_MEMORY_ENV"] = "test"
@@ -35,12 +46,6 @@ if not current_url or current_url == "driver://user:pass@localhost/dbname":
sqlalchemy_url = DatabaseType.get_db_url(
app_config.database_path, DatabaseType.FILESYSTEM, app_config
)
# For Postgres, Alembic needs synchronous driver (psycopg2), not async (asyncpg)
if app_config.database_backend == DatabaseBackend.POSTGRES:
# Convert asyncpg URL to psycopg2 URL for Alembic
sqlalchemy_url = sqlalchemy_url.replace("postgresql+asyncpg://", "postgresql://")
config.set_main_option("sqlalchemy.url", sqlalchemy_url)
# Interpret the config file for Python logging.
@@ -85,28 +90,89 @@ def run_migrations_offline() -> None:
context.run_migrations()
def do_run_migrations(connection):
"""Execute migrations with the given connection."""
context.configure(
connection=connection,
target_metadata=target_metadata,
include_object=include_object,
render_as_batch=True,
compare_type=True,
)
with context.begin_transaction():
context.run_migrations()
async def run_async_migrations(connectable):
"""Run migrations asynchronously with AsyncEngine."""
async with connectable.connect() as connection:
await connection.run_sync(do_run_migrations)
await connectable.dispose()
def run_migrations_online() -> None:
"""Run migrations in 'online' mode.
In this scenario we need to create an Engine
and associate a connection with the context.
Supports both sync engines (SQLite) and async engines (PostgreSQL with asyncpg).
"""
connectable = engine_from_config(
config.get_section(config.config_ini_section, {}),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
# Check if a connection/engine was provided (e.g., from run_migrations)
connectable = context.config.attributes.get("connection", None)
with connectable.connect() as connection:
context.configure(
connection=connection,
target_metadata=target_metadata,
include_object=include_object,
render_as_batch=True,
)
if connectable is None:
# No connection provided, create engine from config
url = context.config.get_main_option("sqlalchemy.url")
with context.begin_transaction():
context.run_migrations()
# Check if it's an async URL (sqlite+aiosqlite or postgresql+asyncpg)
if url and ("+asyncpg" in url or "+aiosqlite" in url):
# Create async engine for asyncpg or aiosqlite
connectable = create_async_engine(
url,
poolclass=pool.NullPool,
future=True,
)
else:
# Create sync engine for regular sqlite or postgresql
connectable = engine_from_config(
context.config.get_section(context.config.config_ini_section, {}),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
# Handle async engines (PostgreSQL with asyncpg)
if isinstance(connectable, AsyncEngine):
# Try to run async migrations
# nest_asyncio allows asyncio.run() from within event loops, but doesn't work with uvloop
try:
asyncio.run(run_async_migrations(connectable))
except RuntimeError as e:
if "cannot be called from a running event loop" in str(e):
# We're in a running event loop (likely uvloop) - need to use a different approach
# Create a new thread to run the async migrations
import concurrent.futures
def run_in_thread():
"""Run async migrations in a new event loop in a separate thread."""
new_loop = asyncio.new_event_loop()
asyncio.set_event_loop(new_loop)
try:
new_loop.run_until_complete(run_async_migrations(connectable))
finally:
new_loop.close()
with concurrent.futures.ThreadPoolExecutor() as executor:
future = executor.submit(run_in_thread)
future.result() # Wait for completion and re-raise any exceptions
else:
raise
else:
# Handle sync engines (SQLite) or sync connections
if hasattr(connectable, "connect"):
# It's an engine, get a connection
with connectable.connect() as connection:
do_run_migrations(connection)
else:
# It's already a connection
do_run_migrations(connectable)
if context.is_offline_mode():
@@ -0,0 +1,56 @@
"""Add cascade delete FK from search_index to entity
Revision ID: a2b3c4d5e6f7
Revises: f8a9b2c3d4e5
Create Date: 2025-12-02 07:00:00.000000
"""
from typing import Sequence, Union
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "a2b3c4d5e6f7"
down_revision: Union[str, None] = "f8a9b2c3d4e5"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Add FK with CASCADE delete from search_index.entity_id to entity.id.
This migration is Postgres-only because:
- SQLite uses FTS5 virtual tables which don't support foreign keys
- The FK enables automatic cleanup of search_index entries when entities are deleted
"""
connection = op.get_bind()
dialect = connection.dialect.name
if dialect == "postgresql":
# First, clean up any orphaned search_index entries where entity no longer exists
op.execute("""
DELETE FROM search_index
WHERE entity_id IS NOT NULL
AND entity_id NOT IN (SELECT id FROM entity)
""")
# Add FK with CASCADE - nullable FK allows search_index entries without entity_id
op.create_foreign_key(
"fk_search_index_entity_id",
"search_index",
"entity",
["entity_id"],
["id"],
ondelete="CASCADE",
)
def downgrade() -> None:
"""Remove the FK constraint."""
connection = op.get_bind()
dialect = connection.dialect.name
if dialect == "postgresql":
op.drop_constraint("fk_search_index_entity_id", "search_index", type_="foreignkey")
@@ -0,0 +1,199 @@
"""Add project_id to relation/observation and pg_trgm for fuzzy link resolution
Revision ID: f8a9b2c3d4e5
Revises: 314f1ea54dc4
Create Date: 2025-12-01 12:00:00.000000
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "f8a9b2c3d4e5"
down_revision: Union[str, None] = "314f1ea54dc4"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Add project_id to relation and observation tables, plus pg_trgm indexes.
This migration:
1. Adds project_id column to relation and observation tables (denormalization)
2. Backfills project_id from the associated entity
3. Enables pg_trgm extension for trigram-based fuzzy matching (Postgres only)
4. Creates GIN indexes on entity title and permalink for fast similarity searches
5. Creates partial index on unresolved relations for efficient bulk resolution
"""
connection = op.get_bind()
dialect = connection.dialect.name
# -------------------------------------------------------------------------
# Add project_id to relation table
# -------------------------------------------------------------------------
# Step 1: Add project_id column as nullable first
op.add_column("relation", sa.Column("project_id", sa.Integer(), nullable=True))
# Step 2: Backfill project_id from entity.project_id via from_id
if dialect == "postgresql":
op.execute("""
UPDATE relation
SET project_id = entity.project_id
FROM entity
WHERE relation.from_id = entity.id
""")
else:
# SQLite syntax
op.execute("""
UPDATE relation
SET project_id = (
SELECT entity.project_id
FROM entity
WHERE entity.id = relation.from_id
)
""")
# Step 3: Make project_id NOT NULL and add foreign key
if dialect == "postgresql":
op.alter_column("relation", "project_id", nullable=False)
op.create_foreign_key(
"fk_relation_project_id",
"relation",
"project",
["project_id"],
["id"],
)
else:
# SQLite requires batch operations for ALTER COLUMN
with op.batch_alter_table("relation") as batch_op:
batch_op.alter_column("project_id", nullable=False)
batch_op.create_foreign_key(
"fk_relation_project_id",
"project",
["project_id"],
["id"],
)
# Step 4: Create index on relation.project_id
op.create_index("ix_relation_project_id", "relation", ["project_id"])
# -------------------------------------------------------------------------
# Add project_id to observation table
# -------------------------------------------------------------------------
# Step 1: Add project_id column as nullable first
op.add_column("observation", sa.Column("project_id", sa.Integer(), nullable=True))
# Step 2: Backfill project_id from entity.project_id via entity_id
if dialect == "postgresql":
op.execute("""
UPDATE observation
SET project_id = entity.project_id
FROM entity
WHERE observation.entity_id = entity.id
""")
else:
# SQLite syntax
op.execute("""
UPDATE observation
SET project_id = (
SELECT entity.project_id
FROM entity
WHERE entity.id = observation.entity_id
)
""")
# Step 3: Make project_id NOT NULL and add foreign key
if dialect == "postgresql":
op.alter_column("observation", "project_id", nullable=False)
op.create_foreign_key(
"fk_observation_project_id",
"observation",
"project",
["project_id"],
["id"],
)
else:
# SQLite requires batch operations for ALTER COLUMN
with op.batch_alter_table("observation") as batch_op:
batch_op.alter_column("project_id", nullable=False)
batch_op.create_foreign_key(
"fk_observation_project_id",
"project",
["project_id"],
["id"],
)
# Step 4: Create index on observation.project_id
op.create_index("ix_observation_project_id", "observation", ["project_id"])
# Postgres-specific: pg_trgm and GIN indexes
if dialect == "postgresql":
# Enable pg_trgm extension for fuzzy string matching
op.execute("CREATE EXTENSION IF NOT EXISTS pg_trgm")
# Create trigram indexes on entity table for fuzzy matching
# GIN indexes with gin_trgm_ops support similarity searches
op.execute("""
CREATE INDEX IF NOT EXISTS idx_entity_title_trgm
ON entity USING gin (title gin_trgm_ops)
""")
op.execute("""
CREATE INDEX IF NOT EXISTS idx_entity_permalink_trgm
ON entity USING gin (permalink gin_trgm_ops)
""")
# Create partial index on unresolved relations for efficient bulk resolution
# This makes "WHERE to_id IS NULL AND project_id = X" queries very fast
op.execute("""
CREATE INDEX IF NOT EXISTS idx_relation_unresolved
ON relation (project_id, to_name)
WHERE to_id IS NULL
""")
# Create index on relation.to_name for join performance in bulk resolution
op.execute("""
CREATE INDEX IF NOT EXISTS idx_relation_to_name
ON relation (to_name)
""")
def downgrade() -> None:
"""Remove project_id from relation/observation and pg_trgm indexes."""
connection = op.get_bind()
dialect = connection.dialect.name
if dialect == "postgresql":
# Drop Postgres-specific indexes
op.execute("DROP INDEX IF EXISTS idx_relation_to_name")
op.execute("DROP INDEX IF EXISTS idx_relation_unresolved")
op.execute("DROP INDEX IF EXISTS idx_entity_permalink_trgm")
op.execute("DROP INDEX IF EXISTS idx_entity_title_trgm")
# Note: We don't drop the pg_trgm extension as other code may depend on it
# Drop project_id from observation
op.drop_index("ix_observation_project_id", table_name="observation")
op.drop_constraint("fk_observation_project_id", "observation", type_="foreignkey")
op.drop_column("observation", "project_id")
# Drop project_id from relation
op.drop_index("ix_relation_project_id", table_name="relation")
op.drop_constraint("fk_relation_project_id", "relation", type_="foreignkey")
op.drop_column("relation", "project_id")
else:
# SQLite requires batch operations
op.drop_index("ix_observation_project_id", table_name="observation")
with op.batch_alter_table("observation") as batch_op:
batch_op.drop_constraint("fk_observation_project_id", type_="foreignkey")
batch_op.drop_column("project_id")
op.drop_index("ix_relation_project_id", table_name="relation")
with op.batch_alter_table("relation") as batch_op:
batch_op.drop_constraint("fk_relation_project_id", type_="foreignkey")
batch_op.drop_column("project_id")
+31 -6
View File
@@ -20,7 +20,17 @@ from basic_memory.api.routers import (
search,
prompt_router,
)
from basic_memory.config import ConfigManager
from basic_memory.api.v2.routers import (
knowledge_router as v2_knowledge,
project_router as v2_project,
memory_router as v2_memory,
search_router as v2_search,
resource_router as v2_resource,
directory_router as v2_directory,
prompt_router as v2_prompt,
importer_router as v2_importer,
)
from basic_memory.config import ConfigManager, init_api_logging
from basic_memory.services.initialization import initialize_file_sync, initialize_app
@@ -28,6 +38,9 @@ from basic_memory.services.initialization import initialize_file_sync, initializ
async def lifespan(app: FastAPI): # pragma: no cover
"""Lifecycle manager for the FastAPI app. Not called in stdio mcp mode"""
# Initialize logging for API (stdout in cloud mode, file otherwise)
init_api_logging()
app_config = ConfigManager().config
logger.info("Starting Basic Memory API")
@@ -46,6 +59,7 @@ async def lifespan(app: FastAPI): # pragma: no cover
app.state.sync_task = asyncio.create_task(initialize_file_sync(app_config))
else:
logger.info("Sync changes disabled. Skipping file sync service.")
app.state.sync_task = None
# proceed with startup
yield
@@ -54,6 +68,10 @@ async def lifespan(app: FastAPI): # pragma: no cover
if app.state.sync_task:
logger.info("Stopping sync...")
app.state.sync_task.cancel() # pyright: ignore
try:
await app.state.sync_task
except asyncio.CancelledError:
logger.info("Sync task cancelled successfully")
await db.shutdown_db()
@@ -66,8 +84,7 @@ app = FastAPI(
lifespan=lifespan,
)
# Include routers
# Include v1 routers
app.include_router(knowledge.router, prefix="/{project}")
app.include_router(memory.router, prefix="/{project}")
app.include_router(resource.router, prefix="/{project}")
@@ -77,12 +94,20 @@ app.include_router(directory_router.router, prefix="/{project}")
app.include_router(prompt_router.router, prefix="/{project}")
app.include_router(importer_router.router, prefix="/{project}")
# Project resource router works accross projects
# Include v2 routers (ID-based paths)
app.include_router(v2_knowledge, prefix="/v2/projects/{project_id}")
app.include_router(v2_memory, prefix="/v2/projects/{project_id}")
app.include_router(v2_search, prefix="/v2/projects/{project_id}")
app.include_router(v2_resource, prefix="/v2/projects/{project_id}")
app.include_router(v2_directory, prefix="/v2/projects/{project_id}")
app.include_router(v2_prompt, prefix="/v2/projects/{project_id}")
app.include_router(v2_importer, prefix="/v2/projects/{project_id}")
app.include_router(v2_project, prefix="/v2")
# Project resource router works across projects
app.include_router(project.project_resource_router)
app.include_router(management.router)
# Auth routes are handled by FastMCP automatically when auth is enabled
@app.exception_handler(Exception)
async def exception_handler(request, exc): # pragma: no cover
@@ -1,4 +1,11 @@
"""Router for knowledge graph operations."""
"""Router for knowledge graph operations.
⚠️ DEPRECATED: This v1 API is deprecated and will be removed on June 30, 2026.
Please migrate to /v2/{project}/knowledge endpoints which use entity IDs instead
of path-based identifiers for improved performance and stability.
Migration guide: See docs/migration/v1-to-v2.md
"""
from typing import Annotated
@@ -25,7 +32,11 @@ from basic_memory.schemas import (
from basic_memory.schemas.request import EditEntityRequest, MoveEntityRequest
from basic_memory.schemas.base import Permalink, Entity
router = APIRouter(prefix="/knowledge", tags=["knowledge"])
router = APIRouter(
prefix="/knowledge",
tags=["knowledge"],
deprecated=True, # Marks entire router as deprecated in OpenAPI docs
)
async def resolve_relations_background(sync_service, entity_id: int, entity_permalink: str) -> None:
+50 -8
View File
@@ -50,6 +50,7 @@ async def get_project(
) # pragma: no cover
return ProjectItem(
id=found_project.id,
name=found_project.name,
path=normalize_project_path(found_project.path),
is_default=found_project.is_default or False,
@@ -80,9 +81,17 @@ async def update_project(
raise HTTPException(status_code=400, detail="Path must be absolute")
# Get original project info for the response
old_project = await project_service.get_project(name)
if not old_project:
raise HTTPException(
status_code=400, detail=f"Project '{name}' not found in configuration"
)
old_project_info = ProjectItem(
name=name,
path=project_service.projects.get(name, ""),
id=old_project.id,
name=old_project.name,
path=old_project.path,
is_default=old_project.is_default or False,
)
if path:
@@ -91,14 +100,21 @@ async def update_project(
await project_service.update_project(name, is_active=is_active)
# Get updated project info
updated_path = path if path else project_service.projects.get(name, "")
updated_project = await project_service.get_project(name)
if not updated_project:
raise HTTPException(status_code=404, detail=f"Project '{name}' not found after update")
return ProjectStatusResponse(
message=f"Project '{name}' updated successfully",
status="success",
default=(name == project_service.default_project),
old_project=old_project_info,
new_project=ProjectItem(name=name, path=updated_path),
new_project=ProjectItem(
id=updated_project.id,
name=updated_project.name,
path=updated_project.path,
is_default=updated_project.is_default or False,
),
)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
@@ -186,6 +202,7 @@ async def list_projects(
project_items = [
ProjectItem(
id=project.id,
name=project.name,
path=normalize_project_path(project.path),
is_default=project.is_default or False,
@@ -232,6 +249,7 @@ async def add_project(
status="success",
default=existing_project.is_default or False,
new_project=ProjectItem(
id=existing_project.id,
name=existing_project.name,
path=existing_project.path,
is_default=existing_project.is_default or False,
@@ -250,12 +268,20 @@ async def add_project(
project_data.name, project_data.path, set_default=project_data.set_default
)
# Fetch the newly created project to get its ID
new_project = await project_service.get_project(project_data.name)
if not new_project:
raise HTTPException(status_code=500, detail="Failed to retrieve newly created project")
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
message=f"Project '{project_data.name}' added successfully",
status="success",
default=project_data.set_default,
new_project=ProjectItem(
name=project_data.name, path=project_data.path, is_default=project_data.set_default
id=new_project.id,
name=new_project.name,
path=new_project.path,
is_default=new_project.is_default or False,
),
)
except ValueError as e: # pragma: no cover
@@ -306,7 +332,12 @@ async def remove_project(
message=f"Project '{name}' removed successfully",
status="success",
default=False,
old_project=ProjectItem(name=old_project.name, path=old_project.path),
old_project=ProjectItem(
id=old_project.id,
name=old_project.name,
path=old_project.path,
is_default=old_project.is_default or False,
),
new_project=None,
)
except ValueError as e: # pragma: no cover
@@ -349,8 +380,14 @@ async def set_default_project(
message=f"Project '{name}' set as default successfully",
status="success",
default=True,
old_project=ProjectItem(name=default_name, path=default_project.path),
old_project=ProjectItem(
id=default_project.id,
name=default_name,
path=default_project.path,
is_default=False,
),
new_project=ProjectItem(
id=new_default_project.id,
name=name,
path=new_default_project.path,
is_default=True,
@@ -378,7 +415,12 @@ async def get_default_project(
status_code=404, detail=f"Default Project: '{default_name}' does not exist"
)
return ProjectItem(name=default_project.name, path=default_project.path, is_default=True)
return ProjectItem(
id=default_project.id,
name=default_project.name,
path=default_project.path,
is_default=True,
)
# Synchronize projects between config and database
+35 -25
View File
@@ -2,9 +2,9 @@
import tempfile
from pathlib import Path
from typing import Annotated
from typing import Annotated, Union
from fastapi import APIRouter, HTTPException, BackgroundTasks, Body
from fastapi import APIRouter, HTTPException, BackgroundTasks, Body, Response
from fastapi.responses import FileResponse, JSONResponse
from loguru import logger
@@ -25,6 +25,17 @@ from datetime import datetime
router = APIRouter(prefix="/resource", tags=["resources"])
def _mtime_to_datetime(entity: EntityModel) -> datetime:
"""Convert entity mtime (file modification time) to datetime.
Returns the file's actual modification time, falling back to updated_at
if mtime is not available.
"""
if entity.mtime:
return datetime.fromtimestamp(entity.mtime).astimezone()
return entity.updated_at
def get_entity_ids(item: SearchIndexRow) -> set[int]:
match item.type:
case SearchItemType.ENTITY:
@@ -39,7 +50,7 @@ def get_entity_ids(item: SearchIndexRow) -> set[int]:
raise ValueError(f"Unexpected type: {item.type}")
@router.get("/{identifier:path}")
@router.get("/{identifier:path}", response_model=None)
async def get_resource_content(
config: ProjectConfigDep,
link_resolver: LinkResolverDep,
@@ -50,7 +61,7 @@ async def get_resource_content(
identifier: str,
page: int = 1,
page_size: int = 10,
) -> FileResponse:
) -> Union[Response, FileResponse]:
"""Get resource content by identifier: name or permalink."""
logger.debug(f"Getting content for: {identifier}")
@@ -81,13 +92,16 @@ async def get_resource_content(
# return single response
if len(results) == 1:
entity = results[0]
file_path = Path(f"{config.home}/{entity.file_path}")
if not file_path.exists():
# Check file exists via file_service (for cloud compatibility)
if not await file_service.exists(entity.file_path):
raise HTTPException(
status_code=404,
detail=f"File not found: {file_path}",
detail=f"File not found: {entity.file_path}",
)
return FileResponse(path=file_path)
# Read content via file_service as bytes (works with both local and S3)
content = await file_service.read_file_bytes(entity.file_path)
content_type = file_service.content_type(entity.file_path)
return Response(content=content, media_type=content_type)
# for multiple files, initialize a temporary file for writing the results
with tempfile.NamedTemporaryFile(delete=False, mode="w", suffix=".md") as tmp_file:
@@ -97,7 +111,7 @@ async def get_resource_content(
# Read content for each entity
content = await file_service.read_entity_content(result)
memory_url = normalize_memory_url(result.permalink)
modified_date = result.updated_at.isoformat()
modified_date = _mtime_to_datetime(result).isoformat()
checksum = result.checksum[:8] if result.checksum else ""
# Prepare the delimited content
@@ -171,21 +185,17 @@ async def write_resource(
else:
content_str = str(content)
# Get full file path
full_path = Path(f"{config.home}/{file_path}")
# Ensure parent directory exists
full_path.parent.mkdir(parents=True, exist_ok=True)
# Write content to file
checksum = await file_service.write_file(full_path, content_str)
# Cloud compatibility: do not assume a local filesystem path structure.
# Delegate directory creation + writes to the configured FileService (local or S3).
await file_service.ensure_directory(Path(file_path).parent)
checksum = await file_service.write_file(file_path, content_str)
# Get file info
file_stats = file_service.file_stats(full_path)
file_metadata = await file_service.get_file_metadata(file_path)
# Determine file details
file_name = Path(file_path).name
content_type = file_service.content_type(full_path)
content_type = file_service.content_type(file_path)
entity_type = "canvas" if file_path.endswith(".canvas") else "file"
@@ -202,7 +212,7 @@ async def write_resource(
"content_type": content_type,
"file_path": file_path,
"checksum": checksum,
"updated_at": datetime.fromtimestamp(file_stats.st_mtime).astimezone(),
"updated_at": file_metadata.modified_at,
},
)
status_code = 200
@@ -214,8 +224,8 @@ async def write_resource(
content_type=content_type,
file_path=file_path,
checksum=checksum,
created_at=datetime.fromtimestamp(file_stats.st_ctime).astimezone(),
updated_at=datetime.fromtimestamp(file_stats.st_mtime).astimezone(),
created_at=file_metadata.created_at,
updated_at=file_metadata.modified_at,
)
entity = await entity_repository.add(entity)
status_code = 201
@@ -229,9 +239,9 @@ async def write_resource(
content={
"file_path": file_path,
"checksum": checksum,
"size": file_stats.st_size,
"created_at": file_stats.st_ctime,
"modified_at": file_stats.st_mtime,
"size": file_metadata.size,
"created_at": file_metadata.created_at.timestamp(),
"modified_at": file_metadata.modified_at.timestamp(),
},
)
except Exception as e: # pragma: no cover
+53 -14
View File
@@ -24,11 +24,30 @@ async def to_graph_context(
page: Optional[int] = None,
page_size: Optional[int] = None,
):
# First pass: collect all entity IDs needed for relations
entity_ids_needed: set[int] = set()
for context_item in context_result.results:
for item in (
[context_item.primary_result] + context_item.observations + context_item.related_results
):
if item.type == SearchItemType.RELATION:
if item.from_id: # pyright: ignore
entity_ids_needed.add(item.from_id) # pyright: ignore
if item.to_id:
entity_ids_needed.add(item.to_id)
# Batch fetch all entities at once
entity_lookup: dict[int, str] = {}
if entity_ids_needed:
entities = await entity_repository.find_by_ids(list(entity_ids_needed))
entity_lookup = {e.id: e.title for e in entities}
# Helper function to convert items to summaries
async def to_summary(item: SearchIndexRow | ContextResultRow):
def to_summary(item: SearchIndexRow | ContextResultRow):
match item.type:
case SearchItemType.ENTITY:
return EntitySummary(
entity_id=item.id,
title=item.title, # pyright: ignore
permalink=item.permalink,
content=item.content,
@@ -37,6 +56,8 @@ async def to_graph_context(
)
case SearchItemType.OBSERVATION:
return ObservationSummary(
observation_id=item.id,
entity_id=item.entity_id, # pyright: ignore
title=item.title, # pyright: ignore
file_path=item.file_path,
category=item.category, # pyright: ignore
@@ -45,15 +66,19 @@ async def to_graph_context(
created_at=item.created_at,
)
case SearchItemType.RELATION:
from_entity = await entity_repository.find_by_id(item.from_id) # pyright: ignore
to_entity = await entity_repository.find_by_id(item.to_id) if item.to_id else None
from_title = entity_lookup.get(item.from_id) if item.from_id else None # pyright: ignore
to_title = entity_lookup.get(item.to_id) if item.to_id else None
return RelationSummary(
relation_id=item.id,
entity_id=item.entity_id, # pyright: ignore
title=item.title, # pyright: ignore
file_path=item.file_path,
permalink=item.permalink, # pyright: ignore
relation_type=item.relation_type, # pyright: ignore
from_entity=from_entity.title if from_entity else None,
to_entity=to_entity.title if to_entity else None,
from_entity=from_title,
from_entity_id=item.from_id, # pyright: ignore
to_entity=to_title,
to_entity_id=item.to_id,
created_at=item.created_at,
)
case _: # pragma: no cover
@@ -63,23 +88,19 @@ async def to_graph_context(
hierarchical_results = []
for context_item in context_result.results:
# Process primary result
primary_result = await to_summary(context_item.primary_result)
primary_result = to_summary(context_item.primary_result)
# Process observations
observations = []
for obs in context_item.observations:
observations.append(await to_summary(obs))
# Process observations (always ObservationSummary, validated by context_service)
observations = [to_summary(obs) for obs in context_item.observations]
# Process related results
related = []
for rel in context_item.related_results:
related.append(await to_summary(rel))
related = [to_summary(rel) for rel in context_item.related_results]
# Add to hierarchical results
hierarchical_results.append(
ContextResult(
primary_result=primary_result,
observations=observations,
observations=observations, # pyright: ignore[reportArgumentType]
related_results=related,
)
)
@@ -111,6 +132,21 @@ async def to_search_results(entity_service: EntityService, results: List[SearchI
search_results = []
for r in results:
entities = await entity_service.get_entities_by_id([r.entity_id, r.from_id, r.to_id]) # pyright: ignore
# Determine which IDs to set based on type
entity_id = None
observation_id = None
relation_id = None
if r.type == SearchItemType.ENTITY:
entity_id = r.id
elif r.type == SearchItemType.OBSERVATION:
observation_id = r.id
entity_id = r.entity_id # Parent entity
elif r.type == SearchItemType.RELATION:
relation_id = r.id
entity_id = r.entity_id # Parent entity
search_results.append(
SearchResult(
title=r.title, # pyright: ignore
@@ -121,6 +157,9 @@ async def to_search_results(entity_service: EntityService, results: List[SearchI
content=r.content,
file_path=r.file_path,
metadata=r.metadata,
entity_id=entity_id,
observation_id=observation_id,
relation_id=relation_id,
category=r.category,
from_entity=entities[0].permalink if entities else None,
to_entity=entities[1].permalink if len(entities) > 1 else None,
+35
View File
@@ -0,0 +1,35 @@
"""API v2 module - ID-based entity references.
Version 2 of the Basic Memory API uses integer entity IDs as the primary
identifier for improved performance and stability.
Key changes from v1:
- Entity lookups use integer IDs instead of paths/permalinks
- Direct database queries instead of cascading resolution
- Stable references that don't change with file moves
- Better caching support
All v2 routers are registered with the /v2 prefix.
"""
from basic_memory.api.v2.routers import (
knowledge_router,
memory_router,
project_router,
resource_router,
search_router,
directory_router,
prompt_router,
importer_router,
)
__all__ = [
"knowledge_router",
"memory_router",
"project_router",
"resource_router",
"search_router",
"directory_router",
"prompt_router",
"importer_router",
]
@@ -0,0 +1,21 @@
"""V2 API routers."""
from basic_memory.api.v2.routers.knowledge_router import router as knowledge_router
from basic_memory.api.v2.routers.project_router import router as project_router
from basic_memory.api.v2.routers.memory_router import router as memory_router
from basic_memory.api.v2.routers.search_router import router as search_router
from basic_memory.api.v2.routers.resource_router import router as resource_router
from basic_memory.api.v2.routers.directory_router import router as directory_router
from basic_memory.api.v2.routers.prompt_router import router as prompt_router
from basic_memory.api.v2.routers.importer_router import router as importer_router
__all__ = [
"knowledge_router",
"project_router",
"memory_router",
"search_router",
"resource_router",
"directory_router",
"prompt_router",
"importer_router",
]
@@ -0,0 +1,93 @@
"""V2 Directory Router - ID-based directory tree operations.
This router provides directory structure browsing for projects using
integer project IDs instead of name-based identifiers.
Key improvements:
- Direct project lookup via integer primary keys
- Consistent with other v2 endpoints
- Better performance through indexed queries
"""
from typing import List, Optional
from fastapi import APIRouter, Query
from basic_memory.deps import DirectoryServiceV2Dep, ProjectIdPathDep
from basic_memory.schemas.directory import DirectoryNode
router = APIRouter(prefix="/directory", tags=["directory-v2"])
@router.get("/tree", response_model=DirectoryNode, response_model_exclude_none=True)
async def get_directory_tree(
directory_service: DirectoryServiceV2Dep,
project_id: ProjectIdPathDep,
):
"""Get hierarchical directory structure from the knowledge base.
Args:
directory_service: Service for directory operations
project_id: Numeric project ID
Returns:
DirectoryNode representing the root of the hierarchical tree structure
"""
# Get a hierarchical directory tree for the specific project
tree = await directory_service.get_directory_tree()
# Return the hierarchical tree
return tree
@router.get("/structure", response_model=DirectoryNode, response_model_exclude_none=True)
async def get_directory_structure(
directory_service: DirectoryServiceV2Dep,
project_id: ProjectIdPathDep,
):
"""Get folder structure for navigation (no files).
Optimized endpoint for folder tree navigation. Returns only directory nodes
without file metadata. For full tree with files, use /directory/tree.
Args:
directory_service: Service for directory operations
project_id: Numeric project ID
Returns:
DirectoryNode tree containing only folders (type="directory")
"""
structure = await directory_service.get_directory_structure()
return structure
@router.get("/list", response_model=List[DirectoryNode], response_model_exclude_none=True)
async def list_directory(
directory_service: DirectoryServiceV2Dep,
project_id: ProjectIdPathDep,
dir_name: str = Query("/", description="Directory path to list"),
depth: int = Query(1, ge=1, le=10, description="Recursion depth (1-10)"),
file_name_glob: Optional[str] = Query(
None, description="Glob pattern for filtering file names"
),
):
"""List directory contents with filtering and depth control.
Args:
directory_service: Service for directory operations
project_id: Numeric project ID
dir_name: Directory path to list (default: root "/")
depth: Recursion depth (1-10, default: 1 for immediate children only)
file_name_glob: Optional glob pattern for filtering file names (e.g., "*.md", "*meeting*")
Returns:
List of DirectoryNode objects matching the criteria
"""
# Get directory listing with filtering
nodes = await directory_service.list_directory(
dir_name=dir_name,
depth=depth,
file_name_glob=file_name_glob,
)
return nodes
@@ -0,0 +1,182 @@
"""V2 Import Router - ID-based data import operations.
This router uses v2 dependencies for consistent project ID handling.
Import endpoints use project_id in the path for consistency with other v2 endpoints.
"""
import json
import logging
from fastapi import APIRouter, Form, HTTPException, UploadFile, status
from basic_memory.deps import (
ChatGPTImporterV2Dep,
ClaudeConversationsImporterV2Dep,
ClaudeProjectsImporterV2Dep,
MemoryJsonImporterV2Dep,
ProjectIdPathDep,
)
from basic_memory.importers import Importer
from basic_memory.schemas.importer import (
ChatImportResult,
EntityImportResult,
ProjectImportResult,
)
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/import", tags=["import-v2"])
@router.post("/chatgpt", response_model=ChatImportResult)
async def import_chatgpt(
project_id: ProjectIdPathDep,
importer: ChatGPTImporterV2Dep,
file: UploadFile,
folder: str = Form("conversations"),
) -> ChatImportResult:
"""Import conversations from ChatGPT JSON export.
Args:
project_id: Validated numeric project ID from URL path
file: The ChatGPT conversations.json file.
folder: The folder to place the files in.
importer: ChatGPT importer instance.
Returns:
ChatImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
logger.info(f"V2 Importing ChatGPT conversations for project {project_id}")
return await import_file(importer, file, folder)
@router.post("/claude/conversations", response_model=ChatImportResult)
async def import_claude_conversations(
project_id: ProjectIdPathDep,
importer: ClaudeConversationsImporterV2Dep,
file: UploadFile,
folder: str = Form("conversations"),
) -> ChatImportResult:
"""Import conversations from Claude conversations.json export.
Args:
project_id: Validated numeric project ID from URL path
file: The Claude conversations.json file.
folder: The folder to place the files in.
importer: Claude conversations importer instance.
Returns:
ChatImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
logger.info(f"V2 Importing Claude conversations for project {project_id}")
return await import_file(importer, file, folder)
@router.post("/claude/projects", response_model=ProjectImportResult)
async def import_claude_projects(
project_id: ProjectIdPathDep,
importer: ClaudeProjectsImporterV2Dep,
file: UploadFile,
folder: str = Form("projects"),
) -> ProjectImportResult:
"""Import projects from Claude projects.json export.
Args:
project_id: Validated numeric project ID from URL path
file: The Claude projects.json file.
folder: The base folder to place the files in.
importer: Claude projects importer instance.
Returns:
ProjectImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
logger.info(f"V2 Importing Claude projects for project {project_id}")
return await import_file(importer, file, folder)
@router.post("/memory-json", response_model=EntityImportResult)
async def import_memory_json(
project_id: ProjectIdPathDep,
importer: MemoryJsonImporterV2Dep,
file: UploadFile,
folder: str = Form("conversations"),
) -> EntityImportResult:
"""Import entities and relations from a memory.json file.
Args:
project_id: Validated numeric project ID from URL path
file: The memory.json file.
folder: Optional destination folder within the project.
importer: Memory JSON importer instance.
Returns:
EntityImportResult with import statistics.
Raises:
HTTPException: If import fails.
"""
logger.info(f"V2 Importing memory.json for project {project_id}")
try:
file_data = []
file_bytes = await file.read()
file_str = file_bytes.decode("utf-8")
for line in file_str.splitlines():
json_data = json.loads(line)
file_data.append(json_data)
result = await importer.import_data(file_data, folder)
if not result.success: # pragma: no cover
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=result.error_message or "Import failed",
)
except Exception as e:
logger.exception("V2 Import failed")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Import failed: {str(e)}",
)
return result
async def import_file(importer: Importer, file: UploadFile, destination_folder: str):
"""Helper function to import a file using an importer instance.
Args:
importer: The importer instance to use
file: The file to import
destination_folder: Destination folder for imported content
Returns:
Import result from the importer
Raises:
HTTPException: If import fails
"""
try:
# Process file
json_data = json.load(file.file)
result = await importer.import_data(json_data, destination_folder)
if not result.success: # pragma: no cover
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=result.error_message or "Import failed",
)
return result
except Exception as e:
logger.exception("V2 Import failed")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Import failed: {str(e)}",
)
@@ -0,0 +1,415 @@
"""V2 Knowledge Router - ID-based entity operations.
This router provides ID-based CRUD operations for entities, replacing the
path-based identifiers used in v1 with direct integer ID lookups.
Key improvements:
- Direct database lookups via integer primary keys
- Stable references that don't change with file moves
- Better performance through indexed queries
- Simplified caching strategies
"""
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Response
from loguru import logger
from basic_memory.deps import (
EntityServiceV2Dep,
SearchServiceV2Dep,
LinkResolverV2Dep,
ProjectConfigV2Dep,
AppConfigDep,
SyncServiceV2Dep,
EntityRepositoryV2Dep,
ProjectIdPathDep,
)
from basic_memory.schemas import DeleteEntitiesResponse
from basic_memory.schemas.base import Entity
from basic_memory.schemas.request import EditEntityRequest
from basic_memory.schemas.v2 import (
EntityResolveRequest,
EntityResolveResponse,
EntityResponseV2,
MoveEntityRequestV2,
)
router = APIRouter(prefix="/knowledge", tags=["knowledge-v2"])
async def resolve_relations_background(sync_service, entity_id: int, entity_permalink: str) -> None:
"""Background task to resolve relations for a specific entity.
This runs asynchronously after the API response is sent, preventing
long delays when creating entities with many relations.
"""
try:
# Only resolve relations for the newly created entity
await sync_service.resolve_relations(entity_id=entity_id)
logger.debug(
f"Background: Resolved relations for entity {entity_permalink} (id={entity_id})"
)
except Exception as e:
# Log but don't fail - this is a background task
logger.warning(
f"Background: Failed to resolve relations for entity {entity_permalink}: {e}"
)
## Resolution endpoint
@router.post("/resolve", response_model=EntityResolveResponse)
async def resolve_identifier(
project_id: ProjectIdPathDep,
data: EntityResolveRequest,
link_resolver: LinkResolverV2Dep,
) -> EntityResolveResponse:
"""Resolve a string identifier (permalink, title, or path) to an entity ID.
This endpoint provides a bridge between v1-style identifiers and v2 entity IDs.
Use this to convert existing references to the new ID-based format.
Args:
data: Request containing the identifier to resolve
Returns:
Entity ID and metadata about how it was resolved
Raises:
HTTPException: 404 if identifier cannot be resolved
Example:
POST /v2/{project}/knowledge/resolve
{"identifier": "specs/search"}
Returns:
{
"entity_id": 123,
"permalink": "specs/search",
"file_path": "specs/search.md",
"title": "Search Specification",
"resolution_method": "permalink"
}
"""
logger.info(f"API v2 request: resolve_identifier for '{data.identifier}'")
# Try to resolve the identifier
entity = await link_resolver.resolve_link(data.identifier)
if not entity:
raise HTTPException(
status_code=404, detail=f"Could not resolve identifier: '{data.identifier}'"
)
# Determine resolution method
resolution_method = "search" # default
if data.identifier.isdigit():
resolution_method = "id"
elif entity.permalink == data.identifier:
resolution_method = "permalink"
elif entity.title == data.identifier:
resolution_method = "title"
elif entity.file_path == data.identifier:
resolution_method = "path"
result = EntityResolveResponse(
entity_id=entity.id,
permalink=entity.permalink,
file_path=entity.file_path,
title=entity.title,
resolution_method=resolution_method,
)
logger.info(
f"API v2 response: resolved '{data.identifier}' to entity_id={result.entity_id} via {resolution_method}"
)
return result
## Read endpoints
@router.get("/entities/{entity_id}", response_model=EntityResponseV2)
async def get_entity_by_id(
project_id: ProjectIdPathDep,
entity_id: int,
entity_repository: EntityRepositoryV2Dep,
) -> EntityResponseV2:
"""Get an entity by its numeric ID.
This is the primary entity retrieval method in v2, using direct database
lookups for maximum performance.
Args:
entity_id: Numeric entity ID
Returns:
Complete entity with observations and relations
Raises:
HTTPException: 404 if entity not found
"""
logger.info(f"API v2 request: get_entity_by_id entity_id={entity_id}")
entity = await entity_repository.get_by_id(entity_id)
if not entity:
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
result = EntityResponseV2.model_validate(entity)
logger.info(f"API v2 response: entity_id={entity_id}, title='{result.title}'")
return result
## Create endpoints
@router.post("/entities", response_model=EntityResponseV2)
async def create_entity(
project_id: ProjectIdPathDep,
data: Entity,
background_tasks: BackgroundTasks,
entity_service: EntityServiceV2Dep,
search_service: SearchServiceV2Dep,
) -> EntityResponseV2:
"""Create a new entity.
Args:
data: Entity data to create
Returns:
Created entity with generated ID
"""
logger.info(
"API v2 request", endpoint="create_entity", entity_type=data.entity_type, title=data.title
)
entity = await entity_service.create_entity(data)
# reindex
await search_service.index_entity(entity, background_tasks=background_tasks)
result = EntityResponseV2.model_validate(entity)
logger.info(
f"API v2 response: endpoint='create_entity' id={entity.id}, title={result.title}, permalink={result.permalink}, status_code=201"
)
return result
## Update endpoints
@router.put("/entities/{entity_id}", response_model=EntityResponseV2)
async def update_entity_by_id(
project_id: ProjectIdPathDep,
entity_id: int,
data: Entity,
response: Response,
background_tasks: BackgroundTasks,
entity_service: EntityServiceV2Dep,
search_service: SearchServiceV2Dep,
sync_service: SyncServiceV2Dep,
entity_repository: EntityRepositoryV2Dep,
) -> EntityResponseV2:
"""Update an entity by ID.
If the entity doesn't exist, it will be created (upsert behavior).
Args:
entity_id: Numeric entity ID
data: Updated entity data
Returns:
Updated entity
"""
logger.info(f"API v2 request: update_entity_by_id entity_id={entity_id}")
# Check if entity exists
existing = await entity_repository.get_by_id(entity_id)
created = existing is None
# Perform update or create
entity, _ = await entity_service.create_or_update_entity(data)
response.status_code = 201 if created else 200
# reindex
await search_service.index_entity(entity, background_tasks=background_tasks)
# Schedule relation resolution for new entities
if created:
background_tasks.add_task(
resolve_relations_background, sync_service, entity.id, entity.permalink or ""
)
result = EntityResponseV2.model_validate(entity)
logger.info(
f"API v2 response: entity_id={entity_id}, created={created}, status_code={response.status_code}"
)
return result
@router.patch("/entities/{entity_id}", response_model=EntityResponseV2)
async def edit_entity_by_id(
project_id: ProjectIdPathDep,
entity_id: int,
data: EditEntityRequest,
background_tasks: BackgroundTasks,
entity_service: EntityServiceV2Dep,
search_service: SearchServiceV2Dep,
entity_repository: EntityRepositoryV2Dep,
) -> EntityResponseV2:
"""Edit an existing entity by ID using operations like append, prepend, etc.
Args:
entity_id: Numeric entity ID
data: Edit operation details
Returns:
Updated entity
Raises:
HTTPException: 404 if entity not found, 400 if edit fails
"""
logger.info(
f"API v2 request: edit_entity_by_id entity_id={entity_id}, operation='{data.operation}'"
)
# Verify entity exists
entity = await entity_repository.get_by_id(entity_id)
if not entity:
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
try:
# Edit using the entity's permalink or path
identifier = entity.permalink or entity.file_path
updated_entity = await entity_service.edit_entity(
identifier=identifier,
operation=data.operation,
content=data.content,
section=data.section,
find_text=data.find_text,
expected_replacements=data.expected_replacements,
)
# Reindex
await search_service.index_entity(updated_entity, background_tasks=background_tasks)
result = EntityResponseV2.model_validate(updated_entity)
logger.info(
f"API v2 response: entity_id={entity_id}, operation='{data.operation}', status_code=200"
)
return result
except Exception as e:
logger.error(f"Error editing entity {entity_id}: {e}")
raise HTTPException(status_code=400, detail=str(e))
## Delete endpoints
@router.delete("/entities/{entity_id}", response_model=DeleteEntitiesResponse)
async def delete_entity_by_id(
project_id: ProjectIdPathDep,
entity_id: int,
background_tasks: BackgroundTasks,
entity_service: EntityServiceV2Dep,
entity_repository: EntityRepositoryV2Dep,
search_service=Depends(lambda: None), # Optional for now
) -> DeleteEntitiesResponse:
"""Delete an entity by ID.
Args:
entity_id: Numeric entity ID
Returns:
Deletion status
Note: Returns deleted=False if entity doesn't exist (idempotent)
"""
logger.info(f"API v2 request: delete_entity_by_id entity_id={entity_id}")
entity = await entity_repository.get_by_id(entity_id)
if entity is None:
logger.info(f"API v2 response: entity_id={entity_id} not found, deleted=False")
return DeleteEntitiesResponse(deleted=False)
# Delete the entity
deleted = await entity_service.delete_entity(entity_id)
# Remove from search index if search service available
if search_service:
background_tasks.add_task(search_service.handle_delete, entity)
logger.info(f"API v2 response: entity_id={entity_id}, deleted={deleted}")
return DeleteEntitiesResponse(deleted=deleted)
## Move endpoint
@router.put("/entities/{entity_id}/move", response_model=EntityResponseV2)
async def move_entity(
project_id: ProjectIdPathDep,
entity_id: int,
data: MoveEntityRequestV2,
background_tasks: BackgroundTasks,
entity_service: EntityServiceV2Dep,
entity_repository: EntityRepositoryV2Dep,
project_config: ProjectConfigV2Dep,
app_config: AppConfigDep,
search_service: SearchServiceV2Dep,
) -> EntityResponseV2:
"""Move an entity to a new file location.
V2 API uses entity ID in the URL path for stable references.
The entity ID will remain stable after the move.
Args:
project_id: Project ID from URL path
entity_id: Entity ID from URL path (primary identifier)
data: Move request with destination path only
Returns:
Updated entity with new file path
"""
logger.info(
f"API v2 request: move_entity entity_id={entity_id}, destination='{data.destination_path}'"
)
try:
# First, get the entity by ID to verify it exists
entity = await entity_repository.find_by_id(entity_id)
if not entity:
raise HTTPException(status_code=404, detail=f"Entity not found: {entity_id}")
# Move the entity using its current file path as identifier
moved_entity = await entity_service.move_entity(
identifier=entity.file_path, # Use file path for resolution
destination_path=data.destination_path,
project_config=project_config,
app_config=app_config,
)
# Reindex at new location
reindexed_entity = await entity_service.link_resolver.resolve_link(data.destination_path)
if reindexed_entity:
await search_service.index_entity(reindexed_entity, background_tasks=background_tasks)
result = EntityResponseV2.model_validate(moved_entity)
logger.info(
f"API v2 response: moved entity_id={moved_entity.id} to '{data.destination_path}'"
)
return result
except HTTPException:
raise
except Exception as e:
logger.error(f"Error moving entity: {e}")
raise HTTPException(status_code=400, detail=str(e))
@@ -0,0 +1,130 @@
"""V2 routes for memory:// URI operations.
This router uses integer project IDs for stable, efficient routing.
V1 uses string-based project names which are less efficient and less stable.
"""
from typing import Annotated, Optional
from fastapi import APIRouter, Query
from loguru import logger
from basic_memory.deps import ContextServiceV2Dep, EntityRepositoryV2Dep, ProjectIdPathDep
from basic_memory.schemas.base import TimeFrame, parse_timeframe
from basic_memory.schemas.memory import (
GraphContext,
normalize_memory_url,
)
from basic_memory.schemas.search import SearchItemType
from basic_memory.api.routers.utils import to_graph_context
# Note: No prefix here - it's added during registration as /v2/{project_id}/memory
router = APIRouter(tags=["memory"])
@router.get("/memory/recent", response_model=GraphContext)
async def recent(
project_id: ProjectIdPathDep,
context_service: ContextServiceV2Dep,
entity_repository: EntityRepositoryV2Dep,
type: Annotated[list[SearchItemType] | None, Query()] = None,
depth: int = 1,
timeframe: TimeFrame = "7d",
page: int = 1,
page_size: int = 10,
max_related: int = 10,
) -> GraphContext:
"""Get recent activity context for a project.
Args:
project_id: Validated numeric project ID from URL path
context_service: Context service scoped to project
entity_repository: Entity repository scoped to project
type: Types of items to include (entities, relations, observations)
depth: How many levels of related entities to include
timeframe: Time window for recent activity (e.g., "7d", "1 week")
page: Page number for pagination
page_size: Number of items per page
max_related: Maximum related entities to include per item
Returns:
GraphContext with recent activity and related entities
"""
# return all types by default
types = (
[SearchItemType.ENTITY, SearchItemType.RELATION, SearchItemType.OBSERVATION]
if not type
else type
)
logger.debug(
f"V2 Getting recent context for project {project_id}: `{types}` depth: `{depth}` timeframe: `{timeframe}` page: `{page}` page_size: `{page_size}` max_related: `{max_related}`"
)
# Parse timeframe
since = parse_timeframe(timeframe)
limit = page_size
offset = (page - 1) * page_size
# Build context
context = await context_service.build_context(
types=types, depth=depth, since=since, limit=limit, offset=offset, max_related=max_related
)
recent_context = await to_graph_context(
context, entity_repository=entity_repository, page=page, page_size=page_size
)
logger.debug(f"V2 Recent context: {recent_context.model_dump_json()}")
return recent_context
# get_memory_context needs to be declared last so other paths can match
@router.get("/memory/{uri:path}", response_model=GraphContext)
async def get_memory_context(
project_id: ProjectIdPathDep,
context_service: ContextServiceV2Dep,
entity_repository: EntityRepositoryV2Dep,
uri: str,
depth: int = 1,
timeframe: Optional[TimeFrame] = None,
page: int = 1,
page_size: int = 10,
max_related: int = 10,
) -> GraphContext:
"""Get rich context from memory:// URI.
V2 supports both legacy path-based URIs and new ID-based URIs:
- Legacy: memory://path/to/note
- ID-based: memory://id/123 or memory://123
Args:
project_id: Validated numeric project ID from URL path
context_service: Context service scoped to project
entity_repository: Entity repository scoped to project
uri: Memory URI path (e.g., "id/123", "123", or "path/to/note")
depth: How many levels of related entities to include
timeframe: Optional time window for filtering related content
page: Page number for pagination
page_size: Number of items per page
max_related: Maximum related entities to include
Returns:
GraphContext with the entity and its related context
"""
logger.debug(
f"V2 Getting context for project {project_id}, URI: `{uri}` depth: `{depth}` timeframe: `{timeframe}` page: `{page}` page_size: `{page_size}` max_related: `{max_related}`"
)
memory_url = normalize_memory_url(uri)
# Parse timeframe
since = parse_timeframe(timeframe) if timeframe else None
limit = page_size
offset = (page - 1) * page_size
# Build context
context = await context_service.build_context(
memory_url, depth=depth, since=since, limit=limit, offset=offset, max_related=max_related
)
return await to_graph_context(
context, entity_repository=entity_repository, page=page, page_size=page_size
)
@@ -0,0 +1,264 @@
"""V2 Project Router - ID-based project management operations.
This router provides ID-based CRUD operations for projects, replacing the
name-based identifiers used in v1 with direct integer ID lookups.
Key improvements:
- Direct database lookups via integer primary keys
- Stable references that don't change with project renames
- Better performance through indexed queries
- Consistent with v2 entity operations
"""
import os
from typing import Optional
from fastapi import APIRouter, HTTPException, Body, Query
from loguru import logger
from basic_memory.deps import (
ProjectServiceDep,
ProjectRepositoryDep,
ProjectIdPathDep,
)
from basic_memory.schemas.project_info import (
ProjectItem,
ProjectStatusResponse,
)
from basic_memory.utils import normalize_project_path
router = APIRouter(prefix="/projects", tags=["project_management-v2"])
@router.get("/{project_id}", response_model=ProjectItem)
async def get_project_by_id(
project_id: ProjectIdPathDep,
project_repository: ProjectRepositoryDep,
) -> ProjectItem:
"""Get project by its numeric ID.
This is the primary project retrieval method in v2, using direct database
lookups for maximum performance.
Args:
project_id: Numeric project ID
Returns:
Project information
Raises:
HTTPException: 404 if project not found
Example:
GET /v2/projects/3
"""
logger.info(f"API v2 request: get_project_by_id for project_id={project_id}")
project = await project_repository.get_by_id(project_id)
if not project:
raise HTTPException(status_code=404, detail=f"Project with ID {project_id} not found")
return ProjectItem(
id=project.id,
name=project.name,
path=normalize_project_path(project.path),
is_default=project.is_default or False,
)
@router.patch("/{project_id}", response_model=ProjectStatusResponse)
async def update_project_by_id(
project_id: ProjectIdPathDep,
project_service: ProjectServiceDep,
project_repository: ProjectRepositoryDep,
path: Optional[str] = Body(None, description="New absolute path for the project"),
is_active: Optional[bool] = Body(None, description="Status of the project (active/inactive)"),
) -> ProjectStatusResponse:
"""Update a project's information by ID.
Args:
project_id: Numeric project ID
path: Optional new absolute path for the project
is_active: Optional status update for the project
Returns:
Response confirming the project was updated
Raises:
HTTPException: 400 if validation fails, 404 if project not found
Example:
PATCH /v2/projects/3
{"path": "/new/path"}
"""
logger.info(f"API v2 request: update_project_by_id for project_id={project_id}")
try:
# Validate that path is absolute if provided
if path and not os.path.isabs(path):
raise HTTPException(status_code=400, detail="Path must be absolute")
# Get original project info for the response
old_project = await project_repository.get_by_id(project_id)
if not old_project:
raise HTTPException(status_code=404, detail=f"Project with ID {project_id} not found")
old_project_info = ProjectItem(
id=old_project.id,
name=old_project.name,
path=old_project.path,
is_default=old_project.is_default or False,
)
# Update using project name (service layer still uses names internally)
if path:
await project_service.move_project(old_project.name, path)
elif is_active is not None:
await project_service.update_project(old_project.name, is_active=is_active)
# Get updated project info
updated_project = await project_repository.get_by_id(project_id)
if not updated_project:
raise HTTPException(
status_code=404, detail=f"Project with ID {project_id} not found after update"
)
return ProjectStatusResponse(
message=f"Project '{updated_project.name}' updated successfully",
status="success",
default=(old_project.name == project_service.default_project),
old_project=old_project_info,
new_project=ProjectItem(
id=updated_project.id,
name=updated_project.name,
path=updated_project.path,
is_default=updated_project.is_default or False,
),
)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
@router.delete("/{project_id}", response_model=ProjectStatusResponse)
async def delete_project_by_id(
project_id: ProjectIdPathDep,
project_service: ProjectServiceDep,
project_repository: ProjectRepositoryDep,
delete_notes: bool = Query(
False, description="If True, delete project directory from filesystem"
),
) -> ProjectStatusResponse:
"""Delete a project by ID.
Args:
project_id: Numeric project ID
delete_notes: If True, delete the project directory from the filesystem
Returns:
Response confirming the project was deleted
Raises:
HTTPException: 400 if trying to delete default project, 404 if not found
Example:
DELETE /v2/projects/3?delete_notes=false
"""
logger.info(
f"API v2 request: delete_project_by_id for project_id={project_id}, delete_notes={delete_notes}"
)
try:
old_project = await project_repository.get_by_id(project_id)
if not old_project:
raise HTTPException(status_code=404, detail=f"Project with ID {project_id} not found")
# Check if trying to delete the default project
if old_project.name == project_service.default_project:
available_projects = await project_service.list_projects()
other_projects = [p.name for p in available_projects if p.id != project_id]
detail = f"Cannot delete default project '{old_project.name}'. "
if other_projects:
detail += (
f"Set another project as default first. Available: {', '.join(other_projects)}"
)
else:
detail += "This is the only project in your configuration."
raise HTTPException(status_code=400, detail=detail)
# Delete using project name (service layer still uses names internally)
await project_service.remove_project(old_project.name, delete_notes=delete_notes)
return ProjectStatusResponse(
message=f"Project '{old_project.name}' removed successfully",
status="success",
default=False,
old_project=ProjectItem(
id=old_project.id,
name=old_project.name,
path=old_project.path,
is_default=old_project.is_default or False,
),
new_project=None,
)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
@router.put("/{project_id}/default", response_model=ProjectStatusResponse)
async def set_default_project_by_id(
project_id: ProjectIdPathDep,
project_service: ProjectServiceDep,
project_repository: ProjectRepositoryDep,
) -> ProjectStatusResponse:
"""Set a project as the default project by ID.
Args:
project_id: Numeric project ID to set as default
Returns:
Response confirming the project was set as default
Raises:
HTTPException: 404 if project not found
Example:
PUT /v2/projects/3/default
"""
logger.info(f"API v2 request: set_default_project_by_id for project_id={project_id}")
try:
# Get the old default project
default_name = project_service.default_project
default_project = await project_service.get_project(default_name)
if not default_project:
raise HTTPException(
status_code=404, detail=f"Default Project: '{default_name}' does not exist"
)
# Get the new default project
new_default_project = await project_repository.get_by_id(project_id)
if not new_default_project:
raise HTTPException(status_code=404, detail=f"Project with ID {project_id} not found")
# Set as default using project name (service layer still uses names internally)
await project_service.set_default_project(new_default_project.name)
return ProjectStatusResponse(
message=f"Project '{new_default_project.name}' set as default successfully",
status="success",
default=True,
old_project=ProjectItem(
id=default_project.id,
name=default_name,
path=default_project.path,
is_default=False,
),
new_project=ProjectItem(
id=new_default_project.id,
name=new_default_project.name,
path=new_default_project.path,
is_default=True,
),
)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
@@ -0,0 +1,270 @@
"""V2 Prompt Router - ID-based prompt generation operations.
This router uses v2 dependencies for consistent project ID handling.
Prompt endpoints are action-based (not resource-based), so they don't
have entity IDs in URLs - they generate formatted prompts from queries.
"""
from datetime import datetime, timezone
from fastapi import APIRouter, HTTPException, status
from loguru import logger
from basic_memory.api.routers.utils import to_graph_context, to_search_results
from basic_memory.api.template_loader import template_loader
from basic_memory.schemas.base import parse_timeframe
from basic_memory.deps import (
ContextServiceV2Dep,
EntityRepositoryV2Dep,
SearchServiceV2Dep,
EntityServiceV2Dep,
ProjectIdPathDep,
)
from basic_memory.schemas.prompt import (
ContinueConversationRequest,
SearchPromptRequest,
PromptResponse,
PromptMetadata,
)
from basic_memory.schemas.search import SearchItemType, SearchQuery
router = APIRouter(prefix="/prompt", tags=["prompt-v2"])
@router.post("/continue-conversation", response_model=PromptResponse)
async def continue_conversation(
project_id: ProjectIdPathDep,
search_service: SearchServiceV2Dep,
entity_service: EntityServiceV2Dep,
context_service: ContextServiceV2Dep,
entity_repository: EntityRepositoryV2Dep,
request: ContinueConversationRequest,
) -> PromptResponse:
"""Generate a prompt for continuing a conversation.
This endpoint takes a topic and/or timeframe and generates a prompt with
relevant context from the knowledge base.
Args:
project_id: Validated numeric project ID from URL path
request: The request parameters
Returns:
Formatted continuation prompt with context
"""
logger.info(
f"V2 Generating continue conversation prompt for project {project_id}, "
f"topic: {request.topic}, timeframe: {request.timeframe}"
)
since = parse_timeframe(request.timeframe) if request.timeframe else None
# Initialize search results
search_results = []
# Get data needed for template
if request.topic:
query = SearchQuery(text=request.topic, after_date=request.timeframe)
results = await search_service.search(query, limit=request.search_items_limit)
search_results = await to_search_results(entity_service, results)
# Build context from results
all_hierarchical_results = []
for result in search_results:
if hasattr(result, "permalink") and result.permalink:
# Get hierarchical context using the new dataclass-based approach
context_result = await context_service.build_context(
result.permalink,
depth=request.depth,
since=since,
max_related=request.related_items_limit,
include_observations=True, # Include observations for entities
)
# Process results into the schema format
graph_context = await to_graph_context(
context_result, entity_repository=entity_repository
)
# Add results to our collection (limit to top results for each permalink)
if graph_context.results:
all_hierarchical_results.extend(graph_context.results[:3])
# Limit to a reasonable number of total results
all_hierarchical_results = all_hierarchical_results[:10]
template_context = {
"topic": request.topic,
"timeframe": request.timeframe,
"hierarchical_results": all_hierarchical_results,
"has_results": len(all_hierarchical_results) > 0,
}
else:
# If no topic, get recent activity
context_result = await context_service.build_context(
types=[SearchItemType.ENTITY],
depth=request.depth,
since=since,
max_related=request.related_items_limit,
include_observations=True,
)
recent_context = await to_graph_context(context_result, entity_repository=entity_repository)
hierarchical_results = recent_context.results[:5] # Limit to top 5 recent items
template_context = {
"topic": f"Recent Activity from ({request.timeframe})",
"timeframe": request.timeframe,
"hierarchical_results": hierarchical_results,
"has_results": len(hierarchical_results) > 0,
}
try:
# Render template
rendered_prompt = await template_loader.render(
"prompts/continue_conversation.hbs", template_context
)
# Calculate metadata
# Count items of different types
observation_count = 0
relation_count = 0
entity_count = 0
# Get the hierarchical results from the template context
hierarchical_results_for_count = template_context.get("hierarchical_results", [])
# For topic-based search
if request.topic:
for item in hierarchical_results_for_count:
if hasattr(item, "observations"):
observation_count += len(item.observations) if item.observations else 0
if hasattr(item, "related_results"):
for related in item.related_results or []:
if hasattr(related, "type"):
if related.type == "relation":
relation_count += 1
elif related.type == "entity": # pragma: no cover
entity_count += 1 # pragma: no cover
# For recent activity
else:
for item in hierarchical_results_for_count:
if hasattr(item, "observations"):
observation_count += len(item.observations) if item.observations else 0
if hasattr(item, "related_results"):
for related in item.related_results or []:
if hasattr(related, "type"):
if related.type == "relation":
relation_count += 1
elif related.type == "entity": # pragma: no cover
entity_count += 1 # pragma: no cover
# Build metadata
metadata = {
"query": request.topic,
"timeframe": request.timeframe,
"search_count": len(search_results)
if request.topic
else 0, # Original search results count
"context_count": len(hierarchical_results_for_count),
"observation_count": observation_count,
"relation_count": relation_count,
"total_items": (
len(hierarchical_results_for_count)
+ observation_count
+ relation_count
+ entity_count
),
"search_limit": request.search_items_limit,
"context_depth": request.depth,
"related_limit": request.related_items_limit,
"generated_at": datetime.now(timezone.utc).isoformat(),
}
prompt_metadata = PromptMetadata(**metadata)
return PromptResponse(
prompt=rendered_prompt, context=template_context, metadata=prompt_metadata
)
except Exception as e:
logger.error(f"Error rendering continue conversation template: {e}")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Error rendering prompt template: {str(e)}",
)
@router.post("/search", response_model=PromptResponse)
async def search_prompt(
project_id: ProjectIdPathDep,
search_service: SearchServiceV2Dep,
entity_service: EntityServiceV2Dep,
request: SearchPromptRequest,
page: int = 1,
page_size: int = 10,
) -> PromptResponse:
"""Generate a prompt for search results.
This endpoint takes a search query and formats the results into a helpful
prompt with context and suggestions.
Args:
project_id: Validated numeric project ID from URL path
request: The search parameters
page: The page number for pagination
page_size: The number of results per page, defaults to 10
Returns:
Formatted search results prompt with context
"""
logger.info(
f"V2 Generating search prompt for project {project_id}, "
f"query: {request.query}, timeframe: {request.timeframe}"
)
limit = page_size
offset = (page - 1) * page_size
query = SearchQuery(text=request.query, after_date=request.timeframe)
results = await search_service.search(query, limit=limit, offset=offset)
search_results = await to_search_results(entity_service, results)
template_context = {
"query": request.query,
"timeframe": request.timeframe,
"results": search_results,
"has_results": len(search_results) > 0,
"result_count": len(search_results),
}
try:
# Render template
rendered_prompt = await template_loader.render("prompts/search.hbs", template_context)
# Build metadata
metadata = {
"query": request.query,
"timeframe": request.timeframe,
"search_count": len(search_results),
"context_count": len(search_results),
"observation_count": 0, # Search results don't include observations
"relation_count": 0, # Search results don't include relations
"total_items": len(search_results),
"search_limit": limit,
"context_depth": 0, # No context depth for basic search
"related_limit": 0, # No related items for basic search
"generated_at": datetime.now(timezone.utc).isoformat(),
}
prompt_metadata = PromptMetadata(**metadata)
return PromptResponse(
prompt=rendered_prompt, context=template_context, metadata=prompt_metadata
)
except Exception as e:
logger.error(f"Error rendering search template: {e}")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Error rendering prompt template: {str(e)}",
)
@@ -0,0 +1,286 @@
"""V2 Resource Router - ID-based resource content operations.
This router uses entity IDs for all operations, with file paths in request bodies
when needed. This is consistent with v2's ID-first design.
Key differences from v1:
- Uses integer entity IDs in URL paths instead of file paths
- File paths are in request bodies for create/update operations
- More RESTful: POST for create, PUT for update, GET for read
"""
from pathlib import Path
from fastapi import APIRouter, HTTPException, Response
from loguru import logger
from basic_memory.deps import (
ProjectConfigV2Dep,
EntityServiceV2Dep,
FileServiceV2Dep,
EntityRepositoryV2Dep,
SearchServiceV2Dep,
ProjectIdPathDep,
)
from basic_memory.models.knowledge import Entity as EntityModel
from basic_memory.schemas.v2.resource import (
CreateResourceRequest,
UpdateResourceRequest,
ResourceResponse,
)
from basic_memory.utils import validate_project_path
router = APIRouter(prefix="/resource", tags=["resources-v2"])
@router.get("/{entity_id}")
async def get_resource_content(
project_id: ProjectIdPathDep,
entity_id: int,
config: ProjectConfigV2Dep,
entity_service: EntityServiceV2Dep,
file_service: FileServiceV2Dep,
) -> Response:
"""Get raw resource content by entity ID.
Args:
project_id: Validated numeric project ID from URL path
entity_id: Numeric entity ID
config: Project configuration
entity_service: Entity service for fetching entity data
file_service: File service for reading file content
Returns:
Response with entity content
Raises:
HTTPException: 404 if entity or file not found
"""
logger.debug(f"V2 Getting content for project {project_id}, entity_id: {entity_id}")
# Get entity by ID
entities = await entity_service.get_entities_by_id([entity_id])
if not entities:
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
entity = entities[0]
# Validate entity file path to prevent path traversal
project_path = Path(config.home)
if not validate_project_path(entity.file_path, project_path):
logger.error(f"Invalid file path in entity {entity.id}: {entity.file_path}")
raise HTTPException(
status_code=500,
detail="Entity contains invalid file path",
)
# Check file exists via file_service (for cloud compatibility)
if not await file_service.exists(entity.file_path):
raise HTTPException(
status_code=404,
detail=f"File not found: {entity.file_path}",
)
# Read content via file_service as bytes (works with both local and S3)
content = await file_service.read_file_bytes(entity.file_path)
content_type = file_service.content_type(entity.file_path)
return Response(content=content, media_type=content_type)
@router.post("", response_model=ResourceResponse)
async def create_resource(
project_id: ProjectIdPathDep,
data: CreateResourceRequest,
config: ProjectConfigV2Dep,
file_service: FileServiceV2Dep,
entity_repository: EntityRepositoryV2Dep,
search_service: SearchServiceV2Dep,
) -> ResourceResponse:
"""Create a new resource file.
Args:
project_id: Validated numeric project ID from URL path
data: Create resource request with file_path and content
config: Project configuration
file_service: File service for writing files
entity_repository: Entity repository for creating entities
search_service: Search service for indexing
Returns:
ResourceResponse with file information including entity_id
Raises:
HTTPException: 400 for invalid file paths, 409 if file already exists
"""
try:
# Validate path to prevent path traversal attacks
project_path = Path(config.home)
if not validate_project_path(data.file_path, project_path):
logger.warning(
f"Invalid file path attempted: {data.file_path} in project {config.name}"
)
raise HTTPException(
status_code=400,
detail=f"Invalid file path: {data.file_path}. "
"Path must be relative and stay within project boundaries.",
)
# Check if entity already exists
existing_entity = await entity_repository.get_by_file_path(data.file_path)
if existing_entity:
raise HTTPException(
status_code=409,
detail=f"Resource already exists at {data.file_path} with entity_id {existing_entity.id}. "
f"Use PUT /resource/{existing_entity.id} to update it.",
)
# Cloud compatibility: avoid assuming a local filesystem path.
# Delegate directory creation + writes to FileService (local or S3).
await file_service.ensure_directory(Path(data.file_path).parent)
checksum = await file_service.write_file(data.file_path, data.content)
# Get file info
file_metadata = await file_service.get_file_metadata(data.file_path)
# Determine file details
file_name = Path(data.file_path).name
content_type = file_service.content_type(data.file_path)
entity_type = "canvas" if data.file_path.endswith(".canvas") else "file"
# Create a new entity model
entity = EntityModel(
title=file_name,
entity_type=entity_type,
content_type=content_type,
file_path=data.file_path,
checksum=checksum,
created_at=file_metadata.created_at,
updated_at=file_metadata.modified_at,
)
entity = await entity_repository.add(entity)
# Index the file for search
await search_service.index_entity(entity) # pyright: ignore
# Return success response
return ResourceResponse(
entity_id=entity.id,
file_path=data.file_path,
checksum=checksum,
size=file_metadata.size,
created_at=file_metadata.created_at.timestamp(),
modified_at=file_metadata.modified_at.timestamp(),
)
except HTTPException:
# Re-raise HTTP exceptions without wrapping
raise
except Exception as e: # pragma: no cover
logger.error(f"Error creating resource {data.file_path}: {e}")
raise HTTPException(status_code=500, detail=f"Failed to create resource: {str(e)}")
@router.put("/{entity_id}", response_model=ResourceResponse)
async def update_resource(
project_id: ProjectIdPathDep,
entity_id: int,
data: UpdateResourceRequest,
config: ProjectConfigV2Dep,
file_service: FileServiceV2Dep,
entity_repository: EntityRepositoryV2Dep,
search_service: SearchServiceV2Dep,
) -> ResourceResponse:
"""Update an existing resource by entity ID.
Can update content and optionally move the file to a new path.
Args:
project_id: Validated numeric project ID from URL path
entity_id: Entity ID of the resource to update
data: Update resource request with content and optional new file_path
config: Project configuration
file_service: File service for writing files
entity_repository: Entity repository for updating entities
search_service: Search service for indexing
Returns:
ResourceResponse with updated file information
Raises:
HTTPException: 404 if entity not found, 400 for invalid paths
"""
try:
# Get existing entity
entity = await entity_repository.get_by_id(entity_id)
if not entity:
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
# Determine target file path
target_file_path = data.file_path if data.file_path else entity.file_path
# Validate path to prevent path traversal attacks
project_path = Path(config.home)
if not validate_project_path(target_file_path, project_path):
logger.warning(
f"Invalid file path attempted: {target_file_path} in project {config.name}"
)
raise HTTPException(
status_code=400,
detail=f"Invalid file path: {target_file_path}. "
"Path must be relative and stay within project boundaries.",
)
# If moving file, handle the move
if data.file_path and data.file_path != entity.file_path:
# Ensure new parent directory exists (no-op for S3)
await file_service.ensure_directory(Path(target_file_path).parent)
# If old file exists, remove it via file_service (for cloud compatibility)
if await file_service.exists(entity.file_path):
await file_service.delete_file(entity.file_path)
else:
# Ensure directory exists for in-place update
await file_service.ensure_directory(Path(target_file_path).parent)
# Write content to target file
checksum = await file_service.write_file(target_file_path, data.content)
# Get file info
file_metadata = await file_service.get_file_metadata(target_file_path)
# Determine file details
file_name = Path(target_file_path).name
content_type = file_service.content_type(target_file_path)
entity_type = "canvas" if target_file_path.endswith(".canvas") else "file"
# Update entity
updated_entity = await entity_repository.update(
entity_id,
{
"title": file_name,
"entity_type": entity_type,
"content_type": content_type,
"file_path": target_file_path,
"checksum": checksum,
"updated_at": file_metadata.modified_at,
},
)
# Index the updated file for search
await search_service.index_entity(updated_entity) # pyright: ignore
# Return success response
return ResourceResponse(
entity_id=entity_id,
file_path=target_file_path,
checksum=checksum,
size=file_metadata.size,
created_at=file_metadata.created_at.timestamp(),
modified_at=file_metadata.modified_at.timestamp(),
)
except HTTPException:
# Re-raise HTTP exceptions without wrapping
raise
except Exception as e: # pragma: no cover
logger.error(f"Error updating resource {entity_id}: {e}")
raise HTTPException(status_code=500, detail=f"Failed to update resource: {str(e)}")
@@ -0,0 +1,73 @@
"""V2 router for search operations.
This router uses integer project IDs for stable, efficient routing.
V1 uses string-based project names which are less efficient and less stable.
"""
from fastapi import APIRouter, BackgroundTasks
from basic_memory.api.routers.utils import to_search_results
from basic_memory.schemas.search import SearchQuery, SearchResponse
from basic_memory.deps import SearchServiceV2Dep, EntityServiceV2Dep, ProjectIdPathDep
# Note: No prefix here - it's added during registration as /v2/{project_id}/search
router = APIRouter(tags=["search"])
@router.post("/search/", response_model=SearchResponse)
async def search(
project_id: ProjectIdPathDep,
query: SearchQuery,
search_service: SearchServiceV2Dep,
entity_service: EntityServiceV2Dep,
page: int = 1,
page_size: int = 10,
):
"""Search across all knowledge and documents in a project.
V2 uses integer project IDs for improved performance and stability.
Args:
project_id: Validated numeric project ID from URL path
query: Search query parameters (text, filters, etc.)
search_service: Search service scoped to project
entity_service: Entity service scoped to project
page: Page number for pagination
page_size: Number of results per page
Returns:
SearchResponse with paginated search results
"""
limit = page_size
offset = (page - 1) * page_size
results = await search_service.search(query, limit=limit, offset=offset)
search_results = await to_search_results(entity_service, results)
return SearchResponse(
results=search_results,
current_page=page,
page_size=page_size,
)
@router.post("/search/reindex")
async def reindex(
project_id: ProjectIdPathDep,
background_tasks: BackgroundTasks,
search_service: SearchServiceV2Dep,
):
"""Recreate and populate the search index for a project.
This is a background operation that rebuilds the search index
from scratch. Useful after bulk updates or if the index becomes
corrupted.
Args:
project_id: Validated numeric project ID from URL path
background_tasks: FastAPI background tasks handler
search_service: Search service scoped to project
Returns:
Status message indicating reindex has been initiated
"""
await search_service.reindex_all(background_tasks=background_tasks)
return {"status": "ok", "message": "Reindex initiated"}
+4 -1
View File
@@ -2,7 +2,7 @@ from typing import Optional
import typer
from basic_memory.config import ConfigManager
from basic_memory.config import ConfigManager, init_cli_logging
def version_callback(value: bool) -> None:
@@ -31,6 +31,9 @@ def app_callback(
) -> None:
"""Basic Memory - Local-first personal knowledge management."""
# Initialize logging for CLI (file only, no stdout)
init_cli_logging()
# Run initialization for every command unless --version was specified
if not version and ctx.invoked_subcommand is not None:
from basic_memory.services.initialization import ensure_initialization
+3 -1
View File
@@ -6,7 +6,7 @@ import typer
from typing import Optional
from basic_memory.cli.app import app
from basic_memory.config import ConfigManager
from basic_memory.config import ConfigManager, init_mcp_logging
# Import mcp instance
from basic_memory.mcp.server import mcp as mcp_server # pragma: no cover
@@ -44,6 +44,8 @@ if not config.cloud_mode_enabled:
- streamable-http: Recommended for web deployments (default)
- sse: Server-Sent Events (for compatibility with existing clients)
"""
# Initialize logging for MCP (file only, stdout breaks protocol)
init_mcp_logging()
# Validate and set project constraint if specified
if project:
+99 -71
View File
@@ -9,10 +9,9 @@ from typing import Any, Dict, Literal, Optional, List, Tuple
from enum import Enum
from loguru import logger
from pydantic import BaseModel, Field, field_validator
from pydantic import BaseModel, Field, model_validator
from pydantic_settings import BaseSettings, SettingsConfigDict
import basic_memory
from basic_memory.utils import setup_logging, generate_permalink
@@ -100,13 +99,32 @@ class BasicMemoryConfig(BaseSettings):
description="Database connection URL. For Postgres, use postgresql+asyncpg://user:pass@host:port/db. If not set, SQLite will use default path.",
)
# Database connection pool configuration (Postgres only)
db_pool_size: int = Field(
default=20,
description="Number of connections to keep in the pool (Postgres only)",
gt=0,
)
db_pool_overflow: int = Field(
default=40,
description="Max additional connections beyond pool_size under load (Postgres only)",
gt=0,
)
db_pool_recycle: int = Field(
default=180,
description="Recycle connections after N seconds to prevent stale connections. Default 180s works well with Neon's ~5 minute scale-to-zero (Postgres only)",
gt=0,
)
# Watch service configuration
sync_delay: int = Field(
default=1000, description="Milliseconds to wait after changes before syncing", gt=0
)
watch_project_reload_interval: int = Field(
default=30, description="Seconds between reloading project list in watch service", gt=0
default=300,
description="Seconds between reloading project list in watch service. Higher values reduce CPU usage by minimizing watcher restarts. Default 300s (5 min) balances efficiency with responsiveness to new projects.",
gt=0,
)
# update permalinks on move
@@ -197,6 +215,36 @@ class BasicMemoryConfig(BaseSettings):
# Fall back to config file value
return self.cloud_mode
@classmethod
def for_cloud_tenant(
cls,
database_url: str,
projects: Optional[Dict[str, str]] = None,
) -> "BasicMemoryConfig":
"""Create config for cloud tenant - no config.json, database is source of truth.
This factory method creates a BasicMemoryConfig suitable for cloud deployments
where:
- Database is Postgres (Neon), not SQLite
- Projects are discovered from the database, not config file
- Path validation is skipped (no local filesystem in cloud)
- Initialization sync is skipped (stateless deployment)
Args:
database_url: Postgres connection URL for tenant database
projects: Optional project mapping (usually empty, discovered from DB)
Returns:
BasicMemoryConfig configured for cloud mode
"""
return cls(
database_backend=DatabaseBackend.POSTGRES,
database_url=database_url,
projects=projects or {},
cloud_mode=True,
skip_initialization_sync=True,
)
model_config = SettingsConfigDict(
env_prefix="BASIC_MEMORY_",
extra="ignore",
@@ -213,6 +261,10 @@ class BasicMemoryConfig(BaseSettings):
def model_post_init(self, __context: Any) -> None:
"""Ensure configuration is valid after initialization."""
# Skip project initialization in cloud mode - projects are discovered from DB
if self.database_backend == DatabaseBackend.POSTGRES:
return
# Ensure at least one project exists; if none exist then create main
if not self.projects: # pragma: no cover
self.projects["main"] = str(
@@ -255,19 +307,26 @@ class BasicMemoryConfig(BaseSettings):
"""Get all configured projects as ProjectConfig objects."""
return [ProjectConfig(name=name, home=Path(path)) for name, path in self.projects.items()]
@field_validator("projects")
@classmethod
def ensure_project_paths_exists(cls, v: Dict[str, str]) -> Dict[str, str]: # pragma: no cover
"""Ensure project path exists."""
for name, path_value in v.items():
@model_validator(mode="after")
def ensure_project_paths_exists(self) -> "BasicMemoryConfig": # pragma: no cover
"""Ensure project paths exist.
Skips path creation when using Postgres backend (cloud mode) since
cloud tenants don't use local filesystem paths.
"""
# Skip path creation for cloud mode - no local filesystem
if self.database_backend == DatabaseBackend.POSTGRES:
return self
for name, path_value in self.projects.items():
path = Path(path_value)
if not Path(path).exists():
if not path.exists():
try:
path.mkdir(parents=True)
except Exception as e:
logger.error(f"Failed to create project path: {e}")
raise e
return v
return self
@property
def data_dir_path(self):
@@ -470,69 +529,38 @@ def save_basic_memory_config(file_path: Path, config: BasicMemoryConfig) -> None
logger.error(f"Failed to save config: {e}")
# setup logging to a single log file in user home directory
user_home = Path.home()
log_dir = user_home / DATA_DIR_NAME
log_dir.mkdir(parents=True, exist_ok=True)
# Logging initialization functions for different entry points
# Process info for logging
def get_process_name(): # pragma: no cover
def init_cli_logging() -> None: # pragma: no cover
"""Initialize logging for CLI commands - file only.
CLI commands should not log to stdout to avoid interfering with
command output and shell integration.
"""
get the type of process for logging
"""
import sys
log_level = os.getenv("BASIC_MEMORY_LOG_LEVEL", "INFO")
setup_logging(log_level=log_level, log_to_file=True)
if "sync" in sys.argv:
return "sync"
elif "mcp" in sys.argv:
return "mcp"
elif "cli" in sys.argv:
return "cli"
def init_mcp_logging() -> None: # pragma: no cover
"""Initialize logging for MCP server - file only.
MCP server must not log to stdout as it would corrupt the
JSON-RPC protocol communication.
"""
log_level = os.getenv("BASIC_MEMORY_LOG_LEVEL", "INFO")
setup_logging(log_level=log_level, log_to_file=True)
def init_api_logging() -> None: # pragma: no cover
"""Initialize logging for API server.
Cloud mode (BASIC_MEMORY_CLOUD_MODE=1): stdout with structured context
Local mode: file only
"""
log_level = os.getenv("BASIC_MEMORY_LOG_LEVEL", "INFO")
cloud_mode = os.getenv("BASIC_MEMORY_CLOUD_MODE", "").lower() in ("1", "true")
if cloud_mode:
setup_logging(log_level=log_level, log_to_stdout=True, structured_context=True)
else:
return "api"
process_name = get_process_name()
# Global flag to track if logging has been set up
_LOGGING_SETUP = False
# Logging
def setup_basic_memory_logging(): # pragma: no cover
"""Set up logging for basic-memory, ensuring it only happens once."""
global _LOGGING_SETUP
if _LOGGING_SETUP:
# We can't log before logging is set up
# print("Skipping duplicate logging setup")
return
# Check for console logging environment variable - accept more truthy values
console_logging_env = os.getenv("BASIC_MEMORY_CONSOLE_LOGGING", "false").lower()
console_logging = console_logging_env in ("true", "1", "yes", "on")
# Check for log level environment variable first, fall back to config
log_level = os.getenv("BASIC_MEMORY_LOG_LEVEL")
if not log_level:
config_manager = ConfigManager()
log_level = config_manager.config.log_level
config_manager = ConfigManager()
config = get_project_config()
setup_logging(
env=config_manager.config.env,
home_dir=user_home, # Use user home for logs
log_level=log_level,
log_file=f"{DATA_DIR_NAME}/basic-memory-{process_name}.log",
console=console_logging,
)
logger.info(f"Basic Memory {basic_memory.__version__} (Project: {config.project})")
_LOGGING_SETUP = True
# Set up logging
setup_basic_memory_logging()
setup_logging(log_level=log_level, log_to_file=True)
+40 -19
View File
@@ -33,6 +33,7 @@ class DatabaseType(Enum):
MEMORY = auto()
FILESYSTEM = auto()
POSTGRES = auto()
@classmethod
def get_db_url(
@@ -42,7 +43,7 @@ class DatabaseType(Enum):
Args:
db_path: Path to SQLite database file (ignored for Postgres)
db_type: Type of database (MEMORY or FILESYSTEM)
db_type: Type of database (MEMORY, FILESYSTEM, or POSTGRES)
config: Optional config to check for database backend and URL
Returns:
@@ -52,16 +53,21 @@ class DatabaseType(Enum):
if config is None:
config = ConfigManager().config
# Check if Postgres backend is configured
# Handle explicit Postgres type
if db_type == cls.POSTGRES:
if not config.database_url:
raise ValueError("DATABASE_URL must be set when using Postgres backend")
logger.info(f"Using Postgres database: {config.database_url}")
return config.database_url
# Check if Postgres backend is configured (for backward compatibility)
if config.database_backend == DatabaseBackend.POSTGRES:
if not config.database_url:
raise ValueError("DATABASE_URL must be set when using Postgres backend")
logger.info(
f"Using Postgres database: {config.database_url.split('@')[1] if '@' in config.database_url else config.database_url}"
)
logger.info(f"Using Postgres database: {config.database_url}")
return config.database_url
# Default to SQLite
# SQLite databases
if db_type == cls.MEMORY:
logger.info("Using in-memory SQLite database")
return "sqlite+aiosqlite://"
@@ -184,21 +190,37 @@ def _create_sqlite_engine(db_url: str, db_type: DatabaseType) -> AsyncEngine:
return engine
def _create_postgres_engine(db_url: str) -> AsyncEngine:
def _create_postgres_engine(db_url: str, config: BasicMemoryConfig) -> AsyncEngine:
"""Create Postgres async engine with appropriate configuration.
Args:
db_url: Postgres connection URL (postgresql+asyncpg://...)
config: BasicMemoryConfig with pool settings
Returns:
Configured async engine for Postgres
"""
# Postgres with asyncpg - use standard async connection
# Use NullPool connection issues.
# Assume connection pooler like PgBouncer handles connection pooling.
engine = create_async_engine(
db_url,
echo=False,
pool_pre_ping=True, # Verify connections before using them
poolclass=NullPool, # No pooling - fresh connection per request
connect_args={
# Disable statement cache to avoid issues with prepared statements on reconnect
"statement_cache_size": 0,
# Allow 30s for commands (Neon cold start can take 2-5s, sometimes longer)
"command_timeout": 30,
# Allow 30s for initial connection (Neon wake-up time)
"timeout": 30,
"server_settings": {
"application_name": "basic-memory",
# Statement timeout for queries (30s to allow for cold start)
"statement_timeout": "30s",
},
},
)
logger.debug("Created Postgres engine with NullPool (no connection pooling)")
return engine
@@ -210,7 +232,7 @@ def _create_engine_and_session(
Args:
db_path: Path to database file (used for SQLite, ignored for Postgres)
db_type: Type of database (MEMORY or FILESYSTEM)
db_type: Type of database (MEMORY, FILESYSTEM, or POSTGRES)
Returns:
Tuple of (engine, session_maker)
@@ -220,8 +242,9 @@ def _create_engine_and_session(
logger.debug(f"Creating engine for db_url: {db_url}")
# Delegate to backend-specific engine creation
if config.database_backend == DatabaseBackend.POSTGRES:
engine = _create_postgres_engine(db_url)
# Check explicit POSTGRES type first, then config setting
if db_type == DatabaseType.POSTGRES or config.database_backend == DatabaseBackend.POSTGRES:
engine = _create_postgres_engine(db_url, config)
else:
engine = _create_sqlite_engine(db_url, db_type)
@@ -326,13 +349,8 @@ async def run_migrations(
config.set_main_option("revision_environment", "false")
# Get the correct database URL based on backend configuration
# No URL conversion needed - env.py now handles both async and sync engines
db_url = DatabaseType.get_db_url(app_config.database_path, database_type, app_config)
# For Postgres, Alembic needs synchronous driver (psycopg2), not async (asyncpg)
if app_config.database_backend == DatabaseBackend.POSTGRES:
# Convert asyncpg URL to psycopg2 URL for Alembic
db_url = db_url.replace("postgresql+asyncpg://", "postgresql://")
config.set_main_option("sqlalchemy.url", db_url)
command.upgrade(config, "head")
@@ -348,7 +366,10 @@ async def run_migrations(
# For SQLite: Create FTS5 virtual table
# For Postgres: No-op (tsvector column added by migrations)
# The project_id is not used for init_search_index, so we pass a dummy value
if app_config.database_backend == DatabaseBackend.POSTGRES:
if (
database_type == DatabaseType.POSTGRES
or app_config.database_backend == DatabaseBackend.POSTGRES
):
await PostgresSearchRepository(session_maker, 1).init_search_index()
else:
await SQLiteSearchRepository(session_maker, 1).init_search_index()
+282 -4
View File
@@ -76,6 +76,34 @@ async def get_project_config(
ProjectConfigDep = Annotated[ProjectConfig, Depends(get_project_config)] # pragma: no cover
async def get_project_config_v2(
project_id: "ProjectIdPathDep", project_repository: "ProjectRepositoryDep"
) -> ProjectConfig: # pragma: no cover
"""Get the project config for v2 API (uses integer project_id from path).
Args:
project_id: The validated numeric project ID from the URL path
project_repository: Repository for project operations
Returns:
The resolved project config
Raises:
HTTPException: If project is not found
"""
project_obj = await project_repository.get_by_id(project_id)
if project_obj:
return ProjectConfig(name=project_obj.name, home=pathlib.Path(project_obj.path))
# Not found (this should not happen since ProjectIdPathDep already validates existence)
raise HTTPException( # pragma: no cover
status_code=status.HTTP_404_NOT_FOUND, detail=f"Project with ID {project_id} not found."
)
ProjectConfigV2Dep = Annotated[ProjectConfig, Depends(get_project_config_v2)] # pragma: no cover
## sqlalchemy
@@ -130,6 +158,38 @@ ProjectRepositoryDep = Annotated[ProjectRepository, Depends(get_project_reposito
ProjectPathDep = Annotated[str, Path()] # Use Path dependency to extract from URL
async def validate_project_id(
project_id: int,
project_repository: ProjectRepositoryDep,
) -> int:
"""Validate that a numeric project ID exists in the database.
This is used for v2 API endpoints that take project IDs as integers in the path.
The project_id parameter will be automatically extracted from the URL path by FastAPI.
Args:
project_id: The numeric project ID from the URL path
project_repository: Repository for project operations
Returns:
The validated project ID
Raises:
HTTPException: If project with that ID is not found
"""
project_obj = await project_repository.get_by_id(project_id)
if not project_obj:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail=f"Project with ID {project_id} not found.",
)
return project_id
# V2 API: Validated integer project ID from path
ProjectIdPathDep = Annotated[int, Depends(validate_project_id)]
async def get_project_id(
project_repository: ProjectRepositoryDep,
project: ProjectPathDep,
@@ -188,6 +248,17 @@ async def get_entity_repository(
EntityRepositoryDep = Annotated[EntityRepository, Depends(get_entity_repository)]
async def get_entity_repository_v2(
session_maker: SessionMakerDep,
project_id: ProjectIdPathDep,
) -> EntityRepository:
"""Create an EntityRepository instance for v2 API (uses integer project_id from path)."""
return EntityRepository(session_maker, project_id=project_id)
EntityRepositoryV2Dep = Annotated[EntityRepository, Depends(get_entity_repository_v2)]
async def get_observation_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
@@ -199,6 +270,19 @@ async def get_observation_repository(
ObservationRepositoryDep = Annotated[ObservationRepository, Depends(get_observation_repository)]
async def get_observation_repository_v2(
session_maker: SessionMakerDep,
project_id: ProjectIdPathDep,
) -> ObservationRepository:
"""Create an ObservationRepository instance for v2 API."""
return ObservationRepository(session_maker, project_id=project_id)
ObservationRepositoryV2Dep = Annotated[
ObservationRepository, Depends(get_observation_repository_v2)
]
async def get_relation_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
@@ -210,6 +294,17 @@ async def get_relation_repository(
RelationRepositoryDep = Annotated[RelationRepository, Depends(get_relation_repository)]
async def get_relation_repository_v2(
session_maker: SessionMakerDep,
project_id: ProjectIdPathDep,
) -> RelationRepository:
"""Create a RelationRepository instance for v2 API."""
return RelationRepository(session_maker, project_id=project_id)
RelationRepositoryV2Dep = Annotated[RelationRepository, Depends(get_relation_repository_v2)]
async def get_search_repository(
session_maker: SessionMakerDep,
project_id: ProjectIdDep,
@@ -225,6 +320,17 @@ async def get_search_repository(
SearchRepositoryDep = Annotated[SearchRepository, Depends(get_search_repository)]
async def get_search_repository_v2(
session_maker: SessionMakerDep,
project_id: ProjectIdPathDep,
) -> SearchRepository:
"""Create a SearchRepository instance for v2 API."""
return create_search_repository(session_maker, project_id=project_id)
SearchRepositoryV2Dep = Annotated[SearchRepository, Depends(get_search_repository_v2)]
# ProjectInfoRepository is deprecated and will be removed in a future version.
# Use ProjectRepository instead, which has the same functionality plus more project-specific operations.
@@ -238,6 +344,13 @@ async def get_entity_parser(project_config: ProjectConfigDep) -> EntityParser:
EntityParserDep = Annotated["EntityParser", Depends(get_entity_parser)]
async def get_entity_parser_v2(project_config: ProjectConfigV2Dep) -> EntityParser:
return EntityParser(project_config.home)
EntityParserV2Dep = Annotated["EntityParser", Depends(get_entity_parser_v2)]
async def get_markdown_processor(entity_parser: EntityParserDep) -> MarkdownProcessor:
return MarkdownProcessor(entity_parser)
@@ -245,20 +358,39 @@ async def get_markdown_processor(entity_parser: EntityParserDep) -> MarkdownProc
MarkdownProcessorDep = Annotated[MarkdownProcessor, Depends(get_markdown_processor)]
async def get_markdown_processor_v2(entity_parser: EntityParserV2Dep) -> MarkdownProcessor:
return MarkdownProcessor(entity_parser)
MarkdownProcessorV2Dep = Annotated[MarkdownProcessor, Depends(get_markdown_processor_v2)]
async def get_file_service(
project_config: ProjectConfigDep, markdown_processor: MarkdownProcessorDep
) -> FileService:
logger.debug(
f"Creating FileService for project: {project_config.name}, base_path: {project_config.home}"
)
file_service = FileService(project_config.home, markdown_processor)
logger.debug(f"Created FileService for project: {file_service} ")
logger.debug(
f"Created FileService for project: {project_config.name}, base_path: {project_config.home} "
)
return file_service
FileServiceDep = Annotated[FileService, Depends(get_file_service)]
async def get_file_service_v2(
project_config: ProjectConfigV2Dep, markdown_processor: MarkdownProcessorV2Dep
) -> FileService:
file_service = FileService(project_config.home, markdown_processor)
logger.debug(
f"Created FileService for project: {project_config.name}, base_path: {project_config.home}"
)
return file_service
FileServiceV2Dep = Annotated[FileService, Depends(get_file_service_v2)]
async def get_entity_service(
entity_repository: EntityRepositoryDep,
observation_repository: ObservationRepositoryDep,
@@ -283,6 +415,30 @@ async def get_entity_service(
EntityServiceDep = Annotated[EntityService, Depends(get_entity_service)]
async def get_entity_service_v2(
entity_repository: EntityRepositoryV2Dep,
observation_repository: ObservationRepositoryV2Dep,
relation_repository: RelationRepositoryV2Dep,
entity_parser: EntityParserV2Dep,
file_service: FileServiceV2Dep,
link_resolver: "LinkResolverV2Dep",
app_config: AppConfigDep,
) -> EntityService:
"""Create EntityService for v2 API."""
return EntityService(
entity_repository=entity_repository,
observation_repository=observation_repository,
relation_repository=relation_repository,
entity_parser=entity_parser,
file_service=file_service,
link_resolver=link_resolver,
app_config=app_config,
)
EntityServiceV2Dep = Annotated[EntityService, Depends(get_entity_service_v2)]
async def get_search_service(
search_repository: SearchRepositoryDep,
entity_repository: EntityRepositoryDep,
@@ -295,6 +451,18 @@ async def get_search_service(
SearchServiceDep = Annotated[SearchService, Depends(get_search_service)]
async def get_search_service_v2(
search_repository: SearchRepositoryV2Dep,
entity_repository: EntityRepositoryV2Dep,
file_service: FileServiceV2Dep,
) -> SearchService:
"""Create SearchService for v2 API."""
return SearchService(search_repository, entity_repository, file_service)
SearchServiceV2Dep = Annotated[SearchService, Depends(get_search_service_v2)]
async def get_link_resolver(
entity_repository: EntityRepositoryDep, search_service: SearchServiceDep
) -> LinkResolver:
@@ -304,6 +472,15 @@ async def get_link_resolver(
LinkResolverDep = Annotated[LinkResolver, Depends(get_link_resolver)]
async def get_link_resolver_v2(
entity_repository: EntityRepositoryV2Dep, search_service: SearchServiceV2Dep
) -> LinkResolver:
return LinkResolver(entity_repository=entity_repository, search_service=search_service)
LinkResolverV2Dep = Annotated[LinkResolver, Depends(get_link_resolver_v2)]
async def get_context_service(
search_repository: SearchRepositoryDep,
entity_repository: EntityRepositoryDep,
@@ -319,6 +496,22 @@ async def get_context_service(
ContextServiceDep = Annotated[ContextService, Depends(get_context_service)]
async def get_context_service_v2(
search_repository: SearchRepositoryV2Dep,
entity_repository: EntityRepositoryV2Dep,
observation_repository: ObservationRepositoryV2Dep,
) -> ContextService:
"""Create ContextService for v2 API."""
return ContextService(
search_repository=search_repository,
entity_repository=entity_repository,
observation_repository=observation_repository,
)
ContextServiceV2Dep = Annotated[ContextService, Depends(get_context_service_v2)]
async def get_sync_service(
app_config: AppConfigDep,
entity_service: EntityServiceDep,
@@ -348,6 +541,32 @@ async def get_sync_service(
SyncServiceDep = Annotated[SyncService, Depends(get_sync_service)]
async def get_sync_service_v2(
app_config: AppConfigDep,
entity_service: EntityServiceV2Dep,
entity_parser: EntityParserV2Dep,
entity_repository: EntityRepositoryV2Dep,
relation_repository: RelationRepositoryV2Dep,
project_repository: ProjectRepositoryDep,
search_service: SearchServiceV2Dep,
file_service: FileServiceV2Dep,
) -> SyncService: # pragma: no cover
"""Create SyncService for v2 API."""
return SyncService(
app_config=app_config,
entity_service=entity_service,
entity_parser=entity_parser,
entity_repository=entity_repository,
relation_repository=relation_repository,
project_repository=project_repository,
search_service=search_service,
file_service=file_service,
)
SyncServiceV2Dep = Annotated[SyncService, Depends(get_sync_service_v2)]
async def get_project_service(
project_repository: ProjectRepositoryDep,
) -> ProjectService:
@@ -370,6 +589,18 @@ async def get_directory_service(
DirectoryServiceDep = Annotated[DirectoryService, Depends(get_directory_service)]
async def get_directory_service_v2(
entity_repository: EntityRepositoryV2Dep,
) -> DirectoryService:
"""Create DirectoryService for v2 API (uses integer project_id from path)."""
return DirectoryService(
entity_repository=entity_repository,
)
DirectoryServiceV2Dep = Annotated[DirectoryService, Depends(get_directory_service_v2)]
# Import
@@ -413,3 +644,50 @@ async def get_memory_json_importer(
MemoryJsonImporterDep = Annotated[MemoryJsonImporter, Depends(get_memory_json_importer)]
# V2 Import dependencies
async def get_chatgpt_importer_v2(
project_config: ProjectConfigV2Dep, markdown_processor: MarkdownProcessorV2Dep
) -> ChatGPTImporter:
"""Create ChatGPTImporter with v2 dependencies."""
return ChatGPTImporter(project_config.home, markdown_processor)
ChatGPTImporterV2Dep = Annotated[ChatGPTImporter, Depends(get_chatgpt_importer_v2)]
async def get_claude_conversations_importer_v2(
project_config: ProjectConfigV2Dep, markdown_processor: MarkdownProcessorV2Dep
) -> ClaudeConversationsImporter:
"""Create ClaudeConversationsImporter with v2 dependencies."""
return ClaudeConversationsImporter(project_config.home, markdown_processor)
ClaudeConversationsImporterV2Dep = Annotated[
ClaudeConversationsImporter, Depends(get_claude_conversations_importer_v2)
]
async def get_claude_projects_importer_v2(
project_config: ProjectConfigV2Dep, markdown_processor: MarkdownProcessorV2Dep
) -> ClaudeProjectsImporter:
"""Create ClaudeProjectsImporter with v2 dependencies."""
return ClaudeProjectsImporter(project_config.home, markdown_processor)
ClaudeProjectsImporterV2Dep = Annotated[
ClaudeProjectsImporter, Depends(get_claude_projects_importer_v2)
]
async def get_memory_json_importer_v2(
project_config: ProjectConfigV2Dep, markdown_processor: MarkdownProcessorV2Dep
) -> MemoryJsonImporter:
"""Create MemoryJsonImporter with v2 dependencies."""
return MemoryJsonImporter(project_config.home, markdown_processor)
MemoryJsonImporterV2Dep = Annotated[MemoryJsonImporter, Depends(get_memory_json_importer_v2)]
+16
View File
@@ -1,6 +1,8 @@
"""Utilities for file operations."""
import hashlib
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
import re
from typing import Any, Dict, Union
@@ -13,6 +15,20 @@ from loguru import logger
from basic_memory.utils import FilePath
@dataclass
class FileMetadata:
"""File metadata for cloud-compatible file operations.
This dataclass provides a cloud-agnostic way to represent file metadata,
enabling S3FileService to return metadata from head_object responses
instead of mock stat_result with zeros.
"""
size: int
created_at: datetime
modified_at: datetime
class FileError(Exception):
"""Base exception for file operations."""
@@ -40,10 +40,13 @@ class ClaudeConversationsImporter(Importer[ChatImportResult]):
chats_imported = 0
for chat in conversations:
# Get name, providing default for unnamed conversations
chat_name = chat.get("name") or f"Conversation {chat.get('uuid', 'untitled')}"
# Convert to entity
entity = self._format_chat_content(
base_path=folder_path,
name=chat["name"],
name=chat_name,
messages=chat["chat_messages"],
created_at=chat["created_at"],
modified_at=chat["updated_at"],
+59 -26
View File
@@ -23,6 +23,7 @@ from basic_memory.markdown.schemas import (
)
from basic_memory.utils import parse_tags
md = MarkdownIt().use(observation_plugin).use(relation_plugin)
@@ -189,35 +190,63 @@ class EntityParser:
return self.base_path / path
async def parse_file_content(self, absolute_path, file_content):
# Parse frontmatter with proper error handling for malformed YAML (issue #185)
try:
post = frontmatter.loads(file_content)
except yaml.YAMLError as e:
# Log the YAML parsing error with file context
logger.warning(
f"Failed to parse YAML frontmatter in {absolute_path}: {e}. "
f"Treating file as plain markdown without frontmatter."
)
# Create a post with no frontmatter - treat entire content as markdown
post = frontmatter.Post(file_content, metadata={})
"""Parse markdown content from file stats.
# Extract file stat info
Delegates to parse_markdown_content() for actual parsing logic.
Exists for backwards compatibility with code that passes file paths.
"""
# Extract file stat info for timestamps
file_stats = absolute_path.stat()
# Normalize frontmatter values to prevent AttributeError on date objects (issue #236)
# PyYAML automatically converts date strings like "2025-10-24" to datetime.date objects
# This normalization converts them back to ISO format strings to ensure compatibility
# with code that expects string values
# Delegate to parse_markdown_content with timestamps from file stats
return await self.parse_markdown_content(
file_path=absolute_path,
content=file_content,
mtime=file_stats.st_mtime,
ctime=file_stats.st_ctime,
)
async def parse_markdown_content(
self,
file_path: Path,
content: str,
mtime: Optional[float] = None,
ctime: Optional[float] = None,
) -> EntityMarkdown:
"""Parse markdown content without requiring file to exist on disk.
Useful for parsing content from S3 or other remote sources where the file
is not available locally.
Args:
file_path: Path for metadata (doesn't need to exist on disk)
content: Markdown content as string
mtime: Optional modification time (Unix timestamp)
ctime: Optional creation time (Unix timestamp)
Returns:
EntityMarkdown with parsed content
"""
# Parse frontmatter with proper error handling for malformed YAML
try:
post = frontmatter.loads(content)
except yaml.YAMLError as e:
logger.warning(
f"Failed to parse YAML frontmatter in {file_path}: {e}. "
f"Treating file as plain markdown without frontmatter."
)
post = frontmatter.Post(content, metadata={})
# Normalize frontmatter values
metadata = normalize_frontmatter_metadata(post.metadata)
# Ensure required fields have defaults (issue #184, #387)
# Handle title - use default if missing, None/null, empty, or string "None"
# Ensure required fields have defaults
title = metadata.get("title")
if not title or title == "None":
metadata["title"] = absolute_path.stem
metadata["title"] = file_path.stem
else:
metadata["title"] = title
# Handle type - use default if missing OR explicitly set to None/null
entity_type = metadata.get("type")
metadata["type"] = entity_type if entity_type is not None else "note"
@@ -225,16 +254,20 @@ class EntityParser:
if tags:
metadata["tags"] = tags
# frontmatter - use metadata with defaults applied
entity_frontmatter = EntityFrontmatter(
metadata=metadata,
)
# Parse content for observations and relations
entity_frontmatter = EntityFrontmatter(metadata=metadata)
entity_content = parse(post.content)
# Use provided timestamps or current time as fallback
now = datetime.now().astimezone()
created = datetime.fromtimestamp(ctime).astimezone() if ctime else now
modified = datetime.fromtimestamp(mtime).astimezone() if mtime else now
return EntityMarkdown(
frontmatter=entity_frontmatter,
content=post.content,
observations=entity_content.observations,
relations=entity_content.relations,
created=datetime.fromtimestamp(file_stats.st_ctime).astimezone(),
modified=datetime.fromtimestamp(file_stats.st_mtime).astimezone(),
created=created,
modified=modified,
)
@@ -5,6 +5,7 @@ from collections import OrderedDict
from frontmatter import Post
from loguru import logger
from basic_memory import file_utils
from basic_memory.file_utils import dump_frontmatter
from basic_memory.markdown.entity_parser import EntityParser
+4 -2
View File
@@ -30,7 +30,9 @@ def is_observation(token: Token) -> bool:
# Check for proper observation format: [category] content
match = re.match(r"^\[([^\[\]()]+)\]\s+(.+)", content)
has_tags = "#" in content
# Check for standalone hashtags (words starting with #)
# This excludes # in HTML attributes like color="#4285F4"
has_tags = any(part.startswith("#") for part in content.split())
return bool(match) or has_tags
@@ -160,7 +162,7 @@ def parse_inline_relations(content: str) -> List[Dict[str, Any]]:
target = content[start + 2 : end].strip()
if target:
relations.append({"type": "links to", "target": target, "context": None})
relations.append({"type": "links_to", "target": target, "context": None})
start = end + 2
+10 -1
View File
@@ -3,6 +3,7 @@
from pathlib import Path
from typing import Any, Optional
from frontmatter import Post
from basic_memory.file_utils import has_frontmatter, remove_frontmatter, parse_frontmatter
@@ -12,7 +13,10 @@ from basic_memory.models import Observation as ObservationModel
def entity_model_from_markdown(
file_path: Path, markdown: EntityMarkdown, entity: Optional[Entity] = None
file_path: Path,
markdown: EntityMarkdown,
entity: Optional[Entity] = None,
project_id: Optional[int] = None,
) -> Entity:
"""
Convert markdown entity to model. Does not include relations.
@@ -21,6 +25,7 @@ def entity_model_from_markdown(
file_path: Path to the markdown file
markdown: Parsed markdown entity
entity: Optional existing entity to update
project_id: Project ID for new observations (uses entity.project_id if not provided)
Returns:
Entity model populated from markdown
@@ -50,9 +55,13 @@ def entity_model_from_markdown(
metadata = markdown.frontmatter.metadata or {}
model.entity_metadata = {k: str(v) for k, v in metadata.items() if v is not None}
# Get project_id from entity if not provided
obs_project_id = project_id or (model.project_id if hasattr(model, "project_id") else None)
# Convert observations
model.observations = [
ObservationModel(
project_id=obs_project_id,
content=obs.content,
category=obs.category,
context=obs.context,
-2
View File
@@ -4,7 +4,6 @@ import basic_memory
from basic_memory.models.base import Base
from basic_memory.models.knowledge import Entity, Observation, Relation
from basic_memory.models.project import Project
from basic_memory.models.search import SearchIndex
__all__ = [
"Base",
@@ -12,6 +11,5 @@ __all__ = [
"Observation",
"Relation",
"Project",
"SearchIndex",
"basic_memory",
]
+9 -2
View File
@@ -129,7 +129,7 @@ class Entity(Base):
return value
def __repr__(self) -> str:
return f"Entity(id={self.id}, name='{self.title}', type='{self.entity_type}'"
return f"Entity(id={self.id}, name='{self.title}', type='{self.entity_type}', checksum='{self.checksum}')"
class Observation(Base):
@@ -145,6 +145,7 @@ class Observation(Base):
)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
project_id: Mapped[int] = mapped_column(Integer, ForeignKey("project.id"), index=True)
entity_id: Mapped[int] = mapped_column(Integer, ForeignKey("entity.id", ondelete="CASCADE"))
content: Mapped[str] = mapped_column(Text)
category: Mapped[str] = mapped_column(String, nullable=False, default="note")
@@ -162,9 +163,14 @@ class Observation(Base):
We can construct these because observations are always defined in
and owned by a single entity.
Content is truncated to 200 chars to stay under PostgreSQL's
btree index limit of 2704 bytes.
"""
# Truncate content to avoid exceeding PostgreSQL's btree index limit
content_for_permalink = self.content[:200] if len(self.content) > 200 else self.content
return generate_permalink(
f"{self.entity.permalink}/observations/{self.category}/{self.content}"
f"{self.entity.permalink}/observations/{self.category}/{content_for_permalink}"
)
def __repr__(self) -> str: # pragma: no cover
@@ -186,6 +192,7 @@ class Relation(Base):
)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
project_id: Mapped[int] = mapped_column(Integer, ForeignKey("project.id"), index=True)
from_id: Mapped[int] = mapped_column(Integer, ForeignKey("entity.id", ondelete="CASCADE"))
to_id: Mapped[Optional[int]] = mapped_column(
Integer, ForeignKey("entity.id", ondelete="CASCADE"), nullable=True
+43 -44
View File
@@ -1,53 +1,52 @@
"""Search models and tables."""
"""Search DDL statements for SQLite and Postgres.
from sqlalchemy import DDL, Column, Integer, String, DateTime, Text
from sqlalchemy.dialects.postgresql import JSONB
from sqlalchemy.types import JSON
The search_index table is created via raw DDL, not ORM models, because:
- SQLite uses FTS5 virtual tables (cannot be represented as ORM)
- Postgres uses composite primary keys and generated tsvector columns
- Both backends use raw SQL for all search operations via SearchIndexRow dataclass
"""
from basic_memory.models.base import Base
from sqlalchemy import DDL
class SearchIndex(Base):
"""Search index table for Postgres only.
# Define Postgres search_index table with composite primary key and tsvector
# This DDL matches the Alembic migration schema (314f1ea54dc4)
# Used by tests to create the table without running full migrations
# NOTE: Split into separate DDL statements because asyncpg doesn't support
# multiple statements in a single execute call.
CREATE_POSTGRES_SEARCH_INDEX_TABLE = DDL("""
CREATE TABLE IF NOT EXISTS search_index (
id INTEGER NOT NULL,
project_id INTEGER NOT NULL,
title TEXT,
content_stems TEXT,
content_snippet TEXT,
permalink VARCHAR,
file_path VARCHAR,
type VARCHAR,
from_id INTEGER,
to_id INTEGER,
relation_type VARCHAR,
entity_id INTEGER,
category VARCHAR,
metadata JSONB,
created_at TIMESTAMP WITH TIME ZONE,
updated_at TIMESTAMP WITH TIME ZONE,
textsearchable_index_col tsvector GENERATED ALWAYS AS (
to_tsvector('english', coalesce(title, '') || ' ' || coalesce(content_stems, ''))
) STORED,
PRIMARY KEY (id, type, project_id),
FOREIGN KEY (project_id) REFERENCES project(id) ON DELETE CASCADE
)
""")
For SQLite: This model is skipped; FTS5 virtual table is created via DDL instead.
For Postgres: This is the actual table structure with tsvector support.
"""
__tablename__ = "search_index"
# Primary key (rowid in SQLite FTS5, explicit id in Postgres)
id = Column(Integer, primary_key=True, autoincrement=True)
# Core searchable fields
title = Column(Text, nullable=True)
content_stems = Column(Text, nullable=True)
content_snippet = Column(Text, nullable=True)
permalink = Column(String(255), nullable=True, index=True)
file_path = Column(Text, nullable=True)
type = Column(String(50), nullable=True)
# Project context
project_id = Column(Integer, nullable=True, index=True)
# Relation fields
from_id = Column(Integer, nullable=True)
to_id = Column(Integer, nullable=True)
relation_type = Column(String(100), nullable=True)
# Observation fields
entity_id = Column(Integer, nullable=True)
category = Column(String(100), nullable=True)
# Common fields
# Use JSONB for Postgres, JSON for SQLite
# Note: 'metadata' is a reserved name in SQLAlchemy, so we use 'metadata_' and map to 'metadata'
metadata_ = Column("metadata", JSON().with_variant(JSONB(), "postgresql"), nullable=True)
created_at = Column(DateTime(timezone=True), nullable=True)
updated_at = Column(DateTime(timezone=True), nullable=True)
# Note: textsearchable_index_col (tsvector) will be added by migration for Postgres only
CREATE_POSTGRES_SEARCH_INDEX_FTS = DDL("""
CREATE INDEX IF NOT EXISTS idx_search_index_fts ON search_index USING gin(textsearchable_index_col)
""")
CREATE_POSTGRES_SEARCH_INDEX_METADATA = DDL("""
CREATE INDEX IF NOT EXISTS idx_search_index_metadata_gin ON search_index USING gin(metadata jsonb_path_ops)
""")
# Define FTS5 virtual table creation for SQLite only
# This DDL is executed separately for SQLite databases
@@ -1,7 +1,8 @@
"""Repository for managing entities in the knowledge graph."""
from pathlib import Path
from typing import List, Optional, Sequence, Union
from typing import List, Optional, Sequence, Union, Any
from loguru import logger
from sqlalchemy import select
@@ -9,6 +10,7 @@ from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from sqlalchemy.orm import selectinload
from sqlalchemy.orm.interfaces import LoaderOption
from sqlalchemy.engine import Row
from basic_memory import db
from basic_memory.models.knowledge import Entity, Observation, Relation
@@ -31,6 +33,18 @@ class EntityRepository(Repository[Entity]):
"""
super().__init__(session_maker, Entity, project_id=project_id)
async def get_by_id(self, entity_id: int) -> Optional[Entity]:
"""Get entity by numeric ID.
Args:
entity_id: Numeric entity ID
Returns:
Entity if found, None otherwise
"""
async with db.scoped_session(self.session_maker) as session:
return await self.select_by_id(session, entity_id)
async def get_by_permalink(self, permalink: str) -> Optional[Entity]:
"""Get entity by permalink.
@@ -63,6 +77,127 @@ class EntityRepository(Repository[Entity]):
)
return await self.find_one(query)
# -------------------------------------------------------------------------
# Lightweight methods for permalink resolution (no eager loading)
# -------------------------------------------------------------------------
async def permalink_exists(self, permalink: str) -> bool:
"""Check if a permalink exists without loading the full entity.
This is much faster than get_by_permalink() as it skips eager loading
of observations and relations. Use for existence checks in bulk operations.
Args:
permalink: Permalink to check
Returns:
True if permalink exists, False otherwise
"""
query = select(Entity.id).where(Entity.permalink == permalink).limit(1)
query = self._add_project_filter(query)
result = await self.execute_query(query, use_query_options=False)
return result.scalar_one_or_none() is not None
async def get_file_path_for_permalink(self, permalink: str) -> Optional[str]:
"""Get the file_path for a permalink without loading the full entity.
Use when you only need the file_path, not the full entity with relations.
Args:
permalink: Permalink to look up
Returns:
file_path string if found, None otherwise
"""
query = select(Entity.file_path).where(Entity.permalink == permalink)
query = self._add_project_filter(query)
result = await self.execute_query(query, use_query_options=False)
return result.scalar_one_or_none()
async def get_permalink_for_file_path(self, file_path: Union[Path, str]) -> Optional[str]:
"""Get the permalink for a file_path without loading the full entity.
Use when you only need the permalink, not the full entity with relations.
Args:
file_path: File path to look up
Returns:
permalink string if found, None otherwise
"""
query = select(Entity.permalink).where(Entity.file_path == Path(file_path).as_posix())
query = self._add_project_filter(query)
result = await self.execute_query(query, use_query_options=False)
return result.scalar_one_or_none()
async def get_all_permalinks(self) -> List[str]:
"""Get all permalinks for this project.
Optimized for bulk operations - returns only permalink strings
without loading entities or relationships.
Returns:
List of all permalinks in the project
"""
query = select(Entity.permalink)
query = self._add_project_filter(query)
result = await self.execute_query(query, use_query_options=False)
return list(result.scalars().all())
async def get_permalink_to_file_path_map(self) -> dict[str, str]:
"""Get a mapping of permalink -> file_path for all entities.
Optimized for bulk permalink resolution - loads minimal data in one query.
Returns:
Dict mapping permalink to file_path
"""
query = select(Entity.permalink, Entity.file_path)
query = self._add_project_filter(query)
result = await self.execute_query(query, use_query_options=False)
return {row.permalink: row.file_path for row in result.all()}
async def get_file_path_to_permalink_map(self) -> dict[str, str]:
"""Get a mapping of file_path -> permalink for all entities.
Optimized for bulk permalink resolution - loads minimal data in one query.
Returns:
Dict mapping file_path to permalink
"""
query = select(Entity.file_path, Entity.permalink)
query = self._add_project_filter(query)
result = await self.execute_query(query, use_query_options=False)
return {row.file_path: row.permalink for row in result.all()}
async def get_by_file_paths(
self, session: AsyncSession, file_paths: Sequence[Union[Path, str]]
) -> List[Row[Any]]:
"""Get file paths and checksums for multiple entities (optimized for change detection).
Only queries file_path and checksum columns, skips loading full entities and relationships.
This is much faster than loading complete Entity objects when you only need checksums.
Args:
session: Database session to use for the query
file_paths: List of file paths to query
Returns:
List of (file_path, checksum) tuples for matching entities
"""
if not file_paths:
return []
# Convert all paths to POSIX strings for consistent comparison
posix_paths = [Path(fp).as_posix() for fp in file_paths]
# Query ONLY file_path and checksum columns (not full Entity objects)
query = select(Entity.file_path, Entity.checksum).where(Entity.file_path.in_(posix_paths))
query = self._add_project_filter(query)
result = await session.execute(query)
return list(result.all())
async def find_by_checksum(self, checksum: str) -> Sequence[Entity]:
"""Find entities with the given checksum.
@@ -80,6 +215,34 @@ class EntityRepository(Repository[Entity]):
result = await self.execute_query(query, use_query_options=False)
return list(result.scalars().all())
async def find_by_checksums(self, checksums: Sequence[str]) -> Sequence[Entity]:
"""Find entities with any of the given checksums (batch query for move detection).
This is a batch-optimized version of find_by_checksum() that queries multiple checksums
in a single database query. Used for efficient move detection in cloud indexing.
Performance: For 1000 new files, this makes 1 query vs 1000 individual queries (~100x faster).
Example:
When processing new files, we check if any are actually moved files by finding
entities with matching checksums at different paths.
Args:
checksums: List of file content checksums to search for
Returns:
Sequence of entities with matching checksums (may be empty).
Multiple entities may have the same checksum if files were copied.
"""
if not checksums:
return []
# Query: SELECT * FROM entities WHERE checksum IN (checksum1, checksum2, ...)
query = self.select().where(Entity.checksum.in_(checksums))
# Don't load relationships for move detection - we only need file_path and checksum
result = await self.execute_query(query, use_query_options=False)
return list(result.scalars().all())
async def delete_by_file_path(self, file_path: Union[Path, str]) -> bool:
"""Delete entity with the provided file_path.
@@ -2,6 +2,7 @@
from typing import Dict, List, Sequence
from sqlalchemy import select
from sqlalchemy.ext.asyncio import async_sessionmaker
@@ -5,6 +5,7 @@ import re
from datetime import datetime
from typing import List, Optional
from loguru import logger
from sqlalchemy import text
@@ -257,7 +258,7 @@ class PostgresSearchRepository(SearchRepositoryBase):
{score_expr} as score
FROM search_index
WHERE {where_clause}
ORDER BY score DESC {order_by_clause}
ORDER BY score DESC, id ASC {order_by_clause}
LIMIT :limit
OFFSET :offset
"""
@@ -311,3 +312,68 @@ class PostgresSearchRepository(SearchRepositoryBase):
)
return results
async def bulk_index_items(self, search_index_rows: List[SearchIndexRow]) -> None:
"""Index multiple items in a single batch operation using UPSERT.
Uses INSERT ... ON CONFLICT DO UPDATE to handle re-indexing of existing
entities (e.g., during forward reference resolution) without requiring
a separate delete operation. This eliminates race conditions between
delete and insert operations in separate transactions.
Args:
search_index_rows: List of SearchIndexRow objects to index
"""
if not search_index_rows:
return
async with db.scoped_session(self.session_maker) as session:
# When using text() raw SQL, always serialize JSON to string
# Both SQLite (TEXT) and Postgres (JSONB) accept JSON strings in raw SQL
# The database driver/column type will handle conversion
insert_data_list = []
for row in search_index_rows:
insert_data = row.to_insert(serialize_json=True)
insert_data["project_id"] = self.project_id
insert_data_list.append(insert_data)
# Use UPSERT (INSERT ... ON CONFLICT) to handle re-indexing
# Primary key is (id, type, project_id)
# This handles race conditions during forward reference resolution
# where an entity might be re-indexed before the delete commits
# Syntax works for both SQLite 3.24+ and PostgreSQL
await session.execute(
text("""
INSERT INTO search_index (
id, title, content_stems, content_snippet, permalink, file_path, type, metadata,
from_id, to_id, relation_type,
entity_id, category,
created_at, updated_at,
project_id
) VALUES (
:id, :title, :content_stems, :content_snippet, :permalink, :file_path, :type, :metadata,
:from_id, :to_id, :relation_type,
:entity_id, :category,
:created_at, :updated_at,
:project_id
)
ON CONFLICT (id, type, project_id) DO UPDATE SET
title = EXCLUDED.title,
content_stems = EXCLUDED.content_stems,
content_snippet = EXCLUDED.content_snippet,
permalink = EXCLUDED.permalink,
file_path = EXCLUDED.file_path,
metadata = EXCLUDED.metadata,
from_id = EXCLUDED.from_id,
to_id = EXCLUDED.to_id,
relation_type = EXCLUDED.relation_type,
entity_id = EXCLUDED.entity_id,
category = EXCLUDED.category,
created_at = EXCLUDED.created_at,
updated_at = EXCLUDED.updated_at
"""),
insert_data_list,
)
logger.debug(f"Bulk indexed {len(search_index_rows)} rows")
await session.commit()
@@ -3,6 +3,7 @@
from pathlib import Path
from typing import Optional, Sequence, Union
from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
@@ -49,6 +50,18 @@ class ProjectRepository(Repository[Project]):
query = self.select().where(Project.path == Path(path).as_posix())
return await self.find_one(query)
async def get_by_id(self, project_id: int) -> Optional[Project]:
"""Get project by numeric ID.
Args:
project_id: Numeric project ID
Returns:
Project if found, None otherwise
"""
async with db.scoped_session(self.session_maker) as session:
return await self.select_by_id(session, project_id)
async def get_default_project(self) -> Optional[Project]:
"""Get the default project (the one marked as is_default=True)."""
query = self.select().where(Project.is_default.is_not(None))
@@ -1,9 +1,11 @@
"""Repository for managing Relation objects."""
from sqlalchemy import and_, delete
from typing import Sequence, List, Optional
from sqlalchemy import select
from sqlalchemy import and_, delete, select
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
from sqlalchemy.ext.asyncio import async_sessionmaker
from sqlalchemy.orm import selectinload, aliased
from sqlalchemy.orm.interfaces import LoaderOption
@@ -86,5 +88,59 @@ class RelationRepository(Repository[Relation]):
result = await self.execute_query(query)
return result.scalars().all()
async def add_all_ignore_duplicates(self, relations: List[Relation]) -> int:
"""Bulk insert relations, ignoring duplicates.
Uses ON CONFLICT DO NOTHING to skip relations that would violate the
unique constraint on (from_id, to_name, relation_type). This is useful
for bulk operations where the same link may appear multiple times in
a document.
Works with both SQLite and PostgreSQL dialects.
Args:
relations: List of Relation objects to insert
Returns:
Number of relations actually inserted (excludes duplicates)
"""
if not relations:
return 0
# Convert Relation objects to dicts for insert
values = [
{
"project_id": r.project_id if r.project_id else self.project_id,
"from_id": r.from_id,
"to_id": r.to_id,
"to_name": r.to_name,
"relation_type": r.relation_type,
"context": r.context,
}
for r in relations
]
async with db.scoped_session(self.session_maker) as session:
# Check dialect to use appropriate insert
dialect_name = session.bind.dialect.name if session.bind else "sqlite"
if dialect_name == "postgresql":
# PostgreSQL: use RETURNING to count inserted rows
# (rowcount is 0 for ON CONFLICT DO NOTHING)
stmt = (
pg_insert(Relation)
.values(values)
.on_conflict_do_nothing()
.returning(Relation.id)
)
result = await session.execute(stmt)
return len(result.fetchall())
else:
# SQLite: rowcount works correctly
stmt = sqlite_insert(Relation).values(values)
stmt = stmt.on_conflict_do_nothing()
result = await session.execute(stmt)
return result.rowcount if result.rowcount > 0 else 0
def get_load_options(self) -> List[LoaderOption]:
return [selectinload(Relation.from_entity), selectinload(Relation.to_entity)]
@@ -2,6 +2,7 @@
from typing import Type, Optional, Any, Sequence, TypeVar, List, Dict
from loguru import logger
from sqlalchemy import (
select,
@@ -4,6 +4,7 @@ from abc import ABC, abstractmethod
from datetime import datetime
from typing import Any, Dict, List, Optional
from loguru import logger
from sqlalchemy import Executable, Result, text
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
@@ -5,6 +5,7 @@ import re
from datetime import datetime
from typing import List, Optional
from loguru import logger
from sqlalchemy import text
+7
View File
@@ -124,6 +124,7 @@ class EntitySummary(BaseModel):
"""Simplified entity representation."""
type: Literal["entity"] = "entity"
entity_id: int # Database ID for v2 API consistency
permalink: Optional[str]
title: str
content: Optional[str] = None
@@ -141,12 +142,16 @@ class RelationSummary(BaseModel):
"""Simplified relation representation."""
type: Literal["relation"] = "relation"
relation_id: int # Database ID for v2 API consistency
entity_id: Optional[int] = None # ID of the entity this relation belongs to
title: str
file_path: str
permalink: str
relation_type: str
from_entity: Optional[str] = None
from_entity_id: Optional[int] = None # ID of source entity
to_entity: Optional[str] = None
to_entity_id: Optional[int] = None # ID of target entity
created_at: Annotated[
datetime, Field(json_schema_extra={"type": "string", "format": "date-time"})
]
@@ -160,6 +165,8 @@ class ObservationSummary(BaseModel):
"""Simplified observation representation."""
type: Literal["observation"] = "observation"
observation_id: int # Database ID for v2 API consistency
entity_id: Optional[int] = None # ID of the entity this observation belongs to
title: str
file_path: str
permalink: str
+1
View File
@@ -173,6 +173,7 @@ class ProjectWatchStatus(BaseModel):
class ProjectItem(BaseModel):
"""Simple representation of a project."""
id: int
name: str
path: str
is_default: bool = False
+5
View File
@@ -97,6 +97,11 @@ class SearchResult(BaseModel):
metadata: Optional[dict] = None
# IDs for v2 API consistency
entity_id: Optional[int] = None # Entity ID (always present for entities)
observation_id: Optional[int] = None # Observation ID (for observation results)
relation_id: Optional[int] = None # Relation ID (for relation results)
# Type-specific fields
category: Optional[str] = None # For observations
from_entity: Optional[Permalink] = None # For relations
+23
View File
@@ -0,0 +1,23 @@
"""V2 API schemas - ID-based entity references."""
from basic_memory.schemas.v2.entity import (
EntityResolveRequest,
EntityResolveResponse,
EntityResponseV2,
MoveEntityRequestV2,
)
from basic_memory.schemas.v2.resource import (
CreateResourceRequest,
UpdateResourceRequest,
ResourceResponse,
)
__all__ = [
"EntityResolveRequest",
"EntityResolveResponse",
"EntityResponseV2",
"MoveEntityRequestV2",
"CreateResourceRequest",
"UpdateResourceRequest",
"ResourceResponse",
]
+96
View File
@@ -0,0 +1,96 @@
"""V2 entity schemas with ID-first design."""
from datetime import datetime
from typing import Dict, List, Literal, Optional
from pydantic import BaseModel, Field, ConfigDict
from basic_memory.schemas.response import ObservationResponse, RelationResponse
class EntityResolveRequest(BaseModel):
"""Request to resolve a string identifier to an entity ID.
Supports resolution of:
- Permalinks (e.g., "specs/search")
- Titles (e.g., "Search Specification")
- File paths (e.g., "specs/search.md")
"""
identifier: str = Field(
...,
description="Entity identifier to resolve (permalink, title, or file path)",
min_length=1,
max_length=500,
)
class EntityResolveResponse(BaseModel):
"""Response from identifier resolution.
Returns the entity ID and associated metadata for the resolved entity.
"""
entity_id: int = Field(..., description="Numeric entity ID (primary identifier)")
permalink: Optional[str] = Field(None, description="Entity permalink")
file_path: str = Field(..., description="Relative file path")
title: str = Field(..., description="Entity title")
resolution_method: Literal["id", "permalink", "title", "path", "search"] = Field(
..., description="How the identifier was resolved"
)
class MoveEntityRequestV2(BaseModel):
"""V2 request schema for moving an entity to a new file location.
In V2 API, the entity ID is provided in the URL path, so this request
only needs the destination path.
"""
destination_path: str = Field(
...,
description="New file path for the entity (relative to project root)",
min_length=1,
max_length=500,
)
class EntityResponseV2(BaseModel):
"""V2 entity response with ID as the primary field.
This response format emphasizes the entity ID as the primary identifier,
with all other fields (permalink, file_path) as secondary metadata.
"""
# ID first - this is the primary identifier in v2
id: int = Field(..., description="Numeric entity ID (primary identifier)")
# Core entity fields
title: str = Field(..., description="Entity title")
entity_type: str = Field(..., description="Entity type")
content_type: str = Field(default="text/markdown", description="Content MIME type")
# Secondary identifiers (for compatibility and convenience)
permalink: Optional[str] = Field(None, description="Entity permalink (may change)")
file_path: str = Field(..., description="Relative file path (may change)")
# Content and metadata
content: Optional[str] = Field(None, description="Entity content")
entity_metadata: Optional[Dict] = Field(None, description="Entity metadata")
# Relationships
observations: List[ObservationResponse] = Field(
default_factory=list, description="Entity observations"
)
relations: List[RelationResponse] = Field(default_factory=list, description="Entity relations")
# Timestamps
created_at: datetime = Field(..., description="Creation timestamp")
updated_at: datetime = Field(..., description="Last update timestamp")
# V2-specific metadata
api_version: Literal["v2"] = Field(
default="v2", description="API version (always 'v2' for this response)"
)
model_config = ConfigDict(from_attributes=True)
+46
View File
@@ -0,0 +1,46 @@
"""V2 resource schemas for file content operations."""
from pydantic import BaseModel, Field
class CreateResourceRequest(BaseModel):
"""Request to create a new resource file.
File path is required for new resources since we need to know where
to create the file.
"""
file_path: str = Field(
...,
description="Path to create the file, relative to project root",
min_length=1,
max_length=500,
)
content: str = Field(..., description="File content to write")
class UpdateResourceRequest(BaseModel):
"""Request to update an existing resource by entity ID.
Only content is required - the file path is already known from the entity.
Optionally can update the file_path to move the file.
"""
content: str = Field(..., description="File content to write")
file_path: str | None = Field(
None,
description="Optional new file path to move the resource",
min_length=1,
max_length=500,
)
class ResourceResponse(BaseModel):
"""Response from resource operations."""
entity_id: int = Field(..., description="Entity ID of the resource")
file_path: str = Field(..., description="File path of the resource")
checksum: str = Field(..., description="File content checksum")
size: int = Field(..., description="File size in bytes")
created_at: float = Field(..., description="Creation timestamp")
modified_at: float = Field(..., description="Modification timestamp")
@@ -4,6 +4,7 @@ from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import List, Optional, Tuple
from loguru import logger
from sqlalchemy import text
+15 -2
View File
@@ -3,8 +3,10 @@
import fnmatch
import logging
import os
from datetime import datetime
from typing import Dict, List, Optional, Sequence
from basic_memory.models import Entity
from basic_memory.repository import EntityRepository
from basic_memory.schemas.directory import DirectoryNode
@@ -12,6 +14,17 @@ from basic_memory.schemas.directory import DirectoryNode
logger = logging.getLogger(__name__)
def _mtime_to_datetime(entity: Entity) -> datetime:
"""Convert entity mtime (file modification time) to datetime.
Returns the file's actual modification time, falling back to updated_at
if mtime is not available.
"""
if entity.mtime:
return datetime.fromtimestamp(entity.mtime).astimezone()
return entity.updated_at
class DirectoryService:
"""Service for working with directory trees."""
@@ -77,7 +90,7 @@ class DirectoryService:
entity_id=file.id,
entity_type=file.entity_type,
content_type=file.content_type,
updated_at=file.updated_at,
updated_at=_mtime_to_datetime(file),
)
# Add to parent directory's children
@@ -241,7 +254,7 @@ class DirectoryService:
entity_id=file.id,
entity_type=file.entity_type,
content_type=file.content_type,
updated_at=file.updated_at,
updated_at=_mtime_to_datetime(file),
)
# Add to parent directory's children
+60 -32
View File
@@ -8,6 +8,7 @@ import yaml
from loguru import logger
from sqlalchemy.exc import IntegrityError
from basic_memory.config import ProjectConfig, BasicMemoryConfig
from basic_memory.file_utils import (
has_frontmatter,
@@ -106,6 +107,9 @@ class EntityService(BaseService[EntityModel]):
4. Generate new unique permalink from file path
Enhanced to detect and handle character-related conflicts.
Note: Uses lightweight repository methods that skip eager loading of
observations and relations for better performance during bulk operations.
"""
file_path_str = Path(file_path).as_posix()
@@ -122,16 +126,20 @@ class EntityService(BaseService[EntityModel]):
# If markdown has explicit permalink, try to validate it
if markdown and markdown.frontmatter.permalink:
desired_permalink = markdown.frontmatter.permalink
existing = await self.repository.get_by_permalink(desired_permalink)
# Use lightweight method - we only need to check file_path
existing_file_path = await self.repository.get_file_path_for_permalink(
desired_permalink
)
# If no conflict or it's our own file, use as is
if not existing or existing.file_path == file_path_str:
if not existing_file_path or existing_file_path == file_path_str:
return desired_permalink
# For existing files, try to find current permalink
existing = await self.repository.get_by_file_path(file_path_str)
if existing:
return existing.permalink
# Use lightweight method - we only need the permalink
existing_permalink = await self.repository.get_permalink_for_file_path(file_path_str)
if existing_permalink:
return existing_permalink
# New file - generate permalink
if markdown and markdown.frontmatter.permalink:
@@ -140,9 +148,10 @@ class EntityService(BaseService[EntityModel]):
desired_permalink = generate_permalink(file_path_str)
# Make unique if needed - enhanced to handle character conflicts
# Use lightweight existence check instead of loading full entity
permalink = desired_permalink
suffix = 1
while await self.repository.get_by_permalink(permalink):
while await self.repository.permalink_exists(permalink):
permalink = f"{desired_permalink}-{suffix}"
suffix += 1
logger.debug(f"creating unique permalink: {permalink}")
@@ -224,8 +233,11 @@ class EntityService(BaseService[EntityModel]):
final_content = dump_frontmatter(post)
checksum = await self.file_service.write_file(file_path, final_content)
# parse entity from file
entity_markdown = await self.entity_parser.parse_file(file_path)
# parse entity from content we just wrote (avoids re-reading file for cloud compatibility)
entity_markdown = await self.entity_parser.parse_markdown_content(
file_path=file_path,
content=final_content,
)
# create entity
created = await self.create_entity_from_markdown(file_path, entity_markdown)
@@ -245,8 +257,12 @@ class EntityService(BaseService[EntityModel]):
# Convert file path string to Path
file_path = Path(entity.file_path)
# Read existing frontmatter from the file if it exists
existing_markdown = await self.entity_parser.parse_file(file_path)
# Read existing content via file_service (for cloud compatibility)
existing_content = await self.file_service.read_file_content(file_path)
existing_markdown = await self.entity_parser.parse_markdown_content(
file_path=file_path,
content=existing_content,
)
# Parse content frontmatter to check for user-specified permalink and entity_type
content_markdown = None
@@ -302,8 +318,11 @@ class EntityService(BaseService[EntityModel]):
final_content = dump_frontmatter(merged_post)
checksum = await self.file_service.write_file(file_path, final_content)
# parse entity from file
entity_markdown = await self.entity_parser.parse_file(file_path)
# parse entity from content we just wrote (avoids re-reading file for cloud compatibility)
entity_markdown = await self.entity_parser.parse_markdown_content(
file_path=file_path,
content=final_content,
)
# update entity in db
entity = await self.update_entity_and_observations(file_path, entity_markdown)
@@ -378,7 +397,9 @@ class EntityService(BaseService[EntityModel]):
Uses UPSERT approach to handle permalink/file_path conflicts cleanly.
"""
logger.debug(f"Creating entity: {markdown.frontmatter.title} file_path: {file_path}")
model = entity_model_from_markdown(file_path, markdown)
model = entity_model_from_markdown(
file_path, markdown, project_id=self.repository.project_id
)
# Mark as incomplete because we still need to add relations
model.checksum = None
@@ -408,6 +429,7 @@ class EntityService(BaseService[EntityModel]):
# add new observations
observations = [
Observation(
project_id=self.observation_repository.project_id,
entity_id=db_entity.id,
content=obs.content,
category=obs.category,
@@ -448,8 +470,11 @@ class EntityService(BaseService[EntityModel]):
import asyncio
# Create tasks for all relation lookups
# Use strict=True to disable fuzzy search - only exact matches should create resolved relations
# This ensures forward references (links to non-existent entities) remain unresolved (to_id=NULL)
lookup_tasks = [
self.link_resolver.resolve_link(rel.target) for rel in markdown.relations
self.link_resolver.resolve_link(rel.target, strict=True)
for rel in markdown.relations
]
# Execute all lookups in parallel
@@ -471,6 +496,7 @@ class EntityService(BaseService[EntityModel]):
# Create the relation
relation = Relation(
project_id=self.relation_repository.project_id,
from_id=db_entity.id,
to_id=target_id,
to_name=target_name,
@@ -543,8 +569,11 @@ class EntityService(BaseService[EntityModel]):
# Write the updated content back to the file
checksum = await self.file_service.write_file(file_path, new_content)
# Parse the updated file to get new observations/relations
entity_markdown = await self.entity_parser.parse_file(file_path)
# Parse the content we just wrote (avoids re-reading file for cloud compatibility)
entity_markdown = await self.entity_parser.parse_markdown_content(
file_path=file_path,
content=new_content,
)
# Update entity and its relationships
entity = await self.update_entity_and_observations(file_path, entity_markdown)
@@ -763,23 +792,20 @@ class EntityService(BaseService[EntityModel]):
raise ValueError(f"Invalid destination path: {destination_path}")
# 3. Validate paths
source_file = project_config.home / current_path
destination_file = project_config.home / destination_path
# Validate source exists
if not source_file.exists():
# NOTE: In tenantless/cloud mode, we cannot rely on local filesystem paths.
# Use FileService for existence checks and moving.
if not await self.file_service.exists(current_path):
raise ValueError(f"Source file not found: {current_path}")
# Check if destination already exists
if destination_file.exists():
if await self.file_service.exists(destination_path):
raise ValueError(f"Destination already exists: {destination_path}")
try:
# 4. Create destination directory if needed
destination_file.parent.mkdir(parents=True, exist_ok=True)
# 4. Ensure destination directory if needed (no-op for S3)
await self.file_service.ensure_directory(Path(destination_path).parent)
# 5. Move physical file
source_file.rename(destination_file)
# 5. Move physical file via FileService (filesystem rename or cloud move)
await self.file_service.move_file(current_path, destination_path)
logger.info(f"Moved file: {current_path} -> {destination_path}")
# 6. Prepare database updates
@@ -818,12 +844,14 @@ class EntityService(BaseService[EntityModel]):
except Exception as e:
# Rollback: try to restore original file location if move succeeded
if destination_file.exists() and not source_file.exists():
try:
destination_file.rename(source_file)
try:
if await self.file_service.exists(
destination_path
) and not await self.file_service.exists(current_path):
await self.file_service.move_file(destination_path, current_path)
logger.info(f"Rolled back file move: {destination_path} -> {current_path}")
except Exception as rollback_error: # pragma: no cover
logger.error(f"Failed to rollback file move: {rollback_error}")
except Exception as rollback_error: # pragma: no cover
logger.error(f"Failed to rollback file move: {rollback_error}")
# Re-raise the original error with context
raise ValueError(f"Move failed: {str(e)}") from e
+91 -7
View File
@@ -3,15 +3,16 @@
import asyncio
import hashlib
import mimetypes
from os import stat_result
from datetime import datetime
from pathlib import Path
from typing import Any, Dict, Tuple, Union
import aiofiles
import yaml
from basic_memory import file_utils
from basic_memory.file_utils import FileError, ParseError
from basic_memory.file_utils import FileError, FileMetadata, ParseError
from basic_memory.markdown.markdown_processor import MarkdownProcessor
from basic_memory.models import Entity as EntityModel
from basic_memory.schemas import Entity as EntitySchema
@@ -220,6 +221,41 @@ class FileService:
logger.exception("File read error", path=str(full_path), error=str(e))
raise FileOperationError(f"Failed to read file: {e}")
async def read_file_bytes(self, path: FilePath) -> bytes:
"""Read file content as bytes using true async I/O with aiofiles.
This method reads files in binary mode, suitable for non-text files
like images, PDFs, etc. For cloud compatibility with S3FileService.
Args:
path: Path to read (Path or string)
Returns:
File content as bytes
Raises:
FileOperationError: If read fails
"""
# Convert string to Path if needed
path_obj = self.base_path / path if isinstance(path, str) else path
full_path = path_obj if path_obj.is_absolute() else self.base_path / path_obj
try:
logger.debug("Reading file bytes", operation="read_file_bytes", path=str(full_path))
async with aiofiles.open(full_path, mode="rb") as f:
content = await f.read()
logger.debug(
"File read completed",
path=str(full_path),
content_length=len(content),
)
return content
except Exception as e:
logger.exception("File read error", path=str(full_path), error=str(e))
raise FileOperationError(f"Failed to read file: {e}")
async def read_file(self, path: FilePath) -> Tuple[str, str]:
"""Read file and compute checksum using true async I/O.
@@ -276,6 +312,43 @@ class FileService:
full_path = path_obj if path_obj.is_absolute() else self.base_path / path_obj
full_path.unlink(missing_ok=True)
async def move_file(self, source: FilePath, destination: FilePath) -> None:
"""Move/rename a file from source to destination.
This method abstracts the underlying storage (filesystem vs cloud).
Default implementation uses atomic filesystem rename, but cloud-backed
implementations (e.g., S3) can override to copy+delete.
Args:
source: Source path (relative to base_path or absolute)
destination: Destination path (relative to base_path or absolute)
Raises:
FileOperationError: If the move fails
"""
# Convert strings to Paths and resolve relative paths against base_path
src_obj = self.base_path / source if isinstance(source, str) else source
dst_obj = self.base_path / destination if isinstance(destination, str) else destination
src_full = src_obj if src_obj.is_absolute() else self.base_path / src_obj
dst_full = dst_obj if dst_obj.is_absolute() else self.base_path / dst_obj
try:
# Ensure destination directory exists
await self.ensure_directory(dst_full.parent)
# Use semaphore for concurrency control and run blocking rename in executor
async with self._file_semaphore:
loop = asyncio.get_event_loop()
await loop.run_in_executor(None, lambda: src_full.rename(dst_full))
except Exception as e:
logger.exception(
"File move error",
source=str(src_full),
destination=str(dst_full),
error=str(e),
)
raise FileOperationError(f"Failed to move file {source} -> {destination}: {e}")
async def update_frontmatter(self, path: FilePath, updates: Dict[str, Any]) -> str:
"""Update frontmatter fields in a file while preserving all content.
@@ -381,20 +454,31 @@ class FileService:
logger.error("Failed to compute checksum", path=str(full_path), error=str(e))
raise FileError(f"Failed to compute checksum for {path}: {e}")
def file_stats(self, path: FilePath) -> stat_result:
"""Return file stats for a given path.
async def get_file_metadata(self, path: FilePath) -> FileMetadata:
"""Return file metadata for a given path.
This method is async to support cloud implementations (S3FileService)
where file metadata requires async operations (head_object).
Args:
path: Path to the file (Path or string)
Returns:
File statistics
FileMetadata with size, created_at, and modified_at
"""
# Convert string to Path if needed
path_obj = self.base_path / path if isinstance(path, str) else path
full_path = path_obj if path_obj.is_absolute() else self.base_path / path_obj
# get file timestamps
return full_path.stat()
# Run blocking stat() in thread pool to maintain async compatibility
loop = asyncio.get_event_loop()
stat_result = await loop.run_in_executor(None, full_path.stat)
return FileMetadata(
size=stat_result.st_size,
created_at=datetime.fromtimestamp(stat_result.st_ctime).astimezone(),
modified_at=datetime.fromtimestamp(stat_result.st_mtime).astimezone(),
)
def content_type(self, path: FilePath) -> str:
"""Return content_type for a given path.
@@ -5,8 +5,10 @@ to ensure consistent application startup across all entry points.
"""
import asyncio
import os
from pathlib import Path
from loguru import logger
from basic_memory import db
@@ -104,6 +106,12 @@ async def initialize_file_sync(
# Get active projects
active_projects = await project_repository.get_active_projects()
# Filter to constrained project if MCP server was started with --project
constrained_project = os.environ.get("BASIC_MEMORY_MCP_PROJECT")
if constrained_project:
active_projects = [p for p in active_projects if p.name == constrained_project]
logger.info(f"Background sync constrained to project: {constrained_project}")
# Start sync for all projects as background tasks (non-blocking)
async def sync_project_background(project: Project):
"""Sync a single project in the background."""
@@ -2,6 +2,7 @@
from typing import Optional, Tuple
from loguru import logger
from basic_memory.models import Entity
+6 -6
View File
@@ -8,6 +8,7 @@ from datetime import datetime
from pathlib import Path
from typing import Dict, Optional, Sequence
from loguru import logger
from sqlalchemy import text
@@ -23,9 +24,6 @@ from basic_memory.config import WATCH_STATUS_JSON, ConfigManager, get_project_co
from basic_memory.utils import generate_permalink
config = ConfigManager().config
class ProjectService:
"""Service for managing Basic Memory projects."""
@@ -143,6 +141,7 @@ class ProjectService:
"""
# If project_root is set, constrain all projects to that directory
project_root = self.config_manager.config.project_root
sanitized_name = None
if project_root:
base_path = Path(project_root)
@@ -199,14 +198,15 @@ class ProjectService:
f"Projects cannot share directory trees."
)
# First add to config file (this will validate the project doesn't exist)
project_config = self.config_manager.add_project(name, resolved_path)
if not self.config_manager.config.cloud_mode:
# First add to config file (this will validate the project doesn't exist)
self.config_manager.add_project(name, resolved_path)
# Then add to database
project_data = {
"name": name,
"path": resolved_path,
"permalink": generate_permalink(project_config.name),
"permalink": sanitized_name,
"is_active": True,
# Don't set is_default=False to avoid UNIQUE constraint issues
# Let it default to NULL, only set to True when explicitly making default
+50 -10
View File
@@ -4,6 +4,7 @@ import ast
from datetime import datetime
from typing import List, Optional, Set
from dateparser import parse
from fastapi import BackgroundTasks
from loguru import logger
@@ -15,6 +16,21 @@ from basic_memory.repository.search_repository import SearchRepository, SearchIn
from basic_memory.schemas.search import SearchQuery, SearchItemType
from basic_memory.services import FileService
# Maximum size for content_stems field to stay under Postgres's 8KB index row limit.
# We use 6000 characters to leave headroom for other indexed columns and overhead.
MAX_CONTENT_STEMS_SIZE = 6000
def _mtime_to_datetime(entity: Entity) -> datetime:
"""Convert entity mtime (file modification time) to datetime.
Returns the file's actual modification time, falling back to updated_at
if mtime is not available.
"""
if entity.mtime:
return datetime.fromtimestamp(entity.mtime).astimezone()
return entity.updated_at
class SearchService:
"""Service for search operations.
@@ -156,22 +172,24 @@ class SearchService:
self,
entity: Entity,
background_tasks: Optional[BackgroundTasks] = None,
content: str | None = None,
) -> None:
if background_tasks:
background_tasks.add_task(self.index_entity_data, entity)
background_tasks.add_task(self.index_entity_data, entity, content)
else:
await self.index_entity_data(entity)
await self.index_entity_data(entity, content)
async def index_entity_data(
self,
entity: Entity,
content: str | None = None,
) -> None:
# delete all search index data associated with entity
await self.repository.delete_by_entity_id(entity_id=entity.id)
# reindex
await self.index_entity_markdown(
entity
entity, content
) if entity.is_markdown else await self.index_entity_file(entity)
async def index_entity_file(
@@ -191,7 +209,7 @@ class SearchService:
"entity_type": entity.entity_type,
},
created_at=entity.created_at,
updated_at=entity.updated_at,
updated_at=_mtime_to_datetime(entity),
project_id=entity.project_id,
)
)
@@ -199,9 +217,14 @@ class SearchService:
async def index_entity_markdown(
self,
entity: Entity,
content: str | None = None,
) -> None:
"""Index an entity and all its observations and relations.
Args:
entity: The entity to index
content: Optional pre-loaded content (avoids file read). If None, will read from file.
Indexing structure:
1. Entities
- permalink: direct from entity (e.g., "specs/search")
@@ -230,7 +253,9 @@ class SearchService:
title_variants = self._generate_variants(entity.title)
content_stems.extend(title_variants)
content = await self.file_service.read_entity_content(entity)
# Use provided content or read from file
if content is None:
content = await self.file_service.read_entity_content(entity)
if content:
content_stems.append(content)
content_snippet = f"{content[:250]}"
@@ -247,6 +272,10 @@ class SearchService:
entity_content_stems = "\n".join(p for p in content_stems if p and p.strip())
# Truncate to stay under Postgres's 8KB index row limit
if len(entity_content_stems) > MAX_CONTENT_STEMS_SIZE:
entity_content_stems = entity_content_stems[:MAX_CONTENT_STEMS_SIZE]
# Add entity row
rows_to_index.append(
SearchIndexRow(
@@ -262,17 +291,28 @@ class SearchService:
"entity_type": entity.entity_type,
},
created_at=entity.created_at,
updated_at=entity.updated_at,
updated_at=_mtime_to_datetime(entity),
project_id=entity.project_id,
)
)
# Add observation rows
# Add observation rows - dedupe by permalink to avoid unique constraint violations
# Two observations with same entity/category/content generate identical permalinks
seen_permalinks: set[str] = {entity.permalink} if entity.permalink else set()
for obs in entity.observations:
obs_permalink = obs.permalink
if obs_permalink in seen_permalinks:
logger.debug(f"Skipping duplicate observation permalink: {obs_permalink}")
continue
seen_permalinks.add(obs_permalink)
# Index with parent entity's file path since that's where it's defined
obs_content_stems = "\n".join(
p for p in self._generate_variants(obs.content) if p and p.strip()
)
# Truncate to stay under Postgres's 8KB index row limit
if len(obs_content_stems) > MAX_CONTENT_STEMS_SIZE:
obs_content_stems = obs_content_stems[:MAX_CONTENT_STEMS_SIZE]
rows_to_index.append(
SearchIndexRow(
id=obs.id,
@@ -280,7 +320,7 @@ class SearchService:
title=f"{obs.category}: {obs.content[:100]}...",
content_stems=obs_content_stems,
content_snippet=obs.content,
permalink=obs.permalink,
permalink=obs_permalink,
file_path=entity.file_path,
category=obs.category,
entity_id=entity.id,
@@ -288,7 +328,7 @@ class SearchService:
"tags": obs.tags,
},
created_at=entity.created_at,
updated_at=entity.updated_at,
updated_at=_mtime_to_datetime(entity),
project_id=entity.project_id,
)
)
@@ -318,7 +358,7 @@ class SearchService:
to_id=rel.to_id,
relation_type=rel.relation_type,
created_at=entity.created_at,
updated_at=entity.updated_at,
updated_at=_mtime_to_datetime(entity),
project_id=entity.project_id,
)
)
+85 -105
View File
@@ -10,7 +10,7 @@ from pathlib import Path
from typing import AsyncIterator, Dict, List, Optional, Set, Tuple
import aiofiles.os
import logfire
from loguru import logger
from sqlalchemy.exc import IntegrityError
@@ -215,17 +215,12 @@ class SyncService:
f"path={path}, error={error}"
)
# Record metric for file failure
logfire.metric_counter("sync.circuit_breaker.failures").add(1)
# Log when threshold is reached
if failure_info.count >= MAX_CONSECUTIVE_FAILURES:
logger.error(
f"File {path} has failed {MAX_CONSECUTIVE_FAILURES} times and will be skipped. "
f"First failure: {failure_info.first_failure}, Last error: {error}"
)
# Record metric for file being blocked by circuit breaker
logfire.metric_counter("sync.circuit_breaker.blocked_files").add(1)
else:
# Create new failure record
self._file_failures[path] = FileFailureInfo(
@@ -255,7 +250,6 @@ class SyncService:
logger.info(f"Clearing failure history for {path} after successful sync")
del self._file_failures[path]
@logfire.instrument()
async def sync(
self, directory: Path, project_name: Optional[str] = None, force_full: bool = False
) -> SyncReport:
@@ -282,63 +276,58 @@ class SyncService:
)
# sync moves first
with logfire.span("process_moves", move_count=len(report.moves)):
for old_path, new_path in report.moves.items():
# in the case where a file has been deleted and replaced by another file
# it will show up in the move and modified lists, so handle it in modified
if new_path in report.modified:
report.modified.remove(new_path)
logger.debug(
f"File marked as moved and modified: old_path={old_path}, new_path={new_path}"
)
else:
await self.handle_move(old_path, new_path)
for old_path, new_path in report.moves.items():
# in the case where a file has been deleted and replaced by another file
# it will show up in the move and modified lists, so handle it in modified
if new_path in report.modified:
report.modified.remove(new_path)
logger.debug(
f"File marked as moved and modified: old_path={old_path}, new_path={new_path}"
)
else:
await self.handle_move(old_path, new_path)
# deleted next
with logfire.span("process_deletes", delete_count=len(report.deleted)):
for path in report.deleted:
await self.handle_delete(path)
for path in report.deleted:
await self.handle_delete(path)
# then new and modified
with logfire.span("process_new_files", new_count=len(report.new)):
for path in report.new:
entity, _ = await self.sync_file(path, new=True)
for path in report.new:
entity, _ = await self.sync_file(path, new=True)
# Track if file was skipped
if entity is None and await self._should_skip_file(path):
failure_info = self._file_failures[path]
report.skipped_files.append(
SkippedFile(
path=path,
reason=failure_info.last_error,
failure_count=failure_info.count,
first_failed=failure_info.first_failure,
)
# Track if file was skipped
if entity is None and await self._should_skip_file(path):
failure_info = self._file_failures[path]
report.skipped_files.append(
SkippedFile(
path=path,
reason=failure_info.last_error,
failure_count=failure_info.count,
first_failed=failure_info.first_failure,
)
)
with logfire.span("process_modified_files", modified_count=len(report.modified)):
for path in report.modified:
entity, _ = await self.sync_file(path, new=False)
for path in report.modified:
entity, _ = await self.sync_file(path, new=False)
# Track if file was skipped
if entity is None and await self._should_skip_file(path):
failure_info = self._file_failures[path]
report.skipped_files.append(
SkippedFile(
path=path,
reason=failure_info.last_error,
failure_count=failure_info.count,
first_failed=failure_info.first_failure,
)
# Track if file was skipped
if entity is None and await self._should_skip_file(path):
failure_info = self._file_failures[path]
report.skipped_files.append(
SkippedFile(
path=path,
reason=failure_info.last_error,
failure_count=failure_info.count,
first_failed=failure_info.first_failure,
)
)
# Only resolve relations if there were actual changes
# If no files changed, no new unresolved relations could have been created
with logfire.span("resolve_relations"):
if report.total > 0:
await self.resolve_relations()
else:
logger.info("Skipping relation resolution - no file changes detected")
if report.total > 0:
await self.resolve_relations()
else:
logger.info("Skipping relation resolution - no file changes detected")
# Update scan watermark after successful sync
# Use the timestamp from sync start (not end) to ensure we catch files
@@ -361,15 +350,6 @@ class SyncService:
duration_ms = int((time.time() - start_time) * 1000)
# Record metrics for sync operation
logfire.metric_histogram("sync.duration", unit="ms").record(duration_ms)
logfire.metric_counter("sync.files.new").add(len(report.new))
logfire.metric_counter("sync.files.modified").add(len(report.modified))
logfire.metric_counter("sync.files.deleted").add(len(report.deleted))
logfire.metric_counter("sync.files.moved").add(len(report.moves))
if report.skipped_files:
logfire.metric_counter("sync.files.skipped").add(len(report.skipped_files))
# Log summary with skipped files if any
if report.skipped_files:
logger.warning(
@@ -390,7 +370,6 @@ class SyncService:
return report
@logfire.instrument()
async def scan(self, directory, force_full: bool = False):
"""Smart scan using watermark and file count for large project optimization.
@@ -472,12 +451,6 @@ class SyncService:
logger.warning("No scan watermark available, falling back to full scan")
file_paths_to_scan = await self._scan_directory_full(directory)
# Record scan type metric
logfire.metric_counter(f"sync.scan.{scan_type}").add(1)
logfire.metric_histogram("sync.scan.files_scanned", unit="files").record(
len(file_paths_to_scan)
)
# Step 3: Process each file with mtime-based comparison
scanned_paths: Set[str] = set()
changed_checksums: Dict[str, str] = {}
@@ -589,7 +562,6 @@ class SyncService:
report.checksums = changed_checksums
scan_duration_ms = int((time.time() - scan_start_time) * 1000)
logfire.metric_histogram("sync.scan.duration", unit="ms").record(scan_duration_ms)
logger.info(
f"Completed {scan_type} scan for directory {directory} in {scan_duration_ms}ms, "
@@ -599,7 +571,6 @@ class SyncService:
)
return report
@logfire.instrument()
async def sync_file(
self, path: str, new: bool = True
) -> Tuple[Optional[Entity], Optional[str]]:
@@ -654,7 +625,6 @@ class SyncService:
return None, None
@logfire.instrument()
async def sync_markdown_file(self, path: str, new: bool = True) -> Tuple[Optional[Entity], str]:
"""Sync a markdown file with full processing.
@@ -672,12 +642,19 @@ class SyncService:
file_contains_frontmatter = has_frontmatter(file_content)
# Get file timestamps for tracking modification times
file_stats = self.file_service.file_stats(path)
created = datetime.fromtimestamp(file_stats.st_ctime).astimezone()
modified = datetime.fromtimestamp(file_stats.st_mtime).astimezone()
file_metadata = await self.file_service.get_file_metadata(path)
created = file_metadata.created_at
modified = file_metadata.modified_at
# entity markdown will always contain front matter, so it can be used up create/update the entity
entity_markdown = await self.entity_parser.parse_file(path)
# Parse markdown content with file metadata (avoids redundant file read/stat)
# This enables cloud implementations (S3FileService) to provide metadata from head_object
abs_path = self.file_service.base_path / path
entity_markdown = await self.entity_parser.parse_markdown_content(
file_path=abs_path,
content=file_content,
mtime=file_metadata.modified_at.timestamp(),
ctime=file_metadata.created_at.timestamp(),
)
# if the file contains frontmatter, resolve a permalink (unless disabled)
if file_contains_frontmatter and not self.app_config.disable_permalinks:
@@ -723,8 +700,8 @@ class SyncService:
"checksum": final_checksum,
"created_at": created,
"updated_at": modified,
"mtime": file_stats.st_mtime,
"size": file_stats.st_size,
"mtime": file_metadata.modified_at.timestamp(),
"size": file_metadata.size,
},
)
@@ -737,7 +714,6 @@ class SyncService:
# Return the final checksum to ensure everything is consistent
return entity, final_checksum
@logfire.instrument()
async def sync_regular_file(self, path: str, new: bool = True) -> Tuple[Optional[Entity], str]:
"""Sync a non-markdown file with basic tracking.
@@ -754,9 +730,9 @@ class SyncService:
await self.entity_service.resolve_permalink(path, skip_conflict_check=True)
# get file timestamps
file_stats = self.file_service.file_stats(path)
created = datetime.fromtimestamp(file_stats.st_ctime).astimezone()
modified = datetime.fromtimestamp(file_stats.st_mtime).astimezone()
file_metadata = await self.file_service.get_file_metadata(path)
created = file_metadata.created_at
modified = file_metadata.modified_at
# get mime type
content_type = self.file_service.content_type(path)
@@ -772,8 +748,8 @@ class SyncService:
created_at=created,
updated_at=modified,
content_type=content_type,
mtime=file_stats.st_mtime,
size=file_stats.st_size,
mtime=file_metadata.modified_at.timestamp(),
size=file_metadata.size,
)
)
return entity, checksum
@@ -789,15 +765,15 @@ class SyncService:
logger.error(f"Entity not found after constraint violation, path={path}")
raise ValueError(f"Entity not found after constraint violation: {path}")
# Re-get file stats since we're in update path
file_stats_for_update = self.file_service.file_stats(path)
# Re-get file metadata since we're in update path
file_metadata_for_update = await self.file_service.get_file_metadata(path)
updated = await self.entity_repository.update(
entity.id,
{
"file_path": path,
"checksum": checksum,
"mtime": file_stats_for_update.st_mtime,
"size": file_stats_for_update.st_size,
"mtime": file_metadata_for_update.modified_at.timestamp(),
"size": file_metadata_for_update.size,
},
)
@@ -811,8 +787,8 @@ class SyncService:
raise
else:
# Get file timestamps for updating modification time
file_stats = self.file_service.file_stats(path)
modified = datetime.fromtimestamp(file_stats.st_mtime).astimezone()
file_metadata = await self.file_service.get_file_metadata(path)
modified = file_metadata.modified_at
entity = await self.entity_repository.get_by_file_path(path)
if entity is None: # pragma: no cover
@@ -827,8 +803,8 @@ class SyncService:
"file_path": path,
"checksum": checksum,
"updated_at": modified,
"mtime": file_stats.st_mtime,
"size": file_stats.st_size,
"mtime": file_metadata.modified_at.timestamp(),
"size": file_metadata.size,
},
)
@@ -838,7 +814,6 @@ class SyncService:
return updated, checksum
@logfire.instrument()
async def handle_delete(self, file_path: str):
"""Handle complete entity deletion including search index cleanup."""
@@ -870,7 +845,6 @@ class SyncService:
else:
await self.search_service.delete_by_entity_id(entity.id)
@logfire.instrument()
async def handle_move(self, old_path, new_path):
logger.debug("Moving entity", old_path=old_path, new_path=new_path)
@@ -975,7 +949,6 @@ class SyncService:
# update search index
await self.search_service.index_entity(updated)
@logfire.instrument()
async def resolve_relations(self, entity_id: int | None = None):
"""Try to resolve unresolved relations.
@@ -1026,16 +999,27 @@ class SyncService:
"to_name": resolved_entity.title,
},
)
except IntegrityError: # pragma: no cover
# update search index only on successful resolution
await self.search_service.index_entity(resolved_entity)
except IntegrityError:
# IntegrityError means a relation with this (from_id, to_id, relation_type)
# already exists. The UPDATE was rolled back, so our unresolved relation
# (to_id=NULL) still exists in the database. We delete it because:
# 1. It's redundant - a resolved relation already captures this relationship
# 2. If we don't delete it, future syncs will try to resolve it again
# and get the same IntegrityError
logger.debug(
"Ignoring duplicate relation "
"Deleting duplicate unresolved relation "
f"relation_id={relation.id} "
f"from_id={relation.from_id} "
f"to_name={relation.to_name}"
f"to_name={relation.to_name} "
f"resolved_to_id={resolved_entity.id}"
)
# update search index
await self.search_service.index_entity(resolved_entity)
try:
await self.relation_repository.delete(relation.id)
except Exception as e:
# Log but don't fail - the relation may have been deleted already
logger.debug(f"Could not delete duplicate relation {relation.id}: {e}")
async def _quick_count_files(self, directory: Path) -> int:
"""Fast file count using find command.
@@ -1063,8 +1047,6 @@ class SyncService:
f"error: {error_msg}. Falling back to manual count. "
f"This will slow down watermark detection!"
)
# Track optimization failures for visibility
logfire.metric_counter("sync.scan.file_count_failure").add(1)
# Fallback: count using scan_directory
count = 0
async for _ in self.scan_directory(directory):
@@ -1105,8 +1087,6 @@ class SyncService:
f"error: {error_msg}. Falling back to full scan. "
f"This will cause slow syncs on large projects!"
)
# Track optimization failures for visibility
logfire.metric_counter("sync.scan.optimization_failure").add(1)
# Fallback to full scan
return await self._scan_directory_full(directory)
+20 -5
View File
@@ -5,7 +5,10 @@ import os
from collections import defaultdict
from datetime import datetime
from pathlib import Path
from typing import List, Optional, Set, Sequence
from typing import List, Optional, Set, Sequence, Callable, Awaitable, TYPE_CHECKING
if TYPE_CHECKING:
from basic_memory.sync.sync_service import SyncService
from basic_memory.config import BasicMemoryConfig, WATCH_STATUS_JSON
from basic_memory.ignore_utils import load_gitignore_patterns, should_ignore_path
@@ -71,12 +74,17 @@ class WatchServiceState(BaseModel):
self.last_error = datetime.now()
# Type alias for sync service factory function
SyncServiceFactory = Callable[[Project], Awaitable["SyncService"]]
class WatchService:
def __init__(
self,
app_config: BasicMemoryConfig,
project_repository: ProjectRepository,
quiet: bool = False,
sync_service_factory: Optional[SyncServiceFactory] = None,
):
self.app_config = app_config
self.project_repository = project_repository
@@ -84,10 +92,20 @@ class WatchService:
self.status_path = Path.home() / ".basic-memory" / WATCH_STATUS_JSON
self.status_path.parent.mkdir(parents=True, exist_ok=True)
self._ignore_patterns_cache: dict[Path, Set[str]] = {}
self._sync_service_factory = sync_service_factory
# quiet mode for mcp so it doesn't mess up stdout
self.console = Console(quiet=quiet)
async def _get_sync_service(self, project: Project) -> "SyncService":
"""Get sync service for a project, using factory if provided."""
if self._sync_service_factory:
return await self._sync_service_factory(project)
# Fall back to default factory
from basic_memory.sync.sync_service import get_sync_service
return await get_sync_service(project)
async def _schedule_restart(self, stop_event: asyncio.Event):
"""Schedule a restart of the watch service after the configured interval."""
await asyncio.sleep(self.app_config.watch_project_reload_interval)
@@ -233,9 +251,6 @@ class WatchService:
async def handle_changes(self, project: Project, changes: Set[FileChange]) -> None:
"""Process a batch of file changes"""
# avoid circular imports
from basic_memory.sync.sync_service import get_sync_service
# Check if project still exists in configuration before processing
# This prevents deleted projects from being recreated by background sync
from basic_memory.config import ConfigManager
@@ -250,7 +265,7 @@ class WatchService:
)
return
sync_service = await get_sync_service(project)
sync_service = await self._get_sync_service(project)
file_service = sync_service.file_service
start_time = time.time()
+65 -64
View File
@@ -5,9 +5,9 @@ import os
import logging
import re
import sys
from datetime import datetime
from datetime import datetime, timezone
from pathlib import Path
from typing import Optional, Protocol, Union, runtime_checkable, List
from typing import Protocol, Union, runtime_checkable, List
from loguru import logger
from unidecode import unidecode
@@ -67,9 +67,6 @@ class PathLike(Protocol):
# This preserves compatibility with existing code while we migrate
FilePath = Union[Path, str]
# Disable the "Queue is full" warning
logging.getLogger("opentelemetry.sdk.metrics._internal.instrument").setLevel(logging.ERROR)
def generate_permalink(file_path: Union[Path, str, PathLike], split_extension: bool = True) -> str:
"""Generate a stable permalink from a file path.
@@ -206,29 +203,35 @@ def generate_permalink(file_path: Union[Path, str, PathLike], split_extension: b
def setup_logging(
env: str,
home_dir: Path,
log_file: Optional[str] = None,
log_level: str = "INFO",
console: bool = True,
log_to_file: bool = False,
log_to_stdout: bool = False,
structured_context: bool = False,
) -> None: # pragma: no cover
"""
Configure logging for the application.
"""Configure logging with explicit settings.
This function provides a simple, explicit interface for configuring logging.
Each entry point (CLI, MCP, API) should call this with appropriate settings.
Args:
env: The environment name (dev, test, prod)
home_dir: The root directory for the application
log_file: The name of the log file to write to
log_level: The logging level to use
console: Whether to log to the console
log_level: DEBUG, INFO, WARNING, ERROR
log_to_file: Write to ~/.basic-memory/basic-memory.log with rotation
log_to_stdout: Write to stderr (for Docker/cloud deployments)
structured_context: Bind tenant_id, fly_region, etc. for cloud observability
"""
# Remove default handler and any existing handlers
logger.remove()
# Add file handler if we are not running tests and a log file is specified
if log_file and env != "test":
# Setup file logger
log_path = home_dir / log_file
# In test mode, only log to stdout regardless of settings
env = os.getenv("BASIC_MEMORY_ENV", "dev")
if env == "test":
logger.add(sys.stderr, level=log_level, backtrace=True, diagnose=True, colorize=True)
return
# Add file handler with rotation
if log_to_file:
log_path = Path.home() / ".basic-memory" / "basic-memory.log"
log_path.parent.mkdir(parents=True, exist_ok=True)
logger.add(
str(log_path),
level=log_level,
@@ -236,42 +239,28 @@ def setup_logging(
retention="10 days",
backtrace=True,
diagnose=True,
enqueue=True,
enqueue=True, # Thread-safe async logging
colorize=False,
)
# Add console logger if requested or in test mode
if env == "test" or console:
# Add stdout handler (for Docker/cloud)
if log_to_stdout:
logger.add(sys.stderr, level=log_level, backtrace=True, diagnose=True, colorize=True)
logger.info(f"ENV: '{env}' Log level: '{log_level}' Logging to {log_file}")
# Bind environment context for structured logging (works in both local and cloud)
tenant_id = os.getenv("BASIC_MEMORY_TENANT_ID", "local")
fly_app_name = os.getenv("FLY_APP_NAME", "local")
fly_machine_id = os.getenv("FLY_MACHINE_ID", "local")
fly_region = os.getenv("FLY_REGION", "local")
logger.configure(
extra={
"tenant_id": tenant_id,
"fly_app_name": fly_app_name,
"fly_machine_id": fly_machine_id,
"fly_region": fly_region,
}
)
# Bind structured context for cloud observability
if structured_context:
logger.configure(
extra={
"tenant_id": os.getenv("BASIC_MEMORY_TENANT_ID", "local"),
"fly_app_name": os.getenv("FLY_APP_NAME", "local"),
"fly_machine_id": os.getenv("FLY_MACHINE_ID", "local"),
"fly_region": os.getenv("FLY_REGION", "local"),
}
)
# Reduce noise from third-party libraries
noisy_loggers = {
# HTTP client logs
"httpx": logging.WARNING,
# File watching logs
"watchfiles.main": logging.WARNING,
}
# Set log levels for noisy loggers
for logger_name, level in noisy_loggers.items():
logging.getLogger(logger_name).setLevel(level)
logging.getLogger("httpx").setLevel(logging.WARNING)
logging.getLogger("watchfiles.main").setLevel(logging.WARNING)
def parse_tags(tags: Union[List[str], str, None]) -> List[str]:
@@ -340,7 +329,7 @@ def normalize_file_path_for_comparison(file_path: str) -> str:
This function normalizes file paths to help detect potential conflicts:
- Converts to lowercase for case-insensitive comparison
- Normalizes Unicode characters
- Handles path separators consistently
- Converts backslashes to forward slashes for cross-platform consistency
Args:
file_path: The file path to normalize
@@ -349,19 +338,15 @@ def normalize_file_path_for_comparison(file_path: str) -> str:
Normalized file path for comparison purposes
"""
import unicodedata
from pathlib import PureWindowsPath
# Convert to lowercase for case-insensitive comparison
normalized = file_path.lower()
# Use PureWindowsPath to ensure backslashes are treated as separators
# regardless of current platform, then convert to POSIX-style
normalized = PureWindowsPath(file_path).as_posix().lower()
# Normalize Unicode characters (NFD normalization)
normalized = unicodedata.normalize("NFD", normalized)
# Replace path separators with forward slashes
normalized = normalized.replace("\\", "/")
# Remove multiple slashes
normalized = re.sub(r"/+", "/", normalized)
return normalized
@@ -445,21 +430,37 @@ def validate_project_path(path: str, project_path: Path) -> bool:
return False
def ensure_timezone_aware(dt: datetime) -> datetime:
"""Ensure a datetime is timezone-aware using system timezone.
def ensure_timezone_aware(dt: datetime, cloud_mode: bool | None = None) -> datetime:
"""Ensure a datetime is timezone-aware.
If the datetime is naive, convert it to timezone-aware using the system's local timezone.
If it's already timezone-aware, return it unchanged.
If the datetime is naive, convert it to timezone-aware. The interpretation
depends on cloud_mode:
- In cloud mode (PostgreSQL/asyncpg): naive datetimes are interpreted as UTC
- In local mode (SQLite): naive datetimes are interpreted as local time
asyncpg uses binary protocol which returns timestamps in UTC but as naive
datetimes. In cloud deployments, cloud_mode=True handles this correctly.
Args:
dt: The datetime to ensure is timezone-aware
cloud_mode: Optional explicit cloud_mode setting. If None, loads from config.
Returns:
A timezone-aware datetime
"""
if dt.tzinfo is None:
# Naive datetime - assume it's in local time and add timezone
return dt.astimezone()
# Determine cloud_mode: use explicit parameter if provided, otherwise load from config
if cloud_mode is None:
from basic_memory.config import ConfigManager
cloud_mode = ConfigManager().config.cloud_mode_enabled
if cloud_mode:
# Cloud/PostgreSQL mode: naive datetimes from asyncpg are already UTC
return dt.replace(tzinfo=timezone.utc)
else:
# Local/SQLite mode: naive datetimes are in local time
return dt.astimezone()
else:
# Already timezone-aware
return dt
+78 -49
View File
@@ -50,18 +50,23 @@ The `app` fixture ensures FastAPI dependency overrides are active, and
`mcp_server` provides the MCP server with proper project session initialization.
"""
import os
from typing import AsyncGenerator, Literal
import pytest
import pytest_asyncio
from pathlib import Path
from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
from sqlalchemy.pool import NullPool
from testcontainers.postgres import PostgresContainer
from httpx import AsyncClient, ASGITransport
from basic_memory.config import BasicMemoryConfig, ProjectConfig, ConfigManager, DatabaseBackend
from basic_memory.db import engine_session_factory, DatabaseType
from basic_memory.models import Project
from basic_memory.models.base import Base
from basic_memory.repository.project_repository import ProjectRepository
from fastapi import FastAPI
@@ -72,25 +77,38 @@ from basic_memory.deps import get_project_config, get_engine_factory, get_app_co
from basic_memory.mcp import tools # noqa: F401
@pytest.fixture(
params=[
pytest.param("sqlite", id="sqlite"),
pytest.param("postgres", id="postgres", marks=pytest.mark.postgres),
]
)
def db_backend(request) -> Literal["sqlite", "postgres"]:
"""Parametrize tests to run against both SQLite and Postgres.
# =============================================================================
# Database Backend Selection (env var approach)
# =============================================================================
# By default, integration tests run against SQLite.
# Set BASIC_MEMORY_TEST_POSTGRES=1 to run against Postgres (uses testcontainers).
Usage:
pytest # Runs tests against SQLite only (default)
pytest -m postgres # Runs tests against Postgres only
pytest -m "not postgres" # Runs tests against SQLite only
pytest --run-all-backends # Runs tests against both backends
Note: Only tests that use database fixtures (engine_factory, session_maker, etc.)
will be parametrized. Tests that don't use the database won't be affected.
@pytest.fixture(scope="session")
def db_backend() -> Literal["sqlite", "postgres"]:
"""Determine database backend from environment variable.
Default: sqlite
Set BASIC_MEMORY_TEST_POSTGRES=1 to use postgres
"""
return request.param
if os.environ.get("BASIC_MEMORY_TEST_POSTGRES", "").lower() in ("1", "true", "yes"):
return "postgres"
return "sqlite"
@pytest.fixture(scope="session")
def postgres_container(db_backend):
"""Session-scoped Postgres container for integration tests.
Uses testcontainers to spin up a real Postgres instance.
Only starts if db_backend is "postgres".
"""
if db_backend != "postgres":
yield None
return
with PostgresContainer("postgres:16-alpine") as postgres:
yield postgres
@pytest_asyncio.fixture
@@ -98,50 +116,57 @@ async def engine_factory(
app_config,
config_manager,
db_backend: Literal["sqlite", "postgres"],
postgres_container,
tmp_path,
) -> AsyncGenerator[tuple, None]:
"""Create engine and session factory for the configured database backend."""
from basic_memory.models.search import CREATE_SEARCH_INDEX
from basic_memory.models.search import (
CREATE_SEARCH_INDEX,
CREATE_POSTGRES_SEARCH_INDEX_TABLE,
CREATE_POSTGRES_SEARCH_INDEX_FTS,
CREATE_POSTGRES_SEARCH_INDEX_METADATA,
)
from basic_memory import db
# Determine database type based on backend
if db_backend == "postgres":
db_type = DatabaseType.FILESYSTEM
else:
db_type = DatabaseType.FILESYSTEM # Integration tests use file-based SQLite
# Postgres mode using testcontainers
sync_url = postgres_container.get_connection_url()
async_url = sync_url.replace("postgresql+psycopg2", "postgresql+asyncpg")
# Use tmp_path for SQLite, use config database_path for Postgres
if db_backend == "sqlite":
db_path = tmp_path / "test.db"
else:
db_path = app_config.database_path
engine = create_async_engine(
async_url,
echo=False,
poolclass=NullPool,
)
if db_backend == "postgres":
# Postgres: Create fresh engine for each test with full schema reset
config_manager._config = app_config
session_maker = async_sessionmaker(
bind=engine,
class_=AsyncSession,
expire_on_commit=False,
autoflush=False,
)
# Use context manager to handle engine disposal properly
async with engine_session_factory(db_path, db_type) as (engine, session_maker):
# Drop and recreate schema for complete isolation
async with engine.begin() as conn:
await conn.execute(text("DROP SCHEMA IF EXISTS public CASCADE"))
await conn.execute(text("CREATE SCHEMA public"))
await conn.execute(text("GRANT ALL ON SCHEMA public TO basic_memory_user"))
await conn.execute(text("GRANT ALL ON SCHEMA public TO public"))
# Drop and recreate all tables for test isolation
async with engine.begin() as conn:
await conn.execute(text("DROP TABLE IF EXISTS search_index CASCADE"))
await conn.run_sync(Base.metadata.drop_all)
await conn.run_sync(Base.metadata.create_all)
# asyncpg requires separate execute calls for each statement
await conn.execute(CREATE_POSTGRES_SEARCH_INDEX_TABLE)
await conn.execute(CREATE_POSTGRES_SEARCH_INDEX_FTS)
await conn.execute(CREATE_POSTGRES_SEARCH_INDEX_METADATA)
# Run migrations to create production tables
from basic_memory.db import run_migrations
yield engine, session_maker
await run_migrations(app_config, db_type)
yield engine, session_maker
await engine.dispose()
else:
# SQLite: Create fresh database (fast with tmp files)
db_path = tmp_path / "test.db"
db_type = DatabaseType.FILESYSTEM
async with engine_session_factory(db_path, db_type) as (engine, session_maker):
# Create all tables via ORM
from basic_memory.models.base import Base
async with engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
@@ -181,7 +206,11 @@ def config_home(tmp_path, monkeypatch) -> Path:
@pytest.fixture
def app_config(
config_home, db_backend: Literal["sqlite", "postgres"], tmp_path, monkeypatch
config_home,
db_backend: Literal["sqlite", "postgres"],
postgres_container,
tmp_path,
monkeypatch,
) -> BasicMemoryConfig:
"""Create test app configuration."""
# Disable cloud mode for CLI tests
@@ -190,12 +219,12 @@ def app_config(
# Create a basic config with test-project like unit tests do
projects = {"test-project": str(config_home)}
# Configure database backend based on test parameter
# Configure database backend based on env var
if db_backend == "postgres":
database_backend = DatabaseBackend.POSTGRES
database_url = (
"postgresql+asyncpg://basic_memory_user:dev_password@localhost:5433/basic_memory_test"
)
# Get URL from testcontainer and convert to asyncpg driver
sync_url = postgres_container.get_connection_url()
database_url = sync_url.replace("postgresql+psycopg2", "postgresql+asyncpg")
else:
database_backend = DatabaseBackend.SQLITE
database_url = None
@@ -0,0 +1,70 @@
"""
Integration test for FastAPI lifespan shutdown behavior.
This test verifies the asyncio cancellation pattern used by the API lifespan:
when the background sync task is cancelled during shutdown, it must be *awaited*
before database shutdown begins. This prevents "hang on exit" scenarios in
`asyncio.run(...)` callers (e.g. CLI/MCP clients using httpx ASGITransport).
"""
import asyncio
from httpx import ASGITransport, AsyncClient
def test_lifespan_shutdown_awaits_sync_task_cancellation(app, monkeypatch):
"""
Ensure lifespan shutdown awaits the cancelled background sync task.
Why this is deterministic:
- Cancelling a task does not make it "done" immediately; it becomes done only
once the event loop schedules it and it processes the CancelledError.
- In the buggy version, shutdown proceeded directly to db.shutdown_db()
immediately after calling cancel(), so at *entry* to shutdown_db the task
is still not done.
- In the fixed version, lifespan does `await sync_task` before shutdown_db,
so by the time shutdown_db is called, the task is done (cancelled).
"""
# Import the *module* (not the package-level FastAPI `basic_memory.api.app` export)
# so monkeypatching affects the exact symbols referenced inside lifespan().
#
# Note: `basic_memory/api/__init__.py` re-exports `app`, so `import basic_memory.api.app`
# can resolve to the FastAPI instance rather than the `basic_memory.api.app` module.
import importlib
api_app_module = importlib.import_module("basic_memory.api.app")
# Keep startup cheap: we don't need real DB init for this ordering test.
async def _noop_initialize_app(_app_config):
return None
async def _fake_get_or_create_db(*_args, **_kwargs):
return object(), object()
monkeypatch.setattr(api_app_module, "initialize_app", _noop_initialize_app)
monkeypatch.setattr(api_app_module.db, "get_or_create_db", _fake_get_or_create_db)
# Make the sync task long-lived so it must be cancelled on shutdown.
async def _fake_initialize_file_sync(_app_config):
await asyncio.Event().wait()
monkeypatch.setattr(api_app_module, "initialize_file_sync", _fake_initialize_file_sync)
# Assert ordering: shutdown_db must be called only after the sync_task is done.
async def _assert_sync_task_done_before_db_shutdown():
assert api_app_module.app.state.sync_task is not None
assert api_app_module.app.state.sync_task.done()
monkeypatch.setattr(api_app_module.db, "shutdown_db", _assert_sync_task_done_before_db_shutdown)
async def _run_client_once():
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
# Any request is sufficient to trigger lifespan startup/shutdown.
await client.get("/__nonexistent__")
# Use asyncio.run to match the CLI/MCP execution model where loop teardown
# would hang if a background task is left running.
asyncio.run(_run_client_once())
@@ -77,7 +77,8 @@ async def test_create_project_basic_operation(mcp_server, app, test_project):
assert "test-new-project" in create_text
assert "Project Details:" in create_text
assert "Name: test-new-project" in create_text
assert "Path: /tmp/test-new-project" in create_text
# Check path contains project name (platform-independent)
assert "Path:" in create_text and "test-new-project" in create_text
assert "Project is now available for use" in create_text
# Verify project appears in project list
@@ -46,3 +46,57 @@ async def test_read_note_after_write(mcp_server, app, test_project):
assert "# Test Note" in result_text
assert "This is test content." in result_text
assert "test/test-note" in result_text # permalink
@pytest.mark.asyncio
async def test_read_note_underscored_folder_by_permalink(mcp_server, app, test_project):
"""Test read_note with permalink from underscored folder.
Reproduces bug #416: read_note fails to find notes when given permalinks
from underscored folder names (e.g., _archive/, _drafts/), even though
the permalink is copied directly from the note's YAML frontmatter.
"""
async with Client(mcp_server) as client:
# Create a note in an underscored folder
write_result = await client.call_tool(
"write_note",
{
"project": test_project.name,
"title": "Example Note",
"folder": "_archive/articles",
"content": "# Example Note\n\nThis is a test note in an underscored folder.",
"tags": "test,archive",
},
)
assert len(write_result.content) == 1
assert write_result.content[0].type == "text"
write_text = write_result.content[0].text
# Verify the file path includes the underscore
assert "_archive/articles/Example Note.md" in write_text
# Verify the permalink has underscores stripped (this is the expected behavior)
assert "archive/articles/example-note" in write_text
# Now try to read the note using the permalink (without underscores)
# This is the exact scenario from the bug report - using the permalink
# that was generated in the YAML frontmatter
read_result = await client.call_tool(
"read_note",
{
"project": test_project.name,
"identifier": "archive/articles/example-note", # permalink without underscores
},
)
# This should succeed - the note should be found by its permalink
assert len(read_result.content) == 1
assert read_result.content[0].type == "text"
result_text = read_result.content[0].text
# Should contain the note content
assert "# Example Note" in result_text
assert "This is a test note in an underscored folder." in result_text
assert "archive/articles/example-note" in result_text # permalink
@@ -437,9 +437,7 @@ async def test_write_note_project_path_validation(mcp_server, app, test_project)
project_with_tilde = ProjectItem(
id=1,
name="Test BiSync", # Name differs from path structure
description="Test",
path="~/Documents/Test BiSync", # Path with tilde
is_active=True,
is_default=False,
)
-18
View File
@@ -5,7 +5,6 @@ and other SQLite configuration settings work correctly in production scenarios.
"""
import pytest
from unittest.mock import patch
from sqlalchemy import text
@@ -142,23 +141,6 @@ async def test_null_pool_on_windows(tmp_path, monkeypatch):
assert isinstance(engine.pool, NullPool)
@pytest.mark.asyncio
@pytest.mark.skipif(
__import__("os").name == "nt", reason="Non-Windows test - cannot mock POSIX paths on Windows"
)
async def test_regular_pool_on_non_windows(tmp_path):
"""Test that regular pooling is used on non-Windows platforms."""
from basic_memory.db import engine_session_factory, DatabaseType
from sqlalchemy.pool import NullPool
db_path = tmp_path / "test_posix_pool.db"
with patch("basic_memory.db.os.name", "posix"):
async with engine_session_factory(db_path, DatabaseType.FILESYSTEM) as (engine, _):
# Engine should NOT be using NullPool on non-Windows
assert not isinstance(engine.pool, NullPool)
@pytest.mark.asyncio
@pytest.mark.windows
@pytest.mark.skipif(
-369
View File
@@ -1,369 +0,0 @@
"""
Performance benchmark tests for sync operations.
These tests measure baseline performance for indexing operations to track
improvements from optimizations. Tests are marked with @pytest.mark.benchmark
and can be run separately.
Usage:
# Run all benchmarks
pytest test-int/test_sync_performance_benchmark.py -v
# Run specific benchmark
pytest test-int/test_sync_performance_benchmark.py::test_benchmark_sync_100_files -v
"""
import asyncio
import time
from pathlib import Path
from textwrap import dedent
import pytest
from basic_memory.config import BasicMemoryConfig, ProjectConfig
from basic_memory.sync.sync_service import get_sync_service
async def create_benchmark_file(path: Path, file_num: int, total_files: int) -> None:
"""Create a realistic test markdown file with observations and relations.
Args:
path: Path to create the file at
file_num: Current file number (for unique content)
total_files: Total number of files being created (for relation targets)
"""
# Create realistic content with varying complexity
has_relations = file_num < (total_files - 1) # Most files have relations
num_observations = min(3 + (file_num % 5), 10) # 3-10 observations per file
# Generate relation targets (some will be forward references)
relations = []
if has_relations:
# Reference 1-3 other files
num_relations = min(1 + (file_num % 3), 3)
for i in range(num_relations):
target_num = (file_num + i + 1) % total_files
relations.append(f"- relates_to [[test-file-{target_num:04d}]]")
content = dedent(f"""
---
type: note
tags: [benchmark, test, category-{file_num % 10}]
---
# Test File {file_num:04d}
This is benchmark test file {file_num} of {total_files}.
It contains realistic markdown content to simulate actual usage.
## Observations
{chr(10).join([f"- [category-{i % 5}] Observation {i} for file {file_num} with some content #tag{i}" for i in range(num_observations)])}
## Relations
{chr(10).join(relations) if relations else "- No relations for this file"}
## Additional Content
This section contains additional prose to simulate real documents.
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua.
### Subsection
More content here to make the file realistic. This helps test the
full indexing pipeline including content extraction and search indexing.
""").strip()
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(content, encoding="utf-8")
async def generate_benchmark_files(project_dir: Path, num_files: int) -> None:
"""Generate benchmark test files.
Args:
project_dir: Directory to create files in
num_files: Number of files to generate
"""
print(f"\nGenerating {num_files} test files...")
start = time.time()
# Create files in batches for faster generation
batch_size = 100
for batch_start in range(0, num_files, batch_size):
batch_end = min(batch_start + batch_size, num_files)
tasks = [
create_benchmark_file(
project_dir / f"category-{i % 10}" / f"test-file-{i:04d}.md", i, num_files
)
for i in range(batch_start, batch_end)
]
await asyncio.gather(*tasks)
print(f" Created files {batch_start}-{batch_end} ({batch_end}/{num_files})")
duration = time.time() - start
print(f" File generation completed in {duration:.2f}s ({num_files / duration:.1f} files/sec)")
def get_db_size(db_path: Path) -> tuple[int, str]:
"""Get database file size.
Returns:
Tuple of (size_bytes, formatted_size)
"""
if not db_path.exists():
return 0, "0 B"
size_bytes = db_path.stat().st_size
# Format size
for unit in ["B", "KB", "MB", "GB"]:
if size_bytes < 1024.0:
return size_bytes, f"{size_bytes:.2f} {unit}"
size_bytes /= 1024.0
return int(size_bytes * 1024**4), f"{size_bytes:.2f} TB"
async def run_sync_benchmark(
project_config: ProjectConfig, app_config: BasicMemoryConfig, num_files: int, test_name: str
) -> dict:
"""Run a sync benchmark and collect metrics.
Args:
project_config: Project configuration
app_config: App configuration
num_files: Number of files to benchmark
test_name: Name of the test for reporting
Returns:
Dictionary with benchmark results
"""
project_dir = project_config.home
db_path = app_config.database_path
print(f"\n{'=' * 70}")
print(f"BENCHMARK: {test_name}")
print(f"{'=' * 70}")
# Generate test files
await generate_benchmark_files(project_dir, num_files)
# Get initial DB size
initial_db_size, initial_db_formatted = get_db_size(db_path)
print(f"\nInitial database size: {initial_db_formatted}")
# Create sync service
from basic_memory.repository import ProjectRepository
from basic_memory import db
_, session_maker = await db.get_or_create_db(
db_path=app_config.database_path,
db_type=db.DatabaseType.FILESYSTEM,
)
project_repository = ProjectRepository(session_maker)
# Get or create project
projects = await project_repository.find_all()
if projects:
project = projects[0]
else:
project = await project_repository.create(
{
"name": project_config.name,
"path": str(project_config.home),
"is_active": True,
"is_default": True,
}
)
sync_service = await get_sync_service(project)
# Initialize search index (required for FTS5 table)
await sync_service.search_service.init_search_index()
# Run sync and measure time
print(f"\nStarting sync of {num_files} files...")
sync_start = time.time()
report = await sync_service.sync(project_dir, project_name=project.name)
sync_duration = time.time() - sync_start
# Get final DB size
final_db_size, final_db_formatted = get_db_size(db_path)
db_growth = final_db_size - initial_db_size
db_growth_formatted = f"{db_growth / 1024 / 1024:.2f} MB"
# Calculate metrics
files_per_sec = num_files / sync_duration if sync_duration > 0 else 0
ms_per_file = (sync_duration * 1000) / num_files if num_files > 0 else 0
# Print results
print(f"\n{'-' * 70}")
print("RESULTS:")
print(f"{'-' * 70}")
print(f"Files processed: {num_files}")
print(f" New: {len(report.new)}")
print(f" Modified: {len(report.modified)}")
print(f" Deleted: {len(report.deleted)}")
print(f" Moved: {len(report.moves)}")
print("\nPerformance:")
print(f" Total time: {sync_duration:.2f}s")
print(f" Files/sec: {files_per_sec:.1f}")
print(f" ms/file: {ms_per_file:.1f}")
print("\nDatabase:")
print(f" Initial size: {initial_db_formatted}")
print(f" Final size: {final_db_formatted}")
print(f" Growth: {db_growth_formatted}")
print(f" Growth per file: {(db_growth / num_files / 1024):.2f} KB")
print(f"{'=' * 70}\n")
return {
"test_name": test_name,
"num_files": num_files,
"sync_duration_sec": sync_duration,
"files_per_sec": files_per_sec,
"ms_per_file": ms_per_file,
"new_files": len(report.new),
"modified_files": len(report.modified),
"deleted_files": len(report.deleted),
"moved_files": len(report.moves),
"initial_db_size": initial_db_size,
"final_db_size": final_db_size,
"db_growth_bytes": db_growth,
"db_growth_per_file_bytes": db_growth / num_files if num_files > 0 else 0,
}
@pytest.mark.benchmark
@pytest.mark.asyncio
async def test_benchmark_sync_100_files(app_config, project_config, config_manager):
"""Benchmark: Sync 100 files (small repository)."""
results = await run_sync_benchmark(
project_config, app_config, num_files=100, test_name="Sync 100 files (small repository)"
)
# Basic assertions to ensure sync worked
# Note: May be slightly more than 100 due to OS-generated files (.DS_Store, etc.)
assert results["new_files"] >= 100
assert results["sync_duration_sec"] > 0
assert results["files_per_sec"] > 0
@pytest.mark.benchmark
@pytest.mark.asyncio
async def test_benchmark_sync_500_files(app_config, project_config, config_manager):
"""Benchmark: Sync 500 files (medium repository)."""
results = await run_sync_benchmark(
project_config, app_config, num_files=500, test_name="Sync 500 files (medium repository)"
)
# Basic assertions
# Note: May be slightly more than 500 due to OS-generated files
assert results["new_files"] >= 500
assert results["sync_duration_sec"] > 0
assert results["files_per_sec"] > 0
@pytest.mark.benchmark
@pytest.mark.asyncio
@pytest.mark.slow
async def test_benchmark_sync_1000_files(app_config, project_config, config_manager):
"""Benchmark: Sync 1000 files (large repository).
This test is marked as 'slow' and can be skipped in regular test runs:
pytest -m "not slow"
"""
results = await run_sync_benchmark(
project_config, app_config, num_files=1000, test_name="Sync 1000 files (large repository)"
)
# Basic assertions
# Note: May be slightly more than 1000 due to OS-generated files
assert results["new_files"] >= 1000
assert results["sync_duration_sec"] > 0
assert results["files_per_sec"] > 0
@pytest.mark.benchmark
@pytest.mark.asyncio
async def test_benchmark_resync_no_changes(app_config, project_config, config_manager):
"""Benchmark: Re-sync with no changes (should be fast).
This tests the performance of scanning files when nothing has changed,
which is important for cloud restarts.
"""
project_dir = project_config.home
num_files = 100
# First sync
print(f"\nFirst sync of {num_files} files...")
await generate_benchmark_files(project_dir, num_files)
from basic_memory.repository import ProjectRepository
from basic_memory import db
_, session_maker = await db.get_or_create_db(
db_path=app_config.database_path,
db_type=db.DatabaseType.FILESYSTEM,
)
project_repository = ProjectRepository(session_maker)
projects = await project_repository.find_all()
if projects:
project = projects[0]
else:
project = await project_repository.create(
{
"name": project_config.name,
"path": str(project_config.home),
"is_active": True,
"is_default": True,
}
)
sync_service = await get_sync_service(project)
# Initialize search index
await sync_service.search_service.init_search_index()
await sync_service.sync(project_dir, project_name=project.name)
# Second sync (no changes)
print("\nRe-sync with no changes...")
resync_start = time.time()
report = await sync_service.sync(project_dir, project_name=project.name)
resync_duration = time.time() - resync_start
print(f"\n{'-' * 70}")
print("RE-SYNC RESULTS (no changes):")
print(f"{'-' * 70}")
print(f"Files scanned: {num_files}")
print(f"Changes detected: {report.total}")
print(f" New: {len(report.new)}")
print(f" Modified: {len(report.modified)}")
print(f" Deleted: {len(report.deleted)}")
print(f" Moved: {len(report.moves)}")
print(f"Duration: {resync_duration:.2f}s")
print(f"Files/sec: {num_files / resync_duration:.1f}")
# Debug: Show what changed
if report.total > 0:
print("\n⚠️ UNEXPECTED CHANGES DETECTED:")
if report.new:
print(f" New files ({len(report.new)}): {list(report.new)[:5]}")
if report.modified:
print(f" Modified files ({len(report.modified)}): {list(report.modified)[:5]}")
if report.deleted:
print(f" Deleted files ({len(report.deleted)}): {list(report.deleted)[:5]}")
if report.moves:
print(f" Moved files ({len(report.moves)}): {dict(list(report.moves.items())[:5])}")
print(f"{'=' * 70}\n")
# Should be no changes
assert report.total == 0, (
f"Expected no changes but got {report.total}: new={len(report.new)}, modified={len(report.modified)}, deleted={len(report.deleted)}, moves={len(report.moves)}"
)
assert len(report.new) == 0
assert len(report.modified) == 0
assert len(report.deleted) == 0
+18 -14
View File
@@ -9,17 +9,19 @@ from basic_memory.mcp.async_client import create_client
def test_create_client_uses_asgi_when_no_remote_env():
"""Test that create_client uses ASGI transport when BASIC_MEMORY_USE_REMOTE_API is not set."""
# Ensure env vars are not set (pop if they exist)
"""Test that create_client uses ASGI transport when cloud mode is disabled."""
# Ensure env vars are not set and config cloud_mode is False
with patch.dict("os.environ", clear=False):
os.environ.pop("BASIC_MEMORY_USE_REMOTE_API", None)
os.environ.pop("BASIC_MEMORY_CLOUD_MODE", None)
client = create_client()
# Also patch the config's cloud_mode to ensure it's False
with patch.object(ConfigManager().config, "cloud_mode", False):
client = create_client()
assert isinstance(client, AsyncClient)
assert isinstance(client._transport, ASGITransport)
assert str(client.base_url) == "http://test"
assert isinstance(client, AsyncClient)
assert isinstance(client._transport, ASGITransport)
assert str(client.base_url) == "http://test"
def test_create_client_uses_http_when_cloud_mode_env_set():
@@ -37,16 +39,18 @@ def test_create_client_uses_http_when_cloud_mode_env_set():
def test_create_client_configures_extended_timeouts():
"""Test that create_client configures 30-second timeouts for long operations."""
# Ensure env vars are not set (pop if they exist)
# Ensure env vars are not set and config cloud_mode is False
with patch.dict("os.environ", clear=False):
os.environ.pop("BASIC_MEMORY_USE_REMOTE_API", None)
os.environ.pop("BASIC_MEMORY_CLOUD_MODE", None)
client = create_client()
# Also patch the config's cloud_mode to ensure it's False
with patch.object(ConfigManager().config, "cloud_mode", False):
client = create_client()
# Verify timeout configuration
assert isinstance(client.timeout, Timeout)
assert client.timeout.connect == 10.0 # 10 seconds for connection
assert client.timeout.read == 30.0 # 30 seconds for reading
assert client.timeout.write == 30.0 # 30 seconds for writing
assert client.timeout.pool == 30.0 # 30 seconds for pool
# Verify timeout configuration
assert isinstance(client.timeout, Timeout)
assert client.timeout.connect == 10.0 # 10 seconds for connection
assert client.timeout.read == 30.0 # 30 seconds for reading
assert client.timeout.write == 30.0 # 30 seconds for writing
assert client.timeout.pool == 30.0 # 30 seconds for pool
@@ -18,6 +18,7 @@ def template_loader():
def entity_summary():
"""Create a sample EntitySummary for testing."""
return EntitySummary(
entity_id=1,
title="Test Entity",
permalink="test/entity",
type=SearchItemType.ENTITY,
@@ -34,6 +35,8 @@ def context_with_results(entity_summary):
# Create an observation for the entity
observation = ObservationSummary(
observation_id=1,
entity_id=1,
title="Test Observation",
permalink="test/entity/observations/1",
category="test",
+5 -1
View File
@@ -218,9 +218,13 @@ async def test_get_resource_entities(client, project_config, entity_repository,
@pytest.mark.asyncio
async def test_get_resource_entities_pagination(
client, project_config, entity_repository, project_url
client, project_config, entity_repository, project_url, db_backend
):
"""Test getting content by permalink match."""
if db_backend == "postgres":
pytest.skip(
"Pagination differs: relations expand to multiple entities, ordering is undefined"
)
# Create entity
content1 = "# Test Content\n"
data = {
+1
View File
@@ -0,0 +1 @@
"""V2 API tests."""
+21
View File
@@ -0,0 +1,21 @@
"""Fixtures for V2 API tests."""
import pytest
from basic_memory.models import Project
@pytest.fixture
def v2_project_url(test_project: Project) -> str:
"""Create a URL prefix for v2 project-scoped routes using project ID.
This helps tests generate the correct URL for v2 project-scoped routes
which use integer project IDs instead of permalinks.
"""
return f"/v2/projects/{test_project.id}"
@pytest.fixture
def v2_projects_url() -> str:
"""Base URL for v2 project management endpoints."""
return "/v2/projects"
+129
View File
@@ -0,0 +1,129 @@
"""Tests for V2 directory API routes (ID-based endpoints)."""
import pytest
from httpx import AsyncClient
from basic_memory.models import Project
from basic_memory.schemas.directory import DirectoryNode
@pytest.mark.asyncio
async def test_get_directory_tree(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
):
"""Test getting directory tree via v2 endpoint."""
response = await client.get(f"{v2_project_url}/directory/tree")
assert response.status_code == 200
tree = DirectoryNode.model_validate(response.json())
assert tree.type == "directory"
@pytest.mark.asyncio
async def test_get_directory_structure(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
):
"""Test getting directory structure (folders only) via v2 endpoint."""
response = await client.get(f"{v2_project_url}/directory/structure")
assert response.status_code == 200
structure = DirectoryNode.model_validate(response.json())
assert structure.type == "directory"
# Structure should only contain directories, not files
if structure.children:
for child in structure.children:
assert child.type == "directory"
@pytest.mark.asyncio
async def test_list_directory_default(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
):
"""Test listing directory contents with default parameters via v2 endpoint."""
response = await client.get(f"{v2_project_url}/directory/list")
assert response.status_code == 200
nodes = response.json()
assert isinstance(nodes, list)
@pytest.mark.asyncio
async def test_list_directory_with_depth(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
):
"""Test listing directory with custom depth via v2 endpoint."""
response = await client.get(f"{v2_project_url}/directory/list?depth=2")
assert response.status_code == 200
nodes = response.json()
assert isinstance(nodes, list)
@pytest.mark.asyncio
async def test_list_directory_with_glob(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
):
"""Test listing directory with file name glob filter via v2 endpoint."""
response = await client.get(f"{v2_project_url}/directory/list?file_name_glob=*.md")
assert response.status_code == 200
nodes = response.json()
assert isinstance(nodes, list)
# All file nodes should have .md extension
for node in nodes:
if node.get("type") == "file":
assert node.get("path", "").endswith(".md")
@pytest.mark.asyncio
async def test_list_directory_with_custom_path(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
):
"""Test listing a specific directory path via v2 endpoint."""
response = await client.get(f"{v2_project_url}/directory/list?dir_name=/")
assert response.status_code == 200
nodes = response.json()
assert isinstance(nodes, list)
@pytest.mark.asyncio
async def test_directory_invalid_project_id(
client: AsyncClient,
):
"""Test directory endpoints with invalid project ID return 404."""
# Test tree endpoint
response = await client.get("/v2/projects/999999/directory/tree")
assert response.status_code == 404
# Test structure endpoint
response = await client.get("/v2/projects/999999/directory/structure")
assert response.status_code == 404
# Test list endpoint
response = await client.get("/v2/projects/999999/directory/list")
assert response.status_code == 404
@pytest.mark.asyncio
async def test_v2_directory_endpoints_use_project_id_not_name(
client: AsyncClient, test_project: Project
):
"""Verify v2 directory endpoints require project ID, not name."""
# Try using project name instead of ID - should fail
response = await client.get(f"/v2/projects/{test_project.name}/directory/tree")
# Should get validation error or 404 because name is not a valid integer
assert response.status_code in [404, 422]
+530
View File
@@ -0,0 +1,530 @@
"""Tests for V2 importer API routes (ID-based endpoints)."""
import json
from pathlib import Path
import pytest
from httpx import AsyncClient
from basic_memory.models import Project
from basic_memory.schemas.importer import (
ChatImportResult,
EntityImportResult,
ProjectImportResult,
)
@pytest.fixture
def chatgpt_json_content():
"""Sample ChatGPT conversation data for testing."""
return [
{
"title": "Test Conversation",
"create_time": 1736616594.24054,
"update_time": 1736616603.164995,
"mapping": {
"root": {"id": "root", "message": None, "parent": None, "children": ["msg1"]},
"msg1": {
"id": "msg1",
"message": {
"id": "msg1",
"author": {"role": "user", "name": None, "metadata": {}},
"create_time": 1736616594.24054,
"content": {
"content_type": "text",
"parts": ["Hello, this is a test message"],
},
"status": "finished_successfully",
"metadata": {},
},
"parent": "root",
"children": ["msg2"],
},
"msg2": {
"id": "msg2",
"message": {
"id": "msg2",
"author": {"role": "assistant", "name": None, "metadata": {}},
"create_time": 1736616603.164995,
"content": {"content_type": "text", "parts": ["This is a test response"]},
"status": "finished_successfully",
"metadata": {},
},
"parent": "msg1",
"children": [],
},
},
}
]
@pytest.fixture
def claude_conversations_json_content():
"""Sample Claude conversations data for testing."""
return [
{
"uuid": "test-uuid",
"name": "Test Conversation",
"created_at": "2025-01-05T20:55:32.499880+00:00",
"updated_at": "2025-01-05T20:56:39.477600+00:00",
"chat_messages": [
{
"uuid": "msg-1",
"text": "Hello, this is a test",
"sender": "human",
"created_at": "2025-01-05T20:55:32.499880+00:00",
"content": [{"type": "text", "text": "Hello, this is a test"}],
},
{
"uuid": "msg-2",
"text": "Response to test",
"sender": "assistant",
"created_at": "2025-01-05T20:55:40.123456+00:00",
"content": [{"type": "text", "text": "Response to test"}],
},
],
}
]
@pytest.fixture
def claude_projects_json_content():
"""Sample Claude projects data for testing."""
return [
{
"uuid": "test-uuid",
"name": "Test Project",
"created_at": "2025-01-05T20:55:32.499880+00:00",
"updated_at": "2025-01-05T20:56:39.477600+00:00",
"prompt_template": "# Test Prompt\n\nThis is a test prompt.",
"docs": [
{
"uuid": "doc-uuid-1",
"filename": "Test Document",
"content": "# Test Document\n\nThis is test content.",
"created_at": "2025-01-05T20:56:39.477600+00:00",
},
{
"uuid": "doc-uuid-2",
"filename": "Another Document",
"content": "# Another Document\n\nMore test content.",
"created_at": "2025-01-05T20:56:39.477600+00:00",
},
],
}
]
@pytest.fixture
def memory_json_content():
"""Sample memory.json data for testing."""
return [
{
"type": "entity",
"name": "test_entity",
"entityType": "test",
"observations": ["Test observation 1", "Test observation 2"],
},
{
"type": "relation",
"from": "test_entity",
"to": "related_entity",
"relationType": "test_relation",
},
]
async def create_test_upload_file(tmp_path, content):
"""Create a test file for upload."""
file_path = tmp_path / "test_import.json"
with open(file_path, "w", encoding="utf-8") as f:
json.dump(content, f)
return file_path
@pytest.mark.asyncio
async def test_import_chatgpt(
project_config,
client: AsyncClient,
tmp_path,
chatgpt_json_content,
file_service,
v2_project_url: str,
):
"""Test importing ChatGPT conversations via v2 endpoint."""
# Create a test file
file_path = await create_test_upload_file(tmp_path, chatgpt_json_content)
# Create a multipart form with the file
with open(file_path, "rb") as f:
files = {"file": ("conversations.json", f, "application/json")}
data = {"folder": "test_chatgpt"}
# Send request
response = await client.post(f"{v2_project_url}/import/chatgpt", files=files, data=data)
# Check response
assert response.status_code == 200
result = ChatImportResult.model_validate(response.json())
assert result.success is True
assert result.conversations == 1
assert result.messages == 2
# Verify files were created
conv_path = Path("test_chatgpt") / "20250111-Test_Conversation.md"
assert await file_service.exists(conv_path)
content, _ = await file_service.read_file(conv_path)
assert "# Test Conversation" in content
assert "Hello, this is a test message" in content
assert "This is a test response" in content
@pytest.mark.asyncio
async def test_import_chatgpt_invalid_file(client: AsyncClient, tmp_path, v2_project_url: str):
"""Test importing invalid ChatGPT file via v2 endpoint."""
# Create invalid file
file_path = tmp_path / "invalid.json"
with open(file_path, "w") as f:
f.write("This is not JSON")
# Create multipart form with invalid file
with open(file_path, "rb") as f:
files = {"file": ("invalid.json", f, "application/json")}
data = {"folder": "test_chatgpt"}
# Send request - this should return an error
response = await client.post(f"{v2_project_url}/import/chatgpt", files=files, data=data)
# Check response
assert response.status_code == 500
assert "Import failed" in response.json()["detail"]
@pytest.mark.asyncio
async def test_import_claude_conversations(
client: AsyncClient,
tmp_path,
claude_conversations_json_content,
file_service,
v2_project_url: str,
):
"""Test importing Claude conversations via v2 endpoint."""
# Create a test file
file_path = await create_test_upload_file(tmp_path, claude_conversations_json_content)
# Create a multipart form with the file
with open(file_path, "rb") as f:
files = {"file": ("conversations.json", f, "application/json")}
data = {"folder": "test_claude_conversations"}
# Send request
response = await client.post(
f"{v2_project_url}/import/claude/conversations", files=files, data=data
)
# Check response
assert response.status_code == 200
result = ChatImportResult.model_validate(response.json())
assert result.success is True
assert result.conversations == 1
assert result.messages == 2
# Verify files were created
conv_path = Path("test_claude_conversations") / "20250105-Test_Conversation.md"
assert await file_service.exists(conv_path)
content, _ = await file_service.read_file(conv_path)
assert "# Test Conversation" in content
assert "Hello, this is a test" in content
assert "Response to test" in content
@pytest.mark.asyncio
async def test_import_claude_conversations_invalid_file(
client: AsyncClient, tmp_path, v2_project_url: str
):
"""Test importing invalid Claude conversations file via v2 endpoint."""
# Create invalid file
file_path = tmp_path / "invalid.json"
with open(file_path, "w") as f:
f.write("This is not JSON")
# Create multipart form with invalid file
with open(file_path, "rb") as f:
files = {"file": ("invalid.json", f, "application/json")}
data = {"folder": "test_claude_conversations"}
# Send request - this should return an error
response = await client.post(
f"{v2_project_url}/import/claude/conversations", files=files, data=data
)
# Check response
assert response.status_code == 500
assert "Import failed" in response.json()["detail"]
@pytest.mark.asyncio
async def test_import_claude_projects(
client: AsyncClient, tmp_path, claude_projects_json_content, file_service, v2_project_url: str
):
"""Test importing Claude projects via v2 endpoint."""
# Create a test file
file_path = await create_test_upload_file(tmp_path, claude_projects_json_content)
# Create a multipart form with the file
with open(file_path, "rb") as f:
files = {"file": ("projects.json", f, "application/json")}
data = {"folder": "test_claude_projects"}
# Send request
response = await client.post(
f"{v2_project_url}/import/claude/projects", files=files, data=data
)
# Check response
assert response.status_code == 200
result = ProjectImportResult.model_validate(response.json())
assert result.success is True
assert result.documents == 2
assert result.prompts == 1
# Verify files were created
project_dir = Path("test_claude_projects") / "Test_Project"
assert await file_service.exists(project_dir / "prompt-template.md")
assert await file_service.exists(project_dir / "docs" / "Test_Document.md")
assert await file_service.exists(project_dir / "docs" / "Another_Document.md")
# Check content
prompt_content, _ = await file_service.read_file(project_dir / "prompt-template.md")
assert "# Test Prompt" in prompt_content
doc_content, _ = await file_service.read_file(project_dir / "docs" / "Test_Document.md")
assert "# Test Document" in doc_content
assert "This is test content" in doc_content
@pytest.mark.asyncio
async def test_import_claude_projects_invalid_file(
client: AsyncClient, tmp_path, v2_project_url: str
):
"""Test importing invalid Claude projects file via v2 endpoint."""
# Create invalid file
file_path = tmp_path / "invalid.json"
with open(file_path, "w") as f:
f.write("This is not JSON")
# Create multipart form with invalid file
with open(file_path, "rb") as f:
files = {"file": ("invalid.json", f, "application/json")}
data = {"folder": "test_claude_projects"}
# Send request - this should return an error
response = await client.post(
f"{v2_project_url}/import/claude/projects", files=files, data=data
)
# Check response
assert response.status_code == 500
assert "Import failed" in response.json()["detail"]
@pytest.mark.asyncio
async def test_import_memory_json(
client: AsyncClient, tmp_path, memory_json_content, file_service, v2_project_url: str
):
"""Test importing memory.json file via v2 endpoint."""
# Create a test file
json_file = tmp_path / "memory.json"
with open(json_file, "w", encoding="utf-8") as f:
for entity in memory_json_content:
f.write(json.dumps(entity) + "\n")
# Create a multipart form with the file
with open(json_file, "rb") as f:
files = {"file": ("memory.json", f, "application/json")}
data = {"folder": "test_memory_json"}
# Send request
response = await client.post(f"{v2_project_url}/import/memory-json", files=files, data=data)
# Check response
assert response.status_code == 200
result = EntityImportResult.model_validate(response.json())
assert result.success is True
assert result.entities == 1
assert result.relations == 1
# Verify files were created
entity_path = Path("test_memory_json") / "test" / "test_entity.md"
assert await file_service.exists(entity_path)
# Check content
content, _ = await file_service.read_file(entity_path)
assert "Test observation 1" in content
assert "Test observation 2" in content
assert "test_relation [[related_entity]]" in content
@pytest.mark.asyncio
async def test_import_memory_json_without_folder(
client: AsyncClient, tmp_path, memory_json_content, file_service, v2_project_url: str
):
"""Test importing memory.json file without specifying a destination folder."""
# Create a test file
json_file = tmp_path / "memory.json"
with open(json_file, "w", encoding="utf-8") as f:
for entity in memory_json_content:
f.write(json.dumps(entity) + "\n")
# Create a multipart form with the file
with open(json_file, "rb") as f:
files = {"file": ("memory.json", f, "application/json")}
# Send request without destination_folder
response = await client.post(f"{v2_project_url}/import/memory-json", files=files)
# Check response
assert response.status_code == 200
result = EntityImportResult.model_validate(response.json())
assert result.success is True
assert result.entities == 1
assert result.relations == 1
# Verify files were created in the default directory
entity_path = Path("conversations") / "test" / "test_entity.md"
assert await file_service.exists(entity_path)
@pytest.mark.asyncio
async def test_import_memory_json_invalid_file(client: AsyncClient, tmp_path, v2_project_url: str):
"""Test importing invalid memory.json file via v2 endpoint."""
# Create invalid file
file_path = tmp_path / "invalid.json"
with open(file_path, "w") as f:
f.write("This is not JSON")
# Create multipart form with invalid file
with open(file_path, "rb") as f:
files = {"file": ("invalid.json", f, "application/json")}
data = {"folder": "test_memory_json"}
# Send request - this should return an error
response = await client.post(f"{v2_project_url}/import/memory-json", files=files, data=data)
# Check response
assert response.status_code == 500
assert "Import failed" in response.json()["detail"]
@pytest.mark.asyncio
async def test_v2_import_endpoints_use_project_id_not_name(
client: AsyncClient, tmp_path, test_project: Project, chatgpt_json_content
):
"""Verify v2 import endpoints require project ID, not name."""
# Create a test file
file_path = await create_test_upload_file(tmp_path, chatgpt_json_content)
# Try using project name instead of ID - should fail
with open(file_path, "rb") as f:
files = {"file": ("conversations.json", f, "application/json")}
data = {"folder": "test"}
response = await client.post(
f"/v2/projects/{test_project.name}/import/chatgpt",
files=files,
data=data,
)
# Should get validation error or 404 because name is not a valid integer
assert response.status_code in [404, 422]
@pytest.mark.asyncio
async def test_import_invalid_project_id(client: AsyncClient, tmp_path, chatgpt_json_content):
"""Test import endpoints with invalid project ID return 404."""
# Create a test file
file_path = await create_test_upload_file(tmp_path, chatgpt_json_content)
# Test all import endpoints
endpoints = [
"/import/chatgpt",
"/import/claude/conversations",
"/import/claude/projects",
"/import/memory-json",
]
for endpoint in endpoints:
with open(file_path, "rb") as f:
files = {"file": ("test.json", f, "application/json")}
data = {"folder": "test"}
response = await client.post(
f"/v2/projects/999999{endpoint}",
files=files,
data=data,
)
assert response.status_code == 404
@pytest.mark.asyncio
async def test_import_missing_file(client: AsyncClient, v2_project_url: str):
"""Test importing with missing file via v2 endpoint."""
# Send a request without a file
response = await client.post(f"{v2_project_url}/import/chatgpt", data={"folder": "test_folder"})
# Check that the request was rejected
assert response.status_code in [400, 422] # Either bad request or unprocessable entity
@pytest.mark.asyncio
async def test_import_empty_file(client: AsyncClient, tmp_path, v2_project_url: str):
"""Test importing an empty file via v2 endpoint."""
# Create an empty file
file_path = tmp_path / "empty.json"
with open(file_path, "w") as f:
f.write("")
# Create multipart form with empty file
with open(file_path, "rb") as f:
files = {"file": ("empty.json", f, "application/json")}
data = {"folder": "test_chatgpt"}
# Send request
response = await client.post(f"{v2_project_url}/import/chatgpt", files=files, data=data)
# Check response
assert response.status_code == 500
assert "Import failed" in response.json()["detail"]
@pytest.mark.asyncio
async def test_import_malformed_json(client: AsyncClient, tmp_path, v2_project_url: str):
"""Test importing malformed JSON for all v2 import endpoints."""
# Create malformed JSON file
file_path = tmp_path / "malformed.json"
with open(file_path, "w") as f:
f.write('{"incomplete": "json"') # Missing closing brace
# Test all import endpoints
endpoints = [
(f"{v2_project_url}/import/chatgpt", {"folder": "test"}),
(f"{v2_project_url}/import/claude/conversations", {"folder": "test"}),
(f"{v2_project_url}/import/claude/projects", {"folder": "test"}),
(f"{v2_project_url}/import/memory-json", {"folder": "test"}),
]
for endpoint, data in endpoints:
# Create multipart form with malformed JSON
with open(file_path, "rb") as f:
files = {"file": ("malformed.json", f, "application/json")}
# Send request
response = await client.post(endpoint, files=files, data=data)
# Check response
assert response.status_code == 500
assert "Import failed" in response.json()["detail"]
+407
View File
@@ -0,0 +1,407 @@
"""Tests for V2 knowledge graph API routes (ID-based endpoints)."""
import pytest
from httpx import AsyncClient
from basic_memory.models import Project
from basic_memory.schemas import DeleteEntitiesResponse
from basic_memory.schemas.v2 import EntityResponseV2, EntityResolveResponse
@pytest.mark.asyncio
async def test_resolve_identifier_by_permalink(
client: AsyncClient, test_graph, v2_project_url, test_project: Project, entity_repository
):
"""Test resolving an identifier by permalink returns correct entity ID."""
# test_graph fixture creates some test entities
# We'll use one of them to test resolution
# Create an entity first
entity_data = {
"title": "TestResolve",
"folder": "test",
"content": "Test content for resolve",
}
response = await client.post(f"{v2_project_url}/knowledge/entities", json=entity_data)
assert response.status_code == 200
created_entity = EntityResponseV2.model_validate(response.json())
# V2 create must return id
assert created_entity.id is not None
entity_id = created_entity.id
# Now resolve it by permalink
resolve_data = {"identifier": created_entity.permalink}
response = await client.post(f"{v2_project_url}/knowledge/resolve", json=resolve_data)
assert response.status_code == 200
resolved = EntityResolveResponse.model_validate(response.json())
assert resolved.entity_id == entity_id
assert resolved.permalink == created_entity.permalink
assert resolved.resolution_method == "permalink"
@pytest.mark.asyncio
async def test_resolve_identifier_not_found(client: AsyncClient, v2_project_url):
"""Test resolving a non-existent identifier returns 404."""
resolve_data = {"identifier": "nonexistent/entity"}
response = await client.post(f"{v2_project_url}/knowledge/resolve", json=resolve_data)
assert response.status_code == 404
assert "Could not resolve identifier" in response.json()["detail"]
@pytest.mark.asyncio
async def test_get_entity_by_id(client: AsyncClient, test_graph, v2_project_url, entity_repository):
"""Test getting an entity by its numeric ID."""
# Create an entity first
entity_data = {
"title": "TestGetById",
"folder": "test",
"content": "Test content for get by ID",
}
response = await client.post(f"{v2_project_url}/knowledge/entities", json=entity_data)
assert response.status_code == 200
created_entity = EntityResponseV2.model_validate(response.json())
# V2 create must return id
assert created_entity.id is not None
entity_id = created_entity.id
# Get it by ID using v2 endpoint
response = await client.get(f"{v2_project_url}/knowledge/entities/{entity_id}")
assert response.status_code == 200
entity = EntityResponseV2.model_validate(response.json())
assert entity.id == entity_id
assert entity.title == "TestGetById"
assert entity.api_version == "v2"
@pytest.mark.asyncio
async def test_get_entity_by_id_not_found(client: AsyncClient, v2_project_url):
"""Test getting a non-existent entity by ID returns 404."""
response = await client.get(f"{v2_project_url}/knowledge/entities/999999")
assert response.status_code == 404
assert "not found" in response.json()["detail"].lower()
@pytest.mark.asyncio
async def test_create_entity(client: AsyncClient, file_service, v2_project_url):
"""Test creating an entity via v2 endpoint."""
data = {
"title": "TestV2Entity",
"folder": "test",
"entity_type": "test",
"content_type": "text/markdown",
"content": "TestContent for V2",
}
response = await client.post(f"{v2_project_url}/knowledge/entities", json=data)
assert response.status_code == 200
entity = EntityResponseV2.model_validate(response.json())
# V2 endpoints must return id field
assert entity.id is not None
assert isinstance(entity.id, int)
assert entity.api_version == "v2"
assert entity.permalink == "test/test-v2-entity"
assert entity.file_path == "test/TestV2Entity.md"
assert entity.entity_type == data["entity_type"]
# Verify file was created
file_path = file_service.get_entity_path(entity)
file_content, _ = await file_service.read_file(file_path)
assert data["content"] in file_content
@pytest.mark.asyncio
async def test_create_entity_with_observations_and_relations(
client: AsyncClient, file_service, v2_project_url
):
"""Test creating an entity with observations and relations via v2."""
data = {
"title": "TestV2Complex",
"folder": "test",
"content": """
# TestV2Complex
## Observations
- [note] This is a test observation #tag1 (context)
- related to [[OtherEntity]]
""",
}
response = await client.post(f"{v2_project_url}/knowledge/entities", json=data)
assert response.status_code == 200
entity = EntityResponseV2.model_validate(response.json())
# V2 endpoints must return id field
assert entity.id is not None
assert isinstance(entity.id, int)
assert entity.api_version == "v2"
assert len(entity.observations) == 1
assert entity.observations[0].category == "note"
assert entity.observations[0].content == "This is a test observation #tag1"
assert entity.observations[0].tags == ["tag1"]
assert len(entity.relations) == 1
assert entity.relations[0].relation_type == "related to"
@pytest.mark.asyncio
async def test_update_entity_by_id(
client: AsyncClient, file_service, v2_project_url, entity_repository
):
"""Test updating an entity by ID using PUT (replace)."""
# Create an entity first
create_data = {
"title": "TestUpdate",
"folder": "test",
"content": "Original content",
}
response = await client.post(f"{v2_project_url}/knowledge/entities", json=create_data)
assert response.status_code == 200
created_entity = EntityResponseV2.model_validate(response.json())
# V2 create must return id
assert created_entity.id is not None
original_id = created_entity.id
# Update it by ID
update_data = {
"title": "TestUpdate",
"folder": "test",
"content": "Updated content via V2",
}
response = await client.put(
f"{v2_project_url}/knowledge/entities/{original_id}",
json=update_data,
)
assert response.status_code == 200
updated_entity = EntityResponseV2.model_validate(response.json())
# V2 update must return id field
assert updated_entity.id is not None
assert isinstance(updated_entity.id, int)
assert updated_entity.api_version == "v2"
# Verify file was updated
file_path = file_service.get_entity_path(updated_entity)
file_content, _ = await file_service.read_file(file_path)
assert "Updated content via V2" in file_content
assert "Original content" not in file_content
@pytest.mark.asyncio
async def test_edit_entity_by_id_append(
client: AsyncClient, file_service, v2_project_url, entity_repository
):
"""Test editing an entity by ID using PATCH (append operation)."""
# Create an entity first
create_data = {
"title": "TestEdit",
"folder": "test",
"content": "# TestEdit\n\nOriginal content",
}
response = await client.post(f"{v2_project_url}/knowledge/entities", json=create_data)
assert response.status_code == 200
created_entity = EntityResponseV2.model_validate(response.json())
# V2 create must return id
assert created_entity.id is not None
original_id = created_entity.id
# Edit it by appending
edit_data = {
"operation": "append",
"content": "\n\n## New Section\n\nAppended content",
}
response = await client.patch(
f"{v2_project_url}/knowledge/entities/{original_id}",
json=edit_data,
)
assert response.status_code == 200
edited_entity = EntityResponseV2.model_validate(response.json())
# V2 patch must return id field
assert edited_entity.id is not None
assert isinstance(edited_entity.id, int)
assert edited_entity.api_version == "v2"
# Verify file has both original and appended content
file_path = file_service.get_entity_path(edited_entity)
file_content, _ = await file_service.read_file(file_path)
assert "Original content" in file_content
assert "Appended content" in file_content
@pytest.mark.asyncio
async def test_edit_entity_by_id_find_replace(
client: AsyncClient, file_service, v2_project_url, entity_repository
):
"""Test editing an entity by ID using PATCH (find/replace operation)."""
# Create an entity first
create_data = {
"title": "TestFindReplace",
"folder": "test",
"content": "# TestFindReplace\n\nOld text that will be replaced",
}
response = await client.post(f"{v2_project_url}/knowledge/entities", json=create_data)
assert response.status_code == 200
created_entity = EntityResponseV2.model_validate(response.json())
# V2 create must return id
assert created_entity.id is not None
original_id = created_entity.id
# Edit using find/replace
edit_data = {
"operation": "find_replace",
"find_text": "Old text",
"content": "New text",
}
response = await client.patch(
f"{v2_project_url}/knowledge/entities/{original_id}",
json=edit_data,
)
assert response.status_code == 200
edited_entity = EntityResponseV2.model_validate(response.json())
# V2 patch must return id field
assert edited_entity.id is not None
assert isinstance(edited_entity.id, int)
assert edited_entity.api_version == "v2"
# Verify replacement
file_path = file_service.get_entity_path(created_entity)
file_content, _ = await file_service.read_file(file_path)
assert "New text" in file_content
assert "Old text" not in file_content
@pytest.mark.asyncio
async def test_delete_entity_by_id(
client: AsyncClient, file_service, v2_project_url, entity_repository
):
"""Test deleting an entity by ID."""
# Create an entity first
create_data = {
"title": "TestDelete",
"folder": "test",
"content": "Content to be deleted",
}
response = await client.post(f"{v2_project_url}/knowledge/entities", json=create_data)
assert response.status_code == 200
created_entity = EntityResponseV2.model_validate(response.json())
# V2 create must return id
assert created_entity.id is not None
entity_id = created_entity.id
# Delete it by ID
response = await client.delete(f"{v2_project_url}/knowledge/entities/{entity_id}")
assert response.status_code == 200
delete_response = DeleteEntitiesResponse.model_validate(response.json())
assert delete_response.deleted is True
# Verify it's gone - trying to get it should return 404
response = await client.get(f"{v2_project_url}/knowledge/entities/{entity_id}")
assert response.status_code == 404
@pytest.mark.asyncio
async def test_delete_entity_by_id_not_found(client: AsyncClient, v2_project_url):
"""Test deleting a non-existent entity returns deleted=False (idempotent)."""
response = await client.delete(f"{v2_project_url}/knowledge/entities/999999")
# Delete is idempotent - returns 200 with deleted=False
assert response.status_code == 200
delete_response = DeleteEntitiesResponse.model_validate(response.json())
assert delete_response.deleted is False
@pytest.mark.asyncio
async def test_move_entity(client: AsyncClient, file_service, v2_project_url, entity_repository):
"""Test moving an entity to a new location."""
# Create an entity first
create_data = {
"title": "TestMove",
"folder": "test",
"content": "Content to be moved",
}
response = await client.post(f"{v2_project_url}/knowledge/entities", json=create_data)
assert response.status_code == 200
created_entity = EntityResponseV2.model_validate(response.json())
# V2 create must return id
assert created_entity.id is not None
original_id = created_entity.id
# Move it to a new folder (V2 uses entity ID in path)
move_data = {
"destination_path": "moved/MovedEntity.md",
}
response = await client.put(
f"{v2_project_url}/knowledge/entities/{created_entity.id}/move", json=move_data
)
assert response.status_code == 200
moved_entity = EntityResponseV2.model_validate(response.json())
# V2 move must return id field
assert moved_entity.id is not None
assert isinstance(moved_entity.id, int)
assert moved_entity.api_version == "v2"
# ID should remain the same (stable reference)
assert moved_entity.id == original_id
assert moved_entity.file_path == "moved/MovedEntity.md"
@pytest.mark.asyncio
async def test_v2_endpoints_use_project_id_not_name(client: AsyncClient, test_project: Project):
"""Verify v2 endpoints require project ID, not name."""
# Try using project name instead of ID - should fail
response = await client.get(f"/v2/{test_project.name}/knowledge/entities/1")
# Should get validation error or 404 because name is not a valid integer
assert response.status_code in [404, 422]
@pytest.mark.asyncio
async def test_entity_response_v2_has_api_version(
client: AsyncClient, v2_project_url, entity_repository
):
"""Test that EntityResponseV2 includes api_version field."""
# Create an entity
entity_data = {
"title": "TestApiVersion",
"folder": "test",
"content": "Test content",
}
response = await client.post(f"{v2_project_url}/knowledge/entities", json=entity_data)
assert response.status_code == 200
created_entity = EntityResponseV2.model_validate(response.json())
# V2 create must return id and api_version
assert created_entity.id is not None
assert created_entity.api_version == "v2"
entity_id = created_entity.id
# Get it via v2 endpoint
response = await client.get(f"{v2_project_url}/knowledge/entities/{entity_id}")
assert response.status_code == 200
entity_v2 = EntityResponseV2.model_validate(response.json())
assert entity_v2.api_version == "v2"
assert entity_v2.id == entity_id
+301
View File
@@ -0,0 +1,301 @@
"""Tests for v2 memory router endpoints."""
import pytest
from httpx import AsyncClient
from pathlib import Path
from basic_memory.models import Project
async def create_test_entity(
test_project, entity_data, entity_repository, search_service, file_service
):
"""Helper to create an entity with file and index it."""
# Create file
test_content = f"# {entity_data['title']}\n\nTest content"
file_path = Path(test_project.path) / entity_data["file_path"]
file_path.parent.mkdir(parents=True, exist_ok=True)
await file_service.write_file(file_path, test_content)
# Create entity
entity = await entity_repository.create(entity_data)
# Index for search
await search_service.index_entity(entity)
return entity
@pytest.mark.asyncio
async def test_get_recent_context(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
entity_repository,
search_service,
file_service,
):
"""Test getting recent activity context."""
entity_data = {
"title": "Recent Test Entity",
"entity_type": "note",
"content_type": "text/markdown",
"file_path": "recent_test.md",
"checksum": "abc123",
}
await create_test_entity(
test_project, entity_data, entity_repository, search_service, file_service
)
# Get recent context
response = await client.get(f"{v2_project_url}/memory/recent")
assert response.status_code == 200
data = response.json()
# Verify response structure (GraphContext uses 'results' not 'entities')
assert "results" in data
assert "metadata" in data
assert "page" in data
assert "page_size" in data
@pytest.mark.asyncio
async def test_get_recent_context_with_pagination(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
entity_repository,
search_service,
file_service,
):
"""Test recent context with pagination parameters."""
# Create multiple test entities
for i in range(5):
entity_data = {
"title": f"Entity {i}",
"entity_type": "note",
"content_type": "text/markdown",
"file_path": f"entity_{i}.md",
"checksum": f"checksum{i}",
}
await create_test_entity(
test_project, entity_data, entity_repository, search_service, file_service
)
# Get recent context with pagination
response = await client.get(
f"{v2_project_url}/memory/recent", params={"page": 1, "page_size": 3}
)
assert response.status_code == 200
data = response.json()
assert "results" in data
assert data["page"] == 1
assert data["page_size"] == 3
@pytest.mark.asyncio
async def test_get_recent_context_with_type_filter(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
entity_repository,
search_service,
file_service,
):
"""Test filtering recent context by type."""
# Create a test entity
entity_data = {
"title": "Filtered Entity",
"entity_type": "note",
"content_type": "text/markdown",
"file_path": "filtered.md",
"checksum": "xyz789",
}
await create_test_entity(
test_project, entity_data, entity_repository, search_service, file_service
)
# Get recent context filtered by type
response = await client.get(f"{v2_project_url}/memory/recent", params={"type": ["entity"]})
assert response.status_code == 200
data = response.json()
assert "results" in data
@pytest.mark.asyncio
async def test_get_recent_context_with_timeframe(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
):
"""Test recent context with custom timeframe."""
response = await client.get(f"{v2_project_url}/memory/recent", params={"timeframe": "1d"})
assert response.status_code == 200
data = response.json()
assert "results" in data
@pytest.mark.asyncio
async def test_get_recent_context_invalid_project_id(
client: AsyncClient,
):
"""Test getting recent context with invalid project ID returns 404."""
response = await client.get("/v2/projects/999999/memory/recent")
assert response.status_code == 404
@pytest.mark.asyncio
async def test_get_memory_context_by_permalink(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
entity_repository,
search_service,
file_service,
):
"""Test getting context for a specific memory URI (permalink)."""
# Create a test entity
entity_data = {
"title": "Context Test",
"entity_type": "note",
"content_type": "text/markdown",
"file_path": "context_test.md",
"checksum": "def456",
"permalink": "context-test",
}
await create_test_entity(
test_project, entity_data, entity_repository, search_service, file_service
)
# Get context for this entity
response = await client.get(f"{v2_project_url}/memory/context-test")
assert response.status_code == 200
data = response.json()
assert "results" in data
@pytest.mark.asyncio
async def test_get_memory_context_by_id(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
entity_repository,
search_service,
file_service,
):
"""Test getting context using ID-based memory URI."""
# Create a test entity
entity_data = {
"title": "ID Context Test",
"entity_type": "note",
"content_type": "text/markdown",
"file_path": "id_context_test.md",
"checksum": "ghi789",
}
created_entity = await create_test_entity(
test_project, entity_data, entity_repository, search_service, file_service
)
# Get context using ID format (memory://id/123 or memory://123)
response = await client.get(f"{v2_project_url}/memory/id/{created_entity.id}")
assert response.status_code == 200
data = response.json()
assert "results" in data
@pytest.mark.asyncio
async def test_get_memory_context_with_depth(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
entity_repository,
search_service,
file_service,
):
"""Test getting context with depth parameter."""
# Create a test entity
entity_data = {
"title": "Depth Test",
"entity_type": "note",
"content_type": "text/markdown",
"file_path": "depth_test.md",
"checksum": "jkl012",
"permalink": "depth-test",
}
await create_test_entity(
test_project, entity_data, entity_repository, search_service, file_service
)
# Get context with depth
response = await client.get(f"{v2_project_url}/memory/depth-test", params={"depth": 2})
assert response.status_code == 200
data = response.json()
assert "results" in data
@pytest.mark.asyncio
async def test_get_memory_context_not_found(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
):
"""Test getting context for non-existent memory URI returns 404."""
response = await client.get(f"{v2_project_url}/memory/nonexistent-uri")
# Note: This might return 200 with empty results depending on implementation
# Adjust assertion based on actual behavior
assert response.status_code in [200, 404]
@pytest.mark.asyncio
async def test_get_memory_context_with_timeframe(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
entity_repository,
search_service,
file_service,
):
"""Test getting context with timeframe filter."""
# Create a test entity
entity_data = {
"title": "Timeframe Test",
"entity_type": "note",
"content_type": "text/markdown",
"file_path": "timeframe_test.md",
"checksum": "mno345",
"permalink": "timeframe-test",
}
await create_test_entity(
test_project, entity_data, entity_repository, search_service, file_service
)
# Get context with timeframe
response = await client.get(
f"{v2_project_url}/memory/timeframe-test", params={"timeframe": "7d"}
)
assert response.status_code == 200
data = response.json()
assert "results" in data
@pytest.mark.asyncio
async def test_v2_memory_endpoints_use_project_id_not_name(
client: AsyncClient,
test_project: Project,
):
"""Test that v2 memory endpoints reject string project names."""
# Try to use project name instead of ID - should fail
response = await client.get(f"/v2/{test_project.name}/memory/recent")
# FastAPI path validation should reject non-integer project_id
assert response.status_code in [404, 422]
+251
View File
@@ -0,0 +1,251 @@
"""Tests for V2 project management API routes (ID-based endpoints)."""
import tempfile
from pathlib import Path
import pytest
from httpx import AsyncClient
from basic_memory.models import Project
from basic_memory.schemas.project_info import ProjectItem, ProjectStatusResponse
@pytest.mark.asyncio
async def test_get_project_by_id(client: AsyncClient, test_project: Project, v2_projects_url):
"""Test getting a project by its numeric ID."""
response = await client.get(f"{v2_projects_url}/{test_project.id}")
assert response.status_code == 200
project = ProjectItem.model_validate(response.json())
assert project.id == test_project.id
assert project.name == test_project.name
assert project.path == test_project.path
assert project.is_default == (test_project.is_default or False)
@pytest.mark.asyncio
async def test_get_project_by_id_not_found(client: AsyncClient, v2_projects_url):
"""Test getting a non-existent project by ID returns 404."""
response = await client.get(f"{v2_projects_url}/999999")
assert response.status_code == 404
assert "not found" in response.json()["detail"].lower()
@pytest.mark.asyncio
async def test_update_project_path_by_id(
client: AsyncClient, test_project: Project, v2_projects_url
):
"""Test updating a project's path by ID."""
with tempfile.TemporaryDirectory() as tmpdir:
new_path = str(Path(tmpdir) / "new-project-location")
Path(new_path).mkdir(parents=True, exist_ok=True)
update_data = {"path": new_path}
response = await client.patch(
f"{v2_projects_url}/{test_project.id}",
json=update_data,
)
assert response.status_code == 200
status_response = ProjectStatusResponse.model_validate(response.json())
assert status_response.status == "success"
assert status_response.new_project.id == test_project.id
# Normalize paths for cross-platform comparison (Windows uses backslashes, API returns forward slashes)
assert Path(status_response.new_project.path) == Path(new_path)
assert status_response.old_project.id == test_project.id
@pytest.mark.asyncio
async def test_update_project_invalid_path(
client: AsyncClient, test_project: Project, v2_projects_url
):
"""Test updating with a relative path returns 400."""
update_data = {"path": "relative/path"}
response = await client.patch(
f"{v2_projects_url}/{test_project.id}",
json=update_data,
)
assert response.status_code == 400
assert "absolute" in response.json()["detail"].lower()
@pytest.mark.asyncio
async def test_update_project_not_found(client: AsyncClient, v2_projects_url):
"""Test updating a non-existent project returns 404."""
update_data = {"path": "/tmp/new-path"}
response = await client.patch(
f"{v2_projects_url}/999999",
json=update_data,
)
assert response.status_code == 404
@pytest.mark.asyncio
async def test_set_default_project_by_id(
client: AsyncClient, test_project: Project, v2_projects_url, project_repository, project_service
):
"""Test setting a project as default by ID."""
# Create a second project to test setting default
await project_service.add_project("second-project", "/tmp/second-project")
# Get the created project from the repository to get its ID
created_project = await project_repository.get_by_name("second-project")
assert created_project is not None
# Set the second project as default
response = await client.put(f"{v2_projects_url}/{created_project.id}/default")
assert response.status_code == 200
status_response = ProjectStatusResponse.model_validate(response.json())
assert status_response.status == "success"
assert status_response.default is True
assert status_response.new_project.id == created_project.id
assert status_response.new_project.is_default is True
assert status_response.old_project.id == test_project.id
assert status_response.old_project.is_default is False
@pytest.mark.asyncio
async def test_set_default_project_not_found(client: AsyncClient, v2_projects_url):
"""Test setting a non-existent project as default returns 404."""
response = await client.put(f"{v2_projects_url}/999999/default")
assert response.status_code == 404
@pytest.mark.asyncio
async def test_delete_project_by_id(
client: AsyncClient, test_project: Project, v2_projects_url, project_repository, project_service
):
"""Test deleting a project by ID."""
# Create a second project since we can't delete the default
await project_service.add_project("to-delete", "/tmp/to-delete")
# Get the created project from the repository to get its ID
created_project = await project_repository.get_by_name("to-delete")
assert created_project is not None
# Delete it
response = await client.delete(f"{v2_projects_url}/{created_project.id}")
assert response.status_code == 200
status_response = ProjectStatusResponse.model_validate(response.json())
assert status_response.status == "success"
assert status_response.old_project.id == created_project.id
assert status_response.new_project is None
# Verify it's deleted - trying to get it should return 404
response = await client.get(f"{v2_projects_url}/{created_project.id}")
assert response.status_code == 404
@pytest.mark.asyncio
async def test_delete_project_with_delete_notes_param(
client: AsyncClient, test_project: Project, v2_projects_url, project_repository, project_service
):
"""Test deleting a project with delete_notes parameter."""
# Create a project in a temp directory
with tempfile.TemporaryDirectory() as tmpdir:
project_path = Path(tmpdir) / "test-delete-notes"
project_path.mkdir(parents=True, exist_ok=True)
# Create a test file in the project
test_file = project_path / "test.md"
test_file.write_text("Test content")
await project_service.add_project("delete-with-notes", str(project_path))
# Get the created project from the repository to get its ID
created_project = await project_repository.get_by_name("delete-with-notes")
assert created_project is not None
# Delete with delete_notes=true
response = await client.delete(f"{v2_projects_url}/{created_project.id}?delete_notes=true")
assert response.status_code == 200
# Verify directory was deleted
assert not project_path.exists()
@pytest.mark.asyncio
async def test_delete_default_project_fails(
client: AsyncClient, test_project: Project, v2_projects_url
):
"""Test that deleting the default project returns 400."""
# test_project is the default project
response = await client.delete(f"{v2_projects_url}/{test_project.id}")
assert response.status_code == 400
assert "default project" in response.json()["detail"].lower()
@pytest.mark.asyncio
async def test_delete_project_not_found(client: AsyncClient, v2_projects_url):
"""Test deleting a non-existent project returns 404."""
response = await client.delete(f"{v2_projects_url}/999999")
assert response.status_code == 404
@pytest.mark.asyncio
async def test_v2_project_endpoints_use_id_not_name(
client: AsyncClient, test_project: Project, v2_projects_url
):
"""Verify v2 project endpoints require project ID, not name."""
# Try using project name instead of ID - should fail
response = await client.get(f"{v2_projects_url}/{test_project.name}")
# Should get 404 or 422 because name is not a valid integer
assert response.status_code in [404, 422]
@pytest.mark.asyncio
async def test_project_id_stability_after_rename(
client: AsyncClient, test_project: Project, v2_projects_url, project_repository
):
"""Test that project ID remains stable even after renaming."""
original_id = test_project.id
original_name = test_project.name
# Get project by ID
response = await client.get(f"{v2_projects_url}/{original_id}")
assert response.status_code == 200
project_before = ProjectItem.model_validate(response.json())
assert project_before.id == original_id
assert project_before.name == original_name
# Even if we renamed the project (not testing rename here, just the concept),
# the ID would stay the same. This test demonstrates the stability.
# Re-fetch by same ID
response = await client.get(f"{v2_projects_url}/{original_id}")
assert response.status_code == 200
project_after = ProjectItem.model_validate(response.json())
assert project_after.id == original_id
@pytest.mark.asyncio
async def test_update_project_active_status(
client: AsyncClient, test_project: Project, v2_projects_url, project_repository, project_service
):
"""Test updating a project's active status by ID."""
# Create a non-default project
await project_service.add_project("test-active", "/tmp/test-active")
# Get the created project from the repository to get its ID
created_project = await project_repository.get_by_name("test-active")
assert created_project is not None
# Update active status
update_data = {"is_active": False}
response = await client.patch(
f"{v2_projects_url}/{created_project.id}",
json=update_data,
)
assert response.status_code == 200
status_response = ProjectStatusResponse.model_validate(response.json())
assert status_response.status == "success"
+212
View File
@@ -0,0 +1,212 @@
"""Tests for V2 prompt router endpoints (ID-based)."""
import pytest
import pytest_asyncio
from httpx import AsyncClient
from basic_memory.models import Project
from basic_memory.services.context_service import ContextService
@pytest_asyncio.fixture
async def context_service(entity_repository, search_service, observation_repository):
"""Create a real context service for testing."""
return ContextService(entity_repository, search_service, observation_repository)
@pytest.mark.asyncio
async def test_continue_conversation_endpoint(
client: AsyncClient,
entity_service,
search_service,
context_service,
entity_repository,
test_graph,
v2_project_url: str,
):
"""Test the v2 continue_conversation endpoint with real services."""
# Create request data
request_data = {
"topic": "Root", # This should match our test entity in test_graph
"timeframe": "7d",
"depth": 1,
"related_items_limit": 2,
}
# Call the endpoint
response = await client.post(
f"{v2_project_url}/prompt/continue-conversation", json=request_data
)
# Verify response
assert response.status_code == 200
result = response.json()
assert "prompt" in result
assert "context" in result
# Check content of context
context = result["context"]
assert context["topic"] == "Root"
assert context["timeframe"] == "7d"
assert context["has_results"] is True
assert len(context["hierarchical_results"]) > 0
# Check content of prompt
prompt = result["prompt"]
assert "Continuing conversation on: Root" in prompt
assert "memory retrieval session" in prompt
@pytest.mark.asyncio
async def test_continue_conversation_without_topic(
client: AsyncClient,
entity_service,
search_service,
context_service,
entity_repository,
test_graph,
v2_project_url: str,
):
"""Test v2 continue_conversation without topic - should use recent activity."""
request_data = {"timeframe": "1d", "depth": 1, "related_items_limit": 2}
response = await client.post(
f"{v2_project_url}/prompt/continue-conversation", json=request_data
)
assert response.status_code == 200
result = response.json()
assert "Recent Activity" in result["context"]["topic"]
@pytest.mark.asyncio
async def test_search_prompt_endpoint(
client: AsyncClient, entity_service, search_service, test_graph, v2_project_url: str
):
"""Test the v2 search_prompt endpoint with real services."""
# Create request data
request_data = {
"query": "Root", # This should match our test entity
"timeframe": "7d",
}
# Call the endpoint
response = await client.post(f"{v2_project_url}/prompt/search", json=request_data)
# Verify response
assert response.status_code == 200
result = response.json()
assert "prompt" in result
assert "context" in result
# Check content of context
context = result["context"]
assert context["query"] == "Root"
assert context["timeframe"] == "7d"
assert context["has_results"] is True
assert len(context["results"]) > 0
# Check content of prompt
prompt = result["prompt"]
assert 'Search Results for: "Root"' in prompt
assert "This is a memory search session" in prompt
@pytest.mark.asyncio
async def test_search_prompt_no_results(
client: AsyncClient, entity_service, search_service, v2_project_url: str
):
"""Test the v2 search_prompt endpoint with a query that returns no results."""
# Create request data with a query that shouldn't match anything
request_data = {"query": "NonExistentQuery12345", "timeframe": "7d"}
# Call the endpoint
response = await client.post(f"{v2_project_url}/prompt/search", json=request_data)
# Verify response
assert response.status_code == 200
result = response.json()
# Check content of context
context = result["context"]
assert context["query"] == "NonExistentQuery12345"
assert context["has_results"] is False
assert len(context["results"]) == 0
# Check content of prompt
prompt = result["prompt"]
assert 'Search Results for: "NonExistentQuery12345"' in prompt
assert "I couldn't find any results for this query" in prompt
assert "Opportunity to Capture Knowledge" in prompt
@pytest.mark.asyncio
async def test_error_handling(client: AsyncClient, monkeypatch, v2_project_url: str):
"""Test error handling in v2 endpoints by breaking the template loader."""
# Patch the template loader to raise an exception
def mock_render(*args, **kwargs):
raise Exception("Template error")
# Apply the patch
monkeypatch.setattr("basic_memory.api.template_loader.TemplateLoader.render", mock_render)
# Test continue_conversation error handling
response = await client.post(
f"{v2_project_url}/prompt/continue-conversation",
json={"topic": "test error", "timeframe": "7d"},
)
assert response.status_code == 500
assert "detail" in response.json()
assert "Template error" in response.json()["detail"]
# Test search_prompt error handling
response = await client.post(
f"{v2_project_url}/prompt/search", json={"query": "test error", "timeframe": "7d"}
)
assert response.status_code == 500
assert "detail" in response.json()
assert "Template error" in response.json()["detail"]
@pytest.mark.asyncio
async def test_v2_prompt_endpoints_use_project_id_not_name(
client: AsyncClient, test_project: Project
):
"""Verify v2 prompt endpoints require project ID, not name."""
# Try using project name instead of ID - should fail
response = await client.post(
f"/v2/projects/{test_project.name}/prompt/continue-conversation",
json={"topic": "test", "timeframe": "7d"},
)
# Should get validation error or 404 because name is not a valid integer
assert response.status_code in [404, 422]
# Also test search endpoint
response = await client.post(
f"/v2/projects/{test_project.name}/prompt/search",
json={"query": "test", "timeframe": "7d"},
)
assert response.status_code in [404, 422]
@pytest.mark.asyncio
async def test_prompt_invalid_project_id(client: AsyncClient):
"""Test prompt endpoints with invalid project ID return 404."""
# Test continue-conversation
response = await client.post(
"/v2/projects/999999/prompt/continue-conversation",
json={"topic": "test", "timeframe": "7d"},
)
assert response.status_code == 404
# Test search
response = await client.post(
"/v2/projects/999999/prompt/search",
json={"query": "test", "timeframe": "7d"},
)
assert response.status_code == 404
+267
View File
@@ -0,0 +1,267 @@
"""Tests for V2 resource API routes (ID-based endpoints)."""
import pytest
from httpx import AsyncClient
from basic_memory.models import Project
from basic_memory.schemas.v2.resource import ResourceResponse
@pytest.mark.asyncio
async def test_create_resource(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
):
"""Test creating a new resource via v2 POST endpoint."""
create_data = {
"file_path": "test-resources/test-file.md",
"content": "# Test Resource\n\nThis is test content.",
}
response = await client.post(
f"{v2_project_url}/resource",
json=create_data,
)
assert response.status_code == 200
result = ResourceResponse.model_validate(response.json())
# V2 must return entity_id
assert result.entity_id is not None
assert isinstance(result.entity_id, int)
assert result.file_path == "test-resources/test-file.md"
assert result.checksum is not None
@pytest.mark.asyncio
async def test_create_resource_duplicate_fails(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
):
"""Test that creating a resource at an existing path returns 409."""
create_data = {
"file_path": "duplicate-test.md",
"content": "First version",
}
# Create first time - should succeed
response = await client.post(f"{v2_project_url}/resource", json=create_data)
assert response.status_code == 200
# Try to create again - should fail with 409
response = await client.post(f"{v2_project_url}/resource", json=create_data)
assert response.status_code == 409
assert "already exists" in response.json()["detail"]
@pytest.mark.asyncio
async def test_get_resource_by_id(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
):
"""Test getting resource content by entity ID."""
# First create a resource
test_content = "# Test Resource\n\nThis is test content."
create_data = {
"file_path": "test-get.md",
"content": test_content,
}
create_response = await client.post(f"{v2_project_url}/resource", json=create_data)
assert create_response.status_code == 200
created = ResourceResponse.model_validate(create_response.json())
# Now get it by entity ID
response = await client.get(f"{v2_project_url}/resource/{created.entity_id}")
assert response.status_code == 200
# Normalize line endings for cross-platform compatibility
assert test_content.replace("\n", "") in response.text.replace("\r\n", "").replace("\n", "")
@pytest.mark.asyncio
async def test_get_resource_not_found(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
):
"""Test getting a non-existent resource returns 404."""
response = await client.get(f"{v2_project_url}/resource/999999")
assert response.status_code == 404
@pytest.mark.asyncio
async def test_update_resource(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
):
"""Test updating resource content by entity ID."""
# Create a resource
create_data = {
"file_path": "test-update.md",
"content": "Original content",
}
create_response = await client.post(f"{v2_project_url}/resource", json=create_data)
assert create_response.status_code == 200
created = ResourceResponse.model_validate(create_response.json())
# Update it
update_data = {
"content": "Updated content",
}
response = await client.put(
f"{v2_project_url}/resource/{created.entity_id}",
json=update_data,
)
assert response.status_code == 200
result = ResourceResponse.model_validate(response.json())
assert result.entity_id == created.entity_id
assert result.file_path == "test-update.md"
# Verify content was updated
get_response = await client.get(f"{v2_project_url}/resource/{created.entity_id}")
assert "Updated content" in get_response.text
@pytest.mark.asyncio
async def test_update_resource_and_move(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
):
"""Test updating resource content and moving it to a new path."""
# Create a resource
create_data = {
"file_path": "original-location.md",
"content": "Original content",
}
create_response = await client.post(f"{v2_project_url}/resource", json=create_data)
assert create_response.status_code == 200
created = ResourceResponse.model_validate(create_response.json())
# Update content and move file
update_data = {
"content": "Updated content in new location",
"file_path": "moved/new-location.md",
}
response = await client.put(
f"{v2_project_url}/resource/{created.entity_id}",
json=update_data,
)
assert response.status_code == 200
result = ResourceResponse.model_validate(response.json())
assert result.entity_id == created.entity_id
assert result.file_path == "moved/new-location.md"
# Verify content at new location
get_response = await client.get(f"{v2_project_url}/resource/{created.entity_id}")
assert "Updated content in new location" in get_response.text
@pytest.mark.asyncio
async def test_update_resource_not_found(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
):
"""Test updating a non-existent resource returns 404."""
update_data = {
"content": "New content",
}
response = await client.put(
f"{v2_project_url}/resource/999999",
json=update_data,
)
assert response.status_code == 404
@pytest.mark.asyncio
async def test_create_resource_invalid_path(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
):
"""Test creating a resource with path traversal attempt fails."""
create_data = {
"file_path": "../../../etc/passwd",
"content": "malicious content",
}
response = await client.post(f"{v2_project_url}/resource", json=create_data)
assert response.status_code == 400
assert "Invalid file path" in response.json()["detail"]
@pytest.mark.asyncio
async def test_update_resource_invalid_path(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
):
"""Test updating a resource with path traversal attempt fails."""
# Create a valid resource first
create_data = {
"file_path": "valid.md",
"content": "Valid content",
}
create_response = await client.post(f"{v2_project_url}/resource", json=create_data)
assert create_response.status_code == 200
created = ResourceResponse.model_validate(create_response.json())
# Try to move it to an invalid path
update_data = {
"content": "Updated content",
"file_path": "../../../etc/passwd",
}
response = await client.put(
f"{v2_project_url}/resource/{created.entity_id}",
json=update_data,
)
assert response.status_code == 400
assert "Invalid file path" in response.json()["detail"]
@pytest.mark.asyncio
async def test_resource_invalid_project_id(
client: AsyncClient,
):
"""Test resource endpoints with invalid project ID return 404."""
# Test create
response = await client.post(
"/v2/projects/999999/resource",
json={"file_path": "test.md", "content": "test"},
)
assert response.status_code == 404
# Test get
response = await client.get("/v2/projects/999999/resource/1")
assert response.status_code == 404
# Test update
response = await client.put(
"/v2/projects/999999/resource/1",
json={"content": "test"},
)
assert response.status_code == 404
@pytest.mark.asyncio
async def test_v2_resource_endpoints_use_project_id_not_name(
client: AsyncClient, test_project: Project
):
"""Verify v2 resource endpoints require project ID, not name."""
# Try using project name instead of ID - should fail
response = await client.get(f"/v2/projects/{test_project.name}/resource/1")
# Should get validation error or 404 because name is not a valid integer
assert response.status_code in [404, 422]
+289
View File
@@ -0,0 +1,289 @@
"""Tests for v2 search router endpoints."""
import pytest
from httpx import AsyncClient
from pathlib import Path
from basic_memory.models import Project
async def create_test_entity(
test_project, entity_data, entity_repository, search_service, file_service
):
"""Helper to create an entity with file and index it."""
# Create file
test_content = f"# {entity_data['title']}\n\nTest content"
file_path = Path(test_project.path) / entity_data["file_path"]
file_path.parent.mkdir(parents=True, exist_ok=True)
await file_service.write_file(file_path, test_content)
# Create entity
entity = await entity_repository.create(entity_data)
# Index for search
await search_service.index_entity(entity)
return entity
@pytest.mark.asyncio
async def test_search_entities(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
entity_repository,
search_service,
file_service,
):
"""Test searching for entities."""
# Create a test entity
entity_data = {
"title": "Searchable Entity",
"entity_type": "note",
"content_type": "text/markdown",
"file_path": "searchable.md",
"checksum": "search123",
}
await create_test_entity(
test_project, entity_data, entity_repository, search_service, file_service
)
# Search for the entity
response = await client.post(f"{v2_project_url}/search/", json={"search_text": "Searchable"})
assert response.status_code == 200
data = response.json()
# Verify response structure
assert "results" in data
assert "current_page" in data
assert "page_size" in data
@pytest.mark.asyncio
async def test_search_with_pagination(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
entity_repository,
search_service,
file_service,
):
"""Test search with pagination parameters."""
# Create multiple test entities
for i in range(5):
entity_data = {
"title": f"Search Entity {i}",
"entity_type": "note",
"content_type": "text/markdown",
"file_path": f"search_{i}.md",
"checksum": f"searchsum{i}",
}
await create_test_entity(
test_project, entity_data, entity_repository, search_service, file_service
)
# Search with pagination
response = await client.post(
f"{v2_project_url}/search/",
json={"search_text": "Search Entity"},
params={"page": 1, "page_size": 3},
)
assert response.status_code == 200
data = response.json()
assert data["current_page"] == 1
assert data["page_size"] == 3
@pytest.mark.asyncio
async def test_search_by_permalink(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
entity_repository,
search_service,
file_service,
):
"""Test searching by permalink."""
# Create a test entity with permalink
entity_data = {
"title": "Permalink Search",
"entity_type": "note",
"content_type": "text/markdown",
"file_path": "permalink_search.md",
"checksum": "perm123",
"permalink": "permalink-search",
}
await create_test_entity(
test_project, entity_data, entity_repository, search_service, file_service
)
# Search by permalink
response = await client.post(
f"{v2_project_url}/search/", json={"permalink": "permalink-search"}
)
assert response.status_code == 200
data = response.json()
assert "results" in data
@pytest.mark.asyncio
async def test_search_by_title(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
entity_repository,
search_service,
file_service,
):
"""Test searching by title."""
# Create a test entity
entity_data = {
"title": "Unique Title For Search",
"entity_type": "note",
"content_type": "text/markdown",
"file_path": "unique_title.md",
"checksum": "title123",
}
await create_test_entity(
test_project, entity_data, entity_repository, search_service, file_service
)
# Search by title
response = await client.post(f"{v2_project_url}/search/", json={"title": "Unique Title"})
assert response.status_code == 200
data = response.json()
assert "results" in data
@pytest.mark.asyncio
async def test_search_with_type_filter(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
entity_repository,
search_service,
file_service,
):
"""Test searching with entity type filter."""
# Create test entities of different types
for entity_type in ["note", "document"]:
entity_data = {
"title": f"Type {entity_type}",
"entity_type": entity_type,
"content_type": "text/markdown",
"file_path": f"type_{entity_type}.md",
"checksum": f"type{entity_type}",
}
await create_test_entity(
test_project, entity_data, entity_repository, search_service, file_service
)
# Search with type filter
response = await client.post(
f"{v2_project_url}/search/", json={"search_text": "Type", "types": ["note"]}
)
assert response.status_code == 200
data = response.json()
assert "results" in data
@pytest.mark.asyncio
async def test_search_with_date_filter(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
entity_repository,
search_service,
file_service,
):
"""Test searching with date filter."""
# Create a test entity
entity_data = {
"title": "Date Filtered",
"entity_type": "note",
"content_type": "text/markdown",
"file_path": "date_filtered.md",
"checksum": "date123",
}
await create_test_entity(
test_project, entity_data, entity_repository, search_service, file_service
)
# Search with date filter
response = await client.post(
f"{v2_project_url}/search/",
json={"search_text": "Date Filtered", "after_date": "2024-01-01T00:00:00Z"},
)
assert response.status_code == 200
data = response.json()
assert "results" in data
@pytest.mark.asyncio
async def test_search_empty_query(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
):
"""Test search with empty query."""
response = await client.post(f"{v2_project_url}/search/", json={})
# Empty query should still be valid (returns all)
assert response.status_code in [200, 422]
@pytest.mark.asyncio
async def test_search_invalid_project_id(
client: AsyncClient,
):
"""Test searching with invalid project ID returns 404."""
response = await client.post("/v2/projects/999999/search/", json={"search_text": "test"})
assert response.status_code == 404
@pytest.mark.asyncio
async def test_reindex(
client: AsyncClient,
test_project: Project,
v2_project_url: str,
):
"""Test reindexing search index."""
response = await client.post(f"{v2_project_url}/search/reindex")
assert response.status_code == 200
data = response.json()
# Verify response structure
assert "status" in data
assert data["status"] == "ok"
assert "message" in data
@pytest.mark.asyncio
async def test_reindex_invalid_project_id(
client: AsyncClient,
):
"""Test reindexing with invalid project ID returns 404."""
response = await client.post("/v2/projects/999999/search/reindex")
assert response.status_code == 404
@pytest.mark.asyncio
async def test_v2_search_endpoints_use_project_id_not_name(
client: AsyncClient,
test_project: Project,
):
"""Test that v2 search endpoints reject string project names."""
# Try to use project name instead of ID - should fail
response = await client.post(f"/v2/{test_project.name}/search/", json={"search_text": "test"})
# FastAPI path validation should reject non-integer project_id
assert response.status_code in [404, 422]
+28 -1
View File
@@ -1,5 +1,8 @@
import os
from pathlib import Path
from typing import AsyncGenerator
import pytest
import pytest_asyncio
from fastapi import FastAPI
from httpx import AsyncClient, ASGITransport
@@ -8,7 +11,31 @@ from basic_memory.api.app import app as fastapi_app
from basic_memory.deps import get_project_config, get_engine_factory, get_app_config
@pytest_asyncio.fixture(autouse=True)
@pytest.fixture(autouse=True)
def isolated_home(tmp_path, monkeypatch) -> Path:
"""Isolate tests from user's HOME directory.
This prevents tests from reading/writing to ~/.basic-memory/.bmignore
or other user-specific configuration.
Sets BASIC_MEMORY_HOME to tmp_path directly so the default project
writes files to tmp_path, which is where tests expect to find them.
"""
# Clear config cache to ensure fresh config for each test
from basic_memory import config as config_module
config_module._CONFIG_CACHE = None
monkeypatch.setenv("HOME", str(tmp_path))
if os.name == "nt":
monkeypatch.setenv("USERPROFILE", str(tmp_path))
# Set to tmp_path directly (not tmp_path/basic-memory) so default project
# home is tmp_path - tests expect to find imported files there
monkeypatch.setenv("BASIC_MEMORY_HOME", str(tmp_path))
return tmp_path
@pytest_asyncio.fixture
async def app(app_config, project_config, engine_factory, test_config, aiolib) -> FastAPI:
"""Create test FastAPI application."""
app = fastapi_app
@@ -55,6 +55,7 @@ def mock_api_client():
"default": False,
"old_project": None,
"new_project": {
"id": 1,
"name": "test-project",
"path": "/test-project",
"is_default": False,
+113 -93
View File
@@ -1,16 +1,23 @@
"""Common test fixtures."""
import os
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from textwrap import dedent
from typing import AsyncGenerator, Literal
from typing import AsyncGenerator
import os
import pytest
import pytest_asyncio
from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncEngine, AsyncSession, async_sessionmaker
from sqlalchemy.ext.asyncio import (
AsyncEngine,
AsyncSession,
async_sessionmaker,
create_async_engine,
)
from sqlalchemy.pool import NullPool
from testcontainers.postgres import PostgresContainer
from basic_memory import db
from basic_memory.config import ProjectConfig, BasicMemoryConfig, ConfigManager, DatabaseBackend
@@ -37,32 +44,47 @@ from basic_memory.sync.sync_service import SyncService
from basic_memory.sync.watch_service import WatchService
# =============================================================================
# Database Backend Selection (env var approach)
# =============================================================================
# By default, tests run against SQLite.
# Set BASIC_MEMORY_TEST_POSTGRES=1 to run against Postgres (uses testcontainers).
# This allows running sqlite/postgres tests in parallel in CI.
@pytest.fixture(scope="session")
def db_backend():
"""Determine database backend from environment variable.
Default: sqlite
Set BASIC_MEMORY_TEST_POSTGRES=1 to use postgres
"""
if os.environ.get("BASIC_MEMORY_TEST_POSTGRES", "").lower() in ("1", "true", "yes"):
return "postgres"
return "sqlite"
@pytest.fixture(scope="session")
def postgres_container(db_backend):
"""Session-scoped Postgres container for tests.
Uses testcontainers to spin up a real Postgres instance in Docker.
The container is started once per test session and shared across all tests.
Only starts if db_backend is "postgres".
"""
if db_backend != "postgres":
yield None
return
with PostgresContainer("postgres:16-alpine") as postgres:
yield postgres
@pytest.fixture
def anyio_backend():
return "asyncio"
@pytest.fixture(
params=[
pytest.param("sqlite", id="sqlite"),
pytest.param("postgres", id="postgres", marks=pytest.mark.postgres),
]
)
def db_backend(request) -> Literal["sqlite", "postgres"]:
"""Parametrize tests to run against both SQLite and Postgres.
Usage:
pytest # Runs tests against SQLite only (default)
pytest -m postgres # Runs tests against Postgres only
pytest -m "not postgres" # Runs tests against SQLite only
pytest --run-all-backends # Runs tests against both backends
Note: Only tests that use database fixtures (engine_factory, session_maker, etc.)
will be parametrized. Tests that don't use the database won't be affected.
"""
return request.param
@pytest.fixture
def project_root() -> Path:
return Path(__file__).parent.parent
@@ -81,24 +103,18 @@ def config_home(tmp_path, monkeypatch) -> Path:
@pytest.fixture(scope="function")
def app_config(
config_home, db_backend: Literal["sqlite", "postgres"], monkeypatch
) -> BasicMemoryConfig:
"""Create test app configuration."""
# Create a basic config without depending on test_project to avoid circular dependency
def app_config(config_home, db_backend, postgres_container, monkeypatch) -> BasicMemoryConfig:
"""Create test app configuration for the appropriate backend."""
projects = {"test-project": str(config_home)}
# Configure database backend based on test parameter
# Set backend based on parameterized db_backend fixture
if db_backend == "postgres":
database_backend = DatabaseBackend.POSTGRES
# Use env var if set, otherwise use default matching docker-compose-postgres.yml
# These are local test credentials only - NOT for production
database_url = os.getenv(
"POSTGRES_TEST_URL",
"postgresql+asyncpg://basic_memory_user:dev_password@localhost:5433/basic_memory_test",
)
backend = DatabaseBackend.POSTGRES
# Get URL from testcontainer and convert to asyncpg driver
sync_url = postgres_container.get_connection_url()
database_url = sync_url.replace("postgresql+psycopg2", "postgresql+asyncpg")
else:
database_backend = DatabaseBackend.SQLITE
backend = DatabaseBackend.SQLITE
database_url = None
app_config = BasicMemoryConfig(
@@ -106,7 +122,7 @@ def app_config(
projects=projects,
default_project="test-project",
update_permalinks_on_move=True,
database_backend=database_backend,
database_backend=backend,
database_url=database_url,
)
@@ -162,76 +178,66 @@ def test_config(config_home, project_config, app_config, config_manager) -> Test
async def engine_factory(
app_config,
config_manager,
db_backend: Literal["sqlite", "postgres"],
db_backend,
postgres_container,
) -> AsyncGenerator[tuple[AsyncEngine, async_sessionmaker[AsyncSession]], None]:
"""Create engine and session factory for the configured database backend."""
"""Engine factory for SQLite or Postgres tests.
Uses parameterized db_backend fixture to run tests against both backends.
"""
from basic_memory.models.search import CREATE_SEARCH_INDEX
if db_backend == "postgres":
# Postgres: Create fresh engine for each test with full schema reset
config_manager._config = app_config
db_type = DatabaseType.FILESYSTEM
# Postgres mode using testcontainers
# Get async connection URL (asyncpg driver - same as production)
sync_url = postgres_container.get_connection_url()
async_url = sync_url.replace("postgresql+psycopg2", "postgresql+asyncpg")
# Use context manager to handle engine disposal properly
async with db.engine_session_factory(db_path=app_config.database_path, db_type=db_type) as (
engine,
session_maker,
):
# Drop and recreate schema for complete isolation
async with engine.begin() as conn:
await conn.execute(text("DROP SCHEMA IF EXISTS public CASCADE"))
await conn.execute(text("CREATE SCHEMA public"))
await conn.execute(text("GRANT ALL ON SCHEMA public TO basic_memory_user"))
await conn.execute(text("GRANT ALL ON SCHEMA public TO public"))
engine = create_async_engine(
async_url,
echo=False,
poolclass=NullPool, # NullPool for better test isolation
)
# Run migrations to create production tables (including search_index with correct schema)
# Alembic handles duplicate migration checks, so it's safe to call this for each test
from basic_memory.db import run_migrations
session_maker = async_sessionmaker(
bind=engine,
class_=AsyncSession,
expire_on_commit=False,
autoflush=False,
)
await run_migrations(app_config, db_type)
from basic_memory.models.search import (
CREATE_POSTGRES_SEARCH_INDEX_TABLE,
CREATE_POSTGRES_SEARCH_INDEX_FTS,
CREATE_POSTGRES_SEARCH_INDEX_METADATA,
)
# For Postgres, migrations create all production tables with correct schemas
# We only need to create test-specific tables (like ModelTest) that aren't in migrations
# Don't create search_index via ORM - it's already created by migration with composite PK
async with engine.begin() as conn:
# List of tables created by migrations - don't recreate them via ORM
production_tables = {
"entity",
"observation",
"relation",
"project",
"search_index",
"alembic_version",
}
# Drop and recreate all tables for test isolation
async with engine.begin() as conn:
# Must drop search_index first (has FK to project, blocks drop_all)
await conn.execute(text("DROP TABLE IF EXISTS search_index CASCADE"))
await conn.run_sync(Base.metadata.drop_all)
await conn.run_sync(Base.metadata.create_all)
# Create search_index via DDL (not ORM - uses composite PK + tsvector)
# asyncpg requires separate execute calls for each statement
await conn.execute(CREATE_POSTGRES_SEARCH_INDEX_TABLE)
await conn.execute(CREATE_POSTGRES_SEARCH_INDEX_FTS)
await conn.execute(CREATE_POSTGRES_SEARCH_INDEX_METADATA)
# Get test-specific tables that aren't created by migrations
test_tables = [
table
for table in Base.metadata.sorted_tables
if table.name not in production_tables
]
if test_tables:
await conn.run_sync(
lambda sync_conn: Base.metadata.create_all(sync_conn, tables=test_tables)
)
yield engine, session_maker
yield engine, session_maker
await engine.dispose()
else:
# SQLite: Create fresh in-memory database for each test
# SQLite mode
db_type = DatabaseType.MEMORY
async with db.engine_session_factory(db_path=app_config.database_path, db_type=db_type) as (
engine,
session_maker,
):
# Create all tables via ORM
# Create all tables via ORM, then add search_index via FTS5 DDL
async with engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
# Drop any SearchIndex ORM table, then create FTS5 virtual table
async with db.scoped_session(session_maker) as session:
await session.execute(text("DROP TABLE IF EXISTS search_index"))
await session.execute(CREATE_SEARCH_INDEX)
await session.commit()
await conn.execute(CREATE_SEARCH_INDEX)
# Yield after setup is complete
yield engine, session_maker
@@ -543,8 +549,22 @@ async def test_graph(
@pytest.fixture
def watch_service(app_config: BasicMemoryConfig, project_repository) -> WatchService:
return WatchService(app_config=app_config, project_repository=project_repository)
def watch_service(app_config: BasicMemoryConfig, project_repository, sync_service) -> WatchService:
"""Create WatchService with injected sync_service factory.
The sync_service_factory allows tests to use the fixture-provided sync_service
instead of the production get_sync_service() which creates its own db connection.
"""
async def sync_service_factory(project):
"""Return the test fixture's sync_service regardless of project."""
return sync_service
return WatchService(
app_config=app_config,
project_repository=project_repository,
sync_service_factory=sync_service_factory,
)
@pytest.fixture
+3 -3
View File
@@ -85,11 +85,11 @@ async def test_parse_complete_file(project_config, entity_parser, valid_entity_c
), "missing [[Auth API Spec]]"
# inline links in content
assert Relation(type="links to", target="Random Link", context=None) in entity.relations, (
assert Relation(type="links_to", target="Random Link", context=None) in entity.relations, (
"missing [[Random Link]]"
)
assert (
Relation(type="links to", target="Random Link with Title|Titled Link", context=None)
Relation(type="links_to", target="Random Link with Title|Titled Link", context=None)
in entity.relations
), "missing [[Random Link with Title|Titled Link]]"
@@ -179,7 +179,7 @@ async def test_parse_file_without_section_headers(project_config, entity_parser)
assert entity.observations[0].tags == ["test"]
assert len(entity.relations) == 2
assert entity.relations[0].type == "links to"
assert entity.relations[0].type == "links_to"
assert entity.relations[0].target == "Random Link"
assert entity.relations[1].type == "references"
+40 -2
View File
@@ -121,6 +121,44 @@ def test_observation_excludes_markdown_and_wiki_links():
assert not is_observation(token), "No space after category should not be valid observation"
def test_observation_excludes_html_color_codes():
"""Test that HTML color codes are NOT interpreted as hashtags.
This test validates the fix for issue #446 where:
- HTML color codes like #4285F4 in attributes were incorrectly
causing lines to be parsed as observations.
"""
# HTML color code in font tag should NOT be an observation
token = Token("inline", '**<font color="#4285F4">Jane:</font>** Welcome to the show', 0)
assert not is_observation(token), "HTML color codes should not trigger hashtag detection"
# Color code in style attribute
token = Token("inline", '<span style="color:#FF5733">Styled text</span>', 0)
assert not is_observation(token), "Color codes in style should not be observations"
# Multiple color codes
token = Token(
"inline", '<font color="#4285F4">Blue</font> and <font color="#EA4335">Red</font>', 0
)
assert not is_observation(token), "Multiple color codes should not be observations"
# Hex color without quotes (edge case)
token = Token("inline", "background-color:#FFFFFF is white", 0)
assert not is_observation(token), "Inline hex colors should not be observations"
# But standalone hashtags SHOULD still work
token = Token("inline", "This has a #realtag in it", 0)
assert is_observation(token), "Standalone hashtags should still work"
# Multiple real hashtags
token = Token("inline", "Tags: #design #feature #important", 0)
assert is_observation(token), "Multiple standalone hashtags should work"
# Mix of color code and real tag - should be observation because of real tag
token = Token("inline", '<font color="#4285F4">Text</font> #actualtag', 0)
assert is_observation(token), "Real hashtag with color code should still be observation"
def test_relation_plugin():
"""Test relation plugin."""
md = MarkdownIt().use(relation_plugin)
@@ -143,7 +181,7 @@ def test_relation_plugin():
token = [t for t in md.parse(content) if t.type == "inline"][0]
rels = token.meta["relations"]
assert len(rels) == 2
assert rels[0]["type"] == "links to"
assert rels[0]["type"] == "links_to"
assert rels[0]["target"] == "Link"
assert rels[1]["target"] == "Another Link"
@@ -208,4 +246,4 @@ def test_combined_plugins():
text_token = inline_tokens[4]
assert "relations" in text_token.meta
link = text_token.meta["relations"][0]
assert link["type"] == "links to"
assert link["type"] == "links_to"
+1 -1
View File
@@ -88,7 +88,7 @@ async def test_missing_sections(tmp_path):
entity = await parser.parse_file(test_file)
assert len(entity.relations) == 1
assert entity.relations[0].target == "links"
assert entity.relations[0].type == "links to"
assert entity.relations[0].type == "links_to"
@pytest.mark.asyncio
+2 -1
View File
@@ -50,6 +50,7 @@ def test_entity_data():
}
@pytest_asyncio.fixture(autouse=True)
@pytest_asyncio.fixture
async def init_search_index(search_service: SearchService):
"""Initialize search index. Request this fixture explicitly in tests that need it."""
await search_service.init_search_index()
+1
View File
@@ -116,6 +116,7 @@ def test_prompt_context_with_file_path_no_permalink():
# Create a mock context with a file that has no permalink (like a binary file)
test_entity = EntitySummary(
entity_id=1,
type="entity",
title="Test File",
permalink=None, # No permalink
+61 -24
View File
@@ -1,7 +1,9 @@
"""Tests for the move_note MCP tool."""
import pytest
from unittest.mock import patch
from pathlib import Path
from unittest.mock import patch, MagicMock
from contextlib import asynccontextmanager
from basic_memory.mcp.tools.move_note import move_note, _format_move_error_response
from basic_memory.mcp.tools.write_note import write_note
@@ -887,36 +889,71 @@ class TestMoveNoteSecurityValidation:
class TestMoveNoteErrorHandling:
"""Test move note exception handling."""
"""Test move note exception handling.
These are pure unit tests that mock get_client and other dependencies.
They don't need the database or ASGI app.
"""
@pytest.fixture
def mock_client(self):
"""Create a mock async client context manager."""
mock = MagicMock()
@asynccontextmanager
async def mock_get_client():
yield mock
return mock_get_client, mock
@pytest.mark.asyncio
async def test_move_note_exception_handling(self):
async def test_move_note_exception_handling(self, mock_client):
"""Test exception handling in move_note."""
with patch("basic_memory.mcp.tools.move_note.get_active_project") as mock_get_project:
mock_get_project.return_value.project_url = "http://test"
mock_get_project.return_value.name = "test-project"
mock_get_client, _ = mock_client
with patch(
"basic_memory.mcp.tools.move_note.call_post",
side_effect=Exception("entity not found"),
):
result = await move_note.fn("test-note", "target/file.md", project="test-project")
with patch("basic_memory.mcp.tools.move_note.get_client", mock_get_client):
with patch("basic_memory.mcp.tools.move_note.get_active_project") as mock_get_project:
mock_get_project.return_value.project_url = "http://test"
mock_get_project.return_value.name = "test-project"
mock_get_project.return_value.home = Path("/tmp/test")
assert isinstance(result, str)
assert "# Move Failed - Note Not Found" in result
with patch(
"basic_memory.mcp.tools.move_note.call_post",
side_effect=Exception("entity not found"),
):
with patch(
"basic_memory.mcp.tools.move_note.call_get",
side_effect=Exception("not found"),
):
result = await move_note.fn(
"test-note", "target/file.md", project="test-project"
)
assert isinstance(result, str)
assert "# Move Failed - Note Not Found" in result
@pytest.mark.asyncio
async def test_move_note_permission_error_handling(self):
async def test_move_note_permission_error_handling(self, mock_client):
"""Test permission error handling in move_note."""
with patch("basic_memory.mcp.tools.move_note.get_active_project") as mock_get_project:
mock_get_project.return_value.project_url = "http://test"
mock_get_project.return_value.name = "test-project"
mock_get_client, _ = mock_client
with patch(
"basic_memory.mcp.tools.move_note.call_post",
side_effect=Exception("permission denied"),
):
result = await move_note.fn("test-note", "target/file.md", project="test-project")
with patch("basic_memory.mcp.tools.move_note.get_client", mock_get_client):
with patch("basic_memory.mcp.tools.move_note.get_active_project") as mock_get_project:
mock_get_project.return_value.project_url = "http://test"
mock_get_project.return_value.name = "test-project"
mock_get_project.return_value.home = Path("/tmp/test")
assert isinstance(result, str)
assert "# Move Failed - Permission Error" in result
with patch(
"basic_memory.mcp.tools.move_note.call_post",
side_effect=Exception("permission denied"),
):
with patch(
"basic_memory.mcp.tools.move_note.call_get",
side_effect=Exception("not found"),
):
result = await move_note.fn(
"test-note", "target/file.md", project="test-project"
)
assert isinstance(result, str)
assert "# Move Failed - Permission Error" in result
+242
View File
@@ -18,10 +18,12 @@ async def entity_with_observations(session_maker, sample_entity):
async with db.scoped_session(session_maker) as session:
observations = [
Observation(
project_id=sample_entity.project_id,
entity_id=sample_entity.id,
content="First observation",
),
Observation(
project_id=sample_entity.project_id,
entity_id=sample_entity.id,
content="Second observation",
),
@@ -59,6 +61,7 @@ async def related_results(session_maker, test_project: Project):
await session.flush()
relation = Relation(
project_id=test_project.id,
from_id=source.id,
to_id=target.id,
to_name=target.title,
@@ -176,6 +179,55 @@ async def test_update_entity(entity_repository: EntityRepository, sample_entity:
assert db_entity.title == "Updated title"
@pytest.mark.asyncio
async def test_update_entity_returns_with_relations_and_observations(
entity_repository: EntityRepository, entity_with_observations, test_project: Project
):
"""Test that update() returns entity with observations and relations eagerly loaded."""
entity = entity_with_observations
# Create a target entity and relation
async with db.scoped_session(entity_repository.session_maker) as session:
target = Entity(
project_id=test_project.id,
title="target",
entity_type="test",
permalink="target/target",
file_path="target/target.md",
content_type="text/markdown",
created_at=datetime.now(timezone.utc),
updated_at=datetime.now(timezone.utc),
)
session.add(target)
await session.flush()
relation = Relation(
project_id=test_project.id,
from_id=entity.id,
to_id=target.id,
to_name=target.title,
relation_type="connects_to",
)
session.add(relation)
# Now update the entity
updated = await entity_repository.update(entity.id, {"title": "Updated with relations"})
# Verify returned entity has observations and relations accessible
# (would raise DetachedInstanceError if not eagerly loaded)
assert updated is not None
assert updated.title == "Updated with relations"
# Access observations - should NOT raise DetachedInstanceError
assert len(updated.observations) == 2
assert updated.observations[0].content in ["First observation", "Second observation"]
# Access relations - should NOT raise DetachedInstanceError
assert len(updated.relations) == 1
assert updated.relations[0].relation_type == "connects_to"
assert updated.relations[0].to_name == "target"
@pytest.mark.asyncio
async def test_delete_entity(entity_repository: EntityRepository, sample_entity):
"""Test deleting an entity."""
@@ -737,6 +789,7 @@ async def test_get_all_file_paths_performance(entity_repository: EntityRepositor
# Add observations to entity1
observation = Observation(
project_id=entity_repository.project_id,
entity_id=entity1.id,
content="Test observation",
category="note",
@@ -745,6 +798,7 @@ async def test_get_all_file_paths_performance(entity_repository: EntityRepositor
# Add relation between entities
relation = Relation(
project_id=entity_repository.project_id,
from_id=entity1.id,
to_id=entity2.id,
to_name=entity2.title,
@@ -810,3 +864,191 @@ async def test_get_all_file_paths_project_isolation(
# Should only include files from project 1
assert len(file_paths) == 1
assert file_paths == ["test/file1.md"]
# -------------------------------------------------------------------------
# Tests for lightweight permalink resolution methods
# -------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_permalink_exists(entity_repository: EntityRepository, sample_entity: Entity):
"""Test checking if a permalink exists without loading full entity."""
# Existing permalink should return True
assert await entity_repository.permalink_exists(sample_entity.permalink) is True # pyright: ignore [reportArgumentType]
# Non-existent permalink should return False
assert await entity_repository.permalink_exists("nonexistent/permalink") is False
@pytest.mark.asyncio
async def test_permalink_exists_project_isolation(
entity_repository: EntityRepository, session_maker
):
"""Test that permalink_exists respects project isolation."""
async with db.scoped_session(session_maker) as session:
# Create entity in repository's project
entity1 = Entity(
project_id=entity_repository.project_id,
title="Project 1 Entity",
entity_type="test",
permalink="test/entity1",
file_path="test/entity1.md",
content_type="text/markdown",
created_at=datetime.now(timezone.utc),
updated_at=datetime.now(timezone.utc),
)
session.add(entity1)
# Create a second project with same permalink
project2 = Project(name="other-project", path="/tmp/other")
session.add(project2)
await session.flush()
entity2 = Entity(
project_id=project2.id,
title="Project 2 Entity",
entity_type="test",
permalink="test/entity2",
file_path="test/entity2.md",
content_type="text/markdown",
created_at=datetime.now(timezone.utc),
updated_at=datetime.now(timezone.utc),
)
session.add(entity2)
# Should find entity1's permalink in project 1
assert await entity_repository.permalink_exists("test/entity1") is True
# Should NOT find entity2's permalink (it's in project 2)
assert await entity_repository.permalink_exists("test/entity2") is False
@pytest.mark.asyncio
async def test_get_file_path_for_permalink(
entity_repository: EntityRepository, sample_entity: Entity
):
"""Test getting file_path for a permalink without loading full entity."""
# Existing permalink should return file_path
file_path = await entity_repository.get_file_path_for_permalink(sample_entity.permalink) # pyright: ignore [reportArgumentType]
assert file_path == sample_entity.file_path
# Non-existent permalink should return None
result = await entity_repository.get_file_path_for_permalink("nonexistent/permalink")
assert result is None
@pytest.mark.asyncio
async def test_get_permalink_for_file_path(
entity_repository: EntityRepository, sample_entity: Entity
):
"""Test getting permalink for a file_path without loading full entity."""
# Existing file_path should return permalink
permalink = await entity_repository.get_permalink_for_file_path(sample_entity.file_path)
assert permalink == sample_entity.permalink
# Non-existent file_path should return None
result = await entity_repository.get_permalink_for_file_path("nonexistent/path.md")
assert result is None
@pytest.mark.asyncio
async def test_get_all_permalinks(entity_repository: EntityRepository, session_maker):
"""Test getting all permalinks without loading full entities."""
async with db.scoped_session(session_maker) as session:
entity1 = Entity(
project_id=entity_repository.project_id,
title="Entity 1",
entity_type="test",
permalink="test/entity1",
file_path="test/entity1.md",
content_type="text/markdown",
created_at=datetime.now(timezone.utc),
updated_at=datetime.now(timezone.utc),
)
entity2 = Entity(
project_id=entity_repository.project_id,
title="Entity 2",
entity_type="test",
permalink="test/entity2",
file_path="test/entity2.md",
content_type="text/markdown",
created_at=datetime.now(timezone.utc),
updated_at=datetime.now(timezone.utc),
)
session.add_all([entity1, entity2])
permalinks = await entity_repository.get_all_permalinks()
assert len(permalinks) == 2
assert set(permalinks) == {"test/entity1", "test/entity2"}
# Results should be strings, not entities
for permalink in permalinks:
assert isinstance(permalink, str)
@pytest.mark.asyncio
async def test_get_permalink_to_file_path_map(entity_repository: EntityRepository, session_maker):
"""Test getting permalink -> file_path mapping for bulk operations."""
async with db.scoped_session(session_maker) as session:
entity1 = Entity(
project_id=entity_repository.project_id,
title="Entity 1",
entity_type="test",
permalink="test/entity1",
file_path="test/entity1.md",
content_type="text/markdown",
created_at=datetime.now(timezone.utc),
updated_at=datetime.now(timezone.utc),
)
entity2 = Entity(
project_id=entity_repository.project_id,
title="Entity 2",
entity_type="test",
permalink="test/entity2",
file_path="test/entity2.md",
content_type="text/markdown",
created_at=datetime.now(timezone.utc),
updated_at=datetime.now(timezone.utc),
)
session.add_all([entity1, entity2])
mapping = await entity_repository.get_permalink_to_file_path_map()
assert len(mapping) == 2
assert mapping["test/entity1"] == "test/entity1.md"
assert mapping["test/entity2"] == "test/entity2.md"
@pytest.mark.asyncio
async def test_get_file_path_to_permalink_map(entity_repository: EntityRepository, session_maker):
"""Test getting file_path -> permalink mapping for bulk operations."""
async with db.scoped_session(session_maker) as session:
entity1 = Entity(
project_id=entity_repository.project_id,
title="Entity 1",
entity_type="test",
permalink="test/entity1",
file_path="test/entity1.md",
content_type="text/markdown",
created_at=datetime.now(timezone.utc),
updated_at=datetime.now(timezone.utc),
)
entity2 = Entity(
project_id=entity_repository.project_id,
title="Entity 2",
entity_type="test",
permalink="test/entity2",
file_path="test/entity2.md",
content_type="text/markdown",
created_at=datetime.now(timezone.utc),
updated_at=datetime.now(timezone.utc),
)
session.add_all([entity1, entity2])
mapping = await entity_repository.get_file_path_to_permalink_map()
assert len(mapping) == 2
assert mapping["test/entity1.md"] == "test/entity1"
assert mapping["test/entity2.md"] == "test/entity2"
@@ -28,6 +28,7 @@ async def test_upsert_entity_with_observations_conflict(entity_repository: Entit
# Add observations to the entity
obs1 = Observation(
project_id=entity_repository.project_id,
content="This is a test observation",
category="testing",
tags=["test"],
@@ -56,11 +57,13 @@ async def test_upsert_entity_with_observations_conflict(entity_repository: Entit
# Add different observations
obs2 = Observation(
project_id=entity_repository.project_id,
content="This is an updated observation",
category="updated",
tags=["updated"],
)
obs3 = Observation(
project_id=entity_repository.project_id,
content="This is a second observation",
category="second",
tags=["second"],
@@ -22,6 +22,7 @@ async def repo(observation_repository):
async def sample_observation(repo, sample_entity: Entity):
"""Create a sample observation for testing"""
observation_data = {
"project_id": sample_entity.project_id,
"entity_id": sample_entity.id,
"content": "Test observation",
"context": "test-context",
@@ -35,6 +36,7 @@ async def test_create_observation(
):
"""Test creating a new observation"""
observation_data = {
"project_id": sample_entity.project_id,
"entity_id": sample_entity.id,
"content": "Test content",
"context": "test-context",
@@ -52,6 +54,7 @@ async def test_create_observation_entity_does_not_exist(
):
"""Test creating a new observation"""
observation_data = {
"project_id": sample_entity.project_id,
"entity_id": 99999, # Non-existent entity ID (integer for Postgres compatibility)
"content": "Test content",
"context": "test-context",
@@ -104,10 +107,12 @@ async def test_delete_observations(session_maker: async_sessionmaker, repo, test
# Create test observations
obs1 = Observation(
project_id=test_project.id,
entity_id=entity.id,
content="Test observation 1",
)
obs2 = Observation(
project_id=test_project.id,
entity_id=entity.id,
content="Test observation 2",
)
@@ -144,6 +149,7 @@ async def test_delete_observation_by_id(
# Create test observation
obs = Observation(
project_id=test_project.id,
entity_id=entity.id,
content="Test observation",
)
@@ -180,10 +186,12 @@ async def test_delete_observation_by_content(
# Create test observations
obs1 = Observation(
project_id=test_project.id,
entity_id=entity.id,
content="Delete this observation",
)
obs2 = Observation(
project_id=test_project.id,
entity_id=entity.id,
content="Keep this observation",
)
@@ -220,16 +228,19 @@ async def test_find_by_category(session_maker: async_sessionmaker, repo, test_pr
# Create test observations with different categories
observations = [
Observation(
project_id=test_project.id,
entity_id=entity.id,
content="Tech observation",
category="tech",
),
Observation(
project_id=test_project.id,
entity_id=entity.id,
content="Design observation",
category="design",
),
Observation(
project_id=test_project.id,
entity_id=entity.id,
content="Another tech observation",
category="tech",
@@ -278,21 +289,25 @@ async def test_observation_categories(
# Create observations with various categories
observations = [
Observation(
project_id=test_project.id,
entity_id=entity.id,
content="First tech note",
category="tech",
),
Observation(
project_id=test_project.id,
entity_id=entity.id,
content="Second tech note",
category="tech", # Duplicate category
),
Observation(
project_id=test_project.id,
entity_id=entity.id,
content="Design note",
category="design",
),
Observation(
project_id=test_project.id,
entity_id=entity.id,
content="Feature note",
category="feature",
@@ -341,6 +356,7 @@ async def test_find_by_category_case_sensitivity(
# Create a test observation
obs = Observation(
project_id=test_project.id,
entity_id=entity.id,
content="Tech note",
category="tech", # lowercase in database
@@ -356,3 +372,97 @@ async def test_find_by_category_case_sensitivity(
upper_case = await repo.find_by_category("TECH")
assert len(upper_case) == 0 # Currently case-sensitive
@pytest.mark.asyncio
async def test_observation_permalink_truncates_long_content(
session_maker: async_sessionmaker, repo, test_project: Project
):
"""Test that observation permalinks truncate long content.
This test validates the fix for issue #446 where:
- Long observation content (like transcript dialogue) created permalinks
exceeding PostgreSQL's btree index limit of 2704 bytes.
- Content is now truncated to 200 chars in the permalink property.
"""
async with db.scoped_session(session_maker) as session:
entity = Entity(
project_id=test_project.id,
title="test_entity",
entity_type="test",
permalink="test/test-entity",
file_path="test/test_entity.md",
content_type="text/markdown",
created_at=datetime.now(timezone.utc),
updated_at=datetime.now(timezone.utc),
)
session.add(entity)
await session.flush()
# Create observation with very long content (5000+ chars to simulate transcript)
long_content = "A" * 5000 # Well over the 200 char limit
obs = Observation(
project_id=test_project.id,
entity_id=entity.id,
content=long_content,
category="transcript",
)
session.add(obs)
await session.flush()
# Access the permalink property
permalink = obs.permalink
# The full content would create a permalink like:
# test/test-entity/observations/transcript/AAAA...5000 chars
# With truncation, it should be much shorter
# Content portion should be truncated to 200 chars
# Permalink format: entity_permalink/observations/category/content
assert len(permalink) < 300 # Should be well under 300 chars total
assert len(long_content[:200]) == 200 # Verify truncation length
# Verify the permalink contains expected parts
assert "test/test-entity" in permalink or "test-entity" in permalink
assert "observations" in permalink
assert "transcript" in permalink
# Full 5000-char content should NOT be in permalink
assert long_content not in permalink
@pytest.mark.asyncio
async def test_observation_permalink_short_content_unchanged(
session_maker: async_sessionmaker, repo, test_project: Project
):
"""Test that short observation content is not unnecessarily truncated."""
async with db.scoped_session(session_maker) as session:
entity = Entity(
project_id=test_project.id,
title="test_entity",
entity_type="test",
permalink="test/test-entity",
file_path="test/test_entity.md",
content_type="text/markdown",
created_at=datetime.now(timezone.utc),
updated_at=datetime.now(timezone.utc),
)
session.add(entity)
await session.flush()
# Create observation with short content
short_content = "Short observation content"
obs = Observation(
project_id=test_project.id,
entity_id=entity.id,
content=short_content,
category="note",
)
session.add(obs)
await session.flush()
permalink = obs.permalink
# Short content should be fully included (after permalink normalization)
# The generate_permalink function normalizes the content
assert "short-observation-content" in permalink.lower()
+156 -1
View File
@@ -50,16 +50,18 @@ async def target_entity(session_maker, test_project: Project):
@pytest_asyncio.fixture
async def test_relations(session_maker, source_entity, target_entity):
async def test_relations(session_maker, source_entity, target_entity, test_project: Project):
"""Create test relations."""
relations = [
Relation(
project_id=test_project.id,
from_id=source_entity.id,
to_id=target_entity.id,
to_name=target_entity.title,
relation_type="connects_to",
),
Relation(
project_id=test_project.id,
from_id=source_entity.id,
to_id=target_entity.id,
to_name=target_entity.title,
@@ -349,3 +351,156 @@ async def test_delete_nonexistent_relation(relation_repository):
"""Test deleting a relation that doesn't exist."""
result = await relation_repository.delete_by_fields(relation_type="nonexistent")
assert result is False
# -------------------------------------------------------------------------
# Tests for add_all_ignore_duplicates
# -------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_add_all_ignore_duplicates_basic(
relation_repository: RelationRepository, sample_entity: Entity, related_entity: Entity
):
"""Test bulk inserting relations with ON CONFLICT DO NOTHING."""
relations = [
Relation(
from_id=sample_entity.id,
to_id=related_entity.id,
to_name=related_entity.title,
relation_type="links_to",
),
Relation(
from_id=sample_entity.id,
to_id=related_entity.id,
to_name=related_entity.title,
relation_type="references",
),
]
inserted = await relation_repository.add_all_ignore_duplicates(relations)
# Both should be inserted
assert inserted == 2
# Verify they exist
found = await relation_repository.find_by_entities(sample_entity.id, related_entity.id)
assert len(found) == 2
relation_types = {r.relation_type for r in found}
assert relation_types == {"links_to", "references"}
@pytest.mark.asyncio
async def test_add_all_ignore_duplicates_skips_duplicates(
relation_repository: RelationRepository, sample_entity: Entity, related_entity: Entity
):
"""Test that duplicate relations are silently ignored."""
# Same relation appearing multiple times (common when same [[link]] appears twice in doc)
relations = [
Relation(
from_id=sample_entity.id,
to_id=None, # Unresolved
to_name="Some Target",
relation_type="links_to",
),
Relation(
from_id=sample_entity.id,
to_id=None,
to_name="Some Target", # Duplicate!
relation_type="links_to",
),
Relation(
from_id=sample_entity.id,
to_id=None,
to_name="Some Target", # Triple duplicate!
relation_type="links_to",
),
]
inserted = await relation_repository.add_all_ignore_duplicates(relations)
# Only 1 should be inserted (duplicates ignored)
assert inserted == 1
# Verify only one exists
all_relations = await relation_repository.find_all()
matching = [r for r in all_relations if r.to_name == "Some Target"]
assert len(matching) == 1
@pytest.mark.asyncio
async def test_add_all_ignore_duplicates_empty_list(relation_repository: RelationRepository):
"""Test with empty list returns 0."""
inserted = await relation_repository.add_all_ignore_duplicates([])
assert inserted == 0
@pytest.mark.asyncio
async def test_add_all_ignore_duplicates_mixed(
relation_repository: RelationRepository, sample_entity: Entity, related_entity: Entity
):
"""Test with mix of new and duplicate relations."""
# First, insert one relation
first_relation = Relation(
from_id=sample_entity.id,
to_id=None,
to_name="Existing Target",
relation_type="links_to",
)
await relation_repository.add_all_ignore_duplicates([first_relation])
# Now try to insert a mix of new and duplicate
relations = [
Relation(
from_id=sample_entity.id,
to_id=None,
to_name="Existing Target", # Duplicate of first_relation
relation_type="links_to",
),
Relation(
from_id=sample_entity.id,
to_id=None,
to_name="New Target 1", # New
relation_type="links_to",
),
Relation(
from_id=sample_entity.id,
to_id=None,
to_name="New Target 2", # New
relation_type="references",
),
]
inserted = await relation_repository.add_all_ignore_duplicates(relations)
# Only 2 new ones should be inserted
assert inserted == 2
# Verify total count
all_relations = await relation_repository.find_all()
from_sample = [r for r in all_relations if r.from_id == sample_entity.id]
assert len(from_sample) == 3 # 1 existing + 2 new
@pytest.mark.asyncio
async def test_add_all_ignore_duplicates_with_context(
relation_repository: RelationRepository, sample_entity: Entity, related_entity: Entity
):
"""Test that context field is properly inserted."""
relations = [
Relation(
from_id=sample_entity.id,
to_id=related_entity.id,
to_name=related_entity.title,
relation_type="links_to",
context="some context here",
),
]
inserted = await relation_repository.add_all_ignore_duplicates(relations)
assert inserted == 1
# Verify context was saved
found = await relation_repository.find_by_entities(sample_entity.id, related_entity.id)
assert len(found) == 1
assert found[0].context == "some context here"
+20 -1
View File
@@ -22,6 +22,7 @@ class TestDateTimeSerialization:
test_datetime = datetime(2023, 12, 8, 10, 30, 0)
entity = EntitySummary(
entity_id=1,
permalink="test/entity",
title="Test Entity",
file_path="test/entity.md",
@@ -41,6 +42,8 @@ class TestDateTimeSerialization:
test_datetime = datetime(2023, 12, 8, 15, 45, 30)
relation = RelationSummary(
relation_id=1,
entity_id=1,
title="Test Relation",
file_path="test/relation.md",
permalink="test/relation",
@@ -63,6 +66,8 @@ class TestDateTimeSerialization:
test_datetime = datetime(2023, 12, 8, 20, 15, 45)
observation = ObservationSummary(
observation_id=1,
entity_id=1,
title="Test Observation",
file_path="test/observation.md",
permalink="test/observation",
@@ -100,6 +105,7 @@ class TestDateTimeSerialization:
test_datetime = datetime(2023, 12, 8, 9, 30, 15)
entity = EntitySummary(
entity_id=1,
permalink="test/entity",
title="Test Entity",
file_path="test/entity.md",
@@ -107,6 +113,8 @@ class TestDateTimeSerialization:
)
observation = ObservationSummary(
observation_id=1,
entity_id=1,
title="Test Observation",
file_path="test/observation.md",
permalink="test/observation",
@@ -131,6 +139,7 @@ class TestDateTimeSerialization:
test_datetime = datetime(2023, 12, 8, 14, 20, 10)
entity = EntitySummary(
entity_id=1,
permalink="test/entity",
title="Test Entity",
file_path="test/entity.md",
@@ -159,6 +168,7 @@ class TestDateTimeSerialization:
test_datetime = datetime(2023, 12, 8, 10, 30, 0, 123456)
entity = EntitySummary(
entity_id=1,
permalink="test/entity",
title="Test Entity",
file_path="test/entity.md",
@@ -176,6 +186,7 @@ class TestDateTimeSerialization:
test_datetime = datetime(2023, 12, 8, 10, 30, 0)
entity = EntitySummary(
entity_id=1,
permalink="test/entity",
title="Test Entity",
file_path="test/entity.md",
@@ -212,10 +223,16 @@ class TestDateTimeSerialization:
if model_class == EntitySummary:
instance = model_class(
permalink="test", title="Test", file_path="test.md", created_at=test_datetime
entity_id=1,
permalink="test",
title="Test",
file_path="test.md",
created_at=test_datetime,
)
elif model_class == RelationSummary:
instance = model_class(
relation_id=1,
entity_id=1,
title="Test",
file_path="test.md",
permalink="test",
@@ -224,6 +241,8 @@ class TestDateTimeSerialization:
)
elif model_class == ObservationSummary:
instance = model_class(
observation_id=1,
entity_id=1,
title="Test",
file_path="test.md",
permalink="test",
+1
View File
@@ -286,6 +286,7 @@ async def test_project_isolation_in_find_related(session_maker, app_config):
# Create relation in project1 (between entities of project1)
relation_p1 = Relation(
project_id=project1.id,
from_id=entity1_p1.id,
to_id=entity2_p1.id,
to_name="Entity2_P1",
+7 -7
View File
@@ -432,14 +432,14 @@ async def test_create_with_content(entity_service: EntityService, file_service:
assert entity.observations[0].context == "Reduces merge conflicts"
assert len(entity.relations) == 4
assert entity.relations[0].relation_type == "links to"
assert entity.relations[0].relation_type == "links_to"
assert entity.relations[0].to_name == "Git"
assert entity.relations[1].relation_type == "links to"
assert entity.relations[1].relation_type == "links_to"
assert entity.relations[1].to_name == "Trunk Based Development"
assert entity.relations[2].relation_type == "implements"
assert entity.relations[2].to_name == "Branch Strategy"
assert entity.relations[2].context == "Our standard workflow"
assert entity.relations[3].relation_type == "links to"
assert entity.relations[3].relation_type == "links_to"
assert entity.relations[3].to_name == "Git Cheat Sheet"
# Verify file has new content but preserved metadata
@@ -557,14 +557,14 @@ async def test_update_with_content(entity_service: EntityService, file_service:
assert entity.observations[0].context == "Reduces merge conflicts"
assert len(entity.relations) == 4
assert entity.relations[0].relation_type == "links to"
assert entity.relations[0].relation_type == "links_to"
assert entity.relations[0].to_name == "Git"
assert entity.relations[1].relation_type == "links to"
assert entity.relations[1].relation_type == "links_to"
assert entity.relations[1].to_name == "Trunk Based Development"
assert entity.relations[2].relation_type == "implements"
assert entity.relations[2].to_name == "Branch Strategy"
assert entity.relations[2].context == "Our standard workflow"
assert entity.relations[3].relation_type == "links to"
assert entity.relations[3].relation_type == "links_to"
assert entity.relations[3].to_name == "Git Cheat Sheet"
# Verify file has new content but preserved metadata
@@ -1772,7 +1772,7 @@ async def test_move_entity_with_complex_observations(
# Check relations
relation_types = {rel.relation_type for rel in moved_entity.relations}
assert "implements" in relation_types
assert "links to" in relation_types
assert "links_to" in relation_types
relation_targets = {rel.to_name for rel in moved_entity.relations}
assert "Branch Strategy" in relation_targets

Some files were not shown because too many files have changed in this diff Show More