mirror of
https://github.com/basicmachines-co/basic-memory
synced 2026-06-21 13:47:35 +00:00
Compare commits
35 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 24dc9a2931 | |||
| 85c63e5a7a | |||
| f227ef6a86 | |||
| 897b1edaa4 | |||
| 0c12a39a98 | |||
| efbc758325 | |||
| a0f20eb102 | |||
| 78673d8e51 | |||
| 126c0495c0 | |||
| 4a43d7df4a | |||
| c462faf046 | |||
| 70bb10be1d | |||
| fbf9045d78 | |||
| 1094210c52 | |||
| 391feb639f | |||
| a920a9ff29 | |||
| 05efe8701c | |||
| 0eaf30bb06 | |||
| 0818bda565 | |||
| 6f99d2e551 | |||
| 73d940e064 | |||
| c3678a11d2 | |||
| 203d684c24 | |||
| a872220924 | |||
| 7d763a66ff | |||
| b5d4fb559c | |||
| 830775276d | |||
| ed894fc3ed | |||
| 704338edcf | |||
| 0ca02a7ebe | |||
| 28cc5225a7 | |||
| 9b7bbc7116 | |||
| 138c283d6c | |||
| 7a8954c37e | |||
| 10c7c19c03 |
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"enabledPlugins": {
|
||||
"basic-memory@basicmachines": true
|
||||
}
|
||||
}
|
||||
@@ -78,21 +78,7 @@ jobs:
|
||||
python-version: [ "3.12", "3.13" ]
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
# Postgres service (only available on Linux runners)
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:17
|
||||
env:
|
||||
POSTGRES_DB: basic_memory_test
|
||||
POSTGRES_USER: basic_memory_user
|
||||
POSTGRES_PASSWORD: dev_password
|
||||
options: >-
|
||||
--health-cmd pg_isready
|
||||
--health-interval 10s
|
||||
--health-timeout 5s
|
||||
--health-retries 5
|
||||
ports:
|
||||
- 5433:5432
|
||||
# Note: No services section needed - testcontainers handles Postgres in Docker
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
@@ -121,7 +107,7 @@ jobs:
|
||||
run: |
|
||||
uv pip install -e .[dev]
|
||||
|
||||
- name: Run tests (Postgres)
|
||||
- name: Run tests (Postgres via testcontainers)
|
||||
run: |
|
||||
uv pip install pytest pytest-cov
|
||||
just test-postgres
|
||||
+2
-1
@@ -52,4 +52,5 @@ ENV/
|
||||
|
||||
# claude action
|
||||
claude-output
|
||||
**/.claude/settings.local.json
|
||||
**/.claude/settings.local.json
|
||||
.mcp.json
|
||||
|
||||
@@ -1,5 +1,98 @@
|
||||
# CHANGELOG
|
||||
|
||||
## v0.16.3 (2025-12-20)
|
||||
|
||||
### Features
|
||||
|
||||
- **#439**: Add PostgreSQL database backend support
|
||||
([`fb5e9e1`](https://github.com/basicmachines-co/basic-memory/commit/fb5e9e1))
|
||||
- Full PostgreSQL/Neon database support as alternative to SQLite
|
||||
- Async connection pooling with asyncpg
|
||||
- Alembic migrations support for both backends
|
||||
- Configurable via `BASIC_MEMORY_DATABASE_BACKEND` environment variable
|
||||
|
||||
- **#441**: Implement API v2 with ID-based endpoints (Phase 1)
|
||||
([`28cc522`](https://github.com/basicmachines-co/basic-memory/commit/28cc522))
|
||||
- New ID-based API endpoints for improved performance
|
||||
- Foundation for future API enhancements
|
||||
- Backward compatible with existing endpoints
|
||||
|
||||
- Add project_id to Relation and Observation for efficient project-scoped queries
|
||||
([`a920a9f`](https://github.com/basicmachines-co/basic-memory/commit/a920a9f))
|
||||
- Enables faster queries in multi-project environments
|
||||
- Improved database schema for cloud deployments
|
||||
|
||||
- Add bulk insert with ON CONFLICT handling for relations
|
||||
([`0818bda`](https://github.com/basicmachines-co/basic-memory/commit/0818bda))
|
||||
- Faster relation creation during sync operations
|
||||
- Handles duplicate relations gracefully
|
||||
|
||||
### Performance
|
||||
|
||||
- Lightweight permalink resolution to avoid eager loading
|
||||
([`6f99d2e`](https://github.com/basicmachines-co/basic-memory/commit/6f99d2e))
|
||||
- Reduces database queries during entity lookups
|
||||
- Improved response times for read operations
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- **#464**: Pin FastMCP to 2.12.3 to fix MCP tools visibility
|
||||
([`f227ef6`](https://github.com/basicmachines-co/basic-memory/commit/f227ef6))
|
||||
- Fixes issue where MCP tools were not visible to Claude
|
||||
- Reverts to last known working FastMCP version
|
||||
|
||||
- **#458**: Reduce watch service CPU usage by increasing reload interval
|
||||
([`897b1ed`](https://github.com/basicmachines-co/basic-memory/commit/897b1ed))
|
||||
- Lowers CPU usage during file watching
|
||||
- More efficient resource utilization
|
||||
|
||||
- **#456**: Await background sync task cancellation in lifespan shutdown
|
||||
([`efbc758`](https://github.com/basicmachines-co/basic-memory/commit/efbc758))
|
||||
- Prevents hanging on shutdown
|
||||
- Clean async task cleanup
|
||||
|
||||
- **#434**: Respect --project flag in background sync
|
||||
([`70bb10b`](https://github.com/basicmachines-co/basic-memory/commit/70bb10b))
|
||||
- Background sync now correctly uses specified project
|
||||
- Fixes multi-project sync issues
|
||||
|
||||
- **#446**: Fix observation parsing and permalink limits
|
||||
([`73d940e`](https://github.com/basicmachines-co/basic-memory/commit/73d940e))
|
||||
- Handles edge cases in observation content
|
||||
- Prevents permalink truncation issues
|
||||
|
||||
- **#424**: Handle periods in kebab_filenames mode
|
||||
([`b004565`](https://github.com/basicmachines-co/basic-memory/commit/b004565))
|
||||
- Fixes filename handling for files with multiple periods
|
||||
- Improved kebab-case conversion
|
||||
|
||||
- Fix Postgres/Neon connection settings and search index dedupe
|
||||
([`b5d4fb5`](https://github.com/basicmachines-co/basic-memory/commit/b5d4fb5))
|
||||
- Optimized connection pooling for Postgres
|
||||
- Prevents duplicate search index entries
|
||||
|
||||
### Testing & CI
|
||||
|
||||
- Replace py-pglite with testcontainers for Postgres testing
|
||||
([`c462faf`](https://github.com/basicmachines-co/basic-memory/commit/c462faf))
|
||||
- More reliable Postgres testing infrastructure
|
||||
- Uses Docker-based test containers
|
||||
|
||||
- Add PostgreSQL testing to GitHub Actions workflow
|
||||
([`66b91b2`](https://github.com/basicmachines-co/basic-memory/commit/66b91b2))
|
||||
- CI now tests both SQLite and PostgreSQL backends
|
||||
- Ensures cross-database compatibility
|
||||
|
||||
- **#416**: Add integration test for read_note with underscored folders
|
||||
([`0c12a39`](https://github.com/basicmachines-co/basic-memory/commit/0c12a39))
|
||||
- Verifies folder name handling edge cases
|
||||
|
||||
### Internal
|
||||
|
||||
- Cloud compatibility fixes and performance improvements (#454)
|
||||
- Remove logfire instrumentation for cleaner production deployments
|
||||
- Truncate content_stems to fix Postgres 8KB index row limit
|
||||
|
||||
## v0.16.2 (2025-11-16)
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
@@ -15,10 +15,14 @@ See the [README.md](README.md) file for a project overview.
|
||||
### Build and Test Commands
|
||||
|
||||
- Install: `just install` or `pip install -e ".[dev]"`
|
||||
- Run all tests (with coverage): `just test` - Runs both unit and integration tests with unified coverage
|
||||
- Run unit tests only: `just test-unit` - Fast, no coverage
|
||||
- Run integration tests only: `just test-int` - Fast, no coverage
|
||||
- Generate HTML coverage: `just coverage` - Opens in browser
|
||||
- Run all tests (SQLite + Postgres): `just test`
|
||||
- Run all tests against SQLite: `just test-sqlite`
|
||||
- Run all tests against Postgres: `just test-postgres` (uses testcontainers)
|
||||
- Run unit tests (SQLite): `just test-unit-sqlite`
|
||||
- Run unit tests (Postgres): `just test-unit-postgres`
|
||||
- Run integration tests (SQLite): `just test-int-sqlite`
|
||||
- Run integration tests (Postgres): `just test-int-postgres`
|
||||
- Generate HTML coverage: `just coverage`
|
||||
- Single test: `pytest tests/path/to/test_file.py::test_function_name`
|
||||
- Run benchmarks: `pytest test-int/test_sync_performance_benchmark.py -v -m "benchmark and not slow"`
|
||||
- Lint: `just lint` or `ruff check . --fix`
|
||||
@@ -30,6 +34,8 @@ See the [README.md](README.md) file for a project overview.
|
||||
|
||||
**Note:** Project requires Python 3.12+ (uses type parameter syntax and `type` aliases introduced in 3.12)
|
||||
|
||||
**Postgres Testing:** Uses [testcontainers](https://testcontainers-python.readthedocs.io/) which automatically spins up a Postgres instance in Docker. No manual database setup required - just have Docker running.
|
||||
|
||||
### Test Structure
|
||||
|
||||
- `tests/` - Unit tests for individual components (mocked, fast)
|
||||
@@ -76,8 +82,10 @@ See the [README.md](README.md) file for a project overview.
|
||||
- SQLite is used for indexing and full text search, files are source of truth
|
||||
- Testing uses pytest with asyncio support (strict mode)
|
||||
- Unit tests (`tests/`) use mocks when necessary; integration tests (`test-int/`) use real implementations
|
||||
- Test database uses in-memory SQLite
|
||||
- Each test runs in a standalone environment with in-memory SQLite and tmp_file directory
|
||||
- By default, tests run against SQLite (fast, no Docker needed)
|
||||
- Set `BASIC_MEMORY_TEST_POSTGRES=1` to run against Postgres (uses testcontainers - Docker required)
|
||||
- Each test runs in a standalone environment with isolated database and tmp_path directory
|
||||
- CI runs SQLite and Postgres tests in parallel for faster feedback
|
||||
- Performance benchmarks are in `test-int/test_sync_performance_benchmark.py`
|
||||
- Use pytest markers: `@pytest.mark.benchmark` for benchmarks, `@pytest.mark.slow` for slow tests
|
||||
|
||||
@@ -229,6 +237,11 @@ of using AI just for code generation, we've developed a true collaborative workf
|
||||
This approach has allowed us to tackle more complex challenges and build a more robust system than either humans or AI
|
||||
could achieve independently.
|
||||
|
||||
**Problem-Solving Guidance:**
|
||||
- If a solution isn't working after reasonable effort, suggest alternative approaches
|
||||
- Don't persist with a problematic library or pattern when better alternatives exist
|
||||
- Example: When py-pglite caused cascading test failures, switching to testcontainers-postgres was the right call
|
||||
|
||||
## GitHub Integration
|
||||
|
||||
Basic Memory has taken AI-Human collaboration to the next level by integrating Claude directly into the development workflow through GitHub:
|
||||
@@ -264,5 +277,6 @@ With GitHub integration, the development workflow includes:
|
||||
2. **Contribution tracking** - All of Claude's contributions are properly attributed in the Git history
|
||||
3. **Branch management** - Claude can create feature branches for implementations
|
||||
4. **Documentation maintenance** - Claude can keep documentation updated as the code evolves
|
||||
5. **Code Commits**: ALWAYS sign off commits with `git commit -s`
|
||||
|
||||
This level of integration represents a new paradigm in AI-human collaboration, where the AI assistant becomes a full-fledged team member rather than just a tool for generating code snippets.
|
||||
|
||||
@@ -433,42 +433,76 @@ See the [Documentation](https://memory.basicmachines.co/) for more info, includi
|
||||
- [Managing multiple Projects](https://docs.basicmemory.com/guides/cli-reference/#project)
|
||||
- [Importing data from OpenAI/Claude Projects](https://docs.basicmemory.com/guides/cli-reference/#import)
|
||||
|
||||
## Logging
|
||||
|
||||
Basic Memory uses [Loguru](https://github.com/Delgan/loguru) for logging. The logging behavior varies by entry point:
|
||||
|
||||
| Entry Point | Default Behavior | Use Case |
|
||||
|-------------|------------------|----------|
|
||||
| CLI commands | File only | Prevents log output from interfering with command output |
|
||||
| MCP server | File only | Stdout would corrupt the JSON-RPC protocol |
|
||||
| API server | File (local) or stdout (cloud) | Docker/cloud deployments use stdout |
|
||||
|
||||
**Log file location:** `~/.basic-memory/basic-memory.log` (10MB rotation, 10 days retention)
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `BASIC_MEMORY_LOG_LEVEL` | `INFO` | Log level: DEBUG, INFO, WARNING, ERROR |
|
||||
| `BASIC_MEMORY_CLOUD_MODE` | `false` | When `true`, API logs to stdout with structured context |
|
||||
| `BASIC_MEMORY_ENV` | `dev` | Set to `test` for test mode (stderr only) |
|
||||
|
||||
### Examples
|
||||
|
||||
```bash
|
||||
# Enable debug logging
|
||||
BASIC_MEMORY_LOG_LEVEL=DEBUG basic-memory sync
|
||||
|
||||
# View logs
|
||||
tail -f ~/.basic-memory/basic-memory.log
|
||||
|
||||
# Cloud/Docker mode (stdout logging with structured context)
|
||||
BASIC_MEMORY_CLOUD_MODE=true uvicorn basic_memory.api.app:app
|
||||
```
|
||||
|
||||
## Development
|
||||
|
||||
### Running Tests
|
||||
|
||||
Basic Memory supports dual database backends (SQLite and Postgres). Tests are parametrized to run against both backends automatically.
|
||||
Basic Memory supports dual database backends (SQLite and Postgres). By default, tests run against SQLite. Set `BASIC_MEMORY_TEST_POSTGRES=1` to run against Postgres (uses testcontainers - Docker required).
|
||||
|
||||
**Quick Start:**
|
||||
```bash
|
||||
# Run SQLite tests (default, no Docker needed)
|
||||
# Run all tests against SQLite (default, fast)
|
||||
just test-sqlite
|
||||
|
||||
# Run Postgres tests (requires Docker)
|
||||
# Run all tests against Postgres (uses testcontainers)
|
||||
just test-postgres
|
||||
|
||||
# Run both SQLite and Postgres tests
|
||||
just test
|
||||
```
|
||||
|
||||
**Available Test Commands:**
|
||||
|
||||
- `just test-sqlite` - Run tests against SQLite only (fastest, no Docker needed)
|
||||
- `just test-postgres` - Run tests against Postgres only (requires Docker)
|
||||
- `just test` - Run all tests against both SQLite and Postgres
|
||||
- `just test-sqlite` - Run all tests against SQLite (fast, no Docker needed)
|
||||
- `just test-postgres` - Run all tests against Postgres (uses testcontainers)
|
||||
- `just test-unit-sqlite` - Run unit tests against SQLite
|
||||
- `just test-unit-postgres` - Run unit tests against Postgres
|
||||
- `just test-int-sqlite` - Run integration tests against SQLite
|
||||
- `just test-int-postgres` - Run integration tests against Postgres
|
||||
- `just test-windows` - Run Windows-specific tests (auto-skips on other platforms)
|
||||
- `just test-benchmark` - Run performance benchmark tests
|
||||
- `just test-all` - Run all tests including Windows, Postgres, and benchmarks
|
||||
|
||||
**Postgres Testing Requirements:**
|
||||
**Postgres Testing:**
|
||||
|
||||
To run Postgres tests, you need to start the test database:
|
||||
```bash
|
||||
docker-compose -f docker-compose-postgres.yml up -d
|
||||
```
|
||||
|
||||
Tests will connect to `localhost:5433/basic_memory_test`.
|
||||
Postgres tests use [testcontainers](https://testcontainers-python.readthedocs.io/) which automatically spins up a Postgres instance in Docker. No manual database setup required - just have Docker running.
|
||||
|
||||
**Test Markers:**
|
||||
|
||||
Tests use pytest markers for selective execution:
|
||||
- `postgres` - Tests that run against Postgres backend
|
||||
- `windows` - Windows-specific database optimizations
|
||||
- `benchmark` - Performance tests (excluded from default runs)
|
||||
|
||||
|
||||
@@ -7,44 +7,51 @@ install:
|
||||
@echo ""
|
||||
@echo "💡 Remember to activate the virtual environment by running: source .venv/bin/activate"
|
||||
|
||||
# Run all tests with unified coverage report
|
||||
test: test-unit test-int
|
||||
|
||||
# Run unit tests only (fast, no coverage)
|
||||
test-unit:
|
||||
uv run pytest -p pytest_mock -v --no-cov tests
|
||||
|
||||
# Run integration tests only (fast, no coverage)
|
||||
test-int:
|
||||
uv run pytest -p pytest_mock -v --no-cov test-int
|
||||
|
||||
# ==============================================================================
|
||||
# DATABASE BACKEND TESTING
|
||||
# ==============================================================================
|
||||
# Basic Memory supports dual database backends (SQLite and Postgres).
|
||||
# Tests are parametrized to run against both backends automatically.
|
||||
# By default, tests run against SQLite (fast, no dependencies).
|
||||
# Set BASIC_MEMORY_TEST_POSTGRES=1 to run against Postgres (uses testcontainers).
|
||||
#
|
||||
# Quick Start:
|
||||
# just test-sqlite # Run SQLite tests (default, no Docker needed)
|
||||
# just test-postgres # Run Postgres tests (requires Docker)
|
||||
# just test # Run all tests against SQLite (default)
|
||||
# just test-sqlite # Run all tests against SQLite
|
||||
# just test-postgres # Run all tests against Postgres (testcontainers)
|
||||
# just test-unit-sqlite # Run unit tests against SQLite
|
||||
# just test-unit-postgres # Run unit tests against Postgres
|
||||
# just test-int-sqlite # Run integration tests against SQLite
|
||||
# just test-int-postgres # Run integration tests against Postgres
|
||||
#
|
||||
# For Postgres tests, first start the database:
|
||||
# docker-compose -f docker-compose-postgres.yml up -d
|
||||
# CI runs both in parallel for faster feedback.
|
||||
# ==============================================================================
|
||||
|
||||
# Run tests against SQLite only (default backend, skip Postgres/Benchmark tests)
|
||||
# This is the fastest option and doesn't require any Docker setup.
|
||||
# Use this for local development and quick feedback.
|
||||
# Includes Windows-specific tests which will auto-skip on non-Windows platforms.
|
||||
test-sqlite:
|
||||
uv run pytest -p pytest_mock -v --no-cov -m "not postgres and not benchmark" tests test-int
|
||||
# Run all tests against SQLite and Postgres
|
||||
test: test-sqlite test-postgres
|
||||
|
||||
# Run tests against Postgres only (requires docker-compose-postgres.yml up)
|
||||
# First start Postgres: docker-compose -f docker-compose-postgres.yml up -d
|
||||
# Tests will connect to localhost:5433/basic_memory_test
|
||||
# To reset the database: just postgres-reset
|
||||
test-postgres:
|
||||
uv run pytest -p pytest_mock -v --no-cov -m "postgres and not benchmark" tests test-int
|
||||
# Run all tests against SQLite
|
||||
test-sqlite: test-unit-sqlite test-int-sqlite
|
||||
|
||||
# Run all tests against Postgres (uses testcontainers)
|
||||
test-postgres: test-unit-postgres test-int-postgres
|
||||
|
||||
# Run unit tests against SQLite
|
||||
test-unit-sqlite:
|
||||
uv run pytest -p pytest_mock -v --no-cov tests
|
||||
|
||||
# Run unit tests against Postgres
|
||||
test-unit-postgres:
|
||||
BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov tests
|
||||
|
||||
# Run integration tests against SQLite
|
||||
test-int-sqlite:
|
||||
uv run pytest -p pytest_mock -v --no-cov test-int
|
||||
|
||||
# Run integration tests against Postgres
|
||||
# Note: Uses timeout due to FastMCP Client + asyncpg cleanup hang (tests pass, process hangs on exit)
|
||||
# See: https://github.com/jlowin/fastmcp/issues/1311
|
||||
test-int-postgres:
|
||||
timeout --signal=KILL 600 bash -c 'BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov test-int' || test $? -eq 137
|
||||
|
||||
# Reset Postgres test database (drops and recreates schema)
|
||||
# Useful when Alembic migration state gets out of sync during development
|
||||
@@ -59,7 +66,7 @@ postgres-reset:
|
||||
postgres-migrate:
|
||||
@cd src/basic_memory/alembic && \
|
||||
BASIC_MEMORY_DATABASE_BACKEND=postgres \
|
||||
BASIC_MEMORY_DATABASE_URL=${POSTGRES_TEST_URL:-postgresql://basic_memory_user:dev_password@localhost:5433/basic_memory_test} \
|
||||
BASIC_MEMORY_DATABASE_URL=${POSTGRES_TEST_URL:-postgresql+asyncpg://basic_memory_user:dev_password@localhost:5433/basic_memory_test} \
|
||||
uv run alembic upgrade head
|
||||
@echo "✅ Migrations applied to Postgres test database"
|
||||
|
||||
|
||||
+7
-6
@@ -15,7 +15,6 @@ dependencies = [
|
||||
"aiosqlite>=0.20.0",
|
||||
"greenlet>=3.1.1",
|
||||
"pydantic[email,timezone]>=2.10.3",
|
||||
"icecream>=2.1.3",
|
||||
"mcp>=1.2.0",
|
||||
"pydantic-settings>=2.6.1",
|
||||
"loguru>=0.7.3",
|
||||
@@ -30,13 +29,15 @@ dependencies = [
|
||||
"alembic>=1.14.1",
|
||||
"pillow>=11.1.0",
|
||||
"pybars3>=0.9.7",
|
||||
"fastmcp>=2.10.2",
|
||||
"fastmcp==2.12.3", # Pinned - 2.14.x breaks MCP tools visibility (issue #463)
|
||||
"pyjwt>=2.10.1",
|
||||
"python-dotenv>=1.1.0",
|
||||
"pytest-aio>=1.9.0",
|
||||
"aiofiles>=24.1.0", # Async file I/O
|
||||
"logfire>=0.73.0", # Optional observability (disabled by default via config)
|
||||
"aiofiles>=24.1.0", # Optional observability (disabled by default via config)
|
||||
"asyncpg>=0.30.0",
|
||||
"nest-asyncio>=1.6.0", # For Alembic migrations with Postgres
|
||||
"pytest-asyncio>=1.2.0",
|
||||
"psycopg==3.3.1",
|
||||
]
|
||||
|
||||
|
||||
@@ -81,8 +82,8 @@ dev = [
|
||||
"pytest-xdist>=3.0.0",
|
||||
"ruff>=0.1.6",
|
||||
"freezegun>=1.5.5",
|
||||
"nest-asyncio>=1.6.0",
|
||||
"psycopg2-binary>=2.9.0", # For Alembic migrations with Postgres
|
||||
"testcontainers[postgres]>=4.0.0",
|
||||
"psycopg>=3.2.0",
|
||||
]
|
||||
|
||||
[tool.hatch.version]
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
"""basic-memory - Local-first knowledge management combining Zettelkasten with knowledge graphs"""
|
||||
|
||||
# Package version - updated by release automation
|
||||
__version__ = "0.16.2"
|
||||
__version__ = "0.16.3"
|
||||
|
||||
# API version for FastAPI - independent of package version
|
||||
__api_version__ = "v0"
|
||||
|
||||
@@ -1,14 +1,25 @@
|
||||
"""Alembic environment configuration."""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
from logging.config import fileConfig
|
||||
|
||||
from sqlalchemy import engine_from_config
|
||||
from sqlalchemy import pool
|
||||
# Allow nested event loops (needed for pytest-asyncio and other async contexts)
|
||||
# Note: nest_asyncio doesn't work with uvloop, so we handle that case separately
|
||||
try:
|
||||
import nest_asyncio
|
||||
|
||||
nest_asyncio.apply()
|
||||
except (ImportError, ValueError):
|
||||
# nest_asyncio not available or can't patch this loop type (e.g., uvloop)
|
||||
pass
|
||||
|
||||
from sqlalchemy import engine_from_config, pool
|
||||
from sqlalchemy.ext.asyncio import AsyncEngine, create_async_engine
|
||||
|
||||
from alembic import context
|
||||
|
||||
from basic_memory.config import ConfigManager, DatabaseBackend
|
||||
from basic_memory.config import ConfigManager
|
||||
|
||||
# set config.env to "test" for pytest to prevent logging to file in utils.setup_logging()
|
||||
os.environ["BASIC_MEMORY_ENV"] = "test"
|
||||
@@ -35,12 +46,6 @@ if not current_url or current_url == "driver://user:pass@localhost/dbname":
|
||||
sqlalchemy_url = DatabaseType.get_db_url(
|
||||
app_config.database_path, DatabaseType.FILESYSTEM, app_config
|
||||
)
|
||||
|
||||
# For Postgres, Alembic needs synchronous driver (psycopg2), not async (asyncpg)
|
||||
if app_config.database_backend == DatabaseBackend.POSTGRES:
|
||||
# Convert asyncpg URL to psycopg2 URL for Alembic
|
||||
sqlalchemy_url = sqlalchemy_url.replace("postgresql+asyncpg://", "postgresql://")
|
||||
|
||||
config.set_main_option("sqlalchemy.url", sqlalchemy_url)
|
||||
|
||||
# Interpret the config file for Python logging.
|
||||
@@ -85,28 +90,89 @@ def run_migrations_offline() -> None:
|
||||
context.run_migrations()
|
||||
|
||||
|
||||
def do_run_migrations(connection):
|
||||
"""Execute migrations with the given connection."""
|
||||
context.configure(
|
||||
connection=connection,
|
||||
target_metadata=target_metadata,
|
||||
include_object=include_object,
|
||||
render_as_batch=True,
|
||||
compare_type=True,
|
||||
)
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
|
||||
|
||||
async def run_async_migrations(connectable):
|
||||
"""Run migrations asynchronously with AsyncEngine."""
|
||||
async with connectable.connect() as connection:
|
||||
await connection.run_sync(do_run_migrations)
|
||||
await connectable.dispose()
|
||||
|
||||
|
||||
def run_migrations_online() -> None:
|
||||
"""Run migrations in 'online' mode.
|
||||
|
||||
In this scenario we need to create an Engine
|
||||
and associate a connection with the context.
|
||||
Supports both sync engines (SQLite) and async engines (PostgreSQL with asyncpg).
|
||||
"""
|
||||
connectable = engine_from_config(
|
||||
config.get_section(config.config_ini_section, {}),
|
||||
prefix="sqlalchemy.",
|
||||
poolclass=pool.NullPool,
|
||||
)
|
||||
# Check if a connection/engine was provided (e.g., from run_migrations)
|
||||
connectable = context.config.attributes.get("connection", None)
|
||||
|
||||
with connectable.connect() as connection:
|
||||
context.configure(
|
||||
connection=connection,
|
||||
target_metadata=target_metadata,
|
||||
include_object=include_object,
|
||||
render_as_batch=True,
|
||||
)
|
||||
if connectable is None:
|
||||
# No connection provided, create engine from config
|
||||
url = context.config.get_main_option("sqlalchemy.url")
|
||||
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
# Check if it's an async URL (sqlite+aiosqlite or postgresql+asyncpg)
|
||||
if url and ("+asyncpg" in url or "+aiosqlite" in url):
|
||||
# Create async engine for asyncpg or aiosqlite
|
||||
connectable = create_async_engine(
|
||||
url,
|
||||
poolclass=pool.NullPool,
|
||||
future=True,
|
||||
)
|
||||
else:
|
||||
# Create sync engine for regular sqlite or postgresql
|
||||
connectable = engine_from_config(
|
||||
context.config.get_section(context.config.config_ini_section, {}),
|
||||
prefix="sqlalchemy.",
|
||||
poolclass=pool.NullPool,
|
||||
)
|
||||
|
||||
# Handle async engines (PostgreSQL with asyncpg)
|
||||
if isinstance(connectable, AsyncEngine):
|
||||
# Try to run async migrations
|
||||
# nest_asyncio allows asyncio.run() from within event loops, but doesn't work with uvloop
|
||||
try:
|
||||
asyncio.run(run_async_migrations(connectable))
|
||||
except RuntimeError as e:
|
||||
if "cannot be called from a running event loop" in str(e):
|
||||
# We're in a running event loop (likely uvloop) - need to use a different approach
|
||||
# Create a new thread to run the async migrations
|
||||
import concurrent.futures
|
||||
|
||||
def run_in_thread():
|
||||
"""Run async migrations in a new event loop in a separate thread."""
|
||||
new_loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(new_loop)
|
||||
try:
|
||||
new_loop.run_until_complete(run_async_migrations(connectable))
|
||||
finally:
|
||||
new_loop.close()
|
||||
|
||||
with concurrent.futures.ThreadPoolExecutor() as executor:
|
||||
future = executor.submit(run_in_thread)
|
||||
future.result() # Wait for completion and re-raise any exceptions
|
||||
else:
|
||||
raise
|
||||
else:
|
||||
# Handle sync engines (SQLite) or sync connections
|
||||
if hasattr(connectable, "connect"):
|
||||
# It's an engine, get a connection
|
||||
with connectable.connect() as connection:
|
||||
do_run_migrations(connection)
|
||||
else:
|
||||
# It's already a connection
|
||||
do_run_migrations(connectable)
|
||||
|
||||
|
||||
if context.is_offline_mode():
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
"""Add cascade delete FK from search_index to entity
|
||||
|
||||
Revision ID: a2b3c4d5e6f7
|
||||
Revises: f8a9b2c3d4e5
|
||||
Create Date: 2025-12-02 07:00:00.000000
|
||||
|
||||
"""
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "a2b3c4d5e6f7"
|
||||
down_revision: Union[str, None] = "f8a9b2c3d4e5"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
"""Add FK with CASCADE delete from search_index.entity_id to entity.id.
|
||||
|
||||
This migration is Postgres-only because:
|
||||
- SQLite uses FTS5 virtual tables which don't support foreign keys
|
||||
- The FK enables automatic cleanup of search_index entries when entities are deleted
|
||||
"""
|
||||
connection = op.get_bind()
|
||||
dialect = connection.dialect.name
|
||||
|
||||
if dialect == "postgresql":
|
||||
# First, clean up any orphaned search_index entries where entity no longer exists
|
||||
op.execute("""
|
||||
DELETE FROM search_index
|
||||
WHERE entity_id IS NOT NULL
|
||||
AND entity_id NOT IN (SELECT id FROM entity)
|
||||
""")
|
||||
|
||||
# Add FK with CASCADE - nullable FK allows search_index entries without entity_id
|
||||
op.create_foreign_key(
|
||||
"fk_search_index_entity_id",
|
||||
"search_index",
|
||||
"entity",
|
||||
["entity_id"],
|
||||
["id"],
|
||||
ondelete="CASCADE",
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
"""Remove the FK constraint."""
|
||||
connection = op.get_bind()
|
||||
dialect = connection.dialect.name
|
||||
|
||||
if dialect == "postgresql":
|
||||
op.drop_constraint("fk_search_index_entity_id", "search_index", type_="foreignkey")
|
||||
+199
@@ -0,0 +1,199 @@
|
||||
"""Add project_id to relation/observation and pg_trgm for fuzzy link resolution
|
||||
|
||||
Revision ID: f8a9b2c3d4e5
|
||||
Revises: 314f1ea54dc4
|
||||
Create Date: 2025-12-01 12:00:00.000000
|
||||
|
||||
"""
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "f8a9b2c3d4e5"
|
||||
down_revision: Union[str, None] = "314f1ea54dc4"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
"""Add project_id to relation and observation tables, plus pg_trgm indexes.
|
||||
|
||||
This migration:
|
||||
1. Adds project_id column to relation and observation tables (denormalization)
|
||||
2. Backfills project_id from the associated entity
|
||||
3. Enables pg_trgm extension for trigram-based fuzzy matching (Postgres only)
|
||||
4. Creates GIN indexes on entity title and permalink for fast similarity searches
|
||||
5. Creates partial index on unresolved relations for efficient bulk resolution
|
||||
"""
|
||||
connection = op.get_bind()
|
||||
dialect = connection.dialect.name
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Add project_id to relation table
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
# Step 1: Add project_id column as nullable first
|
||||
op.add_column("relation", sa.Column("project_id", sa.Integer(), nullable=True))
|
||||
|
||||
# Step 2: Backfill project_id from entity.project_id via from_id
|
||||
if dialect == "postgresql":
|
||||
op.execute("""
|
||||
UPDATE relation
|
||||
SET project_id = entity.project_id
|
||||
FROM entity
|
||||
WHERE relation.from_id = entity.id
|
||||
""")
|
||||
else:
|
||||
# SQLite syntax
|
||||
op.execute("""
|
||||
UPDATE relation
|
||||
SET project_id = (
|
||||
SELECT entity.project_id
|
||||
FROM entity
|
||||
WHERE entity.id = relation.from_id
|
||||
)
|
||||
""")
|
||||
|
||||
# Step 3: Make project_id NOT NULL and add foreign key
|
||||
if dialect == "postgresql":
|
||||
op.alter_column("relation", "project_id", nullable=False)
|
||||
op.create_foreign_key(
|
||||
"fk_relation_project_id",
|
||||
"relation",
|
||||
"project",
|
||||
["project_id"],
|
||||
["id"],
|
||||
)
|
||||
else:
|
||||
# SQLite requires batch operations for ALTER COLUMN
|
||||
with op.batch_alter_table("relation") as batch_op:
|
||||
batch_op.alter_column("project_id", nullable=False)
|
||||
batch_op.create_foreign_key(
|
||||
"fk_relation_project_id",
|
||||
"project",
|
||||
["project_id"],
|
||||
["id"],
|
||||
)
|
||||
|
||||
# Step 4: Create index on relation.project_id
|
||||
op.create_index("ix_relation_project_id", "relation", ["project_id"])
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Add project_id to observation table
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
# Step 1: Add project_id column as nullable first
|
||||
op.add_column("observation", sa.Column("project_id", sa.Integer(), nullable=True))
|
||||
|
||||
# Step 2: Backfill project_id from entity.project_id via entity_id
|
||||
if dialect == "postgresql":
|
||||
op.execute("""
|
||||
UPDATE observation
|
||||
SET project_id = entity.project_id
|
||||
FROM entity
|
||||
WHERE observation.entity_id = entity.id
|
||||
""")
|
||||
else:
|
||||
# SQLite syntax
|
||||
op.execute("""
|
||||
UPDATE observation
|
||||
SET project_id = (
|
||||
SELECT entity.project_id
|
||||
FROM entity
|
||||
WHERE entity.id = observation.entity_id
|
||||
)
|
||||
""")
|
||||
|
||||
# Step 3: Make project_id NOT NULL and add foreign key
|
||||
if dialect == "postgresql":
|
||||
op.alter_column("observation", "project_id", nullable=False)
|
||||
op.create_foreign_key(
|
||||
"fk_observation_project_id",
|
||||
"observation",
|
||||
"project",
|
||||
["project_id"],
|
||||
["id"],
|
||||
)
|
||||
else:
|
||||
# SQLite requires batch operations for ALTER COLUMN
|
||||
with op.batch_alter_table("observation") as batch_op:
|
||||
batch_op.alter_column("project_id", nullable=False)
|
||||
batch_op.create_foreign_key(
|
||||
"fk_observation_project_id",
|
||||
"project",
|
||||
["project_id"],
|
||||
["id"],
|
||||
)
|
||||
|
||||
# Step 4: Create index on observation.project_id
|
||||
op.create_index("ix_observation_project_id", "observation", ["project_id"])
|
||||
|
||||
# Postgres-specific: pg_trgm and GIN indexes
|
||||
if dialect == "postgresql":
|
||||
# Enable pg_trgm extension for fuzzy string matching
|
||||
op.execute("CREATE EXTENSION IF NOT EXISTS pg_trgm")
|
||||
|
||||
# Create trigram indexes on entity table for fuzzy matching
|
||||
# GIN indexes with gin_trgm_ops support similarity searches
|
||||
op.execute("""
|
||||
CREATE INDEX IF NOT EXISTS idx_entity_title_trgm
|
||||
ON entity USING gin (title gin_trgm_ops)
|
||||
""")
|
||||
|
||||
op.execute("""
|
||||
CREATE INDEX IF NOT EXISTS idx_entity_permalink_trgm
|
||||
ON entity USING gin (permalink gin_trgm_ops)
|
||||
""")
|
||||
|
||||
# Create partial index on unresolved relations for efficient bulk resolution
|
||||
# This makes "WHERE to_id IS NULL AND project_id = X" queries very fast
|
||||
op.execute("""
|
||||
CREATE INDEX IF NOT EXISTS idx_relation_unresolved
|
||||
ON relation (project_id, to_name)
|
||||
WHERE to_id IS NULL
|
||||
""")
|
||||
|
||||
# Create index on relation.to_name for join performance in bulk resolution
|
||||
op.execute("""
|
||||
CREATE INDEX IF NOT EXISTS idx_relation_to_name
|
||||
ON relation (to_name)
|
||||
""")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
"""Remove project_id from relation/observation and pg_trgm indexes."""
|
||||
connection = op.get_bind()
|
||||
dialect = connection.dialect.name
|
||||
|
||||
if dialect == "postgresql":
|
||||
# Drop Postgres-specific indexes
|
||||
op.execute("DROP INDEX IF EXISTS idx_relation_to_name")
|
||||
op.execute("DROP INDEX IF EXISTS idx_relation_unresolved")
|
||||
op.execute("DROP INDEX IF EXISTS idx_entity_permalink_trgm")
|
||||
op.execute("DROP INDEX IF EXISTS idx_entity_title_trgm")
|
||||
# Note: We don't drop the pg_trgm extension as other code may depend on it
|
||||
|
||||
# Drop project_id from observation
|
||||
op.drop_index("ix_observation_project_id", table_name="observation")
|
||||
op.drop_constraint("fk_observation_project_id", "observation", type_="foreignkey")
|
||||
op.drop_column("observation", "project_id")
|
||||
|
||||
# Drop project_id from relation
|
||||
op.drop_index("ix_relation_project_id", table_name="relation")
|
||||
op.drop_constraint("fk_relation_project_id", "relation", type_="foreignkey")
|
||||
op.drop_column("relation", "project_id")
|
||||
else:
|
||||
# SQLite requires batch operations
|
||||
op.drop_index("ix_observation_project_id", table_name="observation")
|
||||
with op.batch_alter_table("observation") as batch_op:
|
||||
batch_op.drop_constraint("fk_observation_project_id", type_="foreignkey")
|
||||
batch_op.drop_column("project_id")
|
||||
|
||||
op.drop_index("ix_relation_project_id", table_name="relation")
|
||||
with op.batch_alter_table("relation") as batch_op:
|
||||
batch_op.drop_constraint("fk_relation_project_id", type_="foreignkey")
|
||||
batch_op.drop_column("project_id")
|
||||
@@ -20,7 +20,17 @@ from basic_memory.api.routers import (
|
||||
search,
|
||||
prompt_router,
|
||||
)
|
||||
from basic_memory.config import ConfigManager
|
||||
from basic_memory.api.v2.routers import (
|
||||
knowledge_router as v2_knowledge,
|
||||
project_router as v2_project,
|
||||
memory_router as v2_memory,
|
||||
search_router as v2_search,
|
||||
resource_router as v2_resource,
|
||||
directory_router as v2_directory,
|
||||
prompt_router as v2_prompt,
|
||||
importer_router as v2_importer,
|
||||
)
|
||||
from basic_memory.config import ConfigManager, init_api_logging
|
||||
from basic_memory.services.initialization import initialize_file_sync, initialize_app
|
||||
|
||||
|
||||
@@ -28,6 +38,9 @@ from basic_memory.services.initialization import initialize_file_sync, initializ
|
||||
async def lifespan(app: FastAPI): # pragma: no cover
|
||||
"""Lifecycle manager for the FastAPI app. Not called in stdio mcp mode"""
|
||||
|
||||
# Initialize logging for API (stdout in cloud mode, file otherwise)
|
||||
init_api_logging()
|
||||
|
||||
app_config = ConfigManager().config
|
||||
logger.info("Starting Basic Memory API")
|
||||
|
||||
@@ -46,6 +59,7 @@ async def lifespan(app: FastAPI): # pragma: no cover
|
||||
app.state.sync_task = asyncio.create_task(initialize_file_sync(app_config))
|
||||
else:
|
||||
logger.info("Sync changes disabled. Skipping file sync service.")
|
||||
app.state.sync_task = None
|
||||
|
||||
# proceed with startup
|
||||
yield
|
||||
@@ -54,6 +68,10 @@ async def lifespan(app: FastAPI): # pragma: no cover
|
||||
if app.state.sync_task:
|
||||
logger.info("Stopping sync...")
|
||||
app.state.sync_task.cancel() # pyright: ignore
|
||||
try:
|
||||
await app.state.sync_task
|
||||
except asyncio.CancelledError:
|
||||
logger.info("Sync task cancelled successfully")
|
||||
|
||||
await db.shutdown_db()
|
||||
|
||||
@@ -66,8 +84,7 @@ app = FastAPI(
|
||||
lifespan=lifespan,
|
||||
)
|
||||
|
||||
|
||||
# Include routers
|
||||
# Include v1 routers
|
||||
app.include_router(knowledge.router, prefix="/{project}")
|
||||
app.include_router(memory.router, prefix="/{project}")
|
||||
app.include_router(resource.router, prefix="/{project}")
|
||||
@@ -77,12 +94,20 @@ app.include_router(directory_router.router, prefix="/{project}")
|
||||
app.include_router(prompt_router.router, prefix="/{project}")
|
||||
app.include_router(importer_router.router, prefix="/{project}")
|
||||
|
||||
# Project resource router works accross projects
|
||||
# Include v2 routers (ID-based paths)
|
||||
app.include_router(v2_knowledge, prefix="/v2/projects/{project_id}")
|
||||
app.include_router(v2_memory, prefix="/v2/projects/{project_id}")
|
||||
app.include_router(v2_search, prefix="/v2/projects/{project_id}")
|
||||
app.include_router(v2_resource, prefix="/v2/projects/{project_id}")
|
||||
app.include_router(v2_directory, prefix="/v2/projects/{project_id}")
|
||||
app.include_router(v2_prompt, prefix="/v2/projects/{project_id}")
|
||||
app.include_router(v2_importer, prefix="/v2/projects/{project_id}")
|
||||
app.include_router(v2_project, prefix="/v2")
|
||||
|
||||
# Project resource router works across projects
|
||||
app.include_router(project.project_resource_router)
|
||||
app.include_router(management.router)
|
||||
|
||||
# Auth routes are handled by FastMCP automatically when auth is enabled
|
||||
|
||||
|
||||
@app.exception_handler(Exception)
|
||||
async def exception_handler(request, exc): # pragma: no cover
|
||||
|
||||
@@ -1,4 +1,11 @@
|
||||
"""Router for knowledge graph operations."""
|
||||
"""Router for knowledge graph operations.
|
||||
|
||||
⚠️ DEPRECATED: This v1 API is deprecated and will be removed on June 30, 2026.
|
||||
Please migrate to /v2/{project}/knowledge endpoints which use entity IDs instead
|
||||
of path-based identifiers for improved performance and stability.
|
||||
|
||||
Migration guide: See docs/migration/v1-to-v2.md
|
||||
"""
|
||||
|
||||
from typing import Annotated
|
||||
|
||||
@@ -25,7 +32,11 @@ from basic_memory.schemas import (
|
||||
from basic_memory.schemas.request import EditEntityRequest, MoveEntityRequest
|
||||
from basic_memory.schemas.base import Permalink, Entity
|
||||
|
||||
router = APIRouter(prefix="/knowledge", tags=["knowledge"])
|
||||
router = APIRouter(
|
||||
prefix="/knowledge",
|
||||
tags=["knowledge"],
|
||||
deprecated=True, # Marks entire router as deprecated in OpenAPI docs
|
||||
)
|
||||
|
||||
|
||||
async def resolve_relations_background(sync_service, entity_id: int, entity_permalink: str) -> None:
|
||||
|
||||
@@ -50,6 +50,7 @@ async def get_project(
|
||||
) # pragma: no cover
|
||||
|
||||
return ProjectItem(
|
||||
id=found_project.id,
|
||||
name=found_project.name,
|
||||
path=normalize_project_path(found_project.path),
|
||||
is_default=found_project.is_default or False,
|
||||
@@ -80,9 +81,17 @@ async def update_project(
|
||||
raise HTTPException(status_code=400, detail="Path must be absolute")
|
||||
|
||||
# Get original project info for the response
|
||||
old_project = await project_service.get_project(name)
|
||||
if not old_project:
|
||||
raise HTTPException(
|
||||
status_code=400, detail=f"Project '{name}' not found in configuration"
|
||||
)
|
||||
|
||||
old_project_info = ProjectItem(
|
||||
name=name,
|
||||
path=project_service.projects.get(name, ""),
|
||||
id=old_project.id,
|
||||
name=old_project.name,
|
||||
path=old_project.path,
|
||||
is_default=old_project.is_default or False,
|
||||
)
|
||||
|
||||
if path:
|
||||
@@ -91,14 +100,21 @@ async def update_project(
|
||||
await project_service.update_project(name, is_active=is_active)
|
||||
|
||||
# Get updated project info
|
||||
updated_path = path if path else project_service.projects.get(name, "")
|
||||
updated_project = await project_service.get_project(name)
|
||||
if not updated_project:
|
||||
raise HTTPException(status_code=404, detail=f"Project '{name}' not found after update")
|
||||
|
||||
return ProjectStatusResponse(
|
||||
message=f"Project '{name}' updated successfully",
|
||||
status="success",
|
||||
default=(name == project_service.default_project),
|
||||
old_project=old_project_info,
|
||||
new_project=ProjectItem(name=name, path=updated_path),
|
||||
new_project=ProjectItem(
|
||||
id=updated_project.id,
|
||||
name=updated_project.name,
|
||||
path=updated_project.path,
|
||||
is_default=updated_project.is_default or False,
|
||||
),
|
||||
)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
@@ -186,6 +202,7 @@ async def list_projects(
|
||||
|
||||
project_items = [
|
||||
ProjectItem(
|
||||
id=project.id,
|
||||
name=project.name,
|
||||
path=normalize_project_path(project.path),
|
||||
is_default=project.is_default or False,
|
||||
@@ -232,6 +249,7 @@ async def add_project(
|
||||
status="success",
|
||||
default=existing_project.is_default or False,
|
||||
new_project=ProjectItem(
|
||||
id=existing_project.id,
|
||||
name=existing_project.name,
|
||||
path=existing_project.path,
|
||||
is_default=existing_project.is_default or False,
|
||||
@@ -250,12 +268,20 @@ async def add_project(
|
||||
project_data.name, project_data.path, set_default=project_data.set_default
|
||||
)
|
||||
|
||||
# Fetch the newly created project to get its ID
|
||||
new_project = await project_service.get_project(project_data.name)
|
||||
if not new_project:
|
||||
raise HTTPException(status_code=500, detail="Failed to retrieve newly created project")
|
||||
|
||||
return ProjectStatusResponse( # pyright: ignore [reportCallIssue]
|
||||
message=f"Project '{project_data.name}' added successfully",
|
||||
status="success",
|
||||
default=project_data.set_default,
|
||||
new_project=ProjectItem(
|
||||
name=project_data.name, path=project_data.path, is_default=project_data.set_default
|
||||
id=new_project.id,
|
||||
name=new_project.name,
|
||||
path=new_project.path,
|
||||
is_default=new_project.is_default or False,
|
||||
),
|
||||
)
|
||||
except ValueError as e: # pragma: no cover
|
||||
@@ -306,7 +332,12 @@ async def remove_project(
|
||||
message=f"Project '{name}' removed successfully",
|
||||
status="success",
|
||||
default=False,
|
||||
old_project=ProjectItem(name=old_project.name, path=old_project.path),
|
||||
old_project=ProjectItem(
|
||||
id=old_project.id,
|
||||
name=old_project.name,
|
||||
path=old_project.path,
|
||||
is_default=old_project.is_default or False,
|
||||
),
|
||||
new_project=None,
|
||||
)
|
||||
except ValueError as e: # pragma: no cover
|
||||
@@ -349,8 +380,14 @@ async def set_default_project(
|
||||
message=f"Project '{name}' set as default successfully",
|
||||
status="success",
|
||||
default=True,
|
||||
old_project=ProjectItem(name=default_name, path=default_project.path),
|
||||
old_project=ProjectItem(
|
||||
id=default_project.id,
|
||||
name=default_name,
|
||||
path=default_project.path,
|
||||
is_default=False,
|
||||
),
|
||||
new_project=ProjectItem(
|
||||
id=new_default_project.id,
|
||||
name=name,
|
||||
path=new_default_project.path,
|
||||
is_default=True,
|
||||
@@ -378,7 +415,12 @@ async def get_default_project(
|
||||
status_code=404, detail=f"Default Project: '{default_name}' does not exist"
|
||||
)
|
||||
|
||||
return ProjectItem(name=default_project.name, path=default_project.path, is_default=True)
|
||||
return ProjectItem(
|
||||
id=default_project.id,
|
||||
name=default_project.name,
|
||||
path=default_project.path,
|
||||
is_default=True,
|
||||
)
|
||||
|
||||
|
||||
# Synchronize projects between config and database
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Annotated
|
||||
from typing import Annotated, Union
|
||||
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Body
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Body, Response
|
||||
from fastapi.responses import FileResponse, JSONResponse
|
||||
from loguru import logger
|
||||
|
||||
@@ -25,6 +25,17 @@ from datetime import datetime
|
||||
router = APIRouter(prefix="/resource", tags=["resources"])
|
||||
|
||||
|
||||
def _mtime_to_datetime(entity: EntityModel) -> datetime:
|
||||
"""Convert entity mtime (file modification time) to datetime.
|
||||
|
||||
Returns the file's actual modification time, falling back to updated_at
|
||||
if mtime is not available.
|
||||
"""
|
||||
if entity.mtime:
|
||||
return datetime.fromtimestamp(entity.mtime).astimezone()
|
||||
return entity.updated_at
|
||||
|
||||
|
||||
def get_entity_ids(item: SearchIndexRow) -> set[int]:
|
||||
match item.type:
|
||||
case SearchItemType.ENTITY:
|
||||
@@ -39,7 +50,7 @@ def get_entity_ids(item: SearchIndexRow) -> set[int]:
|
||||
raise ValueError(f"Unexpected type: {item.type}")
|
||||
|
||||
|
||||
@router.get("/{identifier:path}")
|
||||
@router.get("/{identifier:path}", response_model=None)
|
||||
async def get_resource_content(
|
||||
config: ProjectConfigDep,
|
||||
link_resolver: LinkResolverDep,
|
||||
@@ -50,7 +61,7 @@ async def get_resource_content(
|
||||
identifier: str,
|
||||
page: int = 1,
|
||||
page_size: int = 10,
|
||||
) -> FileResponse:
|
||||
) -> Union[Response, FileResponse]:
|
||||
"""Get resource content by identifier: name or permalink."""
|
||||
logger.debug(f"Getting content for: {identifier}")
|
||||
|
||||
@@ -81,13 +92,16 @@ async def get_resource_content(
|
||||
# return single response
|
||||
if len(results) == 1:
|
||||
entity = results[0]
|
||||
file_path = Path(f"{config.home}/{entity.file_path}")
|
||||
if not file_path.exists():
|
||||
# Check file exists via file_service (for cloud compatibility)
|
||||
if not await file_service.exists(entity.file_path):
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"File not found: {file_path}",
|
||||
detail=f"File not found: {entity.file_path}",
|
||||
)
|
||||
return FileResponse(path=file_path)
|
||||
# Read content via file_service as bytes (works with both local and S3)
|
||||
content = await file_service.read_file_bytes(entity.file_path)
|
||||
content_type = file_service.content_type(entity.file_path)
|
||||
return Response(content=content, media_type=content_type)
|
||||
|
||||
# for multiple files, initialize a temporary file for writing the results
|
||||
with tempfile.NamedTemporaryFile(delete=False, mode="w", suffix=".md") as tmp_file:
|
||||
@@ -97,7 +111,7 @@ async def get_resource_content(
|
||||
# Read content for each entity
|
||||
content = await file_service.read_entity_content(result)
|
||||
memory_url = normalize_memory_url(result.permalink)
|
||||
modified_date = result.updated_at.isoformat()
|
||||
modified_date = _mtime_to_datetime(result).isoformat()
|
||||
checksum = result.checksum[:8] if result.checksum else ""
|
||||
|
||||
# Prepare the delimited content
|
||||
@@ -171,21 +185,17 @@ async def write_resource(
|
||||
else:
|
||||
content_str = str(content)
|
||||
|
||||
# Get full file path
|
||||
full_path = Path(f"{config.home}/{file_path}")
|
||||
|
||||
# Ensure parent directory exists
|
||||
full_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Write content to file
|
||||
checksum = await file_service.write_file(full_path, content_str)
|
||||
# Cloud compatibility: do not assume a local filesystem path structure.
|
||||
# Delegate directory creation + writes to the configured FileService (local or S3).
|
||||
await file_service.ensure_directory(Path(file_path).parent)
|
||||
checksum = await file_service.write_file(file_path, content_str)
|
||||
|
||||
# Get file info
|
||||
file_stats = file_service.file_stats(full_path)
|
||||
file_metadata = await file_service.get_file_metadata(file_path)
|
||||
|
||||
# Determine file details
|
||||
file_name = Path(file_path).name
|
||||
content_type = file_service.content_type(full_path)
|
||||
content_type = file_service.content_type(file_path)
|
||||
|
||||
entity_type = "canvas" if file_path.endswith(".canvas") else "file"
|
||||
|
||||
@@ -202,7 +212,7 @@ async def write_resource(
|
||||
"content_type": content_type,
|
||||
"file_path": file_path,
|
||||
"checksum": checksum,
|
||||
"updated_at": datetime.fromtimestamp(file_stats.st_mtime).astimezone(),
|
||||
"updated_at": file_metadata.modified_at,
|
||||
},
|
||||
)
|
||||
status_code = 200
|
||||
@@ -214,8 +224,8 @@ async def write_resource(
|
||||
content_type=content_type,
|
||||
file_path=file_path,
|
||||
checksum=checksum,
|
||||
created_at=datetime.fromtimestamp(file_stats.st_ctime).astimezone(),
|
||||
updated_at=datetime.fromtimestamp(file_stats.st_mtime).astimezone(),
|
||||
created_at=file_metadata.created_at,
|
||||
updated_at=file_metadata.modified_at,
|
||||
)
|
||||
entity = await entity_repository.add(entity)
|
||||
status_code = 201
|
||||
@@ -229,9 +239,9 @@ async def write_resource(
|
||||
content={
|
||||
"file_path": file_path,
|
||||
"checksum": checksum,
|
||||
"size": file_stats.st_size,
|
||||
"created_at": file_stats.st_ctime,
|
||||
"modified_at": file_stats.st_mtime,
|
||||
"size": file_metadata.size,
|
||||
"created_at": file_metadata.created_at.timestamp(),
|
||||
"modified_at": file_metadata.modified_at.timestamp(),
|
||||
},
|
||||
)
|
||||
except Exception as e: # pragma: no cover
|
||||
|
||||
@@ -24,11 +24,30 @@ async def to_graph_context(
|
||||
page: Optional[int] = None,
|
||||
page_size: Optional[int] = None,
|
||||
):
|
||||
# First pass: collect all entity IDs needed for relations
|
||||
entity_ids_needed: set[int] = set()
|
||||
for context_item in context_result.results:
|
||||
for item in (
|
||||
[context_item.primary_result] + context_item.observations + context_item.related_results
|
||||
):
|
||||
if item.type == SearchItemType.RELATION:
|
||||
if item.from_id: # pyright: ignore
|
||||
entity_ids_needed.add(item.from_id) # pyright: ignore
|
||||
if item.to_id:
|
||||
entity_ids_needed.add(item.to_id)
|
||||
|
||||
# Batch fetch all entities at once
|
||||
entity_lookup: dict[int, str] = {}
|
||||
if entity_ids_needed:
|
||||
entities = await entity_repository.find_by_ids(list(entity_ids_needed))
|
||||
entity_lookup = {e.id: e.title for e in entities}
|
||||
|
||||
# Helper function to convert items to summaries
|
||||
async def to_summary(item: SearchIndexRow | ContextResultRow):
|
||||
def to_summary(item: SearchIndexRow | ContextResultRow):
|
||||
match item.type:
|
||||
case SearchItemType.ENTITY:
|
||||
return EntitySummary(
|
||||
entity_id=item.id,
|
||||
title=item.title, # pyright: ignore
|
||||
permalink=item.permalink,
|
||||
content=item.content,
|
||||
@@ -37,6 +56,8 @@ async def to_graph_context(
|
||||
)
|
||||
case SearchItemType.OBSERVATION:
|
||||
return ObservationSummary(
|
||||
observation_id=item.id,
|
||||
entity_id=item.entity_id, # pyright: ignore
|
||||
title=item.title, # pyright: ignore
|
||||
file_path=item.file_path,
|
||||
category=item.category, # pyright: ignore
|
||||
@@ -45,15 +66,19 @@ async def to_graph_context(
|
||||
created_at=item.created_at,
|
||||
)
|
||||
case SearchItemType.RELATION:
|
||||
from_entity = await entity_repository.find_by_id(item.from_id) # pyright: ignore
|
||||
to_entity = await entity_repository.find_by_id(item.to_id) if item.to_id else None
|
||||
from_title = entity_lookup.get(item.from_id) if item.from_id else None # pyright: ignore
|
||||
to_title = entity_lookup.get(item.to_id) if item.to_id else None
|
||||
return RelationSummary(
|
||||
relation_id=item.id,
|
||||
entity_id=item.entity_id, # pyright: ignore
|
||||
title=item.title, # pyright: ignore
|
||||
file_path=item.file_path,
|
||||
permalink=item.permalink, # pyright: ignore
|
||||
relation_type=item.relation_type, # pyright: ignore
|
||||
from_entity=from_entity.title if from_entity else None,
|
||||
to_entity=to_entity.title if to_entity else None,
|
||||
from_entity=from_title,
|
||||
from_entity_id=item.from_id, # pyright: ignore
|
||||
to_entity=to_title,
|
||||
to_entity_id=item.to_id,
|
||||
created_at=item.created_at,
|
||||
)
|
||||
case _: # pragma: no cover
|
||||
@@ -63,23 +88,19 @@ async def to_graph_context(
|
||||
hierarchical_results = []
|
||||
for context_item in context_result.results:
|
||||
# Process primary result
|
||||
primary_result = await to_summary(context_item.primary_result)
|
||||
primary_result = to_summary(context_item.primary_result)
|
||||
|
||||
# Process observations
|
||||
observations = []
|
||||
for obs in context_item.observations:
|
||||
observations.append(await to_summary(obs))
|
||||
# Process observations (always ObservationSummary, validated by context_service)
|
||||
observations = [to_summary(obs) for obs in context_item.observations]
|
||||
|
||||
# Process related results
|
||||
related = []
|
||||
for rel in context_item.related_results:
|
||||
related.append(await to_summary(rel))
|
||||
related = [to_summary(rel) for rel in context_item.related_results]
|
||||
|
||||
# Add to hierarchical results
|
||||
hierarchical_results.append(
|
||||
ContextResult(
|
||||
primary_result=primary_result,
|
||||
observations=observations,
|
||||
observations=observations, # pyright: ignore[reportArgumentType]
|
||||
related_results=related,
|
||||
)
|
||||
)
|
||||
@@ -111,6 +132,21 @@ async def to_search_results(entity_service: EntityService, results: List[SearchI
|
||||
search_results = []
|
||||
for r in results:
|
||||
entities = await entity_service.get_entities_by_id([r.entity_id, r.from_id, r.to_id]) # pyright: ignore
|
||||
|
||||
# Determine which IDs to set based on type
|
||||
entity_id = None
|
||||
observation_id = None
|
||||
relation_id = None
|
||||
|
||||
if r.type == SearchItemType.ENTITY:
|
||||
entity_id = r.id
|
||||
elif r.type == SearchItemType.OBSERVATION:
|
||||
observation_id = r.id
|
||||
entity_id = r.entity_id # Parent entity
|
||||
elif r.type == SearchItemType.RELATION:
|
||||
relation_id = r.id
|
||||
entity_id = r.entity_id # Parent entity
|
||||
|
||||
search_results.append(
|
||||
SearchResult(
|
||||
title=r.title, # pyright: ignore
|
||||
@@ -121,6 +157,9 @@ async def to_search_results(entity_service: EntityService, results: List[SearchI
|
||||
content=r.content,
|
||||
file_path=r.file_path,
|
||||
metadata=r.metadata,
|
||||
entity_id=entity_id,
|
||||
observation_id=observation_id,
|
||||
relation_id=relation_id,
|
||||
category=r.category,
|
||||
from_entity=entities[0].permalink if entities else None,
|
||||
to_entity=entities[1].permalink if len(entities) > 1 else None,
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
"""API v2 module - ID-based entity references.
|
||||
|
||||
Version 2 of the Basic Memory API uses integer entity IDs as the primary
|
||||
identifier for improved performance and stability.
|
||||
|
||||
Key changes from v1:
|
||||
- Entity lookups use integer IDs instead of paths/permalinks
|
||||
- Direct database queries instead of cascading resolution
|
||||
- Stable references that don't change with file moves
|
||||
- Better caching support
|
||||
|
||||
All v2 routers are registered with the /v2 prefix.
|
||||
"""
|
||||
|
||||
from basic_memory.api.v2.routers import (
|
||||
knowledge_router,
|
||||
memory_router,
|
||||
project_router,
|
||||
resource_router,
|
||||
search_router,
|
||||
directory_router,
|
||||
prompt_router,
|
||||
importer_router,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"knowledge_router",
|
||||
"memory_router",
|
||||
"project_router",
|
||||
"resource_router",
|
||||
"search_router",
|
||||
"directory_router",
|
||||
"prompt_router",
|
||||
"importer_router",
|
||||
]
|
||||
@@ -0,0 +1,21 @@
|
||||
"""V2 API routers."""
|
||||
|
||||
from basic_memory.api.v2.routers.knowledge_router import router as knowledge_router
|
||||
from basic_memory.api.v2.routers.project_router import router as project_router
|
||||
from basic_memory.api.v2.routers.memory_router import router as memory_router
|
||||
from basic_memory.api.v2.routers.search_router import router as search_router
|
||||
from basic_memory.api.v2.routers.resource_router import router as resource_router
|
||||
from basic_memory.api.v2.routers.directory_router import router as directory_router
|
||||
from basic_memory.api.v2.routers.prompt_router import router as prompt_router
|
||||
from basic_memory.api.v2.routers.importer_router import router as importer_router
|
||||
|
||||
__all__ = [
|
||||
"knowledge_router",
|
||||
"project_router",
|
||||
"memory_router",
|
||||
"search_router",
|
||||
"resource_router",
|
||||
"directory_router",
|
||||
"prompt_router",
|
||||
"importer_router",
|
||||
]
|
||||
@@ -0,0 +1,93 @@
|
||||
"""V2 Directory Router - ID-based directory tree operations.
|
||||
|
||||
This router provides directory structure browsing for projects using
|
||||
integer project IDs instead of name-based identifiers.
|
||||
|
||||
Key improvements:
|
||||
- Direct project lookup via integer primary keys
|
||||
- Consistent with other v2 endpoints
|
||||
- Better performance through indexed queries
|
||||
"""
|
||||
|
||||
from typing import List, Optional
|
||||
|
||||
from fastapi import APIRouter, Query
|
||||
|
||||
from basic_memory.deps import DirectoryServiceV2Dep, ProjectIdPathDep
|
||||
from basic_memory.schemas.directory import DirectoryNode
|
||||
|
||||
router = APIRouter(prefix="/directory", tags=["directory-v2"])
|
||||
|
||||
|
||||
@router.get("/tree", response_model=DirectoryNode, response_model_exclude_none=True)
|
||||
async def get_directory_tree(
|
||||
directory_service: DirectoryServiceV2Dep,
|
||||
project_id: ProjectIdPathDep,
|
||||
):
|
||||
"""Get hierarchical directory structure from the knowledge base.
|
||||
|
||||
Args:
|
||||
directory_service: Service for directory operations
|
||||
project_id: Numeric project ID
|
||||
|
||||
Returns:
|
||||
DirectoryNode representing the root of the hierarchical tree structure
|
||||
"""
|
||||
# Get a hierarchical directory tree for the specific project
|
||||
tree = await directory_service.get_directory_tree()
|
||||
|
||||
# Return the hierarchical tree
|
||||
return tree
|
||||
|
||||
|
||||
@router.get("/structure", response_model=DirectoryNode, response_model_exclude_none=True)
|
||||
async def get_directory_structure(
|
||||
directory_service: DirectoryServiceV2Dep,
|
||||
project_id: ProjectIdPathDep,
|
||||
):
|
||||
"""Get folder structure for navigation (no files).
|
||||
|
||||
Optimized endpoint for folder tree navigation. Returns only directory nodes
|
||||
without file metadata. For full tree with files, use /directory/tree.
|
||||
|
||||
Args:
|
||||
directory_service: Service for directory operations
|
||||
project_id: Numeric project ID
|
||||
|
||||
Returns:
|
||||
DirectoryNode tree containing only folders (type="directory")
|
||||
"""
|
||||
structure = await directory_service.get_directory_structure()
|
||||
return structure
|
||||
|
||||
|
||||
@router.get("/list", response_model=List[DirectoryNode], response_model_exclude_none=True)
|
||||
async def list_directory(
|
||||
directory_service: DirectoryServiceV2Dep,
|
||||
project_id: ProjectIdPathDep,
|
||||
dir_name: str = Query("/", description="Directory path to list"),
|
||||
depth: int = Query(1, ge=1, le=10, description="Recursion depth (1-10)"),
|
||||
file_name_glob: Optional[str] = Query(
|
||||
None, description="Glob pattern for filtering file names"
|
||||
),
|
||||
):
|
||||
"""List directory contents with filtering and depth control.
|
||||
|
||||
Args:
|
||||
directory_service: Service for directory operations
|
||||
project_id: Numeric project ID
|
||||
dir_name: Directory path to list (default: root "/")
|
||||
depth: Recursion depth (1-10, default: 1 for immediate children only)
|
||||
file_name_glob: Optional glob pattern for filtering file names (e.g., "*.md", "*meeting*")
|
||||
|
||||
Returns:
|
||||
List of DirectoryNode objects matching the criteria
|
||||
"""
|
||||
# Get directory listing with filtering
|
||||
nodes = await directory_service.list_directory(
|
||||
dir_name=dir_name,
|
||||
depth=depth,
|
||||
file_name_glob=file_name_glob,
|
||||
)
|
||||
|
||||
return nodes
|
||||
@@ -0,0 +1,182 @@
|
||||
"""V2 Import Router - ID-based data import operations.
|
||||
|
||||
This router uses v2 dependencies for consistent project ID handling.
|
||||
Import endpoints use project_id in the path for consistency with other v2 endpoints.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
|
||||
from fastapi import APIRouter, Form, HTTPException, UploadFile, status
|
||||
|
||||
from basic_memory.deps import (
|
||||
ChatGPTImporterV2Dep,
|
||||
ClaudeConversationsImporterV2Dep,
|
||||
ClaudeProjectsImporterV2Dep,
|
||||
MemoryJsonImporterV2Dep,
|
||||
ProjectIdPathDep,
|
||||
)
|
||||
from basic_memory.importers import Importer
|
||||
from basic_memory.schemas.importer import (
|
||||
ChatImportResult,
|
||||
EntityImportResult,
|
||||
ProjectImportResult,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/import", tags=["import-v2"])
|
||||
|
||||
|
||||
@router.post("/chatgpt", response_model=ChatImportResult)
|
||||
async def import_chatgpt(
|
||||
project_id: ProjectIdPathDep,
|
||||
importer: ChatGPTImporterV2Dep,
|
||||
file: UploadFile,
|
||||
folder: str = Form("conversations"),
|
||||
) -> ChatImportResult:
|
||||
"""Import conversations from ChatGPT JSON export.
|
||||
|
||||
Args:
|
||||
project_id: Validated numeric project ID from URL path
|
||||
file: The ChatGPT conversations.json file.
|
||||
folder: The folder to place the files in.
|
||||
importer: ChatGPT importer instance.
|
||||
|
||||
Returns:
|
||||
ChatImportResult with import statistics.
|
||||
|
||||
Raises:
|
||||
HTTPException: If import fails.
|
||||
"""
|
||||
logger.info(f"V2 Importing ChatGPT conversations for project {project_id}")
|
||||
return await import_file(importer, file, folder)
|
||||
|
||||
|
||||
@router.post("/claude/conversations", response_model=ChatImportResult)
|
||||
async def import_claude_conversations(
|
||||
project_id: ProjectIdPathDep,
|
||||
importer: ClaudeConversationsImporterV2Dep,
|
||||
file: UploadFile,
|
||||
folder: str = Form("conversations"),
|
||||
) -> ChatImportResult:
|
||||
"""Import conversations from Claude conversations.json export.
|
||||
|
||||
Args:
|
||||
project_id: Validated numeric project ID from URL path
|
||||
file: The Claude conversations.json file.
|
||||
folder: The folder to place the files in.
|
||||
importer: Claude conversations importer instance.
|
||||
|
||||
Returns:
|
||||
ChatImportResult with import statistics.
|
||||
|
||||
Raises:
|
||||
HTTPException: If import fails.
|
||||
"""
|
||||
logger.info(f"V2 Importing Claude conversations for project {project_id}")
|
||||
return await import_file(importer, file, folder)
|
||||
|
||||
|
||||
@router.post("/claude/projects", response_model=ProjectImportResult)
|
||||
async def import_claude_projects(
|
||||
project_id: ProjectIdPathDep,
|
||||
importer: ClaudeProjectsImporterV2Dep,
|
||||
file: UploadFile,
|
||||
folder: str = Form("projects"),
|
||||
) -> ProjectImportResult:
|
||||
"""Import projects from Claude projects.json export.
|
||||
|
||||
Args:
|
||||
project_id: Validated numeric project ID from URL path
|
||||
file: The Claude projects.json file.
|
||||
folder: The base folder to place the files in.
|
||||
importer: Claude projects importer instance.
|
||||
|
||||
Returns:
|
||||
ProjectImportResult with import statistics.
|
||||
|
||||
Raises:
|
||||
HTTPException: If import fails.
|
||||
"""
|
||||
logger.info(f"V2 Importing Claude projects for project {project_id}")
|
||||
return await import_file(importer, file, folder)
|
||||
|
||||
|
||||
@router.post("/memory-json", response_model=EntityImportResult)
|
||||
async def import_memory_json(
|
||||
project_id: ProjectIdPathDep,
|
||||
importer: MemoryJsonImporterV2Dep,
|
||||
file: UploadFile,
|
||||
folder: str = Form("conversations"),
|
||||
) -> EntityImportResult:
|
||||
"""Import entities and relations from a memory.json file.
|
||||
|
||||
Args:
|
||||
project_id: Validated numeric project ID from URL path
|
||||
file: The memory.json file.
|
||||
folder: Optional destination folder within the project.
|
||||
importer: Memory JSON importer instance.
|
||||
|
||||
Returns:
|
||||
EntityImportResult with import statistics.
|
||||
|
||||
Raises:
|
||||
HTTPException: If import fails.
|
||||
"""
|
||||
logger.info(f"V2 Importing memory.json for project {project_id}")
|
||||
try:
|
||||
file_data = []
|
||||
file_bytes = await file.read()
|
||||
file_str = file_bytes.decode("utf-8")
|
||||
for line in file_str.splitlines():
|
||||
json_data = json.loads(line)
|
||||
file_data.append(json_data)
|
||||
|
||||
result = await importer.import_data(file_data, folder)
|
||||
if not result.success: # pragma: no cover
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
|
||||
detail=result.error_message or "Import failed",
|
||||
)
|
||||
except Exception as e:
|
||||
logger.exception("V2 Import failed")
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
|
||||
detail=f"Import failed: {str(e)}",
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
async def import_file(importer: Importer, file: UploadFile, destination_folder: str):
|
||||
"""Helper function to import a file using an importer instance.
|
||||
|
||||
Args:
|
||||
importer: The importer instance to use
|
||||
file: The file to import
|
||||
destination_folder: Destination folder for imported content
|
||||
|
||||
Returns:
|
||||
Import result from the importer
|
||||
|
||||
Raises:
|
||||
HTTPException: If import fails
|
||||
"""
|
||||
try:
|
||||
# Process file
|
||||
json_data = json.load(file.file)
|
||||
result = await importer.import_data(json_data, destination_folder)
|
||||
if not result.success: # pragma: no cover
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
|
||||
detail=result.error_message or "Import failed",
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
except Exception as e:
|
||||
logger.exception("V2 Import failed")
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
|
||||
detail=f"Import failed: {str(e)}",
|
||||
)
|
||||
@@ -0,0 +1,415 @@
|
||||
"""V2 Knowledge Router - ID-based entity operations.
|
||||
|
||||
This router provides ID-based CRUD operations for entities, replacing the
|
||||
path-based identifiers used in v1 with direct integer ID lookups.
|
||||
|
||||
Key improvements:
|
||||
- Direct database lookups via integer primary keys
|
||||
- Stable references that don't change with file moves
|
||||
- Better performance through indexed queries
|
||||
- Simplified caching strategies
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Response
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory.deps import (
|
||||
EntityServiceV2Dep,
|
||||
SearchServiceV2Dep,
|
||||
LinkResolverV2Dep,
|
||||
ProjectConfigV2Dep,
|
||||
AppConfigDep,
|
||||
SyncServiceV2Dep,
|
||||
EntityRepositoryV2Dep,
|
||||
ProjectIdPathDep,
|
||||
)
|
||||
from basic_memory.schemas import DeleteEntitiesResponse
|
||||
from basic_memory.schemas.base import Entity
|
||||
from basic_memory.schemas.request import EditEntityRequest
|
||||
from basic_memory.schemas.v2 import (
|
||||
EntityResolveRequest,
|
||||
EntityResolveResponse,
|
||||
EntityResponseV2,
|
||||
MoveEntityRequestV2,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/knowledge", tags=["knowledge-v2"])
|
||||
|
||||
|
||||
async def resolve_relations_background(sync_service, entity_id: int, entity_permalink: str) -> None:
|
||||
"""Background task to resolve relations for a specific entity.
|
||||
|
||||
This runs asynchronously after the API response is sent, preventing
|
||||
long delays when creating entities with many relations.
|
||||
"""
|
||||
try:
|
||||
# Only resolve relations for the newly created entity
|
||||
await sync_service.resolve_relations(entity_id=entity_id)
|
||||
logger.debug(
|
||||
f"Background: Resolved relations for entity {entity_permalink} (id={entity_id})"
|
||||
)
|
||||
except Exception as e:
|
||||
# Log but don't fail - this is a background task
|
||||
logger.warning(
|
||||
f"Background: Failed to resolve relations for entity {entity_permalink}: {e}"
|
||||
)
|
||||
|
||||
|
||||
## Resolution endpoint
|
||||
|
||||
|
||||
@router.post("/resolve", response_model=EntityResolveResponse)
|
||||
async def resolve_identifier(
|
||||
project_id: ProjectIdPathDep,
|
||||
data: EntityResolveRequest,
|
||||
link_resolver: LinkResolverV2Dep,
|
||||
) -> EntityResolveResponse:
|
||||
"""Resolve a string identifier (permalink, title, or path) to an entity ID.
|
||||
|
||||
This endpoint provides a bridge between v1-style identifiers and v2 entity IDs.
|
||||
Use this to convert existing references to the new ID-based format.
|
||||
|
||||
Args:
|
||||
data: Request containing the identifier to resolve
|
||||
|
||||
Returns:
|
||||
Entity ID and metadata about how it was resolved
|
||||
|
||||
Raises:
|
||||
HTTPException: 404 if identifier cannot be resolved
|
||||
|
||||
Example:
|
||||
POST /v2/{project}/knowledge/resolve
|
||||
{"identifier": "specs/search"}
|
||||
|
||||
Returns:
|
||||
{
|
||||
"entity_id": 123,
|
||||
"permalink": "specs/search",
|
||||
"file_path": "specs/search.md",
|
||||
"title": "Search Specification",
|
||||
"resolution_method": "permalink"
|
||||
}
|
||||
"""
|
||||
logger.info(f"API v2 request: resolve_identifier for '{data.identifier}'")
|
||||
|
||||
# Try to resolve the identifier
|
||||
entity = await link_resolver.resolve_link(data.identifier)
|
||||
if not entity:
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"Could not resolve identifier: '{data.identifier}'"
|
||||
)
|
||||
|
||||
# Determine resolution method
|
||||
resolution_method = "search" # default
|
||||
if data.identifier.isdigit():
|
||||
resolution_method = "id"
|
||||
elif entity.permalink == data.identifier:
|
||||
resolution_method = "permalink"
|
||||
elif entity.title == data.identifier:
|
||||
resolution_method = "title"
|
||||
elif entity.file_path == data.identifier:
|
||||
resolution_method = "path"
|
||||
|
||||
result = EntityResolveResponse(
|
||||
entity_id=entity.id,
|
||||
permalink=entity.permalink,
|
||||
file_path=entity.file_path,
|
||||
title=entity.title,
|
||||
resolution_method=resolution_method,
|
||||
)
|
||||
|
||||
logger.info(
|
||||
f"API v2 response: resolved '{data.identifier}' to entity_id={result.entity_id} via {resolution_method}"
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
## Read endpoints
|
||||
|
||||
|
||||
@router.get("/entities/{entity_id}", response_model=EntityResponseV2)
|
||||
async def get_entity_by_id(
|
||||
project_id: ProjectIdPathDep,
|
||||
entity_id: int,
|
||||
entity_repository: EntityRepositoryV2Dep,
|
||||
) -> EntityResponseV2:
|
||||
"""Get an entity by its numeric ID.
|
||||
|
||||
This is the primary entity retrieval method in v2, using direct database
|
||||
lookups for maximum performance.
|
||||
|
||||
Args:
|
||||
entity_id: Numeric entity ID
|
||||
|
||||
Returns:
|
||||
Complete entity with observations and relations
|
||||
|
||||
Raises:
|
||||
HTTPException: 404 if entity not found
|
||||
"""
|
||||
logger.info(f"API v2 request: get_entity_by_id entity_id={entity_id}")
|
||||
|
||||
entity = await entity_repository.get_by_id(entity_id)
|
||||
if not entity:
|
||||
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
|
||||
|
||||
result = EntityResponseV2.model_validate(entity)
|
||||
logger.info(f"API v2 response: entity_id={entity_id}, title='{result.title}'")
|
||||
|
||||
return result
|
||||
|
||||
|
||||
## Create endpoints
|
||||
|
||||
|
||||
@router.post("/entities", response_model=EntityResponseV2)
|
||||
async def create_entity(
|
||||
project_id: ProjectIdPathDep,
|
||||
data: Entity,
|
||||
background_tasks: BackgroundTasks,
|
||||
entity_service: EntityServiceV2Dep,
|
||||
search_service: SearchServiceV2Dep,
|
||||
) -> EntityResponseV2:
|
||||
"""Create a new entity.
|
||||
|
||||
Args:
|
||||
data: Entity data to create
|
||||
|
||||
Returns:
|
||||
Created entity with generated ID
|
||||
"""
|
||||
logger.info(
|
||||
"API v2 request", endpoint="create_entity", entity_type=data.entity_type, title=data.title
|
||||
)
|
||||
|
||||
entity = await entity_service.create_entity(data)
|
||||
|
||||
# reindex
|
||||
await search_service.index_entity(entity, background_tasks=background_tasks)
|
||||
result = EntityResponseV2.model_validate(entity)
|
||||
|
||||
logger.info(
|
||||
f"API v2 response: endpoint='create_entity' id={entity.id}, title={result.title}, permalink={result.permalink}, status_code=201"
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
## Update endpoints
|
||||
|
||||
|
||||
@router.put("/entities/{entity_id}", response_model=EntityResponseV2)
|
||||
async def update_entity_by_id(
|
||||
project_id: ProjectIdPathDep,
|
||||
entity_id: int,
|
||||
data: Entity,
|
||||
response: Response,
|
||||
background_tasks: BackgroundTasks,
|
||||
entity_service: EntityServiceV2Dep,
|
||||
search_service: SearchServiceV2Dep,
|
||||
sync_service: SyncServiceV2Dep,
|
||||
entity_repository: EntityRepositoryV2Dep,
|
||||
) -> EntityResponseV2:
|
||||
"""Update an entity by ID.
|
||||
|
||||
If the entity doesn't exist, it will be created (upsert behavior).
|
||||
|
||||
Args:
|
||||
entity_id: Numeric entity ID
|
||||
data: Updated entity data
|
||||
|
||||
Returns:
|
||||
Updated entity
|
||||
"""
|
||||
logger.info(f"API v2 request: update_entity_by_id entity_id={entity_id}")
|
||||
|
||||
# Check if entity exists
|
||||
existing = await entity_repository.get_by_id(entity_id)
|
||||
created = existing is None
|
||||
|
||||
# Perform update or create
|
||||
entity, _ = await entity_service.create_or_update_entity(data)
|
||||
response.status_code = 201 if created else 200
|
||||
|
||||
# reindex
|
||||
await search_service.index_entity(entity, background_tasks=background_tasks)
|
||||
|
||||
# Schedule relation resolution for new entities
|
||||
if created:
|
||||
background_tasks.add_task(
|
||||
resolve_relations_background, sync_service, entity.id, entity.permalink or ""
|
||||
)
|
||||
|
||||
result = EntityResponseV2.model_validate(entity)
|
||||
|
||||
logger.info(
|
||||
f"API v2 response: entity_id={entity_id}, created={created}, status_code={response.status_code}"
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
@router.patch("/entities/{entity_id}", response_model=EntityResponseV2)
|
||||
async def edit_entity_by_id(
|
||||
project_id: ProjectIdPathDep,
|
||||
entity_id: int,
|
||||
data: EditEntityRequest,
|
||||
background_tasks: BackgroundTasks,
|
||||
entity_service: EntityServiceV2Dep,
|
||||
search_service: SearchServiceV2Dep,
|
||||
entity_repository: EntityRepositoryV2Dep,
|
||||
) -> EntityResponseV2:
|
||||
"""Edit an existing entity by ID using operations like append, prepend, etc.
|
||||
|
||||
Args:
|
||||
entity_id: Numeric entity ID
|
||||
data: Edit operation details
|
||||
|
||||
Returns:
|
||||
Updated entity
|
||||
|
||||
Raises:
|
||||
HTTPException: 404 if entity not found, 400 if edit fails
|
||||
"""
|
||||
logger.info(
|
||||
f"API v2 request: edit_entity_by_id entity_id={entity_id}, operation='{data.operation}'"
|
||||
)
|
||||
|
||||
# Verify entity exists
|
||||
entity = await entity_repository.get_by_id(entity_id)
|
||||
if not entity:
|
||||
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
|
||||
|
||||
try:
|
||||
# Edit using the entity's permalink or path
|
||||
identifier = entity.permalink or entity.file_path
|
||||
updated_entity = await entity_service.edit_entity(
|
||||
identifier=identifier,
|
||||
operation=data.operation,
|
||||
content=data.content,
|
||||
section=data.section,
|
||||
find_text=data.find_text,
|
||||
expected_replacements=data.expected_replacements,
|
||||
)
|
||||
|
||||
# Reindex
|
||||
await search_service.index_entity(updated_entity, background_tasks=background_tasks)
|
||||
|
||||
result = EntityResponseV2.model_validate(updated_entity)
|
||||
|
||||
logger.info(
|
||||
f"API v2 response: entity_id={entity_id}, operation='{data.operation}', status_code=200"
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error editing entity {entity_id}: {e}")
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
|
||||
|
||||
## Delete endpoints
|
||||
|
||||
|
||||
@router.delete("/entities/{entity_id}", response_model=DeleteEntitiesResponse)
|
||||
async def delete_entity_by_id(
|
||||
project_id: ProjectIdPathDep,
|
||||
entity_id: int,
|
||||
background_tasks: BackgroundTasks,
|
||||
entity_service: EntityServiceV2Dep,
|
||||
entity_repository: EntityRepositoryV2Dep,
|
||||
search_service=Depends(lambda: None), # Optional for now
|
||||
) -> DeleteEntitiesResponse:
|
||||
"""Delete an entity by ID.
|
||||
|
||||
Args:
|
||||
entity_id: Numeric entity ID
|
||||
|
||||
Returns:
|
||||
Deletion status
|
||||
|
||||
Note: Returns deleted=False if entity doesn't exist (idempotent)
|
||||
"""
|
||||
logger.info(f"API v2 request: delete_entity_by_id entity_id={entity_id}")
|
||||
|
||||
entity = await entity_repository.get_by_id(entity_id)
|
||||
if entity is None:
|
||||
logger.info(f"API v2 response: entity_id={entity_id} not found, deleted=False")
|
||||
return DeleteEntitiesResponse(deleted=False)
|
||||
|
||||
# Delete the entity
|
||||
deleted = await entity_service.delete_entity(entity_id)
|
||||
|
||||
# Remove from search index if search service available
|
||||
if search_service:
|
||||
background_tasks.add_task(search_service.handle_delete, entity)
|
||||
|
||||
logger.info(f"API v2 response: entity_id={entity_id}, deleted={deleted}")
|
||||
|
||||
return DeleteEntitiesResponse(deleted=deleted)
|
||||
|
||||
|
||||
## Move endpoint
|
||||
|
||||
|
||||
@router.put("/entities/{entity_id}/move", response_model=EntityResponseV2)
|
||||
async def move_entity(
|
||||
project_id: ProjectIdPathDep,
|
||||
entity_id: int,
|
||||
data: MoveEntityRequestV2,
|
||||
background_tasks: BackgroundTasks,
|
||||
entity_service: EntityServiceV2Dep,
|
||||
entity_repository: EntityRepositoryV2Dep,
|
||||
project_config: ProjectConfigV2Dep,
|
||||
app_config: AppConfigDep,
|
||||
search_service: SearchServiceV2Dep,
|
||||
) -> EntityResponseV2:
|
||||
"""Move an entity to a new file location.
|
||||
|
||||
V2 API uses entity ID in the URL path for stable references.
|
||||
The entity ID will remain stable after the move.
|
||||
|
||||
Args:
|
||||
project_id: Project ID from URL path
|
||||
entity_id: Entity ID from URL path (primary identifier)
|
||||
data: Move request with destination path only
|
||||
|
||||
Returns:
|
||||
Updated entity with new file path
|
||||
"""
|
||||
logger.info(
|
||||
f"API v2 request: move_entity entity_id={entity_id}, destination='{data.destination_path}'"
|
||||
)
|
||||
|
||||
try:
|
||||
# First, get the entity by ID to verify it exists
|
||||
entity = await entity_repository.find_by_id(entity_id)
|
||||
if not entity:
|
||||
raise HTTPException(status_code=404, detail=f"Entity not found: {entity_id}")
|
||||
|
||||
# Move the entity using its current file path as identifier
|
||||
moved_entity = await entity_service.move_entity(
|
||||
identifier=entity.file_path, # Use file path for resolution
|
||||
destination_path=data.destination_path,
|
||||
project_config=project_config,
|
||||
app_config=app_config,
|
||||
)
|
||||
|
||||
# Reindex at new location
|
||||
reindexed_entity = await entity_service.link_resolver.resolve_link(data.destination_path)
|
||||
if reindexed_entity:
|
||||
await search_service.index_entity(reindexed_entity, background_tasks=background_tasks)
|
||||
|
||||
result = EntityResponseV2.model_validate(moved_entity)
|
||||
|
||||
logger.info(
|
||||
f"API v2 response: moved entity_id={moved_entity.id} to '{data.destination_path}'"
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Error moving entity: {e}")
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
@@ -0,0 +1,130 @@
|
||||
"""V2 routes for memory:// URI operations.
|
||||
|
||||
This router uses integer project IDs for stable, efficient routing.
|
||||
V1 uses string-based project names which are less efficient and less stable.
|
||||
"""
|
||||
|
||||
from typing import Annotated, Optional
|
||||
|
||||
from fastapi import APIRouter, Query
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory.deps import ContextServiceV2Dep, EntityRepositoryV2Dep, ProjectIdPathDep
|
||||
from basic_memory.schemas.base import TimeFrame, parse_timeframe
|
||||
from basic_memory.schemas.memory import (
|
||||
GraphContext,
|
||||
normalize_memory_url,
|
||||
)
|
||||
from basic_memory.schemas.search import SearchItemType
|
||||
from basic_memory.api.routers.utils import to_graph_context
|
||||
|
||||
# Note: No prefix here - it's added during registration as /v2/{project_id}/memory
|
||||
router = APIRouter(tags=["memory"])
|
||||
|
||||
|
||||
@router.get("/memory/recent", response_model=GraphContext)
|
||||
async def recent(
|
||||
project_id: ProjectIdPathDep,
|
||||
context_service: ContextServiceV2Dep,
|
||||
entity_repository: EntityRepositoryV2Dep,
|
||||
type: Annotated[list[SearchItemType] | None, Query()] = None,
|
||||
depth: int = 1,
|
||||
timeframe: TimeFrame = "7d",
|
||||
page: int = 1,
|
||||
page_size: int = 10,
|
||||
max_related: int = 10,
|
||||
) -> GraphContext:
|
||||
"""Get recent activity context for a project.
|
||||
|
||||
Args:
|
||||
project_id: Validated numeric project ID from URL path
|
||||
context_service: Context service scoped to project
|
||||
entity_repository: Entity repository scoped to project
|
||||
type: Types of items to include (entities, relations, observations)
|
||||
depth: How many levels of related entities to include
|
||||
timeframe: Time window for recent activity (e.g., "7d", "1 week")
|
||||
page: Page number for pagination
|
||||
page_size: Number of items per page
|
||||
max_related: Maximum related entities to include per item
|
||||
|
||||
Returns:
|
||||
GraphContext with recent activity and related entities
|
||||
"""
|
||||
# return all types by default
|
||||
types = (
|
||||
[SearchItemType.ENTITY, SearchItemType.RELATION, SearchItemType.OBSERVATION]
|
||||
if not type
|
||||
else type
|
||||
)
|
||||
|
||||
logger.debug(
|
||||
f"V2 Getting recent context for project {project_id}: `{types}` depth: `{depth}` timeframe: `{timeframe}` page: `{page}` page_size: `{page_size}` max_related: `{max_related}`"
|
||||
)
|
||||
# Parse timeframe
|
||||
since = parse_timeframe(timeframe)
|
||||
limit = page_size
|
||||
offset = (page - 1) * page_size
|
||||
|
||||
# Build context
|
||||
context = await context_service.build_context(
|
||||
types=types, depth=depth, since=since, limit=limit, offset=offset, max_related=max_related
|
||||
)
|
||||
recent_context = await to_graph_context(
|
||||
context, entity_repository=entity_repository, page=page, page_size=page_size
|
||||
)
|
||||
logger.debug(f"V2 Recent context: {recent_context.model_dump_json()}")
|
||||
return recent_context
|
||||
|
||||
|
||||
# get_memory_context needs to be declared last so other paths can match
|
||||
|
||||
|
||||
@router.get("/memory/{uri:path}", response_model=GraphContext)
|
||||
async def get_memory_context(
|
||||
project_id: ProjectIdPathDep,
|
||||
context_service: ContextServiceV2Dep,
|
||||
entity_repository: EntityRepositoryV2Dep,
|
||||
uri: str,
|
||||
depth: int = 1,
|
||||
timeframe: Optional[TimeFrame] = None,
|
||||
page: int = 1,
|
||||
page_size: int = 10,
|
||||
max_related: int = 10,
|
||||
) -> GraphContext:
|
||||
"""Get rich context from memory:// URI.
|
||||
|
||||
V2 supports both legacy path-based URIs and new ID-based URIs:
|
||||
- Legacy: memory://path/to/note
|
||||
- ID-based: memory://id/123 or memory://123
|
||||
|
||||
Args:
|
||||
project_id: Validated numeric project ID from URL path
|
||||
context_service: Context service scoped to project
|
||||
entity_repository: Entity repository scoped to project
|
||||
uri: Memory URI path (e.g., "id/123", "123", or "path/to/note")
|
||||
depth: How many levels of related entities to include
|
||||
timeframe: Optional time window for filtering related content
|
||||
page: Page number for pagination
|
||||
page_size: Number of items per page
|
||||
max_related: Maximum related entities to include
|
||||
|
||||
Returns:
|
||||
GraphContext with the entity and its related context
|
||||
"""
|
||||
logger.debug(
|
||||
f"V2 Getting context for project {project_id}, URI: `{uri}` depth: `{depth}` timeframe: `{timeframe}` page: `{page}` page_size: `{page_size}` max_related: `{max_related}`"
|
||||
)
|
||||
memory_url = normalize_memory_url(uri)
|
||||
|
||||
# Parse timeframe
|
||||
since = parse_timeframe(timeframe) if timeframe else None
|
||||
limit = page_size
|
||||
offset = (page - 1) * page_size
|
||||
|
||||
# Build context
|
||||
context = await context_service.build_context(
|
||||
memory_url, depth=depth, since=since, limit=limit, offset=offset, max_related=max_related
|
||||
)
|
||||
return await to_graph_context(
|
||||
context, entity_repository=entity_repository, page=page, page_size=page_size
|
||||
)
|
||||
@@ -0,0 +1,264 @@
|
||||
"""V2 Project Router - ID-based project management operations.
|
||||
|
||||
This router provides ID-based CRUD operations for projects, replacing the
|
||||
name-based identifiers used in v1 with direct integer ID lookups.
|
||||
|
||||
Key improvements:
|
||||
- Direct database lookups via integer primary keys
|
||||
- Stable references that don't change with project renames
|
||||
- Better performance through indexed queries
|
||||
- Consistent with v2 entity operations
|
||||
"""
|
||||
|
||||
import os
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Body, Query
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory.deps import (
|
||||
ProjectServiceDep,
|
||||
ProjectRepositoryDep,
|
||||
ProjectIdPathDep,
|
||||
)
|
||||
from basic_memory.schemas.project_info import (
|
||||
ProjectItem,
|
||||
ProjectStatusResponse,
|
||||
)
|
||||
from basic_memory.utils import normalize_project_path
|
||||
|
||||
router = APIRouter(prefix="/projects", tags=["project_management-v2"])
|
||||
|
||||
|
||||
@router.get("/{project_id}", response_model=ProjectItem)
|
||||
async def get_project_by_id(
|
||||
project_id: ProjectIdPathDep,
|
||||
project_repository: ProjectRepositoryDep,
|
||||
) -> ProjectItem:
|
||||
"""Get project by its numeric ID.
|
||||
|
||||
This is the primary project retrieval method in v2, using direct database
|
||||
lookups for maximum performance.
|
||||
|
||||
Args:
|
||||
project_id: Numeric project ID
|
||||
|
||||
Returns:
|
||||
Project information
|
||||
|
||||
Raises:
|
||||
HTTPException: 404 if project not found
|
||||
|
||||
Example:
|
||||
GET /v2/projects/3
|
||||
"""
|
||||
logger.info(f"API v2 request: get_project_by_id for project_id={project_id}")
|
||||
|
||||
project = await project_repository.get_by_id(project_id)
|
||||
if not project:
|
||||
raise HTTPException(status_code=404, detail=f"Project with ID {project_id} not found")
|
||||
|
||||
return ProjectItem(
|
||||
id=project.id,
|
||||
name=project.name,
|
||||
path=normalize_project_path(project.path),
|
||||
is_default=project.is_default or False,
|
||||
)
|
||||
|
||||
|
||||
@router.patch("/{project_id}", response_model=ProjectStatusResponse)
|
||||
async def update_project_by_id(
|
||||
project_id: ProjectIdPathDep,
|
||||
project_service: ProjectServiceDep,
|
||||
project_repository: ProjectRepositoryDep,
|
||||
path: Optional[str] = Body(None, description="New absolute path for the project"),
|
||||
is_active: Optional[bool] = Body(None, description="Status of the project (active/inactive)"),
|
||||
) -> ProjectStatusResponse:
|
||||
"""Update a project's information by ID.
|
||||
|
||||
Args:
|
||||
project_id: Numeric project ID
|
||||
path: Optional new absolute path for the project
|
||||
is_active: Optional status update for the project
|
||||
|
||||
Returns:
|
||||
Response confirming the project was updated
|
||||
|
||||
Raises:
|
||||
HTTPException: 400 if validation fails, 404 if project not found
|
||||
|
||||
Example:
|
||||
PATCH /v2/projects/3
|
||||
{"path": "/new/path"}
|
||||
"""
|
||||
logger.info(f"API v2 request: update_project_by_id for project_id={project_id}")
|
||||
|
||||
try:
|
||||
# Validate that path is absolute if provided
|
||||
if path and not os.path.isabs(path):
|
||||
raise HTTPException(status_code=400, detail="Path must be absolute")
|
||||
|
||||
# Get original project info for the response
|
||||
old_project = await project_repository.get_by_id(project_id)
|
||||
if not old_project:
|
||||
raise HTTPException(status_code=404, detail=f"Project with ID {project_id} not found")
|
||||
|
||||
old_project_info = ProjectItem(
|
||||
id=old_project.id,
|
||||
name=old_project.name,
|
||||
path=old_project.path,
|
||||
is_default=old_project.is_default or False,
|
||||
)
|
||||
|
||||
# Update using project name (service layer still uses names internally)
|
||||
if path:
|
||||
await project_service.move_project(old_project.name, path)
|
||||
elif is_active is not None:
|
||||
await project_service.update_project(old_project.name, is_active=is_active)
|
||||
|
||||
# Get updated project info
|
||||
updated_project = await project_repository.get_by_id(project_id)
|
||||
if not updated_project:
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"Project with ID {project_id} not found after update"
|
||||
)
|
||||
|
||||
return ProjectStatusResponse(
|
||||
message=f"Project '{updated_project.name}' updated successfully",
|
||||
status="success",
|
||||
default=(old_project.name == project_service.default_project),
|
||||
old_project=old_project_info,
|
||||
new_project=ProjectItem(
|
||||
id=updated_project.id,
|
||||
name=updated_project.name,
|
||||
path=updated_project.path,
|
||||
is_default=updated_project.is_default or False,
|
||||
),
|
||||
)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
|
||||
|
||||
@router.delete("/{project_id}", response_model=ProjectStatusResponse)
|
||||
async def delete_project_by_id(
|
||||
project_id: ProjectIdPathDep,
|
||||
project_service: ProjectServiceDep,
|
||||
project_repository: ProjectRepositoryDep,
|
||||
delete_notes: bool = Query(
|
||||
False, description="If True, delete project directory from filesystem"
|
||||
),
|
||||
) -> ProjectStatusResponse:
|
||||
"""Delete a project by ID.
|
||||
|
||||
Args:
|
||||
project_id: Numeric project ID
|
||||
delete_notes: If True, delete the project directory from the filesystem
|
||||
|
||||
Returns:
|
||||
Response confirming the project was deleted
|
||||
|
||||
Raises:
|
||||
HTTPException: 400 if trying to delete default project, 404 if not found
|
||||
|
||||
Example:
|
||||
DELETE /v2/projects/3?delete_notes=false
|
||||
"""
|
||||
logger.info(
|
||||
f"API v2 request: delete_project_by_id for project_id={project_id}, delete_notes={delete_notes}"
|
||||
)
|
||||
|
||||
try:
|
||||
old_project = await project_repository.get_by_id(project_id)
|
||||
if not old_project:
|
||||
raise HTTPException(status_code=404, detail=f"Project with ID {project_id} not found")
|
||||
|
||||
# Check if trying to delete the default project
|
||||
if old_project.name == project_service.default_project:
|
||||
available_projects = await project_service.list_projects()
|
||||
other_projects = [p.name for p in available_projects if p.id != project_id]
|
||||
detail = f"Cannot delete default project '{old_project.name}'. "
|
||||
if other_projects:
|
||||
detail += (
|
||||
f"Set another project as default first. Available: {', '.join(other_projects)}"
|
||||
)
|
||||
else:
|
||||
detail += "This is the only project in your configuration."
|
||||
raise HTTPException(status_code=400, detail=detail)
|
||||
|
||||
# Delete using project name (service layer still uses names internally)
|
||||
await project_service.remove_project(old_project.name, delete_notes=delete_notes)
|
||||
|
||||
return ProjectStatusResponse(
|
||||
message=f"Project '{old_project.name}' removed successfully",
|
||||
status="success",
|
||||
default=False,
|
||||
old_project=ProjectItem(
|
||||
id=old_project.id,
|
||||
name=old_project.name,
|
||||
path=old_project.path,
|
||||
is_default=old_project.is_default or False,
|
||||
),
|
||||
new_project=None,
|
||||
)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
|
||||
|
||||
@router.put("/{project_id}/default", response_model=ProjectStatusResponse)
|
||||
async def set_default_project_by_id(
|
||||
project_id: ProjectIdPathDep,
|
||||
project_service: ProjectServiceDep,
|
||||
project_repository: ProjectRepositoryDep,
|
||||
) -> ProjectStatusResponse:
|
||||
"""Set a project as the default project by ID.
|
||||
|
||||
Args:
|
||||
project_id: Numeric project ID to set as default
|
||||
|
||||
Returns:
|
||||
Response confirming the project was set as default
|
||||
|
||||
Raises:
|
||||
HTTPException: 404 if project not found
|
||||
|
||||
Example:
|
||||
PUT /v2/projects/3/default
|
||||
"""
|
||||
logger.info(f"API v2 request: set_default_project_by_id for project_id={project_id}")
|
||||
|
||||
try:
|
||||
# Get the old default project
|
||||
default_name = project_service.default_project
|
||||
default_project = await project_service.get_project(default_name)
|
||||
if not default_project:
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"Default Project: '{default_name}' does not exist"
|
||||
)
|
||||
|
||||
# Get the new default project
|
||||
new_default_project = await project_repository.get_by_id(project_id)
|
||||
if not new_default_project:
|
||||
raise HTTPException(status_code=404, detail=f"Project with ID {project_id} not found")
|
||||
|
||||
# Set as default using project name (service layer still uses names internally)
|
||||
await project_service.set_default_project(new_default_project.name)
|
||||
|
||||
return ProjectStatusResponse(
|
||||
message=f"Project '{new_default_project.name}' set as default successfully",
|
||||
status="success",
|
||||
default=True,
|
||||
old_project=ProjectItem(
|
||||
id=default_project.id,
|
||||
name=default_name,
|
||||
path=default_project.path,
|
||||
is_default=False,
|
||||
),
|
||||
new_project=ProjectItem(
|
||||
id=new_default_project.id,
|
||||
name=new_default_project.name,
|
||||
path=new_default_project.path,
|
||||
is_default=True,
|
||||
),
|
||||
)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
@@ -0,0 +1,270 @@
|
||||
"""V2 Prompt Router - ID-based prompt generation operations.
|
||||
|
||||
This router uses v2 dependencies for consistent project ID handling.
|
||||
Prompt endpoints are action-based (not resource-based), so they don't
|
||||
have entity IDs in URLs - they generate formatted prompts from queries.
|
||||
"""
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from fastapi import APIRouter, HTTPException, status
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory.api.routers.utils import to_graph_context, to_search_results
|
||||
from basic_memory.api.template_loader import template_loader
|
||||
from basic_memory.schemas.base import parse_timeframe
|
||||
from basic_memory.deps import (
|
||||
ContextServiceV2Dep,
|
||||
EntityRepositoryV2Dep,
|
||||
SearchServiceV2Dep,
|
||||
EntityServiceV2Dep,
|
||||
ProjectIdPathDep,
|
||||
)
|
||||
from basic_memory.schemas.prompt import (
|
||||
ContinueConversationRequest,
|
||||
SearchPromptRequest,
|
||||
PromptResponse,
|
||||
PromptMetadata,
|
||||
)
|
||||
from basic_memory.schemas.search import SearchItemType, SearchQuery
|
||||
|
||||
router = APIRouter(prefix="/prompt", tags=["prompt-v2"])
|
||||
|
||||
|
||||
@router.post("/continue-conversation", response_model=PromptResponse)
|
||||
async def continue_conversation(
|
||||
project_id: ProjectIdPathDep,
|
||||
search_service: SearchServiceV2Dep,
|
||||
entity_service: EntityServiceV2Dep,
|
||||
context_service: ContextServiceV2Dep,
|
||||
entity_repository: EntityRepositoryV2Dep,
|
||||
request: ContinueConversationRequest,
|
||||
) -> PromptResponse:
|
||||
"""Generate a prompt for continuing a conversation.
|
||||
|
||||
This endpoint takes a topic and/or timeframe and generates a prompt with
|
||||
relevant context from the knowledge base.
|
||||
|
||||
Args:
|
||||
project_id: Validated numeric project ID from URL path
|
||||
request: The request parameters
|
||||
|
||||
Returns:
|
||||
Formatted continuation prompt with context
|
||||
"""
|
||||
logger.info(
|
||||
f"V2 Generating continue conversation prompt for project {project_id}, "
|
||||
f"topic: {request.topic}, timeframe: {request.timeframe}"
|
||||
)
|
||||
|
||||
since = parse_timeframe(request.timeframe) if request.timeframe else None
|
||||
|
||||
# Initialize search results
|
||||
search_results = []
|
||||
|
||||
# Get data needed for template
|
||||
if request.topic:
|
||||
query = SearchQuery(text=request.topic, after_date=request.timeframe)
|
||||
results = await search_service.search(query, limit=request.search_items_limit)
|
||||
search_results = await to_search_results(entity_service, results)
|
||||
|
||||
# Build context from results
|
||||
all_hierarchical_results = []
|
||||
for result in search_results:
|
||||
if hasattr(result, "permalink") and result.permalink:
|
||||
# Get hierarchical context using the new dataclass-based approach
|
||||
context_result = await context_service.build_context(
|
||||
result.permalink,
|
||||
depth=request.depth,
|
||||
since=since,
|
||||
max_related=request.related_items_limit,
|
||||
include_observations=True, # Include observations for entities
|
||||
)
|
||||
|
||||
# Process results into the schema format
|
||||
graph_context = await to_graph_context(
|
||||
context_result, entity_repository=entity_repository
|
||||
)
|
||||
|
||||
# Add results to our collection (limit to top results for each permalink)
|
||||
if graph_context.results:
|
||||
all_hierarchical_results.extend(graph_context.results[:3])
|
||||
|
||||
# Limit to a reasonable number of total results
|
||||
all_hierarchical_results = all_hierarchical_results[:10]
|
||||
|
||||
template_context = {
|
||||
"topic": request.topic,
|
||||
"timeframe": request.timeframe,
|
||||
"hierarchical_results": all_hierarchical_results,
|
||||
"has_results": len(all_hierarchical_results) > 0,
|
||||
}
|
||||
else:
|
||||
# If no topic, get recent activity
|
||||
context_result = await context_service.build_context(
|
||||
types=[SearchItemType.ENTITY],
|
||||
depth=request.depth,
|
||||
since=since,
|
||||
max_related=request.related_items_limit,
|
||||
include_observations=True,
|
||||
)
|
||||
recent_context = await to_graph_context(context_result, entity_repository=entity_repository)
|
||||
|
||||
hierarchical_results = recent_context.results[:5] # Limit to top 5 recent items
|
||||
|
||||
template_context = {
|
||||
"topic": f"Recent Activity from ({request.timeframe})",
|
||||
"timeframe": request.timeframe,
|
||||
"hierarchical_results": hierarchical_results,
|
||||
"has_results": len(hierarchical_results) > 0,
|
||||
}
|
||||
|
||||
try:
|
||||
# Render template
|
||||
rendered_prompt = await template_loader.render(
|
||||
"prompts/continue_conversation.hbs", template_context
|
||||
)
|
||||
|
||||
# Calculate metadata
|
||||
# Count items of different types
|
||||
observation_count = 0
|
||||
relation_count = 0
|
||||
entity_count = 0
|
||||
|
||||
# Get the hierarchical results from the template context
|
||||
hierarchical_results_for_count = template_context.get("hierarchical_results", [])
|
||||
|
||||
# For topic-based search
|
||||
if request.topic:
|
||||
for item in hierarchical_results_for_count:
|
||||
if hasattr(item, "observations"):
|
||||
observation_count += len(item.observations) if item.observations else 0
|
||||
|
||||
if hasattr(item, "related_results"):
|
||||
for related in item.related_results or []:
|
||||
if hasattr(related, "type"):
|
||||
if related.type == "relation":
|
||||
relation_count += 1
|
||||
elif related.type == "entity": # pragma: no cover
|
||||
entity_count += 1 # pragma: no cover
|
||||
# For recent activity
|
||||
else:
|
||||
for item in hierarchical_results_for_count:
|
||||
if hasattr(item, "observations"):
|
||||
observation_count += len(item.observations) if item.observations else 0
|
||||
|
||||
if hasattr(item, "related_results"):
|
||||
for related in item.related_results or []:
|
||||
if hasattr(related, "type"):
|
||||
if related.type == "relation":
|
||||
relation_count += 1
|
||||
elif related.type == "entity": # pragma: no cover
|
||||
entity_count += 1 # pragma: no cover
|
||||
|
||||
# Build metadata
|
||||
metadata = {
|
||||
"query": request.topic,
|
||||
"timeframe": request.timeframe,
|
||||
"search_count": len(search_results)
|
||||
if request.topic
|
||||
else 0, # Original search results count
|
||||
"context_count": len(hierarchical_results_for_count),
|
||||
"observation_count": observation_count,
|
||||
"relation_count": relation_count,
|
||||
"total_items": (
|
||||
len(hierarchical_results_for_count)
|
||||
+ observation_count
|
||||
+ relation_count
|
||||
+ entity_count
|
||||
),
|
||||
"search_limit": request.search_items_limit,
|
||||
"context_depth": request.depth,
|
||||
"related_limit": request.related_items_limit,
|
||||
"generated_at": datetime.now(timezone.utc).isoformat(),
|
||||
}
|
||||
|
||||
prompt_metadata = PromptMetadata(**metadata)
|
||||
|
||||
return PromptResponse(
|
||||
prompt=rendered_prompt, context=template_context, metadata=prompt_metadata
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Error rendering continue conversation template: {e}")
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
|
||||
detail=f"Error rendering prompt template: {str(e)}",
|
||||
)
|
||||
|
||||
|
||||
@router.post("/search", response_model=PromptResponse)
|
||||
async def search_prompt(
|
||||
project_id: ProjectIdPathDep,
|
||||
search_service: SearchServiceV2Dep,
|
||||
entity_service: EntityServiceV2Dep,
|
||||
request: SearchPromptRequest,
|
||||
page: int = 1,
|
||||
page_size: int = 10,
|
||||
) -> PromptResponse:
|
||||
"""Generate a prompt for search results.
|
||||
|
||||
This endpoint takes a search query and formats the results into a helpful
|
||||
prompt with context and suggestions.
|
||||
|
||||
Args:
|
||||
project_id: Validated numeric project ID from URL path
|
||||
request: The search parameters
|
||||
page: The page number for pagination
|
||||
page_size: The number of results per page, defaults to 10
|
||||
|
||||
Returns:
|
||||
Formatted search results prompt with context
|
||||
"""
|
||||
logger.info(
|
||||
f"V2 Generating search prompt for project {project_id}, "
|
||||
f"query: {request.query}, timeframe: {request.timeframe}"
|
||||
)
|
||||
|
||||
limit = page_size
|
||||
offset = (page - 1) * page_size
|
||||
|
||||
query = SearchQuery(text=request.query, after_date=request.timeframe)
|
||||
results = await search_service.search(query, limit=limit, offset=offset)
|
||||
search_results = await to_search_results(entity_service, results)
|
||||
|
||||
template_context = {
|
||||
"query": request.query,
|
||||
"timeframe": request.timeframe,
|
||||
"results": search_results,
|
||||
"has_results": len(search_results) > 0,
|
||||
"result_count": len(search_results),
|
||||
}
|
||||
|
||||
try:
|
||||
# Render template
|
||||
rendered_prompt = await template_loader.render("prompts/search.hbs", template_context)
|
||||
|
||||
# Build metadata
|
||||
metadata = {
|
||||
"query": request.query,
|
||||
"timeframe": request.timeframe,
|
||||
"search_count": len(search_results),
|
||||
"context_count": len(search_results),
|
||||
"observation_count": 0, # Search results don't include observations
|
||||
"relation_count": 0, # Search results don't include relations
|
||||
"total_items": len(search_results),
|
||||
"search_limit": limit,
|
||||
"context_depth": 0, # No context depth for basic search
|
||||
"related_limit": 0, # No related items for basic search
|
||||
"generated_at": datetime.now(timezone.utc).isoformat(),
|
||||
}
|
||||
|
||||
prompt_metadata = PromptMetadata(**metadata)
|
||||
|
||||
return PromptResponse(
|
||||
prompt=rendered_prompt, context=template_context, metadata=prompt_metadata
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Error rendering search template: {e}")
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
|
||||
detail=f"Error rendering prompt template: {str(e)}",
|
||||
)
|
||||
@@ -0,0 +1,286 @@
|
||||
"""V2 Resource Router - ID-based resource content operations.
|
||||
|
||||
This router uses entity IDs for all operations, with file paths in request bodies
|
||||
when needed. This is consistent with v2's ID-first design.
|
||||
|
||||
Key differences from v1:
|
||||
- Uses integer entity IDs in URL paths instead of file paths
|
||||
- File paths are in request bodies for create/update operations
|
||||
- More RESTful: POST for create, PUT for update, GET for read
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Response
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory.deps import (
|
||||
ProjectConfigV2Dep,
|
||||
EntityServiceV2Dep,
|
||||
FileServiceV2Dep,
|
||||
EntityRepositoryV2Dep,
|
||||
SearchServiceV2Dep,
|
||||
ProjectIdPathDep,
|
||||
)
|
||||
from basic_memory.models.knowledge import Entity as EntityModel
|
||||
from basic_memory.schemas.v2.resource import (
|
||||
CreateResourceRequest,
|
||||
UpdateResourceRequest,
|
||||
ResourceResponse,
|
||||
)
|
||||
from basic_memory.utils import validate_project_path
|
||||
|
||||
router = APIRouter(prefix="/resource", tags=["resources-v2"])
|
||||
|
||||
|
||||
@router.get("/{entity_id}")
|
||||
async def get_resource_content(
|
||||
project_id: ProjectIdPathDep,
|
||||
entity_id: int,
|
||||
config: ProjectConfigV2Dep,
|
||||
entity_service: EntityServiceV2Dep,
|
||||
file_service: FileServiceV2Dep,
|
||||
) -> Response:
|
||||
"""Get raw resource content by entity ID.
|
||||
|
||||
Args:
|
||||
project_id: Validated numeric project ID from URL path
|
||||
entity_id: Numeric entity ID
|
||||
config: Project configuration
|
||||
entity_service: Entity service for fetching entity data
|
||||
file_service: File service for reading file content
|
||||
|
||||
Returns:
|
||||
Response with entity content
|
||||
|
||||
Raises:
|
||||
HTTPException: 404 if entity or file not found
|
||||
"""
|
||||
logger.debug(f"V2 Getting content for project {project_id}, entity_id: {entity_id}")
|
||||
|
||||
# Get entity by ID
|
||||
entities = await entity_service.get_entities_by_id([entity_id])
|
||||
if not entities:
|
||||
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
|
||||
|
||||
entity = entities[0]
|
||||
|
||||
# Validate entity file path to prevent path traversal
|
||||
project_path = Path(config.home)
|
||||
if not validate_project_path(entity.file_path, project_path):
|
||||
logger.error(f"Invalid file path in entity {entity.id}: {entity.file_path}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Entity contains invalid file path",
|
||||
)
|
||||
|
||||
# Check file exists via file_service (for cloud compatibility)
|
||||
if not await file_service.exists(entity.file_path):
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"File not found: {entity.file_path}",
|
||||
)
|
||||
|
||||
# Read content via file_service as bytes (works with both local and S3)
|
||||
content = await file_service.read_file_bytes(entity.file_path)
|
||||
content_type = file_service.content_type(entity.file_path)
|
||||
|
||||
return Response(content=content, media_type=content_type)
|
||||
|
||||
|
||||
@router.post("", response_model=ResourceResponse)
|
||||
async def create_resource(
|
||||
project_id: ProjectIdPathDep,
|
||||
data: CreateResourceRequest,
|
||||
config: ProjectConfigV2Dep,
|
||||
file_service: FileServiceV2Dep,
|
||||
entity_repository: EntityRepositoryV2Dep,
|
||||
search_service: SearchServiceV2Dep,
|
||||
) -> ResourceResponse:
|
||||
"""Create a new resource file.
|
||||
|
||||
Args:
|
||||
project_id: Validated numeric project ID from URL path
|
||||
data: Create resource request with file_path and content
|
||||
config: Project configuration
|
||||
file_service: File service for writing files
|
||||
entity_repository: Entity repository for creating entities
|
||||
search_service: Search service for indexing
|
||||
|
||||
Returns:
|
||||
ResourceResponse with file information including entity_id
|
||||
|
||||
Raises:
|
||||
HTTPException: 400 for invalid file paths, 409 if file already exists
|
||||
"""
|
||||
try:
|
||||
# Validate path to prevent path traversal attacks
|
||||
project_path = Path(config.home)
|
||||
if not validate_project_path(data.file_path, project_path):
|
||||
logger.warning(
|
||||
f"Invalid file path attempted: {data.file_path} in project {config.name}"
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid file path: {data.file_path}. "
|
||||
"Path must be relative and stay within project boundaries.",
|
||||
)
|
||||
|
||||
# Check if entity already exists
|
||||
existing_entity = await entity_repository.get_by_file_path(data.file_path)
|
||||
if existing_entity:
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail=f"Resource already exists at {data.file_path} with entity_id {existing_entity.id}. "
|
||||
f"Use PUT /resource/{existing_entity.id} to update it.",
|
||||
)
|
||||
|
||||
# Cloud compatibility: avoid assuming a local filesystem path.
|
||||
# Delegate directory creation + writes to FileService (local or S3).
|
||||
await file_service.ensure_directory(Path(data.file_path).parent)
|
||||
checksum = await file_service.write_file(data.file_path, data.content)
|
||||
|
||||
# Get file info
|
||||
file_metadata = await file_service.get_file_metadata(data.file_path)
|
||||
|
||||
# Determine file details
|
||||
file_name = Path(data.file_path).name
|
||||
content_type = file_service.content_type(data.file_path)
|
||||
entity_type = "canvas" if data.file_path.endswith(".canvas") else "file"
|
||||
|
||||
# Create a new entity model
|
||||
entity = EntityModel(
|
||||
title=file_name,
|
||||
entity_type=entity_type,
|
||||
content_type=content_type,
|
||||
file_path=data.file_path,
|
||||
checksum=checksum,
|
||||
created_at=file_metadata.created_at,
|
||||
updated_at=file_metadata.modified_at,
|
||||
)
|
||||
entity = await entity_repository.add(entity)
|
||||
|
||||
# Index the file for search
|
||||
await search_service.index_entity(entity) # pyright: ignore
|
||||
|
||||
# Return success response
|
||||
return ResourceResponse(
|
||||
entity_id=entity.id,
|
||||
file_path=data.file_path,
|
||||
checksum=checksum,
|
||||
size=file_metadata.size,
|
||||
created_at=file_metadata.created_at.timestamp(),
|
||||
modified_at=file_metadata.modified_at.timestamp(),
|
||||
)
|
||||
except HTTPException:
|
||||
# Re-raise HTTP exceptions without wrapping
|
||||
raise
|
||||
except Exception as e: # pragma: no cover
|
||||
logger.error(f"Error creating resource {data.file_path}: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Failed to create resource: {str(e)}")
|
||||
|
||||
|
||||
@router.put("/{entity_id}", response_model=ResourceResponse)
|
||||
async def update_resource(
|
||||
project_id: ProjectIdPathDep,
|
||||
entity_id: int,
|
||||
data: UpdateResourceRequest,
|
||||
config: ProjectConfigV2Dep,
|
||||
file_service: FileServiceV2Dep,
|
||||
entity_repository: EntityRepositoryV2Dep,
|
||||
search_service: SearchServiceV2Dep,
|
||||
) -> ResourceResponse:
|
||||
"""Update an existing resource by entity ID.
|
||||
|
||||
Can update content and optionally move the file to a new path.
|
||||
|
||||
Args:
|
||||
project_id: Validated numeric project ID from URL path
|
||||
entity_id: Entity ID of the resource to update
|
||||
data: Update resource request with content and optional new file_path
|
||||
config: Project configuration
|
||||
file_service: File service for writing files
|
||||
entity_repository: Entity repository for updating entities
|
||||
search_service: Search service for indexing
|
||||
|
||||
Returns:
|
||||
ResourceResponse with updated file information
|
||||
|
||||
Raises:
|
||||
HTTPException: 404 if entity not found, 400 for invalid paths
|
||||
"""
|
||||
try:
|
||||
# Get existing entity
|
||||
entity = await entity_repository.get_by_id(entity_id)
|
||||
if not entity:
|
||||
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
|
||||
|
||||
# Determine target file path
|
||||
target_file_path = data.file_path if data.file_path else entity.file_path
|
||||
|
||||
# Validate path to prevent path traversal attacks
|
||||
project_path = Path(config.home)
|
||||
if not validate_project_path(target_file_path, project_path):
|
||||
logger.warning(
|
||||
f"Invalid file path attempted: {target_file_path} in project {config.name}"
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid file path: {target_file_path}. "
|
||||
"Path must be relative and stay within project boundaries.",
|
||||
)
|
||||
|
||||
# If moving file, handle the move
|
||||
if data.file_path and data.file_path != entity.file_path:
|
||||
# Ensure new parent directory exists (no-op for S3)
|
||||
await file_service.ensure_directory(Path(target_file_path).parent)
|
||||
|
||||
# If old file exists, remove it via file_service (for cloud compatibility)
|
||||
if await file_service.exists(entity.file_path):
|
||||
await file_service.delete_file(entity.file_path)
|
||||
else:
|
||||
# Ensure directory exists for in-place update
|
||||
await file_service.ensure_directory(Path(target_file_path).parent)
|
||||
|
||||
# Write content to target file
|
||||
checksum = await file_service.write_file(target_file_path, data.content)
|
||||
|
||||
# Get file info
|
||||
file_metadata = await file_service.get_file_metadata(target_file_path)
|
||||
|
||||
# Determine file details
|
||||
file_name = Path(target_file_path).name
|
||||
content_type = file_service.content_type(target_file_path)
|
||||
entity_type = "canvas" if target_file_path.endswith(".canvas") else "file"
|
||||
|
||||
# Update entity
|
||||
updated_entity = await entity_repository.update(
|
||||
entity_id,
|
||||
{
|
||||
"title": file_name,
|
||||
"entity_type": entity_type,
|
||||
"content_type": content_type,
|
||||
"file_path": target_file_path,
|
||||
"checksum": checksum,
|
||||
"updated_at": file_metadata.modified_at,
|
||||
},
|
||||
)
|
||||
|
||||
# Index the updated file for search
|
||||
await search_service.index_entity(updated_entity) # pyright: ignore
|
||||
|
||||
# Return success response
|
||||
return ResourceResponse(
|
||||
entity_id=entity_id,
|
||||
file_path=target_file_path,
|
||||
checksum=checksum,
|
||||
size=file_metadata.size,
|
||||
created_at=file_metadata.created_at.timestamp(),
|
||||
modified_at=file_metadata.modified_at.timestamp(),
|
||||
)
|
||||
except HTTPException:
|
||||
# Re-raise HTTP exceptions without wrapping
|
||||
raise
|
||||
except Exception as e: # pragma: no cover
|
||||
logger.error(f"Error updating resource {entity_id}: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Failed to update resource: {str(e)}")
|
||||
@@ -0,0 +1,73 @@
|
||||
"""V2 router for search operations.
|
||||
|
||||
This router uses integer project IDs for stable, efficient routing.
|
||||
V1 uses string-based project names which are less efficient and less stable.
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, BackgroundTasks
|
||||
|
||||
from basic_memory.api.routers.utils import to_search_results
|
||||
from basic_memory.schemas.search import SearchQuery, SearchResponse
|
||||
from basic_memory.deps import SearchServiceV2Dep, EntityServiceV2Dep, ProjectIdPathDep
|
||||
|
||||
# Note: No prefix here - it's added during registration as /v2/{project_id}/search
|
||||
router = APIRouter(tags=["search"])
|
||||
|
||||
|
||||
@router.post("/search/", response_model=SearchResponse)
|
||||
async def search(
|
||||
project_id: ProjectIdPathDep,
|
||||
query: SearchQuery,
|
||||
search_service: SearchServiceV2Dep,
|
||||
entity_service: EntityServiceV2Dep,
|
||||
page: int = 1,
|
||||
page_size: int = 10,
|
||||
):
|
||||
"""Search across all knowledge and documents in a project.
|
||||
|
||||
V2 uses integer project IDs for improved performance and stability.
|
||||
|
||||
Args:
|
||||
project_id: Validated numeric project ID from URL path
|
||||
query: Search query parameters (text, filters, etc.)
|
||||
search_service: Search service scoped to project
|
||||
entity_service: Entity service scoped to project
|
||||
page: Page number for pagination
|
||||
page_size: Number of results per page
|
||||
|
||||
Returns:
|
||||
SearchResponse with paginated search results
|
||||
"""
|
||||
limit = page_size
|
||||
offset = (page - 1) * page_size
|
||||
results = await search_service.search(query, limit=limit, offset=offset)
|
||||
search_results = await to_search_results(entity_service, results)
|
||||
return SearchResponse(
|
||||
results=search_results,
|
||||
current_page=page,
|
||||
page_size=page_size,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/search/reindex")
|
||||
async def reindex(
|
||||
project_id: ProjectIdPathDep,
|
||||
background_tasks: BackgroundTasks,
|
||||
search_service: SearchServiceV2Dep,
|
||||
):
|
||||
"""Recreate and populate the search index for a project.
|
||||
|
||||
This is a background operation that rebuilds the search index
|
||||
from scratch. Useful after bulk updates or if the index becomes
|
||||
corrupted.
|
||||
|
||||
Args:
|
||||
project_id: Validated numeric project ID from URL path
|
||||
background_tasks: FastAPI background tasks handler
|
||||
search_service: Search service scoped to project
|
||||
|
||||
Returns:
|
||||
Status message indicating reindex has been initiated
|
||||
"""
|
||||
await search_service.reindex_all(background_tasks=background_tasks)
|
||||
return {"status": "ok", "message": "Reindex initiated"}
|
||||
@@ -2,7 +2,7 @@ from typing import Optional
|
||||
|
||||
import typer
|
||||
|
||||
from basic_memory.config import ConfigManager
|
||||
from basic_memory.config import ConfigManager, init_cli_logging
|
||||
|
||||
|
||||
def version_callback(value: bool) -> None:
|
||||
@@ -31,6 +31,9 @@ def app_callback(
|
||||
) -> None:
|
||||
"""Basic Memory - Local-first personal knowledge management."""
|
||||
|
||||
# Initialize logging for CLI (file only, no stdout)
|
||||
init_cli_logging()
|
||||
|
||||
# Run initialization for every command unless --version was specified
|
||||
if not version and ctx.invoked_subcommand is not None:
|
||||
from basic_memory.services.initialization import ensure_initialization
|
||||
|
||||
@@ -6,7 +6,7 @@ import typer
|
||||
from typing import Optional
|
||||
|
||||
from basic_memory.cli.app import app
|
||||
from basic_memory.config import ConfigManager
|
||||
from basic_memory.config import ConfigManager, init_mcp_logging
|
||||
|
||||
# Import mcp instance
|
||||
from basic_memory.mcp.server import mcp as mcp_server # pragma: no cover
|
||||
@@ -44,6 +44,8 @@ if not config.cloud_mode_enabled:
|
||||
- streamable-http: Recommended for web deployments (default)
|
||||
- sse: Server-Sent Events (for compatibility with existing clients)
|
||||
"""
|
||||
# Initialize logging for MCP (file only, stdout breaks protocol)
|
||||
init_mcp_logging()
|
||||
|
||||
# Validate and set project constraint if specified
|
||||
if project:
|
||||
|
||||
+99
-77
@@ -9,10 +9,9 @@ from typing import Any, Dict, Literal, Optional, List, Tuple
|
||||
from enum import Enum
|
||||
|
||||
from loguru import logger
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
import basic_memory
|
||||
from basic_memory.utils import setup_logging, generate_permalink
|
||||
|
||||
|
||||
@@ -100,13 +99,32 @@ class BasicMemoryConfig(BaseSettings):
|
||||
description="Database connection URL. For Postgres, use postgresql+asyncpg://user:pass@host:port/db. If not set, SQLite will use default path.",
|
||||
)
|
||||
|
||||
# Database connection pool configuration (Postgres only)
|
||||
db_pool_size: int = Field(
|
||||
default=20,
|
||||
description="Number of connections to keep in the pool (Postgres only)",
|
||||
gt=0,
|
||||
)
|
||||
db_pool_overflow: int = Field(
|
||||
default=40,
|
||||
description="Max additional connections beyond pool_size under load (Postgres only)",
|
||||
gt=0,
|
||||
)
|
||||
db_pool_recycle: int = Field(
|
||||
default=180,
|
||||
description="Recycle connections after N seconds to prevent stale connections. Default 180s works well with Neon's ~5 minute scale-to-zero (Postgres only)",
|
||||
gt=0,
|
||||
)
|
||||
|
||||
# Watch service configuration
|
||||
sync_delay: int = Field(
|
||||
default=1000, description="Milliseconds to wait after changes before syncing", gt=0
|
||||
)
|
||||
|
||||
watch_project_reload_interval: int = Field(
|
||||
default=30, description="Seconds between reloading project list in watch service", gt=0
|
||||
default=300,
|
||||
description="Seconds between reloading project list in watch service. Higher values reduce CPU usage by minimizing watcher restarts. Default 300s (5 min) balances efficiency with responsiveness to new projects.",
|
||||
gt=0,
|
||||
)
|
||||
|
||||
# update permalinks on move
|
||||
@@ -132,12 +150,6 @@ class BasicMemoryConfig(BaseSettings):
|
||||
gt=0,
|
||||
)
|
||||
|
||||
sync_batch_size: int = Field(
|
||||
default=100,
|
||||
description="Number of files to process in a single database transaction during sync. Higher values improve performance with remote databases (Postgres) but increase memory usage. Typical values: 100 (conservative), 500 (balanced), 1000 (aggressive).",
|
||||
gt=0,
|
||||
)
|
||||
|
||||
kebab_filenames: bool = Field(
|
||||
default=False,
|
||||
description="Format for generated filenames. False preserves spaces and special chars, True converts them to hyphens for consistency with permalinks",
|
||||
@@ -203,6 +215,36 @@ class BasicMemoryConfig(BaseSettings):
|
||||
# Fall back to config file value
|
||||
return self.cloud_mode
|
||||
|
||||
@classmethod
|
||||
def for_cloud_tenant(
|
||||
cls,
|
||||
database_url: str,
|
||||
projects: Optional[Dict[str, str]] = None,
|
||||
) -> "BasicMemoryConfig":
|
||||
"""Create config for cloud tenant - no config.json, database is source of truth.
|
||||
|
||||
This factory method creates a BasicMemoryConfig suitable for cloud deployments
|
||||
where:
|
||||
- Database is Postgres (Neon), not SQLite
|
||||
- Projects are discovered from the database, not config file
|
||||
- Path validation is skipped (no local filesystem in cloud)
|
||||
- Initialization sync is skipped (stateless deployment)
|
||||
|
||||
Args:
|
||||
database_url: Postgres connection URL for tenant database
|
||||
projects: Optional project mapping (usually empty, discovered from DB)
|
||||
|
||||
Returns:
|
||||
BasicMemoryConfig configured for cloud mode
|
||||
"""
|
||||
return cls(
|
||||
database_backend=DatabaseBackend.POSTGRES,
|
||||
database_url=database_url,
|
||||
projects=projects or {},
|
||||
cloud_mode=True,
|
||||
skip_initialization_sync=True,
|
||||
)
|
||||
|
||||
model_config = SettingsConfigDict(
|
||||
env_prefix="BASIC_MEMORY_",
|
||||
extra="ignore",
|
||||
@@ -219,6 +261,10 @@ class BasicMemoryConfig(BaseSettings):
|
||||
|
||||
def model_post_init(self, __context: Any) -> None:
|
||||
"""Ensure configuration is valid after initialization."""
|
||||
# Skip project initialization in cloud mode - projects are discovered from DB
|
||||
if self.database_backend == DatabaseBackend.POSTGRES:
|
||||
return
|
||||
|
||||
# Ensure at least one project exists; if none exist then create main
|
||||
if not self.projects: # pragma: no cover
|
||||
self.projects["main"] = str(
|
||||
@@ -261,19 +307,26 @@ class BasicMemoryConfig(BaseSettings):
|
||||
"""Get all configured projects as ProjectConfig objects."""
|
||||
return [ProjectConfig(name=name, home=Path(path)) for name, path in self.projects.items()]
|
||||
|
||||
@field_validator("projects")
|
||||
@classmethod
|
||||
def ensure_project_paths_exists(cls, v: Dict[str, str]) -> Dict[str, str]: # pragma: no cover
|
||||
"""Ensure project path exists."""
|
||||
for name, path_value in v.items():
|
||||
@model_validator(mode="after")
|
||||
def ensure_project_paths_exists(self) -> "BasicMemoryConfig": # pragma: no cover
|
||||
"""Ensure project paths exist.
|
||||
|
||||
Skips path creation when using Postgres backend (cloud mode) since
|
||||
cloud tenants don't use local filesystem paths.
|
||||
"""
|
||||
# Skip path creation for cloud mode - no local filesystem
|
||||
if self.database_backend == DatabaseBackend.POSTGRES:
|
||||
return self
|
||||
|
||||
for name, path_value in self.projects.items():
|
||||
path = Path(path_value)
|
||||
if not Path(path).exists():
|
||||
if not path.exists():
|
||||
try:
|
||||
path.mkdir(parents=True)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to create project path: {e}")
|
||||
raise e
|
||||
return v
|
||||
return self
|
||||
|
||||
@property
|
||||
def data_dir_path(self):
|
||||
@@ -476,69 +529,38 @@ def save_basic_memory_config(file_path: Path, config: BasicMemoryConfig) -> None
|
||||
logger.error(f"Failed to save config: {e}")
|
||||
|
||||
|
||||
# setup logging to a single log file in user home directory
|
||||
user_home = Path.home()
|
||||
log_dir = user_home / DATA_DIR_NAME
|
||||
log_dir.mkdir(parents=True, exist_ok=True)
|
||||
# Logging initialization functions for different entry points
|
||||
|
||||
|
||||
# Process info for logging
|
||||
def get_process_name(): # pragma: no cover
|
||||
def init_cli_logging() -> None: # pragma: no cover
|
||||
"""Initialize logging for CLI commands - file only.
|
||||
|
||||
CLI commands should not log to stdout to avoid interfering with
|
||||
command output and shell integration.
|
||||
"""
|
||||
get the type of process for logging
|
||||
"""
|
||||
import sys
|
||||
log_level = os.getenv("BASIC_MEMORY_LOG_LEVEL", "INFO")
|
||||
setup_logging(log_level=log_level, log_to_file=True)
|
||||
|
||||
if "sync" in sys.argv:
|
||||
return "sync"
|
||||
elif "mcp" in sys.argv:
|
||||
return "mcp"
|
||||
elif "cli" in sys.argv:
|
||||
return "cli"
|
||||
|
||||
def init_mcp_logging() -> None: # pragma: no cover
|
||||
"""Initialize logging for MCP server - file only.
|
||||
|
||||
MCP server must not log to stdout as it would corrupt the
|
||||
JSON-RPC protocol communication.
|
||||
"""
|
||||
log_level = os.getenv("BASIC_MEMORY_LOG_LEVEL", "INFO")
|
||||
setup_logging(log_level=log_level, log_to_file=True)
|
||||
|
||||
|
||||
def init_api_logging() -> None: # pragma: no cover
|
||||
"""Initialize logging for API server.
|
||||
|
||||
Cloud mode (BASIC_MEMORY_CLOUD_MODE=1): stdout with structured context
|
||||
Local mode: file only
|
||||
"""
|
||||
log_level = os.getenv("BASIC_MEMORY_LOG_LEVEL", "INFO")
|
||||
cloud_mode = os.getenv("BASIC_MEMORY_CLOUD_MODE", "").lower() in ("1", "true")
|
||||
if cloud_mode:
|
||||
setup_logging(log_level=log_level, log_to_stdout=True, structured_context=True)
|
||||
else:
|
||||
return "api"
|
||||
|
||||
|
||||
process_name = get_process_name()
|
||||
|
||||
# Global flag to track if logging has been set up
|
||||
_LOGGING_SETUP = False
|
||||
|
||||
|
||||
# Logging
|
||||
|
||||
|
||||
def setup_basic_memory_logging(): # pragma: no cover
|
||||
"""Set up logging for basic-memory, ensuring it only happens once."""
|
||||
global _LOGGING_SETUP
|
||||
if _LOGGING_SETUP:
|
||||
# We can't log before logging is set up
|
||||
# print("Skipping duplicate logging setup")
|
||||
return
|
||||
|
||||
# Check for console logging environment variable - accept more truthy values
|
||||
console_logging_env = os.getenv("BASIC_MEMORY_CONSOLE_LOGGING", "false").lower()
|
||||
console_logging = console_logging_env in ("true", "1", "yes", "on")
|
||||
|
||||
# Check for log level environment variable first, fall back to config
|
||||
log_level = os.getenv("BASIC_MEMORY_LOG_LEVEL")
|
||||
if not log_level:
|
||||
config_manager = ConfigManager()
|
||||
log_level = config_manager.config.log_level
|
||||
|
||||
config_manager = ConfigManager()
|
||||
config = get_project_config()
|
||||
setup_logging(
|
||||
env=config_manager.config.env,
|
||||
home_dir=user_home, # Use user home for logs
|
||||
log_level=log_level,
|
||||
log_file=f"{DATA_DIR_NAME}/basic-memory-{process_name}.log",
|
||||
console=console_logging,
|
||||
)
|
||||
|
||||
logger.info(f"Basic Memory {basic_memory.__version__} (Project: {config.project})")
|
||||
_LOGGING_SETUP = True
|
||||
|
||||
|
||||
# Set up logging
|
||||
setup_basic_memory_logging()
|
||||
setup_logging(log_level=log_level, log_to_file=True)
|
||||
|
||||
+40
-19
@@ -33,6 +33,7 @@ class DatabaseType(Enum):
|
||||
|
||||
MEMORY = auto()
|
||||
FILESYSTEM = auto()
|
||||
POSTGRES = auto()
|
||||
|
||||
@classmethod
|
||||
def get_db_url(
|
||||
@@ -42,7 +43,7 @@ class DatabaseType(Enum):
|
||||
|
||||
Args:
|
||||
db_path: Path to SQLite database file (ignored for Postgres)
|
||||
db_type: Type of database (MEMORY or FILESYSTEM)
|
||||
db_type: Type of database (MEMORY, FILESYSTEM, or POSTGRES)
|
||||
config: Optional config to check for database backend and URL
|
||||
|
||||
Returns:
|
||||
@@ -52,16 +53,21 @@ class DatabaseType(Enum):
|
||||
if config is None:
|
||||
config = ConfigManager().config
|
||||
|
||||
# Check if Postgres backend is configured
|
||||
# Handle explicit Postgres type
|
||||
if db_type == cls.POSTGRES:
|
||||
if not config.database_url:
|
||||
raise ValueError("DATABASE_URL must be set when using Postgres backend")
|
||||
logger.info(f"Using Postgres database: {config.database_url}")
|
||||
return config.database_url
|
||||
|
||||
# Check if Postgres backend is configured (for backward compatibility)
|
||||
if config.database_backend == DatabaseBackend.POSTGRES:
|
||||
if not config.database_url:
|
||||
raise ValueError("DATABASE_URL must be set when using Postgres backend")
|
||||
logger.info(
|
||||
f"Using Postgres database: {config.database_url.split('@')[1] if '@' in config.database_url else config.database_url}"
|
||||
)
|
||||
logger.info(f"Using Postgres database: {config.database_url}")
|
||||
return config.database_url
|
||||
|
||||
# Default to SQLite
|
||||
# SQLite databases
|
||||
if db_type == cls.MEMORY:
|
||||
logger.info("Using in-memory SQLite database")
|
||||
return "sqlite+aiosqlite://"
|
||||
@@ -184,21 +190,37 @@ def _create_sqlite_engine(db_url: str, db_type: DatabaseType) -> AsyncEngine:
|
||||
return engine
|
||||
|
||||
|
||||
def _create_postgres_engine(db_url: str) -> AsyncEngine:
|
||||
def _create_postgres_engine(db_url: str, config: BasicMemoryConfig) -> AsyncEngine:
|
||||
"""Create Postgres async engine with appropriate configuration.
|
||||
|
||||
Args:
|
||||
db_url: Postgres connection URL (postgresql+asyncpg://...)
|
||||
config: BasicMemoryConfig with pool settings
|
||||
|
||||
Returns:
|
||||
Configured async engine for Postgres
|
||||
"""
|
||||
# Postgres with asyncpg - use standard async connection
|
||||
# Use NullPool connection issues.
|
||||
# Assume connection pooler like PgBouncer handles connection pooling.
|
||||
engine = create_async_engine(
|
||||
db_url,
|
||||
echo=False,
|
||||
pool_pre_ping=True, # Verify connections before using them
|
||||
poolclass=NullPool, # No pooling - fresh connection per request
|
||||
connect_args={
|
||||
# Disable statement cache to avoid issues with prepared statements on reconnect
|
||||
"statement_cache_size": 0,
|
||||
# Allow 30s for commands (Neon cold start can take 2-5s, sometimes longer)
|
||||
"command_timeout": 30,
|
||||
# Allow 30s for initial connection (Neon wake-up time)
|
||||
"timeout": 30,
|
||||
"server_settings": {
|
||||
"application_name": "basic-memory",
|
||||
# Statement timeout for queries (30s to allow for cold start)
|
||||
"statement_timeout": "30s",
|
||||
},
|
||||
},
|
||||
)
|
||||
logger.debug("Created Postgres engine with NullPool (no connection pooling)")
|
||||
|
||||
return engine
|
||||
|
||||
@@ -210,7 +232,7 @@ def _create_engine_and_session(
|
||||
|
||||
Args:
|
||||
db_path: Path to database file (used for SQLite, ignored for Postgres)
|
||||
db_type: Type of database (MEMORY or FILESYSTEM)
|
||||
db_type: Type of database (MEMORY, FILESYSTEM, or POSTGRES)
|
||||
|
||||
Returns:
|
||||
Tuple of (engine, session_maker)
|
||||
@@ -220,8 +242,9 @@ def _create_engine_and_session(
|
||||
logger.debug(f"Creating engine for db_url: {db_url}")
|
||||
|
||||
# Delegate to backend-specific engine creation
|
||||
if config.database_backend == DatabaseBackend.POSTGRES:
|
||||
engine = _create_postgres_engine(db_url)
|
||||
# Check explicit POSTGRES type first, then config setting
|
||||
if db_type == DatabaseType.POSTGRES or config.database_backend == DatabaseBackend.POSTGRES:
|
||||
engine = _create_postgres_engine(db_url, config)
|
||||
else:
|
||||
engine = _create_sqlite_engine(db_url, db_type)
|
||||
|
||||
@@ -326,13 +349,8 @@ async def run_migrations(
|
||||
config.set_main_option("revision_environment", "false")
|
||||
|
||||
# Get the correct database URL based on backend configuration
|
||||
# No URL conversion needed - env.py now handles both async and sync engines
|
||||
db_url = DatabaseType.get_db_url(app_config.database_path, database_type, app_config)
|
||||
|
||||
# For Postgres, Alembic needs synchronous driver (psycopg2), not async (asyncpg)
|
||||
if app_config.database_backend == DatabaseBackend.POSTGRES:
|
||||
# Convert asyncpg URL to psycopg2 URL for Alembic
|
||||
db_url = db_url.replace("postgresql+asyncpg://", "postgresql://")
|
||||
|
||||
config.set_main_option("sqlalchemy.url", db_url)
|
||||
|
||||
command.upgrade(config, "head")
|
||||
@@ -348,7 +366,10 @@ async def run_migrations(
|
||||
# For SQLite: Create FTS5 virtual table
|
||||
# For Postgres: No-op (tsvector column added by migrations)
|
||||
# The project_id is not used for init_search_index, so we pass a dummy value
|
||||
if app_config.database_backend == DatabaseBackend.POSTGRES:
|
||||
if (
|
||||
database_type == DatabaseType.POSTGRES
|
||||
or app_config.database_backend == DatabaseBackend.POSTGRES
|
||||
):
|
||||
await PostgresSearchRepository(session_maker, 1).init_search_index()
|
||||
else:
|
||||
await SQLiteSearchRepository(session_maker, 1).init_search_index()
|
||||
|
||||
+282
-4
@@ -76,6 +76,34 @@ async def get_project_config(
|
||||
|
||||
ProjectConfigDep = Annotated[ProjectConfig, Depends(get_project_config)] # pragma: no cover
|
||||
|
||||
|
||||
async def get_project_config_v2(
|
||||
project_id: "ProjectIdPathDep", project_repository: "ProjectRepositoryDep"
|
||||
) -> ProjectConfig: # pragma: no cover
|
||||
"""Get the project config for v2 API (uses integer project_id from path).
|
||||
|
||||
Args:
|
||||
project_id: The validated numeric project ID from the URL path
|
||||
project_repository: Repository for project operations
|
||||
|
||||
Returns:
|
||||
The resolved project config
|
||||
|
||||
Raises:
|
||||
HTTPException: If project is not found
|
||||
"""
|
||||
project_obj = await project_repository.get_by_id(project_id)
|
||||
if project_obj:
|
||||
return ProjectConfig(name=project_obj.name, home=pathlib.Path(project_obj.path))
|
||||
|
||||
# Not found (this should not happen since ProjectIdPathDep already validates existence)
|
||||
raise HTTPException( # pragma: no cover
|
||||
status_code=status.HTTP_404_NOT_FOUND, detail=f"Project with ID {project_id} not found."
|
||||
)
|
||||
|
||||
|
||||
ProjectConfigV2Dep = Annotated[ProjectConfig, Depends(get_project_config_v2)] # pragma: no cover
|
||||
|
||||
## sqlalchemy
|
||||
|
||||
|
||||
@@ -130,6 +158,38 @@ ProjectRepositoryDep = Annotated[ProjectRepository, Depends(get_project_reposito
|
||||
ProjectPathDep = Annotated[str, Path()] # Use Path dependency to extract from URL
|
||||
|
||||
|
||||
async def validate_project_id(
|
||||
project_id: int,
|
||||
project_repository: ProjectRepositoryDep,
|
||||
) -> int:
|
||||
"""Validate that a numeric project ID exists in the database.
|
||||
|
||||
This is used for v2 API endpoints that take project IDs as integers in the path.
|
||||
The project_id parameter will be automatically extracted from the URL path by FastAPI.
|
||||
|
||||
Args:
|
||||
project_id: The numeric project ID from the URL path
|
||||
project_repository: Repository for project operations
|
||||
|
||||
Returns:
|
||||
The validated project ID
|
||||
|
||||
Raises:
|
||||
HTTPException: If project with that ID is not found
|
||||
"""
|
||||
project_obj = await project_repository.get_by_id(project_id)
|
||||
if not project_obj:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail=f"Project with ID {project_id} not found.",
|
||||
)
|
||||
return project_id
|
||||
|
||||
|
||||
# V2 API: Validated integer project ID from path
|
||||
ProjectIdPathDep = Annotated[int, Depends(validate_project_id)]
|
||||
|
||||
|
||||
async def get_project_id(
|
||||
project_repository: ProjectRepositoryDep,
|
||||
project: ProjectPathDep,
|
||||
@@ -188,6 +248,17 @@ async def get_entity_repository(
|
||||
EntityRepositoryDep = Annotated[EntityRepository, Depends(get_entity_repository)]
|
||||
|
||||
|
||||
async def get_entity_repository_v2(
|
||||
session_maker: SessionMakerDep,
|
||||
project_id: ProjectIdPathDep,
|
||||
) -> EntityRepository:
|
||||
"""Create an EntityRepository instance for v2 API (uses integer project_id from path)."""
|
||||
return EntityRepository(session_maker, project_id=project_id)
|
||||
|
||||
|
||||
EntityRepositoryV2Dep = Annotated[EntityRepository, Depends(get_entity_repository_v2)]
|
||||
|
||||
|
||||
async def get_observation_repository(
|
||||
session_maker: SessionMakerDep,
|
||||
project_id: ProjectIdDep,
|
||||
@@ -199,6 +270,19 @@ async def get_observation_repository(
|
||||
ObservationRepositoryDep = Annotated[ObservationRepository, Depends(get_observation_repository)]
|
||||
|
||||
|
||||
async def get_observation_repository_v2(
|
||||
session_maker: SessionMakerDep,
|
||||
project_id: ProjectIdPathDep,
|
||||
) -> ObservationRepository:
|
||||
"""Create an ObservationRepository instance for v2 API."""
|
||||
return ObservationRepository(session_maker, project_id=project_id)
|
||||
|
||||
|
||||
ObservationRepositoryV2Dep = Annotated[
|
||||
ObservationRepository, Depends(get_observation_repository_v2)
|
||||
]
|
||||
|
||||
|
||||
async def get_relation_repository(
|
||||
session_maker: SessionMakerDep,
|
||||
project_id: ProjectIdDep,
|
||||
@@ -210,6 +294,17 @@ async def get_relation_repository(
|
||||
RelationRepositoryDep = Annotated[RelationRepository, Depends(get_relation_repository)]
|
||||
|
||||
|
||||
async def get_relation_repository_v2(
|
||||
session_maker: SessionMakerDep,
|
||||
project_id: ProjectIdPathDep,
|
||||
) -> RelationRepository:
|
||||
"""Create a RelationRepository instance for v2 API."""
|
||||
return RelationRepository(session_maker, project_id=project_id)
|
||||
|
||||
|
||||
RelationRepositoryV2Dep = Annotated[RelationRepository, Depends(get_relation_repository_v2)]
|
||||
|
||||
|
||||
async def get_search_repository(
|
||||
session_maker: SessionMakerDep,
|
||||
project_id: ProjectIdDep,
|
||||
@@ -225,6 +320,17 @@ async def get_search_repository(
|
||||
SearchRepositoryDep = Annotated[SearchRepository, Depends(get_search_repository)]
|
||||
|
||||
|
||||
async def get_search_repository_v2(
|
||||
session_maker: SessionMakerDep,
|
||||
project_id: ProjectIdPathDep,
|
||||
) -> SearchRepository:
|
||||
"""Create a SearchRepository instance for v2 API."""
|
||||
return create_search_repository(session_maker, project_id=project_id)
|
||||
|
||||
|
||||
SearchRepositoryV2Dep = Annotated[SearchRepository, Depends(get_search_repository_v2)]
|
||||
|
||||
|
||||
# ProjectInfoRepository is deprecated and will be removed in a future version.
|
||||
# Use ProjectRepository instead, which has the same functionality plus more project-specific operations.
|
||||
|
||||
@@ -238,6 +344,13 @@ async def get_entity_parser(project_config: ProjectConfigDep) -> EntityParser:
|
||||
EntityParserDep = Annotated["EntityParser", Depends(get_entity_parser)]
|
||||
|
||||
|
||||
async def get_entity_parser_v2(project_config: ProjectConfigV2Dep) -> EntityParser:
|
||||
return EntityParser(project_config.home)
|
||||
|
||||
|
||||
EntityParserV2Dep = Annotated["EntityParser", Depends(get_entity_parser_v2)]
|
||||
|
||||
|
||||
async def get_markdown_processor(entity_parser: EntityParserDep) -> MarkdownProcessor:
|
||||
return MarkdownProcessor(entity_parser)
|
||||
|
||||
@@ -245,20 +358,39 @@ async def get_markdown_processor(entity_parser: EntityParserDep) -> MarkdownProc
|
||||
MarkdownProcessorDep = Annotated[MarkdownProcessor, Depends(get_markdown_processor)]
|
||||
|
||||
|
||||
async def get_markdown_processor_v2(entity_parser: EntityParserV2Dep) -> MarkdownProcessor:
|
||||
return MarkdownProcessor(entity_parser)
|
||||
|
||||
|
||||
MarkdownProcessorV2Dep = Annotated[MarkdownProcessor, Depends(get_markdown_processor_v2)]
|
||||
|
||||
|
||||
async def get_file_service(
|
||||
project_config: ProjectConfigDep, markdown_processor: MarkdownProcessorDep
|
||||
) -> FileService:
|
||||
logger.debug(
|
||||
f"Creating FileService for project: {project_config.name}, base_path: {project_config.home}"
|
||||
)
|
||||
file_service = FileService(project_config.home, markdown_processor)
|
||||
logger.debug(f"Created FileService for project: {file_service} ")
|
||||
logger.debug(
|
||||
f"Created FileService for project: {project_config.name}, base_path: {project_config.home} "
|
||||
)
|
||||
return file_service
|
||||
|
||||
|
||||
FileServiceDep = Annotated[FileService, Depends(get_file_service)]
|
||||
|
||||
|
||||
async def get_file_service_v2(
|
||||
project_config: ProjectConfigV2Dep, markdown_processor: MarkdownProcessorV2Dep
|
||||
) -> FileService:
|
||||
file_service = FileService(project_config.home, markdown_processor)
|
||||
logger.debug(
|
||||
f"Created FileService for project: {project_config.name}, base_path: {project_config.home}"
|
||||
)
|
||||
return file_service
|
||||
|
||||
|
||||
FileServiceV2Dep = Annotated[FileService, Depends(get_file_service_v2)]
|
||||
|
||||
|
||||
async def get_entity_service(
|
||||
entity_repository: EntityRepositoryDep,
|
||||
observation_repository: ObservationRepositoryDep,
|
||||
@@ -283,6 +415,30 @@ async def get_entity_service(
|
||||
EntityServiceDep = Annotated[EntityService, Depends(get_entity_service)]
|
||||
|
||||
|
||||
async def get_entity_service_v2(
|
||||
entity_repository: EntityRepositoryV2Dep,
|
||||
observation_repository: ObservationRepositoryV2Dep,
|
||||
relation_repository: RelationRepositoryV2Dep,
|
||||
entity_parser: EntityParserV2Dep,
|
||||
file_service: FileServiceV2Dep,
|
||||
link_resolver: "LinkResolverV2Dep",
|
||||
app_config: AppConfigDep,
|
||||
) -> EntityService:
|
||||
"""Create EntityService for v2 API."""
|
||||
return EntityService(
|
||||
entity_repository=entity_repository,
|
||||
observation_repository=observation_repository,
|
||||
relation_repository=relation_repository,
|
||||
entity_parser=entity_parser,
|
||||
file_service=file_service,
|
||||
link_resolver=link_resolver,
|
||||
app_config=app_config,
|
||||
)
|
||||
|
||||
|
||||
EntityServiceV2Dep = Annotated[EntityService, Depends(get_entity_service_v2)]
|
||||
|
||||
|
||||
async def get_search_service(
|
||||
search_repository: SearchRepositoryDep,
|
||||
entity_repository: EntityRepositoryDep,
|
||||
@@ -295,6 +451,18 @@ async def get_search_service(
|
||||
SearchServiceDep = Annotated[SearchService, Depends(get_search_service)]
|
||||
|
||||
|
||||
async def get_search_service_v2(
|
||||
search_repository: SearchRepositoryV2Dep,
|
||||
entity_repository: EntityRepositoryV2Dep,
|
||||
file_service: FileServiceV2Dep,
|
||||
) -> SearchService:
|
||||
"""Create SearchService for v2 API."""
|
||||
return SearchService(search_repository, entity_repository, file_service)
|
||||
|
||||
|
||||
SearchServiceV2Dep = Annotated[SearchService, Depends(get_search_service_v2)]
|
||||
|
||||
|
||||
async def get_link_resolver(
|
||||
entity_repository: EntityRepositoryDep, search_service: SearchServiceDep
|
||||
) -> LinkResolver:
|
||||
@@ -304,6 +472,15 @@ async def get_link_resolver(
|
||||
LinkResolverDep = Annotated[LinkResolver, Depends(get_link_resolver)]
|
||||
|
||||
|
||||
async def get_link_resolver_v2(
|
||||
entity_repository: EntityRepositoryV2Dep, search_service: SearchServiceV2Dep
|
||||
) -> LinkResolver:
|
||||
return LinkResolver(entity_repository=entity_repository, search_service=search_service)
|
||||
|
||||
|
||||
LinkResolverV2Dep = Annotated[LinkResolver, Depends(get_link_resolver_v2)]
|
||||
|
||||
|
||||
async def get_context_service(
|
||||
search_repository: SearchRepositoryDep,
|
||||
entity_repository: EntityRepositoryDep,
|
||||
@@ -319,6 +496,22 @@ async def get_context_service(
|
||||
ContextServiceDep = Annotated[ContextService, Depends(get_context_service)]
|
||||
|
||||
|
||||
async def get_context_service_v2(
|
||||
search_repository: SearchRepositoryV2Dep,
|
||||
entity_repository: EntityRepositoryV2Dep,
|
||||
observation_repository: ObservationRepositoryV2Dep,
|
||||
) -> ContextService:
|
||||
"""Create ContextService for v2 API."""
|
||||
return ContextService(
|
||||
search_repository=search_repository,
|
||||
entity_repository=entity_repository,
|
||||
observation_repository=observation_repository,
|
||||
)
|
||||
|
||||
|
||||
ContextServiceV2Dep = Annotated[ContextService, Depends(get_context_service_v2)]
|
||||
|
||||
|
||||
async def get_sync_service(
|
||||
app_config: AppConfigDep,
|
||||
entity_service: EntityServiceDep,
|
||||
@@ -348,6 +541,32 @@ async def get_sync_service(
|
||||
SyncServiceDep = Annotated[SyncService, Depends(get_sync_service)]
|
||||
|
||||
|
||||
async def get_sync_service_v2(
|
||||
app_config: AppConfigDep,
|
||||
entity_service: EntityServiceV2Dep,
|
||||
entity_parser: EntityParserV2Dep,
|
||||
entity_repository: EntityRepositoryV2Dep,
|
||||
relation_repository: RelationRepositoryV2Dep,
|
||||
project_repository: ProjectRepositoryDep,
|
||||
search_service: SearchServiceV2Dep,
|
||||
file_service: FileServiceV2Dep,
|
||||
) -> SyncService: # pragma: no cover
|
||||
"""Create SyncService for v2 API."""
|
||||
return SyncService(
|
||||
app_config=app_config,
|
||||
entity_service=entity_service,
|
||||
entity_parser=entity_parser,
|
||||
entity_repository=entity_repository,
|
||||
relation_repository=relation_repository,
|
||||
project_repository=project_repository,
|
||||
search_service=search_service,
|
||||
file_service=file_service,
|
||||
)
|
||||
|
||||
|
||||
SyncServiceV2Dep = Annotated[SyncService, Depends(get_sync_service_v2)]
|
||||
|
||||
|
||||
async def get_project_service(
|
||||
project_repository: ProjectRepositoryDep,
|
||||
) -> ProjectService:
|
||||
@@ -370,6 +589,18 @@ async def get_directory_service(
|
||||
DirectoryServiceDep = Annotated[DirectoryService, Depends(get_directory_service)]
|
||||
|
||||
|
||||
async def get_directory_service_v2(
|
||||
entity_repository: EntityRepositoryV2Dep,
|
||||
) -> DirectoryService:
|
||||
"""Create DirectoryService for v2 API (uses integer project_id from path)."""
|
||||
return DirectoryService(
|
||||
entity_repository=entity_repository,
|
||||
)
|
||||
|
||||
|
||||
DirectoryServiceV2Dep = Annotated[DirectoryService, Depends(get_directory_service_v2)]
|
||||
|
||||
|
||||
# Import
|
||||
|
||||
|
||||
@@ -413,3 +644,50 @@ async def get_memory_json_importer(
|
||||
|
||||
|
||||
MemoryJsonImporterDep = Annotated[MemoryJsonImporter, Depends(get_memory_json_importer)]
|
||||
|
||||
|
||||
# V2 Import dependencies
|
||||
|
||||
|
||||
async def get_chatgpt_importer_v2(
|
||||
project_config: ProjectConfigV2Dep, markdown_processor: MarkdownProcessorV2Dep
|
||||
) -> ChatGPTImporter:
|
||||
"""Create ChatGPTImporter with v2 dependencies."""
|
||||
return ChatGPTImporter(project_config.home, markdown_processor)
|
||||
|
||||
|
||||
ChatGPTImporterV2Dep = Annotated[ChatGPTImporter, Depends(get_chatgpt_importer_v2)]
|
||||
|
||||
|
||||
async def get_claude_conversations_importer_v2(
|
||||
project_config: ProjectConfigV2Dep, markdown_processor: MarkdownProcessorV2Dep
|
||||
) -> ClaudeConversationsImporter:
|
||||
"""Create ClaudeConversationsImporter with v2 dependencies."""
|
||||
return ClaudeConversationsImporter(project_config.home, markdown_processor)
|
||||
|
||||
|
||||
ClaudeConversationsImporterV2Dep = Annotated[
|
||||
ClaudeConversationsImporter, Depends(get_claude_conversations_importer_v2)
|
||||
]
|
||||
|
||||
|
||||
async def get_claude_projects_importer_v2(
|
||||
project_config: ProjectConfigV2Dep, markdown_processor: MarkdownProcessorV2Dep
|
||||
) -> ClaudeProjectsImporter:
|
||||
"""Create ClaudeProjectsImporter with v2 dependencies."""
|
||||
return ClaudeProjectsImporter(project_config.home, markdown_processor)
|
||||
|
||||
|
||||
ClaudeProjectsImporterV2Dep = Annotated[
|
||||
ClaudeProjectsImporter, Depends(get_claude_projects_importer_v2)
|
||||
]
|
||||
|
||||
|
||||
async def get_memory_json_importer_v2(
|
||||
project_config: ProjectConfigV2Dep, markdown_processor: MarkdownProcessorV2Dep
|
||||
) -> MemoryJsonImporter:
|
||||
"""Create MemoryJsonImporter with v2 dependencies."""
|
||||
return MemoryJsonImporter(project_config.home, markdown_processor)
|
||||
|
||||
|
||||
MemoryJsonImporterV2Dep = Annotated[MemoryJsonImporter, Depends(get_memory_json_importer_v2)]
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
"""Utilities for file operations."""
|
||||
|
||||
import hashlib
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
import re
|
||||
from typing import Any, Dict, Union
|
||||
@@ -13,6 +15,20 @@ from loguru import logger
|
||||
from basic_memory.utils import FilePath
|
||||
|
||||
|
||||
@dataclass
|
||||
class FileMetadata:
|
||||
"""File metadata for cloud-compatible file operations.
|
||||
|
||||
This dataclass provides a cloud-agnostic way to represent file metadata,
|
||||
enabling S3FileService to return metadata from head_object responses
|
||||
instead of mock stat_result with zeros.
|
||||
"""
|
||||
|
||||
size: int
|
||||
created_at: datetime
|
||||
modified_at: datetime
|
||||
|
||||
|
||||
class FileError(Exception):
|
||||
"""Base exception for file operations."""
|
||||
|
||||
|
||||
@@ -40,10 +40,13 @@ class ClaudeConversationsImporter(Importer[ChatImportResult]):
|
||||
chats_imported = 0
|
||||
|
||||
for chat in conversations:
|
||||
# Get name, providing default for unnamed conversations
|
||||
chat_name = chat.get("name") or f"Conversation {chat.get('uuid', 'untitled')}"
|
||||
|
||||
# Convert to entity
|
||||
entity = self._format_chat_content(
|
||||
base_path=folder_path,
|
||||
name=chat["name"],
|
||||
name=chat_name,
|
||||
messages=chat["chat_messages"],
|
||||
created_at=chat["created_at"],
|
||||
modified_at=chat["updated_at"],
|
||||
|
||||
@@ -23,6 +23,7 @@ from basic_memory.markdown.schemas import (
|
||||
)
|
||||
from basic_memory.utils import parse_tags
|
||||
|
||||
|
||||
md = MarkdownIt().use(observation_plugin).use(relation_plugin)
|
||||
|
||||
|
||||
@@ -189,35 +190,63 @@ class EntityParser:
|
||||
return self.base_path / path
|
||||
|
||||
async def parse_file_content(self, absolute_path, file_content):
|
||||
# Parse frontmatter with proper error handling for malformed YAML (issue #185)
|
||||
try:
|
||||
post = frontmatter.loads(file_content)
|
||||
except yaml.YAMLError as e:
|
||||
# Log the YAML parsing error with file context
|
||||
logger.warning(
|
||||
f"Failed to parse YAML frontmatter in {absolute_path}: {e}. "
|
||||
f"Treating file as plain markdown without frontmatter."
|
||||
)
|
||||
# Create a post with no frontmatter - treat entire content as markdown
|
||||
post = frontmatter.Post(file_content, metadata={})
|
||||
"""Parse markdown content from file stats.
|
||||
|
||||
# Extract file stat info
|
||||
Delegates to parse_markdown_content() for actual parsing logic.
|
||||
Exists for backwards compatibility with code that passes file paths.
|
||||
"""
|
||||
# Extract file stat info for timestamps
|
||||
file_stats = absolute_path.stat()
|
||||
|
||||
# Normalize frontmatter values to prevent AttributeError on date objects (issue #236)
|
||||
# PyYAML automatically converts date strings like "2025-10-24" to datetime.date objects
|
||||
# This normalization converts them back to ISO format strings to ensure compatibility
|
||||
# with code that expects string values
|
||||
# Delegate to parse_markdown_content with timestamps from file stats
|
||||
return await self.parse_markdown_content(
|
||||
file_path=absolute_path,
|
||||
content=file_content,
|
||||
mtime=file_stats.st_mtime,
|
||||
ctime=file_stats.st_ctime,
|
||||
)
|
||||
|
||||
async def parse_markdown_content(
|
||||
self,
|
||||
file_path: Path,
|
||||
content: str,
|
||||
mtime: Optional[float] = None,
|
||||
ctime: Optional[float] = None,
|
||||
) -> EntityMarkdown:
|
||||
"""Parse markdown content without requiring file to exist on disk.
|
||||
|
||||
Useful for parsing content from S3 or other remote sources where the file
|
||||
is not available locally.
|
||||
|
||||
Args:
|
||||
file_path: Path for metadata (doesn't need to exist on disk)
|
||||
content: Markdown content as string
|
||||
mtime: Optional modification time (Unix timestamp)
|
||||
ctime: Optional creation time (Unix timestamp)
|
||||
|
||||
Returns:
|
||||
EntityMarkdown with parsed content
|
||||
"""
|
||||
# Parse frontmatter with proper error handling for malformed YAML
|
||||
try:
|
||||
post = frontmatter.loads(content)
|
||||
except yaml.YAMLError as e:
|
||||
logger.warning(
|
||||
f"Failed to parse YAML frontmatter in {file_path}: {e}. "
|
||||
f"Treating file as plain markdown without frontmatter."
|
||||
)
|
||||
post = frontmatter.Post(content, metadata={})
|
||||
|
||||
# Normalize frontmatter values
|
||||
metadata = normalize_frontmatter_metadata(post.metadata)
|
||||
|
||||
# Ensure required fields have defaults (issue #184, #387)
|
||||
# Handle title - use default if missing, None/null, empty, or string "None"
|
||||
# Ensure required fields have defaults
|
||||
title = metadata.get("title")
|
||||
if not title or title == "None":
|
||||
metadata["title"] = absolute_path.stem
|
||||
metadata["title"] = file_path.stem
|
||||
else:
|
||||
metadata["title"] = title
|
||||
# Handle type - use default if missing OR explicitly set to None/null
|
||||
|
||||
entity_type = metadata.get("type")
|
||||
metadata["type"] = entity_type if entity_type is not None else "note"
|
||||
|
||||
@@ -225,16 +254,20 @@ class EntityParser:
|
||||
if tags:
|
||||
metadata["tags"] = tags
|
||||
|
||||
# frontmatter - use metadata with defaults applied
|
||||
entity_frontmatter = EntityFrontmatter(
|
||||
metadata=metadata,
|
||||
)
|
||||
# Parse content for observations and relations
|
||||
entity_frontmatter = EntityFrontmatter(metadata=metadata)
|
||||
entity_content = parse(post.content)
|
||||
|
||||
# Use provided timestamps or current time as fallback
|
||||
now = datetime.now().astimezone()
|
||||
created = datetime.fromtimestamp(ctime).astimezone() if ctime else now
|
||||
modified = datetime.fromtimestamp(mtime).astimezone() if mtime else now
|
||||
|
||||
return EntityMarkdown(
|
||||
frontmatter=entity_frontmatter,
|
||||
content=post.content,
|
||||
observations=entity_content.observations,
|
||||
relations=entity_content.relations,
|
||||
created=datetime.fromtimestamp(file_stats.st_ctime).astimezone(),
|
||||
modified=datetime.fromtimestamp(file_stats.st_mtime).astimezone(),
|
||||
created=created,
|
||||
modified=modified,
|
||||
)
|
||||
|
||||
@@ -5,6 +5,7 @@ from collections import OrderedDict
|
||||
from frontmatter import Post
|
||||
from loguru import logger
|
||||
|
||||
|
||||
from basic_memory import file_utils
|
||||
from basic_memory.file_utils import dump_frontmatter
|
||||
from basic_memory.markdown.entity_parser import EntityParser
|
||||
|
||||
@@ -30,7 +30,9 @@ def is_observation(token: Token) -> bool:
|
||||
|
||||
# Check for proper observation format: [category] content
|
||||
match = re.match(r"^\[([^\[\]()]+)\]\s+(.+)", content)
|
||||
has_tags = "#" in content
|
||||
# Check for standalone hashtags (words starting with #)
|
||||
# This excludes # in HTML attributes like color="#4285F4"
|
||||
has_tags = any(part.startswith("#") for part in content.split())
|
||||
return bool(match) or has_tags
|
||||
|
||||
|
||||
@@ -160,7 +162,7 @@ def parse_inline_relations(content: str) -> List[Dict[str, Any]]:
|
||||
|
||||
target = content[start + 2 : end].strip()
|
||||
if target:
|
||||
relations.append({"type": "links to", "target": target, "context": None})
|
||||
relations.append({"type": "links_to", "target": target, "context": None})
|
||||
|
||||
start = end + 2
|
||||
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional
|
||||
|
||||
|
||||
from frontmatter import Post
|
||||
|
||||
from basic_memory.file_utils import has_frontmatter, remove_frontmatter, parse_frontmatter
|
||||
@@ -12,7 +13,10 @@ from basic_memory.models import Observation as ObservationModel
|
||||
|
||||
|
||||
def entity_model_from_markdown(
|
||||
file_path: Path, markdown: EntityMarkdown, entity: Optional[Entity] = None
|
||||
file_path: Path,
|
||||
markdown: EntityMarkdown,
|
||||
entity: Optional[Entity] = None,
|
||||
project_id: Optional[int] = None,
|
||||
) -> Entity:
|
||||
"""
|
||||
Convert markdown entity to model. Does not include relations.
|
||||
@@ -21,6 +25,7 @@ def entity_model_from_markdown(
|
||||
file_path: Path to the markdown file
|
||||
markdown: Parsed markdown entity
|
||||
entity: Optional existing entity to update
|
||||
project_id: Project ID for new observations (uses entity.project_id if not provided)
|
||||
|
||||
Returns:
|
||||
Entity model populated from markdown
|
||||
@@ -50,9 +55,13 @@ def entity_model_from_markdown(
|
||||
metadata = markdown.frontmatter.metadata or {}
|
||||
model.entity_metadata = {k: str(v) for k, v in metadata.items() if v is not None}
|
||||
|
||||
# Get project_id from entity if not provided
|
||||
obs_project_id = project_id or (model.project_id if hasattr(model, "project_id") else None)
|
||||
|
||||
# Convert observations
|
||||
model.observations = [
|
||||
ObservationModel(
|
||||
project_id=obs_project_id,
|
||||
content=obs.content,
|
||||
category=obs.category,
|
||||
context=obs.context,
|
||||
|
||||
@@ -4,7 +4,6 @@ import basic_memory
|
||||
from basic_memory.models.base import Base
|
||||
from basic_memory.models.knowledge import Entity, Observation, Relation
|
||||
from basic_memory.models.project import Project
|
||||
from basic_memory.models.search import SearchIndex
|
||||
|
||||
__all__ = [
|
||||
"Base",
|
||||
@@ -12,6 +11,5 @@ __all__ = [
|
||||
"Observation",
|
||||
"Relation",
|
||||
"Project",
|
||||
"SearchIndex",
|
||||
"basic_memory",
|
||||
]
|
||||
|
||||
@@ -129,7 +129,7 @@ class Entity(Base):
|
||||
return value
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"Entity(id={self.id}, name='{self.title}', type='{self.entity_type}'"
|
||||
return f"Entity(id={self.id}, name='{self.title}', type='{self.entity_type}', checksum='{self.checksum}')"
|
||||
|
||||
|
||||
class Observation(Base):
|
||||
@@ -145,6 +145,7 @@ class Observation(Base):
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
project_id: Mapped[int] = mapped_column(Integer, ForeignKey("project.id"), index=True)
|
||||
entity_id: Mapped[int] = mapped_column(Integer, ForeignKey("entity.id", ondelete="CASCADE"))
|
||||
content: Mapped[str] = mapped_column(Text)
|
||||
category: Mapped[str] = mapped_column(String, nullable=False, default="note")
|
||||
@@ -162,9 +163,14 @@ class Observation(Base):
|
||||
|
||||
We can construct these because observations are always defined in
|
||||
and owned by a single entity.
|
||||
|
||||
Content is truncated to 200 chars to stay under PostgreSQL's
|
||||
btree index limit of 2704 bytes.
|
||||
"""
|
||||
# Truncate content to avoid exceeding PostgreSQL's btree index limit
|
||||
content_for_permalink = self.content[:200] if len(self.content) > 200 else self.content
|
||||
return generate_permalink(
|
||||
f"{self.entity.permalink}/observations/{self.category}/{self.content}"
|
||||
f"{self.entity.permalink}/observations/{self.category}/{content_for_permalink}"
|
||||
)
|
||||
|
||||
def __repr__(self) -> str: # pragma: no cover
|
||||
@@ -186,6 +192,7 @@ class Relation(Base):
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
project_id: Mapped[int] = mapped_column(Integer, ForeignKey("project.id"), index=True)
|
||||
from_id: Mapped[int] = mapped_column(Integer, ForeignKey("entity.id", ondelete="CASCADE"))
|
||||
to_id: Mapped[Optional[int]] = mapped_column(
|
||||
Integer, ForeignKey("entity.id", ondelete="CASCADE"), nullable=True
|
||||
|
||||
@@ -1,53 +1,52 @@
|
||||
"""Search models and tables."""
|
||||
"""Search DDL statements for SQLite and Postgres.
|
||||
|
||||
from sqlalchemy import DDL, Column, Integer, String, DateTime, Text
|
||||
from sqlalchemy.dialects.postgresql import JSONB
|
||||
from sqlalchemy.types import JSON
|
||||
The search_index table is created via raw DDL, not ORM models, because:
|
||||
- SQLite uses FTS5 virtual tables (cannot be represented as ORM)
|
||||
- Postgres uses composite primary keys and generated tsvector columns
|
||||
- Both backends use raw SQL for all search operations via SearchIndexRow dataclass
|
||||
"""
|
||||
|
||||
from basic_memory.models.base import Base
|
||||
from sqlalchemy import DDL
|
||||
|
||||
|
||||
class SearchIndex(Base):
|
||||
"""Search index table for Postgres only.
|
||||
# Define Postgres search_index table with composite primary key and tsvector
|
||||
# This DDL matches the Alembic migration schema (314f1ea54dc4)
|
||||
# Used by tests to create the table without running full migrations
|
||||
# NOTE: Split into separate DDL statements because asyncpg doesn't support
|
||||
# multiple statements in a single execute call.
|
||||
CREATE_POSTGRES_SEARCH_INDEX_TABLE = DDL("""
|
||||
CREATE TABLE IF NOT EXISTS search_index (
|
||||
id INTEGER NOT NULL,
|
||||
project_id INTEGER NOT NULL,
|
||||
title TEXT,
|
||||
content_stems TEXT,
|
||||
content_snippet TEXT,
|
||||
permalink VARCHAR,
|
||||
file_path VARCHAR,
|
||||
type VARCHAR,
|
||||
from_id INTEGER,
|
||||
to_id INTEGER,
|
||||
relation_type VARCHAR,
|
||||
entity_id INTEGER,
|
||||
category VARCHAR,
|
||||
metadata JSONB,
|
||||
created_at TIMESTAMP WITH TIME ZONE,
|
||||
updated_at TIMESTAMP WITH TIME ZONE,
|
||||
textsearchable_index_col tsvector GENERATED ALWAYS AS (
|
||||
to_tsvector('english', coalesce(title, '') || ' ' || coalesce(content_stems, ''))
|
||||
) STORED,
|
||||
PRIMARY KEY (id, type, project_id),
|
||||
FOREIGN KEY (project_id) REFERENCES project(id) ON DELETE CASCADE
|
||||
)
|
||||
""")
|
||||
|
||||
For SQLite: This model is skipped; FTS5 virtual table is created via DDL instead.
|
||||
For Postgres: This is the actual table structure with tsvector support.
|
||||
"""
|
||||
|
||||
__tablename__ = "search_index"
|
||||
|
||||
# Primary key (rowid in SQLite FTS5, explicit id in Postgres)
|
||||
id = Column(Integer, primary_key=True, autoincrement=True)
|
||||
|
||||
# Core searchable fields
|
||||
title = Column(Text, nullable=True)
|
||||
content_stems = Column(Text, nullable=True)
|
||||
content_snippet = Column(Text, nullable=True)
|
||||
permalink = Column(String(255), nullable=True, index=True)
|
||||
file_path = Column(Text, nullable=True)
|
||||
type = Column(String(50), nullable=True)
|
||||
|
||||
# Project context
|
||||
project_id = Column(Integer, nullable=True, index=True)
|
||||
|
||||
# Relation fields
|
||||
from_id = Column(Integer, nullable=True)
|
||||
to_id = Column(Integer, nullable=True)
|
||||
relation_type = Column(String(100), nullable=True)
|
||||
|
||||
# Observation fields
|
||||
entity_id = Column(Integer, nullable=True)
|
||||
category = Column(String(100), nullable=True)
|
||||
|
||||
# Common fields
|
||||
# Use JSONB for Postgres, JSON for SQLite
|
||||
# Note: 'metadata' is a reserved name in SQLAlchemy, so we use 'metadata_' and map to 'metadata'
|
||||
metadata_ = Column("metadata", JSON().with_variant(JSONB(), "postgresql"), nullable=True)
|
||||
created_at = Column(DateTime(timezone=True), nullable=True)
|
||||
updated_at = Column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
# Note: textsearchable_index_col (tsvector) will be added by migration for Postgres only
|
||||
CREATE_POSTGRES_SEARCH_INDEX_FTS = DDL("""
|
||||
CREATE INDEX IF NOT EXISTS idx_search_index_fts ON search_index USING gin(textsearchable_index_col)
|
||||
""")
|
||||
|
||||
CREATE_POSTGRES_SEARCH_INDEX_METADATA = DDL("""
|
||||
CREATE INDEX IF NOT EXISTS idx_search_index_metadata_gin ON search_index USING gin(metadata jsonb_path_ops)
|
||||
""")
|
||||
|
||||
# Define FTS5 virtual table creation for SQLite only
|
||||
# This DDL is executed separately for SQLite databases
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
"""Repository for managing entities in the knowledge graph."""
|
||||
|
||||
from pathlib import Path
|
||||
from typing import List, Optional, Sequence, Union
|
||||
from typing import List, Optional, Sequence, Union, Any
|
||||
|
||||
|
||||
from loguru import logger
|
||||
from sqlalchemy import select
|
||||
@@ -9,6 +10,7 @@ from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
from sqlalchemy.orm import selectinload
|
||||
from sqlalchemy.orm.interfaces import LoaderOption
|
||||
from sqlalchemy.engine import Row
|
||||
|
||||
from basic_memory import db
|
||||
from basic_memory.models.knowledge import Entity, Observation, Relation
|
||||
@@ -31,6 +33,18 @@ class EntityRepository(Repository[Entity]):
|
||||
"""
|
||||
super().__init__(session_maker, Entity, project_id=project_id)
|
||||
|
||||
async def get_by_id(self, entity_id: int) -> Optional[Entity]:
|
||||
"""Get entity by numeric ID.
|
||||
|
||||
Args:
|
||||
entity_id: Numeric entity ID
|
||||
|
||||
Returns:
|
||||
Entity if found, None otherwise
|
||||
"""
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
return await self.select_by_id(session, entity_id)
|
||||
|
||||
async def get_by_permalink(self, permalink: str) -> Optional[Entity]:
|
||||
"""Get entity by permalink.
|
||||
|
||||
@@ -63,37 +77,126 @@ class EntityRepository(Repository[Entity]):
|
||||
)
|
||||
return await self.find_one(query)
|
||||
|
||||
async def get_by_file_paths_batch(
|
||||
self, file_paths: Sequence[Union[Path, str]]
|
||||
) -> dict[str, Entity]:
|
||||
"""Batch fetch entities by file paths with eager-loaded relationships.
|
||||
# -------------------------------------------------------------------------
|
||||
# Lightweight methods for permalink resolution (no eager loading)
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
Optimized for scan operations - reduces N queries to 1 batched query.
|
||||
Returns entities with relationships already loaded via selectinload.
|
||||
async def permalink_exists(self, permalink: str) -> bool:
|
||||
"""Check if a permalink exists without loading the full entity.
|
||||
|
||||
This is much faster than get_by_permalink() as it skips eager loading
|
||||
of observations and relations. Use for existence checks in bulk operations.
|
||||
|
||||
Args:
|
||||
file_paths: List of file paths to fetch entities for
|
||||
permalink: Permalink to check
|
||||
|
||||
Returns:
|
||||
Dict mapping file_path (as posix string) -> Entity
|
||||
Only includes entities that exist; missing files are not in dict
|
||||
True if permalink exists, False otherwise
|
||||
"""
|
||||
query = select(Entity.id).where(Entity.permalink == permalink).limit(1)
|
||||
query = self._add_project_filter(query)
|
||||
result = await self.execute_query(query, use_query_options=False)
|
||||
return result.scalar_one_or_none() is not None
|
||||
|
||||
async def get_file_path_for_permalink(self, permalink: str) -> Optional[str]:
|
||||
"""Get the file_path for a permalink without loading the full entity.
|
||||
|
||||
Use when you only need the file_path, not the full entity with relations.
|
||||
|
||||
Args:
|
||||
permalink: Permalink to look up
|
||||
|
||||
Returns:
|
||||
file_path string if found, None otherwise
|
||||
"""
|
||||
query = select(Entity.file_path).where(Entity.permalink == permalink)
|
||||
query = self._add_project_filter(query)
|
||||
result = await self.execute_query(query, use_query_options=False)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
async def get_permalink_for_file_path(self, file_path: Union[Path, str]) -> Optional[str]:
|
||||
"""Get the permalink for a file_path without loading the full entity.
|
||||
|
||||
Use when you only need the permalink, not the full entity with relations.
|
||||
|
||||
Args:
|
||||
file_path: File path to look up
|
||||
|
||||
Returns:
|
||||
permalink string if found, None otherwise
|
||||
"""
|
||||
query = select(Entity.permalink).where(Entity.file_path == Path(file_path).as_posix())
|
||||
query = self._add_project_filter(query)
|
||||
result = await self.execute_query(query, use_query_options=False)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
async def get_all_permalinks(self) -> List[str]:
|
||||
"""Get all permalinks for this project.
|
||||
|
||||
Optimized for bulk operations - returns only permalink strings
|
||||
without loading entities or relationships.
|
||||
|
||||
Returns:
|
||||
List of all permalinks in the project
|
||||
"""
|
||||
query = select(Entity.permalink)
|
||||
query = self._add_project_filter(query)
|
||||
result = await self.execute_query(query, use_query_options=False)
|
||||
return list(result.scalars().all())
|
||||
|
||||
async def get_permalink_to_file_path_map(self) -> dict[str, str]:
|
||||
"""Get a mapping of permalink -> file_path for all entities.
|
||||
|
||||
Optimized for bulk permalink resolution - loads minimal data in one query.
|
||||
|
||||
Returns:
|
||||
Dict mapping permalink to file_path
|
||||
"""
|
||||
query = select(Entity.permalink, Entity.file_path)
|
||||
query = self._add_project_filter(query)
|
||||
result = await self.execute_query(query, use_query_options=False)
|
||||
return {row.permalink: row.file_path for row in result.all()}
|
||||
|
||||
async def get_file_path_to_permalink_map(self) -> dict[str, str]:
|
||||
"""Get a mapping of file_path -> permalink for all entities.
|
||||
|
||||
Optimized for bulk permalink resolution - loads minimal data in one query.
|
||||
|
||||
Returns:
|
||||
Dict mapping file_path to permalink
|
||||
"""
|
||||
query = select(Entity.file_path, Entity.permalink)
|
||||
query = self._add_project_filter(query)
|
||||
result = await self.execute_query(query, use_query_options=False)
|
||||
return {row.file_path: row.permalink for row in result.all()}
|
||||
|
||||
async def get_by_file_paths(
|
||||
self, session: AsyncSession, file_paths: Sequence[Union[Path, str]]
|
||||
) -> List[Row[Any]]:
|
||||
"""Get file paths and checksums for multiple entities (optimized for change detection).
|
||||
|
||||
Only queries file_path and checksum columns, skips loading full entities and relationships.
|
||||
This is much faster than loading complete Entity objects when you only need checksums.
|
||||
|
||||
Args:
|
||||
session: Database session to use for the query
|
||||
file_paths: List of file paths to query
|
||||
|
||||
Returns:
|
||||
List of (file_path, checksum) tuples for matching entities
|
||||
"""
|
||||
if not file_paths:
|
||||
return {}
|
||||
return []
|
||||
|
||||
# Convert all paths to posix strings
|
||||
posix_paths = [Path(p).as_posix() for p in file_paths]
|
||||
# Convert all paths to POSIX strings for consistent comparison
|
||||
posix_paths = [Path(fp).as_posix() for fp in file_paths]
|
||||
|
||||
# Batch query with eager loading
|
||||
query = (
|
||||
self.select().where(Entity.file_path.in_(posix_paths)).options(*self.get_load_options())
|
||||
)
|
||||
# Query ONLY file_path and checksum columns (not full Entity objects)
|
||||
query = select(Entity.file_path, Entity.checksum).where(Entity.file_path.in_(posix_paths))
|
||||
query = self._add_project_filter(query)
|
||||
|
||||
result = await self.execute_query(query)
|
||||
entities = list(result.scalars().all())
|
||||
|
||||
# Return as dict for O(1) lookup
|
||||
return {e.file_path: e for e in entities}
|
||||
result = await session.execute(query)
|
||||
return list(result.all())
|
||||
|
||||
async def find_by_checksum(self, checksum: str) -> Sequence[Entity]:
|
||||
"""Find entities with the given checksum.
|
||||
@@ -112,6 +215,34 @@ class EntityRepository(Repository[Entity]):
|
||||
result = await self.execute_query(query, use_query_options=False)
|
||||
return list(result.scalars().all())
|
||||
|
||||
async def find_by_checksums(self, checksums: Sequence[str]) -> Sequence[Entity]:
|
||||
"""Find entities with any of the given checksums (batch query for move detection).
|
||||
|
||||
This is a batch-optimized version of find_by_checksum() that queries multiple checksums
|
||||
in a single database query. Used for efficient move detection in cloud indexing.
|
||||
|
||||
Performance: For 1000 new files, this makes 1 query vs 1000 individual queries (~100x faster).
|
||||
|
||||
Example:
|
||||
When processing new files, we check if any are actually moved files by finding
|
||||
entities with matching checksums at different paths.
|
||||
|
||||
Args:
|
||||
checksums: List of file content checksums to search for
|
||||
|
||||
Returns:
|
||||
Sequence of entities with matching checksums (may be empty).
|
||||
Multiple entities may have the same checksum if files were copied.
|
||||
"""
|
||||
if not checksums:
|
||||
return []
|
||||
|
||||
# Query: SELECT * FROM entities WHERE checksum IN (checksum1, checksum2, ...)
|
||||
query = self.select().where(Entity.checksum.in_(checksums))
|
||||
# Don't load relationships for move detection - we only need file_path and checksum
|
||||
result = await self.execute_query(query, use_query_options=False)
|
||||
return list(result.scalars().all())
|
||||
|
||||
async def delete_by_file_path(self, file_path: Union[Path, str]) -> bool:
|
||||
"""Delete entity with the provided file_path.
|
||||
|
||||
@@ -370,80 +501,3 @@ class EntityRepository(Repository[Entity]):
|
||||
# Re-raise if not a foreign key error
|
||||
raise
|
||||
return entity
|
||||
|
||||
async def upsert_entities(self, entities: List[Entity]) -> List[Entity]:
|
||||
"""Bulk insert or update multiple entities in a single transaction.
|
||||
|
||||
Optimized for batch operations with remote databases (Postgres).
|
||||
Handles conflicts the same way as upsert_entity() but processes
|
||||
all entities in one transaction.
|
||||
|
||||
Args:
|
||||
entities: List of entities to upsert
|
||||
|
||||
Returns:
|
||||
List of upserted entities with relationships loaded
|
||||
|
||||
Raises:
|
||||
SyncFatalError: If any entity references a non-existent project_id
|
||||
"""
|
||||
if not entities:
|
||||
return []
|
||||
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
# Set project_id on all entities if needed
|
||||
for entity in entities:
|
||||
self._set_project_id_if_needed(entity)
|
||||
|
||||
# Try to add all entities
|
||||
for entity in entities:
|
||||
session.add(entity)
|
||||
|
||||
try:
|
||||
await session.flush()
|
||||
|
||||
# Fetch all entities with relationships loaded
|
||||
file_paths = [e.file_path for e in entities]
|
||||
query = (
|
||||
self.select()
|
||||
.where(Entity.file_path.in_(file_paths))
|
||||
.options(*self.get_load_options())
|
||||
)
|
||||
result = await session.execute(query)
|
||||
return list(result.scalars().all())
|
||||
|
||||
except IntegrityError as e:
|
||||
# Check for foreign key constraint failures
|
||||
error_str = str(e)
|
||||
if (
|
||||
"FOREIGN KEY constraint failed" in error_str
|
||||
or "violates foreign key constraint" in error_str
|
||||
):
|
||||
from basic_memory.services.exceptions import SyncFatalError
|
||||
|
||||
raise SyncFatalError(
|
||||
"Cannot sync entities: project_id does not exist in database. "
|
||||
"The project may have been deleted. This sync will be terminated."
|
||||
) from e
|
||||
|
||||
# For other integrity errors (file_path or permalink conflicts),
|
||||
# rollback and fall back to individual processing
|
||||
await session.rollback()
|
||||
|
||||
# Process each entity individually to handle conflicts properly
|
||||
logger.debug(
|
||||
f"Batch upsert failed with IntegrityError, falling back to individual upserts for {len(entities)} entities"
|
||||
)
|
||||
|
||||
result_entities = []
|
||||
for entity in entities:
|
||||
try:
|
||||
upserted = await self.upsert_entity(entity)
|
||||
result_entities.append(upserted)
|
||||
except Exception as individual_error:
|
||||
logger.error(
|
||||
f"Failed to upsert entity {entity.file_path}: {individual_error}"
|
||||
)
|
||||
# Continue with other entities
|
||||
|
||||
return result_entities
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
from typing import Dict, List, Sequence
|
||||
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker
|
||||
|
||||
@@ -70,33 +71,3 @@ class ObservationRepository(Repository[Observation]):
|
||||
observations_by_entity[obs.entity_id].append(obs)
|
||||
|
||||
return observations_by_entity
|
||||
|
||||
async def delete_by_entity_ids(self, entity_ids: List[int]) -> int:
|
||||
"""Delete all observations for multiple entities in a single query.
|
||||
|
||||
Optimized for batch operations - deletes observations for many entities
|
||||
in one database transaction.
|
||||
|
||||
Args:
|
||||
entity_ids: List of entity IDs whose observations should be deleted
|
||||
|
||||
Returns:
|
||||
Number of observations deleted
|
||||
"""
|
||||
if not entity_ids:
|
||||
return 0
|
||||
|
||||
from basic_memory import db
|
||||
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
# Use bulk delete with IN clause
|
||||
query = select(Observation).where(Observation.entity_id.in_(entity_ids))
|
||||
result = await session.execute(query)
|
||||
observations_to_delete = result.scalars().all()
|
||||
|
||||
# Delete all observations
|
||||
for obs in observations_to_delete:
|
||||
await session.delete(obs)
|
||||
|
||||
await session.flush()
|
||||
return len(observations_to_delete)
|
||||
|
||||
@@ -5,6 +5,7 @@ import re
|
||||
from datetime import datetime
|
||||
from typing import List, Optional
|
||||
|
||||
|
||||
from loguru import logger
|
||||
from sqlalchemy import text
|
||||
|
||||
@@ -257,7 +258,7 @@ class PostgresSearchRepository(SearchRepositoryBase):
|
||||
{score_expr} as score
|
||||
FROM search_index
|
||||
WHERE {where_clause}
|
||||
ORDER BY score DESC {order_by_clause}
|
||||
ORDER BY score DESC, id ASC {order_by_clause}
|
||||
LIMIT :limit
|
||||
OFFSET :offset
|
||||
"""
|
||||
@@ -311,3 +312,68 @@ class PostgresSearchRepository(SearchRepositoryBase):
|
||||
)
|
||||
|
||||
return results
|
||||
|
||||
async def bulk_index_items(self, search_index_rows: List[SearchIndexRow]) -> None:
|
||||
"""Index multiple items in a single batch operation using UPSERT.
|
||||
|
||||
Uses INSERT ... ON CONFLICT DO UPDATE to handle re-indexing of existing
|
||||
entities (e.g., during forward reference resolution) without requiring
|
||||
a separate delete operation. This eliminates race conditions between
|
||||
delete and insert operations in separate transactions.
|
||||
|
||||
Args:
|
||||
search_index_rows: List of SearchIndexRow objects to index
|
||||
"""
|
||||
|
||||
if not search_index_rows:
|
||||
return
|
||||
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
# When using text() raw SQL, always serialize JSON to string
|
||||
# Both SQLite (TEXT) and Postgres (JSONB) accept JSON strings in raw SQL
|
||||
# The database driver/column type will handle conversion
|
||||
insert_data_list = []
|
||||
for row in search_index_rows:
|
||||
insert_data = row.to_insert(serialize_json=True)
|
||||
insert_data["project_id"] = self.project_id
|
||||
insert_data_list.append(insert_data)
|
||||
|
||||
# Use UPSERT (INSERT ... ON CONFLICT) to handle re-indexing
|
||||
# Primary key is (id, type, project_id)
|
||||
# This handles race conditions during forward reference resolution
|
||||
# where an entity might be re-indexed before the delete commits
|
||||
# Syntax works for both SQLite 3.24+ and PostgreSQL
|
||||
await session.execute(
|
||||
text("""
|
||||
INSERT INTO search_index (
|
||||
id, title, content_stems, content_snippet, permalink, file_path, type, metadata,
|
||||
from_id, to_id, relation_type,
|
||||
entity_id, category,
|
||||
created_at, updated_at,
|
||||
project_id
|
||||
) VALUES (
|
||||
:id, :title, :content_stems, :content_snippet, :permalink, :file_path, :type, :metadata,
|
||||
:from_id, :to_id, :relation_type,
|
||||
:entity_id, :category,
|
||||
:created_at, :updated_at,
|
||||
:project_id
|
||||
)
|
||||
ON CONFLICT (id, type, project_id) DO UPDATE SET
|
||||
title = EXCLUDED.title,
|
||||
content_stems = EXCLUDED.content_stems,
|
||||
content_snippet = EXCLUDED.content_snippet,
|
||||
permalink = EXCLUDED.permalink,
|
||||
file_path = EXCLUDED.file_path,
|
||||
metadata = EXCLUDED.metadata,
|
||||
from_id = EXCLUDED.from_id,
|
||||
to_id = EXCLUDED.to_id,
|
||||
relation_type = EXCLUDED.relation_type,
|
||||
entity_id = EXCLUDED.entity_id,
|
||||
category = EXCLUDED.category,
|
||||
created_at = EXCLUDED.created_at,
|
||||
updated_at = EXCLUDED.updated_at
|
||||
"""),
|
||||
insert_data_list,
|
||||
)
|
||||
logger.debug(f"Bulk indexed {len(search_index_rows)} rows")
|
||||
await session.commit()
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
from pathlib import Path
|
||||
from typing import Optional, Sequence, Union
|
||||
|
||||
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
@@ -49,6 +50,18 @@ class ProjectRepository(Repository[Project]):
|
||||
query = self.select().where(Project.path == Path(path).as_posix())
|
||||
return await self.find_one(query)
|
||||
|
||||
async def get_by_id(self, project_id: int) -> Optional[Project]:
|
||||
"""Get project by numeric ID.
|
||||
|
||||
Args:
|
||||
project_id: Numeric project ID
|
||||
|
||||
Returns:
|
||||
Project if found, None otherwise
|
||||
"""
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
return await self.select_by_id(session, project_id)
|
||||
|
||||
async def get_default_project(self) -> Optional[Project]:
|
||||
"""Get the default project (the one marked as is_default=True)."""
|
||||
query = self.select().where(Project.is_default.is_not(None))
|
||||
|
||||
@@ -1,9 +1,11 @@
|
||||
"""Repository for managing Relation objects."""
|
||||
|
||||
from sqlalchemy import and_, delete
|
||||
from typing import Sequence, List, Optional
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from sqlalchemy import and_, delete, select
|
||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker
|
||||
from sqlalchemy.orm import selectinload, aliased
|
||||
from sqlalchemy.orm.interfaces import LoaderOption
|
||||
@@ -67,27 +69,6 @@ class RelationRepository(Repository[Relation]):
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
await session.execute(delete(Relation).where(Relation.from_id == entity_id))
|
||||
|
||||
async def delete_outgoing_relations_from_entities(self, entity_ids: List[int]) -> int:
|
||||
"""Delete outgoing relations for multiple entities in a single query.
|
||||
|
||||
Optimized for batch operations - deletes relations for many entities
|
||||
in one database transaction. Only deletes relations where these entities
|
||||
are the source (from_id).
|
||||
|
||||
Args:
|
||||
entity_ids: List of entity IDs whose outgoing relations should be deleted
|
||||
|
||||
Returns:
|
||||
Number of relations deleted
|
||||
"""
|
||||
if not entity_ids:
|
||||
return 0
|
||||
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
# Use bulk delete with IN clause
|
||||
result = await session.execute(delete(Relation).where(Relation.from_id.in_(entity_ids)))
|
||||
return result.rowcount or 0
|
||||
|
||||
async def find_unresolved_relations(self) -> Sequence[Relation]:
|
||||
"""Find all unresolved relations, where to_id is null."""
|
||||
query = select(Relation).filter(Relation.to_id.is_(None))
|
||||
@@ -107,5 +88,59 @@ class RelationRepository(Repository[Relation]):
|
||||
result = await self.execute_query(query)
|
||||
return result.scalars().all()
|
||||
|
||||
async def add_all_ignore_duplicates(self, relations: List[Relation]) -> int:
|
||||
"""Bulk insert relations, ignoring duplicates.
|
||||
|
||||
Uses ON CONFLICT DO NOTHING to skip relations that would violate the
|
||||
unique constraint on (from_id, to_name, relation_type). This is useful
|
||||
for bulk operations where the same link may appear multiple times in
|
||||
a document.
|
||||
|
||||
Works with both SQLite and PostgreSQL dialects.
|
||||
|
||||
Args:
|
||||
relations: List of Relation objects to insert
|
||||
|
||||
Returns:
|
||||
Number of relations actually inserted (excludes duplicates)
|
||||
"""
|
||||
if not relations:
|
||||
return 0
|
||||
|
||||
# Convert Relation objects to dicts for insert
|
||||
values = [
|
||||
{
|
||||
"project_id": r.project_id if r.project_id else self.project_id,
|
||||
"from_id": r.from_id,
|
||||
"to_id": r.to_id,
|
||||
"to_name": r.to_name,
|
||||
"relation_type": r.relation_type,
|
||||
"context": r.context,
|
||||
}
|
||||
for r in relations
|
||||
]
|
||||
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
# Check dialect to use appropriate insert
|
||||
dialect_name = session.bind.dialect.name if session.bind else "sqlite"
|
||||
|
||||
if dialect_name == "postgresql":
|
||||
# PostgreSQL: use RETURNING to count inserted rows
|
||||
# (rowcount is 0 for ON CONFLICT DO NOTHING)
|
||||
stmt = (
|
||||
pg_insert(Relation)
|
||||
.values(values)
|
||||
.on_conflict_do_nothing()
|
||||
.returning(Relation.id)
|
||||
)
|
||||
result = await session.execute(stmt)
|
||||
return len(result.fetchall())
|
||||
else:
|
||||
# SQLite: rowcount works correctly
|
||||
stmt = sqlite_insert(Relation).values(values)
|
||||
stmt = stmt.on_conflict_do_nothing()
|
||||
result = await session.execute(stmt)
|
||||
return result.rowcount if result.rowcount > 0 else 0
|
||||
|
||||
def get_load_options(self) -> List[LoaderOption]:
|
||||
return [selectinload(Relation.from_entity), selectinload(Relation.to_entity)]
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
from typing import Type, Optional, Any, Sequence, TypeVar, List, Dict
|
||||
|
||||
|
||||
from loguru import logger
|
||||
from sqlalchemy import (
|
||||
select,
|
||||
|
||||
@@ -4,6 +4,7 @@ from abc import ABC, abstractmethod
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
|
||||
from loguru import logger
|
||||
from sqlalchemy import Executable, Result, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
@@ -5,6 +5,7 @@ import re
|
||||
from datetime import datetime
|
||||
from typing import List, Optional
|
||||
|
||||
|
||||
from loguru import logger
|
||||
from sqlalchemy import text
|
||||
|
||||
|
||||
@@ -124,6 +124,7 @@ class EntitySummary(BaseModel):
|
||||
"""Simplified entity representation."""
|
||||
|
||||
type: Literal["entity"] = "entity"
|
||||
entity_id: int # Database ID for v2 API consistency
|
||||
permalink: Optional[str]
|
||||
title: str
|
||||
content: Optional[str] = None
|
||||
@@ -141,12 +142,16 @@ class RelationSummary(BaseModel):
|
||||
"""Simplified relation representation."""
|
||||
|
||||
type: Literal["relation"] = "relation"
|
||||
relation_id: int # Database ID for v2 API consistency
|
||||
entity_id: Optional[int] = None # ID of the entity this relation belongs to
|
||||
title: str
|
||||
file_path: str
|
||||
permalink: str
|
||||
relation_type: str
|
||||
from_entity: Optional[str] = None
|
||||
from_entity_id: Optional[int] = None # ID of source entity
|
||||
to_entity: Optional[str] = None
|
||||
to_entity_id: Optional[int] = None # ID of target entity
|
||||
created_at: Annotated[
|
||||
datetime, Field(json_schema_extra={"type": "string", "format": "date-time"})
|
||||
]
|
||||
@@ -160,6 +165,8 @@ class ObservationSummary(BaseModel):
|
||||
"""Simplified observation representation."""
|
||||
|
||||
type: Literal["observation"] = "observation"
|
||||
observation_id: int # Database ID for v2 API consistency
|
||||
entity_id: Optional[int] = None # ID of the entity this observation belongs to
|
||||
title: str
|
||||
file_path: str
|
||||
permalink: str
|
||||
|
||||
@@ -173,6 +173,7 @@ class ProjectWatchStatus(BaseModel):
|
||||
class ProjectItem(BaseModel):
|
||||
"""Simple representation of a project."""
|
||||
|
||||
id: int
|
||||
name: str
|
||||
path: str
|
||||
is_default: bool = False
|
||||
|
||||
@@ -97,6 +97,11 @@ class SearchResult(BaseModel):
|
||||
|
||||
metadata: Optional[dict] = None
|
||||
|
||||
# IDs for v2 API consistency
|
||||
entity_id: Optional[int] = None # Entity ID (always present for entities)
|
||||
observation_id: Optional[int] = None # Observation ID (for observation results)
|
||||
relation_id: Optional[int] = None # Relation ID (for relation results)
|
||||
|
||||
# Type-specific fields
|
||||
category: Optional[str] = None # For observations
|
||||
from_entity: Optional[Permalink] = None # For relations
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
"""V2 API schemas - ID-based entity references."""
|
||||
|
||||
from basic_memory.schemas.v2.entity import (
|
||||
EntityResolveRequest,
|
||||
EntityResolveResponse,
|
||||
EntityResponseV2,
|
||||
MoveEntityRequestV2,
|
||||
)
|
||||
from basic_memory.schemas.v2.resource import (
|
||||
CreateResourceRequest,
|
||||
UpdateResourceRequest,
|
||||
ResourceResponse,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"EntityResolveRequest",
|
||||
"EntityResolveResponse",
|
||||
"EntityResponseV2",
|
||||
"MoveEntityRequestV2",
|
||||
"CreateResourceRequest",
|
||||
"UpdateResourceRequest",
|
||||
"ResourceResponse",
|
||||
]
|
||||
@@ -0,0 +1,96 @@
|
||||
"""V2 entity schemas with ID-first design."""
|
||||
|
||||
from datetime import datetime
|
||||
from typing import Dict, List, Literal, Optional
|
||||
|
||||
from pydantic import BaseModel, Field, ConfigDict
|
||||
|
||||
from basic_memory.schemas.response import ObservationResponse, RelationResponse
|
||||
|
||||
|
||||
class EntityResolveRequest(BaseModel):
|
||||
"""Request to resolve a string identifier to an entity ID.
|
||||
|
||||
Supports resolution of:
|
||||
- Permalinks (e.g., "specs/search")
|
||||
- Titles (e.g., "Search Specification")
|
||||
- File paths (e.g., "specs/search.md")
|
||||
"""
|
||||
|
||||
identifier: str = Field(
|
||||
...,
|
||||
description="Entity identifier to resolve (permalink, title, or file path)",
|
||||
min_length=1,
|
||||
max_length=500,
|
||||
)
|
||||
|
||||
|
||||
class EntityResolveResponse(BaseModel):
|
||||
"""Response from identifier resolution.
|
||||
|
||||
Returns the entity ID and associated metadata for the resolved entity.
|
||||
"""
|
||||
|
||||
entity_id: int = Field(..., description="Numeric entity ID (primary identifier)")
|
||||
permalink: Optional[str] = Field(None, description="Entity permalink")
|
||||
file_path: str = Field(..., description="Relative file path")
|
||||
title: str = Field(..., description="Entity title")
|
||||
resolution_method: Literal["id", "permalink", "title", "path", "search"] = Field(
|
||||
..., description="How the identifier was resolved"
|
||||
)
|
||||
|
||||
|
||||
class MoveEntityRequestV2(BaseModel):
|
||||
"""V2 request schema for moving an entity to a new file location.
|
||||
|
||||
In V2 API, the entity ID is provided in the URL path, so this request
|
||||
only needs the destination path.
|
||||
"""
|
||||
|
||||
destination_path: str = Field(
|
||||
...,
|
||||
description="New file path for the entity (relative to project root)",
|
||||
min_length=1,
|
||||
max_length=500,
|
||||
)
|
||||
|
||||
|
||||
class EntityResponseV2(BaseModel):
|
||||
"""V2 entity response with ID as the primary field.
|
||||
|
||||
This response format emphasizes the entity ID as the primary identifier,
|
||||
with all other fields (permalink, file_path) as secondary metadata.
|
||||
"""
|
||||
|
||||
# ID first - this is the primary identifier in v2
|
||||
id: int = Field(..., description="Numeric entity ID (primary identifier)")
|
||||
|
||||
# Core entity fields
|
||||
title: str = Field(..., description="Entity title")
|
||||
entity_type: str = Field(..., description="Entity type")
|
||||
content_type: str = Field(default="text/markdown", description="Content MIME type")
|
||||
|
||||
# Secondary identifiers (for compatibility and convenience)
|
||||
permalink: Optional[str] = Field(None, description="Entity permalink (may change)")
|
||||
file_path: str = Field(..., description="Relative file path (may change)")
|
||||
|
||||
# Content and metadata
|
||||
content: Optional[str] = Field(None, description="Entity content")
|
||||
entity_metadata: Optional[Dict] = Field(None, description="Entity metadata")
|
||||
|
||||
# Relationships
|
||||
observations: List[ObservationResponse] = Field(
|
||||
default_factory=list, description="Entity observations"
|
||||
)
|
||||
relations: List[RelationResponse] = Field(default_factory=list, description="Entity relations")
|
||||
|
||||
# Timestamps
|
||||
created_at: datetime = Field(..., description="Creation timestamp")
|
||||
updated_at: datetime = Field(..., description="Last update timestamp")
|
||||
|
||||
# V2-specific metadata
|
||||
api_version: Literal["v2"] = Field(
|
||||
default="v2", description="API version (always 'v2' for this response)"
|
||||
)
|
||||
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
@@ -0,0 +1,46 @@
|
||||
"""V2 resource schemas for file content operations."""
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class CreateResourceRequest(BaseModel):
|
||||
"""Request to create a new resource file.
|
||||
|
||||
File path is required for new resources since we need to know where
|
||||
to create the file.
|
||||
"""
|
||||
|
||||
file_path: str = Field(
|
||||
...,
|
||||
description="Path to create the file, relative to project root",
|
||||
min_length=1,
|
||||
max_length=500,
|
||||
)
|
||||
content: str = Field(..., description="File content to write")
|
||||
|
||||
|
||||
class UpdateResourceRequest(BaseModel):
|
||||
"""Request to update an existing resource by entity ID.
|
||||
|
||||
Only content is required - the file path is already known from the entity.
|
||||
Optionally can update the file_path to move the file.
|
||||
"""
|
||||
|
||||
content: str = Field(..., description="File content to write")
|
||||
file_path: str | None = Field(
|
||||
None,
|
||||
description="Optional new file path to move the resource",
|
||||
min_length=1,
|
||||
max_length=500,
|
||||
)
|
||||
|
||||
|
||||
class ResourceResponse(BaseModel):
|
||||
"""Response from resource operations."""
|
||||
|
||||
entity_id: int = Field(..., description="Entity ID of the resource")
|
||||
file_path: str = Field(..., description="File path of the resource")
|
||||
checksum: str = Field(..., description="File content checksum")
|
||||
size: int = Field(..., description="File size in bytes")
|
||||
created_at: float = Field(..., description="Creation timestamp")
|
||||
modified_at: float = Field(..., description="Modification timestamp")
|
||||
@@ -4,6 +4,7 @@ from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from typing import List, Optional, Tuple
|
||||
|
||||
|
||||
from loguru import logger
|
||||
from sqlalchemy import text
|
||||
|
||||
|
||||
@@ -3,8 +3,10 @@
|
||||
import fnmatch
|
||||
import logging
|
||||
import os
|
||||
from datetime import datetime
|
||||
from typing import Dict, List, Optional, Sequence
|
||||
|
||||
|
||||
from basic_memory.models import Entity
|
||||
from basic_memory.repository import EntityRepository
|
||||
from basic_memory.schemas.directory import DirectoryNode
|
||||
@@ -12,6 +14,17 @@ from basic_memory.schemas.directory import DirectoryNode
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _mtime_to_datetime(entity: Entity) -> datetime:
|
||||
"""Convert entity mtime (file modification time) to datetime.
|
||||
|
||||
Returns the file's actual modification time, falling back to updated_at
|
||||
if mtime is not available.
|
||||
"""
|
||||
if entity.mtime:
|
||||
return datetime.fromtimestamp(entity.mtime).astimezone()
|
||||
return entity.updated_at
|
||||
|
||||
|
||||
class DirectoryService:
|
||||
"""Service for working with directory trees."""
|
||||
|
||||
@@ -77,7 +90,7 @@ class DirectoryService:
|
||||
entity_id=file.id,
|
||||
entity_type=file.entity_type,
|
||||
content_type=file.content_type,
|
||||
updated_at=file.updated_at,
|
||||
updated_at=_mtime_to_datetime(file),
|
||||
)
|
||||
|
||||
# Add to parent directory's children
|
||||
@@ -241,7 +254,7 @@ class DirectoryService:
|
||||
entity_id=file.id,
|
||||
entity_type=file.entity_type,
|
||||
content_type=file.content_type,
|
||||
updated_at=file.updated_at,
|
||||
updated_at=_mtime_to_datetime(file),
|
||||
)
|
||||
|
||||
# Add to parent directory's children
|
||||
|
||||
@@ -8,6 +8,7 @@ import yaml
|
||||
from loguru import logger
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
|
||||
|
||||
from basic_memory.config import ProjectConfig, BasicMemoryConfig
|
||||
from basic_memory.file_utils import (
|
||||
has_frontmatter,
|
||||
@@ -106,6 +107,9 @@ class EntityService(BaseService[EntityModel]):
|
||||
4. Generate new unique permalink from file path
|
||||
|
||||
Enhanced to detect and handle character-related conflicts.
|
||||
|
||||
Note: Uses lightweight repository methods that skip eager loading of
|
||||
observations and relations for better performance during bulk operations.
|
||||
"""
|
||||
file_path_str = Path(file_path).as_posix()
|
||||
|
||||
@@ -122,16 +126,20 @@ class EntityService(BaseService[EntityModel]):
|
||||
# If markdown has explicit permalink, try to validate it
|
||||
if markdown and markdown.frontmatter.permalink:
|
||||
desired_permalink = markdown.frontmatter.permalink
|
||||
existing = await self.repository.get_by_permalink(desired_permalink)
|
||||
# Use lightweight method - we only need to check file_path
|
||||
existing_file_path = await self.repository.get_file_path_for_permalink(
|
||||
desired_permalink
|
||||
)
|
||||
|
||||
# If no conflict or it's our own file, use as is
|
||||
if not existing or existing.file_path == file_path_str:
|
||||
if not existing_file_path or existing_file_path == file_path_str:
|
||||
return desired_permalink
|
||||
|
||||
# For existing files, try to find current permalink
|
||||
existing = await self.repository.get_by_file_path(file_path_str)
|
||||
if existing:
|
||||
return existing.permalink
|
||||
# Use lightweight method - we only need the permalink
|
||||
existing_permalink = await self.repository.get_permalink_for_file_path(file_path_str)
|
||||
if existing_permalink:
|
||||
return existing_permalink
|
||||
|
||||
# New file - generate permalink
|
||||
if markdown and markdown.frontmatter.permalink:
|
||||
@@ -140,9 +148,10 @@ class EntityService(BaseService[EntityModel]):
|
||||
desired_permalink = generate_permalink(file_path_str)
|
||||
|
||||
# Make unique if needed - enhanced to handle character conflicts
|
||||
# Use lightweight existence check instead of loading full entity
|
||||
permalink = desired_permalink
|
||||
suffix = 1
|
||||
while await self.repository.get_by_permalink(permalink):
|
||||
while await self.repository.permalink_exists(permalink):
|
||||
permalink = f"{desired_permalink}-{suffix}"
|
||||
suffix += 1
|
||||
logger.debug(f"creating unique permalink: {permalink}")
|
||||
@@ -224,8 +233,11 @@ class EntityService(BaseService[EntityModel]):
|
||||
final_content = dump_frontmatter(post)
|
||||
checksum = await self.file_service.write_file(file_path, final_content)
|
||||
|
||||
# parse entity from file
|
||||
entity_markdown = await self.entity_parser.parse_file(file_path)
|
||||
# parse entity from content we just wrote (avoids re-reading file for cloud compatibility)
|
||||
entity_markdown = await self.entity_parser.parse_markdown_content(
|
||||
file_path=file_path,
|
||||
content=final_content,
|
||||
)
|
||||
|
||||
# create entity
|
||||
created = await self.create_entity_from_markdown(file_path, entity_markdown)
|
||||
@@ -245,8 +257,12 @@ class EntityService(BaseService[EntityModel]):
|
||||
# Convert file path string to Path
|
||||
file_path = Path(entity.file_path)
|
||||
|
||||
# Read existing frontmatter from the file if it exists
|
||||
existing_markdown = await self.entity_parser.parse_file(file_path)
|
||||
# Read existing content via file_service (for cloud compatibility)
|
||||
existing_content = await self.file_service.read_file_content(file_path)
|
||||
existing_markdown = await self.entity_parser.parse_markdown_content(
|
||||
file_path=file_path,
|
||||
content=existing_content,
|
||||
)
|
||||
|
||||
# Parse content frontmatter to check for user-specified permalink and entity_type
|
||||
content_markdown = None
|
||||
@@ -302,8 +318,11 @@ class EntityService(BaseService[EntityModel]):
|
||||
final_content = dump_frontmatter(merged_post)
|
||||
checksum = await self.file_service.write_file(file_path, final_content)
|
||||
|
||||
# parse entity from file
|
||||
entity_markdown = await self.entity_parser.parse_file(file_path)
|
||||
# parse entity from content we just wrote (avoids re-reading file for cloud compatibility)
|
||||
entity_markdown = await self.entity_parser.parse_markdown_content(
|
||||
file_path=file_path,
|
||||
content=final_content,
|
||||
)
|
||||
|
||||
# update entity in db
|
||||
entity = await self.update_entity_and_observations(file_path, entity_markdown)
|
||||
@@ -378,7 +397,9 @@ class EntityService(BaseService[EntityModel]):
|
||||
Uses UPSERT approach to handle permalink/file_path conflicts cleanly.
|
||||
"""
|
||||
logger.debug(f"Creating entity: {markdown.frontmatter.title} file_path: {file_path}")
|
||||
model = entity_model_from_markdown(file_path, markdown)
|
||||
model = entity_model_from_markdown(
|
||||
file_path, markdown, project_id=self.repository.project_id
|
||||
)
|
||||
|
||||
# Mark as incomplete because we still need to add relations
|
||||
model.checksum = None
|
||||
@@ -408,6 +429,7 @@ class EntityService(BaseService[EntityModel]):
|
||||
# add new observations
|
||||
observations = [
|
||||
Observation(
|
||||
project_id=self.observation_repository.project_id,
|
||||
entity_id=db_entity.id,
|
||||
content=obs.content,
|
||||
category=obs.category,
|
||||
@@ -448,8 +470,11 @@ class EntityService(BaseService[EntityModel]):
|
||||
import asyncio
|
||||
|
||||
# Create tasks for all relation lookups
|
||||
# Use strict=True to disable fuzzy search - only exact matches should create resolved relations
|
||||
# This ensures forward references (links to non-existent entities) remain unresolved (to_id=NULL)
|
||||
lookup_tasks = [
|
||||
self.link_resolver.resolve_link(rel.target) for rel in markdown.relations
|
||||
self.link_resolver.resolve_link(rel.target, strict=True)
|
||||
for rel in markdown.relations
|
||||
]
|
||||
|
||||
# Execute all lookups in parallel
|
||||
@@ -471,6 +496,7 @@ class EntityService(BaseService[EntityModel]):
|
||||
|
||||
# Create the relation
|
||||
relation = Relation(
|
||||
project_id=self.relation_repository.project_id,
|
||||
from_id=db_entity.id,
|
||||
to_id=target_id,
|
||||
to_name=target_name,
|
||||
@@ -543,8 +569,11 @@ class EntityService(BaseService[EntityModel]):
|
||||
# Write the updated content back to the file
|
||||
checksum = await self.file_service.write_file(file_path, new_content)
|
||||
|
||||
# Parse the updated file to get new observations/relations
|
||||
entity_markdown = await self.entity_parser.parse_file(file_path)
|
||||
# Parse the content we just wrote (avoids re-reading file for cloud compatibility)
|
||||
entity_markdown = await self.entity_parser.parse_markdown_content(
|
||||
file_path=file_path,
|
||||
content=new_content,
|
||||
)
|
||||
|
||||
# Update entity and its relationships
|
||||
entity = await self.update_entity_and_observations(file_path, entity_markdown)
|
||||
@@ -763,23 +792,20 @@ class EntityService(BaseService[EntityModel]):
|
||||
raise ValueError(f"Invalid destination path: {destination_path}")
|
||||
|
||||
# 3. Validate paths
|
||||
source_file = project_config.home / current_path
|
||||
destination_file = project_config.home / destination_path
|
||||
|
||||
# Validate source exists
|
||||
if not source_file.exists():
|
||||
# NOTE: In tenantless/cloud mode, we cannot rely on local filesystem paths.
|
||||
# Use FileService for existence checks and moving.
|
||||
if not await self.file_service.exists(current_path):
|
||||
raise ValueError(f"Source file not found: {current_path}")
|
||||
|
||||
# Check if destination already exists
|
||||
if destination_file.exists():
|
||||
if await self.file_service.exists(destination_path):
|
||||
raise ValueError(f"Destination already exists: {destination_path}")
|
||||
|
||||
try:
|
||||
# 4. Create destination directory if needed
|
||||
destination_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
# 4. Ensure destination directory if needed (no-op for S3)
|
||||
await self.file_service.ensure_directory(Path(destination_path).parent)
|
||||
|
||||
# 5. Move physical file
|
||||
source_file.rename(destination_file)
|
||||
# 5. Move physical file via FileService (filesystem rename or cloud move)
|
||||
await self.file_service.move_file(current_path, destination_path)
|
||||
logger.info(f"Moved file: {current_path} -> {destination_path}")
|
||||
|
||||
# 6. Prepare database updates
|
||||
@@ -818,12 +844,14 @@ class EntityService(BaseService[EntityModel]):
|
||||
|
||||
except Exception as e:
|
||||
# Rollback: try to restore original file location if move succeeded
|
||||
if destination_file.exists() and not source_file.exists():
|
||||
try:
|
||||
destination_file.rename(source_file)
|
||||
try:
|
||||
if await self.file_service.exists(
|
||||
destination_path
|
||||
) and not await self.file_service.exists(current_path):
|
||||
await self.file_service.move_file(destination_path, current_path)
|
||||
logger.info(f"Rolled back file move: {destination_path} -> {current_path}")
|
||||
except Exception as rollback_error: # pragma: no cover
|
||||
logger.error(f"Failed to rollback file move: {rollback_error}")
|
||||
except Exception as rollback_error: # pragma: no cover
|
||||
logger.error(f"Failed to rollback file move: {rollback_error}")
|
||||
|
||||
# Re-raise the original error with context
|
||||
raise ValueError(f"Move failed: {str(e)}") from e
|
||||
|
||||
@@ -3,15 +3,16 @@
|
||||
import asyncio
|
||||
import hashlib
|
||||
import mimetypes
|
||||
from os import stat_result
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Tuple, Union
|
||||
|
||||
import aiofiles
|
||||
|
||||
import yaml
|
||||
|
||||
from basic_memory import file_utils
|
||||
from basic_memory.file_utils import FileError, ParseError
|
||||
from basic_memory.file_utils import FileError, FileMetadata, ParseError
|
||||
from basic_memory.markdown.markdown_processor import MarkdownProcessor
|
||||
from basic_memory.models import Entity as EntityModel
|
||||
from basic_memory.schemas import Entity as EntitySchema
|
||||
@@ -220,6 +221,41 @@ class FileService:
|
||||
logger.exception("File read error", path=str(full_path), error=str(e))
|
||||
raise FileOperationError(f"Failed to read file: {e}")
|
||||
|
||||
async def read_file_bytes(self, path: FilePath) -> bytes:
|
||||
"""Read file content as bytes using true async I/O with aiofiles.
|
||||
|
||||
This method reads files in binary mode, suitable for non-text files
|
||||
like images, PDFs, etc. For cloud compatibility with S3FileService.
|
||||
|
||||
Args:
|
||||
path: Path to read (Path or string)
|
||||
|
||||
Returns:
|
||||
File content as bytes
|
||||
|
||||
Raises:
|
||||
FileOperationError: If read fails
|
||||
"""
|
||||
# Convert string to Path if needed
|
||||
path_obj = self.base_path / path if isinstance(path, str) else path
|
||||
full_path = path_obj if path_obj.is_absolute() else self.base_path / path_obj
|
||||
|
||||
try:
|
||||
logger.debug("Reading file bytes", operation="read_file_bytes", path=str(full_path))
|
||||
async with aiofiles.open(full_path, mode="rb") as f:
|
||||
content = await f.read()
|
||||
|
||||
logger.debug(
|
||||
"File read completed",
|
||||
path=str(full_path),
|
||||
content_length=len(content),
|
||||
)
|
||||
return content
|
||||
|
||||
except Exception as e:
|
||||
logger.exception("File read error", path=str(full_path), error=str(e))
|
||||
raise FileOperationError(f"Failed to read file: {e}")
|
||||
|
||||
async def read_file(self, path: FilePath) -> Tuple[str, str]:
|
||||
"""Read file and compute checksum using true async I/O.
|
||||
|
||||
@@ -276,6 +312,43 @@ class FileService:
|
||||
full_path = path_obj if path_obj.is_absolute() else self.base_path / path_obj
|
||||
full_path.unlink(missing_ok=True)
|
||||
|
||||
async def move_file(self, source: FilePath, destination: FilePath) -> None:
|
||||
"""Move/rename a file from source to destination.
|
||||
|
||||
This method abstracts the underlying storage (filesystem vs cloud).
|
||||
Default implementation uses atomic filesystem rename, but cloud-backed
|
||||
implementations (e.g., S3) can override to copy+delete.
|
||||
|
||||
Args:
|
||||
source: Source path (relative to base_path or absolute)
|
||||
destination: Destination path (relative to base_path or absolute)
|
||||
|
||||
Raises:
|
||||
FileOperationError: If the move fails
|
||||
"""
|
||||
# Convert strings to Paths and resolve relative paths against base_path
|
||||
src_obj = self.base_path / source if isinstance(source, str) else source
|
||||
dst_obj = self.base_path / destination if isinstance(destination, str) else destination
|
||||
src_full = src_obj if src_obj.is_absolute() else self.base_path / src_obj
|
||||
dst_full = dst_obj if dst_obj.is_absolute() else self.base_path / dst_obj
|
||||
|
||||
try:
|
||||
# Ensure destination directory exists
|
||||
await self.ensure_directory(dst_full.parent)
|
||||
|
||||
# Use semaphore for concurrency control and run blocking rename in executor
|
||||
async with self._file_semaphore:
|
||||
loop = asyncio.get_event_loop()
|
||||
await loop.run_in_executor(None, lambda: src_full.rename(dst_full))
|
||||
except Exception as e:
|
||||
logger.exception(
|
||||
"File move error",
|
||||
source=str(src_full),
|
||||
destination=str(dst_full),
|
||||
error=str(e),
|
||||
)
|
||||
raise FileOperationError(f"Failed to move file {source} -> {destination}: {e}")
|
||||
|
||||
async def update_frontmatter(self, path: FilePath, updates: Dict[str, Any]) -> str:
|
||||
"""Update frontmatter fields in a file while preserving all content.
|
||||
|
||||
@@ -381,20 +454,31 @@ class FileService:
|
||||
logger.error("Failed to compute checksum", path=str(full_path), error=str(e))
|
||||
raise FileError(f"Failed to compute checksum for {path}: {e}")
|
||||
|
||||
def file_stats(self, path: FilePath) -> stat_result:
|
||||
"""Return file stats for a given path.
|
||||
async def get_file_metadata(self, path: FilePath) -> FileMetadata:
|
||||
"""Return file metadata for a given path.
|
||||
|
||||
This method is async to support cloud implementations (S3FileService)
|
||||
where file metadata requires async operations (head_object).
|
||||
|
||||
Args:
|
||||
path: Path to the file (Path or string)
|
||||
|
||||
Returns:
|
||||
File statistics
|
||||
FileMetadata with size, created_at, and modified_at
|
||||
"""
|
||||
# Convert string to Path if needed
|
||||
path_obj = self.base_path / path if isinstance(path, str) else path
|
||||
full_path = path_obj if path_obj.is_absolute() else self.base_path / path_obj
|
||||
# get file timestamps
|
||||
return full_path.stat()
|
||||
|
||||
# Run blocking stat() in thread pool to maintain async compatibility
|
||||
loop = asyncio.get_event_loop()
|
||||
stat_result = await loop.run_in_executor(None, full_path.stat)
|
||||
|
||||
return FileMetadata(
|
||||
size=stat_result.st_size,
|
||||
created_at=datetime.fromtimestamp(stat_result.st_ctime).astimezone(),
|
||||
modified_at=datetime.fromtimestamp(stat_result.st_mtime).astimezone(),
|
||||
)
|
||||
|
||||
def content_type(self, path: FilePath) -> str:
|
||||
"""Return content_type for a given path.
|
||||
|
||||
@@ -5,8 +5,10 @@ to ensure consistent application startup across all entry points.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory import db
|
||||
@@ -104,6 +106,12 @@ async def initialize_file_sync(
|
||||
# Get active projects
|
||||
active_projects = await project_repository.get_active_projects()
|
||||
|
||||
# Filter to constrained project if MCP server was started with --project
|
||||
constrained_project = os.environ.get("BASIC_MEMORY_MCP_PROJECT")
|
||||
if constrained_project:
|
||||
active_projects = [p for p in active_projects if p.name == constrained_project]
|
||||
logger.info(f"Background sync constrained to project: {constrained_project}")
|
||||
|
||||
# Start sync for all projects as background tasks (non-blocking)
|
||||
async def sync_project_background(project: Project):
|
||||
"""Sync a single project in the background."""
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
from typing import Optional, Tuple
|
||||
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory.models import Entity
|
||||
|
||||
@@ -8,6 +8,7 @@ from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Dict, Optional, Sequence
|
||||
|
||||
|
||||
from loguru import logger
|
||||
from sqlalchemy import text
|
||||
|
||||
@@ -23,9 +24,6 @@ from basic_memory.config import WATCH_STATUS_JSON, ConfigManager, get_project_co
|
||||
from basic_memory.utils import generate_permalink
|
||||
|
||||
|
||||
config = ConfigManager().config
|
||||
|
||||
|
||||
class ProjectService:
|
||||
"""Service for managing Basic Memory projects."""
|
||||
|
||||
@@ -143,6 +141,7 @@ class ProjectService:
|
||||
"""
|
||||
# If project_root is set, constrain all projects to that directory
|
||||
project_root = self.config_manager.config.project_root
|
||||
sanitized_name = None
|
||||
if project_root:
|
||||
base_path = Path(project_root)
|
||||
|
||||
@@ -199,14 +198,15 @@ class ProjectService:
|
||||
f"Projects cannot share directory trees."
|
||||
)
|
||||
|
||||
# First add to config file (this will validate the project doesn't exist)
|
||||
project_config = self.config_manager.add_project(name, resolved_path)
|
||||
if not self.config_manager.config.cloud_mode:
|
||||
# First add to config file (this will validate the project doesn't exist)
|
||||
self.config_manager.add_project(name, resolved_path)
|
||||
|
||||
# Then add to database
|
||||
project_data = {
|
||||
"name": name,
|
||||
"path": resolved_path,
|
||||
"permalink": generate_permalink(project_config.name),
|
||||
"permalink": sanitized_name,
|
||||
"is_active": True,
|
||||
# Don't set is_default=False to avoid UNIQUE constraint issues
|
||||
# Let it default to NULL, only set to True when explicitly making default
|
||||
|
||||
@@ -4,6 +4,7 @@ import ast
|
||||
from datetime import datetime
|
||||
from typing import List, Optional, Set
|
||||
|
||||
|
||||
from dateparser import parse
|
||||
from fastapi import BackgroundTasks
|
||||
from loguru import logger
|
||||
@@ -15,6 +16,21 @@ from basic_memory.repository.search_repository import SearchRepository, SearchIn
|
||||
from basic_memory.schemas.search import SearchQuery, SearchItemType
|
||||
from basic_memory.services import FileService
|
||||
|
||||
# Maximum size for content_stems field to stay under Postgres's 8KB index row limit.
|
||||
# We use 6000 characters to leave headroom for other indexed columns and overhead.
|
||||
MAX_CONTENT_STEMS_SIZE = 6000
|
||||
|
||||
|
||||
def _mtime_to_datetime(entity: Entity) -> datetime:
|
||||
"""Convert entity mtime (file modification time) to datetime.
|
||||
|
||||
Returns the file's actual modification time, falling back to updated_at
|
||||
if mtime is not available.
|
||||
"""
|
||||
if entity.mtime:
|
||||
return datetime.fromtimestamp(entity.mtime).astimezone()
|
||||
return entity.updated_at
|
||||
|
||||
|
||||
class SearchService:
|
||||
"""Service for search operations.
|
||||
@@ -156,22 +172,24 @@ class SearchService:
|
||||
self,
|
||||
entity: Entity,
|
||||
background_tasks: Optional[BackgroundTasks] = None,
|
||||
content: str | None = None,
|
||||
) -> None:
|
||||
if background_tasks:
|
||||
background_tasks.add_task(self.index_entity_data, entity)
|
||||
background_tasks.add_task(self.index_entity_data, entity, content)
|
||||
else:
|
||||
await self.index_entity_data(entity)
|
||||
await self.index_entity_data(entity, content)
|
||||
|
||||
async def index_entity_data(
|
||||
self,
|
||||
entity: Entity,
|
||||
content: str | None = None,
|
||||
) -> None:
|
||||
# delete all search index data associated with entity
|
||||
await self.repository.delete_by_entity_id(entity_id=entity.id)
|
||||
|
||||
# reindex
|
||||
await self.index_entity_markdown(
|
||||
entity
|
||||
entity, content
|
||||
) if entity.is_markdown else await self.index_entity_file(entity)
|
||||
|
||||
async def index_entity_file(
|
||||
@@ -191,7 +209,7 @@ class SearchService:
|
||||
"entity_type": entity.entity_type,
|
||||
},
|
||||
created_at=entity.created_at,
|
||||
updated_at=entity.updated_at,
|
||||
updated_at=_mtime_to_datetime(entity),
|
||||
project_id=entity.project_id,
|
||||
)
|
||||
)
|
||||
@@ -199,9 +217,14 @@ class SearchService:
|
||||
async def index_entity_markdown(
|
||||
self,
|
||||
entity: Entity,
|
||||
content: str | None = None,
|
||||
) -> None:
|
||||
"""Index an entity and all its observations and relations.
|
||||
|
||||
Args:
|
||||
entity: The entity to index
|
||||
content: Optional pre-loaded content (avoids file read). If None, will read from file.
|
||||
|
||||
Indexing structure:
|
||||
1. Entities
|
||||
- permalink: direct from entity (e.g., "specs/search")
|
||||
@@ -230,7 +253,9 @@ class SearchService:
|
||||
title_variants = self._generate_variants(entity.title)
|
||||
content_stems.extend(title_variants)
|
||||
|
||||
content = await self.file_service.read_entity_content(entity)
|
||||
# Use provided content or read from file
|
||||
if content is None:
|
||||
content = await self.file_service.read_entity_content(entity)
|
||||
if content:
|
||||
content_stems.append(content)
|
||||
content_snippet = f"{content[:250]}"
|
||||
@@ -247,6 +272,10 @@ class SearchService:
|
||||
|
||||
entity_content_stems = "\n".join(p for p in content_stems if p and p.strip())
|
||||
|
||||
# Truncate to stay under Postgres's 8KB index row limit
|
||||
if len(entity_content_stems) > MAX_CONTENT_STEMS_SIZE:
|
||||
entity_content_stems = entity_content_stems[:MAX_CONTENT_STEMS_SIZE]
|
||||
|
||||
# Add entity row
|
||||
rows_to_index.append(
|
||||
SearchIndexRow(
|
||||
@@ -262,17 +291,28 @@ class SearchService:
|
||||
"entity_type": entity.entity_type,
|
||||
},
|
||||
created_at=entity.created_at,
|
||||
updated_at=entity.updated_at,
|
||||
updated_at=_mtime_to_datetime(entity),
|
||||
project_id=entity.project_id,
|
||||
)
|
||||
)
|
||||
|
||||
# Add observation rows
|
||||
# Add observation rows - dedupe by permalink to avoid unique constraint violations
|
||||
# Two observations with same entity/category/content generate identical permalinks
|
||||
seen_permalinks: set[str] = {entity.permalink} if entity.permalink else set()
|
||||
for obs in entity.observations:
|
||||
obs_permalink = obs.permalink
|
||||
if obs_permalink in seen_permalinks:
|
||||
logger.debug(f"Skipping duplicate observation permalink: {obs_permalink}")
|
||||
continue
|
||||
seen_permalinks.add(obs_permalink)
|
||||
|
||||
# Index with parent entity's file path since that's where it's defined
|
||||
obs_content_stems = "\n".join(
|
||||
p for p in self._generate_variants(obs.content) if p and p.strip()
|
||||
)
|
||||
# Truncate to stay under Postgres's 8KB index row limit
|
||||
if len(obs_content_stems) > MAX_CONTENT_STEMS_SIZE:
|
||||
obs_content_stems = obs_content_stems[:MAX_CONTENT_STEMS_SIZE]
|
||||
rows_to_index.append(
|
||||
SearchIndexRow(
|
||||
id=obs.id,
|
||||
@@ -280,7 +320,7 @@ class SearchService:
|
||||
title=f"{obs.category}: {obs.content[:100]}...",
|
||||
content_stems=obs_content_stems,
|
||||
content_snippet=obs.content,
|
||||
permalink=obs.permalink,
|
||||
permalink=obs_permalink,
|
||||
file_path=entity.file_path,
|
||||
category=obs.category,
|
||||
entity_id=entity.id,
|
||||
@@ -288,7 +328,7 @@ class SearchService:
|
||||
"tags": obs.tags,
|
||||
},
|
||||
created_at=entity.created_at,
|
||||
updated_at=entity.updated_at,
|
||||
updated_at=_mtime_to_datetime(entity),
|
||||
project_id=entity.project_id,
|
||||
)
|
||||
)
|
||||
@@ -318,7 +358,7 @@ class SearchService:
|
||||
to_id=rel.to_id,
|
||||
relation_type=rel.relation_type,
|
||||
created_at=entity.created_at,
|
||||
updated_at=entity.updated_at,
|
||||
updated_at=_mtime_to_datetime(entity),
|
||||
project_id=entity.project_id,
|
||||
)
|
||||
)
|
||||
|
||||
@@ -10,7 +10,7 @@ from pathlib import Path
|
||||
from typing import AsyncIterator, Dict, List, Optional, Set, Tuple
|
||||
|
||||
import aiofiles.os
|
||||
import logfire
|
||||
|
||||
from loguru import logger
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
|
||||
@@ -31,7 +31,6 @@ from basic_memory.services import EntityService, FileService
|
||||
from basic_memory.services.exceptions import SyncFatalError
|
||||
from basic_memory.services.link_resolver import LinkResolver
|
||||
from basic_memory.services.search_service import SearchService
|
||||
from basic_memory.sync.utils import chunks
|
||||
|
||||
# Circuit breaker configuration
|
||||
MAX_CONSECUTIVE_FAILURES = 3
|
||||
@@ -216,17 +215,12 @@ class SyncService:
|
||||
f"path={path}, error={error}"
|
||||
)
|
||||
|
||||
# Record metric for file failure
|
||||
logfire.metric_counter("sync.circuit_breaker.failures").add(1)
|
||||
|
||||
# Log when threshold is reached
|
||||
if failure_info.count >= MAX_CONSECUTIVE_FAILURES:
|
||||
logger.error(
|
||||
f"File {path} has failed {MAX_CONSECUTIVE_FAILURES} times and will be skipped. "
|
||||
f"First failure: {failure_info.first_failure}, Last error: {error}"
|
||||
)
|
||||
# Record metric for file being blocked by circuit breaker
|
||||
logfire.metric_counter("sync.circuit_breaker.blocked_files").add(1)
|
||||
else:
|
||||
# Create new failure record
|
||||
self._file_failures[path] = FileFailureInfo(
|
||||
@@ -256,7 +250,6 @@ class SyncService:
|
||||
logger.info(f"Clearing failure history for {path} after successful sync")
|
||||
del self._file_failures[path]
|
||||
|
||||
@logfire.instrument()
|
||||
async def sync(
|
||||
self, directory: Path, project_name: Optional[str] = None, force_full: bool = False
|
||||
) -> SyncReport:
|
||||
@@ -283,164 +276,58 @@ class SyncService:
|
||||
)
|
||||
|
||||
# sync moves first
|
||||
with logfire.span("process_moves", move_count=len(report.moves)):
|
||||
for old_path, new_path in report.moves.items():
|
||||
# in the case where a file has been deleted and replaced by another file
|
||||
# it will show up in the move and modified lists, so handle it in modified
|
||||
if new_path in report.modified:
|
||||
report.modified.remove(new_path)
|
||||
logger.debug(
|
||||
f"File marked as moved and modified: old_path={old_path}, new_path={new_path}"
|
||||
)
|
||||
else:
|
||||
await self.handle_move(old_path, new_path)
|
||||
for old_path, new_path in report.moves.items():
|
||||
# in the case where a file has been deleted and replaced by another file
|
||||
# it will show up in the move and modified lists, so handle it in modified
|
||||
if new_path in report.modified:
|
||||
report.modified.remove(new_path)
|
||||
logger.debug(
|
||||
f"File marked as moved and modified: old_path={old_path}, new_path={new_path}"
|
||||
)
|
||||
else:
|
||||
await self.handle_move(old_path, new_path)
|
||||
|
||||
# deleted next
|
||||
with logfire.span("process_deletes", delete_count=len(report.deleted)):
|
||||
for path in report.deleted:
|
||||
await self.handle_delete(path)
|
||||
for path in report.deleted:
|
||||
await self.handle_delete(path)
|
||||
|
||||
# then new and modified - process in batches for better performance
|
||||
batch_size = self.app_config.sync_batch_size
|
||||
logger.debug(f"Using batch size of {batch_size} for file processing")
|
||||
# then new and modified
|
||||
for path in report.new:
|
||||
entity, _ = await self.sync_file(path, new=True)
|
||||
|
||||
with logfire.span("process_new_files", new_count=len(report.new)):
|
||||
# Convert set to list for batching
|
||||
new_files_list = list(report.new)
|
||||
# Track if file was skipped
|
||||
if entity is None and await self._should_skip_file(path):
|
||||
failure_info = self._file_failures[path]
|
||||
report.skipped_files.append(
|
||||
SkippedFile(
|
||||
path=path,
|
||||
reason=failure_info.last_error,
|
||||
failure_count=failure_info.count,
|
||||
first_failed=failure_info.first_failure,
|
||||
)
|
||||
)
|
||||
|
||||
for batch in chunks(new_files_list, batch_size):
|
||||
logger.debug(f"Processing batch of {len(batch)} new files")
|
||||
for path in report.modified:
|
||||
entity, _ = await self.sync_file(path, new=False)
|
||||
|
||||
# Separate markdown and non-markdown files
|
||||
markdown_files = [p for p in batch if self.file_service.is_markdown(p)]
|
||||
regular_files = [p for p in batch if not self.file_service.is_markdown(p)]
|
||||
|
||||
# Batch process markdown files
|
||||
if markdown_files:
|
||||
try:
|
||||
batch_results = await self.sync_markdown_batch(markdown_files, new=True)
|
||||
|
||||
# Track skipped files
|
||||
for path, (entity, _) in zip(markdown_files, batch_results):
|
||||
if entity is None and await self._should_skip_file(path):
|
||||
failure_info = self._file_failures[path]
|
||||
report.skipped_files.append(
|
||||
SkippedFile(
|
||||
path=path,
|
||||
reason=failure_info.last_error,
|
||||
failure_count=failure_info.count,
|
||||
first_failed=failure_info.first_failure,
|
||||
)
|
||||
)
|
||||
|
||||
except SyncFatalError:
|
||||
# Re-raise fatal errors immediately
|
||||
raise
|
||||
except Exception as e:
|
||||
# Batch method raised an exception - record failure for all files in batch
|
||||
logger.error(f"Batch sync failed for {len(markdown_files)} files: {e}")
|
||||
for path in markdown_files:
|
||||
await self._record_failure(path, str(e))
|
||||
# Track skipped files
|
||||
if await self._should_skip_file(path):
|
||||
failure_info = self._file_failures[path]
|
||||
report.skipped_files.append(
|
||||
SkippedFile(
|
||||
path=path,
|
||||
reason=failure_info.last_error,
|
||||
failure_count=failure_info.count,
|
||||
first_failed=failure_info.first_failure,
|
||||
)
|
||||
)
|
||||
|
||||
# Process regular files individually (they're already fast)
|
||||
for path in regular_files:
|
||||
entity, _ = await self.sync_file(path, new=True)
|
||||
|
||||
# Track if file was skipped
|
||||
if entity is None and await self._should_skip_file(path):
|
||||
failure_info = self._file_failures[path]
|
||||
report.skipped_files.append(
|
||||
SkippedFile(
|
||||
path=path,
|
||||
reason=failure_info.last_error,
|
||||
failure_count=failure_info.count,
|
||||
first_failed=failure_info.first_failure,
|
||||
)
|
||||
)
|
||||
|
||||
with logfire.span("process_modified_files", modified_count=len(report.modified)):
|
||||
# Convert set to list for batching
|
||||
modified_files_list = list(report.modified)
|
||||
|
||||
for batch in chunks(modified_files_list, batch_size):
|
||||
logger.debug(f"Processing batch of {len(batch)} modified files")
|
||||
|
||||
# Separate markdown and non-markdown files
|
||||
markdown_files = [p for p in batch if self.file_service.is_markdown(p)]
|
||||
regular_files = [p for p in batch if not self.file_service.is_markdown(p)]
|
||||
|
||||
# Batch process markdown files
|
||||
if markdown_files:
|
||||
try:
|
||||
batch_results = await self.sync_markdown_batch(markdown_files, new=False)
|
||||
|
||||
# Track skipped files
|
||||
for path, (entity, _) in zip(markdown_files, batch_results):
|
||||
if entity is None and await self._should_skip_file(path):
|
||||
failure_info = self._file_failures[path]
|
||||
report.skipped_files.append(
|
||||
SkippedFile(
|
||||
path=path,
|
||||
reason=failure_info.last_error,
|
||||
failure_count=failure_info.count,
|
||||
first_failed=failure_info.first_failure,
|
||||
)
|
||||
)
|
||||
|
||||
except SyncFatalError:
|
||||
# Re-raise fatal errors immediately
|
||||
raise
|
||||
except Exception as e:
|
||||
# Batch method raised an exception - record failure for all files in batch
|
||||
logger.error(f"Batch sync failed for {len(markdown_files)} files: {e}")
|
||||
for path in markdown_files:
|
||||
await self._record_failure(path, str(e))
|
||||
# Track skipped files
|
||||
if await self._should_skip_file(path):
|
||||
failure_info = self._file_failures[path]
|
||||
report.skipped_files.append(
|
||||
SkippedFile(
|
||||
path=path,
|
||||
reason=failure_info.last_error,
|
||||
failure_count=failure_info.count,
|
||||
first_failed=failure_info.first_failure,
|
||||
)
|
||||
)
|
||||
|
||||
# Process regular files individually (they're already fast)
|
||||
for path in regular_files:
|
||||
entity, _ = await self.sync_file(path, new=False)
|
||||
|
||||
# Track if file was skipped
|
||||
if entity is None and await self._should_skip_file(path):
|
||||
failure_info = self._file_failures[path]
|
||||
report.skipped_files.append(
|
||||
SkippedFile(
|
||||
path=path,
|
||||
reason=failure_info.last_error,
|
||||
failure_count=failure_info.count,
|
||||
first_failed=failure_info.first_failure,
|
||||
)
|
||||
)
|
||||
# Track if file was skipped
|
||||
if entity is None and await self._should_skip_file(path):
|
||||
failure_info = self._file_failures[path]
|
||||
report.skipped_files.append(
|
||||
SkippedFile(
|
||||
path=path,
|
||||
reason=failure_info.last_error,
|
||||
failure_count=failure_info.count,
|
||||
first_failed=failure_info.first_failure,
|
||||
)
|
||||
)
|
||||
|
||||
# Only resolve relations if there were actual changes
|
||||
# If no files changed, no new unresolved relations could have been created
|
||||
with logfire.span("resolve_relations"):
|
||||
if report.total > 0:
|
||||
await self.resolve_relations()
|
||||
else:
|
||||
logger.info("Skipping relation resolution - no file changes detected")
|
||||
if report.total > 0:
|
||||
await self.resolve_relations()
|
||||
else:
|
||||
logger.info("Skipping relation resolution - no file changes detected")
|
||||
|
||||
# Update scan watermark after successful sync
|
||||
# Use the timestamp from sync start (not end) to ensure we catch files
|
||||
@@ -463,15 +350,6 @@ class SyncService:
|
||||
|
||||
duration_ms = int((time.time() - start_time) * 1000)
|
||||
|
||||
# Record metrics for sync operation
|
||||
logfire.metric_histogram("sync.duration", unit="ms").record(duration_ms)
|
||||
logfire.metric_counter("sync.files.new").add(len(report.new))
|
||||
logfire.metric_counter("sync.files.modified").add(len(report.modified))
|
||||
logfire.metric_counter("sync.files.deleted").add(len(report.deleted))
|
||||
logfire.metric_counter("sync.files.moved").add(len(report.moves))
|
||||
if report.skipped_files:
|
||||
logfire.metric_counter("sync.files.skipped").add(len(report.skipped_files))
|
||||
|
||||
# Log summary with skipped files if any
|
||||
if report.skipped_files:
|
||||
logger.warning(
|
||||
@@ -492,7 +370,6 @@ class SyncService:
|
||||
|
||||
return report
|
||||
|
||||
@logfire.instrument()
|
||||
async def scan(self, directory, force_full: bool = False):
|
||||
"""Smart scan using watermark and file count for large project optimization.
|
||||
|
||||
@@ -574,24 +451,12 @@ class SyncService:
|
||||
logger.warning("No scan watermark available, falling back to full scan")
|
||||
file_paths_to_scan = await self._scan_directory_full(directory)
|
||||
|
||||
# Record scan type metric
|
||||
logfire.metric_counter(f"sync.scan.{scan_type}").add(1)
|
||||
logfire.metric_histogram("sync.scan.files_scanned", unit="files").record(
|
||||
len(file_paths_to_scan)
|
||||
)
|
||||
|
||||
# Step 3: Process each file with mtime-based comparison
|
||||
scanned_paths: Set[str] = set()
|
||||
changed_checksums: Dict[str, str] = {}
|
||||
|
||||
logger.debug(f"Processing {len(file_paths_to_scan)} files with mtime-based comparison")
|
||||
|
||||
# Optimization: Batch fetch all entities for files being scanned
|
||||
# This reduces N queries to 1 batch query (massive performance win for remote DBs)
|
||||
logger.debug(f"Batch fetching entities for {len(file_paths_to_scan)} files")
|
||||
entities_by_path = await self.entity_repository.get_by_file_paths_batch(file_paths_to_scan)
|
||||
logger.debug(f"Found {len(entities_by_path)} existing entities in database")
|
||||
|
||||
for rel_path in file_paths_to_scan:
|
||||
scanned_paths.add(rel_path)
|
||||
|
||||
@@ -603,8 +468,8 @@ class SyncService:
|
||||
|
||||
stat_info = abs_path.stat()
|
||||
|
||||
# O(1) dict lookup instead of database query
|
||||
db_entity = entities_by_path.get(rel_path)
|
||||
# Indexed lookup - single file query (not full table scan)
|
||||
db_entity = await self.entity_repository.get_by_file_path(rel_path)
|
||||
|
||||
if db_entity is None:
|
||||
# New file - need checksum for move detection
|
||||
@@ -697,7 +562,6 @@ class SyncService:
|
||||
report.checksums = changed_checksums
|
||||
|
||||
scan_duration_ms = int((time.time() - scan_start_time) * 1000)
|
||||
logfire.metric_histogram("sync.scan.duration", unit="ms").record(scan_duration_ms)
|
||||
|
||||
logger.info(
|
||||
f"Completed {scan_type} scan for directory {directory} in {scan_duration_ms}ms, "
|
||||
@@ -707,7 +571,6 @@ class SyncService:
|
||||
)
|
||||
return report
|
||||
|
||||
@logfire.instrument()
|
||||
async def sync_file(
|
||||
self, path: str, new: bool = True
|
||||
) -> Tuple[Optional[Entity], Optional[str]]:
|
||||
@@ -762,7 +625,6 @@ class SyncService:
|
||||
|
||||
return None, None
|
||||
|
||||
@logfire.instrument()
|
||||
async def sync_markdown_file(self, path: str, new: bool = True) -> Tuple[Optional[Entity], str]:
|
||||
"""Sync a markdown file with full processing.
|
||||
|
||||
@@ -780,12 +642,19 @@ class SyncService:
|
||||
file_contains_frontmatter = has_frontmatter(file_content)
|
||||
|
||||
# Get file timestamps for tracking modification times
|
||||
file_stats = self.file_service.file_stats(path)
|
||||
created = datetime.fromtimestamp(file_stats.st_ctime).astimezone()
|
||||
modified = datetime.fromtimestamp(file_stats.st_mtime).astimezone()
|
||||
file_metadata = await self.file_service.get_file_metadata(path)
|
||||
created = file_metadata.created_at
|
||||
modified = file_metadata.modified_at
|
||||
|
||||
# entity markdown will always contain front matter, so it can be used up create/update the entity
|
||||
entity_markdown = await self.entity_parser.parse_file(path)
|
||||
# Parse markdown content with file metadata (avoids redundant file read/stat)
|
||||
# This enables cloud implementations (S3FileService) to provide metadata from head_object
|
||||
abs_path = self.file_service.base_path / path
|
||||
entity_markdown = await self.entity_parser.parse_markdown_content(
|
||||
file_path=abs_path,
|
||||
content=file_content,
|
||||
mtime=file_metadata.modified_at.timestamp(),
|
||||
ctime=file_metadata.created_at.timestamp(),
|
||||
)
|
||||
|
||||
# if the file contains frontmatter, resolve a permalink (unless disabled)
|
||||
if file_contains_frontmatter and not self.app_config.disable_permalinks:
|
||||
@@ -831,8 +700,8 @@ class SyncService:
|
||||
"checksum": final_checksum,
|
||||
"created_at": created,
|
||||
"updated_at": modified,
|
||||
"mtime": file_stats.st_mtime,
|
||||
"size": file_stats.st_size,
|
||||
"mtime": file_metadata.modified_at.timestamp(),
|
||||
"size": file_metadata.size,
|
||||
},
|
||||
)
|
||||
|
||||
@@ -845,207 +714,6 @@ class SyncService:
|
||||
# Return the final checksum to ensure everything is consistent
|
||||
return entity, final_checksum
|
||||
|
||||
@logfire.instrument()
|
||||
async def sync_markdown_batch(
|
||||
self, paths: List[str], new: bool = True
|
||||
) -> List[Tuple[Optional[Entity], str]]:
|
||||
"""Sync multiple markdown files in a single batch operation.
|
||||
|
||||
Optimized for remote databases (Postgres) - reduces N queries to 1 batch query.
|
||||
Parses all files first, then does all database operations in one transaction.
|
||||
|
||||
Args:
|
||||
paths: List of paths to markdown files
|
||||
new: Whether these are new files
|
||||
|
||||
Returns:
|
||||
List of tuples (entity, checksum) for each file
|
||||
"""
|
||||
from basic_memory.markdown.utils import entity_model_from_markdown
|
||||
|
||||
if not paths:
|
||||
return []
|
||||
|
||||
logger.debug(f"Batch syncing {len(paths)} markdown files (new={new})")
|
||||
|
||||
# Phase 1: Parse all files (no DB operations)
|
||||
parsed_files = []
|
||||
for path in paths:
|
||||
# Check if file should be skipped due to repeated failures (circuit breaker)
|
||||
if await self._should_skip_file(path):
|
||||
logger.warning(f"Skipping file in batch due to repeated failures: {path}")
|
||||
parsed_files.append(None)
|
||||
continue
|
||||
|
||||
try:
|
||||
file_content = await self.file_service.read_file_content(path)
|
||||
file_contains_frontmatter = has_frontmatter(file_content)
|
||||
|
||||
# Get file timestamps for tracking modification times
|
||||
file_stats = self.file_service.file_stats(path)
|
||||
created = datetime.fromtimestamp(file_stats.st_ctime).astimezone()
|
||||
modified = datetime.fromtimestamp(file_stats.st_mtime).astimezone()
|
||||
|
||||
# Parse markdown to get entity structure
|
||||
entity_markdown = await self.entity_parser.parse_file(path)
|
||||
|
||||
# Resolve permalink if needed (skip conflict checks during batch)
|
||||
permalink = entity_markdown.frontmatter.permalink
|
||||
if file_contains_frontmatter and not self.app_config.disable_permalinks:
|
||||
permalink = await self.entity_service.resolve_permalink(
|
||||
path, markdown=entity_markdown, skip_conflict_check=True
|
||||
)
|
||||
|
||||
# If permalink changed, update the file
|
||||
if permalink != entity_markdown.frontmatter.permalink:
|
||||
logger.info(
|
||||
f"Updating permalink for path: {path}, "
|
||||
f"old_permalink: {entity_markdown.frontmatter.permalink}, "
|
||||
f"new_permalink: {permalink}"
|
||||
)
|
||||
entity_markdown.frontmatter.metadata["permalink"] = permalink
|
||||
await self.file_service.update_frontmatter(path, {"permalink": permalink})
|
||||
|
||||
# Convert to entity model (without saving to DB yet)
|
||||
entity_model = entity_model_from_markdown(Path(path), entity_markdown)
|
||||
entity_model.checksum = None # Will be set after relations are resolved
|
||||
|
||||
parsed_files.append(
|
||||
{
|
||||
"path": path,
|
||||
"entity_model": entity_model,
|
||||
"entity_markdown": entity_markdown,
|
||||
"created": created,
|
||||
"modified": modified,
|
||||
"mtime": file_stats.st_mtime,
|
||||
"size": file_stats.st_size,
|
||||
}
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
# Check if this is a fatal error (or caused by one)
|
||||
# Fatal errors like project deletion should terminate sync immediately
|
||||
if isinstance(e, SyncFatalError) or isinstance(e.__cause__, SyncFatalError):
|
||||
logger.error(
|
||||
f"Fatal sync error encountered during batch parse, terminating sync: path={path}"
|
||||
)
|
||||
raise
|
||||
|
||||
# Otherwise treat as recoverable file-level error
|
||||
logger.error(f"Failed to parse file in batch: path={path}, error={e}")
|
||||
# Track failure for circuit breaker
|
||||
await self._record_failure(path, str(e))
|
||||
parsed_files.append(None) # Mark as failed
|
||||
|
||||
# Phase 2: Batch database operations
|
||||
# Filter out failed parses
|
||||
valid_files = [f for f in parsed_files if f is not None]
|
||||
if not valid_files:
|
||||
logger.warning("No valid files to sync in batch")
|
||||
return [(None, "") for _ in paths]
|
||||
|
||||
# If this is a new batch, upsert all entities at once
|
||||
if new:
|
||||
entities_to_upsert = [f["entity_model"] for f in valid_files]
|
||||
logger.debug(f"Batch upserting {len(entities_to_upsert)} new entities")
|
||||
upserted_entities = await self.entity_repository.upsert_entities(entities_to_upsert)
|
||||
|
||||
# Create lookup by file_path for O(1) access
|
||||
entities_by_path = {e.file_path: e for e in upserted_entities}
|
||||
|
||||
# If updating existing entities, we need to handle observations/relations differently
|
||||
else:
|
||||
logger.debug(f"Batch updating {len(valid_files)} existing entities")
|
||||
# For updates, we need to:
|
||||
# 1. Get existing entities
|
||||
# 2. Delete old observations/relations
|
||||
# 3. Upsert updated entities
|
||||
|
||||
file_paths = [f["path"] for f in valid_files]
|
||||
existing_entities = await self.entity_repository.get_by_file_paths_batch(file_paths)
|
||||
|
||||
# Delete old observations and relations in batch
|
||||
entity_ids = [e.id for e in existing_entities.values() if e.id]
|
||||
if entity_ids:
|
||||
await self.entity_service.observation_repository.delete_by_entity_ids(entity_ids)
|
||||
await self.relation_repository.delete_outgoing_relations_from_entities(entity_ids)
|
||||
|
||||
# Upsert all updated entities
|
||||
entities_to_upsert = [f["entity_model"] for f in valid_files]
|
||||
upserted_entities = await self.entity_repository.upsert_entities(entities_to_upsert)
|
||||
|
||||
# Create lookup by file_path
|
||||
entities_by_path = {e.file_path: e for e in upserted_entities}
|
||||
|
||||
# Phase 3: Post-processing (relations, checksums, search index)
|
||||
results = []
|
||||
for i, path in enumerate(paths):
|
||||
parsed_file = parsed_files[i]
|
||||
|
||||
# Skip failed files
|
||||
if parsed_file is None:
|
||||
results.append((None, ""))
|
||||
continue
|
||||
|
||||
entity = entities_by_path.get(parsed_file["path"])
|
||||
if entity is None:
|
||||
logger.error(f"Entity not found after upsert: {parsed_file['path']}")
|
||||
results.append((None, ""))
|
||||
continue
|
||||
|
||||
try:
|
||||
# Update relations for this entity
|
||||
entity_with_relations = await self.entity_service.update_entity_relations(
|
||||
parsed_file["path"], parsed_file["entity_markdown"]
|
||||
)
|
||||
|
||||
# Compute final checksum after relations are resolved
|
||||
final_checksum = await self.file_service.compute_checksum(parsed_file["path"])
|
||||
|
||||
# Update checksum, timestamps, and file metadata
|
||||
await self.entity_repository.update(
|
||||
entity.id,
|
||||
{
|
||||
"checksum": final_checksum,
|
||||
"created_at": parsed_file["created"],
|
||||
"updated_at": parsed_file["modified"],
|
||||
"mtime": parsed_file["mtime"],
|
||||
"size": parsed_file["size"],
|
||||
},
|
||||
)
|
||||
|
||||
# Index for search
|
||||
await self.search_service.index_entity(entity_with_relations)
|
||||
|
||||
# Clear failure tracking on successful sync
|
||||
self._clear_failure(parsed_file["path"])
|
||||
|
||||
results.append((entity_with_relations, final_checksum))
|
||||
|
||||
logger.debug(
|
||||
f"Batch sync completed for file: path={parsed_file['path']}, "
|
||||
f"entity_id={entity.id}, checksum={final_checksum[:8]}"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
# Check if this is a fatal error
|
||||
if isinstance(e, SyncFatalError) or isinstance(e.__cause__, SyncFatalError):
|
||||
logger.error(
|
||||
f"Fatal sync error during post-processing, terminating sync: path={parsed_file['path']}"
|
||||
)
|
||||
raise
|
||||
|
||||
# Otherwise treat as recoverable file-level error
|
||||
logger.error(
|
||||
f"Failed to complete post-processing for file: path={parsed_file['path']}, error={e}"
|
||||
)
|
||||
await self._record_failure(parsed_file["path"], str(e))
|
||||
results.append((None, ""))
|
||||
continue
|
||||
|
||||
return results
|
||||
|
||||
@logfire.instrument()
|
||||
async def sync_regular_file(self, path: str, new: bool = True) -> Tuple[Optional[Entity], str]:
|
||||
"""Sync a non-markdown file with basic tracking.
|
||||
|
||||
@@ -1062,9 +730,9 @@ class SyncService:
|
||||
await self.entity_service.resolve_permalink(path, skip_conflict_check=True)
|
||||
|
||||
# get file timestamps
|
||||
file_stats = self.file_service.file_stats(path)
|
||||
created = datetime.fromtimestamp(file_stats.st_ctime).astimezone()
|
||||
modified = datetime.fromtimestamp(file_stats.st_mtime).astimezone()
|
||||
file_metadata = await self.file_service.get_file_metadata(path)
|
||||
created = file_metadata.created_at
|
||||
modified = file_metadata.modified_at
|
||||
|
||||
# get mime type
|
||||
content_type = self.file_service.content_type(path)
|
||||
@@ -1080,8 +748,8 @@ class SyncService:
|
||||
created_at=created,
|
||||
updated_at=modified,
|
||||
content_type=content_type,
|
||||
mtime=file_stats.st_mtime,
|
||||
size=file_stats.st_size,
|
||||
mtime=file_metadata.modified_at.timestamp(),
|
||||
size=file_metadata.size,
|
||||
)
|
||||
)
|
||||
return entity, checksum
|
||||
@@ -1097,15 +765,15 @@ class SyncService:
|
||||
logger.error(f"Entity not found after constraint violation, path={path}")
|
||||
raise ValueError(f"Entity not found after constraint violation: {path}")
|
||||
|
||||
# Re-get file stats since we're in update path
|
||||
file_stats_for_update = self.file_service.file_stats(path)
|
||||
# Re-get file metadata since we're in update path
|
||||
file_metadata_for_update = await self.file_service.get_file_metadata(path)
|
||||
updated = await self.entity_repository.update(
|
||||
entity.id,
|
||||
{
|
||||
"file_path": path,
|
||||
"checksum": checksum,
|
||||
"mtime": file_stats_for_update.st_mtime,
|
||||
"size": file_stats_for_update.st_size,
|
||||
"mtime": file_metadata_for_update.modified_at.timestamp(),
|
||||
"size": file_metadata_for_update.size,
|
||||
},
|
||||
)
|
||||
|
||||
@@ -1119,8 +787,8 @@ class SyncService:
|
||||
raise
|
||||
else:
|
||||
# Get file timestamps for updating modification time
|
||||
file_stats = self.file_service.file_stats(path)
|
||||
modified = datetime.fromtimestamp(file_stats.st_mtime).astimezone()
|
||||
file_metadata = await self.file_service.get_file_metadata(path)
|
||||
modified = file_metadata.modified_at
|
||||
|
||||
entity = await self.entity_repository.get_by_file_path(path)
|
||||
if entity is None: # pragma: no cover
|
||||
@@ -1135,8 +803,8 @@ class SyncService:
|
||||
"file_path": path,
|
||||
"checksum": checksum,
|
||||
"updated_at": modified,
|
||||
"mtime": file_stats.st_mtime,
|
||||
"size": file_stats.st_size,
|
||||
"mtime": file_metadata.modified_at.timestamp(),
|
||||
"size": file_metadata.size,
|
||||
},
|
||||
)
|
||||
|
||||
@@ -1146,7 +814,6 @@ class SyncService:
|
||||
|
||||
return updated, checksum
|
||||
|
||||
@logfire.instrument()
|
||||
async def handle_delete(self, file_path: str):
|
||||
"""Handle complete entity deletion including search index cleanup."""
|
||||
|
||||
@@ -1178,7 +845,6 @@ class SyncService:
|
||||
else:
|
||||
await self.search_service.delete_by_entity_id(entity.id)
|
||||
|
||||
@logfire.instrument()
|
||||
async def handle_move(self, old_path, new_path):
|
||||
logger.debug("Moving entity", old_path=old_path, new_path=new_path)
|
||||
|
||||
@@ -1283,7 +949,6 @@ class SyncService:
|
||||
# update search index
|
||||
await self.search_service.index_entity(updated)
|
||||
|
||||
@logfire.instrument()
|
||||
async def resolve_relations(self, entity_id: int | None = None):
|
||||
"""Try to resolve unresolved relations.
|
||||
|
||||
@@ -1334,16 +999,27 @@ class SyncService:
|
||||
"to_name": resolved_entity.title,
|
||||
},
|
||||
)
|
||||
except IntegrityError: # pragma: no cover
|
||||
# update search index only on successful resolution
|
||||
await self.search_service.index_entity(resolved_entity)
|
||||
except IntegrityError:
|
||||
# IntegrityError means a relation with this (from_id, to_id, relation_type)
|
||||
# already exists. The UPDATE was rolled back, so our unresolved relation
|
||||
# (to_id=NULL) still exists in the database. We delete it because:
|
||||
# 1. It's redundant - a resolved relation already captures this relationship
|
||||
# 2. If we don't delete it, future syncs will try to resolve it again
|
||||
# and get the same IntegrityError
|
||||
logger.debug(
|
||||
"Ignoring duplicate relation "
|
||||
"Deleting duplicate unresolved relation "
|
||||
f"relation_id={relation.id} "
|
||||
f"from_id={relation.from_id} "
|
||||
f"to_name={relation.to_name}"
|
||||
f"to_name={relation.to_name} "
|
||||
f"resolved_to_id={resolved_entity.id}"
|
||||
)
|
||||
|
||||
# update search index
|
||||
await self.search_service.index_entity(resolved_entity)
|
||||
try:
|
||||
await self.relation_repository.delete(relation.id)
|
||||
except Exception as e:
|
||||
# Log but don't fail - the relation may have been deleted already
|
||||
logger.debug(f"Could not delete duplicate relation {relation.id}: {e}")
|
||||
|
||||
async def _quick_count_files(self, directory: Path) -> int:
|
||||
"""Fast file count using find command.
|
||||
@@ -1371,8 +1047,6 @@ class SyncService:
|
||||
f"error: {error_msg}. Falling back to manual count. "
|
||||
f"This will slow down watermark detection!"
|
||||
)
|
||||
# Track optimization failures for visibility
|
||||
logfire.metric_counter("sync.scan.file_count_failure").add(1)
|
||||
# Fallback: count using scan_directory
|
||||
count = 0
|
||||
async for _ in self.scan_directory(directory):
|
||||
@@ -1413,8 +1087,6 @@ class SyncService:
|
||||
f"error: {error_msg}. Falling back to full scan. "
|
||||
f"This will cause slow syncs on large projects!"
|
||||
)
|
||||
# Track optimization failures for visibility
|
||||
logfire.metric_counter("sync.scan.optimization_failure").add(1)
|
||||
# Fallback to full scan
|
||||
return await self._scan_directory_full(directory)
|
||||
|
||||
@@ -1442,13 +1114,13 @@ class SyncService:
|
||||
async def _scan_directory_full(self, directory: Path) -> List[str]:
|
||||
"""Full directory scan returning all file paths.
|
||||
|
||||
Uses scan_directory() which respects .bmignore patterns.
|
||||
Uses scan_directory() which respects .bmignore patterns.
|
||||
|
||||
Args:
|
||||
directory: Directory to scan
|
||||
Args:
|
||||
directory: Directory to scan
|
||||
|
||||
Returns:
|
||||
List of relative file paths (respects .bmignore)
|
||||
List of relative file paths (respects .bmignore)
|
||||
"""
|
||||
file_paths = []
|
||||
async for file_path_str, _ in self.scan_directory(directory):
|
||||
|
||||
@@ -1,23 +0,0 @@
|
||||
"""Utilities for sync operations."""
|
||||
|
||||
from typing import Iterator, List, TypeVar
|
||||
|
||||
T = TypeVar("T")
|
||||
|
||||
|
||||
def chunks(items: List[T], size: int) -> Iterator[List[T]]:
|
||||
"""Split a list into chunks of specified size.
|
||||
|
||||
Args:
|
||||
items: List of items to chunk
|
||||
size: Size of each chunk
|
||||
|
||||
Yields:
|
||||
Lists of items, each of specified size (last chunk may be smaller)
|
||||
|
||||
Example:
|
||||
>>> list(chunks([1, 2, 3, 4, 5], 2))
|
||||
[[1, 2], [3, 4], [5]]
|
||||
"""
|
||||
for i in range(0, len(items), size):
|
||||
yield items[i : i + size]
|
||||
@@ -5,7 +5,10 @@ import os
|
||||
from collections import defaultdict
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import List, Optional, Set, Sequence
|
||||
from typing import List, Optional, Set, Sequence, Callable, Awaitable, TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from basic_memory.sync.sync_service import SyncService
|
||||
|
||||
from basic_memory.config import BasicMemoryConfig, WATCH_STATUS_JSON
|
||||
from basic_memory.ignore_utils import load_gitignore_patterns, should_ignore_path
|
||||
@@ -71,12 +74,17 @@ class WatchServiceState(BaseModel):
|
||||
self.last_error = datetime.now()
|
||||
|
||||
|
||||
# Type alias for sync service factory function
|
||||
SyncServiceFactory = Callable[[Project], Awaitable["SyncService"]]
|
||||
|
||||
|
||||
class WatchService:
|
||||
def __init__(
|
||||
self,
|
||||
app_config: BasicMemoryConfig,
|
||||
project_repository: ProjectRepository,
|
||||
quiet: bool = False,
|
||||
sync_service_factory: Optional[SyncServiceFactory] = None,
|
||||
):
|
||||
self.app_config = app_config
|
||||
self.project_repository = project_repository
|
||||
@@ -84,10 +92,20 @@ class WatchService:
|
||||
self.status_path = Path.home() / ".basic-memory" / WATCH_STATUS_JSON
|
||||
self.status_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
self._ignore_patterns_cache: dict[Path, Set[str]] = {}
|
||||
self._sync_service_factory = sync_service_factory
|
||||
|
||||
# quiet mode for mcp so it doesn't mess up stdout
|
||||
self.console = Console(quiet=quiet)
|
||||
|
||||
async def _get_sync_service(self, project: Project) -> "SyncService":
|
||||
"""Get sync service for a project, using factory if provided."""
|
||||
if self._sync_service_factory:
|
||||
return await self._sync_service_factory(project)
|
||||
# Fall back to default factory
|
||||
from basic_memory.sync.sync_service import get_sync_service
|
||||
|
||||
return await get_sync_service(project)
|
||||
|
||||
async def _schedule_restart(self, stop_event: asyncio.Event):
|
||||
"""Schedule a restart of the watch service after the configured interval."""
|
||||
await asyncio.sleep(self.app_config.watch_project_reload_interval)
|
||||
@@ -233,9 +251,6 @@ class WatchService:
|
||||
|
||||
async def handle_changes(self, project: Project, changes: Set[FileChange]) -> None:
|
||||
"""Process a batch of file changes"""
|
||||
# avoid circular imports
|
||||
from basic_memory.sync.sync_service import get_sync_service
|
||||
|
||||
# Check if project still exists in configuration before processing
|
||||
# This prevents deleted projects from being recreated by background sync
|
||||
from basic_memory.config import ConfigManager
|
||||
@@ -250,7 +265,7 @@ class WatchService:
|
||||
)
|
||||
return
|
||||
|
||||
sync_service = await get_sync_service(project)
|
||||
sync_service = await self._get_sync_service(project)
|
||||
file_service = sync_service.file_service
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
+65
-64
@@ -5,9 +5,9 @@ import os
|
||||
import logging
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Optional, Protocol, Union, runtime_checkable, List
|
||||
from typing import Protocol, Union, runtime_checkable, List
|
||||
|
||||
from loguru import logger
|
||||
from unidecode import unidecode
|
||||
@@ -67,9 +67,6 @@ class PathLike(Protocol):
|
||||
# This preserves compatibility with existing code while we migrate
|
||||
FilePath = Union[Path, str]
|
||||
|
||||
# Disable the "Queue is full" warning
|
||||
logging.getLogger("opentelemetry.sdk.metrics._internal.instrument").setLevel(logging.ERROR)
|
||||
|
||||
|
||||
def generate_permalink(file_path: Union[Path, str, PathLike], split_extension: bool = True) -> str:
|
||||
"""Generate a stable permalink from a file path.
|
||||
@@ -206,29 +203,35 @@ def generate_permalink(file_path: Union[Path, str, PathLike], split_extension: b
|
||||
|
||||
|
||||
def setup_logging(
|
||||
env: str,
|
||||
home_dir: Path,
|
||||
log_file: Optional[str] = None,
|
||||
log_level: str = "INFO",
|
||||
console: bool = True,
|
||||
log_to_file: bool = False,
|
||||
log_to_stdout: bool = False,
|
||||
structured_context: bool = False,
|
||||
) -> None: # pragma: no cover
|
||||
"""
|
||||
Configure logging for the application.
|
||||
"""Configure logging with explicit settings.
|
||||
|
||||
This function provides a simple, explicit interface for configuring logging.
|
||||
Each entry point (CLI, MCP, API) should call this with appropriate settings.
|
||||
|
||||
Args:
|
||||
env: The environment name (dev, test, prod)
|
||||
home_dir: The root directory for the application
|
||||
log_file: The name of the log file to write to
|
||||
log_level: The logging level to use
|
||||
console: Whether to log to the console
|
||||
log_level: DEBUG, INFO, WARNING, ERROR
|
||||
log_to_file: Write to ~/.basic-memory/basic-memory.log with rotation
|
||||
log_to_stdout: Write to stderr (for Docker/cloud deployments)
|
||||
structured_context: Bind tenant_id, fly_region, etc. for cloud observability
|
||||
"""
|
||||
# Remove default handler and any existing handlers
|
||||
logger.remove()
|
||||
|
||||
# Add file handler if we are not running tests and a log file is specified
|
||||
if log_file and env != "test":
|
||||
# Setup file logger
|
||||
log_path = home_dir / log_file
|
||||
# In test mode, only log to stdout regardless of settings
|
||||
env = os.getenv("BASIC_MEMORY_ENV", "dev")
|
||||
if env == "test":
|
||||
logger.add(sys.stderr, level=log_level, backtrace=True, diagnose=True, colorize=True)
|
||||
return
|
||||
|
||||
# Add file handler with rotation
|
||||
if log_to_file:
|
||||
log_path = Path.home() / ".basic-memory" / "basic-memory.log"
|
||||
log_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
logger.add(
|
||||
str(log_path),
|
||||
level=log_level,
|
||||
@@ -236,42 +239,28 @@ def setup_logging(
|
||||
retention="10 days",
|
||||
backtrace=True,
|
||||
diagnose=True,
|
||||
enqueue=True,
|
||||
enqueue=True, # Thread-safe async logging
|
||||
colorize=False,
|
||||
)
|
||||
|
||||
# Add console logger if requested or in test mode
|
||||
if env == "test" or console:
|
||||
# Add stdout handler (for Docker/cloud)
|
||||
if log_to_stdout:
|
||||
logger.add(sys.stderr, level=log_level, backtrace=True, diagnose=True, colorize=True)
|
||||
|
||||
logger.info(f"ENV: '{env}' Log level: '{log_level}' Logging to {log_file}")
|
||||
|
||||
# Bind environment context for structured logging (works in both local and cloud)
|
||||
tenant_id = os.getenv("BASIC_MEMORY_TENANT_ID", "local")
|
||||
fly_app_name = os.getenv("FLY_APP_NAME", "local")
|
||||
fly_machine_id = os.getenv("FLY_MACHINE_ID", "local")
|
||||
fly_region = os.getenv("FLY_REGION", "local")
|
||||
|
||||
logger.configure(
|
||||
extra={
|
||||
"tenant_id": tenant_id,
|
||||
"fly_app_name": fly_app_name,
|
||||
"fly_machine_id": fly_machine_id,
|
||||
"fly_region": fly_region,
|
||||
}
|
||||
)
|
||||
# Bind structured context for cloud observability
|
||||
if structured_context:
|
||||
logger.configure(
|
||||
extra={
|
||||
"tenant_id": os.getenv("BASIC_MEMORY_TENANT_ID", "local"),
|
||||
"fly_app_name": os.getenv("FLY_APP_NAME", "local"),
|
||||
"fly_machine_id": os.getenv("FLY_MACHINE_ID", "local"),
|
||||
"fly_region": os.getenv("FLY_REGION", "local"),
|
||||
}
|
||||
)
|
||||
|
||||
# Reduce noise from third-party libraries
|
||||
noisy_loggers = {
|
||||
# HTTP client logs
|
||||
"httpx": logging.WARNING,
|
||||
# File watching logs
|
||||
"watchfiles.main": logging.WARNING,
|
||||
}
|
||||
|
||||
# Set log levels for noisy loggers
|
||||
for logger_name, level in noisy_loggers.items():
|
||||
logging.getLogger(logger_name).setLevel(level)
|
||||
logging.getLogger("httpx").setLevel(logging.WARNING)
|
||||
logging.getLogger("watchfiles.main").setLevel(logging.WARNING)
|
||||
|
||||
|
||||
def parse_tags(tags: Union[List[str], str, None]) -> List[str]:
|
||||
@@ -340,7 +329,7 @@ def normalize_file_path_for_comparison(file_path: str) -> str:
|
||||
This function normalizes file paths to help detect potential conflicts:
|
||||
- Converts to lowercase for case-insensitive comparison
|
||||
- Normalizes Unicode characters
|
||||
- Handles path separators consistently
|
||||
- Converts backslashes to forward slashes for cross-platform consistency
|
||||
|
||||
Args:
|
||||
file_path: The file path to normalize
|
||||
@@ -349,19 +338,15 @@ def normalize_file_path_for_comparison(file_path: str) -> str:
|
||||
Normalized file path for comparison purposes
|
||||
"""
|
||||
import unicodedata
|
||||
from pathlib import PureWindowsPath
|
||||
|
||||
# Convert to lowercase for case-insensitive comparison
|
||||
normalized = file_path.lower()
|
||||
# Use PureWindowsPath to ensure backslashes are treated as separators
|
||||
# regardless of current platform, then convert to POSIX-style
|
||||
normalized = PureWindowsPath(file_path).as_posix().lower()
|
||||
|
||||
# Normalize Unicode characters (NFD normalization)
|
||||
normalized = unicodedata.normalize("NFD", normalized)
|
||||
|
||||
# Replace path separators with forward slashes
|
||||
normalized = normalized.replace("\\", "/")
|
||||
|
||||
# Remove multiple slashes
|
||||
normalized = re.sub(r"/+", "/", normalized)
|
||||
|
||||
return normalized
|
||||
|
||||
|
||||
@@ -445,21 +430,37 @@ def validate_project_path(path: str, project_path: Path) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def ensure_timezone_aware(dt: datetime) -> datetime:
|
||||
"""Ensure a datetime is timezone-aware using system timezone.
|
||||
def ensure_timezone_aware(dt: datetime, cloud_mode: bool | None = None) -> datetime:
|
||||
"""Ensure a datetime is timezone-aware.
|
||||
|
||||
If the datetime is naive, convert it to timezone-aware using the system's local timezone.
|
||||
If it's already timezone-aware, return it unchanged.
|
||||
If the datetime is naive, convert it to timezone-aware. The interpretation
|
||||
depends on cloud_mode:
|
||||
- In cloud mode (PostgreSQL/asyncpg): naive datetimes are interpreted as UTC
|
||||
- In local mode (SQLite): naive datetimes are interpreted as local time
|
||||
|
||||
asyncpg uses binary protocol which returns timestamps in UTC but as naive
|
||||
datetimes. In cloud deployments, cloud_mode=True handles this correctly.
|
||||
|
||||
Args:
|
||||
dt: The datetime to ensure is timezone-aware
|
||||
cloud_mode: Optional explicit cloud_mode setting. If None, loads from config.
|
||||
|
||||
Returns:
|
||||
A timezone-aware datetime
|
||||
"""
|
||||
if dt.tzinfo is None:
|
||||
# Naive datetime - assume it's in local time and add timezone
|
||||
return dt.astimezone()
|
||||
# Determine cloud_mode: use explicit parameter if provided, otherwise load from config
|
||||
if cloud_mode is None:
|
||||
from basic_memory.config import ConfigManager
|
||||
|
||||
cloud_mode = ConfigManager().config.cloud_mode_enabled
|
||||
|
||||
if cloud_mode:
|
||||
# Cloud/PostgreSQL mode: naive datetimes from asyncpg are already UTC
|
||||
return dt.replace(tzinfo=timezone.utc)
|
||||
else:
|
||||
# Local/SQLite mode: naive datetimes are in local time
|
||||
return dt.astimezone()
|
||||
else:
|
||||
# Already timezone-aware
|
||||
return dt
|
||||
|
||||
+78
-49
@@ -50,18 +50,23 @@ The `app` fixture ensures FastAPI dependency overrides are active, and
|
||||
`mcp_server` provides the MCP server with proper project session initialization.
|
||||
"""
|
||||
|
||||
import os
|
||||
from typing import AsyncGenerator, Literal
|
||||
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
from pathlib import Path
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
|
||||
from sqlalchemy.pool import NullPool
|
||||
from testcontainers.postgres import PostgresContainer
|
||||
|
||||
from httpx import AsyncClient, ASGITransport
|
||||
|
||||
from basic_memory.config import BasicMemoryConfig, ProjectConfig, ConfigManager, DatabaseBackend
|
||||
from basic_memory.db import engine_session_factory, DatabaseType
|
||||
from basic_memory.models import Project
|
||||
from basic_memory.models.base import Base
|
||||
from basic_memory.repository.project_repository import ProjectRepository
|
||||
from fastapi import FastAPI
|
||||
|
||||
@@ -72,25 +77,38 @@ from basic_memory.deps import get_project_config, get_engine_factory, get_app_co
|
||||
from basic_memory.mcp import tools # noqa: F401
|
||||
|
||||
|
||||
@pytest.fixture(
|
||||
params=[
|
||||
pytest.param("sqlite", id="sqlite"),
|
||||
pytest.param("postgres", id="postgres", marks=pytest.mark.postgres),
|
||||
]
|
||||
)
|
||||
def db_backend(request) -> Literal["sqlite", "postgres"]:
|
||||
"""Parametrize tests to run against both SQLite and Postgres.
|
||||
# =============================================================================
|
||||
# Database Backend Selection (env var approach)
|
||||
# =============================================================================
|
||||
# By default, integration tests run against SQLite.
|
||||
# Set BASIC_MEMORY_TEST_POSTGRES=1 to run against Postgres (uses testcontainers).
|
||||
|
||||
Usage:
|
||||
pytest # Runs tests against SQLite only (default)
|
||||
pytest -m postgres # Runs tests against Postgres only
|
||||
pytest -m "not postgres" # Runs tests against SQLite only
|
||||
pytest --run-all-backends # Runs tests against both backends
|
||||
|
||||
Note: Only tests that use database fixtures (engine_factory, session_maker, etc.)
|
||||
will be parametrized. Tests that don't use the database won't be affected.
|
||||
@pytest.fixture(scope="session")
|
||||
def db_backend() -> Literal["sqlite", "postgres"]:
|
||||
"""Determine database backend from environment variable.
|
||||
|
||||
Default: sqlite
|
||||
Set BASIC_MEMORY_TEST_POSTGRES=1 to use postgres
|
||||
"""
|
||||
return request.param
|
||||
if os.environ.get("BASIC_MEMORY_TEST_POSTGRES", "").lower() in ("1", "true", "yes"):
|
||||
return "postgres"
|
||||
return "sqlite"
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def postgres_container(db_backend):
|
||||
"""Session-scoped Postgres container for integration tests.
|
||||
|
||||
Uses testcontainers to spin up a real Postgres instance.
|
||||
Only starts if db_backend is "postgres".
|
||||
"""
|
||||
if db_backend != "postgres":
|
||||
yield None
|
||||
return
|
||||
|
||||
with PostgresContainer("postgres:16-alpine") as postgres:
|
||||
yield postgres
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
@@ -98,50 +116,57 @@ async def engine_factory(
|
||||
app_config,
|
||||
config_manager,
|
||||
db_backend: Literal["sqlite", "postgres"],
|
||||
postgres_container,
|
||||
tmp_path,
|
||||
) -> AsyncGenerator[tuple, None]:
|
||||
"""Create engine and session factory for the configured database backend."""
|
||||
from basic_memory.models.search import CREATE_SEARCH_INDEX
|
||||
from basic_memory.models.search import (
|
||||
CREATE_SEARCH_INDEX,
|
||||
CREATE_POSTGRES_SEARCH_INDEX_TABLE,
|
||||
CREATE_POSTGRES_SEARCH_INDEX_FTS,
|
||||
CREATE_POSTGRES_SEARCH_INDEX_METADATA,
|
||||
)
|
||||
from basic_memory import db
|
||||
|
||||
# Determine database type based on backend
|
||||
if db_backend == "postgres":
|
||||
db_type = DatabaseType.FILESYSTEM
|
||||
else:
|
||||
db_type = DatabaseType.FILESYSTEM # Integration tests use file-based SQLite
|
||||
# Postgres mode using testcontainers
|
||||
sync_url = postgres_container.get_connection_url()
|
||||
async_url = sync_url.replace("postgresql+psycopg2", "postgresql+asyncpg")
|
||||
|
||||
# Use tmp_path for SQLite, use config database_path for Postgres
|
||||
if db_backend == "sqlite":
|
||||
db_path = tmp_path / "test.db"
|
||||
else:
|
||||
db_path = app_config.database_path
|
||||
engine = create_async_engine(
|
||||
async_url,
|
||||
echo=False,
|
||||
poolclass=NullPool,
|
||||
)
|
||||
|
||||
if db_backend == "postgres":
|
||||
# Postgres: Create fresh engine for each test with full schema reset
|
||||
config_manager._config = app_config
|
||||
session_maker = async_sessionmaker(
|
||||
bind=engine,
|
||||
class_=AsyncSession,
|
||||
expire_on_commit=False,
|
||||
autoflush=False,
|
||||
)
|
||||
|
||||
# Use context manager to handle engine disposal properly
|
||||
async with engine_session_factory(db_path, db_type) as (engine, session_maker):
|
||||
# Drop and recreate schema for complete isolation
|
||||
async with engine.begin() as conn:
|
||||
await conn.execute(text("DROP SCHEMA IF EXISTS public CASCADE"))
|
||||
await conn.execute(text("CREATE SCHEMA public"))
|
||||
await conn.execute(text("GRANT ALL ON SCHEMA public TO basic_memory_user"))
|
||||
await conn.execute(text("GRANT ALL ON SCHEMA public TO public"))
|
||||
# Drop and recreate all tables for test isolation
|
||||
async with engine.begin() as conn:
|
||||
await conn.execute(text("DROP TABLE IF EXISTS search_index CASCADE"))
|
||||
await conn.run_sync(Base.metadata.drop_all)
|
||||
await conn.run_sync(Base.metadata.create_all)
|
||||
# asyncpg requires separate execute calls for each statement
|
||||
await conn.execute(CREATE_POSTGRES_SEARCH_INDEX_TABLE)
|
||||
await conn.execute(CREATE_POSTGRES_SEARCH_INDEX_FTS)
|
||||
await conn.execute(CREATE_POSTGRES_SEARCH_INDEX_METADATA)
|
||||
|
||||
# Run migrations to create production tables
|
||||
from basic_memory.db import run_migrations
|
||||
yield engine, session_maker
|
||||
|
||||
await run_migrations(app_config, db_type)
|
||||
|
||||
yield engine, session_maker
|
||||
await engine.dispose()
|
||||
|
||||
else:
|
||||
# SQLite: Create fresh database (fast with tmp files)
|
||||
db_path = tmp_path / "test.db"
|
||||
db_type = DatabaseType.FILESYSTEM
|
||||
|
||||
async with engine_session_factory(db_path, db_type) as (engine, session_maker):
|
||||
# Create all tables via ORM
|
||||
from basic_memory.models.base import Base
|
||||
|
||||
async with engine.begin() as conn:
|
||||
await conn.run_sync(Base.metadata.create_all)
|
||||
|
||||
@@ -181,7 +206,11 @@ def config_home(tmp_path, monkeypatch) -> Path:
|
||||
|
||||
@pytest.fixture
|
||||
def app_config(
|
||||
config_home, db_backend: Literal["sqlite", "postgres"], tmp_path, monkeypatch
|
||||
config_home,
|
||||
db_backend: Literal["sqlite", "postgres"],
|
||||
postgres_container,
|
||||
tmp_path,
|
||||
monkeypatch,
|
||||
) -> BasicMemoryConfig:
|
||||
"""Create test app configuration."""
|
||||
# Disable cloud mode for CLI tests
|
||||
@@ -190,12 +219,12 @@ def app_config(
|
||||
# Create a basic config with test-project like unit tests do
|
||||
projects = {"test-project": str(config_home)}
|
||||
|
||||
# Configure database backend based on test parameter
|
||||
# Configure database backend based on env var
|
||||
if db_backend == "postgres":
|
||||
database_backend = DatabaseBackend.POSTGRES
|
||||
database_url = (
|
||||
"postgresql+asyncpg://basic_memory_user:dev_password@localhost:5433/basic_memory_test"
|
||||
)
|
||||
# Get URL from testcontainer and convert to asyncpg driver
|
||||
sync_url = postgres_container.get_connection_url()
|
||||
database_url = sync_url.replace("postgresql+psycopg2", "postgresql+asyncpg")
|
||||
else:
|
||||
database_backend = DatabaseBackend.SQLITE
|
||||
database_url = None
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
"""
|
||||
Integration test for FastAPI lifespan shutdown behavior.
|
||||
|
||||
This test verifies the asyncio cancellation pattern used by the API lifespan:
|
||||
when the background sync task is cancelled during shutdown, it must be *awaited*
|
||||
before database shutdown begins. This prevents "hang on exit" scenarios in
|
||||
`asyncio.run(...)` callers (e.g. CLI/MCP clients using httpx ASGITransport).
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
|
||||
|
||||
def test_lifespan_shutdown_awaits_sync_task_cancellation(app, monkeypatch):
|
||||
"""
|
||||
Ensure lifespan shutdown awaits the cancelled background sync task.
|
||||
|
||||
Why this is deterministic:
|
||||
- Cancelling a task does not make it "done" immediately; it becomes done only
|
||||
once the event loop schedules it and it processes the CancelledError.
|
||||
- In the buggy version, shutdown proceeded directly to db.shutdown_db()
|
||||
immediately after calling cancel(), so at *entry* to shutdown_db the task
|
||||
is still not done.
|
||||
- In the fixed version, lifespan does `await sync_task` before shutdown_db,
|
||||
so by the time shutdown_db is called, the task is done (cancelled).
|
||||
"""
|
||||
|
||||
# Import the *module* (not the package-level FastAPI `basic_memory.api.app` export)
|
||||
# so monkeypatching affects the exact symbols referenced inside lifespan().
|
||||
#
|
||||
# Note: `basic_memory/api/__init__.py` re-exports `app`, so `import basic_memory.api.app`
|
||||
# can resolve to the FastAPI instance rather than the `basic_memory.api.app` module.
|
||||
import importlib
|
||||
|
||||
api_app_module = importlib.import_module("basic_memory.api.app")
|
||||
|
||||
# Keep startup cheap: we don't need real DB init for this ordering test.
|
||||
async def _noop_initialize_app(_app_config):
|
||||
return None
|
||||
|
||||
async def _fake_get_or_create_db(*_args, **_kwargs):
|
||||
return object(), object()
|
||||
|
||||
monkeypatch.setattr(api_app_module, "initialize_app", _noop_initialize_app)
|
||||
monkeypatch.setattr(api_app_module.db, "get_or_create_db", _fake_get_or_create_db)
|
||||
|
||||
# Make the sync task long-lived so it must be cancelled on shutdown.
|
||||
async def _fake_initialize_file_sync(_app_config):
|
||||
await asyncio.Event().wait()
|
||||
|
||||
monkeypatch.setattr(api_app_module, "initialize_file_sync", _fake_initialize_file_sync)
|
||||
|
||||
# Assert ordering: shutdown_db must be called only after the sync_task is done.
|
||||
async def _assert_sync_task_done_before_db_shutdown():
|
||||
assert api_app_module.app.state.sync_task is not None
|
||||
assert api_app_module.app.state.sync_task.done()
|
||||
|
||||
monkeypatch.setattr(api_app_module.db, "shutdown_db", _assert_sync_task_done_before_db_shutdown)
|
||||
|
||||
async def _run_client_once():
|
||||
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
|
||||
# Any request is sufficient to trigger lifespan startup/shutdown.
|
||||
await client.get("/__nonexistent__")
|
||||
|
||||
# Use asyncio.run to match the CLI/MCP execution model where loop teardown
|
||||
# would hang if a background task is left running.
|
||||
asyncio.run(_run_client_once())
|
||||
|
||||
|
||||
@@ -77,7 +77,8 @@ async def test_create_project_basic_operation(mcp_server, app, test_project):
|
||||
assert "test-new-project" in create_text
|
||||
assert "Project Details:" in create_text
|
||||
assert "Name: test-new-project" in create_text
|
||||
assert "Path: /tmp/test-new-project" in create_text
|
||||
# Check path contains project name (platform-independent)
|
||||
assert "Path:" in create_text and "test-new-project" in create_text
|
||||
assert "Project is now available for use" in create_text
|
||||
|
||||
# Verify project appears in project list
|
||||
|
||||
@@ -46,3 +46,57 @@ async def test_read_note_after_write(mcp_server, app, test_project):
|
||||
assert "# Test Note" in result_text
|
||||
assert "This is test content." in result_text
|
||||
assert "test/test-note" in result_text # permalink
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_read_note_underscored_folder_by_permalink(mcp_server, app, test_project):
|
||||
"""Test read_note with permalink from underscored folder.
|
||||
|
||||
Reproduces bug #416: read_note fails to find notes when given permalinks
|
||||
from underscored folder names (e.g., _archive/, _drafts/), even though
|
||||
the permalink is copied directly from the note's YAML frontmatter.
|
||||
"""
|
||||
|
||||
async with Client(mcp_server) as client:
|
||||
# Create a note in an underscored folder
|
||||
write_result = await client.call_tool(
|
||||
"write_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"title": "Example Note",
|
||||
"folder": "_archive/articles",
|
||||
"content": "# Example Note\n\nThis is a test note in an underscored folder.",
|
||||
"tags": "test,archive",
|
||||
},
|
||||
)
|
||||
|
||||
assert len(write_result.content) == 1
|
||||
assert write_result.content[0].type == "text"
|
||||
write_text = write_result.content[0].text
|
||||
|
||||
# Verify the file path includes the underscore
|
||||
assert "_archive/articles/Example Note.md" in write_text
|
||||
|
||||
# Verify the permalink has underscores stripped (this is the expected behavior)
|
||||
assert "archive/articles/example-note" in write_text
|
||||
|
||||
# Now try to read the note using the permalink (without underscores)
|
||||
# This is the exact scenario from the bug report - using the permalink
|
||||
# that was generated in the YAML frontmatter
|
||||
read_result = await client.call_tool(
|
||||
"read_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"identifier": "archive/articles/example-note", # permalink without underscores
|
||||
},
|
||||
)
|
||||
|
||||
# This should succeed - the note should be found by its permalink
|
||||
assert len(read_result.content) == 1
|
||||
assert read_result.content[0].type == "text"
|
||||
result_text = read_result.content[0].text
|
||||
|
||||
# Should contain the note content
|
||||
assert "# Example Note" in result_text
|
||||
assert "This is a test note in an underscored folder." in result_text
|
||||
assert "archive/articles/example-note" in result_text # permalink
|
||||
|
||||
@@ -437,9 +437,7 @@ async def test_write_note_project_path_validation(mcp_server, app, test_project)
|
||||
project_with_tilde = ProjectItem(
|
||||
id=1,
|
||||
name="Test BiSync", # Name differs from path structure
|
||||
description="Test",
|
||||
path="~/Documents/Test BiSync", # Path with tilde
|
||||
is_active=True,
|
||||
is_default=False,
|
||||
)
|
||||
|
||||
|
||||
@@ -5,7 +5,6 @@ and other SQLite configuration settings work correctly in production scenarios.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from unittest.mock import patch
|
||||
from sqlalchemy import text
|
||||
|
||||
|
||||
@@ -142,23 +141,6 @@ async def test_null_pool_on_windows(tmp_path, monkeypatch):
|
||||
assert isinstance(engine.pool, NullPool)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.skipif(
|
||||
__import__("os").name == "nt", reason="Non-Windows test - cannot mock POSIX paths on Windows"
|
||||
)
|
||||
async def test_regular_pool_on_non_windows(tmp_path):
|
||||
"""Test that regular pooling is used on non-Windows platforms."""
|
||||
from basic_memory.db import engine_session_factory, DatabaseType
|
||||
from sqlalchemy.pool import NullPool
|
||||
|
||||
db_path = tmp_path / "test_posix_pool.db"
|
||||
|
||||
with patch("basic_memory.db.os.name", "posix"):
|
||||
async with engine_session_factory(db_path, DatabaseType.FILESYSTEM) as (engine, _):
|
||||
# Engine should NOT be using NullPool on non-Windows
|
||||
assert not isinstance(engine.pool, NullPool)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.windows
|
||||
@pytest.mark.skipif(
|
||||
|
||||
@@ -1,369 +0,0 @@
|
||||
"""
|
||||
Performance benchmark tests for sync operations.
|
||||
|
||||
These tests measure baseline performance for indexing operations to track
|
||||
improvements from optimizations. Tests are marked with @pytest.mark.benchmark
|
||||
and can be run separately.
|
||||
|
||||
Usage:
|
||||
# Run all benchmarks
|
||||
pytest test-int/test_sync_performance_benchmark.py -v
|
||||
|
||||
# Run specific benchmark
|
||||
pytest test-int/test_sync_performance_benchmark.py::test_benchmark_sync_100_files -v
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
from pathlib import Path
|
||||
from textwrap import dedent
|
||||
|
||||
import pytest
|
||||
|
||||
from basic_memory.config import BasicMemoryConfig, ProjectConfig
|
||||
from basic_memory.sync.sync_service import get_sync_service
|
||||
|
||||
|
||||
async def create_benchmark_file(path: Path, file_num: int, total_files: int) -> None:
|
||||
"""Create a realistic test markdown file with observations and relations.
|
||||
|
||||
Args:
|
||||
path: Path to create the file at
|
||||
file_num: Current file number (for unique content)
|
||||
total_files: Total number of files being created (for relation targets)
|
||||
"""
|
||||
# Create realistic content with varying complexity
|
||||
has_relations = file_num < (total_files - 1) # Most files have relations
|
||||
num_observations = min(3 + (file_num % 5), 10) # 3-10 observations per file
|
||||
|
||||
# Generate relation targets (some will be forward references)
|
||||
relations = []
|
||||
if has_relations:
|
||||
# Reference 1-3 other files
|
||||
num_relations = min(1 + (file_num % 3), 3)
|
||||
for i in range(num_relations):
|
||||
target_num = (file_num + i + 1) % total_files
|
||||
relations.append(f"- relates_to [[test-file-{target_num:04d}]]")
|
||||
|
||||
content = dedent(f"""
|
||||
---
|
||||
type: note
|
||||
tags: [benchmark, test, category-{file_num % 10}]
|
||||
---
|
||||
# Test File {file_num:04d}
|
||||
|
||||
This is benchmark test file {file_num} of {total_files}.
|
||||
It contains realistic markdown content to simulate actual usage.
|
||||
|
||||
## Observations
|
||||
{chr(10).join([f"- [category-{i % 5}] Observation {i} for file {file_num} with some content #tag{i}" for i in range(num_observations)])}
|
||||
|
||||
## Relations
|
||||
{chr(10).join(relations) if relations else "- No relations for this file"}
|
||||
|
||||
## Additional Content
|
||||
|
||||
This section contains additional prose to simulate real documents.
|
||||
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed do eiusmod
|
||||
tempor incididunt ut labore et dolore magna aliqua.
|
||||
|
||||
### Subsection
|
||||
|
||||
More content here to make the file realistic. This helps test the
|
||||
full indexing pipeline including content extraction and search indexing.
|
||||
""").strip()
|
||||
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(content, encoding="utf-8")
|
||||
|
||||
|
||||
async def generate_benchmark_files(project_dir: Path, num_files: int) -> None:
|
||||
"""Generate benchmark test files.
|
||||
|
||||
Args:
|
||||
project_dir: Directory to create files in
|
||||
num_files: Number of files to generate
|
||||
"""
|
||||
print(f"\nGenerating {num_files} test files...")
|
||||
start = time.time()
|
||||
|
||||
# Create files in batches for faster generation
|
||||
batch_size = 100
|
||||
for batch_start in range(0, num_files, batch_size):
|
||||
batch_end = min(batch_start + batch_size, num_files)
|
||||
tasks = [
|
||||
create_benchmark_file(
|
||||
project_dir / f"category-{i % 10}" / f"test-file-{i:04d}.md", i, num_files
|
||||
)
|
||||
for i in range(batch_start, batch_end)
|
||||
]
|
||||
await asyncio.gather(*tasks)
|
||||
print(f" Created files {batch_start}-{batch_end} ({batch_end}/{num_files})")
|
||||
|
||||
duration = time.time() - start
|
||||
print(f" File generation completed in {duration:.2f}s ({num_files / duration:.1f} files/sec)")
|
||||
|
||||
|
||||
def get_db_size(db_path: Path) -> tuple[int, str]:
|
||||
"""Get database file size.
|
||||
|
||||
Returns:
|
||||
Tuple of (size_bytes, formatted_size)
|
||||
"""
|
||||
if not db_path.exists():
|
||||
return 0, "0 B"
|
||||
|
||||
size_bytes = db_path.stat().st_size
|
||||
|
||||
# Format size
|
||||
for unit in ["B", "KB", "MB", "GB"]:
|
||||
if size_bytes < 1024.0:
|
||||
return size_bytes, f"{size_bytes:.2f} {unit}"
|
||||
size_bytes /= 1024.0
|
||||
|
||||
return int(size_bytes * 1024**4), f"{size_bytes:.2f} TB"
|
||||
|
||||
|
||||
async def run_sync_benchmark(
|
||||
project_config: ProjectConfig, app_config: BasicMemoryConfig, num_files: int, test_name: str
|
||||
) -> dict:
|
||||
"""Run a sync benchmark and collect metrics.
|
||||
|
||||
Args:
|
||||
project_config: Project configuration
|
||||
app_config: App configuration
|
||||
num_files: Number of files to benchmark
|
||||
test_name: Name of the test for reporting
|
||||
|
||||
Returns:
|
||||
Dictionary with benchmark results
|
||||
"""
|
||||
project_dir = project_config.home
|
||||
db_path = app_config.database_path
|
||||
|
||||
print(f"\n{'=' * 70}")
|
||||
print(f"BENCHMARK: {test_name}")
|
||||
print(f"{'=' * 70}")
|
||||
|
||||
# Generate test files
|
||||
await generate_benchmark_files(project_dir, num_files)
|
||||
|
||||
# Get initial DB size
|
||||
initial_db_size, initial_db_formatted = get_db_size(db_path)
|
||||
print(f"\nInitial database size: {initial_db_formatted}")
|
||||
|
||||
# Create sync service
|
||||
from basic_memory.repository import ProjectRepository
|
||||
from basic_memory import db
|
||||
|
||||
_, session_maker = await db.get_or_create_db(
|
||||
db_path=app_config.database_path,
|
||||
db_type=db.DatabaseType.FILESYSTEM,
|
||||
)
|
||||
project_repository = ProjectRepository(session_maker)
|
||||
|
||||
# Get or create project
|
||||
projects = await project_repository.find_all()
|
||||
if projects:
|
||||
project = projects[0]
|
||||
else:
|
||||
project = await project_repository.create(
|
||||
{
|
||||
"name": project_config.name,
|
||||
"path": str(project_config.home),
|
||||
"is_active": True,
|
||||
"is_default": True,
|
||||
}
|
||||
)
|
||||
|
||||
sync_service = await get_sync_service(project)
|
||||
|
||||
# Initialize search index (required for FTS5 table)
|
||||
await sync_service.search_service.init_search_index()
|
||||
|
||||
# Run sync and measure time
|
||||
print(f"\nStarting sync of {num_files} files...")
|
||||
sync_start = time.time()
|
||||
|
||||
report = await sync_service.sync(project_dir, project_name=project.name)
|
||||
|
||||
sync_duration = time.time() - sync_start
|
||||
|
||||
# Get final DB size
|
||||
final_db_size, final_db_formatted = get_db_size(db_path)
|
||||
db_growth = final_db_size - initial_db_size
|
||||
db_growth_formatted = f"{db_growth / 1024 / 1024:.2f} MB"
|
||||
|
||||
# Calculate metrics
|
||||
files_per_sec = num_files / sync_duration if sync_duration > 0 else 0
|
||||
ms_per_file = (sync_duration * 1000) / num_files if num_files > 0 else 0
|
||||
|
||||
# Print results
|
||||
print(f"\n{'-' * 70}")
|
||||
print("RESULTS:")
|
||||
print(f"{'-' * 70}")
|
||||
print(f"Files processed: {num_files}")
|
||||
print(f" New: {len(report.new)}")
|
||||
print(f" Modified: {len(report.modified)}")
|
||||
print(f" Deleted: {len(report.deleted)}")
|
||||
print(f" Moved: {len(report.moves)}")
|
||||
print("\nPerformance:")
|
||||
print(f" Total time: {sync_duration:.2f}s")
|
||||
print(f" Files/sec: {files_per_sec:.1f}")
|
||||
print(f" ms/file: {ms_per_file:.1f}")
|
||||
print("\nDatabase:")
|
||||
print(f" Initial size: {initial_db_formatted}")
|
||||
print(f" Final size: {final_db_formatted}")
|
||||
print(f" Growth: {db_growth_formatted}")
|
||||
print(f" Growth per file: {(db_growth / num_files / 1024):.2f} KB")
|
||||
print(f"{'=' * 70}\n")
|
||||
|
||||
return {
|
||||
"test_name": test_name,
|
||||
"num_files": num_files,
|
||||
"sync_duration_sec": sync_duration,
|
||||
"files_per_sec": files_per_sec,
|
||||
"ms_per_file": ms_per_file,
|
||||
"new_files": len(report.new),
|
||||
"modified_files": len(report.modified),
|
||||
"deleted_files": len(report.deleted),
|
||||
"moved_files": len(report.moves),
|
||||
"initial_db_size": initial_db_size,
|
||||
"final_db_size": final_db_size,
|
||||
"db_growth_bytes": db_growth,
|
||||
"db_growth_per_file_bytes": db_growth / num_files if num_files > 0 else 0,
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.benchmark
|
||||
@pytest.mark.asyncio
|
||||
async def test_benchmark_sync_100_files(app_config, project_config, config_manager):
|
||||
"""Benchmark: Sync 100 files (small repository)."""
|
||||
results = await run_sync_benchmark(
|
||||
project_config, app_config, num_files=100, test_name="Sync 100 files (small repository)"
|
||||
)
|
||||
|
||||
# Basic assertions to ensure sync worked
|
||||
# Note: May be slightly more than 100 due to OS-generated files (.DS_Store, etc.)
|
||||
assert results["new_files"] >= 100
|
||||
assert results["sync_duration_sec"] > 0
|
||||
assert results["files_per_sec"] > 0
|
||||
|
||||
|
||||
@pytest.mark.benchmark
|
||||
@pytest.mark.asyncio
|
||||
async def test_benchmark_sync_500_files(app_config, project_config, config_manager):
|
||||
"""Benchmark: Sync 500 files (medium repository)."""
|
||||
results = await run_sync_benchmark(
|
||||
project_config, app_config, num_files=500, test_name="Sync 500 files (medium repository)"
|
||||
)
|
||||
|
||||
# Basic assertions
|
||||
# Note: May be slightly more than 500 due to OS-generated files
|
||||
assert results["new_files"] >= 500
|
||||
assert results["sync_duration_sec"] > 0
|
||||
assert results["files_per_sec"] > 0
|
||||
|
||||
|
||||
@pytest.mark.benchmark
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.slow
|
||||
async def test_benchmark_sync_1000_files(app_config, project_config, config_manager):
|
||||
"""Benchmark: Sync 1000 files (large repository).
|
||||
|
||||
This test is marked as 'slow' and can be skipped in regular test runs:
|
||||
pytest -m "not slow"
|
||||
"""
|
||||
results = await run_sync_benchmark(
|
||||
project_config, app_config, num_files=1000, test_name="Sync 1000 files (large repository)"
|
||||
)
|
||||
|
||||
# Basic assertions
|
||||
# Note: May be slightly more than 1000 due to OS-generated files
|
||||
assert results["new_files"] >= 1000
|
||||
assert results["sync_duration_sec"] > 0
|
||||
assert results["files_per_sec"] > 0
|
||||
|
||||
|
||||
@pytest.mark.benchmark
|
||||
@pytest.mark.asyncio
|
||||
async def test_benchmark_resync_no_changes(app_config, project_config, config_manager):
|
||||
"""Benchmark: Re-sync with no changes (should be fast).
|
||||
|
||||
This tests the performance of scanning files when nothing has changed,
|
||||
which is important for cloud restarts.
|
||||
"""
|
||||
project_dir = project_config.home
|
||||
num_files = 100
|
||||
|
||||
# First sync
|
||||
print(f"\nFirst sync of {num_files} files...")
|
||||
await generate_benchmark_files(project_dir, num_files)
|
||||
|
||||
from basic_memory.repository import ProjectRepository
|
||||
from basic_memory import db
|
||||
|
||||
_, session_maker = await db.get_or_create_db(
|
||||
db_path=app_config.database_path,
|
||||
db_type=db.DatabaseType.FILESYSTEM,
|
||||
)
|
||||
project_repository = ProjectRepository(session_maker)
|
||||
projects = await project_repository.find_all()
|
||||
if projects:
|
||||
project = projects[0]
|
||||
else:
|
||||
project = await project_repository.create(
|
||||
{
|
||||
"name": project_config.name,
|
||||
"path": str(project_config.home),
|
||||
"is_active": True,
|
||||
"is_default": True,
|
||||
}
|
||||
)
|
||||
|
||||
sync_service = await get_sync_service(project)
|
||||
|
||||
# Initialize search index
|
||||
await sync_service.search_service.init_search_index()
|
||||
|
||||
await sync_service.sync(project_dir, project_name=project.name)
|
||||
|
||||
# Second sync (no changes)
|
||||
print("\nRe-sync with no changes...")
|
||||
resync_start = time.time()
|
||||
report = await sync_service.sync(project_dir, project_name=project.name)
|
||||
resync_duration = time.time() - resync_start
|
||||
|
||||
print(f"\n{'-' * 70}")
|
||||
print("RE-SYNC RESULTS (no changes):")
|
||||
print(f"{'-' * 70}")
|
||||
print(f"Files scanned: {num_files}")
|
||||
print(f"Changes detected: {report.total}")
|
||||
print(f" New: {len(report.new)}")
|
||||
print(f" Modified: {len(report.modified)}")
|
||||
print(f" Deleted: {len(report.deleted)}")
|
||||
print(f" Moved: {len(report.moves)}")
|
||||
print(f"Duration: {resync_duration:.2f}s")
|
||||
print(f"Files/sec: {num_files / resync_duration:.1f}")
|
||||
|
||||
# Debug: Show what changed
|
||||
if report.total > 0:
|
||||
print("\n⚠️ UNEXPECTED CHANGES DETECTED:")
|
||||
if report.new:
|
||||
print(f" New files ({len(report.new)}): {list(report.new)[:5]}")
|
||||
if report.modified:
|
||||
print(f" Modified files ({len(report.modified)}): {list(report.modified)[:5]}")
|
||||
if report.deleted:
|
||||
print(f" Deleted files ({len(report.deleted)}): {list(report.deleted)[:5]}")
|
||||
if report.moves:
|
||||
print(f" Moved files ({len(report.moves)}): {dict(list(report.moves.items())[:5])}")
|
||||
|
||||
print(f"{'=' * 70}\n")
|
||||
|
||||
# Should be no changes
|
||||
assert report.total == 0, (
|
||||
f"Expected no changes but got {report.total}: new={len(report.new)}, modified={len(report.modified)}, deleted={len(report.deleted)}, moves={len(report.moves)}"
|
||||
)
|
||||
assert len(report.new) == 0
|
||||
assert len(report.modified) == 0
|
||||
assert len(report.deleted) == 0
|
||||
@@ -9,17 +9,19 @@ from basic_memory.mcp.async_client import create_client
|
||||
|
||||
|
||||
def test_create_client_uses_asgi_when_no_remote_env():
|
||||
"""Test that create_client uses ASGI transport when BASIC_MEMORY_USE_REMOTE_API is not set."""
|
||||
# Ensure env vars are not set (pop if they exist)
|
||||
"""Test that create_client uses ASGI transport when cloud mode is disabled."""
|
||||
# Ensure env vars are not set and config cloud_mode is False
|
||||
with patch.dict("os.environ", clear=False):
|
||||
os.environ.pop("BASIC_MEMORY_USE_REMOTE_API", None)
|
||||
os.environ.pop("BASIC_MEMORY_CLOUD_MODE", None)
|
||||
|
||||
client = create_client()
|
||||
# Also patch the config's cloud_mode to ensure it's False
|
||||
with patch.object(ConfigManager().config, "cloud_mode", False):
|
||||
client = create_client()
|
||||
|
||||
assert isinstance(client, AsyncClient)
|
||||
assert isinstance(client._transport, ASGITransport)
|
||||
assert str(client.base_url) == "http://test"
|
||||
assert isinstance(client, AsyncClient)
|
||||
assert isinstance(client._transport, ASGITransport)
|
||||
assert str(client.base_url) == "http://test"
|
||||
|
||||
|
||||
def test_create_client_uses_http_when_cloud_mode_env_set():
|
||||
@@ -37,16 +39,18 @@ def test_create_client_uses_http_when_cloud_mode_env_set():
|
||||
|
||||
def test_create_client_configures_extended_timeouts():
|
||||
"""Test that create_client configures 30-second timeouts for long operations."""
|
||||
# Ensure env vars are not set (pop if they exist)
|
||||
# Ensure env vars are not set and config cloud_mode is False
|
||||
with patch.dict("os.environ", clear=False):
|
||||
os.environ.pop("BASIC_MEMORY_USE_REMOTE_API", None)
|
||||
os.environ.pop("BASIC_MEMORY_CLOUD_MODE", None)
|
||||
|
||||
client = create_client()
|
||||
# Also patch the config's cloud_mode to ensure it's False
|
||||
with patch.object(ConfigManager().config, "cloud_mode", False):
|
||||
client = create_client()
|
||||
|
||||
# Verify timeout configuration
|
||||
assert isinstance(client.timeout, Timeout)
|
||||
assert client.timeout.connect == 10.0 # 10 seconds for connection
|
||||
assert client.timeout.read == 30.0 # 30 seconds for reading
|
||||
assert client.timeout.write == 30.0 # 30 seconds for writing
|
||||
assert client.timeout.pool == 30.0 # 30 seconds for pool
|
||||
# Verify timeout configuration
|
||||
assert isinstance(client.timeout, Timeout)
|
||||
assert client.timeout.connect == 10.0 # 10 seconds for connection
|
||||
assert client.timeout.read == 30.0 # 30 seconds for reading
|
||||
assert client.timeout.write == 30.0 # 30 seconds for writing
|
||||
assert client.timeout.pool == 30.0 # 30 seconds for pool
|
||||
|
||||
@@ -18,6 +18,7 @@ def template_loader():
|
||||
def entity_summary():
|
||||
"""Create a sample EntitySummary for testing."""
|
||||
return EntitySummary(
|
||||
entity_id=1,
|
||||
title="Test Entity",
|
||||
permalink="test/entity",
|
||||
type=SearchItemType.ENTITY,
|
||||
@@ -34,6 +35,8 @@ def context_with_results(entity_summary):
|
||||
|
||||
# Create an observation for the entity
|
||||
observation = ObservationSummary(
|
||||
observation_id=1,
|
||||
entity_id=1,
|
||||
title="Test Observation",
|
||||
permalink="test/entity/observations/1",
|
||||
category="test",
|
||||
|
||||
@@ -218,9 +218,13 @@ async def test_get_resource_entities(client, project_config, entity_repository,
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_resource_entities_pagination(
|
||||
client, project_config, entity_repository, project_url
|
||||
client, project_config, entity_repository, project_url, db_backend
|
||||
):
|
||||
"""Test getting content by permalink match."""
|
||||
if db_backend == "postgres":
|
||||
pytest.skip(
|
||||
"Pagination differs: relations expand to multiple entities, ordering is undefined"
|
||||
)
|
||||
# Create entity
|
||||
content1 = "# Test Content\n"
|
||||
data = {
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
"""V2 API tests."""
|
||||
@@ -0,0 +1,21 @@
|
||||
"""Fixtures for V2 API tests."""
|
||||
|
||||
import pytest
|
||||
|
||||
from basic_memory.models import Project
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def v2_project_url(test_project: Project) -> str:
|
||||
"""Create a URL prefix for v2 project-scoped routes using project ID.
|
||||
|
||||
This helps tests generate the correct URL for v2 project-scoped routes
|
||||
which use integer project IDs instead of permalinks.
|
||||
"""
|
||||
return f"/v2/projects/{test_project.id}"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def v2_projects_url() -> str:
|
||||
"""Base URL for v2 project management endpoints."""
|
||||
return "/v2/projects"
|
||||
@@ -0,0 +1,129 @@
|
||||
"""Tests for V2 directory API routes (ID-based endpoints)."""
|
||||
|
||||
import pytest
|
||||
from httpx import AsyncClient
|
||||
|
||||
from basic_memory.models import Project
|
||||
from basic_memory.schemas.directory import DirectoryNode
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_directory_tree(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
):
|
||||
"""Test getting directory tree via v2 endpoint."""
|
||||
response = await client.get(f"{v2_project_url}/directory/tree")
|
||||
|
||||
assert response.status_code == 200
|
||||
tree = DirectoryNode.model_validate(response.json())
|
||||
assert tree.type == "directory"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_directory_structure(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
):
|
||||
"""Test getting directory structure (folders only) via v2 endpoint."""
|
||||
response = await client.get(f"{v2_project_url}/directory/structure")
|
||||
|
||||
assert response.status_code == 200
|
||||
structure = DirectoryNode.model_validate(response.json())
|
||||
assert structure.type == "directory"
|
||||
# Structure should only contain directories, not files
|
||||
if structure.children:
|
||||
for child in structure.children:
|
||||
assert child.type == "directory"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_list_directory_default(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
):
|
||||
"""Test listing directory contents with default parameters via v2 endpoint."""
|
||||
response = await client.get(f"{v2_project_url}/directory/list")
|
||||
|
||||
assert response.status_code == 200
|
||||
nodes = response.json()
|
||||
assert isinstance(nodes, list)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_list_directory_with_depth(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
):
|
||||
"""Test listing directory with custom depth via v2 endpoint."""
|
||||
response = await client.get(f"{v2_project_url}/directory/list?depth=2")
|
||||
|
||||
assert response.status_code == 200
|
||||
nodes = response.json()
|
||||
assert isinstance(nodes, list)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_list_directory_with_glob(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
):
|
||||
"""Test listing directory with file name glob filter via v2 endpoint."""
|
||||
response = await client.get(f"{v2_project_url}/directory/list?file_name_glob=*.md")
|
||||
|
||||
assert response.status_code == 200
|
||||
nodes = response.json()
|
||||
assert isinstance(nodes, list)
|
||||
# All file nodes should have .md extension
|
||||
for node in nodes:
|
||||
if node.get("type") == "file":
|
||||
assert node.get("path", "").endswith(".md")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_list_directory_with_custom_path(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
):
|
||||
"""Test listing a specific directory path via v2 endpoint."""
|
||||
response = await client.get(f"{v2_project_url}/directory/list?dir_name=/")
|
||||
|
||||
assert response.status_code == 200
|
||||
nodes = response.json()
|
||||
assert isinstance(nodes, list)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_directory_invalid_project_id(
|
||||
client: AsyncClient,
|
||||
):
|
||||
"""Test directory endpoints with invalid project ID return 404."""
|
||||
# Test tree endpoint
|
||||
response = await client.get("/v2/projects/999999/directory/tree")
|
||||
assert response.status_code == 404
|
||||
|
||||
# Test structure endpoint
|
||||
response = await client.get("/v2/projects/999999/directory/structure")
|
||||
assert response.status_code == 404
|
||||
|
||||
# Test list endpoint
|
||||
response = await client.get("/v2/projects/999999/directory/list")
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_v2_directory_endpoints_use_project_id_not_name(
|
||||
client: AsyncClient, test_project: Project
|
||||
):
|
||||
"""Verify v2 directory endpoints require project ID, not name."""
|
||||
# Try using project name instead of ID - should fail
|
||||
response = await client.get(f"/v2/projects/{test_project.name}/directory/tree")
|
||||
|
||||
# Should get validation error or 404 because name is not a valid integer
|
||||
assert response.status_code in [404, 422]
|
||||
@@ -0,0 +1,530 @@
|
||||
"""Tests for V2 importer API routes (ID-based endpoints)."""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from httpx import AsyncClient
|
||||
|
||||
from basic_memory.models import Project
|
||||
from basic_memory.schemas.importer import (
|
||||
ChatImportResult,
|
||||
EntityImportResult,
|
||||
ProjectImportResult,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def chatgpt_json_content():
|
||||
"""Sample ChatGPT conversation data for testing."""
|
||||
return [
|
||||
{
|
||||
"title": "Test Conversation",
|
||||
"create_time": 1736616594.24054,
|
||||
"update_time": 1736616603.164995,
|
||||
"mapping": {
|
||||
"root": {"id": "root", "message": None, "parent": None, "children": ["msg1"]},
|
||||
"msg1": {
|
||||
"id": "msg1",
|
||||
"message": {
|
||||
"id": "msg1",
|
||||
"author": {"role": "user", "name": None, "metadata": {}},
|
||||
"create_time": 1736616594.24054,
|
||||
"content": {
|
||||
"content_type": "text",
|
||||
"parts": ["Hello, this is a test message"],
|
||||
},
|
||||
"status": "finished_successfully",
|
||||
"metadata": {},
|
||||
},
|
||||
"parent": "root",
|
||||
"children": ["msg2"],
|
||||
},
|
||||
"msg2": {
|
||||
"id": "msg2",
|
||||
"message": {
|
||||
"id": "msg2",
|
||||
"author": {"role": "assistant", "name": None, "metadata": {}},
|
||||
"create_time": 1736616603.164995,
|
||||
"content": {"content_type": "text", "parts": ["This is a test response"]},
|
||||
"status": "finished_successfully",
|
||||
"metadata": {},
|
||||
},
|
||||
"parent": "msg1",
|
||||
"children": [],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def claude_conversations_json_content():
|
||||
"""Sample Claude conversations data for testing."""
|
||||
return [
|
||||
{
|
||||
"uuid": "test-uuid",
|
||||
"name": "Test Conversation",
|
||||
"created_at": "2025-01-05T20:55:32.499880+00:00",
|
||||
"updated_at": "2025-01-05T20:56:39.477600+00:00",
|
||||
"chat_messages": [
|
||||
{
|
||||
"uuid": "msg-1",
|
||||
"text": "Hello, this is a test",
|
||||
"sender": "human",
|
||||
"created_at": "2025-01-05T20:55:32.499880+00:00",
|
||||
"content": [{"type": "text", "text": "Hello, this is a test"}],
|
||||
},
|
||||
{
|
||||
"uuid": "msg-2",
|
||||
"text": "Response to test",
|
||||
"sender": "assistant",
|
||||
"created_at": "2025-01-05T20:55:40.123456+00:00",
|
||||
"content": [{"type": "text", "text": "Response to test"}],
|
||||
},
|
||||
],
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def claude_projects_json_content():
|
||||
"""Sample Claude projects data for testing."""
|
||||
return [
|
||||
{
|
||||
"uuid": "test-uuid",
|
||||
"name": "Test Project",
|
||||
"created_at": "2025-01-05T20:55:32.499880+00:00",
|
||||
"updated_at": "2025-01-05T20:56:39.477600+00:00",
|
||||
"prompt_template": "# Test Prompt\n\nThis is a test prompt.",
|
||||
"docs": [
|
||||
{
|
||||
"uuid": "doc-uuid-1",
|
||||
"filename": "Test Document",
|
||||
"content": "# Test Document\n\nThis is test content.",
|
||||
"created_at": "2025-01-05T20:56:39.477600+00:00",
|
||||
},
|
||||
{
|
||||
"uuid": "doc-uuid-2",
|
||||
"filename": "Another Document",
|
||||
"content": "# Another Document\n\nMore test content.",
|
||||
"created_at": "2025-01-05T20:56:39.477600+00:00",
|
||||
},
|
||||
],
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def memory_json_content():
|
||||
"""Sample memory.json data for testing."""
|
||||
return [
|
||||
{
|
||||
"type": "entity",
|
||||
"name": "test_entity",
|
||||
"entityType": "test",
|
||||
"observations": ["Test observation 1", "Test observation 2"],
|
||||
},
|
||||
{
|
||||
"type": "relation",
|
||||
"from": "test_entity",
|
||||
"to": "related_entity",
|
||||
"relationType": "test_relation",
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
async def create_test_upload_file(tmp_path, content):
|
||||
"""Create a test file for upload."""
|
||||
file_path = tmp_path / "test_import.json"
|
||||
with open(file_path, "w", encoding="utf-8") as f:
|
||||
json.dump(content, f)
|
||||
|
||||
return file_path
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_import_chatgpt(
|
||||
project_config,
|
||||
client: AsyncClient,
|
||||
tmp_path,
|
||||
chatgpt_json_content,
|
||||
file_service,
|
||||
v2_project_url: str,
|
||||
):
|
||||
"""Test importing ChatGPT conversations via v2 endpoint."""
|
||||
# Create a test file
|
||||
file_path = await create_test_upload_file(tmp_path, chatgpt_json_content)
|
||||
|
||||
# Create a multipart form with the file
|
||||
with open(file_path, "rb") as f:
|
||||
files = {"file": ("conversations.json", f, "application/json")}
|
||||
data = {"folder": "test_chatgpt"}
|
||||
|
||||
# Send request
|
||||
response = await client.post(f"{v2_project_url}/import/chatgpt", files=files, data=data)
|
||||
|
||||
# Check response
|
||||
assert response.status_code == 200
|
||||
result = ChatImportResult.model_validate(response.json())
|
||||
assert result.success is True
|
||||
assert result.conversations == 1
|
||||
assert result.messages == 2
|
||||
|
||||
# Verify files were created
|
||||
conv_path = Path("test_chatgpt") / "20250111-Test_Conversation.md"
|
||||
assert await file_service.exists(conv_path)
|
||||
|
||||
content, _ = await file_service.read_file(conv_path)
|
||||
assert "# Test Conversation" in content
|
||||
assert "Hello, this is a test message" in content
|
||||
assert "This is a test response" in content
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_import_chatgpt_invalid_file(client: AsyncClient, tmp_path, v2_project_url: str):
|
||||
"""Test importing invalid ChatGPT file via v2 endpoint."""
|
||||
# Create invalid file
|
||||
file_path = tmp_path / "invalid.json"
|
||||
with open(file_path, "w") as f:
|
||||
f.write("This is not JSON")
|
||||
|
||||
# Create multipart form with invalid file
|
||||
with open(file_path, "rb") as f:
|
||||
files = {"file": ("invalid.json", f, "application/json")}
|
||||
data = {"folder": "test_chatgpt"}
|
||||
|
||||
# Send request - this should return an error
|
||||
response = await client.post(f"{v2_project_url}/import/chatgpt", files=files, data=data)
|
||||
|
||||
# Check response
|
||||
assert response.status_code == 500
|
||||
assert "Import failed" in response.json()["detail"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_import_claude_conversations(
|
||||
client: AsyncClient,
|
||||
tmp_path,
|
||||
claude_conversations_json_content,
|
||||
file_service,
|
||||
v2_project_url: str,
|
||||
):
|
||||
"""Test importing Claude conversations via v2 endpoint."""
|
||||
# Create a test file
|
||||
file_path = await create_test_upload_file(tmp_path, claude_conversations_json_content)
|
||||
|
||||
# Create a multipart form with the file
|
||||
with open(file_path, "rb") as f:
|
||||
files = {"file": ("conversations.json", f, "application/json")}
|
||||
data = {"folder": "test_claude_conversations"}
|
||||
|
||||
# Send request
|
||||
response = await client.post(
|
||||
f"{v2_project_url}/import/claude/conversations", files=files, data=data
|
||||
)
|
||||
|
||||
# Check response
|
||||
assert response.status_code == 200
|
||||
result = ChatImportResult.model_validate(response.json())
|
||||
assert result.success is True
|
||||
assert result.conversations == 1
|
||||
assert result.messages == 2
|
||||
|
||||
# Verify files were created
|
||||
conv_path = Path("test_claude_conversations") / "20250105-Test_Conversation.md"
|
||||
assert await file_service.exists(conv_path)
|
||||
|
||||
content, _ = await file_service.read_file(conv_path)
|
||||
assert "# Test Conversation" in content
|
||||
assert "Hello, this is a test" in content
|
||||
assert "Response to test" in content
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_import_claude_conversations_invalid_file(
|
||||
client: AsyncClient, tmp_path, v2_project_url: str
|
||||
):
|
||||
"""Test importing invalid Claude conversations file via v2 endpoint."""
|
||||
# Create invalid file
|
||||
file_path = tmp_path / "invalid.json"
|
||||
with open(file_path, "w") as f:
|
||||
f.write("This is not JSON")
|
||||
|
||||
# Create multipart form with invalid file
|
||||
with open(file_path, "rb") as f:
|
||||
files = {"file": ("invalid.json", f, "application/json")}
|
||||
data = {"folder": "test_claude_conversations"}
|
||||
|
||||
# Send request - this should return an error
|
||||
response = await client.post(
|
||||
f"{v2_project_url}/import/claude/conversations", files=files, data=data
|
||||
)
|
||||
|
||||
# Check response
|
||||
assert response.status_code == 500
|
||||
assert "Import failed" in response.json()["detail"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_import_claude_projects(
|
||||
client: AsyncClient, tmp_path, claude_projects_json_content, file_service, v2_project_url: str
|
||||
):
|
||||
"""Test importing Claude projects via v2 endpoint."""
|
||||
# Create a test file
|
||||
file_path = await create_test_upload_file(tmp_path, claude_projects_json_content)
|
||||
|
||||
# Create a multipart form with the file
|
||||
with open(file_path, "rb") as f:
|
||||
files = {"file": ("projects.json", f, "application/json")}
|
||||
data = {"folder": "test_claude_projects"}
|
||||
|
||||
# Send request
|
||||
response = await client.post(
|
||||
f"{v2_project_url}/import/claude/projects", files=files, data=data
|
||||
)
|
||||
|
||||
# Check response
|
||||
assert response.status_code == 200
|
||||
result = ProjectImportResult.model_validate(response.json())
|
||||
assert result.success is True
|
||||
assert result.documents == 2
|
||||
assert result.prompts == 1
|
||||
|
||||
# Verify files were created
|
||||
project_dir = Path("test_claude_projects") / "Test_Project"
|
||||
assert await file_service.exists(project_dir / "prompt-template.md")
|
||||
assert await file_service.exists(project_dir / "docs" / "Test_Document.md")
|
||||
assert await file_service.exists(project_dir / "docs" / "Another_Document.md")
|
||||
|
||||
# Check content
|
||||
prompt_content, _ = await file_service.read_file(project_dir / "prompt-template.md")
|
||||
assert "# Test Prompt" in prompt_content
|
||||
|
||||
doc_content, _ = await file_service.read_file(project_dir / "docs" / "Test_Document.md")
|
||||
assert "# Test Document" in doc_content
|
||||
assert "This is test content" in doc_content
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_import_claude_projects_invalid_file(
|
||||
client: AsyncClient, tmp_path, v2_project_url: str
|
||||
):
|
||||
"""Test importing invalid Claude projects file via v2 endpoint."""
|
||||
# Create invalid file
|
||||
file_path = tmp_path / "invalid.json"
|
||||
with open(file_path, "w") as f:
|
||||
f.write("This is not JSON")
|
||||
|
||||
# Create multipart form with invalid file
|
||||
with open(file_path, "rb") as f:
|
||||
files = {"file": ("invalid.json", f, "application/json")}
|
||||
data = {"folder": "test_claude_projects"}
|
||||
|
||||
# Send request - this should return an error
|
||||
response = await client.post(
|
||||
f"{v2_project_url}/import/claude/projects", files=files, data=data
|
||||
)
|
||||
|
||||
# Check response
|
||||
assert response.status_code == 500
|
||||
assert "Import failed" in response.json()["detail"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_import_memory_json(
|
||||
client: AsyncClient, tmp_path, memory_json_content, file_service, v2_project_url: str
|
||||
):
|
||||
"""Test importing memory.json file via v2 endpoint."""
|
||||
# Create a test file
|
||||
json_file = tmp_path / "memory.json"
|
||||
with open(json_file, "w", encoding="utf-8") as f:
|
||||
for entity in memory_json_content:
|
||||
f.write(json.dumps(entity) + "\n")
|
||||
|
||||
# Create a multipart form with the file
|
||||
with open(json_file, "rb") as f:
|
||||
files = {"file": ("memory.json", f, "application/json")}
|
||||
data = {"folder": "test_memory_json"}
|
||||
|
||||
# Send request
|
||||
response = await client.post(f"{v2_project_url}/import/memory-json", files=files, data=data)
|
||||
|
||||
# Check response
|
||||
assert response.status_code == 200
|
||||
result = EntityImportResult.model_validate(response.json())
|
||||
assert result.success is True
|
||||
assert result.entities == 1
|
||||
assert result.relations == 1
|
||||
|
||||
# Verify files were created
|
||||
entity_path = Path("test_memory_json") / "test" / "test_entity.md"
|
||||
assert await file_service.exists(entity_path)
|
||||
|
||||
# Check content
|
||||
content, _ = await file_service.read_file(entity_path)
|
||||
assert "Test observation 1" in content
|
||||
assert "Test observation 2" in content
|
||||
assert "test_relation [[related_entity]]" in content
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_import_memory_json_without_folder(
|
||||
client: AsyncClient, tmp_path, memory_json_content, file_service, v2_project_url: str
|
||||
):
|
||||
"""Test importing memory.json file without specifying a destination folder."""
|
||||
# Create a test file
|
||||
json_file = tmp_path / "memory.json"
|
||||
with open(json_file, "w", encoding="utf-8") as f:
|
||||
for entity in memory_json_content:
|
||||
f.write(json.dumps(entity) + "\n")
|
||||
|
||||
# Create a multipart form with the file
|
||||
with open(json_file, "rb") as f:
|
||||
files = {"file": ("memory.json", f, "application/json")}
|
||||
|
||||
# Send request without destination_folder
|
||||
response = await client.post(f"{v2_project_url}/import/memory-json", files=files)
|
||||
|
||||
# Check response
|
||||
assert response.status_code == 200
|
||||
result = EntityImportResult.model_validate(response.json())
|
||||
assert result.success is True
|
||||
assert result.entities == 1
|
||||
assert result.relations == 1
|
||||
|
||||
# Verify files were created in the default directory
|
||||
entity_path = Path("conversations") / "test" / "test_entity.md"
|
||||
assert await file_service.exists(entity_path)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_import_memory_json_invalid_file(client: AsyncClient, tmp_path, v2_project_url: str):
|
||||
"""Test importing invalid memory.json file via v2 endpoint."""
|
||||
# Create invalid file
|
||||
file_path = tmp_path / "invalid.json"
|
||||
with open(file_path, "w") as f:
|
||||
f.write("This is not JSON")
|
||||
|
||||
# Create multipart form with invalid file
|
||||
with open(file_path, "rb") as f:
|
||||
files = {"file": ("invalid.json", f, "application/json")}
|
||||
data = {"folder": "test_memory_json"}
|
||||
|
||||
# Send request - this should return an error
|
||||
response = await client.post(f"{v2_project_url}/import/memory-json", files=files, data=data)
|
||||
|
||||
# Check response
|
||||
assert response.status_code == 500
|
||||
assert "Import failed" in response.json()["detail"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_v2_import_endpoints_use_project_id_not_name(
|
||||
client: AsyncClient, tmp_path, test_project: Project, chatgpt_json_content
|
||||
):
|
||||
"""Verify v2 import endpoints require project ID, not name."""
|
||||
# Create a test file
|
||||
file_path = await create_test_upload_file(tmp_path, chatgpt_json_content)
|
||||
|
||||
# Try using project name instead of ID - should fail
|
||||
with open(file_path, "rb") as f:
|
||||
files = {"file": ("conversations.json", f, "application/json")}
|
||||
data = {"folder": "test"}
|
||||
|
||||
response = await client.post(
|
||||
f"/v2/projects/{test_project.name}/import/chatgpt",
|
||||
files=files,
|
||||
data=data,
|
||||
)
|
||||
|
||||
# Should get validation error or 404 because name is not a valid integer
|
||||
assert response.status_code in [404, 422]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_import_invalid_project_id(client: AsyncClient, tmp_path, chatgpt_json_content):
|
||||
"""Test import endpoints with invalid project ID return 404."""
|
||||
# Create a test file
|
||||
file_path = await create_test_upload_file(tmp_path, chatgpt_json_content)
|
||||
|
||||
# Test all import endpoints
|
||||
endpoints = [
|
||||
"/import/chatgpt",
|
||||
"/import/claude/conversations",
|
||||
"/import/claude/projects",
|
||||
"/import/memory-json",
|
||||
]
|
||||
|
||||
for endpoint in endpoints:
|
||||
with open(file_path, "rb") as f:
|
||||
files = {"file": ("test.json", f, "application/json")}
|
||||
data = {"folder": "test"}
|
||||
|
||||
response = await client.post(
|
||||
f"/v2/projects/999999{endpoint}",
|
||||
files=files,
|
||||
data=data,
|
||||
)
|
||||
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_import_missing_file(client: AsyncClient, v2_project_url: str):
|
||||
"""Test importing with missing file via v2 endpoint."""
|
||||
# Send a request without a file
|
||||
response = await client.post(f"{v2_project_url}/import/chatgpt", data={"folder": "test_folder"})
|
||||
|
||||
# Check that the request was rejected
|
||||
assert response.status_code in [400, 422] # Either bad request or unprocessable entity
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_import_empty_file(client: AsyncClient, tmp_path, v2_project_url: str):
|
||||
"""Test importing an empty file via v2 endpoint."""
|
||||
# Create an empty file
|
||||
file_path = tmp_path / "empty.json"
|
||||
with open(file_path, "w") as f:
|
||||
f.write("")
|
||||
|
||||
# Create multipart form with empty file
|
||||
with open(file_path, "rb") as f:
|
||||
files = {"file": ("empty.json", f, "application/json")}
|
||||
data = {"folder": "test_chatgpt"}
|
||||
|
||||
# Send request
|
||||
response = await client.post(f"{v2_project_url}/import/chatgpt", files=files, data=data)
|
||||
|
||||
# Check response
|
||||
assert response.status_code == 500
|
||||
assert "Import failed" in response.json()["detail"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_import_malformed_json(client: AsyncClient, tmp_path, v2_project_url: str):
|
||||
"""Test importing malformed JSON for all v2 import endpoints."""
|
||||
# Create malformed JSON file
|
||||
file_path = tmp_path / "malformed.json"
|
||||
with open(file_path, "w") as f:
|
||||
f.write('{"incomplete": "json"') # Missing closing brace
|
||||
|
||||
# Test all import endpoints
|
||||
endpoints = [
|
||||
(f"{v2_project_url}/import/chatgpt", {"folder": "test"}),
|
||||
(f"{v2_project_url}/import/claude/conversations", {"folder": "test"}),
|
||||
(f"{v2_project_url}/import/claude/projects", {"folder": "test"}),
|
||||
(f"{v2_project_url}/import/memory-json", {"folder": "test"}),
|
||||
]
|
||||
|
||||
for endpoint, data in endpoints:
|
||||
# Create multipart form with malformed JSON
|
||||
with open(file_path, "rb") as f:
|
||||
files = {"file": ("malformed.json", f, "application/json")}
|
||||
|
||||
# Send request
|
||||
response = await client.post(endpoint, files=files, data=data)
|
||||
|
||||
# Check response
|
||||
assert response.status_code == 500
|
||||
assert "Import failed" in response.json()["detail"]
|
||||
@@ -0,0 +1,407 @@
|
||||
"""Tests for V2 knowledge graph API routes (ID-based endpoints)."""
|
||||
|
||||
import pytest
|
||||
from httpx import AsyncClient
|
||||
|
||||
from basic_memory.models import Project
|
||||
from basic_memory.schemas import DeleteEntitiesResponse
|
||||
from basic_memory.schemas.v2 import EntityResponseV2, EntityResolveResponse
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_resolve_identifier_by_permalink(
|
||||
client: AsyncClient, test_graph, v2_project_url, test_project: Project, entity_repository
|
||||
):
|
||||
"""Test resolving an identifier by permalink returns correct entity ID."""
|
||||
# test_graph fixture creates some test entities
|
||||
# We'll use one of them to test resolution
|
||||
|
||||
# Create an entity first
|
||||
entity_data = {
|
||||
"title": "TestResolve",
|
||||
"folder": "test",
|
||||
"content": "Test content for resolve",
|
||||
}
|
||||
response = await client.post(f"{v2_project_url}/knowledge/entities", json=entity_data)
|
||||
assert response.status_code == 200
|
||||
created_entity = EntityResponseV2.model_validate(response.json())
|
||||
|
||||
# V2 create must return id
|
||||
assert created_entity.id is not None
|
||||
entity_id = created_entity.id
|
||||
|
||||
# Now resolve it by permalink
|
||||
resolve_data = {"identifier": created_entity.permalink}
|
||||
response = await client.post(f"{v2_project_url}/knowledge/resolve", json=resolve_data)
|
||||
|
||||
assert response.status_code == 200
|
||||
resolved = EntityResolveResponse.model_validate(response.json())
|
||||
assert resolved.entity_id == entity_id
|
||||
assert resolved.permalink == created_entity.permalink
|
||||
assert resolved.resolution_method == "permalink"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_resolve_identifier_not_found(client: AsyncClient, v2_project_url):
|
||||
"""Test resolving a non-existent identifier returns 404."""
|
||||
resolve_data = {"identifier": "nonexistent/entity"}
|
||||
response = await client.post(f"{v2_project_url}/knowledge/resolve", json=resolve_data)
|
||||
|
||||
assert response.status_code == 404
|
||||
assert "Could not resolve identifier" in response.json()["detail"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_entity_by_id(client: AsyncClient, test_graph, v2_project_url, entity_repository):
|
||||
"""Test getting an entity by its numeric ID."""
|
||||
# Create an entity first
|
||||
entity_data = {
|
||||
"title": "TestGetById",
|
||||
"folder": "test",
|
||||
"content": "Test content for get by ID",
|
||||
}
|
||||
response = await client.post(f"{v2_project_url}/knowledge/entities", json=entity_data)
|
||||
assert response.status_code == 200
|
||||
created_entity = EntityResponseV2.model_validate(response.json())
|
||||
|
||||
# V2 create must return id
|
||||
assert created_entity.id is not None
|
||||
entity_id = created_entity.id
|
||||
|
||||
# Get it by ID using v2 endpoint
|
||||
response = await client.get(f"{v2_project_url}/knowledge/entities/{entity_id}")
|
||||
|
||||
assert response.status_code == 200
|
||||
entity = EntityResponseV2.model_validate(response.json())
|
||||
assert entity.id == entity_id
|
||||
assert entity.title == "TestGetById"
|
||||
assert entity.api_version == "v2"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_entity_by_id_not_found(client: AsyncClient, v2_project_url):
|
||||
"""Test getting a non-existent entity by ID returns 404."""
|
||||
response = await client.get(f"{v2_project_url}/knowledge/entities/999999")
|
||||
|
||||
assert response.status_code == 404
|
||||
assert "not found" in response.json()["detail"].lower()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_create_entity(client: AsyncClient, file_service, v2_project_url):
|
||||
"""Test creating an entity via v2 endpoint."""
|
||||
data = {
|
||||
"title": "TestV2Entity",
|
||||
"folder": "test",
|
||||
"entity_type": "test",
|
||||
"content_type": "text/markdown",
|
||||
"content": "TestContent for V2",
|
||||
}
|
||||
|
||||
response = await client.post(f"{v2_project_url}/knowledge/entities", json=data)
|
||||
|
||||
assert response.status_code == 200
|
||||
entity = EntityResponseV2.model_validate(response.json())
|
||||
|
||||
# V2 endpoints must return id field
|
||||
assert entity.id is not None
|
||||
assert isinstance(entity.id, int)
|
||||
assert entity.api_version == "v2"
|
||||
|
||||
assert entity.permalink == "test/test-v2-entity"
|
||||
assert entity.file_path == "test/TestV2Entity.md"
|
||||
assert entity.entity_type == data["entity_type"]
|
||||
|
||||
# Verify file was created
|
||||
file_path = file_service.get_entity_path(entity)
|
||||
file_content, _ = await file_service.read_file(file_path)
|
||||
assert data["content"] in file_content
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_create_entity_with_observations_and_relations(
|
||||
client: AsyncClient, file_service, v2_project_url
|
||||
):
|
||||
"""Test creating an entity with observations and relations via v2."""
|
||||
data = {
|
||||
"title": "TestV2Complex",
|
||||
"folder": "test",
|
||||
"content": """
|
||||
# TestV2Complex
|
||||
|
||||
## Observations
|
||||
- [note] This is a test observation #tag1 (context)
|
||||
- related to [[OtherEntity]]
|
||||
""",
|
||||
}
|
||||
|
||||
response = await client.post(f"{v2_project_url}/knowledge/entities", json=data)
|
||||
|
||||
assert response.status_code == 200
|
||||
entity = EntityResponseV2.model_validate(response.json())
|
||||
|
||||
# V2 endpoints must return id field
|
||||
assert entity.id is not None
|
||||
assert isinstance(entity.id, int)
|
||||
assert entity.api_version == "v2"
|
||||
|
||||
assert len(entity.observations) == 1
|
||||
assert entity.observations[0].category == "note"
|
||||
assert entity.observations[0].content == "This is a test observation #tag1"
|
||||
assert entity.observations[0].tags == ["tag1"]
|
||||
|
||||
assert len(entity.relations) == 1
|
||||
assert entity.relations[0].relation_type == "related to"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_entity_by_id(
|
||||
client: AsyncClient, file_service, v2_project_url, entity_repository
|
||||
):
|
||||
"""Test updating an entity by ID using PUT (replace)."""
|
||||
# Create an entity first
|
||||
create_data = {
|
||||
"title": "TestUpdate",
|
||||
"folder": "test",
|
||||
"content": "Original content",
|
||||
}
|
||||
response = await client.post(f"{v2_project_url}/knowledge/entities", json=create_data)
|
||||
assert response.status_code == 200
|
||||
created_entity = EntityResponseV2.model_validate(response.json())
|
||||
|
||||
# V2 create must return id
|
||||
assert created_entity.id is not None
|
||||
original_id = created_entity.id
|
||||
|
||||
# Update it by ID
|
||||
update_data = {
|
||||
"title": "TestUpdate",
|
||||
"folder": "test",
|
||||
"content": "Updated content via V2",
|
||||
}
|
||||
response = await client.put(
|
||||
f"{v2_project_url}/knowledge/entities/{original_id}",
|
||||
json=update_data,
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
updated_entity = EntityResponseV2.model_validate(response.json())
|
||||
|
||||
# V2 update must return id field
|
||||
assert updated_entity.id is not None
|
||||
assert isinstance(updated_entity.id, int)
|
||||
assert updated_entity.api_version == "v2"
|
||||
|
||||
# Verify file was updated
|
||||
file_path = file_service.get_entity_path(updated_entity)
|
||||
file_content, _ = await file_service.read_file(file_path)
|
||||
assert "Updated content via V2" in file_content
|
||||
assert "Original content" not in file_content
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_edit_entity_by_id_append(
|
||||
client: AsyncClient, file_service, v2_project_url, entity_repository
|
||||
):
|
||||
"""Test editing an entity by ID using PATCH (append operation)."""
|
||||
# Create an entity first
|
||||
create_data = {
|
||||
"title": "TestEdit",
|
||||
"folder": "test",
|
||||
"content": "# TestEdit\n\nOriginal content",
|
||||
}
|
||||
response = await client.post(f"{v2_project_url}/knowledge/entities", json=create_data)
|
||||
assert response.status_code == 200
|
||||
created_entity = EntityResponseV2.model_validate(response.json())
|
||||
|
||||
# V2 create must return id
|
||||
assert created_entity.id is not None
|
||||
original_id = created_entity.id
|
||||
|
||||
# Edit it by appending
|
||||
edit_data = {
|
||||
"operation": "append",
|
||||
"content": "\n\n## New Section\n\nAppended content",
|
||||
}
|
||||
response = await client.patch(
|
||||
f"{v2_project_url}/knowledge/entities/{original_id}",
|
||||
json=edit_data,
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
edited_entity = EntityResponseV2.model_validate(response.json())
|
||||
|
||||
# V2 patch must return id field
|
||||
assert edited_entity.id is not None
|
||||
assert isinstance(edited_entity.id, int)
|
||||
assert edited_entity.api_version == "v2"
|
||||
|
||||
# Verify file has both original and appended content
|
||||
file_path = file_service.get_entity_path(edited_entity)
|
||||
file_content, _ = await file_service.read_file(file_path)
|
||||
assert "Original content" in file_content
|
||||
assert "Appended content" in file_content
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_edit_entity_by_id_find_replace(
|
||||
client: AsyncClient, file_service, v2_project_url, entity_repository
|
||||
):
|
||||
"""Test editing an entity by ID using PATCH (find/replace operation)."""
|
||||
# Create an entity first
|
||||
create_data = {
|
||||
"title": "TestFindReplace",
|
||||
"folder": "test",
|
||||
"content": "# TestFindReplace\n\nOld text that will be replaced",
|
||||
}
|
||||
response = await client.post(f"{v2_project_url}/knowledge/entities", json=create_data)
|
||||
assert response.status_code == 200
|
||||
created_entity = EntityResponseV2.model_validate(response.json())
|
||||
|
||||
# V2 create must return id
|
||||
assert created_entity.id is not None
|
||||
original_id = created_entity.id
|
||||
|
||||
# Edit using find/replace
|
||||
edit_data = {
|
||||
"operation": "find_replace",
|
||||
"find_text": "Old text",
|
||||
"content": "New text",
|
||||
}
|
||||
response = await client.patch(
|
||||
f"{v2_project_url}/knowledge/entities/{original_id}",
|
||||
json=edit_data,
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
edited_entity = EntityResponseV2.model_validate(response.json())
|
||||
|
||||
# V2 patch must return id field
|
||||
assert edited_entity.id is not None
|
||||
assert isinstance(edited_entity.id, int)
|
||||
assert edited_entity.api_version == "v2"
|
||||
|
||||
# Verify replacement
|
||||
file_path = file_service.get_entity_path(created_entity)
|
||||
file_content, _ = await file_service.read_file(file_path)
|
||||
assert "New text" in file_content
|
||||
assert "Old text" not in file_content
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_delete_entity_by_id(
|
||||
client: AsyncClient, file_service, v2_project_url, entity_repository
|
||||
):
|
||||
"""Test deleting an entity by ID."""
|
||||
# Create an entity first
|
||||
create_data = {
|
||||
"title": "TestDelete",
|
||||
"folder": "test",
|
||||
"content": "Content to be deleted",
|
||||
}
|
||||
response = await client.post(f"{v2_project_url}/knowledge/entities", json=create_data)
|
||||
assert response.status_code == 200
|
||||
created_entity = EntityResponseV2.model_validate(response.json())
|
||||
|
||||
# V2 create must return id
|
||||
assert created_entity.id is not None
|
||||
entity_id = created_entity.id
|
||||
|
||||
# Delete it by ID
|
||||
response = await client.delete(f"{v2_project_url}/knowledge/entities/{entity_id}")
|
||||
|
||||
assert response.status_code == 200
|
||||
delete_response = DeleteEntitiesResponse.model_validate(response.json())
|
||||
assert delete_response.deleted is True
|
||||
|
||||
# Verify it's gone - trying to get it should return 404
|
||||
response = await client.get(f"{v2_project_url}/knowledge/entities/{entity_id}")
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_delete_entity_by_id_not_found(client: AsyncClient, v2_project_url):
|
||||
"""Test deleting a non-existent entity returns deleted=False (idempotent)."""
|
||||
response = await client.delete(f"{v2_project_url}/knowledge/entities/999999")
|
||||
|
||||
# Delete is idempotent - returns 200 with deleted=False
|
||||
assert response.status_code == 200
|
||||
delete_response = DeleteEntitiesResponse.model_validate(response.json())
|
||||
assert delete_response.deleted is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_move_entity(client: AsyncClient, file_service, v2_project_url, entity_repository):
|
||||
"""Test moving an entity to a new location."""
|
||||
# Create an entity first
|
||||
create_data = {
|
||||
"title": "TestMove",
|
||||
"folder": "test",
|
||||
"content": "Content to be moved",
|
||||
}
|
||||
response = await client.post(f"{v2_project_url}/knowledge/entities", json=create_data)
|
||||
assert response.status_code == 200
|
||||
created_entity = EntityResponseV2.model_validate(response.json())
|
||||
|
||||
# V2 create must return id
|
||||
assert created_entity.id is not None
|
||||
original_id = created_entity.id
|
||||
|
||||
# Move it to a new folder (V2 uses entity ID in path)
|
||||
move_data = {
|
||||
"destination_path": "moved/MovedEntity.md",
|
||||
}
|
||||
response = await client.put(
|
||||
f"{v2_project_url}/knowledge/entities/{created_entity.id}/move", json=move_data
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
moved_entity = EntityResponseV2.model_validate(response.json())
|
||||
|
||||
# V2 move must return id field
|
||||
assert moved_entity.id is not None
|
||||
assert isinstance(moved_entity.id, int)
|
||||
assert moved_entity.api_version == "v2"
|
||||
|
||||
# ID should remain the same (stable reference)
|
||||
assert moved_entity.id == original_id
|
||||
assert moved_entity.file_path == "moved/MovedEntity.md"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_v2_endpoints_use_project_id_not_name(client: AsyncClient, test_project: Project):
|
||||
"""Verify v2 endpoints require project ID, not name."""
|
||||
# Try using project name instead of ID - should fail
|
||||
response = await client.get(f"/v2/{test_project.name}/knowledge/entities/1")
|
||||
|
||||
# Should get validation error or 404 because name is not a valid integer
|
||||
assert response.status_code in [404, 422]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_entity_response_v2_has_api_version(
|
||||
client: AsyncClient, v2_project_url, entity_repository
|
||||
):
|
||||
"""Test that EntityResponseV2 includes api_version field."""
|
||||
# Create an entity
|
||||
entity_data = {
|
||||
"title": "TestApiVersion",
|
||||
"folder": "test",
|
||||
"content": "Test content",
|
||||
}
|
||||
response = await client.post(f"{v2_project_url}/knowledge/entities", json=entity_data)
|
||||
assert response.status_code == 200
|
||||
created_entity = EntityResponseV2.model_validate(response.json())
|
||||
|
||||
# V2 create must return id and api_version
|
||||
assert created_entity.id is not None
|
||||
assert created_entity.api_version == "v2"
|
||||
entity_id = created_entity.id
|
||||
|
||||
# Get it via v2 endpoint
|
||||
response = await client.get(f"{v2_project_url}/knowledge/entities/{entity_id}")
|
||||
assert response.status_code == 200
|
||||
|
||||
entity_v2 = EntityResponseV2.model_validate(response.json())
|
||||
assert entity_v2.api_version == "v2"
|
||||
assert entity_v2.id == entity_id
|
||||
@@ -0,0 +1,301 @@
|
||||
"""Tests for v2 memory router endpoints."""
|
||||
|
||||
import pytest
|
||||
from httpx import AsyncClient
|
||||
from pathlib import Path
|
||||
|
||||
from basic_memory.models import Project
|
||||
|
||||
|
||||
async def create_test_entity(
|
||||
test_project, entity_data, entity_repository, search_service, file_service
|
||||
):
|
||||
"""Helper to create an entity with file and index it."""
|
||||
# Create file
|
||||
test_content = f"# {entity_data['title']}\n\nTest content"
|
||||
file_path = Path(test_project.path) / entity_data["file_path"]
|
||||
file_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
await file_service.write_file(file_path, test_content)
|
||||
|
||||
# Create entity
|
||||
entity = await entity_repository.create(entity_data)
|
||||
|
||||
# Index for search
|
||||
await search_service.index_entity(entity)
|
||||
|
||||
return entity
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_recent_context(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
entity_repository,
|
||||
search_service,
|
||||
file_service,
|
||||
):
|
||||
"""Test getting recent activity context."""
|
||||
entity_data = {
|
||||
"title": "Recent Test Entity",
|
||||
"entity_type": "note",
|
||||
"content_type": "text/markdown",
|
||||
"file_path": "recent_test.md",
|
||||
"checksum": "abc123",
|
||||
}
|
||||
await create_test_entity(
|
||||
test_project, entity_data, entity_repository, search_service, file_service
|
||||
)
|
||||
|
||||
# Get recent context
|
||||
response = await client.get(f"{v2_project_url}/memory/recent")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
|
||||
# Verify response structure (GraphContext uses 'results' not 'entities')
|
||||
assert "results" in data
|
||||
assert "metadata" in data
|
||||
assert "page" in data
|
||||
assert "page_size" in data
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_recent_context_with_pagination(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
entity_repository,
|
||||
search_service,
|
||||
file_service,
|
||||
):
|
||||
"""Test recent context with pagination parameters."""
|
||||
# Create multiple test entities
|
||||
for i in range(5):
|
||||
entity_data = {
|
||||
"title": f"Entity {i}",
|
||||
"entity_type": "note",
|
||||
"content_type": "text/markdown",
|
||||
"file_path": f"entity_{i}.md",
|
||||
"checksum": f"checksum{i}",
|
||||
}
|
||||
await create_test_entity(
|
||||
test_project, entity_data, entity_repository, search_service, file_service
|
||||
)
|
||||
|
||||
# Get recent context with pagination
|
||||
response = await client.get(
|
||||
f"{v2_project_url}/memory/recent", params={"page": 1, "page_size": 3}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "results" in data
|
||||
assert data["page"] == 1
|
||||
assert data["page_size"] == 3
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_recent_context_with_type_filter(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
entity_repository,
|
||||
search_service,
|
||||
file_service,
|
||||
):
|
||||
"""Test filtering recent context by type."""
|
||||
# Create a test entity
|
||||
entity_data = {
|
||||
"title": "Filtered Entity",
|
||||
"entity_type": "note",
|
||||
"content_type": "text/markdown",
|
||||
"file_path": "filtered.md",
|
||||
"checksum": "xyz789",
|
||||
}
|
||||
await create_test_entity(
|
||||
test_project, entity_data, entity_repository, search_service, file_service
|
||||
)
|
||||
|
||||
# Get recent context filtered by type
|
||||
response = await client.get(f"{v2_project_url}/memory/recent", params={"type": ["entity"]})
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "results" in data
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_recent_context_with_timeframe(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
):
|
||||
"""Test recent context with custom timeframe."""
|
||||
response = await client.get(f"{v2_project_url}/memory/recent", params={"timeframe": "1d"})
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "results" in data
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_recent_context_invalid_project_id(
|
||||
client: AsyncClient,
|
||||
):
|
||||
"""Test getting recent context with invalid project ID returns 404."""
|
||||
response = await client.get("/v2/projects/999999/memory/recent")
|
||||
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_memory_context_by_permalink(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
entity_repository,
|
||||
search_service,
|
||||
file_service,
|
||||
):
|
||||
"""Test getting context for a specific memory URI (permalink)."""
|
||||
# Create a test entity
|
||||
entity_data = {
|
||||
"title": "Context Test",
|
||||
"entity_type": "note",
|
||||
"content_type": "text/markdown",
|
||||
"file_path": "context_test.md",
|
||||
"checksum": "def456",
|
||||
"permalink": "context-test",
|
||||
}
|
||||
await create_test_entity(
|
||||
test_project, entity_data, entity_repository, search_service, file_service
|
||||
)
|
||||
|
||||
# Get context for this entity
|
||||
response = await client.get(f"{v2_project_url}/memory/context-test")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "results" in data
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_memory_context_by_id(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
entity_repository,
|
||||
search_service,
|
||||
file_service,
|
||||
):
|
||||
"""Test getting context using ID-based memory URI."""
|
||||
# Create a test entity
|
||||
entity_data = {
|
||||
"title": "ID Context Test",
|
||||
"entity_type": "note",
|
||||
"content_type": "text/markdown",
|
||||
"file_path": "id_context_test.md",
|
||||
"checksum": "ghi789",
|
||||
}
|
||||
created_entity = await create_test_entity(
|
||||
test_project, entity_data, entity_repository, search_service, file_service
|
||||
)
|
||||
|
||||
# Get context using ID format (memory://id/123 or memory://123)
|
||||
response = await client.get(f"{v2_project_url}/memory/id/{created_entity.id}")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "results" in data
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_memory_context_with_depth(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
entity_repository,
|
||||
search_service,
|
||||
file_service,
|
||||
):
|
||||
"""Test getting context with depth parameter."""
|
||||
# Create a test entity
|
||||
entity_data = {
|
||||
"title": "Depth Test",
|
||||
"entity_type": "note",
|
||||
"content_type": "text/markdown",
|
||||
"file_path": "depth_test.md",
|
||||
"checksum": "jkl012",
|
||||
"permalink": "depth-test",
|
||||
}
|
||||
await create_test_entity(
|
||||
test_project, entity_data, entity_repository, search_service, file_service
|
||||
)
|
||||
|
||||
# Get context with depth
|
||||
response = await client.get(f"{v2_project_url}/memory/depth-test", params={"depth": 2})
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "results" in data
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_memory_context_not_found(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
):
|
||||
"""Test getting context for non-existent memory URI returns 404."""
|
||||
response = await client.get(f"{v2_project_url}/memory/nonexistent-uri")
|
||||
|
||||
# Note: This might return 200 with empty results depending on implementation
|
||||
# Adjust assertion based on actual behavior
|
||||
assert response.status_code in [200, 404]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_memory_context_with_timeframe(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
entity_repository,
|
||||
search_service,
|
||||
file_service,
|
||||
):
|
||||
"""Test getting context with timeframe filter."""
|
||||
# Create a test entity
|
||||
entity_data = {
|
||||
"title": "Timeframe Test",
|
||||
"entity_type": "note",
|
||||
"content_type": "text/markdown",
|
||||
"file_path": "timeframe_test.md",
|
||||
"checksum": "mno345",
|
||||
"permalink": "timeframe-test",
|
||||
}
|
||||
await create_test_entity(
|
||||
test_project, entity_data, entity_repository, search_service, file_service
|
||||
)
|
||||
|
||||
# Get context with timeframe
|
||||
response = await client.get(
|
||||
f"{v2_project_url}/memory/timeframe-test", params={"timeframe": "7d"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "results" in data
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_v2_memory_endpoints_use_project_id_not_name(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
):
|
||||
"""Test that v2 memory endpoints reject string project names."""
|
||||
# Try to use project name instead of ID - should fail
|
||||
response = await client.get(f"/v2/{test_project.name}/memory/recent")
|
||||
|
||||
# FastAPI path validation should reject non-integer project_id
|
||||
assert response.status_code in [404, 422]
|
||||
@@ -0,0 +1,251 @@
|
||||
"""Tests for V2 project management API routes (ID-based endpoints)."""
|
||||
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from httpx import AsyncClient
|
||||
|
||||
from basic_memory.models import Project
|
||||
from basic_memory.schemas.project_info import ProjectItem, ProjectStatusResponse
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_project_by_id(client: AsyncClient, test_project: Project, v2_projects_url):
|
||||
"""Test getting a project by its numeric ID."""
|
||||
response = await client.get(f"{v2_projects_url}/{test_project.id}")
|
||||
|
||||
assert response.status_code == 200
|
||||
project = ProjectItem.model_validate(response.json())
|
||||
assert project.id == test_project.id
|
||||
assert project.name == test_project.name
|
||||
assert project.path == test_project.path
|
||||
assert project.is_default == (test_project.is_default or False)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_project_by_id_not_found(client: AsyncClient, v2_projects_url):
|
||||
"""Test getting a non-existent project by ID returns 404."""
|
||||
response = await client.get(f"{v2_projects_url}/999999")
|
||||
|
||||
assert response.status_code == 404
|
||||
assert "not found" in response.json()["detail"].lower()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_project_path_by_id(
|
||||
client: AsyncClient, test_project: Project, v2_projects_url
|
||||
):
|
||||
"""Test updating a project's path by ID."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
new_path = str(Path(tmpdir) / "new-project-location")
|
||||
Path(new_path).mkdir(parents=True, exist_ok=True)
|
||||
|
||||
update_data = {"path": new_path}
|
||||
response = await client.patch(
|
||||
f"{v2_projects_url}/{test_project.id}",
|
||||
json=update_data,
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
status_response = ProjectStatusResponse.model_validate(response.json())
|
||||
assert status_response.status == "success"
|
||||
assert status_response.new_project.id == test_project.id
|
||||
# Normalize paths for cross-platform comparison (Windows uses backslashes, API returns forward slashes)
|
||||
assert Path(status_response.new_project.path) == Path(new_path)
|
||||
assert status_response.old_project.id == test_project.id
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_project_invalid_path(
|
||||
client: AsyncClient, test_project: Project, v2_projects_url
|
||||
):
|
||||
"""Test updating with a relative path returns 400."""
|
||||
update_data = {"path": "relative/path"}
|
||||
response = await client.patch(
|
||||
f"{v2_projects_url}/{test_project.id}",
|
||||
json=update_data,
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
assert "absolute" in response.json()["detail"].lower()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_project_not_found(client: AsyncClient, v2_projects_url):
|
||||
"""Test updating a non-existent project returns 404."""
|
||||
update_data = {"path": "/tmp/new-path"}
|
||||
response = await client.patch(
|
||||
f"{v2_projects_url}/999999",
|
||||
json=update_data,
|
||||
)
|
||||
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_set_default_project_by_id(
|
||||
client: AsyncClient, test_project: Project, v2_projects_url, project_repository, project_service
|
||||
):
|
||||
"""Test setting a project as default by ID."""
|
||||
# Create a second project to test setting default
|
||||
await project_service.add_project("second-project", "/tmp/second-project")
|
||||
|
||||
# Get the created project from the repository to get its ID
|
||||
created_project = await project_repository.get_by_name("second-project")
|
||||
assert created_project is not None
|
||||
|
||||
# Set the second project as default
|
||||
response = await client.put(f"{v2_projects_url}/{created_project.id}/default")
|
||||
|
||||
assert response.status_code == 200
|
||||
status_response = ProjectStatusResponse.model_validate(response.json())
|
||||
assert status_response.status == "success"
|
||||
assert status_response.default is True
|
||||
assert status_response.new_project.id == created_project.id
|
||||
assert status_response.new_project.is_default is True
|
||||
assert status_response.old_project.id == test_project.id
|
||||
assert status_response.old_project.is_default is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_set_default_project_not_found(client: AsyncClient, v2_projects_url):
|
||||
"""Test setting a non-existent project as default returns 404."""
|
||||
response = await client.put(f"{v2_projects_url}/999999/default")
|
||||
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_delete_project_by_id(
|
||||
client: AsyncClient, test_project: Project, v2_projects_url, project_repository, project_service
|
||||
):
|
||||
"""Test deleting a project by ID."""
|
||||
# Create a second project since we can't delete the default
|
||||
await project_service.add_project("to-delete", "/tmp/to-delete")
|
||||
|
||||
# Get the created project from the repository to get its ID
|
||||
created_project = await project_repository.get_by_name("to-delete")
|
||||
assert created_project is not None
|
||||
|
||||
# Delete it
|
||||
response = await client.delete(f"{v2_projects_url}/{created_project.id}")
|
||||
|
||||
assert response.status_code == 200
|
||||
status_response = ProjectStatusResponse.model_validate(response.json())
|
||||
assert status_response.status == "success"
|
||||
assert status_response.old_project.id == created_project.id
|
||||
assert status_response.new_project is None
|
||||
|
||||
# Verify it's deleted - trying to get it should return 404
|
||||
response = await client.get(f"{v2_projects_url}/{created_project.id}")
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_delete_project_with_delete_notes_param(
|
||||
client: AsyncClient, test_project: Project, v2_projects_url, project_repository, project_service
|
||||
):
|
||||
"""Test deleting a project with delete_notes parameter."""
|
||||
# Create a project in a temp directory
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
project_path = Path(tmpdir) / "test-delete-notes"
|
||||
project_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Create a test file in the project
|
||||
test_file = project_path / "test.md"
|
||||
test_file.write_text("Test content")
|
||||
|
||||
await project_service.add_project("delete-with-notes", str(project_path))
|
||||
|
||||
# Get the created project from the repository to get its ID
|
||||
created_project = await project_repository.get_by_name("delete-with-notes")
|
||||
assert created_project is not None
|
||||
|
||||
# Delete with delete_notes=true
|
||||
response = await client.delete(f"{v2_projects_url}/{created_project.id}?delete_notes=true")
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
# Verify directory was deleted
|
||||
assert not project_path.exists()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_delete_default_project_fails(
|
||||
client: AsyncClient, test_project: Project, v2_projects_url
|
||||
):
|
||||
"""Test that deleting the default project returns 400."""
|
||||
# test_project is the default project
|
||||
response = await client.delete(f"{v2_projects_url}/{test_project.id}")
|
||||
|
||||
assert response.status_code == 400
|
||||
assert "default project" in response.json()["detail"].lower()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_delete_project_not_found(client: AsyncClient, v2_projects_url):
|
||||
"""Test deleting a non-existent project returns 404."""
|
||||
response = await client.delete(f"{v2_projects_url}/999999")
|
||||
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_v2_project_endpoints_use_id_not_name(
|
||||
client: AsyncClient, test_project: Project, v2_projects_url
|
||||
):
|
||||
"""Verify v2 project endpoints require project ID, not name."""
|
||||
# Try using project name instead of ID - should fail
|
||||
response = await client.get(f"{v2_projects_url}/{test_project.name}")
|
||||
|
||||
# Should get 404 or 422 because name is not a valid integer
|
||||
assert response.status_code in [404, 422]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_project_id_stability_after_rename(
|
||||
client: AsyncClient, test_project: Project, v2_projects_url, project_repository
|
||||
):
|
||||
"""Test that project ID remains stable even after renaming."""
|
||||
original_id = test_project.id
|
||||
original_name = test_project.name
|
||||
|
||||
# Get project by ID
|
||||
response = await client.get(f"{v2_projects_url}/{original_id}")
|
||||
assert response.status_code == 200
|
||||
project_before = ProjectItem.model_validate(response.json())
|
||||
assert project_before.id == original_id
|
||||
assert project_before.name == original_name
|
||||
|
||||
# Even if we renamed the project (not testing rename here, just the concept),
|
||||
# the ID would stay the same. This test demonstrates the stability.
|
||||
# Re-fetch by same ID
|
||||
response = await client.get(f"{v2_projects_url}/{original_id}")
|
||||
assert response.status_code == 200
|
||||
project_after = ProjectItem.model_validate(response.json())
|
||||
assert project_after.id == original_id
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_project_active_status(
|
||||
client: AsyncClient, test_project: Project, v2_projects_url, project_repository, project_service
|
||||
):
|
||||
"""Test updating a project's active status by ID."""
|
||||
# Create a non-default project
|
||||
await project_service.add_project("test-active", "/tmp/test-active")
|
||||
|
||||
# Get the created project from the repository to get its ID
|
||||
created_project = await project_repository.get_by_name("test-active")
|
||||
assert created_project is not None
|
||||
|
||||
# Update active status
|
||||
update_data = {"is_active": False}
|
||||
response = await client.patch(
|
||||
f"{v2_projects_url}/{created_project.id}",
|
||||
json=update_data,
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
status_response = ProjectStatusResponse.model_validate(response.json())
|
||||
assert status_response.status == "success"
|
||||
@@ -0,0 +1,212 @@
|
||||
"""Tests for V2 prompt router endpoints (ID-based)."""
|
||||
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
from httpx import AsyncClient
|
||||
|
||||
from basic_memory.models import Project
|
||||
from basic_memory.services.context_service import ContextService
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def context_service(entity_repository, search_service, observation_repository):
|
||||
"""Create a real context service for testing."""
|
||||
return ContextService(entity_repository, search_service, observation_repository)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_continue_conversation_endpoint(
|
||||
client: AsyncClient,
|
||||
entity_service,
|
||||
search_service,
|
||||
context_service,
|
||||
entity_repository,
|
||||
test_graph,
|
||||
v2_project_url: str,
|
||||
):
|
||||
"""Test the v2 continue_conversation endpoint with real services."""
|
||||
# Create request data
|
||||
request_data = {
|
||||
"topic": "Root", # This should match our test entity in test_graph
|
||||
"timeframe": "7d",
|
||||
"depth": 1,
|
||||
"related_items_limit": 2,
|
||||
}
|
||||
|
||||
# Call the endpoint
|
||||
response = await client.post(
|
||||
f"{v2_project_url}/prompt/continue-conversation", json=request_data
|
||||
)
|
||||
|
||||
# Verify response
|
||||
assert response.status_code == 200
|
||||
result = response.json()
|
||||
assert "prompt" in result
|
||||
assert "context" in result
|
||||
|
||||
# Check content of context
|
||||
context = result["context"]
|
||||
assert context["topic"] == "Root"
|
||||
assert context["timeframe"] == "7d"
|
||||
assert context["has_results"] is True
|
||||
assert len(context["hierarchical_results"]) > 0
|
||||
|
||||
# Check content of prompt
|
||||
prompt = result["prompt"]
|
||||
assert "Continuing conversation on: Root" in prompt
|
||||
assert "memory retrieval session" in prompt
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_continue_conversation_without_topic(
|
||||
client: AsyncClient,
|
||||
entity_service,
|
||||
search_service,
|
||||
context_service,
|
||||
entity_repository,
|
||||
test_graph,
|
||||
v2_project_url: str,
|
||||
):
|
||||
"""Test v2 continue_conversation without topic - should use recent activity."""
|
||||
request_data = {"timeframe": "1d", "depth": 1, "related_items_limit": 2}
|
||||
|
||||
response = await client.post(
|
||||
f"{v2_project_url}/prompt/continue-conversation", json=request_data
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
result = response.json()
|
||||
assert "Recent Activity" in result["context"]["topic"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_prompt_endpoint(
|
||||
client: AsyncClient, entity_service, search_service, test_graph, v2_project_url: str
|
||||
):
|
||||
"""Test the v2 search_prompt endpoint with real services."""
|
||||
# Create request data
|
||||
request_data = {
|
||||
"query": "Root", # This should match our test entity
|
||||
"timeframe": "7d",
|
||||
}
|
||||
|
||||
# Call the endpoint
|
||||
response = await client.post(f"{v2_project_url}/prompt/search", json=request_data)
|
||||
|
||||
# Verify response
|
||||
assert response.status_code == 200
|
||||
result = response.json()
|
||||
assert "prompt" in result
|
||||
assert "context" in result
|
||||
|
||||
# Check content of context
|
||||
context = result["context"]
|
||||
assert context["query"] == "Root"
|
||||
assert context["timeframe"] == "7d"
|
||||
assert context["has_results"] is True
|
||||
assert len(context["results"]) > 0
|
||||
|
||||
# Check content of prompt
|
||||
prompt = result["prompt"]
|
||||
assert 'Search Results for: "Root"' in prompt
|
||||
assert "This is a memory search session" in prompt
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_prompt_no_results(
|
||||
client: AsyncClient, entity_service, search_service, v2_project_url: str
|
||||
):
|
||||
"""Test the v2 search_prompt endpoint with a query that returns no results."""
|
||||
# Create request data with a query that shouldn't match anything
|
||||
request_data = {"query": "NonExistentQuery12345", "timeframe": "7d"}
|
||||
|
||||
# Call the endpoint
|
||||
response = await client.post(f"{v2_project_url}/prompt/search", json=request_data)
|
||||
|
||||
# Verify response
|
||||
assert response.status_code == 200
|
||||
result = response.json()
|
||||
|
||||
# Check content of context
|
||||
context = result["context"]
|
||||
assert context["query"] == "NonExistentQuery12345"
|
||||
assert context["has_results"] is False
|
||||
assert len(context["results"]) == 0
|
||||
|
||||
# Check content of prompt
|
||||
prompt = result["prompt"]
|
||||
assert 'Search Results for: "NonExistentQuery12345"' in prompt
|
||||
assert "I couldn't find any results for this query" in prompt
|
||||
assert "Opportunity to Capture Knowledge" in prompt
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_error_handling(client: AsyncClient, monkeypatch, v2_project_url: str):
|
||||
"""Test error handling in v2 endpoints by breaking the template loader."""
|
||||
|
||||
# Patch the template loader to raise an exception
|
||||
def mock_render(*args, **kwargs):
|
||||
raise Exception("Template error")
|
||||
|
||||
# Apply the patch
|
||||
monkeypatch.setattr("basic_memory.api.template_loader.TemplateLoader.render", mock_render)
|
||||
|
||||
# Test continue_conversation error handling
|
||||
response = await client.post(
|
||||
f"{v2_project_url}/prompt/continue-conversation",
|
||||
json={"topic": "test error", "timeframe": "7d"},
|
||||
)
|
||||
|
||||
assert response.status_code == 500
|
||||
assert "detail" in response.json()
|
||||
assert "Template error" in response.json()["detail"]
|
||||
|
||||
# Test search_prompt error handling
|
||||
response = await client.post(
|
||||
f"{v2_project_url}/prompt/search", json={"query": "test error", "timeframe": "7d"}
|
||||
)
|
||||
|
||||
assert response.status_code == 500
|
||||
assert "detail" in response.json()
|
||||
assert "Template error" in response.json()["detail"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_v2_prompt_endpoints_use_project_id_not_name(
|
||||
client: AsyncClient, test_project: Project
|
||||
):
|
||||
"""Verify v2 prompt endpoints require project ID, not name."""
|
||||
# Try using project name instead of ID - should fail
|
||||
response = await client.post(
|
||||
f"/v2/projects/{test_project.name}/prompt/continue-conversation",
|
||||
json={"topic": "test", "timeframe": "7d"},
|
||||
)
|
||||
|
||||
# Should get validation error or 404 because name is not a valid integer
|
||||
assert response.status_code in [404, 422]
|
||||
|
||||
# Also test search endpoint
|
||||
response = await client.post(
|
||||
f"/v2/projects/{test_project.name}/prompt/search",
|
||||
json={"query": "test", "timeframe": "7d"},
|
||||
)
|
||||
|
||||
assert response.status_code in [404, 422]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_prompt_invalid_project_id(client: AsyncClient):
|
||||
"""Test prompt endpoints with invalid project ID return 404."""
|
||||
# Test continue-conversation
|
||||
response = await client.post(
|
||||
"/v2/projects/999999/prompt/continue-conversation",
|
||||
json={"topic": "test", "timeframe": "7d"},
|
||||
)
|
||||
assert response.status_code == 404
|
||||
|
||||
# Test search
|
||||
response = await client.post(
|
||||
"/v2/projects/999999/prompt/search",
|
||||
json={"query": "test", "timeframe": "7d"},
|
||||
)
|
||||
assert response.status_code == 404
|
||||
@@ -0,0 +1,267 @@
|
||||
"""Tests for V2 resource API routes (ID-based endpoints)."""
|
||||
|
||||
import pytest
|
||||
from httpx import AsyncClient
|
||||
|
||||
from basic_memory.models import Project
|
||||
from basic_memory.schemas.v2.resource import ResourceResponse
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_create_resource(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
):
|
||||
"""Test creating a new resource via v2 POST endpoint."""
|
||||
create_data = {
|
||||
"file_path": "test-resources/test-file.md",
|
||||
"content": "# Test Resource\n\nThis is test content.",
|
||||
}
|
||||
|
||||
response = await client.post(
|
||||
f"{v2_project_url}/resource",
|
||||
json=create_data,
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
result = ResourceResponse.model_validate(response.json())
|
||||
|
||||
# V2 must return entity_id
|
||||
assert result.entity_id is not None
|
||||
assert isinstance(result.entity_id, int)
|
||||
assert result.file_path == "test-resources/test-file.md"
|
||||
assert result.checksum is not None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_create_resource_duplicate_fails(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
):
|
||||
"""Test that creating a resource at an existing path returns 409."""
|
||||
create_data = {
|
||||
"file_path": "duplicate-test.md",
|
||||
"content": "First version",
|
||||
}
|
||||
|
||||
# Create first time - should succeed
|
||||
response = await client.post(f"{v2_project_url}/resource", json=create_data)
|
||||
assert response.status_code == 200
|
||||
|
||||
# Try to create again - should fail with 409
|
||||
response = await client.post(f"{v2_project_url}/resource", json=create_data)
|
||||
assert response.status_code == 409
|
||||
assert "already exists" in response.json()["detail"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_resource_by_id(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
):
|
||||
"""Test getting resource content by entity ID."""
|
||||
# First create a resource
|
||||
test_content = "# Test Resource\n\nThis is test content."
|
||||
create_data = {
|
||||
"file_path": "test-get.md",
|
||||
"content": test_content,
|
||||
}
|
||||
|
||||
create_response = await client.post(f"{v2_project_url}/resource", json=create_data)
|
||||
assert create_response.status_code == 200
|
||||
created = ResourceResponse.model_validate(create_response.json())
|
||||
|
||||
# Now get it by entity ID
|
||||
response = await client.get(f"{v2_project_url}/resource/{created.entity_id}")
|
||||
|
||||
assert response.status_code == 200
|
||||
# Normalize line endings for cross-platform compatibility
|
||||
assert test_content.replace("\n", "") in response.text.replace("\r\n", "").replace("\n", "")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_resource_not_found(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
):
|
||||
"""Test getting a non-existent resource returns 404."""
|
||||
response = await client.get(f"{v2_project_url}/resource/999999")
|
||||
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_resource(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
):
|
||||
"""Test updating resource content by entity ID."""
|
||||
# Create a resource
|
||||
create_data = {
|
||||
"file_path": "test-update.md",
|
||||
"content": "Original content",
|
||||
}
|
||||
create_response = await client.post(f"{v2_project_url}/resource", json=create_data)
|
||||
assert create_response.status_code == 200
|
||||
created = ResourceResponse.model_validate(create_response.json())
|
||||
|
||||
# Update it
|
||||
update_data = {
|
||||
"content": "Updated content",
|
||||
}
|
||||
response = await client.put(
|
||||
f"{v2_project_url}/resource/{created.entity_id}",
|
||||
json=update_data,
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
result = ResourceResponse.model_validate(response.json())
|
||||
assert result.entity_id == created.entity_id
|
||||
assert result.file_path == "test-update.md"
|
||||
|
||||
# Verify content was updated
|
||||
get_response = await client.get(f"{v2_project_url}/resource/{created.entity_id}")
|
||||
assert "Updated content" in get_response.text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_resource_and_move(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
):
|
||||
"""Test updating resource content and moving it to a new path."""
|
||||
# Create a resource
|
||||
create_data = {
|
||||
"file_path": "original-location.md",
|
||||
"content": "Original content",
|
||||
}
|
||||
create_response = await client.post(f"{v2_project_url}/resource", json=create_data)
|
||||
assert create_response.status_code == 200
|
||||
created = ResourceResponse.model_validate(create_response.json())
|
||||
|
||||
# Update content and move file
|
||||
update_data = {
|
||||
"content": "Updated content in new location",
|
||||
"file_path": "moved/new-location.md",
|
||||
}
|
||||
response = await client.put(
|
||||
f"{v2_project_url}/resource/{created.entity_id}",
|
||||
json=update_data,
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
result = ResourceResponse.model_validate(response.json())
|
||||
assert result.entity_id == created.entity_id
|
||||
assert result.file_path == "moved/new-location.md"
|
||||
|
||||
# Verify content at new location
|
||||
get_response = await client.get(f"{v2_project_url}/resource/{created.entity_id}")
|
||||
assert "Updated content in new location" in get_response.text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_resource_not_found(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
):
|
||||
"""Test updating a non-existent resource returns 404."""
|
||||
update_data = {
|
||||
"content": "New content",
|
||||
}
|
||||
response = await client.put(
|
||||
f"{v2_project_url}/resource/999999",
|
||||
json=update_data,
|
||||
)
|
||||
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_create_resource_invalid_path(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
):
|
||||
"""Test creating a resource with path traversal attempt fails."""
|
||||
create_data = {
|
||||
"file_path": "../../../etc/passwd",
|
||||
"content": "malicious content",
|
||||
}
|
||||
|
||||
response = await client.post(f"{v2_project_url}/resource", json=create_data)
|
||||
|
||||
assert response.status_code == 400
|
||||
assert "Invalid file path" in response.json()["detail"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_resource_invalid_path(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
):
|
||||
"""Test updating a resource with path traversal attempt fails."""
|
||||
# Create a valid resource first
|
||||
create_data = {
|
||||
"file_path": "valid.md",
|
||||
"content": "Valid content",
|
||||
}
|
||||
create_response = await client.post(f"{v2_project_url}/resource", json=create_data)
|
||||
assert create_response.status_code == 200
|
||||
created = ResourceResponse.model_validate(create_response.json())
|
||||
|
||||
# Try to move it to an invalid path
|
||||
update_data = {
|
||||
"content": "Updated content",
|
||||
"file_path": "../../../etc/passwd",
|
||||
}
|
||||
response = await client.put(
|
||||
f"{v2_project_url}/resource/{created.entity_id}",
|
||||
json=update_data,
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
assert "Invalid file path" in response.json()["detail"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_resource_invalid_project_id(
|
||||
client: AsyncClient,
|
||||
):
|
||||
"""Test resource endpoints with invalid project ID return 404."""
|
||||
# Test create
|
||||
response = await client.post(
|
||||
"/v2/projects/999999/resource",
|
||||
json={"file_path": "test.md", "content": "test"},
|
||||
)
|
||||
assert response.status_code == 404
|
||||
|
||||
# Test get
|
||||
response = await client.get("/v2/projects/999999/resource/1")
|
||||
assert response.status_code == 404
|
||||
|
||||
# Test update
|
||||
response = await client.put(
|
||||
"/v2/projects/999999/resource/1",
|
||||
json={"content": "test"},
|
||||
)
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_v2_resource_endpoints_use_project_id_not_name(
|
||||
client: AsyncClient, test_project: Project
|
||||
):
|
||||
"""Verify v2 resource endpoints require project ID, not name."""
|
||||
# Try using project name instead of ID - should fail
|
||||
response = await client.get(f"/v2/projects/{test_project.name}/resource/1")
|
||||
|
||||
# Should get validation error or 404 because name is not a valid integer
|
||||
assert response.status_code in [404, 422]
|
||||
@@ -0,0 +1,289 @@
|
||||
"""Tests for v2 search router endpoints."""
|
||||
|
||||
import pytest
|
||||
from httpx import AsyncClient
|
||||
from pathlib import Path
|
||||
|
||||
from basic_memory.models import Project
|
||||
|
||||
|
||||
async def create_test_entity(
|
||||
test_project, entity_data, entity_repository, search_service, file_service
|
||||
):
|
||||
"""Helper to create an entity with file and index it."""
|
||||
# Create file
|
||||
test_content = f"# {entity_data['title']}\n\nTest content"
|
||||
file_path = Path(test_project.path) / entity_data["file_path"]
|
||||
file_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
await file_service.write_file(file_path, test_content)
|
||||
|
||||
# Create entity
|
||||
entity = await entity_repository.create(entity_data)
|
||||
|
||||
# Index for search
|
||||
await search_service.index_entity(entity)
|
||||
|
||||
return entity
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_entities(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
entity_repository,
|
||||
search_service,
|
||||
file_service,
|
||||
):
|
||||
"""Test searching for entities."""
|
||||
# Create a test entity
|
||||
entity_data = {
|
||||
"title": "Searchable Entity",
|
||||
"entity_type": "note",
|
||||
"content_type": "text/markdown",
|
||||
"file_path": "searchable.md",
|
||||
"checksum": "search123",
|
||||
}
|
||||
await create_test_entity(
|
||||
test_project, entity_data, entity_repository, search_service, file_service
|
||||
)
|
||||
|
||||
# Search for the entity
|
||||
response = await client.post(f"{v2_project_url}/search/", json={"search_text": "Searchable"})
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
|
||||
# Verify response structure
|
||||
assert "results" in data
|
||||
assert "current_page" in data
|
||||
assert "page_size" in data
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_with_pagination(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
entity_repository,
|
||||
search_service,
|
||||
file_service,
|
||||
):
|
||||
"""Test search with pagination parameters."""
|
||||
# Create multiple test entities
|
||||
for i in range(5):
|
||||
entity_data = {
|
||||
"title": f"Search Entity {i}",
|
||||
"entity_type": "note",
|
||||
"content_type": "text/markdown",
|
||||
"file_path": f"search_{i}.md",
|
||||
"checksum": f"searchsum{i}",
|
||||
}
|
||||
await create_test_entity(
|
||||
test_project, entity_data, entity_repository, search_service, file_service
|
||||
)
|
||||
|
||||
# Search with pagination
|
||||
response = await client.post(
|
||||
f"{v2_project_url}/search/",
|
||||
json={"search_text": "Search Entity"},
|
||||
params={"page": 1, "page_size": 3},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["current_page"] == 1
|
||||
assert data["page_size"] == 3
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_by_permalink(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
entity_repository,
|
||||
search_service,
|
||||
file_service,
|
||||
):
|
||||
"""Test searching by permalink."""
|
||||
# Create a test entity with permalink
|
||||
entity_data = {
|
||||
"title": "Permalink Search",
|
||||
"entity_type": "note",
|
||||
"content_type": "text/markdown",
|
||||
"file_path": "permalink_search.md",
|
||||
"checksum": "perm123",
|
||||
"permalink": "permalink-search",
|
||||
}
|
||||
await create_test_entity(
|
||||
test_project, entity_data, entity_repository, search_service, file_service
|
||||
)
|
||||
|
||||
# Search by permalink
|
||||
response = await client.post(
|
||||
f"{v2_project_url}/search/", json={"permalink": "permalink-search"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "results" in data
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_by_title(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
entity_repository,
|
||||
search_service,
|
||||
file_service,
|
||||
):
|
||||
"""Test searching by title."""
|
||||
# Create a test entity
|
||||
entity_data = {
|
||||
"title": "Unique Title For Search",
|
||||
"entity_type": "note",
|
||||
"content_type": "text/markdown",
|
||||
"file_path": "unique_title.md",
|
||||
"checksum": "title123",
|
||||
}
|
||||
await create_test_entity(
|
||||
test_project, entity_data, entity_repository, search_service, file_service
|
||||
)
|
||||
|
||||
# Search by title
|
||||
response = await client.post(f"{v2_project_url}/search/", json={"title": "Unique Title"})
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "results" in data
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_with_type_filter(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
entity_repository,
|
||||
search_service,
|
||||
file_service,
|
||||
):
|
||||
"""Test searching with entity type filter."""
|
||||
# Create test entities of different types
|
||||
for entity_type in ["note", "document"]:
|
||||
entity_data = {
|
||||
"title": f"Type {entity_type}",
|
||||
"entity_type": entity_type,
|
||||
"content_type": "text/markdown",
|
||||
"file_path": f"type_{entity_type}.md",
|
||||
"checksum": f"type{entity_type}",
|
||||
}
|
||||
await create_test_entity(
|
||||
test_project, entity_data, entity_repository, search_service, file_service
|
||||
)
|
||||
|
||||
# Search with type filter
|
||||
response = await client.post(
|
||||
f"{v2_project_url}/search/", json={"search_text": "Type", "types": ["note"]}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "results" in data
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_with_date_filter(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
entity_repository,
|
||||
search_service,
|
||||
file_service,
|
||||
):
|
||||
"""Test searching with date filter."""
|
||||
# Create a test entity
|
||||
entity_data = {
|
||||
"title": "Date Filtered",
|
||||
"entity_type": "note",
|
||||
"content_type": "text/markdown",
|
||||
"file_path": "date_filtered.md",
|
||||
"checksum": "date123",
|
||||
}
|
||||
await create_test_entity(
|
||||
test_project, entity_data, entity_repository, search_service, file_service
|
||||
)
|
||||
|
||||
# Search with date filter
|
||||
response = await client.post(
|
||||
f"{v2_project_url}/search/",
|
||||
json={"search_text": "Date Filtered", "after_date": "2024-01-01T00:00:00Z"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "results" in data
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_empty_query(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
):
|
||||
"""Test search with empty query."""
|
||||
response = await client.post(f"{v2_project_url}/search/", json={})
|
||||
|
||||
# Empty query should still be valid (returns all)
|
||||
assert response.status_code in [200, 422]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_invalid_project_id(
|
||||
client: AsyncClient,
|
||||
):
|
||||
"""Test searching with invalid project ID returns 404."""
|
||||
response = await client.post("/v2/projects/999999/search/", json={"search_text": "test"})
|
||||
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reindex(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
v2_project_url: str,
|
||||
):
|
||||
"""Test reindexing search index."""
|
||||
response = await client.post(f"{v2_project_url}/search/reindex")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
|
||||
# Verify response structure
|
||||
assert "status" in data
|
||||
assert data["status"] == "ok"
|
||||
assert "message" in data
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reindex_invalid_project_id(
|
||||
client: AsyncClient,
|
||||
):
|
||||
"""Test reindexing with invalid project ID returns 404."""
|
||||
response = await client.post("/v2/projects/999999/search/reindex")
|
||||
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_v2_search_endpoints_use_project_id_not_name(
|
||||
client: AsyncClient,
|
||||
test_project: Project,
|
||||
):
|
||||
"""Test that v2 search endpoints reject string project names."""
|
||||
# Try to use project name instead of ID - should fail
|
||||
response = await client.post(f"/v2/{test_project.name}/search/", json={"search_text": "test"})
|
||||
|
||||
# FastAPI path validation should reject non-integer project_id
|
||||
assert response.status_code in [404, 422]
|
||||
+28
-1
@@ -1,5 +1,8 @@
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import AsyncGenerator
|
||||
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
from fastapi import FastAPI
|
||||
from httpx import AsyncClient, ASGITransport
|
||||
@@ -8,7 +11,31 @@ from basic_memory.api.app import app as fastapi_app
|
||||
from basic_memory.deps import get_project_config, get_engine_factory, get_app_config
|
||||
|
||||
|
||||
@pytest_asyncio.fixture(autouse=True)
|
||||
@pytest.fixture(autouse=True)
|
||||
def isolated_home(tmp_path, monkeypatch) -> Path:
|
||||
"""Isolate tests from user's HOME directory.
|
||||
|
||||
This prevents tests from reading/writing to ~/.basic-memory/.bmignore
|
||||
or other user-specific configuration.
|
||||
|
||||
Sets BASIC_MEMORY_HOME to tmp_path directly so the default project
|
||||
writes files to tmp_path, which is where tests expect to find them.
|
||||
"""
|
||||
# Clear config cache to ensure fresh config for each test
|
||||
from basic_memory import config as config_module
|
||||
|
||||
config_module._CONFIG_CACHE = None
|
||||
|
||||
monkeypatch.setenv("HOME", str(tmp_path))
|
||||
if os.name == "nt":
|
||||
monkeypatch.setenv("USERPROFILE", str(tmp_path))
|
||||
# Set to tmp_path directly (not tmp_path/basic-memory) so default project
|
||||
# home is tmp_path - tests expect to find imported files there
|
||||
monkeypatch.setenv("BASIC_MEMORY_HOME", str(tmp_path))
|
||||
return tmp_path
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def app(app_config, project_config, engine_factory, test_config, aiolib) -> FastAPI:
|
||||
"""Create test FastAPI application."""
|
||||
app = fastapi_app
|
||||
|
||||
@@ -55,6 +55,7 @@ def mock_api_client():
|
||||
"default": False,
|
||||
"old_project": None,
|
||||
"new_project": {
|
||||
"id": 1,
|
||||
"name": "test-project",
|
||||
"path": "/test-project",
|
||||
"is_default": False,
|
||||
|
||||
+113
-93
@@ -1,16 +1,23 @@
|
||||
"""Common test fixtures."""
|
||||
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from textwrap import dedent
|
||||
from typing import AsyncGenerator, Literal
|
||||
from typing import AsyncGenerator
|
||||
|
||||
import os
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.ext.asyncio import AsyncEngine, AsyncSession, async_sessionmaker
|
||||
from sqlalchemy.ext.asyncio import (
|
||||
AsyncEngine,
|
||||
AsyncSession,
|
||||
async_sessionmaker,
|
||||
create_async_engine,
|
||||
)
|
||||
from sqlalchemy.pool import NullPool
|
||||
from testcontainers.postgres import PostgresContainer
|
||||
|
||||
from basic_memory import db
|
||||
from basic_memory.config import ProjectConfig, BasicMemoryConfig, ConfigManager, DatabaseBackend
|
||||
@@ -37,32 +44,47 @@ from basic_memory.sync.sync_service import SyncService
|
||||
from basic_memory.sync.watch_service import WatchService
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Database Backend Selection (env var approach)
|
||||
# =============================================================================
|
||||
# By default, tests run against SQLite.
|
||||
# Set BASIC_MEMORY_TEST_POSTGRES=1 to run against Postgres (uses testcontainers).
|
||||
# This allows running sqlite/postgres tests in parallel in CI.
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def db_backend():
|
||||
"""Determine database backend from environment variable.
|
||||
|
||||
Default: sqlite
|
||||
Set BASIC_MEMORY_TEST_POSTGRES=1 to use postgres
|
||||
"""
|
||||
if os.environ.get("BASIC_MEMORY_TEST_POSTGRES", "").lower() in ("1", "true", "yes"):
|
||||
return "postgres"
|
||||
return "sqlite"
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def postgres_container(db_backend):
|
||||
"""Session-scoped Postgres container for tests.
|
||||
|
||||
Uses testcontainers to spin up a real Postgres instance in Docker.
|
||||
The container is started once per test session and shared across all tests.
|
||||
Only starts if db_backend is "postgres".
|
||||
"""
|
||||
if db_backend != "postgres":
|
||||
yield None
|
||||
return
|
||||
|
||||
with PostgresContainer("postgres:16-alpine") as postgres:
|
||||
yield postgres
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def anyio_backend():
|
||||
return "asyncio"
|
||||
|
||||
|
||||
@pytest.fixture(
|
||||
params=[
|
||||
pytest.param("sqlite", id="sqlite"),
|
||||
pytest.param("postgres", id="postgres", marks=pytest.mark.postgres),
|
||||
]
|
||||
)
|
||||
def db_backend(request) -> Literal["sqlite", "postgres"]:
|
||||
"""Parametrize tests to run against both SQLite and Postgres.
|
||||
|
||||
Usage:
|
||||
pytest # Runs tests against SQLite only (default)
|
||||
pytest -m postgres # Runs tests against Postgres only
|
||||
pytest -m "not postgres" # Runs tests against SQLite only
|
||||
pytest --run-all-backends # Runs tests against both backends
|
||||
|
||||
Note: Only tests that use database fixtures (engine_factory, session_maker, etc.)
|
||||
will be parametrized. Tests that don't use the database won't be affected.
|
||||
"""
|
||||
return request.param
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def project_root() -> Path:
|
||||
return Path(__file__).parent.parent
|
||||
@@ -81,24 +103,18 @@ def config_home(tmp_path, monkeypatch) -> Path:
|
||||
|
||||
|
||||
@pytest.fixture(scope="function")
|
||||
def app_config(
|
||||
config_home, db_backend: Literal["sqlite", "postgres"], monkeypatch
|
||||
) -> BasicMemoryConfig:
|
||||
"""Create test app configuration."""
|
||||
# Create a basic config without depending on test_project to avoid circular dependency
|
||||
def app_config(config_home, db_backend, postgres_container, monkeypatch) -> BasicMemoryConfig:
|
||||
"""Create test app configuration for the appropriate backend."""
|
||||
projects = {"test-project": str(config_home)}
|
||||
|
||||
# Configure database backend based on test parameter
|
||||
# Set backend based on parameterized db_backend fixture
|
||||
if db_backend == "postgres":
|
||||
database_backend = DatabaseBackend.POSTGRES
|
||||
# Use env var if set, otherwise use default matching docker-compose-postgres.yml
|
||||
# These are local test credentials only - NOT for production
|
||||
database_url = os.getenv(
|
||||
"POSTGRES_TEST_URL",
|
||||
"postgresql+asyncpg://basic_memory_user:dev_password@localhost:5433/basic_memory_test",
|
||||
)
|
||||
backend = DatabaseBackend.POSTGRES
|
||||
# Get URL from testcontainer and convert to asyncpg driver
|
||||
sync_url = postgres_container.get_connection_url()
|
||||
database_url = sync_url.replace("postgresql+psycopg2", "postgresql+asyncpg")
|
||||
else:
|
||||
database_backend = DatabaseBackend.SQLITE
|
||||
backend = DatabaseBackend.SQLITE
|
||||
database_url = None
|
||||
|
||||
app_config = BasicMemoryConfig(
|
||||
@@ -106,7 +122,7 @@ def app_config(
|
||||
projects=projects,
|
||||
default_project="test-project",
|
||||
update_permalinks_on_move=True,
|
||||
database_backend=database_backend,
|
||||
database_backend=backend,
|
||||
database_url=database_url,
|
||||
)
|
||||
|
||||
@@ -162,76 +178,66 @@ def test_config(config_home, project_config, app_config, config_manager) -> Test
|
||||
async def engine_factory(
|
||||
app_config,
|
||||
config_manager,
|
||||
db_backend: Literal["sqlite", "postgres"],
|
||||
db_backend,
|
||||
postgres_container,
|
||||
) -> AsyncGenerator[tuple[AsyncEngine, async_sessionmaker[AsyncSession]], None]:
|
||||
"""Create engine and session factory for the configured database backend."""
|
||||
"""Engine factory for SQLite or Postgres tests.
|
||||
|
||||
Uses parameterized db_backend fixture to run tests against both backends.
|
||||
"""
|
||||
from basic_memory.models.search import CREATE_SEARCH_INDEX
|
||||
|
||||
if db_backend == "postgres":
|
||||
# Postgres: Create fresh engine for each test with full schema reset
|
||||
config_manager._config = app_config
|
||||
db_type = DatabaseType.FILESYSTEM
|
||||
# Postgres mode using testcontainers
|
||||
# Get async connection URL (asyncpg driver - same as production)
|
||||
sync_url = postgres_container.get_connection_url()
|
||||
async_url = sync_url.replace("postgresql+psycopg2", "postgresql+asyncpg")
|
||||
|
||||
# Use context manager to handle engine disposal properly
|
||||
async with db.engine_session_factory(db_path=app_config.database_path, db_type=db_type) as (
|
||||
engine,
|
||||
session_maker,
|
||||
):
|
||||
# Drop and recreate schema for complete isolation
|
||||
async with engine.begin() as conn:
|
||||
await conn.execute(text("DROP SCHEMA IF EXISTS public CASCADE"))
|
||||
await conn.execute(text("CREATE SCHEMA public"))
|
||||
await conn.execute(text("GRANT ALL ON SCHEMA public TO basic_memory_user"))
|
||||
await conn.execute(text("GRANT ALL ON SCHEMA public TO public"))
|
||||
engine = create_async_engine(
|
||||
async_url,
|
||||
echo=False,
|
||||
poolclass=NullPool, # NullPool for better test isolation
|
||||
)
|
||||
|
||||
# Run migrations to create production tables (including search_index with correct schema)
|
||||
# Alembic handles duplicate migration checks, so it's safe to call this for each test
|
||||
from basic_memory.db import run_migrations
|
||||
session_maker = async_sessionmaker(
|
||||
bind=engine,
|
||||
class_=AsyncSession,
|
||||
expire_on_commit=False,
|
||||
autoflush=False,
|
||||
)
|
||||
|
||||
await run_migrations(app_config, db_type)
|
||||
from basic_memory.models.search import (
|
||||
CREATE_POSTGRES_SEARCH_INDEX_TABLE,
|
||||
CREATE_POSTGRES_SEARCH_INDEX_FTS,
|
||||
CREATE_POSTGRES_SEARCH_INDEX_METADATA,
|
||||
)
|
||||
|
||||
# For Postgres, migrations create all production tables with correct schemas
|
||||
# We only need to create test-specific tables (like ModelTest) that aren't in migrations
|
||||
# Don't create search_index via ORM - it's already created by migration with composite PK
|
||||
async with engine.begin() as conn:
|
||||
# List of tables created by migrations - don't recreate them via ORM
|
||||
production_tables = {
|
||||
"entity",
|
||||
"observation",
|
||||
"relation",
|
||||
"project",
|
||||
"search_index",
|
||||
"alembic_version",
|
||||
}
|
||||
# Drop and recreate all tables for test isolation
|
||||
async with engine.begin() as conn:
|
||||
# Must drop search_index first (has FK to project, blocks drop_all)
|
||||
await conn.execute(text("DROP TABLE IF EXISTS search_index CASCADE"))
|
||||
await conn.run_sync(Base.metadata.drop_all)
|
||||
await conn.run_sync(Base.metadata.create_all)
|
||||
# Create search_index via DDL (not ORM - uses composite PK + tsvector)
|
||||
# asyncpg requires separate execute calls for each statement
|
||||
await conn.execute(CREATE_POSTGRES_SEARCH_INDEX_TABLE)
|
||||
await conn.execute(CREATE_POSTGRES_SEARCH_INDEX_FTS)
|
||||
await conn.execute(CREATE_POSTGRES_SEARCH_INDEX_METADATA)
|
||||
|
||||
# Get test-specific tables that aren't created by migrations
|
||||
test_tables = [
|
||||
table
|
||||
for table in Base.metadata.sorted_tables
|
||||
if table.name not in production_tables
|
||||
]
|
||||
if test_tables:
|
||||
await conn.run_sync(
|
||||
lambda sync_conn: Base.metadata.create_all(sync_conn, tables=test_tables)
|
||||
)
|
||||
yield engine, session_maker
|
||||
|
||||
yield engine, session_maker
|
||||
await engine.dispose()
|
||||
else:
|
||||
# SQLite: Create fresh in-memory database for each test
|
||||
# SQLite mode
|
||||
db_type = DatabaseType.MEMORY
|
||||
async with db.engine_session_factory(db_path=app_config.database_path, db_type=db_type) as (
|
||||
engine,
|
||||
session_maker,
|
||||
):
|
||||
# Create all tables via ORM
|
||||
# Create all tables via ORM, then add search_index via FTS5 DDL
|
||||
async with engine.begin() as conn:
|
||||
await conn.run_sync(Base.metadata.create_all)
|
||||
|
||||
# Drop any SearchIndex ORM table, then create FTS5 virtual table
|
||||
async with db.scoped_session(session_maker) as session:
|
||||
await session.execute(text("DROP TABLE IF EXISTS search_index"))
|
||||
await session.execute(CREATE_SEARCH_INDEX)
|
||||
await session.commit()
|
||||
await conn.execute(CREATE_SEARCH_INDEX)
|
||||
|
||||
# Yield after setup is complete
|
||||
yield engine, session_maker
|
||||
@@ -543,8 +549,22 @@ async def test_graph(
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def watch_service(app_config: BasicMemoryConfig, project_repository) -> WatchService:
|
||||
return WatchService(app_config=app_config, project_repository=project_repository)
|
||||
def watch_service(app_config: BasicMemoryConfig, project_repository, sync_service) -> WatchService:
|
||||
"""Create WatchService with injected sync_service factory.
|
||||
|
||||
The sync_service_factory allows tests to use the fixture-provided sync_service
|
||||
instead of the production get_sync_service() which creates its own db connection.
|
||||
"""
|
||||
|
||||
async def sync_service_factory(project):
|
||||
"""Return the test fixture's sync_service regardless of project."""
|
||||
return sync_service
|
||||
|
||||
return WatchService(
|
||||
app_config=app_config,
|
||||
project_repository=project_repository,
|
||||
sync_service_factory=sync_service_factory,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
|
||||
@@ -85,11 +85,11 @@ async def test_parse_complete_file(project_config, entity_parser, valid_entity_c
|
||||
), "missing [[Auth API Spec]]"
|
||||
|
||||
# inline links in content
|
||||
assert Relation(type="links to", target="Random Link", context=None) in entity.relations, (
|
||||
assert Relation(type="links_to", target="Random Link", context=None) in entity.relations, (
|
||||
"missing [[Random Link]]"
|
||||
)
|
||||
assert (
|
||||
Relation(type="links to", target="Random Link with Title|Titled Link", context=None)
|
||||
Relation(type="links_to", target="Random Link with Title|Titled Link", context=None)
|
||||
in entity.relations
|
||||
), "missing [[Random Link with Title|Titled Link]]"
|
||||
|
||||
@@ -179,7 +179,7 @@ async def test_parse_file_without_section_headers(project_config, entity_parser)
|
||||
assert entity.observations[0].tags == ["test"]
|
||||
|
||||
assert len(entity.relations) == 2
|
||||
assert entity.relations[0].type == "links to"
|
||||
assert entity.relations[0].type == "links_to"
|
||||
assert entity.relations[0].target == "Random Link"
|
||||
|
||||
assert entity.relations[1].type == "references"
|
||||
|
||||
@@ -121,6 +121,44 @@ def test_observation_excludes_markdown_and_wiki_links():
|
||||
assert not is_observation(token), "No space after category should not be valid observation"
|
||||
|
||||
|
||||
def test_observation_excludes_html_color_codes():
|
||||
"""Test that HTML color codes are NOT interpreted as hashtags.
|
||||
|
||||
This test validates the fix for issue #446 where:
|
||||
- HTML color codes like #4285F4 in attributes were incorrectly
|
||||
causing lines to be parsed as observations.
|
||||
"""
|
||||
# HTML color code in font tag should NOT be an observation
|
||||
token = Token("inline", '**<font color="#4285F4">Jane:</font>** Welcome to the show', 0)
|
||||
assert not is_observation(token), "HTML color codes should not trigger hashtag detection"
|
||||
|
||||
# Color code in style attribute
|
||||
token = Token("inline", '<span style="color:#FF5733">Styled text</span>', 0)
|
||||
assert not is_observation(token), "Color codes in style should not be observations"
|
||||
|
||||
# Multiple color codes
|
||||
token = Token(
|
||||
"inline", '<font color="#4285F4">Blue</font> and <font color="#EA4335">Red</font>', 0
|
||||
)
|
||||
assert not is_observation(token), "Multiple color codes should not be observations"
|
||||
|
||||
# Hex color without quotes (edge case)
|
||||
token = Token("inline", "background-color:#FFFFFF is white", 0)
|
||||
assert not is_observation(token), "Inline hex colors should not be observations"
|
||||
|
||||
# But standalone hashtags SHOULD still work
|
||||
token = Token("inline", "This has a #realtag in it", 0)
|
||||
assert is_observation(token), "Standalone hashtags should still work"
|
||||
|
||||
# Multiple real hashtags
|
||||
token = Token("inline", "Tags: #design #feature #important", 0)
|
||||
assert is_observation(token), "Multiple standalone hashtags should work"
|
||||
|
||||
# Mix of color code and real tag - should be observation because of real tag
|
||||
token = Token("inline", '<font color="#4285F4">Text</font> #actualtag', 0)
|
||||
assert is_observation(token), "Real hashtag with color code should still be observation"
|
||||
|
||||
|
||||
def test_relation_plugin():
|
||||
"""Test relation plugin."""
|
||||
md = MarkdownIt().use(relation_plugin)
|
||||
@@ -143,7 +181,7 @@ def test_relation_plugin():
|
||||
token = [t for t in md.parse(content) if t.type == "inline"][0]
|
||||
rels = token.meta["relations"]
|
||||
assert len(rels) == 2
|
||||
assert rels[0]["type"] == "links to"
|
||||
assert rels[0]["type"] == "links_to"
|
||||
assert rels[0]["target"] == "Link"
|
||||
assert rels[1]["target"] == "Another Link"
|
||||
|
||||
@@ -208,4 +246,4 @@ def test_combined_plugins():
|
||||
text_token = inline_tokens[4]
|
||||
assert "relations" in text_token.meta
|
||||
link = text_token.meta["relations"][0]
|
||||
assert link["type"] == "links to"
|
||||
assert link["type"] == "links_to"
|
||||
|
||||
@@ -88,7 +88,7 @@ async def test_missing_sections(tmp_path):
|
||||
entity = await parser.parse_file(test_file)
|
||||
assert len(entity.relations) == 1
|
||||
assert entity.relations[0].target == "links"
|
||||
assert entity.relations[0].type == "links to"
|
||||
assert entity.relations[0].type == "links_to"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
@@ -50,6 +50,7 @@ def test_entity_data():
|
||||
}
|
||||
|
||||
|
||||
@pytest_asyncio.fixture(autouse=True)
|
||||
@pytest_asyncio.fixture
|
||||
async def init_search_index(search_service: SearchService):
|
||||
"""Initialize search index. Request this fixture explicitly in tests that need it."""
|
||||
await search_service.init_search_index()
|
||||
|
||||
@@ -116,6 +116,7 @@ def test_prompt_context_with_file_path_no_permalink():
|
||||
|
||||
# Create a mock context with a file that has no permalink (like a binary file)
|
||||
test_entity = EntitySummary(
|
||||
entity_id=1,
|
||||
type="entity",
|
||||
title="Test File",
|
||||
permalink=None, # No permalink
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
"""Tests for the move_note MCP tool."""
|
||||
|
||||
import pytest
|
||||
from unittest.mock import patch
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch, MagicMock
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from basic_memory.mcp.tools.move_note import move_note, _format_move_error_response
|
||||
from basic_memory.mcp.tools.write_note import write_note
|
||||
@@ -887,36 +889,71 @@ class TestMoveNoteSecurityValidation:
|
||||
|
||||
|
||||
class TestMoveNoteErrorHandling:
|
||||
"""Test move note exception handling."""
|
||||
"""Test move note exception handling.
|
||||
|
||||
These are pure unit tests that mock get_client and other dependencies.
|
||||
They don't need the database or ASGI app.
|
||||
"""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_client(self):
|
||||
"""Create a mock async client context manager."""
|
||||
mock = MagicMock()
|
||||
|
||||
@asynccontextmanager
|
||||
async def mock_get_client():
|
||||
yield mock
|
||||
|
||||
return mock_get_client, mock
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_move_note_exception_handling(self):
|
||||
async def test_move_note_exception_handling(self, mock_client):
|
||||
"""Test exception handling in move_note."""
|
||||
with patch("basic_memory.mcp.tools.move_note.get_active_project") as mock_get_project:
|
||||
mock_get_project.return_value.project_url = "http://test"
|
||||
mock_get_project.return_value.name = "test-project"
|
||||
mock_get_client, _ = mock_client
|
||||
|
||||
with patch(
|
||||
"basic_memory.mcp.tools.move_note.call_post",
|
||||
side_effect=Exception("entity not found"),
|
||||
):
|
||||
result = await move_note.fn("test-note", "target/file.md", project="test-project")
|
||||
with patch("basic_memory.mcp.tools.move_note.get_client", mock_get_client):
|
||||
with patch("basic_memory.mcp.tools.move_note.get_active_project") as mock_get_project:
|
||||
mock_get_project.return_value.project_url = "http://test"
|
||||
mock_get_project.return_value.name = "test-project"
|
||||
mock_get_project.return_value.home = Path("/tmp/test")
|
||||
|
||||
assert isinstance(result, str)
|
||||
assert "# Move Failed - Note Not Found" in result
|
||||
with patch(
|
||||
"basic_memory.mcp.tools.move_note.call_post",
|
||||
side_effect=Exception("entity not found"),
|
||||
):
|
||||
with patch(
|
||||
"basic_memory.mcp.tools.move_note.call_get",
|
||||
side_effect=Exception("not found"),
|
||||
):
|
||||
result = await move_note.fn(
|
||||
"test-note", "target/file.md", project="test-project"
|
||||
)
|
||||
|
||||
assert isinstance(result, str)
|
||||
assert "# Move Failed - Note Not Found" in result
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_move_note_permission_error_handling(self):
|
||||
async def test_move_note_permission_error_handling(self, mock_client):
|
||||
"""Test permission error handling in move_note."""
|
||||
with patch("basic_memory.mcp.tools.move_note.get_active_project") as mock_get_project:
|
||||
mock_get_project.return_value.project_url = "http://test"
|
||||
mock_get_project.return_value.name = "test-project"
|
||||
mock_get_client, _ = mock_client
|
||||
|
||||
with patch(
|
||||
"basic_memory.mcp.tools.move_note.call_post",
|
||||
side_effect=Exception("permission denied"),
|
||||
):
|
||||
result = await move_note.fn("test-note", "target/file.md", project="test-project")
|
||||
with patch("basic_memory.mcp.tools.move_note.get_client", mock_get_client):
|
||||
with patch("basic_memory.mcp.tools.move_note.get_active_project") as mock_get_project:
|
||||
mock_get_project.return_value.project_url = "http://test"
|
||||
mock_get_project.return_value.name = "test-project"
|
||||
mock_get_project.return_value.home = Path("/tmp/test")
|
||||
|
||||
assert isinstance(result, str)
|
||||
assert "# Move Failed - Permission Error" in result
|
||||
with patch(
|
||||
"basic_memory.mcp.tools.move_note.call_post",
|
||||
side_effect=Exception("permission denied"),
|
||||
):
|
||||
with patch(
|
||||
"basic_memory.mcp.tools.move_note.call_get",
|
||||
side_effect=Exception("not found"),
|
||||
):
|
||||
result = await move_note.fn(
|
||||
"test-note", "target/file.md", project="test-project"
|
||||
)
|
||||
|
||||
assert isinstance(result, str)
|
||||
assert "# Move Failed - Permission Error" in result
|
||||
|
||||
@@ -18,10 +18,12 @@ async def entity_with_observations(session_maker, sample_entity):
|
||||
async with db.scoped_session(session_maker) as session:
|
||||
observations = [
|
||||
Observation(
|
||||
project_id=sample_entity.project_id,
|
||||
entity_id=sample_entity.id,
|
||||
content="First observation",
|
||||
),
|
||||
Observation(
|
||||
project_id=sample_entity.project_id,
|
||||
entity_id=sample_entity.id,
|
||||
content="Second observation",
|
||||
),
|
||||
@@ -59,6 +61,7 @@ async def related_results(session_maker, test_project: Project):
|
||||
await session.flush()
|
||||
|
||||
relation = Relation(
|
||||
project_id=test_project.id,
|
||||
from_id=source.id,
|
||||
to_id=target.id,
|
||||
to_name=target.title,
|
||||
@@ -176,6 +179,55 @@ async def test_update_entity(entity_repository: EntityRepository, sample_entity:
|
||||
assert db_entity.title == "Updated title"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_entity_returns_with_relations_and_observations(
|
||||
entity_repository: EntityRepository, entity_with_observations, test_project: Project
|
||||
):
|
||||
"""Test that update() returns entity with observations and relations eagerly loaded."""
|
||||
entity = entity_with_observations
|
||||
|
||||
# Create a target entity and relation
|
||||
async with db.scoped_session(entity_repository.session_maker) as session:
|
||||
target = Entity(
|
||||
project_id=test_project.id,
|
||||
title="target",
|
||||
entity_type="test",
|
||||
permalink="target/target",
|
||||
file_path="target/target.md",
|
||||
content_type="text/markdown",
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
)
|
||||
session.add(target)
|
||||
await session.flush()
|
||||
|
||||
relation = Relation(
|
||||
project_id=test_project.id,
|
||||
from_id=entity.id,
|
||||
to_id=target.id,
|
||||
to_name=target.title,
|
||||
relation_type="connects_to",
|
||||
)
|
||||
session.add(relation)
|
||||
|
||||
# Now update the entity
|
||||
updated = await entity_repository.update(entity.id, {"title": "Updated with relations"})
|
||||
|
||||
# Verify returned entity has observations and relations accessible
|
||||
# (would raise DetachedInstanceError if not eagerly loaded)
|
||||
assert updated is not None
|
||||
assert updated.title == "Updated with relations"
|
||||
|
||||
# Access observations - should NOT raise DetachedInstanceError
|
||||
assert len(updated.observations) == 2
|
||||
assert updated.observations[0].content in ["First observation", "Second observation"]
|
||||
|
||||
# Access relations - should NOT raise DetachedInstanceError
|
||||
assert len(updated.relations) == 1
|
||||
assert updated.relations[0].relation_type == "connects_to"
|
||||
assert updated.relations[0].to_name == "target"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_delete_entity(entity_repository: EntityRepository, sample_entity):
|
||||
"""Test deleting an entity."""
|
||||
@@ -737,6 +789,7 @@ async def test_get_all_file_paths_performance(entity_repository: EntityRepositor
|
||||
|
||||
# Add observations to entity1
|
||||
observation = Observation(
|
||||
project_id=entity_repository.project_id,
|
||||
entity_id=entity1.id,
|
||||
content="Test observation",
|
||||
category="note",
|
||||
@@ -745,6 +798,7 @@ async def test_get_all_file_paths_performance(entity_repository: EntityRepositor
|
||||
|
||||
# Add relation between entities
|
||||
relation = Relation(
|
||||
project_id=entity_repository.project_id,
|
||||
from_id=entity1.id,
|
||||
to_id=entity2.id,
|
||||
to_name=entity2.title,
|
||||
@@ -810,3 +864,191 @@ async def test_get_all_file_paths_project_isolation(
|
||||
# Should only include files from project 1
|
||||
assert len(file_paths) == 1
|
||||
assert file_paths == ["test/file1.md"]
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Tests for lightweight permalink resolution methods
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_permalink_exists(entity_repository: EntityRepository, sample_entity: Entity):
|
||||
"""Test checking if a permalink exists without loading full entity."""
|
||||
# Existing permalink should return True
|
||||
assert await entity_repository.permalink_exists(sample_entity.permalink) is True # pyright: ignore [reportArgumentType]
|
||||
|
||||
# Non-existent permalink should return False
|
||||
assert await entity_repository.permalink_exists("nonexistent/permalink") is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_permalink_exists_project_isolation(
|
||||
entity_repository: EntityRepository, session_maker
|
||||
):
|
||||
"""Test that permalink_exists respects project isolation."""
|
||||
async with db.scoped_session(session_maker) as session:
|
||||
# Create entity in repository's project
|
||||
entity1 = Entity(
|
||||
project_id=entity_repository.project_id,
|
||||
title="Project 1 Entity",
|
||||
entity_type="test",
|
||||
permalink="test/entity1",
|
||||
file_path="test/entity1.md",
|
||||
content_type="text/markdown",
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
)
|
||||
session.add(entity1)
|
||||
|
||||
# Create a second project with same permalink
|
||||
project2 = Project(name="other-project", path="/tmp/other")
|
||||
session.add(project2)
|
||||
await session.flush()
|
||||
|
||||
entity2 = Entity(
|
||||
project_id=project2.id,
|
||||
title="Project 2 Entity",
|
||||
entity_type="test",
|
||||
permalink="test/entity2",
|
||||
file_path="test/entity2.md",
|
||||
content_type="text/markdown",
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
)
|
||||
session.add(entity2)
|
||||
|
||||
# Should find entity1's permalink in project 1
|
||||
assert await entity_repository.permalink_exists("test/entity1") is True
|
||||
|
||||
# Should NOT find entity2's permalink (it's in project 2)
|
||||
assert await entity_repository.permalink_exists("test/entity2") is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_file_path_for_permalink(
|
||||
entity_repository: EntityRepository, sample_entity: Entity
|
||||
):
|
||||
"""Test getting file_path for a permalink without loading full entity."""
|
||||
# Existing permalink should return file_path
|
||||
file_path = await entity_repository.get_file_path_for_permalink(sample_entity.permalink) # pyright: ignore [reportArgumentType]
|
||||
assert file_path == sample_entity.file_path
|
||||
|
||||
# Non-existent permalink should return None
|
||||
result = await entity_repository.get_file_path_for_permalink("nonexistent/permalink")
|
||||
assert result is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_permalink_for_file_path(
|
||||
entity_repository: EntityRepository, sample_entity: Entity
|
||||
):
|
||||
"""Test getting permalink for a file_path without loading full entity."""
|
||||
# Existing file_path should return permalink
|
||||
permalink = await entity_repository.get_permalink_for_file_path(sample_entity.file_path)
|
||||
assert permalink == sample_entity.permalink
|
||||
|
||||
# Non-existent file_path should return None
|
||||
result = await entity_repository.get_permalink_for_file_path("nonexistent/path.md")
|
||||
assert result is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_all_permalinks(entity_repository: EntityRepository, session_maker):
|
||||
"""Test getting all permalinks without loading full entities."""
|
||||
async with db.scoped_session(session_maker) as session:
|
||||
entity1 = Entity(
|
||||
project_id=entity_repository.project_id,
|
||||
title="Entity 1",
|
||||
entity_type="test",
|
||||
permalink="test/entity1",
|
||||
file_path="test/entity1.md",
|
||||
content_type="text/markdown",
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
)
|
||||
entity2 = Entity(
|
||||
project_id=entity_repository.project_id,
|
||||
title="Entity 2",
|
||||
entity_type="test",
|
||||
permalink="test/entity2",
|
||||
file_path="test/entity2.md",
|
||||
content_type="text/markdown",
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
)
|
||||
session.add_all([entity1, entity2])
|
||||
|
||||
permalinks = await entity_repository.get_all_permalinks()
|
||||
|
||||
assert len(permalinks) == 2
|
||||
assert set(permalinks) == {"test/entity1", "test/entity2"}
|
||||
|
||||
# Results should be strings, not entities
|
||||
for permalink in permalinks:
|
||||
assert isinstance(permalink, str)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_permalink_to_file_path_map(entity_repository: EntityRepository, session_maker):
|
||||
"""Test getting permalink -> file_path mapping for bulk operations."""
|
||||
async with db.scoped_session(session_maker) as session:
|
||||
entity1 = Entity(
|
||||
project_id=entity_repository.project_id,
|
||||
title="Entity 1",
|
||||
entity_type="test",
|
||||
permalink="test/entity1",
|
||||
file_path="test/entity1.md",
|
||||
content_type="text/markdown",
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
)
|
||||
entity2 = Entity(
|
||||
project_id=entity_repository.project_id,
|
||||
title="Entity 2",
|
||||
entity_type="test",
|
||||
permalink="test/entity2",
|
||||
file_path="test/entity2.md",
|
||||
content_type="text/markdown",
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
)
|
||||
session.add_all([entity1, entity2])
|
||||
|
||||
mapping = await entity_repository.get_permalink_to_file_path_map()
|
||||
|
||||
assert len(mapping) == 2
|
||||
assert mapping["test/entity1"] == "test/entity1.md"
|
||||
assert mapping["test/entity2"] == "test/entity2.md"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_file_path_to_permalink_map(entity_repository: EntityRepository, session_maker):
|
||||
"""Test getting file_path -> permalink mapping for bulk operations."""
|
||||
async with db.scoped_session(session_maker) as session:
|
||||
entity1 = Entity(
|
||||
project_id=entity_repository.project_id,
|
||||
title="Entity 1",
|
||||
entity_type="test",
|
||||
permalink="test/entity1",
|
||||
file_path="test/entity1.md",
|
||||
content_type="text/markdown",
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
)
|
||||
entity2 = Entity(
|
||||
project_id=entity_repository.project_id,
|
||||
title="Entity 2",
|
||||
entity_type="test",
|
||||
permalink="test/entity2",
|
||||
file_path="test/entity2.md",
|
||||
content_type="text/markdown",
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
)
|
||||
session.add_all([entity1, entity2])
|
||||
|
||||
mapping = await entity_repository.get_file_path_to_permalink_map()
|
||||
|
||||
assert len(mapping) == 2
|
||||
assert mapping["test/entity1.md"] == "test/entity1"
|
||||
assert mapping["test/entity2.md"] == "test/entity2"
|
||||
|
||||
@@ -28,6 +28,7 @@ async def test_upsert_entity_with_observations_conflict(entity_repository: Entit
|
||||
|
||||
# Add observations to the entity
|
||||
obs1 = Observation(
|
||||
project_id=entity_repository.project_id,
|
||||
content="This is a test observation",
|
||||
category="testing",
|
||||
tags=["test"],
|
||||
@@ -56,11 +57,13 @@ async def test_upsert_entity_with_observations_conflict(entity_repository: Entit
|
||||
|
||||
# Add different observations
|
||||
obs2 = Observation(
|
||||
project_id=entity_repository.project_id,
|
||||
content="This is an updated observation",
|
||||
category="updated",
|
||||
tags=["updated"],
|
||||
)
|
||||
obs3 = Observation(
|
||||
project_id=entity_repository.project_id,
|
||||
content="This is a second observation",
|
||||
category="second",
|
||||
tags=["second"],
|
||||
|
||||
@@ -22,6 +22,7 @@ async def repo(observation_repository):
|
||||
async def sample_observation(repo, sample_entity: Entity):
|
||||
"""Create a sample observation for testing"""
|
||||
observation_data = {
|
||||
"project_id": sample_entity.project_id,
|
||||
"entity_id": sample_entity.id,
|
||||
"content": "Test observation",
|
||||
"context": "test-context",
|
||||
@@ -35,6 +36,7 @@ async def test_create_observation(
|
||||
):
|
||||
"""Test creating a new observation"""
|
||||
observation_data = {
|
||||
"project_id": sample_entity.project_id,
|
||||
"entity_id": sample_entity.id,
|
||||
"content": "Test content",
|
||||
"context": "test-context",
|
||||
@@ -52,6 +54,7 @@ async def test_create_observation_entity_does_not_exist(
|
||||
):
|
||||
"""Test creating a new observation"""
|
||||
observation_data = {
|
||||
"project_id": sample_entity.project_id,
|
||||
"entity_id": 99999, # Non-existent entity ID (integer for Postgres compatibility)
|
||||
"content": "Test content",
|
||||
"context": "test-context",
|
||||
@@ -104,10 +107,12 @@ async def test_delete_observations(session_maker: async_sessionmaker, repo, test
|
||||
|
||||
# Create test observations
|
||||
obs1 = Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content="Test observation 1",
|
||||
)
|
||||
obs2 = Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content="Test observation 2",
|
||||
)
|
||||
@@ -144,6 +149,7 @@ async def test_delete_observation_by_id(
|
||||
|
||||
# Create test observation
|
||||
obs = Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content="Test observation",
|
||||
)
|
||||
@@ -180,10 +186,12 @@ async def test_delete_observation_by_content(
|
||||
|
||||
# Create test observations
|
||||
obs1 = Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content="Delete this observation",
|
||||
)
|
||||
obs2 = Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content="Keep this observation",
|
||||
)
|
||||
@@ -220,16 +228,19 @@ async def test_find_by_category(session_maker: async_sessionmaker, repo, test_pr
|
||||
# Create test observations with different categories
|
||||
observations = [
|
||||
Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content="Tech observation",
|
||||
category="tech",
|
||||
),
|
||||
Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content="Design observation",
|
||||
category="design",
|
||||
),
|
||||
Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content="Another tech observation",
|
||||
category="tech",
|
||||
@@ -278,21 +289,25 @@ async def test_observation_categories(
|
||||
# Create observations with various categories
|
||||
observations = [
|
||||
Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content="First tech note",
|
||||
category="tech",
|
||||
),
|
||||
Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content="Second tech note",
|
||||
category="tech", # Duplicate category
|
||||
),
|
||||
Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content="Design note",
|
||||
category="design",
|
||||
),
|
||||
Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content="Feature note",
|
||||
category="feature",
|
||||
@@ -341,6 +356,7 @@ async def test_find_by_category_case_sensitivity(
|
||||
|
||||
# Create a test observation
|
||||
obs = Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content="Tech note",
|
||||
category="tech", # lowercase in database
|
||||
@@ -356,3 +372,97 @@ async def test_find_by_category_case_sensitivity(
|
||||
|
||||
upper_case = await repo.find_by_category("TECH")
|
||||
assert len(upper_case) == 0 # Currently case-sensitive
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_observation_permalink_truncates_long_content(
|
||||
session_maker: async_sessionmaker, repo, test_project: Project
|
||||
):
|
||||
"""Test that observation permalinks truncate long content.
|
||||
|
||||
This test validates the fix for issue #446 where:
|
||||
- Long observation content (like transcript dialogue) created permalinks
|
||||
exceeding PostgreSQL's btree index limit of 2704 bytes.
|
||||
- Content is now truncated to 200 chars in the permalink property.
|
||||
"""
|
||||
async with db.scoped_session(session_maker) as session:
|
||||
entity = Entity(
|
||||
project_id=test_project.id,
|
||||
title="test_entity",
|
||||
entity_type="test",
|
||||
permalink="test/test-entity",
|
||||
file_path="test/test_entity.md",
|
||||
content_type="text/markdown",
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
)
|
||||
session.add(entity)
|
||||
await session.flush()
|
||||
|
||||
# Create observation with very long content (5000+ chars to simulate transcript)
|
||||
long_content = "A" * 5000 # Well over the 200 char limit
|
||||
obs = Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content=long_content,
|
||||
category="transcript",
|
||||
)
|
||||
session.add(obs)
|
||||
await session.flush()
|
||||
|
||||
# Access the permalink property
|
||||
permalink = obs.permalink
|
||||
|
||||
# The full content would create a permalink like:
|
||||
# test/test-entity/observations/transcript/AAAA...5000 chars
|
||||
# With truncation, it should be much shorter
|
||||
|
||||
# Content portion should be truncated to 200 chars
|
||||
# Permalink format: entity_permalink/observations/category/content
|
||||
assert len(permalink) < 300 # Should be well under 300 chars total
|
||||
assert len(long_content[:200]) == 200 # Verify truncation length
|
||||
|
||||
# Verify the permalink contains expected parts
|
||||
assert "test/test-entity" in permalink or "test-entity" in permalink
|
||||
assert "observations" in permalink
|
||||
assert "transcript" in permalink
|
||||
|
||||
# Full 5000-char content should NOT be in permalink
|
||||
assert long_content not in permalink
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_observation_permalink_short_content_unchanged(
|
||||
session_maker: async_sessionmaker, repo, test_project: Project
|
||||
):
|
||||
"""Test that short observation content is not unnecessarily truncated."""
|
||||
async with db.scoped_session(session_maker) as session:
|
||||
entity = Entity(
|
||||
project_id=test_project.id,
|
||||
title="test_entity",
|
||||
entity_type="test",
|
||||
permalink="test/test-entity",
|
||||
file_path="test/test_entity.md",
|
||||
content_type="text/markdown",
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
)
|
||||
session.add(entity)
|
||||
await session.flush()
|
||||
|
||||
# Create observation with short content
|
||||
short_content = "Short observation content"
|
||||
obs = Observation(
|
||||
project_id=test_project.id,
|
||||
entity_id=entity.id,
|
||||
content=short_content,
|
||||
category="note",
|
||||
)
|
||||
session.add(obs)
|
||||
await session.flush()
|
||||
|
||||
permalink = obs.permalink
|
||||
|
||||
# Short content should be fully included (after permalink normalization)
|
||||
# The generate_permalink function normalizes the content
|
||||
assert "short-observation-content" in permalink.lower()
|
||||
|
||||
@@ -50,16 +50,18 @@ async def target_entity(session_maker, test_project: Project):
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def test_relations(session_maker, source_entity, target_entity):
|
||||
async def test_relations(session_maker, source_entity, target_entity, test_project: Project):
|
||||
"""Create test relations."""
|
||||
relations = [
|
||||
Relation(
|
||||
project_id=test_project.id,
|
||||
from_id=source_entity.id,
|
||||
to_id=target_entity.id,
|
||||
to_name=target_entity.title,
|
||||
relation_type="connects_to",
|
||||
),
|
||||
Relation(
|
||||
project_id=test_project.id,
|
||||
from_id=source_entity.id,
|
||||
to_id=target_entity.id,
|
||||
to_name=target_entity.title,
|
||||
@@ -349,3 +351,156 @@ async def test_delete_nonexistent_relation(relation_repository):
|
||||
"""Test deleting a relation that doesn't exist."""
|
||||
result = await relation_repository.delete_by_fields(relation_type="nonexistent")
|
||||
assert result is False
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Tests for add_all_ignore_duplicates
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_add_all_ignore_duplicates_basic(
|
||||
relation_repository: RelationRepository, sample_entity: Entity, related_entity: Entity
|
||||
):
|
||||
"""Test bulk inserting relations with ON CONFLICT DO NOTHING."""
|
||||
relations = [
|
||||
Relation(
|
||||
from_id=sample_entity.id,
|
||||
to_id=related_entity.id,
|
||||
to_name=related_entity.title,
|
||||
relation_type="links_to",
|
||||
),
|
||||
Relation(
|
||||
from_id=sample_entity.id,
|
||||
to_id=related_entity.id,
|
||||
to_name=related_entity.title,
|
||||
relation_type="references",
|
||||
),
|
||||
]
|
||||
|
||||
inserted = await relation_repository.add_all_ignore_duplicates(relations)
|
||||
|
||||
# Both should be inserted
|
||||
assert inserted == 2
|
||||
|
||||
# Verify they exist
|
||||
found = await relation_repository.find_by_entities(sample_entity.id, related_entity.id)
|
||||
assert len(found) == 2
|
||||
relation_types = {r.relation_type for r in found}
|
||||
assert relation_types == {"links_to", "references"}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_add_all_ignore_duplicates_skips_duplicates(
|
||||
relation_repository: RelationRepository, sample_entity: Entity, related_entity: Entity
|
||||
):
|
||||
"""Test that duplicate relations are silently ignored."""
|
||||
# Same relation appearing multiple times (common when same [[link]] appears twice in doc)
|
||||
relations = [
|
||||
Relation(
|
||||
from_id=sample_entity.id,
|
||||
to_id=None, # Unresolved
|
||||
to_name="Some Target",
|
||||
relation_type="links_to",
|
||||
),
|
||||
Relation(
|
||||
from_id=sample_entity.id,
|
||||
to_id=None,
|
||||
to_name="Some Target", # Duplicate!
|
||||
relation_type="links_to",
|
||||
),
|
||||
Relation(
|
||||
from_id=sample_entity.id,
|
||||
to_id=None,
|
||||
to_name="Some Target", # Triple duplicate!
|
||||
relation_type="links_to",
|
||||
),
|
||||
]
|
||||
|
||||
inserted = await relation_repository.add_all_ignore_duplicates(relations)
|
||||
|
||||
# Only 1 should be inserted (duplicates ignored)
|
||||
assert inserted == 1
|
||||
|
||||
# Verify only one exists
|
||||
all_relations = await relation_repository.find_all()
|
||||
matching = [r for r in all_relations if r.to_name == "Some Target"]
|
||||
assert len(matching) == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_add_all_ignore_duplicates_empty_list(relation_repository: RelationRepository):
|
||||
"""Test with empty list returns 0."""
|
||||
inserted = await relation_repository.add_all_ignore_duplicates([])
|
||||
assert inserted == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_add_all_ignore_duplicates_mixed(
|
||||
relation_repository: RelationRepository, sample_entity: Entity, related_entity: Entity
|
||||
):
|
||||
"""Test with mix of new and duplicate relations."""
|
||||
# First, insert one relation
|
||||
first_relation = Relation(
|
||||
from_id=sample_entity.id,
|
||||
to_id=None,
|
||||
to_name="Existing Target",
|
||||
relation_type="links_to",
|
||||
)
|
||||
await relation_repository.add_all_ignore_duplicates([first_relation])
|
||||
|
||||
# Now try to insert a mix of new and duplicate
|
||||
relations = [
|
||||
Relation(
|
||||
from_id=sample_entity.id,
|
||||
to_id=None,
|
||||
to_name="Existing Target", # Duplicate of first_relation
|
||||
relation_type="links_to",
|
||||
),
|
||||
Relation(
|
||||
from_id=sample_entity.id,
|
||||
to_id=None,
|
||||
to_name="New Target 1", # New
|
||||
relation_type="links_to",
|
||||
),
|
||||
Relation(
|
||||
from_id=sample_entity.id,
|
||||
to_id=None,
|
||||
to_name="New Target 2", # New
|
||||
relation_type="references",
|
||||
),
|
||||
]
|
||||
|
||||
inserted = await relation_repository.add_all_ignore_duplicates(relations)
|
||||
|
||||
# Only 2 new ones should be inserted
|
||||
assert inserted == 2
|
||||
|
||||
# Verify total count
|
||||
all_relations = await relation_repository.find_all()
|
||||
from_sample = [r for r in all_relations if r.from_id == sample_entity.id]
|
||||
assert len(from_sample) == 3 # 1 existing + 2 new
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_add_all_ignore_duplicates_with_context(
|
||||
relation_repository: RelationRepository, sample_entity: Entity, related_entity: Entity
|
||||
):
|
||||
"""Test that context field is properly inserted."""
|
||||
relations = [
|
||||
Relation(
|
||||
from_id=sample_entity.id,
|
||||
to_id=related_entity.id,
|
||||
to_name=related_entity.title,
|
||||
relation_type="links_to",
|
||||
context="some context here",
|
||||
),
|
||||
]
|
||||
|
||||
inserted = await relation_repository.add_all_ignore_duplicates(relations)
|
||||
assert inserted == 1
|
||||
|
||||
# Verify context was saved
|
||||
found = await relation_repository.find_by_entities(sample_entity.id, related_entity.id)
|
||||
assert len(found) == 1
|
||||
assert found[0].context == "some context here"
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user