basicmachines-co-basic-memory/docs/post-v0.18.0-test-plan.md

# Post-v0.18.0 Test Plan and Acceptance Criteria

## Goal

Define a complete validation plan for all major features merged after `v0.18.0`, combining:

- Coverage-gap-driven automated tests
- Real MCP server integration tests (no mocks for target flows)
- Manual MCP verification via LLM-driven tool calls

This plan is based on commits in `v0.18.0..HEAD` and the latest `just check` coverage output.

## Scope Window

- Start tag: `v0.18.0` (2026-01-28)
- End: current `main`
- Change volume: 12 feature commits + 14 bug-fix commits (+ release chores/hotfixes)

## Execution Strategy

1. Stabilize all feature-level acceptance criteria in automated tests first.
2. Add black-box MCP integration tests for semantic search + schema (real server startup).
3. Run manual MCP tool-call verification to confirm real UX and routing behavior.
4. Re-run full gate: `just check` + targeted integration packs.

## Global Quality Gates

- Feature criteria below must all pass.
- No regressions in existing suites.
- Coverage improves in targeted low-coverage feature modules.
- SQLite and Postgres parity for search/semantic features.

## Priority Coverage Gaps (from latest run)

These are the most important post-`v0.18.0` feature modules currently under-covered:

- `src/basic_memory/mcp/tools/schema.py` (27%)
- `src/basic_memory/mcp/clients/schema.py` (36%)
- `src/basic_memory/mcp/tools/ui_sdk.py` (43%)
- `src/basic_memory/mcp/tools/search.py` (73%)
- `src/basic_memory/repository/postgres_search_repository.py` (63%)
- `src/basic_memory/mcp/async_client.py` (82%)
- `src/basic_memory/api/v2/routers/schema_router.py` (80%)

## Feature Acceptance Criteria and Test Plan

### 1) Schema System (`c97733d`) — DONE

### Acceptance criteria

- `schema_validate`, `schema_infer`, and `schema_diff` produce consistent outcomes across CLI/API/MCP for the same fixture set.
- Strict validation fails deterministically on required-field/type violations.
- Validation warnings are stable and machine-readable in non-strict mode.
- Inference output is deterministic for unchanged input corpus.
- Drift diff output is deterministic and identifies missing/extra/type-mismatch fields correctly.

### Existing coverage anchor points

- `tests/schema/*`
- `tests/api/v2/test_schema_router.py`
- `test-int/test_schema/*`

### Gaps to close — DONE

- ~~MCP schema tool branches (`src/basic_memory/mcp/tools/schema.py`)~~ — 18 tests in `tests/mcp/test_tool_schema.py`
- ~~MCP schema client behavior (`src/basic_memory/mcp/clients/schema.py`)~~ — `tests/mcp/test_client_schema.py`
- ~~Schema router error-path branches (`src/basic_memory/api/v2/routers/schema_router.py`)~~ — `tests/api/v2/test_schema_router.py`

### Planned additions — DONE

- ~~Add MCP tool tests for `schema_validate` strict + non-strict result shapes.~~ **DONE**
- ~~Add MCP tool tests for `schema_infer` with explicit `entity_type` and inferred type fallback.~~ **DONE**
- ~~Add MCP tool tests for `schema_diff` empty-diff and non-empty-diff paths.~~ **DONE**
- ~~Add API tests for schema router invalid payload/edge error handling.~~ **DONE**
- Add integration test that starts MCP server and calls schema tools end-to-end on fixture notes. — deferred to backlog item 4.

### 2) Semantic Search (`0777879`, `1428d18`, `344e651`) — DONE

### Acceptance criteria

- `search_type=text|vector|hybrid` returns expected ranked results on canonical semantic corpus.
- Missing semantic dependencies fail fast with actionable install guidance.
- Reindex and provider/model changes produce valid vectors without dimension mismatch.
- SQLite and Postgres produce equivalent behavior for semantic modes on the same dataset.
- Generated-column migration path is valid on SQLite environments in use.

### Existing coverage anchor points

- `tests/repository/test_sqlite_vector_search_repository.py`
- `tests/repository/test_postgres_search_repository.py`
- `tests/services/test_semantic_search.py`
- `tests/mcp/test_tool_search.py`
- `test-int/test_search_performance_benchmark.py`

### Gaps to close — DONE

- ~~Uncovered Postgres vector/hybrid branches~~ — 20 tests in `tests/repository/test_postgres_search_repository_unit.py` + 5 integration tests in `test-int/semantic/test_semantic_coverage.py`
- ~~MCP search semantic/output branches~~ — expanded `tests/mcp/test_tool_search.py`

### Planned additions — DONE

- ~~Expand Postgres repository tests for vector query composition edge cases.~~ **DONE**
- ~~Expand Postgres repository tests for hybrid fusion ranking and pagination branches.~~ **DONE**
- ~~Expand Postgres repository tests for embedding/provider error handling branches.~~ **DONE**
- ~~Expand MCP search tool tests for vector/hybrid output formatting branches.~~ **DONE**
- ~~Expand MCP search tool tests for semantic-disabled and missing-dependency failures.~~ **DONE**
- Add MCP integration tests that start server and execute semantic `search_notes` tool calls. — deferred to backlog item 4.

### Semantic search quality benchmarks (NEW)

Full benchmark suite in `test-int/semantic/` covering 5 backend×provider combinations:
- `sqlite-fts`, `sqlite-fastembed`, `postgres-fts`, `postgres-fastembed`, `postgres-openai`
- Quality metrics: hit@1, recall@5, MRR@10 with per-query timing
- Realistic corpus with cross-topic vocabulary overlap (240 notes, 4 topics)
- Rich CLI viewer: `just semantic-report`
- JSON artifact output: `just test-semantic-report`

Key finding: **FastEmbed (384-d local ONNX) matches or exceeds OpenAI (1536-d) quality at 30x lower latency.** Recommending FastEmbed as default for both local and cloud deployments.

### 3) Per-Project Local/Cloud Routing + API Key Auth (`d84708c`, `ed94877`, `312662f`) — DONE

### Acceptance criteria

- Project mode (`local`/`cloud`) persists and displays correctly.
- Routing selects ASGI for local projects and HTTP+Bearer for cloud projects.
- Cloud project without key fails with explicit remediation (`cloud set-key`/`cloud create-key`).
- Resolution precedence is correct (factory > force-local > per-project cloud > global fallback > local).
- Watch/sync only run for local projects.

### Existing coverage anchor points

- `tests/mcp/test_async_client_modes.py`
- `tests/cli/test_project_set_cloud_local.py`
- `tests/mcp/test_project_context.py`
- `tests/test_project_resolver.py`
- `tests/sync/test_watch_service_reload.py`

### Gaps to close — DONE

- ~~Cloud routing branch gaps in `src/basic_memory/mcp/async_client.py`~~ — expanded `tests/mcp/test_async_client_modes.py`

### Planned additions — DONE

- ~~Add branch-focused tests for all unresolved routing branches in `get_client()`.~~ **DONE**
- Add MCP integration scenario with mixed local/cloud project config — deferred to backlog item 4.

### 4) Project-Prefixed Permalinks + Memory URL Routing (`545804f`) — DONE

### Acceptance criteria

- Project-prefixed permalinks are generated consistently on create/update/import flows.
- Memory URLs resolve to the correct project/entity even with duplicate note titles.
- `read_note`, `search`, `build_context`, write/edit/move flows preserve project identity correctly.
- Link resolution remains correct for context-aware wikilinks.

### Existing coverage anchor points

- `tests/utils/test_permalink_formatting.py`
- `tests/mcp/test_tool_read_note.py`
- `tests/mcp/test_tool_search.py`
- `tests/services/test_context_service.py`
- `test-int/mcp/test_read_note_integration.py`

### Gaps to close

- No major coverage alarm in report, but keep as regression-critical due broad impact surface.

### Planned additions — DONE

- ~~Add one integration test with colliding titles across two projects and assert URL routing invariants.~~ **DONE** — `test-int/mcp/test_permalink_collision_integration.py` (2 tests: collision across projects + memory:// URL routing with project prefix)

### 5) MCP UI Variants + TUI Output (`8bc03d1`) — DONE

### Acceptance criteria

- UI resource variant selection (`tool-ui`, `vanilla`, `mcp-ui`) follows env configuration.
- `search_notes` and `read_note` expose expected resource metadata for UI hosts.
- `ascii`/`ansi` outputs are deterministic and stable for terminal clients.

### Existing coverage anchor points

- `tests/mcp/test_tool_contracts.py`
- `test-int/mcp/test_output_format_json_integration.py`
- `test-int/mcp/test_ui_sdk_integration.py`

### Gaps to close — DONE

- ~~`src/basic_memory/mcp/tools/ui_sdk.py` branch coverage~~ — `tests/mcp/test_ui_sdk.py`
- ~~`src/basic_memory/mcp/ui/sdk.py` and `src/basic_memory/mcp/ui/templates.py` branch coverage~~ — `tests/mcp/test_ui_templates.py` + `tests/mcp/test_ui_resources.py`

### Planned additions — DONE

- ~~Add unit tests for UI SDK metadata generation and template selection branches.~~ **DONE** — 31 tests
- ~~Add integration assertion for variant-specific resource URIs and metadata payload shape.~~ **DONE**

### 6) Watch Command (`8df88e4`) — DONE

### Acceptance criteria

- `basic-memory watch` starts and processes create/update/delete events.
- Watch restart/reload path does not duplicate watchers.
- Cloud-mode projects are excluded from active watcher set.

### Existing coverage anchor points

- `tests/cli/test_watch.py`
- `tests/sync/test_coordinator.py`
- `tests/sync/test_watch_service_reload.py`

### Planned additions — DONE

- ~~Add one stress-style integration test for rapid file changes and watcher stability.~~ **DONE** — `tests/sync/test_watch_service_stress.py` (3 tests: 50-file batch, mixed add/modify/delete batch, rapid modifications to same file)

### 7) CLI JSON Output (`a47c9c0`) — DONE

### Acceptance criteria

- `--format json` returns valid JSON with stable keys for success paths.
- Error paths also return JSON-shaped output with correct non-zero exits.
- Default human output remains unchanged.

### Existing coverage anchor points

- `tests/cli/test_cli_tool_json_output.py`
- `test-int/cli/test_cli_tool_json_integration.py`

### Planned additions — DONE

- ~~Add one failure-path integration test per high-use tool command.~~ **DONE** — `test-int/cli/test_cli_tool_json_failure_integration.py` (4 tests: read-note not found, write-note missing content, write→read roundtrip, recent-activity empty project)

### 8) Search/Edit and Metadata Fixes (`530cbac`, `f1d50c2`, `8838571`, `009e849`) — DONE

### Acceptance criteria

- Metadata filters produce consistent results on SQLite and Postgres.
- `tag:` shorthand works alone and with mixed query terms.
- Fast write/edit paths preserve `external_id` and metadata integrity.

### Existing coverage anchor points

- `tests/repository/test_metadata_filters.py`
- `tests/repository/test_search_repository.py`
- `tests/services/test_search_service.py`

### Planned additions — DONE

- ~~Add Postgres-specific metadata filter edge-case tests to mirror SQLite assertions exactly.~~ **DONE** — `tests/repository/test_metadata_filters_edge_cases.py` (6 tests: missing field, AND logic, contains single-element array, nested path missing intermediate, $gte/$lte boundaries, $between inclusive — all pass on both SQLite and Postgres)

### 9) Compatibility and Hotfix Regression Pack (`c46d7a6`, `a0e754b`, `343a6e1`, `24ca5f6`, `e3ced49`, `8489a3d`, `b609c4e`, `f6e0a5b`, `7624a20`)

### Acceptance criteria

- Legacy endpoints required by older CLI versions function without `405` (`GET /projects/projects`, `POST /projects/projects`, `POST /projects/config/sync`).
- Entity creation conflicts map to conflict status (not 500).
- `recent_activity` prompt defaults are correct.
- No spurious `metadata: {}` in serialized frontmatter.
- Tigris/rclone uses global consistency headers for all transaction types.
- `bm --version` fast path avoids heavy import path and remains responsive.
- Default SQLite DB path is isolated by config dir.

### Gaps to close

- ~~Commits with no direct tests added (`c46d7a6`, `344e651`, `f6e0a5b`) need explicit regression tests.~~ **DONE**

### Planned additions — DONE

- ~~Add API compat test covering all legacy endpoint methods and payloads.~~ **DONE** — `test_legacy_v1_add_project_endpoint`, `test_legacy_v1_sync_config_endpoint`
- ~~Add CLI fast-path test for `--version` import behavior/performance guard.~~ **DONE** — `test_bm_version_does_not_import_heavy_modules`
- ~~Add empty metadata serialization regression test.~~ **DONE** — `test_schema_to_markdown_empty_metadata_no_metadata_key`
- Add migration safety test for SQLite generated columns (`VIRTUAL` expectation) — deferred, low risk.

## MCP Manual Verification Plan (LLM Tool Calls)

Run after automated tests pass.

### Setup

- Start MCP server: `basic-memory mcp --transport stdio`
- Use an MCP-capable client and issue tool calls directly.

### Manual scenarios

- Schema: call `schema_validate`, `schema_infer`, and `schema_diff` on known fixtures.
- Schema: verify error and success payloads match acceptance criteria.
- Semantic search: call `search_notes` with `search_type=text|vector|hybrid`.
- Semantic search: verify ranking relevance on semantic fixture queries.
- Routing: call tools with explicit project on mixed local/cloud setup.
- Routing: verify success/failure paths with and without API key.
- Permalink routing: read/write/search notes across projects with colliding titles.
- Permalink routing: verify memory URL routing correctness.
- UI/TUI: call `search_notes` and `read_note` with UI variants and `output_format=text|json`.
- UI/TUI: verify payload/resource format and metadata completeness.

## Implementation Backlog (Ordered)

1. ~~Fill schema MCP/client/router coverage gaps.~~ **DONE** — 18 tests in `test_tool_schema.py` + `test_client_schema.py`
2. ~~Fill semantic search MCP + Postgres repository gaps.~~ **DONE** — 20 tests in `test_postgres_search_repository_unit.py` + `test_tool_search.py`
3. ~~Add compatibility regression tests (legacy endpoints, migration, version fast path).~~ **DONE** — 5 tests across 3 files (see below)
4. ~~Add feature-level integration tests (permalinks, watch, CLI JSON, metadata filters).~~ **DONE** — 15 tests across 4 files (see items 4, 6, 7, 8 above)
5. ~~Expand UI SDK and template branch tests.~~ **DONE** — 31 tests in `test_ui_templates.py` + `test_ui_sdk.py` + `test_ui_resources.py`
6. ~~Run full gate and capture results in a short release readiness summary.~~ **DONE** — see results below

### Full Gate Results (`just check`)

| Phase | Result |
|-------|--------|
| lint | PASS |
| format | PASS |
| typecheck | PASS |
| Unit tests (SQLite) | 1788 passed, 15 skipped |
| Integration tests (SQLite) | 243 passed, 4 skipped, 10 deselected |
| Unit tests (Postgres) | 1760 passed, 28 skipped |
| Integration tests (Postgres) | 234 passed, 13 skipped, 10 deselected |

**0 failures. 10 deselected = semantic benchmark tests (run separately via `just test-semantic`).**

### Item 3 Details — Compatibility Regression Tests

| Test | File | What it covers |
|------|------|----------------|
| `test_legacy_v1_add_project_endpoint` | `tests/api/v2/test_project_router.py` | POST `/projects/projects` legacy route reachable (idempotent path) |
| `test_legacy_v1_sync_config_endpoint` | `tests/api/v2/test_project_router.py` | POST `/projects/config/sync` legacy route reachable |
| `test_bm_version_does_not_import_heavy_modules` | `tests/cli/test_cli_exit.py` | `bm --version` fast path does not load `basic_memory.mcp` |
| `test_schema_to_markdown_empty_metadata_no_metadata_key` | `tests/markdown/test_entity_parser_error_handling.py` | `schema_to_markdown()` with `entity_metadata={}` emits no `metadata:` key |
| `test_legacy_v1_list_projects_endpoint` | `tests/api/v2/test_project_router.py` | (pre-existing) GET `/projects/projects` legacy route |

**Suite totals after item 3: 1764 passed, 15 skipped, 0 failures.**

## Suggested Commands

- Full suite: `just check`
- Fast loop: `just fast-check`
- E2E consistency: `just doctor`
- SQLite focused: `just test-sqlite`
- Postgres focused: `just test-postgres`
- Schema integration: `pytest test-int/test_schema -q`
- Semantic + repo focus: `pytest tests/repository/test_postgres_search_repository.py tests/mcp/test_tool_search.py tests/services/test_semantic_search.py -q`
- MCP integration focus: `pytest test-int/mcp -q`

## Exit Criteria for This Plan

- All feature acceptance criteria above are validated.
- All identified high-priority coverage gaps are addressed or explicitly documented as intentional.
- Manual MCP verification scenarios complete with no P0/P1 findings.