basicmachines-co-basic-memory/docs/releases/v0.19.0.md

# v0.19.0 Release Notes

## Overview

v0.19.0 is a major release that introduces semantic vector search, a schema validation system,
project-prefixed permalinks, per-project cloud routing, and a significant upgrade to FastMCP 3.0.
It includes 90+ commits since v0.18.0 spanning new features, architectural improvements, and
stability fixes across both SQLite and Postgres backends.

---

## Major Features

### Semantic Vector Search

Full vector and hybrid search for SQLite (via sqlite-vec) and Postgres (via pgvector).

- **Hybrid search mode** combines full-text search (FTS) with vector similarity for best results
- **Score-based fusion** replaces RRF for hybrid ranking — `max(vec, fts) + 0.3 * min(vec, fts)` preserves dominant signals and rewards dual-source agreement (#577)
- **Default search mode** is now `hybrid` when semantic search is enabled, `text` when disabled
- Embedding providers: FastEmbed (local, default) or OpenAI API
- Configurable similarity threshold via `semantic_min_similarity` (default 0.55)
- Per-query `min_similarity` override on `search_notes` tool
- Auto-backfill: existing entities get embeddings generated on first startup
- Backend-specific distance-to-similarity conversion (cosine for SQLite, inner product for Postgres)
- FTS fallback: if semantic dependencies are missing, search gracefully degrades to text-only
- sqlite-vec knn `k` parameter capped at 4096 to prevent backend errors

**Configuration:**
```json
{
  "semantic_search_enabled": true,
  "semantic_embedding_provider": "fastembed",
  "semantic_embedding_model": "bge-small-en-v1.5",
  "semantic_min_similarity": 0.55
}
```

**Usage:**
```
search_notes("machine learning concepts", search_type="hybrid")
search_notes("similar to my notes on coffee", search_type="vector")
search_notes("exact phrase match", search_type="text")
search_notes("broad search", min_similarity=0.3)  # lower threshold for more results
```

### Schema System

Validate note structure against user-defined schemas with frontmatter-based rules.

- Define schemas as YAML in note frontmatter with field types, required fields, and constraints
- Frontmatter validation during sync — malformed notes get clear error messages
- Schema inference from existing notes to bootstrap schemas from your content
- Schema diff to compare two schemas and see changes
- Available via MCP tools and CLI

### Project-Prefixed Permalinks

Permalinks now include the project name for unambiguous cross-project references.

- Memory URLs like `memory://project-name/folder/note` route to the correct project
- Existing non-prefixed permalinks continue to work (backwards compatible)
- Controlled by `permalinks_include_project` config (default: true)
- `build_context` and `search_notes` auto-detect project from URL prefix

### Per-Project Cloud Routing

Individual projects can be routed through the cloud while others stay local.

- Set a project to cloud mode: `bm project set-cloud research`
- Revert to local: `bm project set-local research`
- Uses API key authentication: `bm cloud set-key bmc_abc123...`
- MCP tools automatically route based on each project's mode
- Local MCP server (`bm mcp`) still uses local routing for all projects by default
- `--local` and `--cloud` CLI flags override per-command

### Workspace Selection

Cloud projects can target specific workspaces for multi-tenant environments.

- `workspace` parameter on MCP tools for explicit workspace targeting
- CLI workspace-aware project listing with `bm project list`
- Spinner feedback while fetching cloud projects

---

## New Tools and Capabilities

### Dashboard (`bm project info`)

`bm project info` now displays an htop-inspired compact dashboard with:

- Horizontal bar charts for note types (top 5)
- Embedding coverage bar with Unicode block characters
- Colored status dots for at-a-glance health
- `EmbeddingStatus` schema and `get_embedding_status()` service method for programmatic access

### Unified Metadata Search

`search_by_metadata` has been merged into `search_notes` — one tool for all searches.
`query` is now optional, so you can search purely by frontmatter metadata.

```
search_notes(metadata_filters={"status": "in-progress"})
search_notes(metadata_filters={"tags": ["security", "oauth"]})
search_notes(metadata_filters={"priority": {"$in": ["high", "critical"]}})
search_notes(metadata_filters={"schema.confidence": {"$gt": 0.7}})
search_notes(tags=["security"])  # convenience shorthand
search_notes(status="draft")     # convenience shorthand
```

### JSON Output Mode

All MCP tools now support `output_format="json"` for machine-readable responses.

- Default remains `"text"` for human-readable output (no breaking changes)
- `build_context` defaults to `"json"` with slimmed payloads (redundant fields stripped)
- CLI tool commands support `--format json` flag

### `tag:` Search Shorthand

Search by tag using convenient shorthand syntax.

```
search_notes("tag:security")
search_notes("tag:coffee AND tag:brewing")
```

### Entity User Tracking

Entities now track `created_by` and `last_updated_by` fields for attribution.

### Improved Search Result Content (#609)

Search results now surface more relevant context:

- `matched_chunk_text` populated for FTS-only hybrid results (no more fallback to truncated content)
- `TOP_CHUNKS_PER_RESULT` increased from 3 to 5, catching answers deeper in large notes (~2700 → ~4500 chars)
- `CONTENT_DISPLAY_LIMIT` doubled from 2000 to 4000 chars for results without matched chunks

### `write_note` Overwrite Guard (#632)

`write_note` is now non-idempotent by default. If a note already exists, the tool returns an
error instead of silently overwriting. Pass `overwrite=True` to replace, or use `edit_note`
for incremental updates. Config option `write_note_overwrite_default` restores the old upsert
behavior.

---

## Architecture Changes

### Score-Based Hybrid Fusion (#577)

RRF (Reciprocal Rank Fusion) compressed all fused scores to ~0.016, destroying ranking
differentiation. The new formula `max(vec, fts) + FUSION_BONUS * min(vec, fts)` preserves
dominant signals and rewards dual-source agreement. Zero-score results now produce zero
fused score instead of receiving a 0.1 weight floor.

### FastMCP 3.0 Upgrade

Upgraded from FastMCP 2.12.3 to 3.0.1.

- Tool annotations (`readOnlyHint`, `openWorldHint`) for better client integration
- Improved MCP protocol compliance
- Better error handling and context management

### Prompts Call MCP Tools Directly

MCP prompts (`search`, `continue_conversation`) now call MCP tools directly instead of
going through API endpoints. This fixes empty results in discovery mode and ensures prompts
use the same resolution logic as tools (including LinkResolver fallback).

### build_context LinkResolver Fallback

`build_context` now falls back to LinkResolver when an exact permalink lookup returns empty.
This uses the same 7-strategy resolution pipeline as `read_note`, so callers no longer get
empty results for valid note identifiers that don't match exact permalinks.

### Sync Handles Semantic Dependency Errors Gracefully

When sqlite-vec or another embedding provider is unavailable, `sync_file` now catches
`SemanticDependenciesMissingError` separately. The entity is created and FTS-indexed
successfully — only vector embeddings are skipped, with a clear warning:

```
WARNING: Semantic search dependencies missing — vector embeddings skipped for path=note.md.
Run 'bm reindex --embeddings' after resolving the dependency issue.
```

### Unified Project Path

Cloud projects with bisync now store the local filesystem path in `path` (not the Docker
container path). Config migration automatically promotes `local_sync_path` → `path` for
existing configs.

---

## CLI Improvements

### Status and Doctor Default to Local Routing

`bm status` and `bm doctor` now default to local routing since they scan the local filesystem.
Previously, cloud-mode projects would route these commands to the cloud API, which returned
Docker-internal paths that don't exist locally.

### `--format json` for CLI Tool Commands

All `bm tool` subcommands support `--format json` for machine-readable output, enabling
integration with scripts and plugins.

### `--json` for Top-Level CLI Commands

Five additional CLI commands now support `--json` for machine-readable output:

- `bm status --json` — sync report with new/modified/deleted/moved files and skipped files
- `bm project list --json` — structured project list with name, paths, routing mode, and defaults
- `bm schema validate --json` — validation report with per-note pass/fail, warnings, and errors
- `bm schema infer --json` — field frequency analysis and suggested schema definition
- `bm schema diff --json` — drift report with new fields, dropped fields, and cardinality changes

This complements the existing `bm project info --json` and `bm tool --format json` support,
making all major CLI commands scriptable for CI pipelines and automation.

### Cloud Promo and Analytics

- Cloud promo panel shown on first run or version bump with OSS discount code
- Anonymous usage telemetry via Umami Cloud (promo/login funnel events only)
- Opt out with `BASIC_MEMORY_NO_PROMOS=1`
- No PII, no file contents, no per-command tracking
- See [Telemetry](https://github.com/basicmachines-co/basic-memory#telemetry) in README

---

## Bug Fixes

- **#577**: RRF fusion compressed all hybrid scores to ~0.016, destroying ranking differentiation
- **#582**: build_context returns empty results on valid note identifiers
- **#575**: Remove hardcoded "main" default from default_project
- **#595**: recent_activity dedup and pagination across MCP tools
- **#593**: Backend-specific distance-to-similarity conversion
- **#592**: Strip NUL bytes from content before PostgreSQL search indexing
- **#562**: Use VIRTUAL instead of STORED columns in SQLite migration
- **#558**: Add X-Tigris-Consistent headers to all rclone commands
- **#541**: Handle EntityCreationError as conflict
- **#536**: Stabilize metadata filters on Postgres
- **#533**: Fix recent_activity prompt defaults
- **#530**: Prevent spurious `metadata: {}` in frontmatter output
- **#601**: Return matched chunk text in search results
- **#606**: Accept `null` for `expected_replacements` in `edit_note`
- **#579, #607**: Guard against closed streams in promo panel and missing vector tables on shutdown
- **#609**: FTS-only hybrid results missing `matched_chunk_text`; content limits too conservative
- **#631**: `build_context` related_results schema validation failure — replaced fragile `_slim_context()` stripping with Pydantic `exclude=True` field config
- **#630**: Skip workspace resolution when client factory is active — prevents 401 errors in cloud MCP server mode
- **#30**: `tag:` prefix query fails with hybrid search — moved tag prefix parsing to MCP tool level so it works with all search modes
- **#31**: `search_notes` returns cluttered observation/relation-level results — now defaults to entity-level results
- **#28**: `schema_infer` and `schema_diff` return raw Pydantic models as "undefined" in LLM output — added markdown formatters
- Fix `schema_validate` identifier resolution (now uses LinkResolver) and text rendering (markdown formatter)
- **#634**: `schema_validate` and `schema_diff` use stale database metadata instead of reading schema definitions from file — now reads frontmatter directly from the file with fallback to database metadata
- Fix `Post(**metadata)` crash when frontmatter contains `content` or `handler` keys
- Fix list-valued frontmatter fields (`title`, `type`) crashing on `.strip()` — now coerced to strings
- Cap sqlite-vec knn `k` parameter at 4096 to prevent backend errors
- Parameterize SQL queries in search repository type filters
- Double-default display in project list
- `ensure_frontmatter_on_sync` default changed to `True`
- Status/doctor commands fail with cloud-mode projects (Docker path error)
- Prompts return "0 projects" in discovery mode

---

## Security

- Upgrade `cryptography` for CVE advisory
- Upgrade `python-multipart` for security advisory

---

## Internal / Developer

- **#598**: Upgrade FastMCP 2.12.3 → 3.0.1 with tool annotations
- **#594**: Add `ty` as supplemental type checker
- **#538**: Add fast feedback loop tooling (`just fast-check`, `just doctor`, `just testmon`)
- **#600**: Rename `entity_type` to `note_type` for consistency
- **#596**: Fix CLI runtime defects and audit regressions
- CLI refactoring and workspace-aware cloud project listing
- Split and speed up PR test matrix in CI
- Fix CI: collect coverage from test jobs instead of re-running all tests
- Create `search_vector_chunks` in test fixtures for Postgres compatibility

---

## Configuration Changes

| Setting | Old Default | New Default | Notes |
|---------|-------------|-------------|-------|
| `semantic_search_enabled` | `false` | `true` | Semantic search on by default |
| `ensure_frontmatter_on_sync` | `false` | `true` | Frontmatter added during sync |
| `permalinks_include_project` | `false` | `true` | Project prefix in permalinks |

---

## Upgrade Notes

- **Semantic search dependencies** are now included by default. If sqlite-vec fails to load,
  search gracefully falls back to FTS. Run `bm reindex --embeddings` to generate embeddings
  for existing content.
- **Hybrid search scoring** has changed from RRF to score-based fusion. Search result ordering
  may differ — results should be more accurate with better score differentiation.
- **`search_by_metadata`** is removed as a standalone tool. Use `search_notes` with
  `metadata_filters` instead (same parameters, same behavior).
- **Project-prefixed permalinks** are enabled by default. Existing notes keep their current
  permalinks until modified. Set `permalinks_include_project: false` to disable.
- **Frontmatter on sync** is now enabled by default. Files without frontmatter will have it
  added on next sync. Set `ensure_frontmatter_on_sync: false` to preserve old behavior.
- **Config migration** runs automatically for cloud projects with bisync — `local_sync_path`
  is promoted to `path` so filesystem operations work correctly.
- **`write_note` is no longer idempotent** — calls to `write_note` for existing notes now
  return an error unless `overwrite=True` is passed. Use `edit_note` for incremental changes,
  or set `write_note_overwrite_default: true` in config to restore the old behavior.