mirror of
https://github.com/basicmachines-co/basic-memory
synced 2026-06-21 13:47:35 +00:00
Compare commits
6 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 60ec6728de | |||
| a84e77f4af | |||
| 6910620968 | |||
| 214a54d740 | |||
| 5a90c7cedf | |||
| 9e3fe26a83 |
@@ -63,25 +63,63 @@ jobs:
|
||||
# Only run for stable releases (not dev, beta, or rc versions)
|
||||
if: ${{ !contains(github.ref_name, 'dev') && !contains(github.ref_name, 'b') && !contains(github.ref_name, 'rc') }}
|
||||
permissions:
|
||||
contents: write
|
||||
actions: read
|
||||
contents: read
|
||||
steps:
|
||||
# Inline bump replaces mislav/bump-homebrew-formula-action@v4.x.
|
||||
# The action does a HEAD request to api.github.com /repos/.../tarball/<ref>
|
||||
# with the bearer token and expects a 302 redirect. GitHub now returns
|
||||
# 303 on that endpoint when authenticated, which the action treats as a
|
||||
# fatal error. Re-implementing the bump as plain git+sed keeps the same
|
||||
# contract (update url + sha256, commit, push) with no third-party action.
|
||||
- name: Update Homebrew formula
|
||||
uses: mislav/bump-homebrew-formula-action@v4.1
|
||||
with:
|
||||
# Formula name in homebrew-basic-memory repo
|
||||
formula-name: basic-memory
|
||||
# The tap repository
|
||||
homebrew-tap: basicmachines-co/homebrew-basic-memory
|
||||
# Base branch of the tap repository
|
||||
base-branch: main
|
||||
# Download URL will be automatically constructed from the tag
|
||||
download-url: https://github.com/basicmachines-co/basic-memory/archive/refs/tags/${{ github.ref_name }}.tar.gz
|
||||
# Commit message for the formula update
|
||||
commit-message: |
|
||||
{{formulaName}} {{version}}
|
||||
|
||||
Created by https://github.com/basicmachines-co/basic-memory/actions/runs/${{ github.run_id }}
|
||||
env:
|
||||
# Personal Access Token with repo scope for homebrew-basic-memory repo
|
||||
COMMITTER_TOKEN: ${{ secrets.HOMEBREW_TOKEN }}
|
||||
HOMEBREW_TOKEN: ${{ secrets.HOMEBREW_TOKEN }}
|
||||
REF: ${{ github.ref_name }}
|
||||
REPO: ${{ github.repository }}
|
||||
RUN_URL: https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
VERSION="${REF#v}"
|
||||
ARCHIVE_URL="https://github.com/${REPO}/archive/refs/tags/${REF}.tar.gz"
|
||||
|
||||
echo "::group::Compute tarball sha256"
|
||||
SHA256="$(curl --fail --silent --location "$ARCHIVE_URL" | sha256sum | awk '{print $1}')"
|
||||
test -n "$SHA256"
|
||||
echo "sha256: $SHA256"
|
||||
echo "::endgroup::"
|
||||
|
||||
echo "::group::Clone tap"
|
||||
git clone \
|
||||
--depth 1 \
|
||||
"https://x-access-token:${HOMEBREW_TOKEN}@github.com/basicmachines-co/homebrew-basic-memory.git" \
|
||||
tap
|
||||
cd tap
|
||||
git config user.name "github-actions[bot]"
|
||||
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
|
||||
echo "::endgroup::"
|
||||
|
||||
echo "::group::Patch Formula/basic-memory.rb"
|
||||
# Pipe-delimited sed because the URL contains slashes. The Formula
|
||||
# only has one `url` and one `sha256` directive, so a first-match
|
||||
# replacement is unambiguous. POSIX character classes ([[:space:]])
|
||||
# keep this portable across BSD and GNU sed.
|
||||
sed -i -E \
|
||||
-e "s|^([[:space:]]*url[[:space:]]+)\"[^\"]+\"|\1\"${ARCHIVE_URL}\"|" \
|
||||
-e "s|^([[:space:]]*sha256[[:space:]]+)\"[^\"]+\"|\1\"${SHA256}\"|" \
|
||||
Formula/basic-memory.rb
|
||||
git --no-pager diff Formula/basic-memory.rb
|
||||
echo "::endgroup::"
|
||||
|
||||
if git diff --quiet Formula/basic-memory.rb; then
|
||||
echo "Formula already at ${REF}; nothing to do."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "::group::Commit & push"
|
||||
git add Formula/basic-memory.rb
|
||||
git commit -m "basic-memory ${VERSION}
|
||||
|
||||
Created by ${RUN_URL}"
|
||||
git push origin HEAD:main
|
||||
echo "::endgroup::"
|
||||
|
||||
+122
-1
@@ -1,6 +1,24 @@
|
||||
# CHANGELOG
|
||||
|
||||
## Unreleased
|
||||
## v0.21.1 (2026-05-16)
|
||||
|
||||
CI-only release. No user-facing changes.
|
||||
|
||||
### Maintenance
|
||||
|
||||
- **#833**: Replace `mislav/bump-homebrew-formula-action` with an inline
|
||||
bash bump step. The action's `resolveRedirect` HEAD-requests
|
||||
`api.github.com /repos/.../tarball/<ref>` expecting HTTP 302; GitHub now
|
||||
returns 303 for authenticated requests on that endpoint, which broke
|
||||
the v0.21.0 Homebrew job and required a manual tap bump. The inline
|
||||
step does the same work (curl + sha256sum, clone tap, sed-bump
|
||||
`url`/`sha256`, commit, push) without any third-party dependency.
|
||||
|
||||
## v0.21.0 (2026-05-16)
|
||||
|
||||
Workspace-aware everywhere: every MCP tool and CLI command now routes through the
|
||||
same workspace/project model, the search and sync paths are noticeably faster,
|
||||
and a handful of long-standing parsing and routing footguns are gone.
|
||||
|
||||
### Breaking Changes
|
||||
|
||||
@@ -11,6 +29,109 @@
|
||||
- Bare list wikilinks like `- [[Target]]` now index as `links_to`.
|
||||
- Prose list items like `- some other thing [[Target]]` now index as `links_to`.
|
||||
- To preserve existing multi-word relation types on re-sync, quote them before upgrading.
|
||||
- See **#824**.
|
||||
|
||||
### Features
|
||||
|
||||
- **#816**: `bm orphan` CLI command surfaces entities whose underlying markdown
|
||||
files are gone, with a flag to clean them out.
|
||||
- **#789**: Create projects directly by cloud workspace slug from MCP
|
||||
(`create_memory_project(workspace=...)`).
|
||||
- **#757**: Discover projects across every accessible cloud workspace in MCP's
|
||||
project list — no more per-workspace blind spots.
|
||||
- **#766**: MCP tools accept training-data-friendly parameter aliases
|
||||
(`q`/`search`/`text` for `query`, etc.) so models reach for them naturally.
|
||||
- **#776**: `bm db reset` refuses to run while a `basic-memory mcp` process is
|
||||
alive, so resets can no longer corrupt an open session.
|
||||
- **#791**: Search responses include result totals so pagers can stop guessing.
|
||||
- **#719**: Cloud `note_content` tenant schema primitive lands on the backend.
|
||||
- **#715**: `bm project add` accepts a `--visibility` flag for cloud projects.
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
#### Search and recent activity
|
||||
- **#832**: SQLite project deletion now sweeps `search_index`,
|
||||
`search_vector_chunks`, and `search_vector_embeddings`, so a project that
|
||||
reuses a recycled auto-increment id can't inherit the previous tenant's content.
|
||||
- **#812**: `recent_activity` orders and filters by `updated_at`, so edits bubble
|
||||
to the top instead of staying pinned to creation time.
|
||||
- **#807**: Multi-project `search_notes` is opt-in (`search_all_projects=True`);
|
||||
default search stays scoped to the resolved project.
|
||||
- **#785**: `recent_activity` caps responses and emits an explicit truncation
|
||||
footer instead of silently dropping rows.
|
||||
- **#713**: Eliminated an N+1 query in search result hydration.
|
||||
|
||||
#### Workspace / project routing
|
||||
- **#822**: `bm project list` now includes projects from every workspace, not
|
||||
just the current one.
|
||||
- **#813**, **#808**, **#806**, **#803**, **#801**, **#795**, **#790**, **#778**,
|
||||
**#777**, **#722**, **#712**, **#704**: Workspace-qualified permalink routing
|
||||
is centralized and applied consistently across `edit_note`, `delete_project`,
|
||||
`build_context`, `memory://` URLs, factory-mode project listing, cloud uploads,
|
||||
and the API client.
|
||||
|
||||
#### Sync
|
||||
- **#827**: `rclone bisync` filters from `.bmignore` are preserved across syncs.
|
||||
- **#815**: Watch service ignores hidden paths relative to the watched project,
|
||||
not just relative to `$HOME`.
|
||||
- **#814**: `scan` subprocesses no longer go through the shell, avoiding quoting
|
||||
issues with paths that contain special characters.
|
||||
- **#759**: Watch service stays inside `--project` scope.
|
||||
- **#746**: Canonical markdown is preserved during single-file sync.
|
||||
|
||||
#### Parsing
|
||||
- **#796**: Picoschema modifier descriptions (`field?(modifier, description)`)
|
||||
parse correctly.
|
||||
- **#769**: Obsidian callout blocks are skipped by the observation parser
|
||||
instead of being mis-extracted as observations.
|
||||
|
||||
#### CLI
|
||||
- **#775**: `bm project set-cloud` / `set-local` cleans up local DB state for
|
||||
the affected project.
|
||||
- **#773**: `bm cloud logout` clears `default_workspace`.
|
||||
- **#780**: `bm cloud setup` hint points at `bm cloud sync-setup`.
|
||||
- **#734**: `bm project info` shows cloud index freshness.
|
||||
- **#718**: Private cloud projects display under their `display_name` instead of
|
||||
raw UUID.
|
||||
- **#768**: `read_note` / `view_note` drop no-op pagination params from their
|
||||
signatures.
|
||||
|
||||
#### Stability
|
||||
- **#774**: `sqlite-vec` failures during init degrade gracefully to keyword-only
|
||||
search instead of crashing startup.
|
||||
- **#733**: Delete-vector and cloud-sync cleanup is now consistent.
|
||||
- **#702**: Race conditions in concurrent `delete_entity` are resolved.
|
||||
- **#724**: `external_id` is preserved when entities are re-upserted during a
|
||||
re-index.
|
||||
- **#728**: Vector init no longer issues runtime `ALTER TABLE`.
|
||||
- **#744**: `BASIC_MEMORY_CONFIG_DIR` is honored across remaining call sites.
|
||||
- **#743**: FastEmbed cache lives under the data dir instead of `/tmp`.
|
||||
- **#752**: Cloud projects report `source=cloud` in factory mode.
|
||||
- Stripped null bytes from markdown content before DB insert.
|
||||
- Allowed long `relation_type` values in API responses.
|
||||
|
||||
#### Installer
|
||||
- **#772**: Docker-compose config volume mounts under the `appuser` home.
|
||||
- **#695**: Bumped `brew outdated` timeout from 15s to 60s.
|
||||
|
||||
### Performance
|
||||
|
||||
- **#828**: CLI startup no longer pulls in the local ASGI FastAPI app when it
|
||||
isn't needed.
|
||||
- **#751**, **#726**, **#717**, **#714**, single-file/batch indexing: marked
|
||||
speedups on the sync hot path; unchanged markdown is skipped entirely.
|
||||
- **#731**, **#723**: Vector sync is faster on both backends, with tuned
|
||||
fastembed defaults.
|
||||
|
||||
### Maintenance
|
||||
|
||||
- **#825**: Dependency refresh + security hardening.
|
||||
- Updated to `fastmcp` 3.3.1.
|
||||
- **#754**: Removed in-house telemetry wrappers in favor of direct `logfire`
|
||||
usage.
|
||||
- **#736**: `ty` is now the default typechecker.
|
||||
- **#771**, **#770**, **#716**: New regression guards for vector-row cleanup,
|
||||
long relation types, and recent activity hydration.
|
||||
|
||||
## v0.20.3 (2026-03-26)
|
||||
|
||||
|
||||
+2
-2
@@ -6,12 +6,12 @@
|
||||
"url": "https://github.com/basicmachines-co/basic-memory.git",
|
||||
"source": "github"
|
||||
},
|
||||
"version": "0.20.3",
|
||||
"version": "0.21.1",
|
||||
"packages": [
|
||||
{
|
||||
"registryType": "pypi",
|
||||
"identifier": "basic-memory",
|
||||
"version": "0.20.3",
|
||||
"version": "0.21.1",
|
||||
"runtimeHint": "uvx",
|
||||
"runtimeArguments": [
|
||||
{"type": "positional", "value": "basic-memory"},
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
"""basic-memory - Local-first knowledge management combining Zettelkasten with knowledge graphs"""
|
||||
|
||||
# Package version - updated by release automation
|
||||
__version__ = "0.20.3"
|
||||
__version__ = "0.21.1"
|
||||
|
||||
# API version for FastAPI - independent of package version
|
||||
__api_version__ = "v0"
|
||||
|
||||
@@ -0,0 +1,86 @@
|
||||
"""Remove orphaned search rows whose project was already deleted.
|
||||
|
||||
Revision ID: n7i8j9k0l1m2
|
||||
Revises: m6h7i8j9k0l1
|
||||
Create Date: 2026-05-15 18:30:00.000000
|
||||
|
||||
"""
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy import inspect
|
||||
|
||||
|
||||
revision: str = "n7i8j9k0l1m2"
|
||||
down_revision: Union[str, None] = "m6h7i8j9k0l1"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def _table_exists(connection, table_name: str) -> bool:
|
||||
"""Inspector-based table check, dialect agnostic.
|
||||
|
||||
Trigger: SQLite creates search_index as an FTS5 virtual table at runtime
|
||||
via SearchRepository.init_search_index, not through Alembic, so fresh
|
||||
installs hit this migration before the table exists.
|
||||
Why: a blind DELETE against a missing table fails the whole upgrade.
|
||||
Outcome: callers skip the sweep when the table isn't present yet — the
|
||||
runtime-created table on a fresh DB has no orphans to clean.
|
||||
"""
|
||||
return table_name in inspect(connection).get_table_names()
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
"""Purge orphaned search rows left over from prior project deletions.
|
||||
|
||||
Trigger: project deletion on SQLite never removed the derived FTS rows,
|
||||
because the FTS5 virtual table can't carry a foreign key. The leak shows
|
||||
up in two shapes:
|
||||
1. project_id no longer exists in `project` (deleted project, id never
|
||||
reused).
|
||||
2. project_id still exists but `entity_id` no longer exists in `entity`
|
||||
— auto-increment handed the id to a brand-new project and the FTS
|
||||
rows from the deleted predecessor masquerade as the new tenant's data.
|
||||
Why: search_index.project_id is the only scope predicate the search
|
||||
repository applies, so leftover rows surface under the wrong project on
|
||||
every search.
|
||||
Outcome: a one-time sweep deletes both shapes, from the FTS index and
|
||||
from search_vector_chunks. Postgres already cascaded on FK delete, so
|
||||
these statements are no-ops there.
|
||||
"""
|
||||
connection = op.get_bind()
|
||||
|
||||
if _table_exists(connection, "search_index"):
|
||||
op.execute(
|
||||
"""
|
||||
DELETE FROM search_index
|
||||
WHERE project_id NOT IN (SELECT id FROM project)
|
||||
"""
|
||||
)
|
||||
op.execute(
|
||||
"""
|
||||
DELETE FROM search_index
|
||||
WHERE entity_id IS NOT NULL
|
||||
AND entity_id NOT IN (SELECT id FROM entity)
|
||||
"""
|
||||
)
|
||||
|
||||
if _table_exists(connection, "search_vector_chunks"):
|
||||
op.execute(
|
||||
"""
|
||||
DELETE FROM search_vector_chunks
|
||||
WHERE project_id NOT IN (SELECT id FROM project)
|
||||
"""
|
||||
)
|
||||
op.execute(
|
||||
"""
|
||||
DELETE FROM search_vector_chunks
|
||||
WHERE entity_id NOT IN (SELECT id FROM entity)
|
||||
"""
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
"""No-op: orphan rows cannot be reconstructed."""
|
||||
pass
|
||||
@@ -10,8 +10,13 @@ from loguru import logger
|
||||
from fastmcp import Context
|
||||
from pydantic import AliasChoices, BeforeValidator, Field
|
||||
|
||||
from basic_memory.config import ConfigManager
|
||||
from basic_memory.config import ConfigManager, has_cloud_credentials
|
||||
from basic_memory.utils import build_canonical_permalink, coerce_dict, coerce_list
|
||||
from basic_memory.mcp.async_client import (
|
||||
_explicit_routing,
|
||||
_force_local_mode,
|
||||
is_factory_mode,
|
||||
)
|
||||
from basic_memory.mcp.container import get_container
|
||||
from basic_memory.mcp.project_context import (
|
||||
detect_project_from_identifier_prefix,
|
||||
@@ -479,12 +484,30 @@ async def _search_all_projects(
|
||||
total = 0
|
||||
any_project_has_more = False
|
||||
|
||||
# Trigger: caller asked for an account-wide search.
|
||||
# Why: project_id (external UUID) routes through the cloud v2 API path,
|
||||
# which 401s on local installs because there's no JWT to present.
|
||||
# Project names route through the local-ASGI path and work for both
|
||||
# backends — cloud disambiguates names via the workspace/project
|
||||
# qualified_name already baked into project_ref["project"].
|
||||
# Outcome: forward project_id only when the same signals get_project_client
|
||||
# uses to pick a cloud route are present. Mirrors the cloud_available
|
||||
# composite in project_context.get_project_client (single source of
|
||||
# truth for "can we route to cloud?").
|
||||
config = ConfigManager().config
|
||||
use_cloud_routing = (
|
||||
is_factory_mode()
|
||||
or (_explicit_routing() and not _force_local_mode())
|
||||
or has_cloud_credentials(config)
|
||||
)
|
||||
|
||||
for project_ref in project_refs:
|
||||
recursive_project_id = project_ref["project_id"] if use_cloud_routing else None
|
||||
try:
|
||||
results = await search_notes(
|
||||
query=query,
|
||||
project=project_ref["project"],
|
||||
project_id=project_ref["project_id"],
|
||||
project_id=recursive_project_id,
|
||||
page=1,
|
||||
page_size=per_project_page_size,
|
||||
search_type=search_type,
|
||||
|
||||
@@ -4,7 +4,9 @@ from pathlib import Path
|
||||
from typing import Optional, Sequence, Union
|
||||
|
||||
|
||||
from sqlalchemy import text
|
||||
from loguru import logger
|
||||
from sqlalchemy import inspect as sa_inspect, select, text
|
||||
from sqlalchemy.exc import NoResultFound, OperationalError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
from basic_memory import db
|
||||
@@ -12,6 +14,65 @@ from basic_memory.models.project import Project
|
||||
from basic_memory.repository.repository import Repository
|
||||
|
||||
|
||||
async def _load_sqlite_vec_on_session(session) -> bool:
|
||||
"""Ensure the sqlite-vec extension is loaded on this session's connection.
|
||||
|
||||
Returns True when vec0 is available after the call. Returns False when the
|
||||
extension can't be loaded on this Python build (e.g., python.org macOS or
|
||||
Windows interpreters without `enable_load_extension`) — every connection in
|
||||
the pool shares the same interpreter, so a False here also means no
|
||||
embedding row could ever have been written, and skipping the embeddings
|
||||
purge is safe.
|
||||
|
||||
Mirrors SQLiteSearchRepository._ensure_sqlite_vec_loaded but as a free
|
||||
function: we don't have a SearchRepository instance during project delete,
|
||||
and the per-connection nature of extension loading means a pooled connection
|
||||
routed to this session might not have vec loaded even when other
|
||||
connections wrote embeddings.
|
||||
"""
|
||||
try:
|
||||
await session.execute(text("SELECT vec_version()"))
|
||||
return True
|
||||
except OperationalError:
|
||||
pass
|
||||
|
||||
try:
|
||||
import sqlite_vec # type: ignore[import-not-found]
|
||||
except ImportError:
|
||||
logger.debug("sqlite-vec package not installed; skipping vec purge")
|
||||
return False
|
||||
|
||||
async_connection = await session.connection()
|
||||
raw_connection = await async_connection.get_raw_connection()
|
||||
driver_connection = raw_connection.driver_connection
|
||||
|
||||
if not hasattr(driver_connection, "enable_load_extension"):
|
||||
# Trigger: CPython build without sqlite extension support (#711).
|
||||
# Why: load_extension is unavailable, so no connection in this pool
|
||||
# can host vec0. No embeddings exist anywhere.
|
||||
# Outcome: skip the embeddings purge entirely.
|
||||
logger.debug(
|
||||
"Skipping search_vector_embeddings purge: this Python build does "
|
||||
"not support SQLite extension loading"
|
||||
)
|
||||
return False
|
||||
|
||||
try:
|
||||
await driver_connection.enable_load_extension(True)
|
||||
await driver_connection.load_extension(sqlite_vec.loadable_path())
|
||||
await driver_connection.enable_load_extension(False)
|
||||
await session.execute(text("SELECT vec_version()"))
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"Failed to load sqlite-vec for project delete cleanup; "
|
||||
"skipping embeddings purge: {}",
|
||||
exc,
|
||||
)
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
|
||||
class ProjectRepository(Repository[Project]):
|
||||
"""Repository for Project model.
|
||||
|
||||
@@ -121,6 +182,83 @@ class ProjectRepository(Repository[Project]):
|
||||
return target_project
|
||||
return None # pragma: no cover
|
||||
|
||||
async def delete(self, entity_id: int) -> bool:
|
||||
"""Delete a project and its derived search rows in one transaction.
|
||||
|
||||
The cascade picture differs by backend:
|
||||
|
||||
- search_index → project: Postgres has ON DELETE CASCADE FK; SQLite
|
||||
stores search_index as an FTS5 virtual table and can't carry FKs,
|
||||
so it needs explicit cleanup.
|
||||
- search_vector_chunks → project: neither backend has an FK here, so
|
||||
both need an explicit DELETE.
|
||||
- search_vector_embeddings → search_vector_chunks: Postgres has an FK
|
||||
(chunk_id REFERENCES … ON DELETE CASCADE); SQLite stores embeddings
|
||||
in a vec0 virtual table keyed by rowid with no cascade. On SQLite
|
||||
the embeddings must be purged before the chunk rows, otherwise
|
||||
`_run_vector_query` keeps returning stale vectors that crowd out
|
||||
live results.
|
||||
|
||||
Each derived table is created lazily (search_index by
|
||||
SearchRepository.init_search_index, the vector tables once semantic
|
||||
search initializes), so any of them may be absent on minimal test DBs.
|
||||
Inspect the connection once and skip whichever is missing.
|
||||
"""
|
||||
logger.debug(f"Deleting Project and search rows for project_id: {entity_id}")
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
try:
|
||||
result = await session.execute(
|
||||
select(self.Model).filter(self.primary_key == entity_id)
|
||||
)
|
||||
project = result.scalars().one()
|
||||
except NoResultFound:
|
||||
logger.debug(f"No Project found to delete: {entity_id}")
|
||||
return False
|
||||
|
||||
dialect_name = session.bind.dialect.name if session.bind else "sqlite"
|
||||
is_sqlite = dialect_name == "sqlite"
|
||||
|
||||
existing_tables = await session.run_sync(
|
||||
lambda sync_session: set(sa_inspect(sync_session.connection()).get_table_names())
|
||||
)
|
||||
|
||||
# search_index: SQLite has no FK on the FTS5 virtual table; Postgres
|
||||
# cascades from the project FK, so the explicit DELETE is redundant.
|
||||
if is_sqlite and "search_index" in existing_tables:
|
||||
await session.execute(
|
||||
text("DELETE FROM search_index WHERE project_id = :project_id"),
|
||||
{"project_id": entity_id},
|
||||
)
|
||||
|
||||
# search_vector_chunks: no FK to project on either backend, so both
|
||||
# backends need this. SQLite must purge vec0 embeddings first
|
||||
# (rowid pseudocolumn — Postgres uses chunk_id and would 500 here);
|
||||
# Postgres' chunk_id FK CASCADE handles its embeddings cleanup when
|
||||
# we delete the chunk rows below.
|
||||
if "search_vector_chunks" in existing_tables:
|
||||
if is_sqlite and "search_vector_embeddings" in existing_tables:
|
||||
# Extension loading is per-connection. We must load vec0 on
|
||||
# *this* session before the DELETE; otherwise a different
|
||||
# pooled connection might have written embeddings that we'd
|
||||
# silently leave behind.
|
||||
if await _load_sqlite_vec_on_session(session):
|
||||
await session.execute(
|
||||
text(
|
||||
"DELETE FROM search_vector_embeddings WHERE rowid IN ("
|
||||
"SELECT id FROM search_vector_chunks "
|
||||
"WHERE project_id = :project_id)"
|
||||
),
|
||||
{"project_id": entity_id},
|
||||
)
|
||||
await session.execute(
|
||||
text("DELETE FROM search_vector_chunks WHERE project_id = :project_id"),
|
||||
{"project_id": entity_id},
|
||||
)
|
||||
|
||||
await session.delete(project)
|
||||
logger.debug(f"Deleted Project and search rows for project_id: {entity_id}")
|
||||
return True
|
||||
|
||||
async def update_path(self, project_id: int, new_path: str) -> Optional[Project]:
|
||||
"""Update project path.
|
||||
|
||||
|
||||
@@ -8,8 +8,38 @@ import pytest
|
||||
from basic_memory.schemas.search import SearchItemType, SearchResponse, SearchResult
|
||||
|
||||
|
||||
def _stub_routing_mode(monkeypatch, *, cloud: bool) -> None:
|
||||
"""Pin the three cloud-route signals search.py reads.
|
||||
|
||||
`_search_all_projects` only forwards project_id (external UUID) when a
|
||||
cloud route is available. The composite mirrors get_project_client:
|
||||
factory mode OR explicit --cloud OR has_cloud_credentials. Tests stub
|
||||
all three so a dev box with OAuth tokens on disk can't bleed into the
|
||||
local-mode case.
|
||||
"""
|
||||
search_mod = importlib.import_module("basic_memory.mcp.tools.search")
|
||||
monkeypatch.setattr(search_mod, "is_factory_mode", lambda: False)
|
||||
monkeypatch.setattr(search_mod, "_explicit_routing", lambda: cloud)
|
||||
monkeypatch.setattr(search_mod, "_force_local_mode", lambda: False)
|
||||
monkeypatch.setattr(search_mod, "has_cloud_credentials", lambda config: cloud)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def cloud_routing(monkeypatch):
|
||||
"""Force the cloud-routing path for multi-project search tests."""
|
||||
_stub_routing_mode(monkeypatch, cloud=True)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def local_routing(monkeypatch):
|
||||
"""Force the local-routing path for multi-project search tests."""
|
||||
_stub_routing_mode(monkeypatch, cloud=False)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_notes_search_all_projects_qualifies_result_permalinks(monkeypatch):
|
||||
async def test_search_notes_search_all_projects_qualifies_result_permalinks(
|
||||
monkeypatch, cloud_routing
|
||||
):
|
||||
"""Multi-project search belongs to search_notes and keeps result ids routable."""
|
||||
clients_mod = importlib.import_module("basic_memory.mcp.clients")
|
||||
search_mod = importlib.import_module("basic_memory.mcp.tools.search")
|
||||
@@ -169,7 +199,7 @@ async def test_search_notes_search_all_projects_with_no_refs_returns_empty_all_p
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_notes_search_all_projects_continues_after_project_failure(
|
||||
monkeypatch,
|
||||
monkeypatch, cloud_routing
|
||||
):
|
||||
"""One failing project should not discard successful all-project search results."""
|
||||
clients_mod = importlib.import_module("basic_memory.mcp.clients")
|
||||
@@ -254,3 +284,85 @@ async def test_search_notes_search_all_projects_continues_after_project_failure(
|
||||
assert result["total"] == 1
|
||||
assert any("team-paul/main" in warning for warning in warnings)
|
||||
assert any("team index unavailable" in warning for warning in warnings)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_notes_search_all_projects_local_omits_project_id(
|
||||
monkeypatch, local_routing
|
||||
):
|
||||
"""Without a cloud route, fan-out must address each project by name only.
|
||||
|
||||
project_id (external UUID) routes through the cloud v2 API path, which
|
||||
returns 401 on local installs because there's no JWT to present. Local
|
||||
fan-out has to fall back to the name-routed path so each per-project
|
||||
search actually returns results instead of silently failing.
|
||||
"""
|
||||
clients_mod = importlib.import_module("basic_memory.mcp.clients")
|
||||
search_mod = importlib.import_module("basic_memory.mcp.tools.search")
|
||||
|
||||
project_refs = [
|
||||
{
|
||||
"project": "alpha",
|
||||
"project_id": "11111111-1111-1111-1111-111111111111",
|
||||
},
|
||||
{
|
||||
"project": "beta",
|
||||
"project_id": "22222222-2222-2222-2222-222222222222",
|
||||
},
|
||||
]
|
||||
searched_projects: list[tuple[str | None, str | None]] = []
|
||||
|
||||
async def fake_load_search_project_refs(context=None):
|
||||
return project_refs
|
||||
|
||||
class StubProject:
|
||||
def __init__(self, name: str | None, external_id: str | None):
|
||||
self.name = name or "main"
|
||||
self.external_id = external_id or "local-main"
|
||||
|
||||
@asynccontextmanager
|
||||
async def fake_get_project_client(project=None, context=None, project_id=None):
|
||||
searched_projects.append((project, project_id))
|
||||
yield object(), StubProject(project, project_id)
|
||||
|
||||
async def fake_resolve_project_and_path(client, identifier, project=None, context=None):
|
||||
return StubProject(project, None), identifier, False
|
||||
|
||||
class MockSearchClient:
|
||||
def __init__(self, client, project_id):
|
||||
self.project_id = project_id
|
||||
|
||||
async def search(self, payload, page, page_size):
|
||||
return SearchResponse(
|
||||
results=[
|
||||
SearchResult(
|
||||
title=f"Note in {self.project_id or 'local'}",
|
||||
permalink="notes/example",
|
||||
content="",
|
||||
type=SearchItemType.ENTITY,
|
||||
score=0.5,
|
||||
file_path="/notes/example.md",
|
||||
)
|
||||
],
|
||||
current_page=page,
|
||||
page_size=page_size,
|
||||
total=1,
|
||||
)
|
||||
|
||||
monkeypatch.setattr(search_mod, "_load_search_project_refs", fake_load_search_project_refs)
|
||||
monkeypatch.setattr(search_mod, "get_project_client", fake_get_project_client)
|
||||
monkeypatch.setattr(search_mod, "resolve_project_and_path", fake_resolve_project_and_path)
|
||||
monkeypatch.setattr(clients_mod, "SearchClient", MockSearchClient)
|
||||
|
||||
result = await search_mod.search_notes(
|
||||
query="anything",
|
||||
search_all_projects=True,
|
||||
output_format="json",
|
||||
)
|
||||
|
||||
assert isinstance(result, dict)
|
||||
assert searched_projects == [("alpha", None), ("beta", None)], (
|
||||
"Local fan-out must omit project_id so the recursive search_notes calls "
|
||||
"take the name-routed path."
|
||||
)
|
||||
assert result["total"] == 2
|
||||
|
||||
@@ -6,7 +6,9 @@ from datetime import timezone, datetime
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import text
|
||||
|
||||
from basic_memory import db
|
||||
from basic_memory.services.project_service import ProjectService
|
||||
|
||||
|
||||
@@ -136,3 +138,215 @@ async def test_remove_project_with_related_entities(project_service: ProjectServ
|
||||
project = await project_service.get_project(test_project_name)
|
||||
if project:
|
||||
await project_service.repository.delete(project.id)
|
||||
|
||||
|
||||
async def _table_exists(session_maker, table: str) -> bool:
|
||||
"""Return True if the named table is present on the current connection."""
|
||||
from sqlalchemy import inspect as sa_inspect
|
||||
|
||||
async with db.scoped_session(session_maker) as session:
|
||||
return await session.run_sync(
|
||||
lambda sync_session: table in sa_inspect(sync_session.connection()).get_table_names()
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_remove_project_purges_search_rows(project_service: ProjectService):
|
||||
"""Project deletion must sweep the derived search tables.
|
||||
|
||||
SQLite stores search_index as an FTS5 virtual table, which cannot carry a
|
||||
foreign key, so without an explicit purge the FTS rows survive the project
|
||||
and leak into the next project that reuses the same auto-increment id.
|
||||
Postgres has the cascade FK, but we expect the same end-state on either
|
||||
backend. This test fails on the pre-fix code: search_index still holds the
|
||||
project's rows after remove_project completes.
|
||||
"""
|
||||
test_project_name = f"test-search-cleanup-{os.urandom(4).hex()}"
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
test_project_path = str(Path(temp_dir) / "test-search-cleanup")
|
||||
os.makedirs(test_project_path, exist_ok=True)
|
||||
|
||||
await project_service.add_project(test_project_name, test_project_path)
|
||||
project = await project_service.get_project(test_project_name)
|
||||
assert project is not None
|
||||
project_id = project.id
|
||||
|
||||
# Seed both derived tables directly. The bug is in the cleanup path,
|
||||
# not the indexer, so a synthetic row is enough to prove the sweep.
|
||||
async with db.scoped_session(project_service.repository.session_maker) as session:
|
||||
await session.execute(
|
||||
text(
|
||||
"INSERT INTO search_index "
|
||||
"(id, title, content_stems, content_snippet, permalink, "
|
||||
" file_path, type, project_id) "
|
||||
"VALUES (:id, :title, :stems, :snippet, :permalink, "
|
||||
" :file_path, :type, :project_id)"
|
||||
),
|
||||
{
|
||||
"id": 999_001,
|
||||
"title": "leak canary",
|
||||
"stems": "leak canary",
|
||||
"snippet": "leak canary",
|
||||
"permalink": f"leak-canary-{project_id}",
|
||||
"file_path": "leak-canary.md",
|
||||
"type": "entity",
|
||||
"project_id": project_id,
|
||||
},
|
||||
)
|
||||
await session.execute(
|
||||
text(
|
||||
"INSERT INTO search_vector_chunks "
|
||||
"(entity_id, project_id, chunk_key, chunk_text, source_hash, "
|
||||
" entity_fingerprint, embedding_model) "
|
||||
"VALUES (:entity_id, :project_id, :chunk_key, :chunk_text, "
|
||||
" :source_hash, :entity_fingerprint, :embedding_model)"
|
||||
),
|
||||
{
|
||||
"entity_id": 999_001,
|
||||
"project_id": project_id,
|
||||
"chunk_key": "canary",
|
||||
"chunk_text": "leak canary",
|
||||
"source_hash": "abc",
|
||||
"entity_fingerprint": "",
|
||||
"embedding_model": "",
|
||||
},
|
||||
)
|
||||
|
||||
async with db.scoped_session(project_service.repository.session_maker) as session:
|
||||
pre_index = (
|
||||
await session.execute(
|
||||
text("SELECT COUNT(*) FROM search_index WHERE project_id = :pid"),
|
||||
{"pid": project_id},
|
||||
)
|
||||
).scalar_one()
|
||||
pre_chunks = (
|
||||
await session.execute(
|
||||
text("SELECT COUNT(*) FROM search_vector_chunks WHERE project_id = :pid"),
|
||||
{"pid": project_id},
|
||||
)
|
||||
).scalar_one()
|
||||
assert pre_index >= 1, "seed row should exist before removal"
|
||||
assert pre_chunks >= 1, "seed chunk should exist before removal"
|
||||
|
||||
await project_service.remove_project(test_project_name)
|
||||
|
||||
async with db.scoped_session(project_service.repository.session_maker) as session:
|
||||
post_index = (
|
||||
await session.execute(
|
||||
text("SELECT COUNT(*) FROM search_index WHERE project_id = :pid"),
|
||||
{"pid": project_id},
|
||||
)
|
||||
).scalar_one()
|
||||
post_chunks = (
|
||||
await session.execute(
|
||||
text("SELECT COUNT(*) FROM search_vector_chunks WHERE project_id = :pid"),
|
||||
{"pid": project_id},
|
||||
)
|
||||
).scalar_one()
|
||||
|
||||
assert post_index == 0, (
|
||||
f"search_index still has {post_index} rows for deleted project_id={project_id} "
|
||||
"— project deletion did not sweep the FTS table."
|
||||
)
|
||||
assert post_chunks == 0, (
|
||||
f"search_vector_chunks still has {post_chunks} rows for deleted "
|
||||
f"project_id={project_id}."
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_delete_returns_false_for_missing_project_id(project_service: ProjectService):
|
||||
"""ProjectRepository.delete must return False when the project id is gone.
|
||||
|
||||
The override loses the base Repository.delete contract if the NoResultFound
|
||||
branch isn't covered — a silent True would mislead callers into thinking
|
||||
a non-existent project was removed.
|
||||
"""
|
||||
result = await project_service.repository.delete(9_999_999)
|
||||
assert result is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_remove_project_purges_vector_embeddings(project_service: ProjectService):
|
||||
"""Project deletion must also drop sqlite-vec embeddings keyed by chunk rowid.
|
||||
|
||||
sqlite-vec stores vectors in a vec0 virtual table that has no cascade
|
||||
behavior. If embeddings linger after the chunks they reference are gone,
|
||||
`_run_vector_query` pulls them as top-k candidates and crowds out live
|
||||
results. The test only runs when the embeddings table is present, which
|
||||
matches the install path that exercises semantic search.
|
||||
"""
|
||||
test_project_name = f"test-vec-cleanup-{os.urandom(4).hex()}"
|
||||
session_maker = project_service.repository.session_maker
|
||||
|
||||
# The embeddings table only exists once semantic search has initialized.
|
||||
# Skipping when it's absent keeps this test honest on minimal CI DBs.
|
||||
if not await _table_exists(session_maker, "search_vector_embeddings"):
|
||||
pytest.skip("search_vector_embeddings is not present on this connection")
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
test_project_path = str(Path(temp_dir) / "test-vec-cleanup")
|
||||
os.makedirs(test_project_path, exist_ok=True)
|
||||
|
||||
await project_service.add_project(test_project_name, test_project_path)
|
||||
project = await project_service.get_project(test_project_name)
|
||||
assert project is not None
|
||||
project_id = project.id
|
||||
|
||||
async with db.scoped_session(session_maker) as session:
|
||||
await session.execute(
|
||||
text(
|
||||
"INSERT INTO search_vector_chunks "
|
||||
"(id, entity_id, project_id, chunk_key, chunk_text, source_hash, "
|
||||
" entity_fingerprint, embedding_model) "
|
||||
"VALUES (:id, :entity_id, :project_id, :chunk_key, :chunk_text, "
|
||||
" :source_hash, :entity_fingerprint, :embedding_model)"
|
||||
),
|
||||
{
|
||||
"id": 999_201,
|
||||
"entity_id": 999_201,
|
||||
"project_id": project_id,
|
||||
"chunk_key": "vec-canary",
|
||||
"chunk_text": "vec canary",
|
||||
"source_hash": "abc",
|
||||
"entity_fingerprint": "",
|
||||
"embedding_model": "",
|
||||
},
|
||||
)
|
||||
# vec0 requires a vector matching the configured dimensions, but the
|
||||
# delete path filters by rowid; a non-existing dimension would block
|
||||
# this seed step. Skip the insert if the embeddings DDL hasn't run.
|
||||
try:
|
||||
await session.execute(
|
||||
text(
|
||||
"INSERT INTO search_vector_embeddings (rowid, embedding) "
|
||||
"VALUES (:rowid, :embedding)"
|
||||
),
|
||||
{"rowid": 999_201, "embedding": "[" + ",".join(["0.0"] * 384) + "]"},
|
||||
)
|
||||
except Exception:
|
||||
pytest.skip("search_vector_embeddings rejected the synthetic seed row")
|
||||
|
||||
async with db.scoped_session(session_maker) as session:
|
||||
pre = (
|
||||
await session.execute(
|
||||
text("SELECT COUNT(*) FROM search_vector_embeddings WHERE rowid = :rowid"),
|
||||
{"rowid": 999_201},
|
||||
)
|
||||
).scalar_one()
|
||||
assert pre >= 1, "seed embedding should exist before removal"
|
||||
|
||||
await project_service.remove_project(test_project_name)
|
||||
|
||||
async with db.scoped_session(session_maker) as session:
|
||||
post = (
|
||||
await session.execute(
|
||||
text("SELECT COUNT(*) FROM search_vector_embeddings WHERE rowid = :rowid"),
|
||||
{"rowid": 999_201},
|
||||
)
|
||||
).scalar_one()
|
||||
|
||||
assert post == 0, (
|
||||
f"search_vector_embeddings still has {post} rows for rowid 999_201 "
|
||||
"— project deletion did not sweep the sqlite-vec embeddings table."
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user