Compare commits

..

10 Commits

Author SHA1 Message Date
Paul Hernandez b5f13d6903 Update README.md
add devin deep wiki badge for weekly updates

Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
2026-06-14 11:08:45 -05:00
Drew Cain 6d06c4a4e7 docs(ci): fix stale basicmachines.co post-release step
The marketing site moved to basicmemory.com (repo basicmachines-co/basicmemory.com,
now Astro + React) and no longer renders a hardcoded version number anywhere in
its UI — src/components/sections/hero.tsx has no version string. The post-release
instruction to bump the version there is obsolete. Replace it across the runbook
(release.md), AGENTS.md post-release tasks, and the release/beta justfile reminders
with the current reality: no version bump; optionally add a dated blog post under
src/content/blog/ for significant releases; skip for patches.

Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-13 14:53:52 -05:00
Drew Cain 16869867da fix(core): gate hybrid FTS relaxation with the service's eligibility rules
Addresses Codex review on #994: the hybrid FTS branch opted into
OR-relaxation for every query shape, but SearchService's relaxed FTS
path deliberately rejects short queries and numeric identifiers because
OR-relaxing them over-broadens — and in hybrid the relaxed FTS-only rows
normalize to 1.0 and can outrank the vector result the user wanted
(e.g. "SPEC 16", "root note 1", "New Feature").

relaxed_query_words now enforces the same eligibility as
SearchService._is_relaxed_fts_fallback_eligible: tokenize on
[A-Za-z0-9]+ and return None when there are fewer than three tokens or
any token is a pure digit (in addition to the existing quoted/boolean
guards). Both the SQLite and Postgres relaxed retries route through this
helper, so the hybrid path now relaxes exactly the query shapes the
service does.

Tests updated for the tightened eligibility (short + numeric queries no
longer relax); SQLite + Postgres relaxation suites green, full
repository/search-service/link-resolver suite green (442 passed), ty
clean.

Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-13 14:50:15 -05:00
Drew Cain 18da6194f9 fix(core): Postgres parity for FTS punctuation/relaxation (CI green)
CI Postgres shard caught two issues invisible to the local SQLite suite:

1. Postgres _prepare_single_term regression: the new edge-punctuation
   strip ran after special-character cleaning, so an all-special-char
   term ("()&!:") collapsed to empty and skipped the existing
   NOSPECIALCHARS:* guard, emitting a malformed ":*". Folded the strip
   into the word handlers so every guard survives, and added a
   single-word empty guard.

2. Backend-specific test assumptions. Four tests in
   test_search_repository.py (run under both backends via the
   search_repository fixture) asserted SQLite FTS5 syntax and
   SQLite-only strict-miss behavior. Postgres to_tsquery('english', ...)
   auto-strips stopwords, so "When did Melanie paint a sunrise?" already
   matches under strict AND. Made the four tests backend-aware via the
   existing is_postgres_backend() helper, and switched the relaxation
   integration test to a query with a word absent from the doc
   ("hiking") so the strict miss holds on both backends.

Reproduced and fixed against real Postgres (testcontainers): full
search test surface green on both backends (53 passed Postgres,
2968 SQLite), ruff + ty clean.

Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-13 14:50:15 -05:00
Drew Cain a6d0784335 fix(core): repair FTS half of hybrid search for natural-language queries
Hybrid search was silently running vector-only on natural-language
queries — the FTS branch contributed zero candidates. Two causes in the
SQLite (and parallel Postgres) FTS query preparation:

1. Sentence punctuation forced phrase matching. A question like
   "When did Melanie paint a sunrise?" reached FTS5 as the exact phrase
   '"When did Melanie paint a sunrise?"*', which matches no document.
   The FTS5 tokenizer ignores this punctuation in the index, so
   stripping it from word edges loses nothing — but leaving it disabled
   the entire FTS contribution. _prepare_single_term now strips
   ?!.,;: from word edges of multi-word queries (interior characters —
   hyphens, slashes in permalinks/paths — untouched).

2. No relaxation when strict all-terms-AND matched nothing. Questions
   rarely have every word in one document, so even after (1) the
   strict AND returned zero rows. The hybrid path now retries once with
   an OR-joined, stopword-filtered, content-term query when the strict
   query is empty. bm25/ts_rank still rank multi-term matches first, and
   fusion with the vector branch keeps relaxed lexical candidates from
   dominating precision.

The relaxation is gated behind a new allow_relaxed=False parameter on
SearchRepositoryBase.search; only _search_hybrid opts in. Strict FTS
behavior (search_type=text, title, permalink, link resolution) is
unchanged — the service layer keeps its own conservative fallback.
No config flag, default-safe.

Discovered via the benchmark harness: two different fusion algorithms
produced byte-identical rankings across 1,986 queries (impossible with
two live sources), and instrumentation confirmed fts=0 on 40/40
sampled LoCoMo queries.

Benchmark impact (corrected LoCoMo, 1,986 queries, same index,
retrieval metrics — every category improves, no regression):
  recall@5  0.745 -> 0.823  (+7.9)
  MRR       0.618 -> 0.718  (+10.0)
  headline  r5 0.734 -> 0.801, MRR 0.621 -> 0.706
Largest gains on open_domain (+0.10 r5) and adversarial (+0.12 r5);
smallest on temporal (+0.003 r5 / +0.02 MRR).

Tests: punctuation no longer phrase-quotes; relaxation builds the
expected OR query and respects boolean/quoted/short-query intent; the
hybrid opt-in surfaces a partial-overlap document while the default
strict path still returns empty. Parallel coverage for Postgres. Full
SQLite unit suite green (2968 passed); ty + ruff clean.

Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-13 14:50:15 -05:00
Drew Cain 232f469065 chore: update version to 0.22.1 for v0.22.1 release
Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-12 22:34:04 -05:00
Drew Cain 5a08cfd9ac docs(core): add v0.22.1 changelog entry
Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-12 22:33:34 -05:00
Drew Cain b997d858cd fix(sync): also exclude orphan DB projects absent from config (#949)
Address Codex review feedback. The previous path-only filter dropped an
implicit protection: get_project_mode() defaults projects missing from
config to CLOUD, so the old mode-based guard skipped stale DB rows that
had been removed from config. With a path-only check, an orphan row with
an absolute path would pass and background sync/watch could still mutate
a directory the user already removed from config (config is the source of
truth) if reconciliation was skipped or failed.

Introduce BasicMemoryConfig.is_locally_syncable(name, path), which
requires both config membership and an absolute path, and use it from
both the background sync selection and the watch cycle so the two paths
cannot diverge. Add direct unit tests for the helper plus an orphan-row
regression test for the watch selection.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-12 18:02:18 -05:00
Drew Cain 598965c389 test(sync): use OS-absolute paths in watch selection tests
The absolute-path sync guard from the previous commit now checks every
project's path, not just cloud-mode ones. Three existing watch-selection
tests hardcoded POSIX paths like "/tmp/alpha", which are absolute on
Linux/macOS but not on Windows (no drive letter), so the guard filtered
them out and the tests failed on windows-latest.

Build the project paths from the tmp_path fixture so they are absolute on
every platform. Production paths are unaffected: real local projects are
always resolved to OS-absolute paths at creation.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-12 18:02:18 -05:00
Drew Cain 8dd6451dfe fix(sync): skip projects without an absolute local path (#949)
A project entry in config.json with an empty path (e.g. `{"path": ""}`)
caused background sync and the watch service to adopt the process cwd as
the project root, injecting Basic Memory frontmatter into unrelated
markdown files.

The existing guards only skipped a project when get_project_mode()
returned CLOUD. But ProjectEntry.mode defaults to LOCAL, so an empty- or
relative-path entry without an explicit mode slipped through, and
Path("") resolves against the current working directory.

Gate local sync and watching on the path itself: any project whose path
is not absolute is excluded, regardless of mode. Legitimate local
projects are always resolved to absolute paths at creation, and cloud
projects with a real local bisync copy keep their absolute path and are
still synced/watched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-06-12 18:02:18 -05:00
32 changed files with 567 additions and 867 deletions
+2 -2
View File
@@ -6,14 +6,14 @@
},
"metadata": {
"description": "Official Basic Memory plugins from the canonical basic-memory repository",
"version": "0.22.0"
"version": "0.22.1"
},
"plugins": [
{
"name": "basic-memory",
"source": "./plugins/claude-code",
"description": "The bridge between Claude's working memory and Basic Memory's durable knowledge graph \u2014 session briefings, pre-compaction checkpoints, and capture reflexes",
"version": "0.22.0",
"version": "0.22.1",
"author": {
"name": "Basic Machines"
},
+16 -14
View File
@@ -116,22 +116,24 @@ After PyPI release is published, update the MCP registry:
#### Website Updates
**1. basicmachines.co** (sibling `basicmachines.co` repo)
- **Goal**: Update version number displayed on the homepage
- **Location**: Search for "Basic Memory v0." in the codebase to find version displays
- **What to update**:
- Hero section heading that shows "Basic Memory v{VERSION}"
- "What's New in v{VERSION}" section heading
- Feature highlights array (look for array of features with title/description)
- **Process**:
**1. basicmemory.com** (sibling `basicmemory.com` repo
`basicmachines-co/basicmemory.com`, formerly `basicmachines.co`)
- **No version bump needed.** The marketing site is an Astro + React app and
carries **no hardcoded Basic Memory version number** anywhere in its UI
(`hero.tsx` and the rest of the site have no version string). The old
instruction to bump `src/components/sections/hero.tsx` is obsolete — that
file no longer holds a version. Release announcements are dated blog posts,
not an in-place edit.
- **Skip entirely for patch releases.**
- **Significant releases only — optional announcement post**:
1. Pull latest from GitHub: `git pull origin main`
2. Create release branch: `git checkout -b release/v{VERSION}`
3. Search codebase for current version number (e.g., "v0.16.1")
4. Update version numbers to new release version
5. Update feature highlights with 3-5 key features from this release (extract from CHANGELOG.md)
6. Commit changes: `git commit -m "chore: update to v{VERSION}"`
7. Push branch: `git push origin release/v{VERSION}`
- **Deploy**: Follow deployment process for basicmachines.co
3. Add a dated post under `src/content/blog/` modeled on an existing
release post (e.g. `basic-memory-v0-19-0-release.md`), summarizing 35
headline features from `CHANGELOG.md`
4. Commit (`git commit -s -m "..."`), push, and open a PR against
`basicmachines-co/basicmemory.com`
- **Deploy**: follow that repo's deployment process.
**2. docs.basicmemory.com** (sibling `docs.basicmemory.com` repo)
- **Goal**: Add a What's New page for the release and bump the homepage badge
+6 -1
View File
@@ -322,7 +322,12 @@ The recipe runs `just lint` + `just typecheck`, then updates every release manif
**Post-release tasks** the recipe surfaces but doesn't run:
- `docs.basicmemory.com` — add a What's New page under `content/2.whats-new/` and bump the version badge in `content/index.md` (the changelog page auto-fetches GitHub releases; see that repo's CLAUDE.md version-bump checklist)
- `basicmachines.co`bump version in `src/components/sections/hero.tsx`
- `basicmemory.com`the marketing site (Astro + React, repo
`basicmachines-co/basicmemory.com`, formerly `basicmachines.co`) carries **no
hardcoded version number** in its UI, so there is nothing to bump. For a
significant release, optionally add a dated announcement post under
`src/content/blog/` (model it on an existing `basic-memory-vX-Y-Z-release.md`).
Skip entirely for routine patch releases.
- MCP Registry — `mcp-publisher publish` from the repo root
See `.claude/commands/release/release.md` (and `beta.md`, `release-check.md`, `changelog.md` alongside it) for the full release + post-release runbook, including the slash commands.
+44
View File
@@ -1,5 +1,49 @@
# CHANGELOG
## v0.22.1 (2026-06-12)
Follow-up patch to v0.22.0. Fixes project and default-project resolution on
fresh installs, MCP workspace routing, sync project selection, and CLI startup
latency, plus a few MCP parity additions.
### Features
- Added a `workspace` parameter to `write_note` for parity with `edit_note`.
- **#826**: Added `title` and `tags` annotations to all MCP tool decorators
(phase 1).
- **#930**: `search_notes` now comma-splits `note_types`, `entity_types`, and
`categories`.
- **#971**: Added the manual-pages flow — manpage seed schema, flow docs, and
verification fixes.
### Bug Fixes
- Fresh installs no longer fail when the projects table is empty: resolve now
points them at project setup, the first project is promoted to default when
the config default is missing from the database, the promoted default state
is returned from the project-create API, and a default can be set when none
is currently set. An existing database default is preserved when repairing a
missing config default.
- **#949**: Sync skips projects without an absolute local path and excludes
orphan DB projects that are absent from config.
- **#952 / #981**: Resolved workspace display names and tenant ids in qualified
project routes, closing out the manual verification findings.
- `note_types`/`entity_types`/`categories` are normalized on the direct-call
path, with non-string list elements rejected.
- Vector-search hydration keys on `(type, id)` to prevent id collisions.
- `file_utils` requires line-anchored frontmatter fences.
- CLI startup is faster: FastAPI and app imports are deferred out of CLI
startup, and rich/typer modules are preloaded before an in-place upgrade.
- `config.json` is written atomically.
- In-memory SQLite sessions are serialized so concurrent rollbacks cannot
destroy writes.
### Maintenance
- Release recipes route through PRs and wait for the release PR merge to land
before tagging; the release runbook is refreshed and stripped of
user-specific absolute paths.
## v0.22.0 (2026-06-11)
Team-safe cloud sync. New additive `bm cloud push` and `bm cloud pull`
+1
View File
@@ -6,6 +6,7 @@
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
![](https://badge.mcpx.dev?type=server 'MCP Server')
![](https://badge.mcpx.dev?type=dev 'MCP Dev')
[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/basicmachines-co/basic-memory)
## Skip the install — try Basic Memory in the cloud
-70
View File
@@ -107,76 +107,6 @@ All settings are fields on `BasicMemoryConfig` and can be set via environment va
| `semantic_embedding_document_input_type` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_DOCUMENT_INPUT_TYPE` | Auto for known LiteLLM models | Optional LiteLLM `input_type` for indexed document/passages. |
| `semantic_embedding_query_input_type` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_QUERY_INPUT_TYPE` | Auto for known LiteLLM models | Optional LiteLLM `input_type` for search queries. |
| `semantic_vector_k` | `BASIC_MEMORY_SEMANTIC_VECTOR_K` | `100` | Candidate count for vector nearest-neighbour retrieval. Higher values improve recall at the cost of latency. |
| `search_entity_boost_enabled` | `BASIC_MEMORY_SEARCH_ENTITY_BOOST_ENABLED` | `false` | Enable the entity-aware ranking boost in hybrid search (see below). Default off: benchmark-validated as inert on LoCoMo and prone to Title-Case false positives. |
| `search_entity_boost_weight` | `BASIC_MEMORY_SEARCH_ENTITY_BOOST_WEIGHT` | `0.15` | Per-matched-term multiplier strength for the entity boost. A candidate matching N query entity terms is scaled by `1 + weight * min(N, max_terms)`. |
| `search_entity_boost_max_terms` | `BASIC_MEMORY_SEARCH_ENTITY_BOOST_MAX_TERMS` | `3` | Maximum number of distinct matched entity terms that contribute to the boost, bounding the multiplier. |
## Entity-Aware Ranking Boost
Hybrid search fuses keyword (FTS) and vector similarity, but proper nouns in a query
carry no special weight against generic semantic similarity. As a result, a document
about a *different* entity on the same topic can outrank the document that actually
names the queried entity — e.g. "What are Joanna's hobbies?" surfacing a generic
hobbies note ahead of Joanna's note (see
[#951](https://github.com/basicmachines-co/basic-memory/issues/951)).
When `search_entity_boost_enabled=true`, hybrid retrieval performs a final,
lexical-only re-scoring pass:
1. It extracts candidate entity terms from the query — capitalized / proper-noun
tokens that are not common stopwords (e.g. `Joanna`, `Anthony`, `NASA`).
2. For each fused candidate, it counts how many distinct query entity terms appear in
the candidate's entity name (its title) or in a relation row's linked entity names.
3. Matching candidates have their fused score multiplied by
`1 + weight * min(matches, max_terms)`, so an entity-matching document can be
promoted above a higher-similarity non-matching one.
The boost adds **no model inference** — it is pure index/lexical lookup, so per-query
latency overhead is trivial. It only affects `hybrid` retrieval; `text` and `vector`
modes are unchanged. Non-matching candidates keep their original scores, so ordering
among them is preserved.
```bash
export BASIC_MEMORY_SEARCH_ENTITY_BOOST_ENABLED=true
# Optional tuning:
export BASIC_MEMORY_SEARCH_ENTITY_BOOST_WEIGHT=0.15
export BASIC_MEMORY_SEARCH_ENTITY_BOOST_MAX_TERMS=3
```
> **Default off.** This setting is disabled by default. See the benchmark
> findings below for why the default stays off and where the boost helps.
### Benchmark findings
The boost was benchmarked against LoCoMo (the
[basic-memory-benchmarks](https://github.com/basicmachines-co/basic-memory-benchmarks)
retrieval suite, hybrid mode) and a hand-built adversarial corpus. Two results
drove the decision to keep the default **off** and leave the weight at `0.15`:
1. **LoCoMo is insensitive to the boost.** Sweeping the weight across
`0.15, 0.3, 0.5, 1.0, 2.0` produced *identical* recall@5, recall@10, MRR, and
content-hit at every point — no query reordered, no score changed. LoCoMo's
documents are titled by conversation/session id and expose speaker names only
in body text, never as entity titles or relation names. Because the boost
matches query proper nouns against a candidate's **title or linked relation
names**, it never fires on this corpus. LoCoMo therefore provides no signal to
raise the weight, and the boost neither helps nor harms it.
2. **A capitalization-only heuristic has false positives.** On a corpus where
entity terms appear in titles, the boost correctly promotes the right document
for clean proper nouns (e.g. `Katze`) and is correctly inert on
lowercase-leading identifiers (e.g. `getUserById`, ignored). But **Title-Case
queries can regress**: a query like `What Is The Plan For Q3` extracts `Q3` as
an entity term, and even at weight `0.15` it promotes a document that
*literally* contains "Q3" above the more relevant document that says "third
quarter". Since entity detection is lexical (capitalization, no NER), any
capitalized non-entity token in a query is a potential false positive.
**Guidance.** Enable the boost only on entity-heavy corpora where your queries
name entities that are themselves note titles or linked relations (the #951
"Joanna" case). Prefer natural-case queries (`What are Joanna's hobbies?`) over
Title-Cased phrasing, which can inject spurious entity terms. Leave it off for
conversational / body-text-keyed corpora like LoCoMo, where it cannot help.
## Embedding Providers
+1 -1
View File
@@ -38,7 +38,7 @@ from typing import Any, Callable
from agent.memory_provider import MemoryProvider
from tools.registry import tool_error
__version__ = "0.22.0"
__version__ = "0.22.1"
logger = logging.getLogger("hermes.memory.basic-memory")
+1 -1
View File
@@ -1,5 +1,5 @@
name: basic-memory
version: 0.22.0
version: 0.22.1
description: "Basic Memory — persistent knowledge graph backed by the basic-memory MCP server"
pip_dependencies:
- mcp
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@basicmemory/openclaw-basic-memory",
"version": "0.22.0",
"version": "0.22.1",
"type": "module",
"main": "./dist/index.js",
"types": "./dist/index.d.ts",
+4 -2
View File
@@ -468,7 +468,8 @@ release version:
echo "📝 REMINDER: Post-release tasks:"
echo " 1. docs.basicmemory.com - Add a What's New page under content/2.whats-new/"
echo " and bump the badge in content/index.md (see that repo's CLAUDE.md)"
echo " 2. basicmachines.co - Update version in src/components/sections/hero.tsx"
echo " 2. basicmemory.com - No version number in the site UI; for a significant"
echo " release optionally add a post under src/content/blog/. Skip for patches."
echo " 3. MCP Registry - Run: mcp-publisher publish"
echo " See: .claude/commands/release/release.md for detailed instructions"
@@ -585,7 +586,8 @@ beta version:
echo "📝 REMINDER: For stable releases, update documentation sites:"
echo " 1. docs.basicmemory.com - Add a What's New page under content/2.whats-new/"
echo " and bump the badge in content/index.md (see that repo's CLAUDE.md)"
echo " 2. basicmachines.co - Update version in src/components/sections/hero.tsx"
echo " 2. basicmemory.com - No version number in the site UI; for a significant"
echo " release optionally add a post under src/content/blog/. Skip for patches."
echo " See: .claude/commands/release/release.md for detailed instructions"
# List all available recipes
@@ -6,14 +6,14 @@
},
"metadata": {
"description": "Official plugins from Basic Machines for knowledge management and AI-assisted development",
"version": "0.22.0"
"version": "0.22.1"
},
"plugins": [
{
"name": "basic-memory",
"source": "./",
"description": "The bridge between Claude's working memory and Basic Memory's durable knowledge graph \u2014 session briefings, pre-compaction checkpoints, and capture reflexes",
"version": "0.22.0",
"version": "0.22.1",
"author": {
"name": "Basic Machines"
},
@@ -1,7 +1,7 @@
{
"name": "basic-memory",
"description": "The bridge between Claude's working memory and Basic Memory's durable knowledge graph \u2014 session briefings, pre-compaction checkpoints, and capture reflexes",
"version": "0.22.0",
"version": "0.22.1",
"author": {
"name": "Basic Machines"
},
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "codex",
"version": "0.22.0",
"version": "0.22.1",
"description": "A Codex-native bridge to Basic Memory for durable engineering context, decisions, and resumable checkpoints.",
"author": {
"name": "Basic Machines",
+2 -2
View File
@@ -6,12 +6,12 @@
"url": "https://github.com/basicmachines-co/basic-memory.git",
"source": "github"
},
"version": "0.22.0",
"version": "0.22.1",
"packages": [
{
"registryType": "pypi",
"identifier": "basic-memory",
"version": "0.22.0",
"version": "0.22.1",
"runtimeHint": "uvx",
"runtimeArguments": [
{
+1 -1
View File
@@ -1,7 +1,7 @@
"""basic-memory - Local-first knowledge management combining Zettelkasten with knowledge graphs"""
# Package version - updated by release automation
__version__ = "0.22.0"
__version__ = "0.22.1"
# API version for FastAPI - independent of package version
__api_version__ = "v0"
+20 -36
View File
@@ -346,42 +346,6 @@ class BasicMemoryConfig(BaseSettings):
"Valid values: text, vector, hybrid. "
"When unset, defaults to 'hybrid' if semantic search is enabled, otherwise 'text'.",
)
# Entity-aware ranking boost (hybrid retrieval).
# Trigger: proper nouns in a query (e.g. "Joanna") carry no extra weight against
# generic semantic similarity, so documents from the wrong conversation can outrank
# the gold document during hybrid fusion (#951).
# Why: entities are first-class in Basic Memory, so a candidate whose title or linked
# relation names contain a query proper noun is a stronger answer than a same-topic
# document about a different entity.
# Outcome: when enabled, hybrid fusion multiplies a candidate's fused score by a small
# bonus for each distinct query entity term it matches lexically (no model inference).
# Default OFF: LoCoMo benchmarking showed the boost is inert there (its docs are keyed
# by session id, not entity titles) and an adversarial check found Title-Case queries
# can inject spurious entity terms (e.g. "Q3") that regress ranking. See
# docs/semantic-search.md "Benchmark findings".
search_entity_boost_enabled: bool = Field(
default=False,
description="Enable entity-aware ranking boost in hybrid search. When enabled, "
"hybrid candidates whose title or linked relation names contain a proper-noun "
"term from the query are boosted in the final ranking. Lexical-only; adds no "
"model inference. Default off: benchmark-validated as inert on LoCoMo and prone "
"to Title-Case false positives (see docs/semantic-search.md).",
)
search_entity_boost_weight: float = Field(
default=0.15,
description="Per-matched-term multiplier strength for the entity-aware ranking "
"boost. A candidate matching N distinct query entity terms has its fused score "
"multiplied by (1 + weight * N), capped at search_entity_boost_max_terms terms. "
"Only applies when search_entity_boost_enabled is true.",
ge=0.0,
)
search_entity_boost_max_terms: int = Field(
default=3,
description="Maximum number of distinct matched entity terms that contribute to "
"the entity-aware ranking boost, bounding the multiplier so a single candidate "
"cannot run away with the ranking.",
gt=0,
)
# Database connection pool configuration (Postgres only)
db_pool_size: int = Field(
@@ -694,6 +658,26 @@ class BasicMemoryConfig(BaseSettings):
entry = self.projects.get(project_name)
return entry.mode if entry else ProjectMode.CLOUD
def is_locally_syncable(self, project_name: str, project_path: str) -> bool:
"""Whether a project should be synced/watched on the local filesystem.
Both conditions are required (issue #949):
* The project is present in config. Config is the source of truth, so a
stale database row that was removed from config — but whose deletion
has not yet been reconciled, or whose reconciliation failed — must
not be synced even though it still has a real directory on disk.
* Its path is absolute. An empty or relative path resolves against the
process cwd, so syncing it would adopt whatever directory the server
was launched from as the project root and mutate unrelated files.
Cloud-only projects (empty/slug path) and cloud projects with a real
local bisync copy (absolute path) are handled correctly by these two
conditions, so no separate mode check is needed.
"""
entry = self.projects.get(project_name)
return entry is not None and Path(project_path).is_absolute()
def set_project_mode(self, project_name: str, mode: ProjectMode) -> None:
"""Set the routing mode for a project.
@@ -18,6 +18,7 @@ from basic_memory.repository.search_index_row import SearchIndexRow
from basic_memory.repository.search_repository_base import (
SearchRepositoryBase,
VectorChunkState,
relaxed_query_words,
)
from basic_memory.repository.metadata_filters import parse_metadata_filters
from basic_memory.repository.semantic_errors import SemanticDependenciesMissingError
@@ -67,9 +68,6 @@ class PostgresSearchRepository(SearchRepositoryBase):
self._semantic_postgres_prepare_concurrency = (
self._app_config.semantic_postgres_prepare_concurrency
)
self._entity_boost_enabled = self._app_config.search_entity_boost_enabled
self._entity_boost_weight = self._app_config.search_entity_boost_weight
self._entity_boost_max_terms = self._app_config.search_entity_boost_max_terms
self._embedding_provider = embedding_provider
self._vector_dimensions = 384
self._vector_tables_initialized = False
@@ -179,6 +177,14 @@ class PostgresSearchRepository(SearchRepositoryBase):
# For non-Boolean queries, prepare single term
return self._prepare_single_term(term, is_prefix)
@staticmethod
def _relaxed_tsquery_text(search_text: Optional[str]) -> Optional[str]:
"""OR-relaxed tsquery expression for a failed strict query, or None."""
words = relaxed_query_words(search_text)
if not words:
return None
return " | ".join(f"{word}:*" for word in words)
def _prepare_boolean_query(self, query: str) -> str:
"""Convert Boolean query to tsquery format.
@@ -239,7 +245,12 @@ class PostgresSearchRepository(SearchRepositoryBase):
# Handle multi-word queries
if " " in cleaned_term:
words = [w for w in cleaned_term.split() if w.strip()]
# Strip sentence punctuation from word edges so question-form
# queries produce clean lexemes (parity with SQLite FTS5 prep).
# The tsquery tokenizer ignores this punctuation anyway; leaving it
# in only risks tsquery syntax errors. Interior characters are kept.
words = [w.strip("?!.,;") for w in cleaned_term.split()]
words = [w for w in words if w]
if not words:
# All characters were special chars, search won't match anything
# Return a safe search term that won't cause syntax errors
@@ -252,8 +263,11 @@ class PostgresSearchRepository(SearchRepositoryBase):
# Join with AND operator
return " & ".join(prepared_words)
# Single word
cleaned_term = cleaned_term.strip()
# Single word: strip edge punctuation; guard the now-empty case so a
# bare ":*"/"" never reaches tsquery.
cleaned_term = cleaned_term.strip().strip("?!.,;")
if not cleaned_term:
return "NOSPECIALCHARS:*"
if is_prefix:
return f"{cleaned_term}:*"
else:
@@ -911,6 +925,7 @@ class PostgresSearchRepository(SearchRepositoryBase):
min_similarity: Optional[float] = None,
limit: int = 10,
offset: int = 0,
allow_relaxed: bool = False,
) -> List[SearchIndexRow]:
"""Search across all indexed content using PostgreSQL tsvector."""
# --- Dispatch vector / hybrid modes (shared logic) ---
@@ -985,6 +1000,20 @@ class PostgresSearchRepository(SearchRepositoryBase):
async with db.scoped_session(self.session_maker) as session:
result = await session.execute(text(sql), params)
rows = result.fetchall()
# Trigger: multi-word natural-language query matched nothing
# under the default all-terms-AND tsquery semantics.
# Why: questions rarely have every word in one document;
# without relaxation the FTS half of hybrid search contributes
# zero candidates (parity with the SQLite path).
# Outcome: one retry with OR-joined prefix lexemes; ts_rank
# still ranks multi-term matches first.
relaxed = (
self._relaxed_tsquery_text(search_text) if allow_relaxed and not rows else None
)
if relaxed and params.get("text"):
params["text"] = relaxed
result = await session.execute(text(sql), params)
rows = result.fetchall()
except Exception as e:
if self._is_tsquery_syntax_error(e):
logger.warning(f"tsquery syntax error for search term: {search_text}, error: {e}")
@@ -40,69 +40,55 @@ BULLET_PATTERN = re.compile(r"^[\-\*]\s+")
OVERSIZED_ENTITY_VECTOR_SHARD_SIZE = 256
_SQLITE_MAX_PREPARE_WINDOW = 8
# Interrogative/function words contribute lexical noise when a strict
# full-text query is relaxed: "when OR did OR a" matches loud wrong documents
# that displace genuine results from the ranking window.
RELAXATION_STOPWORDS = frozenset(
"a an and are as at be but by did do does for from had has have how i in is it of on "
"or that the their they this to was we were what when where which who whom whose why "
"will with you your".split()
)
def relaxed_query_words(search_text: Optional[str]) -> Optional[list[str]]:
"""Content-bearing words for OR-relaxing a strict full-text query.
Returns None when relaxation must not apply. These eligibility rules match
SearchService._is_relaxed_fts_fallback_eligible so the hybrid FTS branch
relaxes exactly the same query shapes as the service-level FTS path:
- empty / quoted / explicit-boolean queries (user intent is not
second-guessed);
- fewer than three alphanumeric tokens (short queries like "New Feature"
over-broaden under OR — and in hybrid the relaxed FTS-only rows normalize
to 1.0 and can outrank the vector result the user wanted);
- any pure-digit token ("root note 1", "SPEC 16") — identifier-like queries
over-broaden and create false positives under OR.
"""
if not search_text:
return None
stripped = search_text.strip()
if '"' in stripped or any(op in f" {stripped} " for op in (" AND ", " OR ", " NOT ")):
return None
# Eligibility checks run on raw alphanumeric tokens (parity with the
# service), before stopword filtering.
tokens = re.findall(r"[A-Za-z0-9]+", stripped.lower())
if len(tokens) < 3 or any(token.isdigit() for token in tokens):
return None
words = [word.strip("?!.,;:") for word in stripped.split()]
words = [
word
for word in words
if word and word.isalnum() and word.lower() not in RELAXATION_STOPWORDS
]
return words or None
# Entity, observation, and relation rows in search_index carry ids from independent
# auto-increment sequences, so a bare id is ambiguous across row types. Every map in
# the vector/hybrid retrieval path must key rows by (type, id) to avoid collisions.
type SearchIndexKey = tuple[str, int]
# --- Entity-aware ranking boost (#951) ---
# Match word tokens (allowing internal apostrophes/hyphens) so we can inspect
# their capitalization to detect proper-noun-like query terms.
_ENTITY_TERM_TOKEN_PATTERN = re.compile(r"[A-Za-z][A-Za-z'\-]*")
# Common capitalized sentence-starters and interrogatives that look like proper
# nouns but are not entity references. Kept lowercase for case-insensitive checks.
# Intentionally small: a candidate term only boosts a row when it actually matches
# that row's title/relation names, so a stray non-entity term simply does nothing.
_ENTITY_TERM_STOPWORDS = frozenset(
{
"a",
"an",
"and",
"are",
"as",
"at",
"be",
"but",
"by",
"do",
"does",
"for",
"from",
"has",
"have",
"how",
"i",
"in",
"is",
"it",
"of",
"on",
"or",
"the",
"their",
"they",
"this",
"to",
"was",
"we",
"were",
"what",
"when",
"where",
"which",
"who",
"whom",
"whose",
"why",
"will",
"with",
"you",
"your",
}
)
@dataclass
class VectorSyncBatchResult:
@@ -224,13 +210,6 @@ class SearchRepositoryBase(ABC):
_vector_dimensions: int
_vector_tables_initialized: bool
# Entity-aware ranking boost (#951). Defaults keep the feature off for any
# subclass or test double that does not explicitly configure it. Concrete
# backends overwrite these from BasicMemoryConfig in their __init__.
_entity_boost_enabled: bool = False
_entity_boost_weight: float = 0.0
_entity_boost_max_terms: int = 1
def __init__(self, session_maker: async_sessionmaker[AsyncSession], project_id: int):
"""Initialize with session maker and project_id filter.
@@ -294,6 +273,7 @@ class SearchRepositoryBase(ABC):
min_similarity: Optional[float] = None,
limit: int = 10,
offset: int = 0,
allow_relaxed: bool = False,
) -> List[SearchIndexRow]:
"""Search across all indexed content.
@@ -2212,105 +2192,6 @@ class SearchRepositoryBase(ABC):
# Shared semantic search: hybrid score-based fusion
# ------------------------------------------------------------------
# --- Entity-aware ranking boost (#951) ---
@staticmethod
def _extract_query_entity_terms(search_text: Optional[str]) -> set[str]:
"""Extract candidate entity (proper-noun) terms from a query string.
Heuristic, lexical only (no model inference): a token is a candidate entity
term when it is title-cased or all-caps and is not a common stopword. The
result is lowercased so downstream matching is case-insensitive.
Examples:
"What are Joanna's hobbies?" -> {"joanna"}
"Who is Anthony?" -> {"anthony"}
"Deborah and Jolene" -> {"deborah", "jolene"}
"what is the weather" -> set() (no proper nouns)
"""
if not search_text:
return set()
terms: set[str] = set()
for match in _ENTITY_TERM_TOKEN_PATTERN.finditer(search_text):
token = match.group(0)
# Trigger: token begins with an uppercase letter (Title-Case or ALL-CAPS).
# Why: proper nouns and named entities are conventionally capitalized; this
# is the cheapest reliable signal without a NER model.
# Outcome: lowercase, non-capitalized words are ignored as generic terms.
if not token[0].isupper():
continue
normalized = token.lower()
# Strip a trailing possessive so "Joanna's" matches the entity "Joanna".
if normalized.endswith("'s"):
normalized = normalized[:-2]
if normalized in _ENTITY_TERM_STOPWORDS:
continue
# Single characters (e.g. a stray "I") carry no entity signal.
if len(normalized) < 2:
continue
terms.add(normalized)
return terms
@staticmethod
def _row_entity_match_count(row: SearchIndexRow, entity_terms: set[str]) -> int:
"""Count distinct query entity terms that a candidate row references.
Matches against the row's own entity name (title) and the names embedded in
a relation row's title (``"From -> To"``). These are the fields where Basic
Memory's first-class entity names surface, so a match here is strong evidence
the candidate is about the queried entity rather than a same-topic document.
"""
if not entity_terms:
return 0
haystack_parts = [row.title or ""]
# Relation rows encode linked entity names in their title ("From -> To");
# the relation_type itself is not an entity name, so it is excluded.
haystack = " ".join(part for part in haystack_parts if part)
if not haystack:
return 0
haystack_tokens: set[str] = set()
for match in _ENTITY_TERM_TOKEN_PATTERN.finditer(haystack):
token = match.group(0).lower()
# Mirror the query-side possessive stripping so a doc titled
# "Joanna's Hobbies" matches the query entity term "joanna".
if token.endswith("'s"):
token = token[:-2]
haystack_tokens.add(token)
return len(entity_terms & haystack_tokens)
def _apply_entity_boost(
self,
fused_scores: dict[SearchIndexKey, float],
rows_by_key: dict[SearchIndexKey, SearchIndexRow],
entity_terms: set[str],
) -> dict[SearchIndexKey, float]:
"""Multiply fused scores by a per-matched-term bonus for entity-matching rows.
Trigger: entity boosting is enabled and the query contains proper-noun terms.
Why: a candidate whose entity/relation names contain a queried proper noun is a
stronger answer than a generic same-topic document (#951 cross-conversation
confusion).
Outcome: ``score * (1 + weight * min(matches, max_terms))``. Rows that match no
query entity term are returned unchanged, so relative order among non-matching
rows is preserved.
"""
if not self._entity_boost_enabled or not entity_terms or self._entity_boost_weight <= 0:
return fused_scores
boosted: dict[SearchIndexKey, float] = {}
for row_key, score in fused_scores.items():
row = rows_by_key.get(row_key)
matches = self._row_entity_match_count(row, entity_terms) if row is not None else 0
if matches <= 0:
boosted[row_key] = score
continue
capped_matches = min(matches, self._entity_boost_max_terms)
boosted[row_key] = score * (1.0 + self._entity_boost_weight * capped_matches)
return boosted
async def _search_hybrid(
self,
*,
@@ -2338,6 +2219,9 @@ class SearchRepositoryBase(ABC):
query_start = time.perf_counter()
candidate_limit = max(self._semantic_vector_k, (limit + offset) * 10)
fts_start = time.perf_counter()
# allow_relaxed: question-form queries rarely AND-match, and a dead FTS
# branch silently degrades hybrid to vector-only ranking. Fusion plus
# bm25 keep relaxed lexical candidates from dominating precision.
fts_results = await self.search(
search_text=search_text,
permalink=permalink,
@@ -2351,6 +2235,7 @@ class SearchRepositoryBase(ABC):
retrieval_mode=SearchRetrievalMode.FTS,
limit=candidate_limit,
offset=0,
allow_relaxed=True,
)
fts_ms = (time.perf_counter() - fts_start) * 1000
vector_start = time.perf_counter()
@@ -2414,15 +2299,6 @@ class SearchRepositoryBase(ABC):
f = fts_scores.get(row_key, 0.0)
fused_scores[row_key] = max(v, f) + FUSION_BONUS * min(v, f)
# Entity-aware ranking boost (#951): runs over the full fused candidate set
# before the limit/offset cut, so a boosted entity-matching candidate can be
# promoted into the returned window. No-op when the feature is disabled or the
# query contains no proper-noun terms, preserving the existing ordering.
entity_terms = (
self._extract_query_entity_terms(query_text) if self._entity_boost_enabled else set()
)
fused_scores = self._apply_entity_boost(fused_scores, rows_by_key, entity_terms)
ranked = sorted(fused_scores.items(), key=lambda item: item[1], reverse=True)
output: list[SearchIndexRow] = []
for row_key, fused_score in ranked[offset : offset + limit]:
@@ -23,7 +23,10 @@ from basic_memory.models.search import (
from basic_memory.repository.embedding_provider import EmbeddingProvider
from basic_memory.repository.embedding_provider_factory import create_embedding_provider
from basic_memory.repository.search_index_row import SearchIndexRow
from basic_memory.repository.search_repository_base import SearchRepositoryBase
from basic_memory.repository.search_repository_base import (
SearchRepositoryBase,
relaxed_query_words,
)
from basic_memory.repository.metadata_filters import parse_metadata_filters, build_sqlite_json_path
from basic_memory.repository.semantic_errors import SemanticDependenciesMissingError
from basic_memory.schemas.search import SearchItemType, SearchRetrievalMode
@@ -55,9 +58,6 @@ class SQLiteSearchRepository(SearchRepositoryBase):
self._semantic_embedding_sync_batch_size = (
self._app_config.semantic_embedding_sync_batch_size
)
self._entity_boost_enabled = self._app_config.search_entity_boost_enabled
self._entity_boost_weight = self._app_config.search_entity_boost_weight
self._entity_boost_max_terms = self._app_config.search_entity_boost_max_terms
self._embedding_provider = embedding_provider
self._sqlite_vec_load_lock = asyncio.Lock()
self._sqlite_prepare_write_lock = asyncio.Lock()
@@ -258,6 +258,19 @@ class SQLiteSearchRepository(SearchRepositoryBase):
if "*" in term and all(c.isalnum() or c in "*_-" for c in term):
return term
# Natural-language queries arrive with sentence punctuation that FTS5
# treats as syntax ("When did Melanie paint a sunrise?"). The tokenizer
# ignores this punctuation in the INDEX, so stripping it from word
# edges loses nothing — but leaving it forces the whole question into
# an exact-phrase match that returns zero rows, silently disabling the
# FTS half of hybrid search. Interior characters (hyphens, slashes —
# permalinks and paths) are untouched.
if " " in term:
words = [word.strip("?!.,;:") for word in term.split()]
term = " ".join(word for word in words if word)
if not term:
return ""
# Characters that can cause FTS5 syntax errors when used as operators
# We're more conservative here - only quote when we detect problematic patterns
problematic_chars = [
@@ -354,6 +367,14 @@ class SQLiteSearchRepository(SearchRepositoryBase):
# For non-Boolean queries, use the single term preparation logic
return self._prepare_single_term(term, is_prefix)
@staticmethod
def _relaxed_fts_text(search_text: Optional[str]) -> Optional[str]:
"""OR-relaxed FTS5 expression for a failed strict query, or None."""
words = relaxed_query_words(search_text)
if not words:
return None
return " OR ".join(f"{word}*" for word in words)
# ------------------------------------------------------------------
# sqlite-vec extension loading (SQLite-specific)
# ------------------------------------------------------------------
@@ -956,8 +977,15 @@ class SQLiteSearchRepository(SearchRepositoryBase):
min_similarity: Optional[float] = None,
limit: int = 10,
offset: int = 0,
allow_relaxed: bool = False,
) -> List[SearchIndexRow]:
"""Search across all indexed content using SQLite FTS5."""
"""Search across all indexed content using SQLite FTS5.
``allow_relaxed=True`` retries a zero-result strict multi-word query
with OR-joined content terms. Only the hybrid path opts in: its FTS
branch otherwise contributes nothing for question-form queries.
Service-level FTS searches keep their own conservative fallback.
"""
# --- Dispatch vector / hybrid modes (shared logic) ---
dispatched = await self._dispatch_retrieval_mode(
search_text=search_text,
@@ -1024,6 +1052,21 @@ class SQLiteSearchRepository(SearchRepositoryBase):
async with db.scoped_session(self.session_maker) as session:
result = await session.execute(text(sql), params)
rows = result.fetchall()
# Trigger: multi-word natural-language query matched nothing
# under the default all-terms-AND semantics.
# Why: questions ("when did X do Y") rarely have every word in
# one document; without relaxation the FTS half of hybrid
# search contributes zero candidates and ranking degrades to
# vector-only.
# Outcome: one retry with OR-joined prefix terms; bm25 still
# ranks multi-term matches first.
relaxed = (
self._relaxed_fts_text(search_text) if allow_relaxed and not rows else None
)
if relaxed and params.get("text"):
params["text"] = relaxed
result = await session.execute(text(sql), params)
rows = result.fetchall()
except Exception as e:
# Handle FTS5 syntax errors and provide user-friendly feedback
if self._is_fts5_syntax_error(e): # pragma: no cover
+9 -13
View File
@@ -13,7 +13,7 @@ from pathlib import Path
from loguru import logger
from basic_memory import db
from basic_memory.config import BasicMemoryConfig, DatabaseBackend, ProjectMode
from basic_memory.config import BasicMemoryConfig, DatabaseBackend
from basic_memory.models import Project
from basic_memory.repository import (
ProjectRepository,
@@ -120,18 +120,14 @@ async def initialize_file_sync(
active_projects = [p for p in active_projects if p.name == constrained_project]
logger.info(f"Background sync constrained to project: {constrained_project}")
# Skip cloud-mode projects that have no local directory.
# Cloud projects with a local bisync copy (absolute path) are kept for local sync.
cloud_skip = []
for p in active_projects:
if app_config.get_project_mode(p.name) == ProjectMode.CLOUD:
entry = app_config.projects.get(p.name)
if entry and Path(entry.path).is_absolute():
continue # Cloud project with local bisync copy — keep for local sync
cloud_skip.append(p.name)
if cloud_skip:
active_projects = [p for p in active_projects if p.name not in cloud_skip]
logger.info(f"Skipping cloud-mode projects for local sync: {cloud_skip}")
# Only sync projects that are in config (source of truth) and have an
# absolute local path; see BasicMemoryConfig.is_locally_syncable. This keeps
# background sync from adopting the process cwd as a project root and
# mutating unrelated files (issue #949).
skip = [p.name for p in active_projects if not app_config.is_locally_syncable(p.name, p.path)]
if skip:
active_projects = [p for p in active_projects if p.name not in skip]
logger.info(f"Skipping projects that are not locally syncable for sync: {skip}")
# Start sync for all projects as background tasks (non-blocking)
async def sync_project_background(project: Project):
+13 -13
View File
@@ -10,7 +10,7 @@ from typing import List, Optional, Set, Sequence, Callable, Awaitable, TYPE_CHEC
if TYPE_CHECKING:
from basic_memory.sync.sync_service import SyncService
from basic_memory.config import BasicMemoryConfig, ProjectMode, WATCH_STATUS_JSON
from basic_memory.config import BasicMemoryConfig, WATCH_STATUS_JSON
from basic_memory.ignore_utils import load_gitignore_patterns, should_ignore_path
from basic_memory.models import Project
from basic_memory.repository import ProjectRepository
@@ -184,24 +184,24 @@ class WatchService:
``--project``, only that project is watched. This keeps concurrent
MCP processes from producing duplicate watchers that race on the
same files.
2. Cloud-only projects without a local bisync copy are skipped so we
don't watch a path that does not exist on disk.
2. Projects that are not locally syncable are skipped — those missing
from config (config is the source of truth, so stale DB rows must
not be watched) or with a non-absolute path (which would resolve
against the process cwd and make the watcher observe and mutate the
directory the server was launched from). See
``BasicMemoryConfig.is_locally_syncable`` (issue #949). Cloud
projects with a local bisync copy keep their absolute path and are
still watched.
"""
projects = await self.project_repository.get_active_projects()
if self.constrained_project:
projects = [p for p in projects if p.name == self.constrained_project]
cloud_skip: list[str] = []
for p in projects:
if self.app_config.get_project_mode(p.name) == ProjectMode.CLOUD:
entry = self.app_config.projects.get(p.name)
if entry and Path(entry.path).is_absolute():
continue # Cloud project with local bisync copy — keep watching
cloud_skip.append(p.name)
if cloud_skip:
projects = [p for p in projects if p.name not in cloud_skip]
logger.debug(f"Skipping cloud-mode projects in watch cycle: {cloud_skip}")
skip = [p.name for p in projects if not self.app_config.is_locally_syncable(p.name, p.path)]
if skip:
projects = [p for p in projects if p.name not in skip]
logger.debug(f"Skipping projects that are not locally syncable in watch cycle: {skip}")
return list(projects)
@@ -1,226 +0,0 @@
"""Service-level integration test for the entity-aware ranking boost (#951).
Drives a fully wired SearchService over a real database with a deterministic stub
embedding provider so vector similarity is controlled. Verifies that when the boost
is enabled, an entity-matching document outranks a higher-similarity non-matching
document, and that ordering is unchanged when the boost is disabled.
No model inference is involved: the stub provider returns fixed unit vectors, so the
test is fast and deterministic on both SQLite and Postgres.
"""
from __future__ import annotations
import math
from typing import Any
import pytest
import pytest_asyncio
from basic_memory.config import BasicMemoryConfig
from basic_memory.repository.entity_repository import EntityRepository
from basic_memory.repository.search_repository import SearchRepository
from basic_memory.schemas.base import Entity as EntitySchema
from basic_memory.schemas.search import SearchQuery, SearchRetrievalMode
from basic_memory.services.entity_service import EntityService
from basic_memory.services.file_service import FileService
from basic_memory.services.search_service import SearchService
# --- Deterministic stub embedding provider ---
_STUB_DIMENSIONS = 4
def _unit(vector: list[float]) -> list[float]:
norm = math.sqrt(sum(component * component for component in vector)) or 1.0
return [component / norm for component in vector]
class _StubEmbeddingProvider:
"""Maps known text fragments to fixed unit vectors for controlled similarity.
The query is engineered to sit closer (cosine) to the non-matching "hobbies"
document than to the gold "Joanna" document, reproducing the #951 failure where
generic semantic similarity outranks the entity-matching gold doc.
"""
model_name = "stub-entity-boost"
dimensions = _STUB_DIMENSIONS
def _vector_for(self, text: str) -> list[float]:
lowered = text.lower()
if "joanna" in lowered:
# Gold doc: shares some direction with the query but less than the decoy.
return _unit([0.6, 0.8, 0.0, 0.0])
if "hobbies" in lowered or "pastime" in lowered:
# Decoy doc: closest to the query direction.
return _unit([0.95, 0.31, 0.0, 0.0])
return _unit([0.0, 0.0, 1.0, 0.0])
async def embed_query(self, text: str) -> list[float]:
# Query direction is closest to the decoy vector above.
return _unit([0.97, 0.24, 0.0, 0.0])
async def embed_documents(self, texts: list[str]) -> list[list[float]]:
return [self._vector_for(text) for text in texts]
def runtime_log_attrs(self) -> dict[str, Any]:
return {}
# --- Fixtures ---
async def _build_search_service(
*,
session_maker,
test_project,
base_app_config: BasicMemoryConfig,
file_service: FileService,
entity_repository: EntityRepository,
boost_enabled: bool,
) -> SearchService:
"""Build a SearchService with semantic search + a deterministic stub provider."""
from basic_memory.repository.sqlite_search_repository import SQLiteSearchRepository
from basic_memory.repository.postgres_search_repository import PostgresSearchRepository
from basic_memory.config import DatabaseBackend
app_config = base_app_config.model_copy(
update={
"semantic_search_enabled": True,
"semantic_min_similarity": 0.0,
"search_entity_boost_enabled": boost_enabled,
"search_entity_boost_weight": 0.3,
"search_entity_boost_max_terms": 3,
}
)
provider = _StubEmbeddingProvider()
if app_config.database_backend == DatabaseBackend.POSTGRES: # pragma: no cover
search_repo: SearchRepository = PostgresSearchRepository(
session_maker,
project_id=test_project.id,
app_config=app_config,
embedding_provider=provider,
)
else:
# Pass the stub provider at construction time so __init__ does not
# instantiate the real configured provider when semantic_search_enabled=True.
repo = SQLiteSearchRepository(
session_maker,
project_id=test_project.id,
app_config=app_config,
embedding_provider=provider,
)
search_repo = repo
service = SearchService(search_repo, entity_repository, file_service)
await service.init_search_index()
return service
@pytest_asyncio.fixture
async def boost_entities(
entity_service: EntityService,
):
"""Index two entities: a decoy 'hobbies' doc and the gold 'Joanna' doc."""
decoy, _ = await entity_service.create_or_update_entity(
EntitySchema(
title="Common Hobbies and Pastimes",
note_type="note",
directory="people",
content="A general overview of hobbies and pastimes people enjoy.",
)
)
gold, _ = await entity_service.create_or_update_entity(
EntitySchema(
title="Joanna",
note_type="note",
directory="people",
content="Notes about Joanna and what she likes to do.",
)
)
return decoy, gold
# --- Tests ---
async def _sync_vectors(service: SearchService, entity_ids: list[int]) -> None:
"""Embed the indexed entities via the stub provider."""
await service.sync_entity_vectors_batch(entity_ids)
@pytest.mark.asyncio
async def test_entity_boost_enabled_promotes_gold_doc(
session_maker,
test_project,
app_config,
file_service,
entity_repository,
boost_entities,
):
decoy, gold = boost_entities
service = await _build_search_service(
session_maker=session_maker,
test_project=test_project,
base_app_config=app_config,
file_service=file_service,
entity_repository=entity_repository,
boost_enabled=True,
)
# Re-index the entities through this service so vector tables exist for it.
for entity in (decoy, gold):
await service.index_entity(entity)
await _sync_vectors(service, [decoy.id, gold.id])
results = await service.search(
SearchQuery(
text="What are Joanna's hobbies?",
retrieval_mode=SearchRetrievalMode.HYBRID,
),
limit=10,
)
entity_ids = [r.entity_id for r in results]
assert gold.id in entity_ids and decoy.id in entity_ids
# With the boost on, the entity-matching gold doc ranks ahead of the
# higher-similarity decoy.
assert entity_ids.index(gold.id) < entity_ids.index(decoy.id)
@pytest.mark.asyncio
async def test_entity_boost_disabled_keeps_similarity_order(
session_maker,
test_project,
app_config,
file_service,
entity_repository,
boost_entities,
):
decoy, gold = boost_entities
service = await _build_search_service(
session_maker=session_maker,
test_project=test_project,
base_app_config=app_config,
file_service=file_service,
entity_repository=entity_repository,
boost_enabled=False,
)
for entity in (decoy, gold):
await service.index_entity(entity)
await _sync_vectors(service, [decoy.id, gold.id])
results = await service.search(
SearchQuery(
text="What are Joanna's hobbies?",
retrieval_mode=SearchRetrievalMode.HYBRID,
),
limit=10,
)
entity_ids = [r.entity_id for r in results]
assert gold.id in entity_ids and decoy.id in entity_ids
# With the boost off, the higher-similarity decoy ranks ahead of the gold doc.
assert entity_ids.index(decoy.id) < entity_ids.index(gold.id)
+1 -109
View File
@@ -77,6 +77,7 @@ class ConcreteSearchRepo(SearchRepositoryBase):
min_similarity: Optional[float] = None,
limit: int = 10,
offset: int = 0,
allow_relaxed: bool = False,
) -> list[SearchIndexRow]:
return [] # pragma: no cover
@@ -134,115 +135,6 @@ HYBRID_KWARGS: dict[str, Any] = dict(
)
def _hybrid_kwargs(**overrides: Any) -> dict[str, Any]:
"""Return HYBRID_KWARGS with overrides applied, typed as dict[str, Any].
Keeps the splat into the keyword-only _search_hybrid signature type-clean.
"""
merged: dict[str, Any] = {**HYBRID_KWARGS, **overrides}
return merged
@pytest.mark.asyncio
async def test_entity_boost_promotes_matching_doc_when_enabled():
"""With entity boost enabled, an entity-matching doc outranks a higher-similarity
non-matching doc.
Reproduces the #951 cross-conversation confusion: a generic same-topic document
(higher raw similarity) initially outranks the gold doc whose title names the
queried entity. Enabling the boost flips the order.
"""
repo = ConcreteSearchRepo()
repo._entity_boost_enabled = True
repo._entity_boost_weight = 0.15
repo._entity_boost_max_terms = 3
# Row 1: generic hobbies doc from the wrong conversation, higher vector similarity.
# Row 2: the gold doc whose title names the queried entity "Joanna".
fts_results = []
vector_results = [
FakeRow(id=1, score=0.80, title="Hobbies and pastimes"),
FakeRow(id=2, score=0.72, title="Joanna profile"),
]
with (
patch.object(repo, "search", new_callable=AsyncMock, return_value=fts_results),
patch.object(
repo, "_search_vector_only", new_callable=AsyncMock, return_value=vector_results
),
):
results = await repo._search_hybrid(
**_hybrid_kwargs(search_text="What are Joanna's hobbies?")
)
# Boost: row 2 -> 0.72 * 1.15 = 0.828 > row 1's 0.80
assert [r.id for r in results] == [2, 1]
assert results[0].score == pytest.approx(0.72 * 1.15, rel=1e-6)
assert results[1].score == pytest.approx(0.80, rel=1e-6)
@pytest.mark.asyncio
async def test_entity_boost_disabled_preserves_ordering():
"""With entity boost disabled (default), ordering matches pure similarity."""
repo = ConcreteSearchRepo()
# Defaults from the base class keep boosting off; assert explicitly.
assert repo._entity_boost_enabled is False
fts_results = []
vector_results = [
FakeRow(id=1, score=0.80, title="Hobbies and pastimes"),
FakeRow(id=2, score=0.72, title="Joanna profile"),
]
with (
patch.object(repo, "search", new_callable=AsyncMock, return_value=fts_results),
patch.object(
repo, "_search_vector_only", new_callable=AsyncMock, return_value=vector_results
),
):
results = await repo._search_hybrid(
**_hybrid_kwargs(search_text="What are Joanna's hobbies?")
)
# No boost: original similarity order is preserved, scores unchanged.
assert [r.id for r in results] == [1, 2]
assert results[0].score == pytest.approx(0.80, rel=1e-6)
assert results[1].score == pytest.approx(0.72, rel=1e-6)
@pytest.mark.asyncio
async def test_entity_boost_promotes_doc_into_limited_window():
"""Boosting runs before the limit cut, so a matching doc ranked below the cutoff
can be promoted into the returned window."""
repo = ConcreteSearchRepo()
repo._entity_boost_enabled = True
repo._entity_boost_weight = 0.6
repo._entity_boost_max_terms = 3
fts_results = []
# Three non-matching docs above the gold doc, which matches "Anthony".
vector_results = [
FakeRow(id=1, score=0.90, title="conversation six"),
FakeRow(id=2, score=0.85, title="conversation one"),
FakeRow(id=3, score=0.60, title="Anthony introduces himself"),
]
with (
patch.object(repo, "search", new_callable=AsyncMock, return_value=fts_results),
patch.object(
repo, "_search_vector_only", new_callable=AsyncMock, return_value=vector_results
),
):
results = await repo._search_hybrid(
**_hybrid_kwargs(search_text="Who is Anthony?", limit=1)
)
# Gold doc boost: 0.60 * 1.6 = 0.96 > row 1's 0.90, so it is promoted into the
# top-1 window even though it was ranked third before boosting.
assert len(results) == 1
assert results[0].id == 3
@pytest.mark.asyncio
async def test_high_fts_score_boosts_ranking():
"""FTS-only: a high normalized score should outscore a low normalized score."""
@@ -1001,3 +1001,59 @@ async def test_postgres_search_categories_exact_match(session_maker, test_projec
# Multiple categories union.
multi = await repo.search(categories=["requirement", "decision"])
assert {r.id for r in multi} == {70101, 70102}
@pytest.mark.asyncio
async def test_postgres_question_punctuation_and_relaxation(session_maker, test_project):
"""Question-form queries must produce clean lexemes and a usable relaxation.
Parity with SQLite: sentence punctuation previously reached tsquery terms,
and a strict all-AND miss had no relaxed retry, silently disabling the FTS
half of hybrid search for natural-language questions.
"""
repo = PostgresSearchRepository(session_maker, project_id=test_project.id)
# Edge punctuation stripped before lexeme formatting.
prepared = repo._prepare_search_term("When did Melanie paint a sunrise?")
assert "?" not in prepared
assert "sunrise:*" in prepared
# Relaxation drops stopwords and OR-joins content terms.
relaxed = repo._relaxed_tsquery_text("When did Melanie paint a sunrise?")
assert relaxed == "Melanie:* | paint:* | sunrise:*"
# User intent is not second-guessed.
assert repo._relaxed_tsquery_text("alpha AND beta") is None
assert repo._relaxed_tsquery_text('"exact phrase"') is None
assert repo._relaxed_tsquery_text(None) is None
@pytest.mark.asyncio
async def test_postgres_multiword_query_relaxes_on_strict_miss(session_maker, test_project):
repo = PostgresSearchRepository(session_maker, project_id=test_project.id)
now = datetime.now(timezone.utc)
await repo.index_item(
SearchIndexRow(
project_id=test_project.id,
id=77,
title="Trip plans",
content_stems="melanie painted a sunrise over the lake last year",
content_snippet="Melanie painted a sunrise over the lake last year.",
permalink="docs/trip-plans",
file_path="docs/trip-plans.md",
type="entity",
metadata={"note_type": "note"},
created_at=now,
updated_at=now,
)
)
# A content word absent from the doc ("hiking") makes the strict
# all-terms-AND query miss even after Postgres drops stopwords — without
# it, to_tsquery('english', ...) already strips "when/did/a" and matches.
strict = await repo.search(search_text="Did Melanie go hiking at sunrise?")
assert strict == []
# The hybrid FTS branch opts in; OR-relaxation surfaces the partial match.
results = await repo.search(search_text="Did Melanie go hiking at sunrise?", allow_relaxed=True)
assert any(r.id == 77 for r in results)
@@ -1124,3 +1124,83 @@ async def test_search_categories_exact_match(search_repository, search_entity):
# Multiple categories union: both observations come back.
multi = await search_repository.search(categories=["requirement", "decision"])
assert {r.id for r in multi} == {70001, 70002}
@pytest.mark.asyncio
async def test_question_punctuation_does_not_phrase_quote(search_repository):
"""Sentence punctuation must not force exact-phrase matching (#hybrid-fts).
'When did Melanie paint a sunrise?' previously became the FTS5 phrase
'"When did Melanie paint a sunrise?"*' zero rows for any corpus which
silently disabled the FTS half of hybrid search for question queries.
"""
prepared = search_repository._prepare_single_term("When did Melanie paint a sunrise?")
assert '"' not in prepared
# Prefix syntax differs by backend: FTS5 uses '*', tsquery uses ':*'.
if is_postgres_backend(search_repository):
assert "sunrise:*" in prepared
else:
assert "sunrise*" in prepared
@pytest.mark.asyncio
async def test_relaxed_query_drops_stopwords(search_repository):
"""Relaxation keys on content-bearing terms in each backend's syntax."""
if is_postgres_backend(search_repository):
relaxed = search_repository._relaxed_tsquery_text("When did Melanie paint a sunrise?")
assert relaxed == "Melanie:* | paint:* | sunrise:*"
else:
relaxed = search_repository._relaxed_fts_text("When did Melanie paint a sunrise?")
assert relaxed == "Melanie* OR paint* OR sunrise*"
@pytest.mark.asyncio
async def test_relaxed_query_respects_user_intent(search_repository):
# Eligibility matches the service-level relaxation (both backends): quoted,
# boolean, short (<3 tokens), and numeric-identifier queries are not relaxed.
relaxer = (
search_repository._relaxed_tsquery_text
if is_postgres_backend(search_repository)
else search_repository._relaxed_fts_text
)
assert relaxer("alpha AND beta") is None
assert relaxer('"exact phrase"') is None
assert relaxer("single") is None # < 3 tokens
assert relaxer("New Feature") is None # < 3 tokens (link title)
assert relaxer("root note 1") is None # numeric identifier token
assert relaxer("SPEC 16 design") is None # numeric token
assert relaxer(None) is None
@pytest.mark.asyncio
async def test_multiword_query_relaxes_to_or_when_strict_misses(search_repository, search_entity):
"""A question sharing only SOME words with a doc still surfaces it."""
from basic_memory.repository.search_index_row import SearchIndexRow
from basic_memory.schemas.search import SearchItemType
row = SearchIndexRow(
project_id=search_repository.project_id,
id=search_entity.id,
type=SearchItemType.ENTITY.value,
title="Trip plans",
content_snippet="Melanie painted a sunrise over the lake last year.",
content_stems="melanie painted a sunrise over the lake last year",
permalink=search_entity.permalink,
file_path=search_entity.file_path,
entity_id=search_entity.id,
metadata={"note_type": search_entity.note_type},
created_at=search_entity.created_at,
updated_at=search_entity.updated_at,
)
await search_repository.index_item(row)
# "hiking" is absent from the doc, so strict all-terms-AND misses on both
# backends (Postgres's stopword stripping can't rescue it either).
strict = await search_repository.search(search_text="Did Melanie go hiking at sunrise?")
assert strict == []
# The hybrid FTS branch opts in; OR-relaxation surfaces the partial match.
results = await search_repository.search(
search_text="Did Melanie go hiking at sunrise?", allow_relaxed=True
)
assert any(r.entity_id == search_entity.id for r in results)
+1 -179
View File
@@ -89,6 +89,7 @@ class _ConcreteRepo(SearchRepositoryBase):
min_similarity: float | None = None,
limit: int = 10,
offset: int = 0,
allow_relaxed: bool = False,
) -> list[SearchIndexRow]:
return []
@@ -809,182 +810,3 @@ async def test_sync_entity_vectors_batch_logs_resolved_fastembed_runtime_setting
assert runtime_logs[0]["threads"] == 4
assert runtime_logs[0]["configured_parallel"] == 2
assert runtime_logs[0]["effective_parallel"] == 2
# --- Entity-aware ranking boost (#951) ---
def _make_index_row(
*,
row_id: int,
title: str,
row_type: str = SearchItemType.ENTITY.value,
) -> SearchIndexRow:
"""Build a real SearchIndexRow for entity-boost matching tests."""
now = datetime(2026, 1, 1)
return SearchIndexRow(
project_id=1,
id=row_id,
type=row_type,
file_path=f"notes/{row_id}.md",
created_at=now,
updated_at=now,
permalink=f"notes/{row_id}",
title=title,
)
class TestExtractQueryEntityTerms:
"""Verify proper-noun extraction from query strings."""
def test_extracts_single_proper_noun(self):
terms = SearchRepositoryBase._extract_query_entity_terms("What are Joanna's hobbies?")
assert terms == {"joanna"}
def test_extracts_multiple_proper_nouns(self):
terms = SearchRepositoryBase._extract_query_entity_terms(
"What symbolic gifts do Deborah and Jolene have from their mothers?"
)
assert terms == {"deborah", "jolene"}
def test_who_is_anthony(self):
terms = SearchRepositoryBase._extract_query_entity_terms("Who is Anthony?")
assert terms == {"anthony"}
def test_all_lowercase_query_has_no_entity_terms(self):
terms = SearchRepositoryBase._extract_query_entity_terms("what is the weather today")
assert terms == set()
def test_capitalized_stopword_at_sentence_start_is_ignored(self):
# "What" and "Who" are capitalized interrogatives, not entity names.
terms = SearchRepositoryBase._extract_query_entity_terms("What does Sarah like?")
assert terms == {"sarah"}
def test_all_caps_token_is_extracted(self):
terms = SearchRepositoryBase._extract_query_entity_terms("who works at NASA")
assert terms == {"nasa"}
def test_possessive_is_stripped(self):
terms = SearchRepositoryBase._extract_query_entity_terms("Joanna's mother")
assert terms == {"joanna"}
def test_single_capital_letter_is_ignored(self):
# "A" lowercases to a stopword; "X" is a non-stopword single letter that
# still carries no entity signal and must be dropped by the length guard.
terms = SearchRepositoryBase._extract_query_entity_terms("A is for X and Apple")
assert terms == {"apple"}
def test_empty_and_none_inputs(self):
assert SearchRepositoryBase._extract_query_entity_terms("") == set()
assert SearchRepositoryBase._extract_query_entity_terms(None) == set()
class TestRowEntityMatchCount:
"""Verify lexical match counting between query terms and row entity names."""
def test_title_match_counts_one(self):
row = _make_index_row(row_id=1, title="Joanna's Hobbies")
assert SearchRepositoryBase._row_entity_match_count(row, {"joanna"}) == 1
def test_relation_title_matches_both_endpoints(self):
row = _make_index_row(
row_id=2,
title="Deborah -> Jolene",
row_type=SearchItemType.RELATION.value,
)
count = SearchRepositoryBase._row_entity_match_count(row, {"deborah", "jolene"})
assert count == 2
def test_no_match_returns_zero(self):
row = _make_index_row(row_id=3, title="Anthony Profile")
assert SearchRepositoryBase._row_entity_match_count(row, {"joanna"}) == 0
def test_match_is_case_insensitive(self):
row = _make_index_row(row_id=4, title="JOANNA notes")
assert SearchRepositoryBase._row_entity_match_count(row, {"joanna"}) == 1
def test_empty_terms_returns_zero(self):
row = _make_index_row(row_id=5, title="Joanna")
assert SearchRepositoryBase._row_entity_match_count(row, set()) == 0
def test_missing_title_returns_zero(self):
row = _make_index_row(row_id=6, title="")
row.title = None
assert SearchRepositoryBase._row_entity_match_count(row, {"joanna"}) == 0
def test_distinct_terms_not_double_counted(self):
# A title containing the same term twice still counts as one distinct match.
row = _make_index_row(row_id=7, title="Joanna and Joanna")
assert SearchRepositoryBase._row_entity_match_count(row, {"joanna"}) == 1
class TestApplyEntityBoost:
"""Verify the entity-boost score math and gating."""
def _repo(self, *, enabled: bool, weight: float = 0.15, max_terms: int = 3) -> _ConcreteRepo:
repo = _ConcreteRepo()
repo._entity_boost_enabled = enabled
repo._entity_boost_weight = weight
repo._entity_boost_max_terms = max_terms
return repo
def test_disabled_returns_scores_unchanged(self):
repo = self._repo(enabled=False)
key = ("entity", 1)
rows = {key: _make_index_row(row_id=1, title="Joanna")}
scores = {key: 0.5}
assert repo._apply_entity_boost(scores, rows, {"joanna"}) == {key: 0.5}
def test_matching_row_is_boosted(self):
repo = self._repo(enabled=True, weight=0.2)
key = ("entity", 1)
rows = {key: _make_index_row(row_id=1, title="Joanna")}
boosted = repo._apply_entity_boost({key: 0.5}, rows, {"joanna"})
assert boosted[key] == pytest.approx(0.5 * 1.2)
def test_non_matching_row_unchanged(self):
repo = self._repo(enabled=True, weight=0.2)
key = ("entity", 1)
rows = {key: _make_index_row(row_id=1, title="Anthony")}
boosted = repo._apply_entity_boost({key: 0.5}, rows, {"joanna"})
assert boosted[key] == pytest.approx(0.5)
def test_boost_can_reorder_lower_scored_match_above_higher_non_match(self):
repo = self._repo(enabled=True, weight=0.5)
generic_key = ("entity", 1)
joanna_key = ("entity", 2)
rows = {
generic_key: _make_index_row(row_id=1, title="Generic topic about hobbies"),
joanna_key: _make_index_row(row_id=2, title="Joanna"),
}
# The generic row starts higher (0.6) but does not match; "Joanna" (0.5) matches.
boosted = repo._apply_entity_boost({generic_key: 0.6, joanna_key: 0.5}, rows, {"joanna"})
assert boosted[joanna_key] > boosted[generic_key]
def test_multiple_matches_scale_boost(self):
repo = self._repo(enabled=True, weight=0.1, max_terms=3)
key = ("relation", 1)
rows = {key: _make_index_row(row_id=1, title="Deborah -> Jolene")}
boosted = repo._apply_entity_boost({key: 1.0}, rows, {"deborah", "jolene"})
# Two matched terms -> 1 + 0.1 * 2 = 1.2
assert boosted[key] == pytest.approx(1.2)
def test_max_terms_caps_the_boost(self):
repo = self._repo(enabled=True, weight=0.1, max_terms=1)
key = ("relation", 1)
rows = {key: _make_index_row(row_id=1, title="Deborah -> Jolene")}
boosted = repo._apply_entity_boost({key: 1.0}, rows, {"deborah", "jolene"})
# Capped at 1 term -> 1 + 0.1 * 1 = 1.1
assert boosted[key] == pytest.approx(1.1)
def test_zero_weight_is_noop(self):
repo = self._repo(enabled=True, weight=0.0)
key = ("entity", 1)
rows = {key: _make_index_row(row_id=1, title="Joanna")}
assert repo._apply_entity_boost({key: 0.5}, rows, {"joanna"}) == {key: 0.5}
def test_empty_entity_terms_is_noop(self):
repo = self._repo(enabled=True, weight=0.2)
key = ("entity", 1)
rows = {key: _make_index_row(row_id=1, title="Joanna")}
assert repo._apply_entity_boost({key: 0.5}, rows, set()) == {key: 0.5}
@@ -62,6 +62,7 @@ class ConcreteSearchRepo(SearchRepositoryBase):
min_similarity: float | None = None,
limit: int = 10,
offset: int = 0,
allow_relaxed: bool = False,
) -> list[SearchIndexRow]:
return [] # pragma: no cover
@@ -66,6 +66,7 @@ class ConcreteSearchRepo(SearchRepositoryBase):
min_similarity: Optional[float] = None,
limit: int = 10,
offset: int = 0,
allow_relaxed: bool = False,
) -> list[SearchIndexRow]:
return [] # pragma: no cover
+51
View File
@@ -222,6 +222,57 @@ async def test_initialize_file_sync_no_constraint_when_env_unset(
assert _FakeWatchService.last_kwargs.get("constrained_project") is None
@pytest.mark.asyncio
async def test_initialize_file_sync_skips_project_with_non_absolute_path(
app_config: BasicMemoryConfig, config_manager, config_home, monkeypatch
):
"""Projects without an absolute local path are excluded from background sync (issue #949).
A config entry of ``{"path": ""}`` defaults to LOCAL mode and is not
recognized as cloud, yet Path("") resolves to the process cwd. Syncing it
would inject frontmatter into unrelated files, so it must be skipped.
"""
await db.shutdown_db()
try:
from basic_memory.config import ProjectEntry
good = config_home / "good"
good.mkdir(parents=True, exist_ok=True)
updated = app_config.model_copy(
update={
"projects": {
"good": ProjectEntry(path=str(good)),
# No mode -> defaults to LOCAL, empty (cwd-relative) path.
"empty-path": ProjectEntry(path=""),
},
"default_project": "good",
}
)
config_manager.save_config(updated)
await initialize_database(updated)
await reconcile_projects_with_config(updated)
_disable_test_env_short_circuit(monkeypatch)
monkeypatch.setattr("basic_memory.sync.WatchService", _FakeWatchService)
infos: list[str] = []
monkeypatch.setattr(
"basic_memory.services.initialization.logger.info",
lambda message, *args, **kwargs: infos.append(message),
)
await initialize_file_sync(updated, quiet=True)
skip_logs = [m for m in infos if "not locally syncable" in m]
assert skip_logs, "expected a skip log for the empty-path project"
assert "empty-path" in skip_logs[0]
assert "good" not in skip_logs[0]
finally:
await db.shutdown_db()
@pytest.mark.asyncio
async def test_initialize_app_no_precedence_warning_when_not_conflicting(
app_config: BasicMemoryConfig, monkeypatch
+11 -8
View File
@@ -71,15 +71,18 @@ async def _register_local_projects(
@pytest.mark.asyncio
async def test_select_projects_to_watch_returns_all_when_unconstrained(
app_config: BasicMemoryConfig, project_repository
app_config: BasicMemoryConfig, project_repository, tmp_path
):
"""Without a --project constraint, every active project is watched."""
# Use tmp_path so the project paths are OS-absolute on Windows too — a
# POSIX-style "/tmp/alpha" is not absolute on Windows (no drive letter),
# and _select_projects_to_watch now skips non-absolute paths (issue #949).
await _register_local_projects(
app_config,
project_repository,
[
{"name": "project-alpha", "path": "/tmp/alpha"},
{"name": "project-beta", "path": "/tmp/beta"},
{"name": "project-alpha", "path": str(tmp_path / "alpha")},
{"name": "project-beta", "path": str(tmp_path / "beta")},
],
)
@@ -94,7 +97,7 @@ async def test_select_projects_to_watch_returns_all_when_unconstrained(
@pytest.mark.asyncio
async def test_select_projects_to_watch_filters_to_constrained_project(
app_config: BasicMemoryConfig, project_repository
app_config: BasicMemoryConfig, project_repository, tmp_path
):
"""With ``constrained_project`` set, only that project is returned.
@@ -106,8 +109,8 @@ async def test_select_projects_to_watch_filters_to_constrained_project(
app_config,
project_repository,
[
{"name": "project-alpha", "path": "/tmp/alpha"},
{"name": "project-beta", "path": "/tmp/beta"},
{"name": "project-alpha", "path": str(tmp_path / "alpha")},
{"name": "project-beta", "path": str(tmp_path / "beta")},
],
)
@@ -124,13 +127,13 @@ async def test_select_projects_to_watch_filters_to_constrained_project(
@pytest.mark.asyncio
async def test_select_projects_to_watch_empty_when_constrained_project_missing(
app_config: BasicMemoryConfig, project_repository
app_config: BasicMemoryConfig, project_repository, tmp_path
):
"""An unknown constraint yields an empty watch set rather than watching everything."""
await _register_local_projects(
app_config,
project_repository,
[{"name": "project-alpha", "path": "/tmp/alpha"}],
[{"name": "project-alpha", "path": str(tmp_path / "alpha")}],
)
service = WatchService(
+84
View File
@@ -236,6 +236,90 @@ async def test_run_keeps_cloud_projects_with_local_bisync(monkeypatch, tmp_path)
assert seen_project_names == [["local-project", "cloud-bisync"]]
@pytest.mark.asyncio
async def test_run_filters_empty_path_local_mode_project(monkeypatch, tmp_path):
"""A project with an empty path is skipped even when mode is LOCAL (issue #949).
ProjectEntry.mode defaults to LOCAL, so a hand-edited config entry of
``{"path": ""}`` is not recognized as cloud. The watch cycle must still skip
it: Path("") resolves to the process cwd, and watching that would mutate
whatever directory the server was launched from.
"""
config = BasicMemoryConfig(
watch_project_reload_interval=1,
projects={
"local-project": {"path": str(tmp_path / "local"), "mode": "local"},
# No explicit mode -> defaults to LOCAL, with an empty (cwd-relative) path.
"empty-path": {"path": ""},
},
)
repo = _Repo(
projects_return=[
Project(id=1, name="local-project", path=str(tmp_path / "local"), permalink="local"),
Project(id=2, name="empty-path", path="", permalink="empty-path"),
]
)
watch_service = _watch_service(config, repo)
seen_project_names: list[list[str]] = []
async def watch_cycle_stub(projects, stop_event):
seen_project_names.append([p.name for p in projects])
watch_service.state.running = False
stop_event.set()
async def fake_write_status():
return None
monkeypatch.setattr(watch_service, "_watch_projects_cycle", watch_cycle_stub)
monkeypatch.setattr(watch_service, "write_status", fake_write_status)
await watch_service.run()
assert seen_project_names == [["local-project"]]
@pytest.mark.asyncio
async def test_run_filters_orphan_db_project_absent_from_config(monkeypatch, tmp_path):
"""A DB row not present in config is skipped even with an absolute path.
Config is the source of truth. Reconciliation normally deletes orphan rows,
but if it is skipped or fails a stale row could remain; watching it would
mutate a directory the user already removed from config.
"""
config = BasicMemoryConfig(
watch_project_reload_interval=1,
projects={
"local-project": {"path": str(tmp_path / "local"), "mode": "local"},
},
)
repo = _Repo(
projects_return=[
Project(id=1, name="local-project", path=str(tmp_path / "local"), permalink="local"),
# Absolute path, but no matching entry in config -> stale/orphan row.
Project(id=2, name="orphan", path=str(tmp_path / "orphan"), permalink="orphan"),
]
)
watch_service = _watch_service(config, repo)
seen_project_names: list[list[str]] = []
async def watch_cycle_stub(projects, stop_event):
seen_project_names.append([p.name for p in projects])
watch_service.state.running = False
stop_event.set()
async def fake_write_status():
return None
monkeypatch.setattr(watch_service, "_watch_projects_cycle", watch_cycle_stub)
monkeypatch.setattr(watch_service, "write_status", fake_write_status)
await watch_service.run()
assert seen_project_names == [["local-project"]]
@pytest.mark.asyncio
async def test_run_continues_after_cycle_error(monkeypatch, tmp_path):
config = BasicMemoryConfig(
+24
View File
@@ -1245,6 +1245,30 @@ class TestProjectMode:
assert config.projects["research"].mode == ProjectMode.LOCAL
assert config.get_project_mode("research") == ProjectMode.LOCAL
def test_is_locally_syncable_true_for_config_project_with_absolute_path(self, tmp_path):
"""A project in config with an absolute path is locally syncable."""
abs_path = str(tmp_path / "research")
config = BasicMemoryConfig(projects={"research": ProjectEntry(path=abs_path)})
assert config.is_locally_syncable("research", abs_path) is True
def test_is_locally_syncable_false_for_empty_path(self):
"""An empty path resolves to cwd, so it is never locally syncable (#949)."""
config = BasicMemoryConfig(projects={"empty": ProjectEntry(path="")})
assert config.is_locally_syncable("empty", "") is False
def test_is_locally_syncable_false_for_relative_path(self):
"""A relative (slug) path, as used by cloud-only projects, is not syncable."""
config = BasicMemoryConfig(projects={"cloud": ProjectEntry(path="cloud-slug")})
assert config.is_locally_syncable("cloud", "cloud-slug") is False
def test_is_locally_syncable_false_for_orphan_not_in_config(self, tmp_path):
"""A DB row absent from config is not syncable even with an absolute path.
Config is the source of truth; stale rows must not be synced (#949).
"""
config = BasicMemoryConfig(projects={})
assert config.is_locally_syncable("orphan", str(tmp_path / "orphan")) is False
def test_cloud_api_key_defaults_to_none(self):
"""Test that cloud_api_key defaults to None."""
config = BasicMemoryConfig()