mirror of
https://github.com/basicmachines-co/basic-memory
synced 2026-06-21 13:47:35 +00:00
Compare commits
10 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| b5f13d6903 | |||
| 6d06c4a4e7 | |||
| 16869867da | |||
| 18da6194f9 | |||
| a6d0784335 | |||
| 232f469065 | |||
| 5a08cfd9ac | |||
| b997d858cd | |||
| 598965c389 | |||
| 8dd6451dfe |
@@ -6,14 +6,14 @@
|
||||
},
|
||||
"metadata": {
|
||||
"description": "Official Basic Memory plugins from the canonical basic-memory repository",
|
||||
"version": "0.22.0"
|
||||
"version": "0.22.1"
|
||||
},
|
||||
"plugins": [
|
||||
{
|
||||
"name": "basic-memory",
|
||||
"source": "./plugins/claude-code",
|
||||
"description": "The bridge between Claude's working memory and Basic Memory's durable knowledge graph \u2014 session briefings, pre-compaction checkpoints, and capture reflexes",
|
||||
"version": "0.22.0",
|
||||
"version": "0.22.1",
|
||||
"author": {
|
||||
"name": "Basic Machines"
|
||||
},
|
||||
|
||||
@@ -116,22 +116,24 @@ After PyPI release is published, update the MCP registry:
|
||||
|
||||
#### Website Updates
|
||||
|
||||
**1. basicmachines.co** (sibling `basicmachines.co` repo)
|
||||
- **Goal**: Update version number displayed on the homepage
|
||||
- **Location**: Search for "Basic Memory v0." in the codebase to find version displays
|
||||
- **What to update**:
|
||||
- Hero section heading that shows "Basic Memory v{VERSION}"
|
||||
- "What's New in v{VERSION}" section heading
|
||||
- Feature highlights array (look for array of features with title/description)
|
||||
- **Process**:
|
||||
**1. basicmemory.com** (sibling `basicmemory.com` repo —
|
||||
`basicmachines-co/basicmemory.com`, formerly `basicmachines.co`)
|
||||
- **No version bump needed.** The marketing site is an Astro + React app and
|
||||
carries **no hardcoded Basic Memory version number** anywhere in its UI
|
||||
(`hero.tsx` and the rest of the site have no version string). The old
|
||||
instruction to bump `src/components/sections/hero.tsx` is obsolete — that
|
||||
file no longer holds a version. Release announcements are dated blog posts,
|
||||
not an in-place edit.
|
||||
- **Skip entirely for patch releases.**
|
||||
- **Significant releases only — optional announcement post**:
|
||||
1. Pull latest from GitHub: `git pull origin main`
|
||||
2. Create release branch: `git checkout -b release/v{VERSION}`
|
||||
3. Search codebase for current version number (e.g., "v0.16.1")
|
||||
4. Update version numbers to new release version
|
||||
5. Update feature highlights with 3-5 key features from this release (extract from CHANGELOG.md)
|
||||
6. Commit changes: `git commit -m "chore: update to v{VERSION}"`
|
||||
7. Push branch: `git push origin release/v{VERSION}`
|
||||
- **Deploy**: Follow deployment process for basicmachines.co
|
||||
3. Add a dated post under `src/content/blog/` modeled on an existing
|
||||
release post (e.g. `basic-memory-v0-19-0-release.md`), summarizing 3–5
|
||||
headline features from `CHANGELOG.md`
|
||||
4. Commit (`git commit -s -m "..."`), push, and open a PR against
|
||||
`basicmachines-co/basicmemory.com`
|
||||
- **Deploy**: follow that repo's deployment process.
|
||||
|
||||
**2. docs.basicmemory.com** (sibling `docs.basicmemory.com` repo)
|
||||
- **Goal**: Add a What's New page for the release and bump the homepage badge
|
||||
|
||||
@@ -322,7 +322,12 @@ The recipe runs `just lint` + `just typecheck`, then updates every release manif
|
||||
|
||||
**Post-release tasks** the recipe surfaces but doesn't run:
|
||||
- `docs.basicmemory.com` — add a What's New page under `content/2.whats-new/` and bump the version badge in `content/index.md` (the changelog page auto-fetches GitHub releases; see that repo's CLAUDE.md version-bump checklist)
|
||||
- `basicmachines.co` — bump version in `src/components/sections/hero.tsx`
|
||||
- `basicmemory.com` — the marketing site (Astro + React, repo
|
||||
`basicmachines-co/basicmemory.com`, formerly `basicmachines.co`) carries **no
|
||||
hardcoded version number** in its UI, so there is nothing to bump. For a
|
||||
significant release, optionally add a dated announcement post under
|
||||
`src/content/blog/` (model it on an existing `basic-memory-vX-Y-Z-release.md`).
|
||||
Skip entirely for routine patch releases.
|
||||
- MCP Registry — `mcp-publisher publish` from the repo root
|
||||
|
||||
See `.claude/commands/release/release.md` (and `beta.md`, `release-check.md`, `changelog.md` alongside it) for the full release + post-release runbook, including the slash commands.
|
||||
|
||||
@@ -1,5 +1,49 @@
|
||||
# CHANGELOG
|
||||
|
||||
## v0.22.1 (2026-06-12)
|
||||
|
||||
Follow-up patch to v0.22.0. Fixes project and default-project resolution on
|
||||
fresh installs, MCP workspace routing, sync project selection, and CLI startup
|
||||
latency, plus a few MCP parity additions.
|
||||
|
||||
### Features
|
||||
|
||||
- Added a `workspace` parameter to `write_note` for parity with `edit_note`.
|
||||
- **#826**: Added `title` and `tags` annotations to all MCP tool decorators
|
||||
(phase 1).
|
||||
- **#930**: `search_notes` now comma-splits `note_types`, `entity_types`, and
|
||||
`categories`.
|
||||
- **#971**: Added the manual-pages flow — manpage seed schema, flow docs, and
|
||||
verification fixes.
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Fresh installs no longer fail when the projects table is empty: resolve now
|
||||
points them at project setup, the first project is promoted to default when
|
||||
the config default is missing from the database, the promoted default state
|
||||
is returned from the project-create API, and a default can be set when none
|
||||
is currently set. An existing database default is preserved when repairing a
|
||||
missing config default.
|
||||
- **#949**: Sync skips projects without an absolute local path and excludes
|
||||
orphan DB projects that are absent from config.
|
||||
- **#952 / #981**: Resolved workspace display names and tenant ids in qualified
|
||||
project routes, closing out the manual verification findings.
|
||||
- `note_types`/`entity_types`/`categories` are normalized on the direct-call
|
||||
path, with non-string list elements rejected.
|
||||
- Vector-search hydration keys on `(type, id)` to prevent id collisions.
|
||||
- `file_utils` requires line-anchored frontmatter fences.
|
||||
- CLI startup is faster: FastAPI and app imports are deferred out of CLI
|
||||
startup, and rich/typer modules are preloaded before an in-place upgrade.
|
||||
- `config.json` is written atomically.
|
||||
- In-memory SQLite sessions are serialized so concurrent rollbacks cannot
|
||||
destroy writes.
|
||||
|
||||
### Maintenance
|
||||
|
||||
- Release recipes route through PRs and wait for the release PR merge to land
|
||||
before tagging; the release runbook is refreshed and stripped of
|
||||
user-specific absolute paths.
|
||||
|
||||
## v0.22.0 (2026-06-11)
|
||||
|
||||
Team-safe cloud sync. New additive `bm cloud push` and `bm cloud pull`
|
||||
|
||||
@@ -6,6 +6,7 @@
|
||||
[](https://github.com/astral-sh/ruff)
|
||||

|
||||

|
||||
[](https://deepwiki.com/basicmachines-co/basic-memory)
|
||||
|
||||
## Skip the install — try Basic Memory in the cloud
|
||||
|
||||
|
||||
@@ -107,76 +107,6 @@ All settings are fields on `BasicMemoryConfig` and can be set via environment va
|
||||
| `semantic_embedding_document_input_type` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_DOCUMENT_INPUT_TYPE` | Auto for known LiteLLM models | Optional LiteLLM `input_type` for indexed document/passages. |
|
||||
| `semantic_embedding_query_input_type` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_QUERY_INPUT_TYPE` | Auto for known LiteLLM models | Optional LiteLLM `input_type` for search queries. |
|
||||
| `semantic_vector_k` | `BASIC_MEMORY_SEMANTIC_VECTOR_K` | `100` | Candidate count for vector nearest-neighbour retrieval. Higher values improve recall at the cost of latency. |
|
||||
| `search_entity_boost_enabled` | `BASIC_MEMORY_SEARCH_ENTITY_BOOST_ENABLED` | `false` | Enable the entity-aware ranking boost in hybrid search (see below). Default off: benchmark-validated as inert on LoCoMo and prone to Title-Case false positives. |
|
||||
| `search_entity_boost_weight` | `BASIC_MEMORY_SEARCH_ENTITY_BOOST_WEIGHT` | `0.15` | Per-matched-term multiplier strength for the entity boost. A candidate matching N query entity terms is scaled by `1 + weight * min(N, max_terms)`. |
|
||||
| `search_entity_boost_max_terms` | `BASIC_MEMORY_SEARCH_ENTITY_BOOST_MAX_TERMS` | `3` | Maximum number of distinct matched entity terms that contribute to the boost, bounding the multiplier. |
|
||||
|
||||
## Entity-Aware Ranking Boost
|
||||
|
||||
Hybrid search fuses keyword (FTS) and vector similarity, but proper nouns in a query
|
||||
carry no special weight against generic semantic similarity. As a result, a document
|
||||
about a *different* entity on the same topic can outrank the document that actually
|
||||
names the queried entity — e.g. "What are Joanna's hobbies?" surfacing a generic
|
||||
hobbies note ahead of Joanna's note (see
|
||||
[#951](https://github.com/basicmachines-co/basic-memory/issues/951)).
|
||||
|
||||
When `search_entity_boost_enabled=true`, hybrid retrieval performs a final,
|
||||
lexical-only re-scoring pass:
|
||||
|
||||
1. It extracts candidate entity terms from the query — capitalized / proper-noun
|
||||
tokens that are not common stopwords (e.g. `Joanna`, `Anthony`, `NASA`).
|
||||
2. For each fused candidate, it counts how many distinct query entity terms appear in
|
||||
the candidate's entity name (its title) or in a relation row's linked entity names.
|
||||
3. Matching candidates have their fused score multiplied by
|
||||
`1 + weight * min(matches, max_terms)`, so an entity-matching document can be
|
||||
promoted above a higher-similarity non-matching one.
|
||||
|
||||
The boost adds **no model inference** — it is pure index/lexical lookup, so per-query
|
||||
latency overhead is trivial. It only affects `hybrid` retrieval; `text` and `vector`
|
||||
modes are unchanged. Non-matching candidates keep their original scores, so ordering
|
||||
among them is preserved.
|
||||
|
||||
```bash
|
||||
export BASIC_MEMORY_SEARCH_ENTITY_BOOST_ENABLED=true
|
||||
# Optional tuning:
|
||||
export BASIC_MEMORY_SEARCH_ENTITY_BOOST_WEIGHT=0.15
|
||||
export BASIC_MEMORY_SEARCH_ENTITY_BOOST_MAX_TERMS=3
|
||||
```
|
||||
|
||||
> **Default off.** This setting is disabled by default. See the benchmark
|
||||
> findings below for why the default stays off and where the boost helps.
|
||||
|
||||
### Benchmark findings
|
||||
|
||||
The boost was benchmarked against LoCoMo (the
|
||||
[basic-memory-benchmarks](https://github.com/basicmachines-co/basic-memory-benchmarks)
|
||||
retrieval suite, hybrid mode) and a hand-built adversarial corpus. Two results
|
||||
drove the decision to keep the default **off** and leave the weight at `0.15`:
|
||||
|
||||
1. **LoCoMo is insensitive to the boost.** Sweeping the weight across
|
||||
`0.15, 0.3, 0.5, 1.0, 2.0` produced *identical* recall@5, recall@10, MRR, and
|
||||
content-hit at every point — no query reordered, no score changed. LoCoMo's
|
||||
documents are titled by conversation/session id and expose speaker names only
|
||||
in body text, never as entity titles or relation names. Because the boost
|
||||
matches query proper nouns against a candidate's **title or linked relation
|
||||
names**, it never fires on this corpus. LoCoMo therefore provides no signal to
|
||||
raise the weight, and the boost neither helps nor harms it.
|
||||
|
||||
2. **A capitalization-only heuristic has false positives.** On a corpus where
|
||||
entity terms appear in titles, the boost correctly promotes the right document
|
||||
for clean proper nouns (e.g. `Katze`) and is correctly inert on
|
||||
lowercase-leading identifiers (e.g. `getUserById`, ignored). But **Title-Case
|
||||
queries can regress**: a query like `What Is The Plan For Q3` extracts `Q3` as
|
||||
an entity term, and even at weight `0.15` it promotes a document that
|
||||
*literally* contains "Q3" above the more relevant document that says "third
|
||||
quarter". Since entity detection is lexical (capitalization, no NER), any
|
||||
capitalized non-entity token in a query is a potential false positive.
|
||||
|
||||
**Guidance.** Enable the boost only on entity-heavy corpora where your queries
|
||||
name entities that are themselves note titles or linked relations (the #951
|
||||
"Joanna" case). Prefer natural-case queries (`What are Joanna's hobbies?`) over
|
||||
Title-Cased phrasing, which can inject spurious entity terms. Leave it off for
|
||||
conversational / body-text-keyed corpora like LoCoMo, where it cannot help.
|
||||
|
||||
## Embedding Providers
|
||||
|
||||
|
||||
@@ -38,7 +38,7 @@ from typing import Any, Callable
|
||||
from agent.memory_provider import MemoryProvider
|
||||
from tools.registry import tool_error
|
||||
|
||||
__version__ = "0.22.0"
|
||||
__version__ = "0.22.1"
|
||||
|
||||
logger = logging.getLogger("hermes.memory.basic-memory")
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
name: basic-memory
|
||||
version: 0.22.0
|
||||
version: 0.22.1
|
||||
description: "Basic Memory — persistent knowledge graph backed by the basic-memory MCP server"
|
||||
pip_dependencies:
|
||||
- mcp
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@basicmemory/openclaw-basic-memory",
|
||||
"version": "0.22.0",
|
||||
"version": "0.22.1",
|
||||
"type": "module",
|
||||
"main": "./dist/index.js",
|
||||
"types": "./dist/index.d.ts",
|
||||
|
||||
@@ -468,7 +468,8 @@ release version:
|
||||
echo "📝 REMINDER: Post-release tasks:"
|
||||
echo " 1. docs.basicmemory.com - Add a What's New page under content/2.whats-new/"
|
||||
echo " and bump the badge in content/index.md (see that repo's CLAUDE.md)"
|
||||
echo " 2. basicmachines.co - Update version in src/components/sections/hero.tsx"
|
||||
echo " 2. basicmemory.com - No version number in the site UI; for a significant"
|
||||
echo " release optionally add a post under src/content/blog/. Skip for patches."
|
||||
echo " 3. MCP Registry - Run: mcp-publisher publish"
|
||||
echo " See: .claude/commands/release/release.md for detailed instructions"
|
||||
|
||||
@@ -585,7 +586,8 @@ beta version:
|
||||
echo "📝 REMINDER: For stable releases, update documentation sites:"
|
||||
echo " 1. docs.basicmemory.com - Add a What's New page under content/2.whats-new/"
|
||||
echo " and bump the badge in content/index.md (see that repo's CLAUDE.md)"
|
||||
echo " 2. basicmachines.co - Update version in src/components/sections/hero.tsx"
|
||||
echo " 2. basicmemory.com - No version number in the site UI; for a significant"
|
||||
echo " release optionally add a post under src/content/blog/. Skip for patches."
|
||||
echo " See: .claude/commands/release/release.md for detailed instructions"
|
||||
|
||||
# List all available recipes
|
||||
|
||||
@@ -6,14 +6,14 @@
|
||||
},
|
||||
"metadata": {
|
||||
"description": "Official plugins from Basic Machines for knowledge management and AI-assisted development",
|
||||
"version": "0.22.0"
|
||||
"version": "0.22.1"
|
||||
},
|
||||
"plugins": [
|
||||
{
|
||||
"name": "basic-memory",
|
||||
"source": "./",
|
||||
"description": "The bridge between Claude's working memory and Basic Memory's durable knowledge graph \u2014 session briefings, pre-compaction checkpoints, and capture reflexes",
|
||||
"version": "0.22.0",
|
||||
"version": "0.22.1",
|
||||
"author": {
|
||||
"name": "Basic Machines"
|
||||
},
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "basic-memory",
|
||||
"description": "The bridge between Claude's working memory and Basic Memory's durable knowledge graph \u2014 session briefings, pre-compaction checkpoints, and capture reflexes",
|
||||
"version": "0.22.0",
|
||||
"version": "0.22.1",
|
||||
"author": {
|
||||
"name": "Basic Machines"
|
||||
},
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "codex",
|
||||
"version": "0.22.0",
|
||||
"version": "0.22.1",
|
||||
"description": "A Codex-native bridge to Basic Memory for durable engineering context, decisions, and resumable checkpoints.",
|
||||
"author": {
|
||||
"name": "Basic Machines",
|
||||
|
||||
+2
-2
@@ -6,12 +6,12 @@
|
||||
"url": "https://github.com/basicmachines-co/basic-memory.git",
|
||||
"source": "github"
|
||||
},
|
||||
"version": "0.22.0",
|
||||
"version": "0.22.1",
|
||||
"packages": [
|
||||
{
|
||||
"registryType": "pypi",
|
||||
"identifier": "basic-memory",
|
||||
"version": "0.22.0",
|
||||
"version": "0.22.1",
|
||||
"runtimeHint": "uvx",
|
||||
"runtimeArguments": [
|
||||
{
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
"""basic-memory - Local-first knowledge management combining Zettelkasten with knowledge graphs"""
|
||||
|
||||
# Package version - updated by release automation
|
||||
__version__ = "0.22.0"
|
||||
__version__ = "0.22.1"
|
||||
|
||||
# API version for FastAPI - independent of package version
|
||||
__api_version__ = "v0"
|
||||
|
||||
+20
-36
@@ -346,42 +346,6 @@ class BasicMemoryConfig(BaseSettings):
|
||||
"Valid values: text, vector, hybrid. "
|
||||
"When unset, defaults to 'hybrid' if semantic search is enabled, otherwise 'text'.",
|
||||
)
|
||||
# Entity-aware ranking boost (hybrid retrieval).
|
||||
# Trigger: proper nouns in a query (e.g. "Joanna") carry no extra weight against
|
||||
# generic semantic similarity, so documents from the wrong conversation can outrank
|
||||
# the gold document during hybrid fusion (#951).
|
||||
# Why: entities are first-class in Basic Memory, so a candidate whose title or linked
|
||||
# relation names contain a query proper noun is a stronger answer than a same-topic
|
||||
# document about a different entity.
|
||||
# Outcome: when enabled, hybrid fusion multiplies a candidate's fused score by a small
|
||||
# bonus for each distinct query entity term it matches lexically (no model inference).
|
||||
# Default OFF: LoCoMo benchmarking showed the boost is inert there (its docs are keyed
|
||||
# by session id, not entity titles) and an adversarial check found Title-Case queries
|
||||
# can inject spurious entity terms (e.g. "Q3") that regress ranking. See
|
||||
# docs/semantic-search.md "Benchmark findings".
|
||||
search_entity_boost_enabled: bool = Field(
|
||||
default=False,
|
||||
description="Enable entity-aware ranking boost in hybrid search. When enabled, "
|
||||
"hybrid candidates whose title or linked relation names contain a proper-noun "
|
||||
"term from the query are boosted in the final ranking. Lexical-only; adds no "
|
||||
"model inference. Default off: benchmark-validated as inert on LoCoMo and prone "
|
||||
"to Title-Case false positives (see docs/semantic-search.md).",
|
||||
)
|
||||
search_entity_boost_weight: float = Field(
|
||||
default=0.15,
|
||||
description="Per-matched-term multiplier strength for the entity-aware ranking "
|
||||
"boost. A candidate matching N distinct query entity terms has its fused score "
|
||||
"multiplied by (1 + weight * N), capped at search_entity_boost_max_terms terms. "
|
||||
"Only applies when search_entity_boost_enabled is true.",
|
||||
ge=0.0,
|
||||
)
|
||||
search_entity_boost_max_terms: int = Field(
|
||||
default=3,
|
||||
description="Maximum number of distinct matched entity terms that contribute to "
|
||||
"the entity-aware ranking boost, bounding the multiplier so a single candidate "
|
||||
"cannot run away with the ranking.",
|
||||
gt=0,
|
||||
)
|
||||
|
||||
# Database connection pool configuration (Postgres only)
|
||||
db_pool_size: int = Field(
|
||||
@@ -694,6 +658,26 @@ class BasicMemoryConfig(BaseSettings):
|
||||
entry = self.projects.get(project_name)
|
||||
return entry.mode if entry else ProjectMode.CLOUD
|
||||
|
||||
def is_locally_syncable(self, project_name: str, project_path: str) -> bool:
|
||||
"""Whether a project should be synced/watched on the local filesystem.
|
||||
|
||||
Both conditions are required (issue #949):
|
||||
|
||||
* The project is present in config. Config is the source of truth, so a
|
||||
stale database row that was removed from config — but whose deletion
|
||||
has not yet been reconciled, or whose reconciliation failed — must
|
||||
not be synced even though it still has a real directory on disk.
|
||||
* Its path is absolute. An empty or relative path resolves against the
|
||||
process cwd, so syncing it would adopt whatever directory the server
|
||||
was launched from as the project root and mutate unrelated files.
|
||||
|
||||
Cloud-only projects (empty/slug path) and cloud projects with a real
|
||||
local bisync copy (absolute path) are handled correctly by these two
|
||||
conditions, so no separate mode check is needed.
|
||||
"""
|
||||
entry = self.projects.get(project_name)
|
||||
return entry is not None and Path(project_path).is_absolute()
|
||||
|
||||
def set_project_mode(self, project_name: str, mode: ProjectMode) -> None:
|
||||
"""Set the routing mode for a project.
|
||||
|
||||
|
||||
@@ -18,6 +18,7 @@ from basic_memory.repository.search_index_row import SearchIndexRow
|
||||
from basic_memory.repository.search_repository_base import (
|
||||
SearchRepositoryBase,
|
||||
VectorChunkState,
|
||||
relaxed_query_words,
|
||||
)
|
||||
from basic_memory.repository.metadata_filters import parse_metadata_filters
|
||||
from basic_memory.repository.semantic_errors import SemanticDependenciesMissingError
|
||||
@@ -67,9 +68,6 @@ class PostgresSearchRepository(SearchRepositoryBase):
|
||||
self._semantic_postgres_prepare_concurrency = (
|
||||
self._app_config.semantic_postgres_prepare_concurrency
|
||||
)
|
||||
self._entity_boost_enabled = self._app_config.search_entity_boost_enabled
|
||||
self._entity_boost_weight = self._app_config.search_entity_boost_weight
|
||||
self._entity_boost_max_terms = self._app_config.search_entity_boost_max_terms
|
||||
self._embedding_provider = embedding_provider
|
||||
self._vector_dimensions = 384
|
||||
self._vector_tables_initialized = False
|
||||
@@ -179,6 +177,14 @@ class PostgresSearchRepository(SearchRepositoryBase):
|
||||
# For non-Boolean queries, prepare single term
|
||||
return self._prepare_single_term(term, is_prefix)
|
||||
|
||||
@staticmethod
|
||||
def _relaxed_tsquery_text(search_text: Optional[str]) -> Optional[str]:
|
||||
"""OR-relaxed tsquery expression for a failed strict query, or None."""
|
||||
words = relaxed_query_words(search_text)
|
||||
if not words:
|
||||
return None
|
||||
return " | ".join(f"{word}:*" for word in words)
|
||||
|
||||
def _prepare_boolean_query(self, query: str) -> str:
|
||||
"""Convert Boolean query to tsquery format.
|
||||
|
||||
@@ -239,7 +245,12 @@ class PostgresSearchRepository(SearchRepositoryBase):
|
||||
|
||||
# Handle multi-word queries
|
||||
if " " in cleaned_term:
|
||||
words = [w for w in cleaned_term.split() if w.strip()]
|
||||
# Strip sentence punctuation from word edges so question-form
|
||||
# queries produce clean lexemes (parity with SQLite FTS5 prep).
|
||||
# The tsquery tokenizer ignores this punctuation anyway; leaving it
|
||||
# in only risks tsquery syntax errors. Interior characters are kept.
|
||||
words = [w.strip("?!.,;") for w in cleaned_term.split()]
|
||||
words = [w for w in words if w]
|
||||
if not words:
|
||||
# All characters were special chars, search won't match anything
|
||||
# Return a safe search term that won't cause syntax errors
|
||||
@@ -252,8 +263,11 @@ class PostgresSearchRepository(SearchRepositoryBase):
|
||||
# Join with AND operator
|
||||
return " & ".join(prepared_words)
|
||||
|
||||
# Single word
|
||||
cleaned_term = cleaned_term.strip()
|
||||
# Single word: strip edge punctuation; guard the now-empty case so a
|
||||
# bare ":*"/"" never reaches tsquery.
|
||||
cleaned_term = cleaned_term.strip().strip("?!.,;")
|
||||
if not cleaned_term:
|
||||
return "NOSPECIALCHARS:*"
|
||||
if is_prefix:
|
||||
return f"{cleaned_term}:*"
|
||||
else:
|
||||
@@ -911,6 +925,7 @@ class PostgresSearchRepository(SearchRepositoryBase):
|
||||
min_similarity: Optional[float] = None,
|
||||
limit: int = 10,
|
||||
offset: int = 0,
|
||||
allow_relaxed: bool = False,
|
||||
) -> List[SearchIndexRow]:
|
||||
"""Search across all indexed content using PostgreSQL tsvector."""
|
||||
# --- Dispatch vector / hybrid modes (shared logic) ---
|
||||
@@ -985,6 +1000,20 @@ class PostgresSearchRepository(SearchRepositoryBase):
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
result = await session.execute(text(sql), params)
|
||||
rows = result.fetchall()
|
||||
# Trigger: multi-word natural-language query matched nothing
|
||||
# under the default all-terms-AND tsquery semantics.
|
||||
# Why: questions rarely have every word in one document;
|
||||
# without relaxation the FTS half of hybrid search contributes
|
||||
# zero candidates (parity with the SQLite path).
|
||||
# Outcome: one retry with OR-joined prefix lexemes; ts_rank
|
||||
# still ranks multi-term matches first.
|
||||
relaxed = (
|
||||
self._relaxed_tsquery_text(search_text) if allow_relaxed and not rows else None
|
||||
)
|
||||
if relaxed and params.get("text"):
|
||||
params["text"] = relaxed
|
||||
result = await session.execute(text(sql), params)
|
||||
rows = result.fetchall()
|
||||
except Exception as e:
|
||||
if self._is_tsquery_syntax_error(e):
|
||||
logger.warning(f"tsquery syntax error for search term: {search_text}, error: {e}")
|
||||
|
||||
@@ -40,69 +40,55 @@ BULLET_PATTERN = re.compile(r"^[\-\*]\s+")
|
||||
OVERSIZED_ENTITY_VECTOR_SHARD_SIZE = 256
|
||||
_SQLITE_MAX_PREPARE_WINDOW = 8
|
||||
|
||||
# Interrogative/function words contribute lexical noise when a strict
|
||||
# full-text query is relaxed: "when OR did OR a" matches loud wrong documents
|
||||
# that displace genuine results from the ranking window.
|
||||
RELAXATION_STOPWORDS = frozenset(
|
||||
"a an and are as at be but by did do does for from had has have how i in is it of on "
|
||||
"or that the their they this to was we were what when where which who whom whose why "
|
||||
"will with you your".split()
|
||||
)
|
||||
|
||||
|
||||
def relaxed_query_words(search_text: Optional[str]) -> Optional[list[str]]:
|
||||
"""Content-bearing words for OR-relaxing a strict full-text query.
|
||||
|
||||
Returns None when relaxation must not apply. These eligibility rules match
|
||||
SearchService._is_relaxed_fts_fallback_eligible so the hybrid FTS branch
|
||||
relaxes exactly the same query shapes as the service-level FTS path:
|
||||
|
||||
- empty / quoted / explicit-boolean queries (user intent is not
|
||||
second-guessed);
|
||||
- fewer than three alphanumeric tokens (short queries like "New Feature"
|
||||
over-broaden under OR — and in hybrid the relaxed FTS-only rows normalize
|
||||
to 1.0 and can outrank the vector result the user wanted);
|
||||
- any pure-digit token ("root note 1", "SPEC 16") — identifier-like queries
|
||||
over-broaden and create false positives under OR.
|
||||
"""
|
||||
if not search_text:
|
||||
return None
|
||||
stripped = search_text.strip()
|
||||
if '"' in stripped or any(op in f" {stripped} " for op in (" AND ", " OR ", " NOT ")):
|
||||
return None
|
||||
# Eligibility checks run on raw alphanumeric tokens (parity with the
|
||||
# service), before stopword filtering.
|
||||
tokens = re.findall(r"[A-Za-z0-9]+", stripped.lower())
|
||||
if len(tokens) < 3 or any(token.isdigit() for token in tokens):
|
||||
return None
|
||||
words = [word.strip("?!.,;:") for word in stripped.split()]
|
||||
words = [
|
||||
word
|
||||
for word in words
|
||||
if word and word.isalnum() and word.lower() not in RELAXATION_STOPWORDS
|
||||
]
|
||||
return words or None
|
||||
|
||||
|
||||
# Entity, observation, and relation rows in search_index carry ids from independent
|
||||
# auto-increment sequences, so a bare id is ambiguous across row types. Every map in
|
||||
# the vector/hybrid retrieval path must key rows by (type, id) to avoid collisions.
|
||||
type SearchIndexKey = tuple[str, int]
|
||||
|
||||
# --- Entity-aware ranking boost (#951) ---
|
||||
|
||||
# Match word tokens (allowing internal apostrophes/hyphens) so we can inspect
|
||||
# their capitalization to detect proper-noun-like query terms.
|
||||
_ENTITY_TERM_TOKEN_PATTERN = re.compile(r"[A-Za-z][A-Za-z'\-]*")
|
||||
|
||||
# Common capitalized sentence-starters and interrogatives that look like proper
|
||||
# nouns but are not entity references. Kept lowercase for case-insensitive checks.
|
||||
# Intentionally small: a candidate term only boosts a row when it actually matches
|
||||
# that row's title/relation names, so a stray non-entity term simply does nothing.
|
||||
_ENTITY_TERM_STOPWORDS = frozenset(
|
||||
{
|
||||
"a",
|
||||
"an",
|
||||
"and",
|
||||
"are",
|
||||
"as",
|
||||
"at",
|
||||
"be",
|
||||
"but",
|
||||
"by",
|
||||
"do",
|
||||
"does",
|
||||
"for",
|
||||
"from",
|
||||
"has",
|
||||
"have",
|
||||
"how",
|
||||
"i",
|
||||
"in",
|
||||
"is",
|
||||
"it",
|
||||
"of",
|
||||
"on",
|
||||
"or",
|
||||
"the",
|
||||
"their",
|
||||
"they",
|
||||
"this",
|
||||
"to",
|
||||
"was",
|
||||
"we",
|
||||
"were",
|
||||
"what",
|
||||
"when",
|
||||
"where",
|
||||
"which",
|
||||
"who",
|
||||
"whom",
|
||||
"whose",
|
||||
"why",
|
||||
"will",
|
||||
"with",
|
||||
"you",
|
||||
"your",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class VectorSyncBatchResult:
|
||||
@@ -224,13 +210,6 @@ class SearchRepositoryBase(ABC):
|
||||
_vector_dimensions: int
|
||||
_vector_tables_initialized: bool
|
||||
|
||||
# Entity-aware ranking boost (#951). Defaults keep the feature off for any
|
||||
# subclass or test double that does not explicitly configure it. Concrete
|
||||
# backends overwrite these from BasicMemoryConfig in their __init__.
|
||||
_entity_boost_enabled: bool = False
|
||||
_entity_boost_weight: float = 0.0
|
||||
_entity_boost_max_terms: int = 1
|
||||
|
||||
def __init__(self, session_maker: async_sessionmaker[AsyncSession], project_id: int):
|
||||
"""Initialize with session maker and project_id filter.
|
||||
|
||||
@@ -294,6 +273,7 @@ class SearchRepositoryBase(ABC):
|
||||
min_similarity: Optional[float] = None,
|
||||
limit: int = 10,
|
||||
offset: int = 0,
|
||||
allow_relaxed: bool = False,
|
||||
) -> List[SearchIndexRow]:
|
||||
"""Search across all indexed content.
|
||||
|
||||
@@ -2212,105 +2192,6 @@ class SearchRepositoryBase(ABC):
|
||||
# Shared semantic search: hybrid score-based fusion
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
# --- Entity-aware ranking boost (#951) ---
|
||||
|
||||
@staticmethod
|
||||
def _extract_query_entity_terms(search_text: Optional[str]) -> set[str]:
|
||||
"""Extract candidate entity (proper-noun) terms from a query string.
|
||||
|
||||
Heuristic, lexical only (no model inference): a token is a candidate entity
|
||||
term when it is title-cased or all-caps and is not a common stopword. The
|
||||
result is lowercased so downstream matching is case-insensitive.
|
||||
|
||||
Examples:
|
||||
"What are Joanna's hobbies?" -> {"joanna"}
|
||||
"Who is Anthony?" -> {"anthony"}
|
||||
"Deborah and Jolene" -> {"deborah", "jolene"}
|
||||
"what is the weather" -> set() (no proper nouns)
|
||||
"""
|
||||
if not search_text:
|
||||
return set()
|
||||
|
||||
terms: set[str] = set()
|
||||
for match in _ENTITY_TERM_TOKEN_PATTERN.finditer(search_text):
|
||||
token = match.group(0)
|
||||
# Trigger: token begins with an uppercase letter (Title-Case or ALL-CAPS).
|
||||
# Why: proper nouns and named entities are conventionally capitalized; this
|
||||
# is the cheapest reliable signal without a NER model.
|
||||
# Outcome: lowercase, non-capitalized words are ignored as generic terms.
|
||||
if not token[0].isupper():
|
||||
continue
|
||||
normalized = token.lower()
|
||||
# Strip a trailing possessive so "Joanna's" matches the entity "Joanna".
|
||||
if normalized.endswith("'s"):
|
||||
normalized = normalized[:-2]
|
||||
if normalized in _ENTITY_TERM_STOPWORDS:
|
||||
continue
|
||||
# Single characters (e.g. a stray "I") carry no entity signal.
|
||||
if len(normalized) < 2:
|
||||
continue
|
||||
terms.add(normalized)
|
||||
return terms
|
||||
|
||||
@staticmethod
|
||||
def _row_entity_match_count(row: SearchIndexRow, entity_terms: set[str]) -> int:
|
||||
"""Count distinct query entity terms that a candidate row references.
|
||||
|
||||
Matches against the row's own entity name (title) and the names embedded in
|
||||
a relation row's title (``"From -> To"``). These are the fields where Basic
|
||||
Memory's first-class entity names surface, so a match here is strong evidence
|
||||
the candidate is about the queried entity rather than a same-topic document.
|
||||
"""
|
||||
if not entity_terms:
|
||||
return 0
|
||||
|
||||
haystack_parts = [row.title or ""]
|
||||
# Relation rows encode linked entity names in their title ("From -> To");
|
||||
# the relation_type itself is not an entity name, so it is excluded.
|
||||
haystack = " ".join(part for part in haystack_parts if part)
|
||||
if not haystack:
|
||||
return 0
|
||||
|
||||
haystack_tokens: set[str] = set()
|
||||
for match in _ENTITY_TERM_TOKEN_PATTERN.finditer(haystack):
|
||||
token = match.group(0).lower()
|
||||
# Mirror the query-side possessive stripping so a doc titled
|
||||
# "Joanna's Hobbies" matches the query entity term "joanna".
|
||||
if token.endswith("'s"):
|
||||
token = token[:-2]
|
||||
haystack_tokens.add(token)
|
||||
return len(entity_terms & haystack_tokens)
|
||||
|
||||
def _apply_entity_boost(
|
||||
self,
|
||||
fused_scores: dict[SearchIndexKey, float],
|
||||
rows_by_key: dict[SearchIndexKey, SearchIndexRow],
|
||||
entity_terms: set[str],
|
||||
) -> dict[SearchIndexKey, float]:
|
||||
"""Multiply fused scores by a per-matched-term bonus for entity-matching rows.
|
||||
|
||||
Trigger: entity boosting is enabled and the query contains proper-noun terms.
|
||||
Why: a candidate whose entity/relation names contain a queried proper noun is a
|
||||
stronger answer than a generic same-topic document (#951 cross-conversation
|
||||
confusion).
|
||||
Outcome: ``score * (1 + weight * min(matches, max_terms))``. Rows that match no
|
||||
query entity term are returned unchanged, so relative order among non-matching
|
||||
rows is preserved.
|
||||
"""
|
||||
if not self._entity_boost_enabled or not entity_terms or self._entity_boost_weight <= 0:
|
||||
return fused_scores
|
||||
|
||||
boosted: dict[SearchIndexKey, float] = {}
|
||||
for row_key, score in fused_scores.items():
|
||||
row = rows_by_key.get(row_key)
|
||||
matches = self._row_entity_match_count(row, entity_terms) if row is not None else 0
|
||||
if matches <= 0:
|
||||
boosted[row_key] = score
|
||||
continue
|
||||
capped_matches = min(matches, self._entity_boost_max_terms)
|
||||
boosted[row_key] = score * (1.0 + self._entity_boost_weight * capped_matches)
|
||||
return boosted
|
||||
|
||||
async def _search_hybrid(
|
||||
self,
|
||||
*,
|
||||
@@ -2338,6 +2219,9 @@ class SearchRepositoryBase(ABC):
|
||||
query_start = time.perf_counter()
|
||||
candidate_limit = max(self._semantic_vector_k, (limit + offset) * 10)
|
||||
fts_start = time.perf_counter()
|
||||
# allow_relaxed: question-form queries rarely AND-match, and a dead FTS
|
||||
# branch silently degrades hybrid to vector-only ranking. Fusion plus
|
||||
# bm25 keep relaxed lexical candidates from dominating precision.
|
||||
fts_results = await self.search(
|
||||
search_text=search_text,
|
||||
permalink=permalink,
|
||||
@@ -2351,6 +2235,7 @@ class SearchRepositoryBase(ABC):
|
||||
retrieval_mode=SearchRetrievalMode.FTS,
|
||||
limit=candidate_limit,
|
||||
offset=0,
|
||||
allow_relaxed=True,
|
||||
)
|
||||
fts_ms = (time.perf_counter() - fts_start) * 1000
|
||||
vector_start = time.perf_counter()
|
||||
@@ -2414,15 +2299,6 @@ class SearchRepositoryBase(ABC):
|
||||
f = fts_scores.get(row_key, 0.0)
|
||||
fused_scores[row_key] = max(v, f) + FUSION_BONUS * min(v, f)
|
||||
|
||||
# Entity-aware ranking boost (#951): runs over the full fused candidate set
|
||||
# before the limit/offset cut, so a boosted entity-matching candidate can be
|
||||
# promoted into the returned window. No-op when the feature is disabled or the
|
||||
# query contains no proper-noun terms, preserving the existing ordering.
|
||||
entity_terms = (
|
||||
self._extract_query_entity_terms(query_text) if self._entity_boost_enabled else set()
|
||||
)
|
||||
fused_scores = self._apply_entity_boost(fused_scores, rows_by_key, entity_terms)
|
||||
|
||||
ranked = sorted(fused_scores.items(), key=lambda item: item[1], reverse=True)
|
||||
output: list[SearchIndexRow] = []
|
||||
for row_key, fused_score in ranked[offset : offset + limit]:
|
||||
|
||||
@@ -23,7 +23,10 @@ from basic_memory.models.search import (
|
||||
from basic_memory.repository.embedding_provider import EmbeddingProvider
|
||||
from basic_memory.repository.embedding_provider_factory import create_embedding_provider
|
||||
from basic_memory.repository.search_index_row import SearchIndexRow
|
||||
from basic_memory.repository.search_repository_base import SearchRepositoryBase
|
||||
from basic_memory.repository.search_repository_base import (
|
||||
SearchRepositoryBase,
|
||||
relaxed_query_words,
|
||||
)
|
||||
from basic_memory.repository.metadata_filters import parse_metadata_filters, build_sqlite_json_path
|
||||
from basic_memory.repository.semantic_errors import SemanticDependenciesMissingError
|
||||
from basic_memory.schemas.search import SearchItemType, SearchRetrievalMode
|
||||
@@ -55,9 +58,6 @@ class SQLiteSearchRepository(SearchRepositoryBase):
|
||||
self._semantic_embedding_sync_batch_size = (
|
||||
self._app_config.semantic_embedding_sync_batch_size
|
||||
)
|
||||
self._entity_boost_enabled = self._app_config.search_entity_boost_enabled
|
||||
self._entity_boost_weight = self._app_config.search_entity_boost_weight
|
||||
self._entity_boost_max_terms = self._app_config.search_entity_boost_max_terms
|
||||
self._embedding_provider = embedding_provider
|
||||
self._sqlite_vec_load_lock = asyncio.Lock()
|
||||
self._sqlite_prepare_write_lock = asyncio.Lock()
|
||||
@@ -258,6 +258,19 @@ class SQLiteSearchRepository(SearchRepositoryBase):
|
||||
if "*" in term and all(c.isalnum() or c in "*_-" for c in term):
|
||||
return term
|
||||
|
||||
# Natural-language queries arrive with sentence punctuation that FTS5
|
||||
# treats as syntax ("When did Melanie paint a sunrise?"). The tokenizer
|
||||
# ignores this punctuation in the INDEX, so stripping it from word
|
||||
# edges loses nothing — but leaving it forces the whole question into
|
||||
# an exact-phrase match that returns zero rows, silently disabling the
|
||||
# FTS half of hybrid search. Interior characters (hyphens, slashes —
|
||||
# permalinks and paths) are untouched.
|
||||
if " " in term:
|
||||
words = [word.strip("?!.,;:") for word in term.split()]
|
||||
term = " ".join(word for word in words if word)
|
||||
if not term:
|
||||
return ""
|
||||
|
||||
# Characters that can cause FTS5 syntax errors when used as operators
|
||||
# We're more conservative here - only quote when we detect problematic patterns
|
||||
problematic_chars = [
|
||||
@@ -354,6 +367,14 @@ class SQLiteSearchRepository(SearchRepositoryBase):
|
||||
# For non-Boolean queries, use the single term preparation logic
|
||||
return self._prepare_single_term(term, is_prefix)
|
||||
|
||||
@staticmethod
|
||||
def _relaxed_fts_text(search_text: Optional[str]) -> Optional[str]:
|
||||
"""OR-relaxed FTS5 expression for a failed strict query, or None."""
|
||||
words = relaxed_query_words(search_text)
|
||||
if not words:
|
||||
return None
|
||||
return " OR ".join(f"{word}*" for word in words)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# sqlite-vec extension loading (SQLite-specific)
|
||||
# ------------------------------------------------------------------
|
||||
@@ -956,8 +977,15 @@ class SQLiteSearchRepository(SearchRepositoryBase):
|
||||
min_similarity: Optional[float] = None,
|
||||
limit: int = 10,
|
||||
offset: int = 0,
|
||||
allow_relaxed: bool = False,
|
||||
) -> List[SearchIndexRow]:
|
||||
"""Search across all indexed content using SQLite FTS5."""
|
||||
"""Search across all indexed content using SQLite FTS5.
|
||||
|
||||
``allow_relaxed=True`` retries a zero-result strict multi-word query
|
||||
with OR-joined content terms. Only the hybrid path opts in: its FTS
|
||||
branch otherwise contributes nothing for question-form queries.
|
||||
Service-level FTS searches keep their own conservative fallback.
|
||||
"""
|
||||
# --- Dispatch vector / hybrid modes (shared logic) ---
|
||||
dispatched = await self._dispatch_retrieval_mode(
|
||||
search_text=search_text,
|
||||
@@ -1024,6 +1052,21 @@ class SQLiteSearchRepository(SearchRepositoryBase):
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
result = await session.execute(text(sql), params)
|
||||
rows = result.fetchall()
|
||||
# Trigger: multi-word natural-language query matched nothing
|
||||
# under the default all-terms-AND semantics.
|
||||
# Why: questions ("when did X do Y") rarely have every word in
|
||||
# one document; without relaxation the FTS half of hybrid
|
||||
# search contributes zero candidates and ranking degrades to
|
||||
# vector-only.
|
||||
# Outcome: one retry with OR-joined prefix terms; bm25 still
|
||||
# ranks multi-term matches first.
|
||||
relaxed = (
|
||||
self._relaxed_fts_text(search_text) if allow_relaxed and not rows else None
|
||||
)
|
||||
if relaxed and params.get("text"):
|
||||
params["text"] = relaxed
|
||||
result = await session.execute(text(sql), params)
|
||||
rows = result.fetchall()
|
||||
except Exception as e:
|
||||
# Handle FTS5 syntax errors and provide user-friendly feedback
|
||||
if self._is_fts5_syntax_error(e): # pragma: no cover
|
||||
|
||||
@@ -13,7 +13,7 @@ from pathlib import Path
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory import db
|
||||
from basic_memory.config import BasicMemoryConfig, DatabaseBackend, ProjectMode
|
||||
from basic_memory.config import BasicMemoryConfig, DatabaseBackend
|
||||
from basic_memory.models import Project
|
||||
from basic_memory.repository import (
|
||||
ProjectRepository,
|
||||
@@ -120,18 +120,14 @@ async def initialize_file_sync(
|
||||
active_projects = [p for p in active_projects if p.name == constrained_project]
|
||||
logger.info(f"Background sync constrained to project: {constrained_project}")
|
||||
|
||||
# Skip cloud-mode projects that have no local directory.
|
||||
# Cloud projects with a local bisync copy (absolute path) are kept for local sync.
|
||||
cloud_skip = []
|
||||
for p in active_projects:
|
||||
if app_config.get_project_mode(p.name) == ProjectMode.CLOUD:
|
||||
entry = app_config.projects.get(p.name)
|
||||
if entry and Path(entry.path).is_absolute():
|
||||
continue # Cloud project with local bisync copy — keep for local sync
|
||||
cloud_skip.append(p.name)
|
||||
if cloud_skip:
|
||||
active_projects = [p for p in active_projects if p.name not in cloud_skip]
|
||||
logger.info(f"Skipping cloud-mode projects for local sync: {cloud_skip}")
|
||||
# Only sync projects that are in config (source of truth) and have an
|
||||
# absolute local path; see BasicMemoryConfig.is_locally_syncable. This keeps
|
||||
# background sync from adopting the process cwd as a project root and
|
||||
# mutating unrelated files (issue #949).
|
||||
skip = [p.name for p in active_projects if not app_config.is_locally_syncable(p.name, p.path)]
|
||||
if skip:
|
||||
active_projects = [p for p in active_projects if p.name not in skip]
|
||||
logger.info(f"Skipping projects that are not locally syncable for sync: {skip}")
|
||||
|
||||
# Start sync for all projects as background tasks (non-blocking)
|
||||
async def sync_project_background(project: Project):
|
||||
|
||||
@@ -10,7 +10,7 @@ from typing import List, Optional, Set, Sequence, Callable, Awaitable, TYPE_CHEC
|
||||
if TYPE_CHECKING:
|
||||
from basic_memory.sync.sync_service import SyncService
|
||||
|
||||
from basic_memory.config import BasicMemoryConfig, ProjectMode, WATCH_STATUS_JSON
|
||||
from basic_memory.config import BasicMemoryConfig, WATCH_STATUS_JSON
|
||||
from basic_memory.ignore_utils import load_gitignore_patterns, should_ignore_path
|
||||
from basic_memory.models import Project
|
||||
from basic_memory.repository import ProjectRepository
|
||||
@@ -184,24 +184,24 @@ class WatchService:
|
||||
``--project``, only that project is watched. This keeps concurrent
|
||||
MCP processes from producing duplicate watchers that race on the
|
||||
same files.
|
||||
2. Cloud-only projects without a local bisync copy are skipped so we
|
||||
don't watch a path that does not exist on disk.
|
||||
2. Projects that are not locally syncable are skipped — those missing
|
||||
from config (config is the source of truth, so stale DB rows must
|
||||
not be watched) or with a non-absolute path (which would resolve
|
||||
against the process cwd and make the watcher observe and mutate the
|
||||
directory the server was launched from). See
|
||||
``BasicMemoryConfig.is_locally_syncable`` (issue #949). Cloud
|
||||
projects with a local bisync copy keep their absolute path and are
|
||||
still watched.
|
||||
"""
|
||||
projects = await self.project_repository.get_active_projects()
|
||||
|
||||
if self.constrained_project:
|
||||
projects = [p for p in projects if p.name == self.constrained_project]
|
||||
|
||||
cloud_skip: list[str] = []
|
||||
for p in projects:
|
||||
if self.app_config.get_project_mode(p.name) == ProjectMode.CLOUD:
|
||||
entry = self.app_config.projects.get(p.name)
|
||||
if entry and Path(entry.path).is_absolute():
|
||||
continue # Cloud project with local bisync copy — keep watching
|
||||
cloud_skip.append(p.name)
|
||||
if cloud_skip:
|
||||
projects = [p for p in projects if p.name not in cloud_skip]
|
||||
logger.debug(f"Skipping cloud-mode projects in watch cycle: {cloud_skip}")
|
||||
skip = [p.name for p in projects if not self.app_config.is_locally_syncable(p.name, p.path)]
|
||||
if skip:
|
||||
projects = [p for p in projects if p.name not in skip]
|
||||
logger.debug(f"Skipping projects that are not locally syncable in watch cycle: {skip}")
|
||||
|
||||
return list(projects)
|
||||
|
||||
|
||||
@@ -1,226 +0,0 @@
|
||||
"""Service-level integration test for the entity-aware ranking boost (#951).
|
||||
|
||||
Drives a fully wired SearchService over a real database with a deterministic stub
|
||||
embedding provider so vector similarity is controlled. Verifies that when the boost
|
||||
is enabled, an entity-matching document outranks a higher-similarity non-matching
|
||||
document, and that ordering is unchanged when the boost is disabled.
|
||||
|
||||
No model inference is involved: the stub provider returns fixed unit vectors, so the
|
||||
test is fast and deterministic on both SQLite and Postgres.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
|
||||
from basic_memory.config import BasicMemoryConfig
|
||||
from basic_memory.repository.entity_repository import EntityRepository
|
||||
from basic_memory.repository.search_repository import SearchRepository
|
||||
from basic_memory.schemas.base import Entity as EntitySchema
|
||||
from basic_memory.schemas.search import SearchQuery, SearchRetrievalMode
|
||||
from basic_memory.services.entity_service import EntityService
|
||||
from basic_memory.services.file_service import FileService
|
||||
from basic_memory.services.search_service import SearchService
|
||||
|
||||
|
||||
# --- Deterministic stub embedding provider ---
|
||||
|
||||
_STUB_DIMENSIONS = 4
|
||||
|
||||
|
||||
def _unit(vector: list[float]) -> list[float]:
|
||||
norm = math.sqrt(sum(component * component for component in vector)) or 1.0
|
||||
return [component / norm for component in vector]
|
||||
|
||||
|
||||
class _StubEmbeddingProvider:
|
||||
"""Maps known text fragments to fixed unit vectors for controlled similarity.
|
||||
|
||||
The query is engineered to sit closer (cosine) to the non-matching "hobbies"
|
||||
document than to the gold "Joanna" document, reproducing the #951 failure where
|
||||
generic semantic similarity outranks the entity-matching gold doc.
|
||||
"""
|
||||
|
||||
model_name = "stub-entity-boost"
|
||||
dimensions = _STUB_DIMENSIONS
|
||||
|
||||
def _vector_for(self, text: str) -> list[float]:
|
||||
lowered = text.lower()
|
||||
if "joanna" in lowered:
|
||||
# Gold doc: shares some direction with the query but less than the decoy.
|
||||
return _unit([0.6, 0.8, 0.0, 0.0])
|
||||
if "hobbies" in lowered or "pastime" in lowered:
|
||||
# Decoy doc: closest to the query direction.
|
||||
return _unit([0.95, 0.31, 0.0, 0.0])
|
||||
return _unit([0.0, 0.0, 1.0, 0.0])
|
||||
|
||||
async def embed_query(self, text: str) -> list[float]:
|
||||
# Query direction is closest to the decoy vector above.
|
||||
return _unit([0.97, 0.24, 0.0, 0.0])
|
||||
|
||||
async def embed_documents(self, texts: list[str]) -> list[list[float]]:
|
||||
return [self._vector_for(text) for text in texts]
|
||||
|
||||
def runtime_log_attrs(self) -> dict[str, Any]:
|
||||
return {}
|
||||
|
||||
|
||||
# --- Fixtures ---
|
||||
|
||||
|
||||
async def _build_search_service(
|
||||
*,
|
||||
session_maker,
|
||||
test_project,
|
||||
base_app_config: BasicMemoryConfig,
|
||||
file_service: FileService,
|
||||
entity_repository: EntityRepository,
|
||||
boost_enabled: bool,
|
||||
) -> SearchService:
|
||||
"""Build a SearchService with semantic search + a deterministic stub provider."""
|
||||
from basic_memory.repository.sqlite_search_repository import SQLiteSearchRepository
|
||||
from basic_memory.repository.postgres_search_repository import PostgresSearchRepository
|
||||
from basic_memory.config import DatabaseBackend
|
||||
|
||||
app_config = base_app_config.model_copy(
|
||||
update={
|
||||
"semantic_search_enabled": True,
|
||||
"semantic_min_similarity": 0.0,
|
||||
"search_entity_boost_enabled": boost_enabled,
|
||||
"search_entity_boost_weight": 0.3,
|
||||
"search_entity_boost_max_terms": 3,
|
||||
}
|
||||
)
|
||||
|
||||
provider = _StubEmbeddingProvider()
|
||||
if app_config.database_backend == DatabaseBackend.POSTGRES: # pragma: no cover
|
||||
search_repo: SearchRepository = PostgresSearchRepository(
|
||||
session_maker,
|
||||
project_id=test_project.id,
|
||||
app_config=app_config,
|
||||
embedding_provider=provider,
|
||||
)
|
||||
else:
|
||||
# Pass the stub provider at construction time so __init__ does not
|
||||
# instantiate the real configured provider when semantic_search_enabled=True.
|
||||
repo = SQLiteSearchRepository(
|
||||
session_maker,
|
||||
project_id=test_project.id,
|
||||
app_config=app_config,
|
||||
embedding_provider=provider,
|
||||
)
|
||||
search_repo = repo
|
||||
|
||||
service = SearchService(search_repo, entity_repository, file_service)
|
||||
await service.init_search_index()
|
||||
return service
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def boost_entities(
|
||||
entity_service: EntityService,
|
||||
):
|
||||
"""Index two entities: a decoy 'hobbies' doc and the gold 'Joanna' doc."""
|
||||
decoy, _ = await entity_service.create_or_update_entity(
|
||||
EntitySchema(
|
||||
title="Common Hobbies and Pastimes",
|
||||
note_type="note",
|
||||
directory="people",
|
||||
content="A general overview of hobbies and pastimes people enjoy.",
|
||||
)
|
||||
)
|
||||
gold, _ = await entity_service.create_or_update_entity(
|
||||
EntitySchema(
|
||||
title="Joanna",
|
||||
note_type="note",
|
||||
directory="people",
|
||||
content="Notes about Joanna and what she likes to do.",
|
||||
)
|
||||
)
|
||||
return decoy, gold
|
||||
|
||||
|
||||
# --- Tests ---
|
||||
|
||||
|
||||
async def _sync_vectors(service: SearchService, entity_ids: list[int]) -> None:
|
||||
"""Embed the indexed entities via the stub provider."""
|
||||
await service.sync_entity_vectors_batch(entity_ids)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_entity_boost_enabled_promotes_gold_doc(
|
||||
session_maker,
|
||||
test_project,
|
||||
app_config,
|
||||
file_service,
|
||||
entity_repository,
|
||||
boost_entities,
|
||||
):
|
||||
decoy, gold = boost_entities
|
||||
service = await _build_search_service(
|
||||
session_maker=session_maker,
|
||||
test_project=test_project,
|
||||
base_app_config=app_config,
|
||||
file_service=file_service,
|
||||
entity_repository=entity_repository,
|
||||
boost_enabled=True,
|
||||
)
|
||||
# Re-index the entities through this service so vector tables exist for it.
|
||||
for entity in (decoy, gold):
|
||||
await service.index_entity(entity)
|
||||
await _sync_vectors(service, [decoy.id, gold.id])
|
||||
|
||||
results = await service.search(
|
||||
SearchQuery(
|
||||
text="What are Joanna's hobbies?",
|
||||
retrieval_mode=SearchRetrievalMode.HYBRID,
|
||||
),
|
||||
limit=10,
|
||||
)
|
||||
|
||||
entity_ids = [r.entity_id for r in results]
|
||||
assert gold.id in entity_ids and decoy.id in entity_ids
|
||||
# With the boost on, the entity-matching gold doc ranks ahead of the
|
||||
# higher-similarity decoy.
|
||||
assert entity_ids.index(gold.id) < entity_ids.index(decoy.id)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_entity_boost_disabled_keeps_similarity_order(
|
||||
session_maker,
|
||||
test_project,
|
||||
app_config,
|
||||
file_service,
|
||||
entity_repository,
|
||||
boost_entities,
|
||||
):
|
||||
decoy, gold = boost_entities
|
||||
service = await _build_search_service(
|
||||
session_maker=session_maker,
|
||||
test_project=test_project,
|
||||
base_app_config=app_config,
|
||||
file_service=file_service,
|
||||
entity_repository=entity_repository,
|
||||
boost_enabled=False,
|
||||
)
|
||||
for entity in (decoy, gold):
|
||||
await service.index_entity(entity)
|
||||
await _sync_vectors(service, [decoy.id, gold.id])
|
||||
|
||||
results = await service.search(
|
||||
SearchQuery(
|
||||
text="What are Joanna's hobbies?",
|
||||
retrieval_mode=SearchRetrievalMode.HYBRID,
|
||||
),
|
||||
limit=10,
|
||||
)
|
||||
|
||||
entity_ids = [r.entity_id for r in results]
|
||||
assert gold.id in entity_ids and decoy.id in entity_ids
|
||||
# With the boost off, the higher-similarity decoy ranks ahead of the gold doc.
|
||||
assert entity_ids.index(decoy.id) < entity_ids.index(gold.id)
|
||||
@@ -77,6 +77,7 @@ class ConcreteSearchRepo(SearchRepositoryBase):
|
||||
min_similarity: Optional[float] = None,
|
||||
limit: int = 10,
|
||||
offset: int = 0,
|
||||
allow_relaxed: bool = False,
|
||||
) -> list[SearchIndexRow]:
|
||||
return [] # pragma: no cover
|
||||
|
||||
@@ -134,115 +135,6 @@ HYBRID_KWARGS: dict[str, Any] = dict(
|
||||
)
|
||||
|
||||
|
||||
def _hybrid_kwargs(**overrides: Any) -> dict[str, Any]:
|
||||
"""Return HYBRID_KWARGS with overrides applied, typed as dict[str, Any].
|
||||
|
||||
Keeps the splat into the keyword-only _search_hybrid signature type-clean.
|
||||
"""
|
||||
merged: dict[str, Any] = {**HYBRID_KWARGS, **overrides}
|
||||
return merged
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_entity_boost_promotes_matching_doc_when_enabled():
|
||||
"""With entity boost enabled, an entity-matching doc outranks a higher-similarity
|
||||
non-matching doc.
|
||||
|
||||
Reproduces the #951 cross-conversation confusion: a generic same-topic document
|
||||
(higher raw similarity) initially outranks the gold doc whose title names the
|
||||
queried entity. Enabling the boost flips the order.
|
||||
"""
|
||||
repo = ConcreteSearchRepo()
|
||||
repo._entity_boost_enabled = True
|
||||
repo._entity_boost_weight = 0.15
|
||||
repo._entity_boost_max_terms = 3
|
||||
|
||||
# Row 1: generic hobbies doc from the wrong conversation, higher vector similarity.
|
||||
# Row 2: the gold doc whose title names the queried entity "Joanna".
|
||||
fts_results = []
|
||||
vector_results = [
|
||||
FakeRow(id=1, score=0.80, title="Hobbies and pastimes"),
|
||||
FakeRow(id=2, score=0.72, title="Joanna profile"),
|
||||
]
|
||||
|
||||
with (
|
||||
patch.object(repo, "search", new_callable=AsyncMock, return_value=fts_results),
|
||||
patch.object(
|
||||
repo, "_search_vector_only", new_callable=AsyncMock, return_value=vector_results
|
||||
),
|
||||
):
|
||||
results = await repo._search_hybrid(
|
||||
**_hybrid_kwargs(search_text="What are Joanna's hobbies?")
|
||||
)
|
||||
|
||||
# Boost: row 2 -> 0.72 * 1.15 = 0.828 > row 1's 0.80
|
||||
assert [r.id for r in results] == [2, 1]
|
||||
assert results[0].score == pytest.approx(0.72 * 1.15, rel=1e-6)
|
||||
assert results[1].score == pytest.approx(0.80, rel=1e-6)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_entity_boost_disabled_preserves_ordering():
|
||||
"""With entity boost disabled (default), ordering matches pure similarity."""
|
||||
repo = ConcreteSearchRepo()
|
||||
# Defaults from the base class keep boosting off; assert explicitly.
|
||||
assert repo._entity_boost_enabled is False
|
||||
|
||||
fts_results = []
|
||||
vector_results = [
|
||||
FakeRow(id=1, score=0.80, title="Hobbies and pastimes"),
|
||||
FakeRow(id=2, score=0.72, title="Joanna profile"),
|
||||
]
|
||||
|
||||
with (
|
||||
patch.object(repo, "search", new_callable=AsyncMock, return_value=fts_results),
|
||||
patch.object(
|
||||
repo, "_search_vector_only", new_callable=AsyncMock, return_value=vector_results
|
||||
),
|
||||
):
|
||||
results = await repo._search_hybrid(
|
||||
**_hybrid_kwargs(search_text="What are Joanna's hobbies?")
|
||||
)
|
||||
|
||||
# No boost: original similarity order is preserved, scores unchanged.
|
||||
assert [r.id for r in results] == [1, 2]
|
||||
assert results[0].score == pytest.approx(0.80, rel=1e-6)
|
||||
assert results[1].score == pytest.approx(0.72, rel=1e-6)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_entity_boost_promotes_doc_into_limited_window():
|
||||
"""Boosting runs before the limit cut, so a matching doc ranked below the cutoff
|
||||
can be promoted into the returned window."""
|
||||
repo = ConcreteSearchRepo()
|
||||
repo._entity_boost_enabled = True
|
||||
repo._entity_boost_weight = 0.6
|
||||
repo._entity_boost_max_terms = 3
|
||||
|
||||
fts_results = []
|
||||
# Three non-matching docs above the gold doc, which matches "Anthony".
|
||||
vector_results = [
|
||||
FakeRow(id=1, score=0.90, title="conversation six"),
|
||||
FakeRow(id=2, score=0.85, title="conversation one"),
|
||||
FakeRow(id=3, score=0.60, title="Anthony introduces himself"),
|
||||
]
|
||||
|
||||
with (
|
||||
patch.object(repo, "search", new_callable=AsyncMock, return_value=fts_results),
|
||||
patch.object(
|
||||
repo, "_search_vector_only", new_callable=AsyncMock, return_value=vector_results
|
||||
),
|
||||
):
|
||||
results = await repo._search_hybrid(
|
||||
**_hybrid_kwargs(search_text="Who is Anthony?", limit=1)
|
||||
)
|
||||
|
||||
# Gold doc boost: 0.60 * 1.6 = 0.96 > row 1's 0.90, so it is promoted into the
|
||||
# top-1 window even though it was ranked third before boosting.
|
||||
assert len(results) == 1
|
||||
assert results[0].id == 3
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_high_fts_score_boosts_ranking():
|
||||
"""FTS-only: a high normalized score should outscore a low normalized score."""
|
||||
|
||||
@@ -1001,3 +1001,59 @@ async def test_postgres_search_categories_exact_match(session_maker, test_projec
|
||||
# Multiple categories union.
|
||||
multi = await repo.search(categories=["requirement", "decision"])
|
||||
assert {r.id for r in multi} == {70101, 70102}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_postgres_question_punctuation_and_relaxation(session_maker, test_project):
|
||||
"""Question-form queries must produce clean lexemes and a usable relaxation.
|
||||
|
||||
Parity with SQLite: sentence punctuation previously reached tsquery terms,
|
||||
and a strict all-AND miss had no relaxed retry, silently disabling the FTS
|
||||
half of hybrid search for natural-language questions.
|
||||
"""
|
||||
repo = PostgresSearchRepository(session_maker, project_id=test_project.id)
|
||||
|
||||
# Edge punctuation stripped before lexeme formatting.
|
||||
prepared = repo._prepare_search_term("When did Melanie paint a sunrise?")
|
||||
assert "?" not in prepared
|
||||
assert "sunrise:*" in prepared
|
||||
|
||||
# Relaxation drops stopwords and OR-joins content terms.
|
||||
relaxed = repo._relaxed_tsquery_text("When did Melanie paint a sunrise?")
|
||||
assert relaxed == "Melanie:* | paint:* | sunrise:*"
|
||||
|
||||
# User intent is not second-guessed.
|
||||
assert repo._relaxed_tsquery_text("alpha AND beta") is None
|
||||
assert repo._relaxed_tsquery_text('"exact phrase"') is None
|
||||
assert repo._relaxed_tsquery_text(None) is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_postgres_multiword_query_relaxes_on_strict_miss(session_maker, test_project):
|
||||
repo = PostgresSearchRepository(session_maker, project_id=test_project.id)
|
||||
now = datetime.now(timezone.utc)
|
||||
await repo.index_item(
|
||||
SearchIndexRow(
|
||||
project_id=test_project.id,
|
||||
id=77,
|
||||
title="Trip plans",
|
||||
content_stems="melanie painted a sunrise over the lake last year",
|
||||
content_snippet="Melanie painted a sunrise over the lake last year.",
|
||||
permalink="docs/trip-plans",
|
||||
file_path="docs/trip-plans.md",
|
||||
type="entity",
|
||||
metadata={"note_type": "note"},
|
||||
created_at=now,
|
||||
updated_at=now,
|
||||
)
|
||||
)
|
||||
|
||||
# A content word absent from the doc ("hiking") makes the strict
|
||||
# all-terms-AND query miss even after Postgres drops stopwords — without
|
||||
# it, to_tsquery('english', ...) already strips "when/did/a" and matches.
|
||||
strict = await repo.search(search_text="Did Melanie go hiking at sunrise?")
|
||||
assert strict == []
|
||||
|
||||
# The hybrid FTS branch opts in; OR-relaxation surfaces the partial match.
|
||||
results = await repo.search(search_text="Did Melanie go hiking at sunrise?", allow_relaxed=True)
|
||||
assert any(r.id == 77 for r in results)
|
||||
|
||||
@@ -1124,3 +1124,83 @@ async def test_search_categories_exact_match(search_repository, search_entity):
|
||||
# Multiple categories union: both observations come back.
|
||||
multi = await search_repository.search(categories=["requirement", "decision"])
|
||||
assert {r.id for r in multi} == {70001, 70002}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_question_punctuation_does_not_phrase_quote(search_repository):
|
||||
"""Sentence punctuation must not force exact-phrase matching (#hybrid-fts).
|
||||
|
||||
'When did Melanie paint a sunrise?' previously became the FTS5 phrase
|
||||
'"When did Melanie paint a sunrise?"*' — zero rows for any corpus — which
|
||||
silently disabled the FTS half of hybrid search for question queries.
|
||||
"""
|
||||
prepared = search_repository._prepare_single_term("When did Melanie paint a sunrise?")
|
||||
assert '"' not in prepared
|
||||
# Prefix syntax differs by backend: FTS5 uses '*', tsquery uses ':*'.
|
||||
if is_postgres_backend(search_repository):
|
||||
assert "sunrise:*" in prepared
|
||||
else:
|
||||
assert "sunrise*" in prepared
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_relaxed_query_drops_stopwords(search_repository):
|
||||
"""Relaxation keys on content-bearing terms in each backend's syntax."""
|
||||
if is_postgres_backend(search_repository):
|
||||
relaxed = search_repository._relaxed_tsquery_text("When did Melanie paint a sunrise?")
|
||||
assert relaxed == "Melanie:* | paint:* | sunrise:*"
|
||||
else:
|
||||
relaxed = search_repository._relaxed_fts_text("When did Melanie paint a sunrise?")
|
||||
assert relaxed == "Melanie* OR paint* OR sunrise*"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_relaxed_query_respects_user_intent(search_repository):
|
||||
# Eligibility matches the service-level relaxation (both backends): quoted,
|
||||
# boolean, short (<3 tokens), and numeric-identifier queries are not relaxed.
|
||||
relaxer = (
|
||||
search_repository._relaxed_tsquery_text
|
||||
if is_postgres_backend(search_repository)
|
||||
else search_repository._relaxed_fts_text
|
||||
)
|
||||
assert relaxer("alpha AND beta") is None
|
||||
assert relaxer('"exact phrase"') is None
|
||||
assert relaxer("single") is None # < 3 tokens
|
||||
assert relaxer("New Feature") is None # < 3 tokens (link title)
|
||||
assert relaxer("root note 1") is None # numeric identifier token
|
||||
assert relaxer("SPEC 16 design") is None # numeric token
|
||||
assert relaxer(None) is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_multiword_query_relaxes_to_or_when_strict_misses(search_repository, search_entity):
|
||||
"""A question sharing only SOME words with a doc still surfaces it."""
|
||||
from basic_memory.repository.search_index_row import SearchIndexRow
|
||||
from basic_memory.schemas.search import SearchItemType
|
||||
|
||||
row = SearchIndexRow(
|
||||
project_id=search_repository.project_id,
|
||||
id=search_entity.id,
|
||||
type=SearchItemType.ENTITY.value,
|
||||
title="Trip plans",
|
||||
content_snippet="Melanie painted a sunrise over the lake last year.",
|
||||
content_stems="melanie painted a sunrise over the lake last year",
|
||||
permalink=search_entity.permalink,
|
||||
file_path=search_entity.file_path,
|
||||
entity_id=search_entity.id,
|
||||
metadata={"note_type": search_entity.note_type},
|
||||
created_at=search_entity.created_at,
|
||||
updated_at=search_entity.updated_at,
|
||||
)
|
||||
await search_repository.index_item(row)
|
||||
|
||||
# "hiking" is absent from the doc, so strict all-terms-AND misses on both
|
||||
# backends (Postgres's stopword stripping can't rescue it either).
|
||||
strict = await search_repository.search(search_text="Did Melanie go hiking at sunrise?")
|
||||
assert strict == []
|
||||
|
||||
# The hybrid FTS branch opts in; OR-relaxation surfaces the partial match.
|
||||
results = await search_repository.search(
|
||||
search_text="Did Melanie go hiking at sunrise?", allow_relaxed=True
|
||||
)
|
||||
assert any(r.entity_id == search_entity.id for r in results)
|
||||
|
||||
@@ -89,6 +89,7 @@ class _ConcreteRepo(SearchRepositoryBase):
|
||||
min_similarity: float | None = None,
|
||||
limit: int = 10,
|
||||
offset: int = 0,
|
||||
allow_relaxed: bool = False,
|
||||
) -> list[SearchIndexRow]:
|
||||
return []
|
||||
|
||||
@@ -809,182 +810,3 @@ async def test_sync_entity_vectors_batch_logs_resolved_fastembed_runtime_setting
|
||||
assert runtime_logs[0]["threads"] == 4
|
||||
assert runtime_logs[0]["configured_parallel"] == 2
|
||||
assert runtime_logs[0]["effective_parallel"] == 2
|
||||
|
||||
|
||||
# --- Entity-aware ranking boost (#951) ---
|
||||
|
||||
|
||||
def _make_index_row(
|
||||
*,
|
||||
row_id: int,
|
||||
title: str,
|
||||
row_type: str = SearchItemType.ENTITY.value,
|
||||
) -> SearchIndexRow:
|
||||
"""Build a real SearchIndexRow for entity-boost matching tests."""
|
||||
now = datetime(2026, 1, 1)
|
||||
return SearchIndexRow(
|
||||
project_id=1,
|
||||
id=row_id,
|
||||
type=row_type,
|
||||
file_path=f"notes/{row_id}.md",
|
||||
created_at=now,
|
||||
updated_at=now,
|
||||
permalink=f"notes/{row_id}",
|
||||
title=title,
|
||||
)
|
||||
|
||||
|
||||
class TestExtractQueryEntityTerms:
|
||||
"""Verify proper-noun extraction from query strings."""
|
||||
|
||||
def test_extracts_single_proper_noun(self):
|
||||
terms = SearchRepositoryBase._extract_query_entity_terms("What are Joanna's hobbies?")
|
||||
assert terms == {"joanna"}
|
||||
|
||||
def test_extracts_multiple_proper_nouns(self):
|
||||
terms = SearchRepositoryBase._extract_query_entity_terms(
|
||||
"What symbolic gifts do Deborah and Jolene have from their mothers?"
|
||||
)
|
||||
assert terms == {"deborah", "jolene"}
|
||||
|
||||
def test_who_is_anthony(self):
|
||||
terms = SearchRepositoryBase._extract_query_entity_terms("Who is Anthony?")
|
||||
assert terms == {"anthony"}
|
||||
|
||||
def test_all_lowercase_query_has_no_entity_terms(self):
|
||||
terms = SearchRepositoryBase._extract_query_entity_terms("what is the weather today")
|
||||
assert terms == set()
|
||||
|
||||
def test_capitalized_stopword_at_sentence_start_is_ignored(self):
|
||||
# "What" and "Who" are capitalized interrogatives, not entity names.
|
||||
terms = SearchRepositoryBase._extract_query_entity_terms("What does Sarah like?")
|
||||
assert terms == {"sarah"}
|
||||
|
||||
def test_all_caps_token_is_extracted(self):
|
||||
terms = SearchRepositoryBase._extract_query_entity_terms("who works at NASA")
|
||||
assert terms == {"nasa"}
|
||||
|
||||
def test_possessive_is_stripped(self):
|
||||
terms = SearchRepositoryBase._extract_query_entity_terms("Joanna's mother")
|
||||
assert terms == {"joanna"}
|
||||
|
||||
def test_single_capital_letter_is_ignored(self):
|
||||
# "A" lowercases to a stopword; "X" is a non-stopword single letter that
|
||||
# still carries no entity signal and must be dropped by the length guard.
|
||||
terms = SearchRepositoryBase._extract_query_entity_terms("A is for X and Apple")
|
||||
assert terms == {"apple"}
|
||||
|
||||
def test_empty_and_none_inputs(self):
|
||||
assert SearchRepositoryBase._extract_query_entity_terms("") == set()
|
||||
assert SearchRepositoryBase._extract_query_entity_terms(None) == set()
|
||||
|
||||
|
||||
class TestRowEntityMatchCount:
|
||||
"""Verify lexical match counting between query terms and row entity names."""
|
||||
|
||||
def test_title_match_counts_one(self):
|
||||
row = _make_index_row(row_id=1, title="Joanna's Hobbies")
|
||||
assert SearchRepositoryBase._row_entity_match_count(row, {"joanna"}) == 1
|
||||
|
||||
def test_relation_title_matches_both_endpoints(self):
|
||||
row = _make_index_row(
|
||||
row_id=2,
|
||||
title="Deborah -> Jolene",
|
||||
row_type=SearchItemType.RELATION.value,
|
||||
)
|
||||
count = SearchRepositoryBase._row_entity_match_count(row, {"deborah", "jolene"})
|
||||
assert count == 2
|
||||
|
||||
def test_no_match_returns_zero(self):
|
||||
row = _make_index_row(row_id=3, title="Anthony Profile")
|
||||
assert SearchRepositoryBase._row_entity_match_count(row, {"joanna"}) == 0
|
||||
|
||||
def test_match_is_case_insensitive(self):
|
||||
row = _make_index_row(row_id=4, title="JOANNA notes")
|
||||
assert SearchRepositoryBase._row_entity_match_count(row, {"joanna"}) == 1
|
||||
|
||||
def test_empty_terms_returns_zero(self):
|
||||
row = _make_index_row(row_id=5, title="Joanna")
|
||||
assert SearchRepositoryBase._row_entity_match_count(row, set()) == 0
|
||||
|
||||
def test_missing_title_returns_zero(self):
|
||||
row = _make_index_row(row_id=6, title="")
|
||||
row.title = None
|
||||
assert SearchRepositoryBase._row_entity_match_count(row, {"joanna"}) == 0
|
||||
|
||||
def test_distinct_terms_not_double_counted(self):
|
||||
# A title containing the same term twice still counts as one distinct match.
|
||||
row = _make_index_row(row_id=7, title="Joanna and Joanna")
|
||||
assert SearchRepositoryBase._row_entity_match_count(row, {"joanna"}) == 1
|
||||
|
||||
|
||||
class TestApplyEntityBoost:
|
||||
"""Verify the entity-boost score math and gating."""
|
||||
|
||||
def _repo(self, *, enabled: bool, weight: float = 0.15, max_terms: int = 3) -> _ConcreteRepo:
|
||||
repo = _ConcreteRepo()
|
||||
repo._entity_boost_enabled = enabled
|
||||
repo._entity_boost_weight = weight
|
||||
repo._entity_boost_max_terms = max_terms
|
||||
return repo
|
||||
|
||||
def test_disabled_returns_scores_unchanged(self):
|
||||
repo = self._repo(enabled=False)
|
||||
key = ("entity", 1)
|
||||
rows = {key: _make_index_row(row_id=1, title="Joanna")}
|
||||
scores = {key: 0.5}
|
||||
assert repo._apply_entity_boost(scores, rows, {"joanna"}) == {key: 0.5}
|
||||
|
||||
def test_matching_row_is_boosted(self):
|
||||
repo = self._repo(enabled=True, weight=0.2)
|
||||
key = ("entity", 1)
|
||||
rows = {key: _make_index_row(row_id=1, title="Joanna")}
|
||||
boosted = repo._apply_entity_boost({key: 0.5}, rows, {"joanna"})
|
||||
assert boosted[key] == pytest.approx(0.5 * 1.2)
|
||||
|
||||
def test_non_matching_row_unchanged(self):
|
||||
repo = self._repo(enabled=True, weight=0.2)
|
||||
key = ("entity", 1)
|
||||
rows = {key: _make_index_row(row_id=1, title="Anthony")}
|
||||
boosted = repo._apply_entity_boost({key: 0.5}, rows, {"joanna"})
|
||||
assert boosted[key] == pytest.approx(0.5)
|
||||
|
||||
def test_boost_can_reorder_lower_scored_match_above_higher_non_match(self):
|
||||
repo = self._repo(enabled=True, weight=0.5)
|
||||
generic_key = ("entity", 1)
|
||||
joanna_key = ("entity", 2)
|
||||
rows = {
|
||||
generic_key: _make_index_row(row_id=1, title="Generic topic about hobbies"),
|
||||
joanna_key: _make_index_row(row_id=2, title="Joanna"),
|
||||
}
|
||||
# The generic row starts higher (0.6) but does not match; "Joanna" (0.5) matches.
|
||||
boosted = repo._apply_entity_boost({generic_key: 0.6, joanna_key: 0.5}, rows, {"joanna"})
|
||||
assert boosted[joanna_key] > boosted[generic_key]
|
||||
|
||||
def test_multiple_matches_scale_boost(self):
|
||||
repo = self._repo(enabled=True, weight=0.1, max_terms=3)
|
||||
key = ("relation", 1)
|
||||
rows = {key: _make_index_row(row_id=1, title="Deborah -> Jolene")}
|
||||
boosted = repo._apply_entity_boost({key: 1.0}, rows, {"deborah", "jolene"})
|
||||
# Two matched terms -> 1 + 0.1 * 2 = 1.2
|
||||
assert boosted[key] == pytest.approx(1.2)
|
||||
|
||||
def test_max_terms_caps_the_boost(self):
|
||||
repo = self._repo(enabled=True, weight=0.1, max_terms=1)
|
||||
key = ("relation", 1)
|
||||
rows = {key: _make_index_row(row_id=1, title="Deborah -> Jolene")}
|
||||
boosted = repo._apply_entity_boost({key: 1.0}, rows, {"deborah", "jolene"})
|
||||
# Capped at 1 term -> 1 + 0.1 * 1 = 1.1
|
||||
assert boosted[key] == pytest.approx(1.1)
|
||||
|
||||
def test_zero_weight_is_noop(self):
|
||||
repo = self._repo(enabled=True, weight=0.0)
|
||||
key = ("entity", 1)
|
||||
rows = {key: _make_index_row(row_id=1, title="Joanna")}
|
||||
assert repo._apply_entity_boost({key: 0.5}, rows, {"joanna"}) == {key: 0.5}
|
||||
|
||||
def test_empty_entity_terms_is_noop(self):
|
||||
repo = self._repo(enabled=True, weight=0.2)
|
||||
key = ("entity", 1)
|
||||
rows = {key: _make_index_row(row_id=1, title="Joanna")}
|
||||
assert repo._apply_entity_boost({key: 0.5}, rows, set()) == {key: 0.5}
|
||||
|
||||
@@ -62,6 +62,7 @@ class ConcreteSearchRepo(SearchRepositoryBase):
|
||||
min_similarity: float | None = None,
|
||||
limit: int = 10,
|
||||
offset: int = 0,
|
||||
allow_relaxed: bool = False,
|
||||
) -> list[SearchIndexRow]:
|
||||
return [] # pragma: no cover
|
||||
|
||||
|
||||
@@ -66,6 +66,7 @@ class ConcreteSearchRepo(SearchRepositoryBase):
|
||||
min_similarity: Optional[float] = None,
|
||||
limit: int = 10,
|
||||
offset: int = 0,
|
||||
allow_relaxed: bool = False,
|
||||
) -> list[SearchIndexRow]:
|
||||
return [] # pragma: no cover
|
||||
|
||||
|
||||
@@ -222,6 +222,57 @@ async def test_initialize_file_sync_no_constraint_when_env_unset(
|
||||
assert _FakeWatchService.last_kwargs.get("constrained_project") is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_initialize_file_sync_skips_project_with_non_absolute_path(
|
||||
app_config: BasicMemoryConfig, config_manager, config_home, monkeypatch
|
||||
):
|
||||
"""Projects without an absolute local path are excluded from background sync (issue #949).
|
||||
|
||||
A config entry of ``{"path": ""}`` defaults to LOCAL mode and is not
|
||||
recognized as cloud, yet Path("") resolves to the process cwd. Syncing it
|
||||
would inject frontmatter into unrelated files, so it must be skipped.
|
||||
"""
|
||||
await db.shutdown_db()
|
||||
try:
|
||||
from basic_memory.config import ProjectEntry
|
||||
|
||||
good = config_home / "good"
|
||||
good.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
updated = app_config.model_copy(
|
||||
update={
|
||||
"projects": {
|
||||
"good": ProjectEntry(path=str(good)),
|
||||
# No mode -> defaults to LOCAL, empty (cwd-relative) path.
|
||||
"empty-path": ProjectEntry(path=""),
|
||||
},
|
||||
"default_project": "good",
|
||||
}
|
||||
)
|
||||
config_manager.save_config(updated)
|
||||
|
||||
await initialize_database(updated)
|
||||
await reconcile_projects_with_config(updated)
|
||||
|
||||
_disable_test_env_short_circuit(monkeypatch)
|
||||
monkeypatch.setattr("basic_memory.sync.WatchService", _FakeWatchService)
|
||||
|
||||
infos: list[str] = []
|
||||
monkeypatch.setattr(
|
||||
"basic_memory.services.initialization.logger.info",
|
||||
lambda message, *args, **kwargs: infos.append(message),
|
||||
)
|
||||
|
||||
await initialize_file_sync(updated, quiet=True)
|
||||
|
||||
skip_logs = [m for m in infos if "not locally syncable" in m]
|
||||
assert skip_logs, "expected a skip log for the empty-path project"
|
||||
assert "empty-path" in skip_logs[0]
|
||||
assert "good" not in skip_logs[0]
|
||||
finally:
|
||||
await db.shutdown_db()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_initialize_app_no_precedence_warning_when_not_conflicting(
|
||||
app_config: BasicMemoryConfig, monkeypatch
|
||||
|
||||
@@ -71,15 +71,18 @@ async def _register_local_projects(
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_select_projects_to_watch_returns_all_when_unconstrained(
|
||||
app_config: BasicMemoryConfig, project_repository
|
||||
app_config: BasicMemoryConfig, project_repository, tmp_path
|
||||
):
|
||||
"""Without a --project constraint, every active project is watched."""
|
||||
# Use tmp_path so the project paths are OS-absolute on Windows too — a
|
||||
# POSIX-style "/tmp/alpha" is not absolute on Windows (no drive letter),
|
||||
# and _select_projects_to_watch now skips non-absolute paths (issue #949).
|
||||
await _register_local_projects(
|
||||
app_config,
|
||||
project_repository,
|
||||
[
|
||||
{"name": "project-alpha", "path": "/tmp/alpha"},
|
||||
{"name": "project-beta", "path": "/tmp/beta"},
|
||||
{"name": "project-alpha", "path": str(tmp_path / "alpha")},
|
||||
{"name": "project-beta", "path": str(tmp_path / "beta")},
|
||||
],
|
||||
)
|
||||
|
||||
@@ -94,7 +97,7 @@ async def test_select_projects_to_watch_returns_all_when_unconstrained(
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_select_projects_to_watch_filters_to_constrained_project(
|
||||
app_config: BasicMemoryConfig, project_repository
|
||||
app_config: BasicMemoryConfig, project_repository, tmp_path
|
||||
):
|
||||
"""With ``constrained_project`` set, only that project is returned.
|
||||
|
||||
@@ -106,8 +109,8 @@ async def test_select_projects_to_watch_filters_to_constrained_project(
|
||||
app_config,
|
||||
project_repository,
|
||||
[
|
||||
{"name": "project-alpha", "path": "/tmp/alpha"},
|
||||
{"name": "project-beta", "path": "/tmp/beta"},
|
||||
{"name": "project-alpha", "path": str(tmp_path / "alpha")},
|
||||
{"name": "project-beta", "path": str(tmp_path / "beta")},
|
||||
],
|
||||
)
|
||||
|
||||
@@ -124,13 +127,13 @@ async def test_select_projects_to_watch_filters_to_constrained_project(
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_select_projects_to_watch_empty_when_constrained_project_missing(
|
||||
app_config: BasicMemoryConfig, project_repository
|
||||
app_config: BasicMemoryConfig, project_repository, tmp_path
|
||||
):
|
||||
"""An unknown constraint yields an empty watch set rather than watching everything."""
|
||||
await _register_local_projects(
|
||||
app_config,
|
||||
project_repository,
|
||||
[{"name": "project-alpha", "path": "/tmp/alpha"}],
|
||||
[{"name": "project-alpha", "path": str(tmp_path / "alpha")}],
|
||||
)
|
||||
|
||||
service = WatchService(
|
||||
|
||||
@@ -236,6 +236,90 @@ async def test_run_keeps_cloud_projects_with_local_bisync(monkeypatch, tmp_path)
|
||||
assert seen_project_names == [["local-project", "cloud-bisync"]]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_filters_empty_path_local_mode_project(monkeypatch, tmp_path):
|
||||
"""A project with an empty path is skipped even when mode is LOCAL (issue #949).
|
||||
|
||||
ProjectEntry.mode defaults to LOCAL, so a hand-edited config entry of
|
||||
``{"path": ""}`` is not recognized as cloud. The watch cycle must still skip
|
||||
it: Path("") resolves to the process cwd, and watching that would mutate
|
||||
whatever directory the server was launched from.
|
||||
"""
|
||||
config = BasicMemoryConfig(
|
||||
watch_project_reload_interval=1,
|
||||
projects={
|
||||
"local-project": {"path": str(tmp_path / "local"), "mode": "local"},
|
||||
# No explicit mode -> defaults to LOCAL, with an empty (cwd-relative) path.
|
||||
"empty-path": {"path": ""},
|
||||
},
|
||||
)
|
||||
repo = _Repo(
|
||||
projects_return=[
|
||||
Project(id=1, name="local-project", path=str(tmp_path / "local"), permalink="local"),
|
||||
Project(id=2, name="empty-path", path="", permalink="empty-path"),
|
||||
]
|
||||
)
|
||||
watch_service = _watch_service(config, repo)
|
||||
|
||||
seen_project_names: list[list[str]] = []
|
||||
|
||||
async def watch_cycle_stub(projects, stop_event):
|
||||
seen_project_names.append([p.name for p in projects])
|
||||
watch_service.state.running = False
|
||||
stop_event.set()
|
||||
|
||||
async def fake_write_status():
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(watch_service, "_watch_projects_cycle", watch_cycle_stub)
|
||||
monkeypatch.setattr(watch_service, "write_status", fake_write_status)
|
||||
|
||||
await watch_service.run()
|
||||
|
||||
assert seen_project_names == [["local-project"]]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_filters_orphan_db_project_absent_from_config(monkeypatch, tmp_path):
|
||||
"""A DB row not present in config is skipped even with an absolute path.
|
||||
|
||||
Config is the source of truth. Reconciliation normally deletes orphan rows,
|
||||
but if it is skipped or fails a stale row could remain; watching it would
|
||||
mutate a directory the user already removed from config.
|
||||
"""
|
||||
config = BasicMemoryConfig(
|
||||
watch_project_reload_interval=1,
|
||||
projects={
|
||||
"local-project": {"path": str(tmp_path / "local"), "mode": "local"},
|
||||
},
|
||||
)
|
||||
repo = _Repo(
|
||||
projects_return=[
|
||||
Project(id=1, name="local-project", path=str(tmp_path / "local"), permalink="local"),
|
||||
# Absolute path, but no matching entry in config -> stale/orphan row.
|
||||
Project(id=2, name="orphan", path=str(tmp_path / "orphan"), permalink="orphan"),
|
||||
]
|
||||
)
|
||||
watch_service = _watch_service(config, repo)
|
||||
|
||||
seen_project_names: list[list[str]] = []
|
||||
|
||||
async def watch_cycle_stub(projects, stop_event):
|
||||
seen_project_names.append([p.name for p in projects])
|
||||
watch_service.state.running = False
|
||||
stop_event.set()
|
||||
|
||||
async def fake_write_status():
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(watch_service, "_watch_projects_cycle", watch_cycle_stub)
|
||||
monkeypatch.setattr(watch_service, "write_status", fake_write_status)
|
||||
|
||||
await watch_service.run()
|
||||
|
||||
assert seen_project_names == [["local-project"]]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_continues_after_cycle_error(monkeypatch, tmp_path):
|
||||
config = BasicMemoryConfig(
|
||||
|
||||
@@ -1245,6 +1245,30 @@ class TestProjectMode:
|
||||
assert config.projects["research"].mode == ProjectMode.LOCAL
|
||||
assert config.get_project_mode("research") == ProjectMode.LOCAL
|
||||
|
||||
def test_is_locally_syncable_true_for_config_project_with_absolute_path(self, tmp_path):
|
||||
"""A project in config with an absolute path is locally syncable."""
|
||||
abs_path = str(tmp_path / "research")
|
||||
config = BasicMemoryConfig(projects={"research": ProjectEntry(path=abs_path)})
|
||||
assert config.is_locally_syncable("research", abs_path) is True
|
||||
|
||||
def test_is_locally_syncable_false_for_empty_path(self):
|
||||
"""An empty path resolves to cwd, so it is never locally syncable (#949)."""
|
||||
config = BasicMemoryConfig(projects={"empty": ProjectEntry(path="")})
|
||||
assert config.is_locally_syncable("empty", "") is False
|
||||
|
||||
def test_is_locally_syncable_false_for_relative_path(self):
|
||||
"""A relative (slug) path, as used by cloud-only projects, is not syncable."""
|
||||
config = BasicMemoryConfig(projects={"cloud": ProjectEntry(path="cloud-slug")})
|
||||
assert config.is_locally_syncable("cloud", "cloud-slug") is False
|
||||
|
||||
def test_is_locally_syncable_false_for_orphan_not_in_config(self, tmp_path):
|
||||
"""A DB row absent from config is not syncable even with an absolute path.
|
||||
|
||||
Config is the source of truth; stale rows must not be synced (#949).
|
||||
"""
|
||||
config = BasicMemoryConfig(projects={})
|
||||
assert config.is_locally_syncable("orphan", str(tmp_path / "orphan")) is False
|
||||
|
||||
def test_cloud_api_key_defaults_to_none(self):
|
||||
"""Test that cloud_api_key defaults to None."""
|
||||
config = BasicMemoryConfig()
|
||||
|
||||
Reference in New Issue
Block a user